Ingest

HiIn an effort to resolve intermittent ODX errors whilst retrieving data back from Oracle JDE tables I am trying to understand whether reducing the ‘concurrent execution threads’ setting in the Oracle data source connections could be a potential solution.I have multiple Oracle connections set up in TX, with each connection pulling a subset of tables (e.g. all the JDE master file tables in one connection, all the transaction-related GL tables in another etc). All these connection transfer tasks are then wrapped up in a TX job scheduled to run each evening.The ‘concurrent execution threads’ default value across all the Oracle data connections I had set up for this was ‘8’.My thinking was that if TimeXtender is running multiple data pulls simultaneously then this could be causing resource contention issues on the source Oracle server, and thereby intermittently triggering data pull failures with generic error messages that don’t precisely indicate root cause.I have since changed the ‘conc

I just upgraded TimeXtender to the latest versions for a client. (7017 and 7047).In doing so most of the datasource stopped working unless I upgraded the datasources to the latest versions. I decided that I should just do them all to be consistent. This has broken my CData Rest sources because the new drivers are not installed! This is a massive issue, and i was not warned in any significant way about this particular issue in the upgrade.I need to get the new CData drivers, I am not in a position where I can transition multiple CData Rest sources as part of an upgrade! This is a production issue and I need to resolve it ASAP!

HiOver the weekend one on our ODX jobs failed with the following error; Executing table [PRODDTA].[F0101]:failed with error:Oracle.ManagedDataAccess.Client.OracleException (0x80004005): ORA-50000: Connection request timed out   at OracleInternal.ConnectionPool.PoolManager`3.Get(ConnectionString csWithDiffOrNewPassw, Boolean bGetForApp, OracleConnection connRefForCriteria, String affinityInstanceName, Boolean bForceMatch, Boolean bAsync)   at OracleInternal.ConnectionPool.OraclePoolManager.Get(ConnectionString csWithNewPassw, Boolean bGetForApp, OracleConnection connRefForCriteria, String affinityInstanceName, Boolean bForceMatch, Boolean bAsync)   at OracleInternal.ConnectionPool.OracleConnectionDispenser`3.Get(ConnectionString cs, PM conPM, ConnectionString pmCS, OracleOpaqueString securedPassw, OracleOpaqueString securedProxyPassw, OracleConnection connRefForCriteria, Boolean bAsync)   at Oracle.ManagedDataAccess.Client.OracleConnectionInternal.OpenInternal(Boolean bAsync)   at Oracl

Good Day,We have a client that is on V20 that would like to incorporate some Sensitive Data within their TX environment that some of the TX Developers should not have access to. As far as I am aware, if you have access to the TX Repository then the developers can open all projects.How would I handle the solution of the Sensitive Data in TX?Kerry

Does anyone know exactly what happens when we ingest data from e.g. a SQL Server into Fabric Lakehouse? It seems concurrency impacts performance (more can be less) but there might also be Fabric assets in play that could act like bottlenecks, and which requires “capacity” management.Any best recommendations/practices in this space?

I am loading a table incrementally into my ODX storage in an Azure Data Lake, where new Parquet files are added daily. This approach is because the source only holds data every two weeks, and I want to maintain a log in the ODX. The Parquet storage is very compact.However, for downstream analysis, I only need to retrieve data from the last 1 to 2 days into my prepare instance. I am using a data selection rule on the mapping, and I have also tried applying it directly on the table. Both approaches take a very long time to complete (+1 hour), whereas running the same query on the source SQL database filtering for 2 days of data completes in about 10 seconds.I suspect that the prepare instance is scanning through all the Parquet files, including older days, causing the slow performance.My question:Is there a way to configure the TX prepare instance to only process the most recent X Parquet files (e.g., the last 2 days) instead of scanning all files? This would significantly improve the se

Version 7026.1 - Ingest into LakehouseI’ve got a SQL data source where some column names contain spaces. This causes the TX SQL connector (23.0.3.0) to throw the following error when I try to ingest data into a Lakehouse:Executing table dbo_tx_profitcentreinfo:failed with error:Found invalid character(s) among ' ,;{}()\n\t=' in the column names of your schema. Please enable Column Mapping on your Delta table with mapping mode 'name'.You can use one of the following commands.If your table is already on the required protocol version:ALTER TABLE table_name SET TBLPROPERTIES ('delta.columnMapping.mode' = 'name')If your table is not on the required protocol version and requires a protocol upgrade:ALTER TABLE table_name SET TBLPROPERTIES (   'delta.columnMapping.mode' = 'name',   'delta.minReaderVersion' = '2',   'delta.minWriterVersion' = '5')I can read what it says 😀 but I thought I’d ask the Community for advice. Has anyone encountered and resolved this issue? If so, what do you recommen

Hi,TX ver: latest I am getting an error when import metadata or doing table flattening. This worked fine earlier, before 3 june. Works fine using Postman. Any suggestion what I am missing? “One or more errors occurred.Failed to execute request for endpoint 'Get Charge Points'.One or more errors occurred.Cannot send a content-body with this verb-type.System.AggregateException: One or more errors occurred. ---> System.Net.ProtocolViolationException: Cannot send a content-body with this verb-type.   at System.Net.HttpWebRequest.CheckProtocol(Boolean onRequestStream)   at System.Net.HttpWebRequest.BeginGetRequestStream(AsyncCallback callback, Object state)” /Bjørn A.

I am fetching refresh logs from power bi using the following endpoint: https://api.powerbi.com/v1.0/myorg/groups/{groupid}/datasets/{id}/refreshes(https://learn.microsoft.com/en-us/rest/api/power-bi/datasets/get-refresh-history-in-group#refresh)In my use case, the groupid is hard coded and I iterate over the datasetid {id} using dynamic values FromEndpointTable.The setup for this is the same as in this guide: My problem is that the dataset id is not included in the payload for the refresh log endpoint, so I have no way of knowing which power bi dataset has been refreshed.Could I somehow include the dynamic endpoint as a field in the result set? Or is there any other way to solve this?Best regards,Pontus B

Suddenly all the excel sheets are not loading anymore however some of them no one touched 6 months ago.the error is:Executing table excel_sheet:failed with error:Exception Type: System.ExceptionMessage: Invalid column count             Stack Trace: at DataStorageEngine.DataLakeGen2.DataLakeGen2Transfer.<>c__DisplayClass41_5.<UploadData>b__14(DataTable dataTable, Partition partition)                          at DataStorageEngine.DataLakeGen2.DataLakeGen2Transfer.UploadData(ADLSGen2APIManager api, DataSourceModel dataSourceModel, DataLakeGen2ExecutionMethod executionMethod, Action`2 writeExecutionLog)                          at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.<>c__DisplayClass35_2.<TransferDataAsync>b__7(DataLakeGen2ExecutionMethod executionMethod)                          at DataSourceEngine.Custom.CustomSourceEngine.InvokeDataSourceExecution(Action`1 action)                          at DataSourceEngine.Custom.CustomSourceEngine.RunExecu

Dear Sir,Our client is running TimeXtender 6963.1.So want a rowcount, count(*) on our tables loaded in de ODX.Our ODX is configured with ADSL (Azure Datalake Gen2 Storage).So we want to use the ODX TimeXtender Parquet Data Source connector (23.1.1.0 64bit).Is it possible to run a count(*) query to our tables when we setup this connection to our ADSL?We need to know how many records are loaded in all of our tables.We have a lot of incremental tables, so we worry of all the increment files in the lake. Do we need to configure some sort of a aggregation ?Is there documentation available how to configure this ? The connector has a lot of options, we don’t know all the details.Hope somebody has some experience with this Parquet connector.Thank you in advance for looking into this.Regards,Arthur

Hi,We've updatet our Data Integration version from 6766.1 to 6963.1. Which means we had to update our outdated providers. So we followed the steps mentioned in this post Change data source provider | Community (timextender.com).This results in an error when we try to full load changed data source. It keeps giving is the error: “The schema …. already exists on the SQL storage and cannot be used” Which actually means we can’t update our providers/ data sources. We have to create new ones en have to map all our tables in de odx. Anyone else facing this problem and have a solution for this?

I'm currently loading Parquet "tables" in TX Classic with the following setup: Using Synapse Serverless to read Parquet files and expose them as external tables Connecting via SQL Server authentication This works, but I’m looking for a better approach.Is there a way to read Parquet "tables" directly in TX Classic? The Enhanced Data Source seems limited to reading or aggregating files, not treating them as tables.Alternatively, could I replace Synapse Serverless with Fabric Serverless? The main blocker is authentication—TX doesn't appear to support SQL providers with service principal (OAuth) authentication, only SQL Server authentication, which we’d prefer to avoid.Any suggestions, recommendations or workarounds would be appreciated. 

Hi! We are currently setting up TimeXtender with Snowflake and have a bit of an issue when it comes TimeXtender not having the capability of managing the ODX in Snowflake (internal stage).  As you know an external stage is needed via Azure Data Lake Gen2.  The issue for us is that we would like to use private endpoint when creating this data lake in Azure but that requires business critical subscription to Snowflake which is 30 percent more expensive.   Are there any plans/timeline for when TimeXtender will be capable of handling the Ingest instance in Snowflake and if so could you please give a rough estimate on when that might be available?  We need to make some decisions on our end on how to proceed and would like to know the answer before we move forward.

As title suggests, I wonder if anyone in this community has experience with ingesting TX with the data source Adnamics and their API. Thanks in advance!

Hey, I use the REST connector to consume a CSV file. In my query parameters I got one for auth key. So when I try to load the file a get an error and the reason is that parameter get encoded (% → %25) .  How can I avoid encoding in the query? /Bjørn A.  

​Hi,I’m sorry for this somewhat lengthy post but I have a problem that I don’t really know how to solve and I’m just looking to see if anyone have run into the same issue, or maybe already solved it.We are going to fetch data from a time management system that have a SOAP API and all the information I find in the community refers to RSD files, that is not to be used anymore. I don’t know how I would solve the problem that I have even with an RSD file, but that’s somewhat beside the point here. 😊The issue is that we must take the request XML, encrypt the whole thing using AES, into a fingerprint that in turn will need to be inserted into the actual request that is then sent to the API.I have received some code in C# that does the following steps:Generates a timestamp and an MD5 hash of the request XML Generates the fingerprint XML using the timestamp and the newly created MD5 hash with the schema like in the attached picture fingerprint_schema Encrypts the fingerprint XML with AES, usi

I am trying to replace the CDATA Dynamics365 adapter with the TX REST connector (8.0.0) and I am having some issues where empty values from the API is interpreted as empty strings rather than NULL and as a result creates tables with incorrect datatypesHow do i set up a way to replace these empty strings with NULL values and keep the correct data types as wellI have tried with various combination of “Search and replace" on the endpoint as well as the individual nodes in the table builder without any luck.I also have the “Empty fields as null” option enabled.  

Dear sir, Our client wants to upgrade to the latest and the greatest of TimeXtender.Right now they are on TX version 6766.1.So they have to migrate their CData datasources to the new TX datasources.Question: Is there a ‘datasource migration-tool’ to help them with that ?Or do they have to setup/configure the new TimeXtender datasources manually ?Hope you can shine a light on this.Regards,Arthur

Hi,I'm using the TimeXtender CSV Data Source provider and was wondering if it's possible to combine this provider with either incremental load settings or history settings. I’ve tried enabling incremental load—both in the ODX and by setting the table as incremental in the first data area—but I receive the error: "The provider does not support incremental load."Interestingly, I don't get the same error when I set the table as a history table instead, but when I try to execute the table, the result is empty.I'm wondering if this could be due to incorrect natural key configuration from my side, or if the CSV provider simply doesn't support historical loading.Any clarification would be appreciated. Thank you!  

The Get Schema function is not working for the REST endpoints / table flattening. I am trying to use version: 8.0.0.0. I get the following message: Any ideas what is happening?

I’ve an Azure storage account with a blob Container that has folders and files. How can I configure a data source to use it?  

I use the CSV provider to load a folder with multiple CSV files. I know one of the files has 17471 rows. Only the ingest task only loads the first 14521 rows. I don't have any filtering in the ingest task.Does anyone know how I can load the full file?

I am extracting data from the MS Graph API using the REST connector in TimeXtender. The issue I am facing is that I can easily retrieve the members of the groups but unfortunately, the GroupId is not included in the Query and I do not see any way to include the {id} parameter into the saved results.If anyone has solved this, can you please help with some information on how to solve this?

Have anyone an example on a successful connector regarding a normal Active Directory - not Azure AD.Can’t find any information about it but I guess many have done a connection using TX for that. RegardsAnders Bengtsson