Ingest

Hello,I have a question about the RAW tables in TimeXtender. I noticed that the data types of the fields in the RAW table change when I modify them in the tool. I was under the impression that RAW tables should keep the original data types from the source, since this would be “RAW”.I ran into trouble when loading transferring from the ODX. I changed the datatype of a field and even added a transformation that resulted in a CAST in the transformation view. However, I got an error message saying: The given value of type String from the data source cannot be converted to type float of the specified target column.This should not be, because the source contains only data that is/should be convertible to a float. The Raw table has no data, which I find strange. Changing the fields data type to float has also changed the datatype of the Raw table. I would expect the Raw table to always have the same datatype as the source and that the changing of datatypes is h

I have an issue, and i have not been able to identify if this is a TX issue or a DWH issue.This is how TX is set up:I have a MS PowerApp that writes to the APP database.I can see that the records are in the DWH, like in the table below.  I have confirmed that the entetried are in APP-stack.At 07:30 in the morning, execution package runs, then this happens:Meaning, the DW_ID that were previously Bob’s and Marlen’s are now overwritten by Kim and Tom.There are no traced of DW_ID 101 and 103 being registrered by Bob and Marlen

In my TimeXtender Data Integration (Ingest) instance, data sources seem to be listed alphabetically. I’d like to set a custom order instead. I tried drag-and-drop, but it didn’t work, and I don’t see any option in the portal to reorder them.Is there a way to change the display order (e.g. sort settings, or another workaround)?If this isn’t supported, a confirmation would also be helpful. Thanks! 😊 

Need help with TimeXtender data transformation issue. We're pulling sales data from SQL Server but getting errors during the transform step. The source has decimal values but TimeXtender seems to be reading them as text.Also need to set up a simple incremental load based on ModifiedDate field. Should be 2-3 hours remote work to fix the data types and configure the incremental logic.Need working by Tuesday for our weekly data refresh.Please Contact Me Asap.

Hi, When trying to replicate the use of “Importing OpenAPI” from this TimeXtender Tuesdays video i get bellow Time Out error:  I am wondering if i am doing something wrong, or one could replicate this issue aswell.Regards,Devin

Hi,TDI ver 7026.1TimeXtender Ingest Service Configuration 7017.1 We are using Azure Datalake Gen2 Storage Container as our Ingest instance. I have set a load job (incr_load) from a sql server database to the ingest instance.This works fine during the day. When get into the next day the same job does a full load on my tables in the database. Why is this happening?  I really need to solve this asap :-) regards,Bjørn A. 

We are using the provider Azure Data Factory - SQL Server for a lof connections where the instances for these connections contains almost the same data but each time another customer. What we have noticed when the incremental load is setup to look at a datetime field called DW_MODIFIED_date. it will never load that 1 record which is in the table. When I exclude that table from the incremental load then I have 1 record loaded.When I include the table again in the incremental load it is removing the first folder from the previous load and don’t write a new file because it was not updated.When I remove the folder from the container and I incremental load that client table again I get no record at al while it should be the first load. Even if the incremental load is on and there is no file  then I expect to have in the initial load the data.

We are unable to open our instance as we are prompted with the following error message: TX Version 7047.1 

Hi there! We have had a small problem recently whereas Table A is on the staging/DSA layer and contains the following columns: Column 1 (PK)Column 2 (PK)Column 3Column 4 … etcNow this table has three sources - three Business Central instances. In two of the sources the original order of columns is Column 1, Column 2 but in the third source the order is Column 2, Column 1.We get the following error when incremental load is enabled:  "TIMEXTENDER COPY DATA LOOP", "Failure happened on 'Sink' side, Column 2 contains an invalid value. We have been able to manage this error previously by ordering the columns in the same way as in the source.However, now the sources have different orders of the columns and therefore not possible to use that way to get around this. Any ideas or suggestions? 

Hello, TimeXtender 20.10.67 with ODX. We have a source (azure datalake) that will geta json file dumped with the following format table.name/”date-when-file-dumped”. The files will be15million + rows and dumped atleast once a week.What I want to achieve is for the connector to only extract the latest file and have it dynamically change. I can’t figure out a way to effectively do this in the connector, as the uri connection string wants to point to a specific filename or take everything in a concat.I can’t figure out a way to change the URI dynamically and the only option I can think of in theory is concatenating all the json in the connector, including resourcecolumns and doing a custom query table on the connector with a DSR rule based on the MAX filedate.However, in a year or two when there will be 50+ files (750 million rows in theory with concat), I am not sure how the performance or scalability of that solution will be

Hi all,Since about a week ago, we’re seeing random but frequent (almost daily) failures in ODX extract tasks from two sources: TimeXtender SQL Data Source (version 22.0.2.1) (SQL database) – runs from 05:00 to ~05:45 (45 min) UTC. Error: Cannot access a closed Stream (HttpRequestException). TimeXtender SQL Data Source (version 22.0.2.1) (Synapse Severless SQL pool)  – runs from 05:00 to ~05:00:30 (30 sec) UTC. Error: System.TimeoutException: The operation has timed out. Manual retries later in the morning succeed without issue.The Synapse serverless SQL pool is pre-triggered ~15 minutes in advance to avoid cold start, and both the ODX environment and the data sources have generous timeout settings.We use Azure Data Lake Gen2 storage as our ODX storage.Any idea what might be causing these recent failures?Thanks,sva

Hi,I need to add authorization basic auth (username, password) to every endpoint in my rest api(Tripletex).I have added dynamic values from the endpoint “TokenSessionCreate Create” creating a session token.This token is my password and using a fixed username to every endpoint.In my endpoint “LedgerPosting” am using the section “Override Headers” to add the token to the “Authorization” and the value equal “Basic user:pwd”. Well, I cant get this to work.I proberly nedd to convert usr:pwd to base64 encoding….Does anyone have any suggestion?best regards,Bjørn A.    

Hi,i have a gen2 storage account with a blob container with multiple parquet files. when i do a sync with odx it’s succesfull, the transfer gives me for 4 files the message “ is used by another table”. all other files in the same map go ok. have tried to recreate the blob container, and recreate the parquet files again, same results.Exact message:The table folder 'parquet_taak_status' is used by another table with the id 9a510177-a130-4bcc-aaaa-5abefd42c1b9 how to solve / find the other table that is referenced to? thanks!  

I recently updated the connection provider version on Hubspot, also for ODX/TX, and I have a problem getting the custom objects in. I checked, they still exist, postman can access them without issues, but I get errors when trying to get them to TimeXtender.  Error messages as below:   

Hello,We are looking to implement incremental data fetching from REST API endpoints, specifically by dynamically filtering with an updatedAt parameter. The goal is to use the maximum updatedAt value previously fetched from each API call, which is stored in our ODX tables.While I'm aware of the capability to use dynamic parameters based on endpoints and queries within TimeXtender, I'm struggling to find a method to reference the maximum updatedAt value already loaded in the ODX tables for this purpose.Has anyone implemented a similar dynamic data fetching mechanism in TimeXtender? If so, could you share how you managed to reference data from the ODX tables to set the parameters for the next API call?Thanks!

HiIn an effort to resolve intermittent ODX errors whilst retrieving data back from Oracle JDE tables I am trying to understand whether reducing the ‘concurrent execution threads’ setting in the Oracle data source connections could be a potential solution.I have multiple Oracle connections set up in TX, with each connection pulling a subset of tables (e.g. all the JDE master file tables in one connection, all the transaction-related GL tables in another etc). All these connection transfer tasks are then wrapped up in a TX job scheduled to run each evening.The ‘concurrent execution threads’ default value across all the Oracle data connections I had set up for this was ‘8’.My thinking was that if TimeXtender is running multiple data pulls simultaneously then this could be causing resource contention issues on the source Oracle server, and thereby intermittently triggering data pull failures with generic error messages that don’t precisely indicate root cause

I just upgraded TimeXtender to the latest versions for a client. (7017 and 7047).In doing so most of the datasource stopped working unless I upgraded the datasources to the latest versions. I decided that I should just do them all to be consistent. This has broken my CData Rest sources because the new drivers are not installed! This is a massive issue, and i was not warned in any significant way about this particular issue in the upgrade.I need to get the new CData drivers, I am not in a position where I can transition multiple CData Rest sources as part of an upgrade! This is a production issue and I need to resolve it ASAP!

HiOver the weekend one on our ODX jobs failed with the following error; Executing table [PRODDTA].[F0101]:failed with error:Oracle.ManagedDataAccess.Client.OracleException (0x80004005): ORA-50000: Connection request timed out   at OracleInternal.ConnectionPool.PoolManager`3.Get(ConnectionString csWithDiffOrNewPassw, Boolean bGetForApp, OracleConnection connRefForCriteria, String affinityInstanceName, Boolean bForceMatch, Boolean bAsync)   at OracleInternal.ConnectionPool.OraclePoolManager.Get(ConnectionString csWithNewPassw, Boolean bGetForApp, OracleConnection connRefForCriteria, String affinityInstanceName, Boolean bForceMatch, Boolean bAsync)   at OracleInternal.ConnectionPool.OracleConnectionDispenser`3.Get(ConnectionString cs, PM conPM, ConnectionString pmCS, OracleOpaqueString securedPassw, OracleOpaqueString securedProxyPassw, OracleConnection connRefForCriteria, Boolean bAsync)   at Oracle.ManagedDataAccess.Client.Orac

Good Day,We have a client that is on V20 that would like to incorporate some Sensitive Data within their TX environment that some of the TX Developers should not have access to. As far as I am aware, if you have access to the TX Repository then the developers can open all projects.How would I handle the solution of the Sensitive Data in TX?Kerry

Does anyone know exactly what happens when we ingest data from e.g. a SQL Server into Fabric Lakehouse? It seems concurrency impacts performance (more can be less) but there might also be Fabric assets in play that could act like bottlenecks, and which requires “capacity” management.Any best recommendations/practices in this space?

I am loading a table incrementally into my ODX storage in an Azure Data Lake, where new Parquet files are added daily. This approach is because the source only holds data every two weeks, and I want to maintain a log in the ODX. The Parquet storage is very compact.However, for downstream analysis, I only need to retrieve data from the last 1 to 2 days into my prepare instance. I am using a data selection rule on the mapping, and I have also tried applying it directly on the table. Both approaches take a very long time to complete (+1 hour), whereas running the same query on the source SQL database filtering for 2 days of data completes in about 10 seconds.I suspect that the prepare instance is scanning through all the Parquet files, including older days, causing the slow performance.My question:Is there a way to configure the TX prepare instance to only process the most recent X Parquet files (e.g., the last 2 days) instead of scanning all

Version 7026.1 - Ingest into LakehouseI’ve got a SQL data source where some column names contain spaces. This causes the TX SQL connector (23.0.3.0) to throw the following error when I try to ingest data into a Lakehouse:Executing table dbo_tx_profitcentreinfo:failed with error:Found invalid character(s) among ' ,;{}()\n\t=' in the column names of your schema. Please enable Column Mapping on your Delta table with mapping mode 'name'.You can use one of the following commands.If your table is already on the required protocol version:ALTER TABLE table_name SET TBLPROPERTIES ('delta.columnMapping.mode' = 'name')If your table is not on the required protocol version and requires a protocol upgrade:ALTER TABLE table_name SET TBLPROPERTIES (   'delta.columnMapping.mode' = 'name',   'delta.minReaderVersion' = '2',   'delta.minWriterVersion' = '5')I can read what it says 😀 but I

Hi,TX ver: latest I am getting an error when import metadata or doing table flattening. This worked fine earlier, before 3 june. Works fine using Postman. Any suggestion what I am missing? “One or more errors occurred.Failed to execute request for endpoint 'Get Charge Points'.One or more errors occurred.Cannot send a content-body with this verb-type.System.AggregateException: One or more errors occurred. ---> System.Net.ProtocolViolationException: Cannot send a content-body with this verb-type.   at System.Net.HttpWebRequest.CheckProtocol(Boolean onRequestStream)   at System.Net.HttpWebRequest.BeginGetRequestStream(AsyncCallback callback, Object state)” /Bjørn A.

I am fetching refresh logs from power bi using the following endpoint: https://api.powerbi.com/v1.0/myorg/groups/{groupid}/datasets/{id}/refreshes(https://learn.microsoft.com/en-us/rest/api/power-bi/datasets/get-refresh-history-in-group#refresh)In my use case, the groupid is hard coded and I iterate over the datasetid {id} using dynamic values FromEndpointTable.The setup for this is the same as in this guide: 

Suddenly all the excel sheets are not loading anymore however some of them no one touched 6 months ago.the error is:Executing table excel_sheet:failed with error:Exception Type: System.ExceptionMessage: Invalid column count             Stack Trace: at DataStorageEngine.DataLakeGen2.DataLakeGen2Transfer.<>c__DisplayClass41_5.<UploadData>b__14(DataTable dataTable, Partition partition)                          at DataStorageEngine.DataLakeGen2.DataLakeGen2Transfer.UploadData(ADLSGen2APIManager api, DataSourceModel dataSourceModel, DataLakeGen2ExecutionMethod executionMethod, Action`2 writeExecutionLog)                          at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.<>c__DisplayClass35_2.<TransferDataAsync>b__7(DataLakeGen2ExecutionMethod executionMethod)                          at DataSourceEngine.Custom.CustomSourceEngine.InvokeDataSourceExecution(Action`1 action)