Ingest

HelloEvery day, I have a new Excel file in the Azure storage or OneDrive account. I use OneDrive to test phase. These file’s name change with a new day. How can I this multiple Excel files to one? Is there a option for Excel like aggregate files option in CSV?Best Regards

We have a REST API that we use to connect to several endpoints. The default name of all the tables is: REST.Jsondata. This causes issues in our Azure Gen2 datalake. It gives an error saying the table name is already in use by a table with a different ID.I almost feel like there must be some really easy way to rename the table name before it lands in the Gen2 datalake but so far I haven’t been able to figure out how.The workaround I use now is to use a QueryTable but that is not the nicest way to do it I guess. 

Hello,My organisation recently had to redeploy our development virtual machine and reinstall timextender / odx on it. Since then I am not allowed to click on certain buttons inside the application, and options are missing when I right-click on certain objects.For example I am not allowed to click on the [Stop Task] button in the ODX Execution Queue window. And when I right-click on an object in the Manage ODX tab, all I can see it ‘View Properties’ and not any of the many other options that used to be there.I do have these functions available in our other environments. And my collegues have no issues on our dev machine.I have tried the following so far:Enable/disable Deprecated Features. Delete the hidden Default.rdp file and create a new rdp connection to the machine. Any tips?

Currently I’m using one ODX instance with this setup:  All the mapped data sources connections has selected the option “Enabled data on demand”:  I am using three instances of Data Warehouse: Development → Testing → Production.My question is:Can I create a new ODX instance connected to the same Storage container?The idea is that the same data sources in the DW instances for development and test use the "Enable data on demand" OFF option. In this way the storage container only will be refreshed one time (when we refresh the DWH production instance) because the data source is the same. 

Hello Guys. I tried to select table from ODX but it deosnt show none (except once query table).the transfer task work fine (it show succefull) but once I triy to select table in order to send them to the datawerhouse I show nothing.- even if it work fine when I use query tool. Any suggestion please? thanks@Thomas Lind   

I am loading a CSV (comma separated) file with the ODX. After upgrading the ODX to the latest version, transferring any table with this datasource gives the following error: Syncing does work.ODX version: 6521.1TX version: 6536.1CSV Provider version: tried both 22.0.8276.0 and 23.0.8749.0 but same error.

Hi,We have only source in which we are using a rest source and incrementally loading the data in the odx. The api cannot handle a date range that is too big. When we trying to preview the data in the odx, we see the data that we need. However, when we run the odx transfer task, with syncing it does not give all the updates in the table like it should. I saw that there were some fixes with the previous versions with incremental loading. We upgraded today to 6536.1. Also we upgraded the rest data source to the newest version. However, because the preview is fine, I don't think the problem is the source connector. 

I need some guidence saving contents ofan endpoint as BLOOB. I can see that the version we use doesnot have an option for BLOOB but may be Text or Binary with maximum length can give me what i need.So I have this RSD that i created <api:script xmlns:api="http://apiscript.com/ns?v1" xmlns:xs="http://www.w3.org/2001/XMLSchema"> <!-- See Column Definitions to specify column behavior and use XPaths to extract column values from JSON. --> <api:info title="GetAllStops" desc="Generated schema file." xmlns:other="http://apiscript.com/ns?v1"> <!-- You can modify the name, type, and column size here. --> <attr name="kortnavn" xs:type="string" readonly="false" other:xPath="/json/kortnavn" /> <attr name="tittel" xs:type="string" readonly="false" other:xPath="/json/tittel" /> <attr name="virksomhet" xs:type="string" readonly="false" other:xPath="/jso

We are running TX version 6536.1.Our data source is AX Dynamics 2012 on premise. We have the issue both with the SQL Server connector and the OLEDB connector.After synchronizing, we can't select any tables with right click on the data source → Select Tables → Search.It also takes > 30 minutes before the table selector pops up.Our only option is to select all tables.The Transfer task - Select table option does give us the option to select tables. However we can't select any individual columns. Neither does this give us the option to define any incremental selection rules or row filters.

Hello, TimeXtender: 20.10.40.64ODX: 20.10.31We are using the ODX and TimeXtenders SQL database connector to get data from a synpase database. Sometimes the synpase databas seems to have problems, where a query runs and doesn’t complete (a seperate issue we are working on).However what happens in TimeXtender in the ODX is that the transfer task keeps running indefinitely and never times out, leading timextender execution package to run but just sits there waiting for the odx transfer task to finish. However nothing fails anywhere, so we do not get an alert.We need some way to be able to capture this.Is there anyway to set up one or both of these things in TimeXtender or if you have any other ideas?1. Kill the transfer task if it takes more then X minutes2. Get an alert on failure or long running taskThank you! 

Hi,I try to connect data source to ODX from TimeXtender Portal. I tried with different data provider (Postgresql and CSV). Both data source are stored in the same machine as ODX server and TimeXtender Server. While I got the error of “Test connection request timed out. Please make sure your ODX server is running and try again.”.The TimeXtender server and ODX server are both running. I checked the connection with command, and it is ok. Test-NetConnection timextender-repositories.database.windows.net -Port 1433

HI Team, Please can you help me understand how to connect TX with common data service connector? I am trying to bring the data from D365 field service to TX. With Power BI i can use common data service connector and extract the data from FS. But i want to use TX to do ETL. Please help!!

Hey all,  We use incremental load in the ODX on a datetime field and we use an offset of 3 days (again in ODX). Now we've noticed that when we load a table from ODX to the data warehouse (even when full loading) that the _R table can contain duplicate records. In our Data Lake each increment is a parquet file of the last 3 days of data, since we load every day these files can overlap by 2 days. I believe this overlap is the cause of the duplicates in our _R table.  What happens next is that the Error report gets spammed with “Violation of primary key constraint” (which is true). But our Valid table does have the expected result. A full load transfer task solves the issue, but of course we do not want to do that all the time.   Is this the intended design? Is there a page that explains how the ODX handles these parquet files before inserting them into the Raw table? Any suggestions or insights on how to handle this issue would be greatly appreciated.

EditOffset value in incremental rule is not available for my Tx version I suppose. UpdateDate is a DateTime column and my UpdateTS an Integer. I’ll see if I can combine them in a single field somehow in the backend.TX20.10.35.64I try to set up incremental loading with a separate Date & Time column but can't seem to find the right way. Found an older source below indicating it's not possible, since it results in a OR clause instead of a AND clause.I am also not able to set up a distraction on my date column to say UpdateDate - 1 day for some reason. Is there another way to do this in the meantime instead of only the date column?Is source based incremental load possible on separate date and time fields? – TimeXtender Support  

I have a new Azure VM that I am installing with the latest version of the ODX server in preparation of upgrading from TX 20.10 to the new version.I have a new adlsgen2 storage account available as storage for the ODX. A service principal (enterprise application) in the Entra ID is created with a (confirmed working) secret.I have added the ODX Instance in the portal with the following configuration:  The app reg has Owner rights on the adlsgen2:  Still I get a 403 error when trying to create the storage container (the container does not already exist):  

Hi team,  How is the process to set my account as TX administrator? Now I am a user and I can't run ODX 's transfer option or create a new schedule

 I am trying to set up access to some tables in the ODX. when I try to add a role and search for an Azure AD user I get the error: Service request failed: Code: Authorization_RequestDenied ...Module: TimeXtender.ODX.EngineTimeXtender.ODX.Engine.ODXFaultException   at TimeXtender.ODX.Engine.ODXEngine.SendServiceRequest[C,T](WcfServerSettings serverSettings, Func`3 func)   at TimeXtender.DataManager.AddODXSecurityRoleWizard_MemberSelectStep.<>c__DisplayClass11_0.<SearchClicked>b__0()   at TimeXtender.DataManager.ConnectingThread.ExecuteConnectingThread(Object dummy)Service request failed: Code: Authorization_RequestDenied ...Module: timeXtenderTXModelInterface.ExceptionWrapperException   at TimeXtender.DataManager.ConnectingThread.HandleError()   at TimeXtender.DataManager.ConnectingThread.Execute(String title, Int32 progressSteps, List`1 actions)   at TimeXtender.DataManager.AddODXSecurityRoleWizard_MemberSelectStep.SearchClicked(Object sender, EventArgs e) This setup is a V

We are facing a problem with a ODX data lake storage table. The problems are appearing at two points in our Data Warehouse:The ODX transfer tasks gives the warning “Table folder X is used by another table with id Y” When using the ‘Preview’ option from TX interface in ODX storage we get the error “The azure data lake gen2 data storage table folder has invalid meta data. The ODX must transfer the table again”Using ‘Synchronize objects’ on ODX server does not do anything at all. Moreover, executing the table in DSA still works without errors and seems to contain new data. Also, in our Azure storage account we can see the transfer task adds new versions of the table every day (scheduled transfer task) even though it gives us the warning 1).By the way, the table ID that the error on ODX transfer task gives us does not correspond with the TableId that we can see in the _model.json file in the corresponding folder in our Azure storage account. Where should the table ID from the first error p

Hi, I have created and populated a small table in Dataverse using a Power App.There are no problems accessing and reading data from the table with Power BI Desktop.But I don't really understand how to use CData's connector (Microsoft Dataverse (23.0.8749.0) 64bit). Does anyone know which fields to fill in for the authentication itself? Username and password or equivalent. BRAnders

Hi,I have a working connection to an S3 bucket with CSV files, using the CData CSV connector.I would like to create a solution where I read only the latest file added to the bucket.Calling the stored procedure “ListFiles” (https://cdn.cdata.com/help/RVJ/ado/pg_sp-listfiles.htm) in a query table returns a table of the files, with filename and created timestamp.Is it possible to call ListFiles in an .rsd file? If so, I think it would be possible to access the created timestamp in another .rsd file using a query slicer which can be used to get the latest file.Have anyone solved this problem, or something similar? I am using the latest version, 6512.1. 

We are using the Azure Data Factory - PostgreSQL 14.0.0.0 data source connector and cannot run the synchronize task. We have selected ‘All tables’ in the data source setup but get an error in ADF:“Failure happened on 'Source' side. ErrorCode=UserErrorInvalidValueInPayload,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to convert the value in 'queryTimeout' property to 'System.TimeSpan' type. Please make sure the payload structure and value are correct.,Source=Microsoft.DataTransfer.DataContracts,''Type=System.ArgumentException,Message=Object of type 'System.Int64' cannot be converted to type 'System.String'.,Source=mscorlib”The error seems to be related to the ‘Command timeout’ setting in the TX portal, which we set to 1440 but ADF wants it to be a ‘System.TimeSpan’ type?When we want to use ‘Select tables’ on the ODX data source the screen runs ‘Search tables’ for a split second and then the results remain empty, no tables appear at all.The Linked Ser

I have a rest interface loading via ODX in V20 into an Azure Data Lake.Recently an infrastructure change to the server made the Rest Endpoint timeout (it has an IP whitelist and the server IP was removed).This resulted in the Synchronization task determining that there were no tables available to load, even though there are multiple custom RSD files that have been generated and the CData Rest driver has been configured to Never generate Schema files. The IP has been restored to the whitelist and the Rest interface is working again (after multiple days of the ODX loading nothing). Unfortunately the Meta Data for every table has been flagged as invalid. When I run the ODX load every table comes back with an error that the table has already been used by a table with a different GUID. And If I run the synchronize task on the ODX it deletes the mapping of every table connected to the particular ODX source (40+ tables).I cannot even run a preview of the data from the Gen 2 storage through Ti

Hello,Our ODX gets stuck whenever we make a request to it (be it synch, transfer, or viewing a log). Then we see the error message in attachment and usually we see a second ODX all of the sudden. Any idea what may be causing it? 

Is there a way to get the timetamp of when a table was transferred from source to ODX Server? Like the DW_Timestamp. The only system field i can find is odx_batch_number. 

Is there a way to remove tables from ODX storage that have been deleted in the desktop, similar to the SQL Database cleanup tool available in the DW? My use case is when SQL Storage is used for the ODX.The “Remove unused object” option does not drop tables it seems. The Storage Management Task is good of course, but we can’t use it for deleted data sources.Let’s say I have transferred a large table hundreds of times into my ODX and then deleted the data source. Can I remove them without having to drop the tables manually in SSMS?Best,Pontus