To homepage To homepage

- - Knowledge base overview
Product Updates
Training
Portal
For Partners
TimeXtender Classic

Home
Community overview
TimeXtender Desktop Q&A
Ingest

Ingest

Ask questions about Ingest Instances in TimeXtender Data Integration

253 Topics
1,360 Replies

When you subscribe we will email you when there is a new topic in this category

253 Topics

Most replies

Most views

Newest first

MarcodenijsContributor

asked in Ingest

Rename table from REST API

We have a REST API that we use to connect to several endpoints. The default name of all the tables is: REST.Jsondata. This causes issues in our Azure Gen2 datalake. It gives an error saying the table name is already in use by a table with a different ID.I almost feel like there must be some really easy way to rename the table name before it lands in the Gen2 datalake but so far I haven’t been able to figure out how.The workaround I use now is to use a QueryTable but that is not the nicest way to do it I guess.

632

8 months ago

MvestExplorer

asked in Ingest

Graphical bug or authorisation issue? Certain functionalities missing or disabled.

Hello,My organisation recently had to redeploy our development virtual machine and reinstall timextender / odx on it. Since then I am not allowed to click on certain buttons inside the application, and options are missing when I right-click on certain objects.For example I am not allowed to click on the [Stop Task] button in the ODX Execution Queue window. And when I right-click on an object in the Manage ODX tab, all I can see it ‘View Properties’ and not any of the many other options that used to be there.I do have these functions available in our other environments. And my collegues have no issues on our dev machine.I have tried the following so far:Enable/disable Deprecated Features. Delete the hidden Default.rdp file and create a new rdp connection to the machine. Any tips?

314

8 months ago

rvgfoxProblem Solver

asked in Ingest

Two ODX in the same Storage container

Currently I’m using one ODX instance with this setup: All the mapped data sources connections has selected the option “Enabled data on demand”: I am using three instances of Data Warehouse: Development → Testing → Production.My question is:Can I create a new ODX instance connected to the same Storage container?The idea is that the same data sources in the DW instances for development and test use the "Enable data on demand" OFF option. In this way the storage container only will be refreshed one time (when we refresh the DWH production instance) because the data source is the same.

475

8 months ago

majeddigStarter

asked in Ingest

Transfer task doesnt work - sql database - doesnt show any table in ODX select table

Hello Guys. I tried to select table from ODX but it deosnt show none (except once query table).the transfer task work fine (it show succefull) but once I triy to select table in order to send them to the datawerhouse I show nothing.- even if it work fine when I use query tool. Any suggestion please? thanks@Thomas Lind

465

8 months ago

wouter.goslingaTimeXtender Xpert

asked in Ingest

CSV load fails with operation could not be found error

I am loading a CSV (comma separated) file with the ODX. After upgrading the ODX to the latest version, transferring any table with this datasource gives the following error: Syncing does work.ODX version: 6521.1TX version: 6536.1CSV Provider version: tried both 22.0.8276.0 and 23.0.8749.0 but same error.

13513

8 months ago

spenningContributor

asked in Ingest

TimeXtender Desktop 6536.1 issue with ODX incremental load?

Hi,We have only source in which we are using a rest source and incrementally loading the data in the odx. The api cannot handle a date range that is too big. When we trying to preview the data in the odx, we see the data that we need. However, when we run the odx transfer task, with syncing it does not give all the updates in the table like it should. I saw that there were some fixes with the previous versions with incremental loading. We upgraded today to 6536.1. Also we upgraded the rest data source to the newest version. However, because the preview is fine, I don't think the problem is the source connector.

812

8 months ago

aftabpContributor

asked in Ingest

Getting BLOOB file into DVH

I need some guidence saving contents ofan endpoint as BLOOB. I can see that the version we use doesnot have an option for BLOOB but may be Text or Binary with maximum length can give me what i need.So I have this RSD that i created <api:script xmlns:api="http://apiscript.com/ns?v1" xmlns:xs="http://www.w3.org/2001/XMLSchema">  <api:info title="GetAllStops" desc="Generated schema file." xmlns:other="http://apiscript.com/ns?v1">  <attr name="kortnavn" xs:type="string" readonly="false" other:xPath="/json/kortnavn" /> <attr name="tittel" xs:type="string" readonly="false" other:xPath="/json/tittel" /> <attr name="virksomhet" xs:type="string" readonly="false" other:xPath="/jso

898

9 months ago

JeroenSParticipant

asked in Ingest

No tables visible in Select Table Option on Data Source after synchronizing

We are running TX version 6536.1.Our data source is AX Dynamics 2012 on premise. We have the issue both with the SQL Server connector and the OLEDB connector.After synchronizing, we can't select any tables with right click on the data source → Select Tables → Search.It also takes > 30 minutes before the table selector pops up.Our only option is to select all tables.The Transfer task - Select table option does give us the option to select tables. However we can't select any individual columns. Neither does this give us the option to define any incremental selection rules or row filters.

14013

9 months ago

VictorParticipant

asked in Ingest

ODX Transfer task running indefinitely

Hello, TimeXtender: 20.10.40.64ODX: 20.10.31We are using the ODX and TimeXtenders SQL database connector to get data from a synpase database. Sometimes the synpase databas seems to have problems, where a query runs and doesn’t complete (a seperate issue we are working on).However what happens in TimeXtender in the ODX is that the transfer task keeps running indefinitely and never times out, leading timextender execution package to run but just sits there waiting for the odx transfer task to finish. However nothing fails anywhere, so we do not get an alert.We need some way to be able to capture this.Is there anyway to set up one or both of these things in TimeXtender or if you have any other ideas?1. Kill the transfer task if it takes more then X minutes2. Get an alert on failure or long running taskThank you!

24220

9 months ago

junweiStarter

asked in Ingest

Issue of data source connection

Hi,I try to connect data source to ODX from TimeXtender Portal. I tried with different data provider (Postgresql and CSV). Both data source are stored in the same machine as ODX server and TimeXtender Server. While I got the error of “Test connection request timed out. Please make sure your ODX server is running and try again.”.The TimeXtender server and ODX server are both running. I checked the connection with command, and it is ok. Test-NetConnection timextender-repositories.database.windows.net -Port 1433

1178

9 months ago

jyothi.dosiContributor

asked in Ingest

CommonDataService Connector

HI Team, Please can you help me understand how to connect TX with common data service connector? I am trying to bring the data from D365 field service to TX. With Power BI i can use common data service connector and extract the data from FS. But i want to use TX to do ETL. Please help!!

592

9 months ago

sierd.zuiderveldContributor

asked in Ingest

Primary Key Violation using Incremental Offset

Hey all, We use incremental load in the ODX on a datetime field and we use an offset of 3 days (again in ODX). Now we've noticed that when we load a table from ODX to the data warehouse (even when full loading) that the _R table can contain duplicate records. In our Data Lake each increment is a parquet file of the last 3 days of data, since we load every day these files can overlap by 2 days. I believe this overlap is the cause of the duplicates in our _R table. What happens next is that the Error report gets spammed with “Violation of primary key constraint” (which is true). But our Valid table does have the expected result. A full load transfer task solves the issue, but of course we do not want to do that all the time. Is this the intended design? Is there a page that explains how the ODX handles these parquet files before inserting them into the Raw table? Any suggestions or insights on how to handle this issue would be greatly appreciated.

1718

9 months ago

KCMTContributor

asked in Ingest

How to set up Source based incremental load with seperate Date and Timestamp column

EditOffset value in incremental rule is not available for my Tx version I suppose. UpdateDate is a DateTime column and my UpdateTS an Integer. I’ll see if I can combine them in a single field somehow in the backend.TX20.10.35.64I try to set up incremental loading with a separate Date & Time column but can't seem to find the right way. Found an older source below indicating it's not possible, since it results in a OR clause instead of a AND clause.I am also not able to set up a distraction on my date column to say UpdateDate - 1 day for some reason. Is there another way to do this in the meantime instead of only the date column?Is source based incremental load possible on separate date and time fields? – TimeXtender Support

1287

10 months ago

wouter.goslingaTimeXtender Xpert

asked in Ingest

Issue with connecting adlsgen2 to ODX

I have a new Azure VM that I am installing with the latest version of the ODX server in preparation of upgrading from TX 20.10 to the new version.I have a new adlsgen2 storage account available as storage for the ODX. A service principal (enterprise application) in the Entra ID is created with a (confirmed working) secret.I have added the ODX Instance in the portal with the following configuration: The app reg has Owner rights on the adlsgen2: Still I get a 403 error when trying to create the storage container (the container does not already exist):

522

10 months ago

ignacioContributor

asked in Ingest

How to set as TX administrator

Hi team, How is the process to set my account as TX administrator? Now I am a user and I can't run ODX 's transfer option or create a new schedule

422

10 months ago

rory.smithTimeXtender Xpert

asked in Ingest

What rights are needed to be able to create roles in the ODX Server to define access

I am trying to set up access to some tables in the ODX. when I try to add a role and search for an Azure AD user I get the error: Service request failed: Code: Authorization_RequestDenied ...Module: TimeXtender.ODX.EngineTimeXtender.ODX.Engine.ODXFaultException at TimeXtender.ODX.Engine.ODXEngine.SendServiceRequest[C,T](WcfServerSettings serverSettings, Func`3 func) at TimeXtender.DataManager.AddODXSecurityRoleWizard_MemberSelectStep.<>c__DisplayClass11_0.<SearchClicked>b__0() at TimeXtender.DataManager.ConnectingThread.ExecuteConnectingThread(Object dummy)Service request failed: Code: Authorization_RequestDenied ...Module: timeXtenderTXModelInterface.ExceptionWrapperException at TimeXtender.DataManager.ConnectingThread.HandleError() at TimeXtender.DataManager.ConnectingThread.Execute(String title, Int32 progressSteps, List`1 actions) at TimeXtender.DataManager.AddODXSecurityRoleWizard_MemberSelectStep.SearchClicked(Object sender, EventArgs e) This setup is a V

21716

10 months ago

KajEmergoContributor

asked in Ingest

Azure data storage table folder has invalid meta data

We are facing a problem with a ODX data lake storage table. The problems are appearing at two points in our Data Warehouse:The ODX transfer tasks gives the warning “Table folder X is used by another table with id Y” When using the ‘Preview’ option from TX interface in ODX storage we get the error “The azure data lake gen2 data storage table folder has invalid meta data. The ODX must transfer the table again”Using ‘Synchronize objects’ on ODX server does not do anything at all. Moreover, executing the table in DSA still works without errors and seems to contain new data. Also, in our Azure storage account we can see the transfer task adds new versions of the table every day (scheduled transfer task) even though it gives us the warning 1).By the way, the table ID that the error on ODX transfer task gives us does not correspond with the TableId that we can see in the _model.json file in the corresponding folder in our Azure storage account. Where should the table ID from the first error p

2117

10 months ago

anders.e.jonssonContributor

asked in Ingest

Microsoft Dataverse Connection

Hi, I have created and populated a small table in Dataverse using a Power App.There are no problems accessing and reading data from the table with Power BI Desktop.But I don't really understand how to use CData's connector (Microsoft Dataverse (23.0.8749.0) 64bit). Does anyone know which fields to fill in for the authentication itself? Username and password or equivalent. BRAnders

2593

10 months ago

pontus.berglundContributor

asked in Ingest

Read latest CSV file (from S3 bucket)

Hi,I have a working connection to an S3 bucket with CSV files, using the CData CSV connector.I would like to create a solution where I read only the latest file added to the bucket.Calling the stored procedure “ListFiles” (https://cdn.cdata.com/help/RVJ/ado/pg_sp-listfiles.htm) in a query table returns a table of the files, with filename and created timestamp.Is it possible to call ListFiles in an .rsd file? If so, I think it would be possible to access the created timestamp in another .rsd file using a query slicer which can be used to get the latest file.Have anyone solved this problem, or something similar? I am using the latest version, 6512.1.

24810

10 months ago

KajEmergoContributor

asked in Ingest

ADF - PostGreSQL 'queryTimeout' error

We are using the Azure Data Factory - PostgreSQL 14.0.0.0 data source connector and cannot run the synchronize task. We have selected ‘All tables’ in the data source setup but get an error in ADF:“Failure happened on 'Source' side. ErrorCode=UserErrorInvalidValueInPayload,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to convert the value in 'queryTimeout' property to 'System.TimeSpan' type. Please make sure the payload structure and value are correct.,Source=Microsoft.DataTransfer.DataContracts,''Type=System.ArgumentException,Message=Object of type 'System.Int64' cannot be converted to type 'System.String'.,Source=mscorlib”The error seems to be related to the ‘Command timeout’ setting in the TX portal, which we set to 1440 but ADF wants it to be a ‘System.TimeSpan’ type?When we want to use ‘Select tables’ on the ODX data source the screen runs ‘Search tables’ for a split second and then the results remain empty, no tables appear at all.The Linked Ser

692

10 months ago

Paul McLeod Bespoke AnalyticsTimeXtender Xpert

asked in Ingest

V20 ODX loosing Meta Data for REST data source

I have a rest interface loading via ODX in V20 into an Azure Data Lake.Recently an infrastructure change to the server made the Rest Endpoint timeout (it has an IP whitelist and the server IP was removed).This resulted in the Synchronization task determining that there were no tables available to load, even though there are multiple custom RSD files that have been generated and the CData Rest driver has been configured to Never generate Schema files. The IP has been restored to the whitelist and the Rest interface is working again (after multiple days of the ODX loading nothing). Unfortunately the Meta Data for every table has been flagged as invalid. When I run the ODX load every table comes back with an error that the table has already been used by a table with a different GUID. And If I run the synchronize task on the ODX it deletes the mapping of every table connected to the particular ODX source (40+ tables).I cannot even run a preview of the data from the Gen 2 storage through Ti

ofirbExplorer

Hello,Our ODX gets stuck whenever we make a request to it (be it synch, transfer, or viewing a log). Then we see the error message in attachment and usually we see a second ODX all of the sudden. Any idea what may be causing it?

1634

10 months ago

magnus.jorstadVisitor

asked in Ingest

ODX Server timestamp

Is there a way to get the timetamp of when a table was transferred from source to ODX Server? Like the DW_Timestamp. The only system field i can find is odx_batch_number.

1475

10 months ago

pontus.berglundContributor

asked in Ingest

SQL Database cleanup tool for ODX

Is there a way to remove tables from ODX storage that have been deleted in the desktop, similar to the SQL Database cleanup tool available in the DW? My use case is when SQL Storage is used for the ODX.The “Remove unused object” option does not drop tables it seems. The Storage Management Task is good of course, but we can’t use it for deleted data sources.Let’s say I have transferred a large table hundreds of times into my ODX and then deleted the data source. Can I remove them without having to drop the tables manually in SSMS?Best,Pontus

1063

10 months ago

bas.hopstakenTimeXtender Xpert

asked in Ingest

PK table in ODX database is growing rapidly in terms of disk space

Hi community,I’ve a question about the ‘PK’ table that is created in the ODX when you enabled the ‘Handle primary key updates’ and ‘Handle primary key deletes’.I’ve noticed the ODX database (SQL Database) is growing rapidly in disk space after enable an incremental load schedule to reload data every 5 minutes. When I used the default SQL ‘Disk usage by top tables’ I noticed the PK tables are the biggest in terms of disk space.My PK table of the GL Entry table contains 4.5 billion rows! And is 95 GB. While my DATA table contains only 118 million records.When I query the PK table and filter the primay key in this table on 1 value. This value is saved 75 times in this table. For every odx_batch_number (odx_batchnumber 0 – 74). Is this normal? I think it is a bit strange that my PK tables are the biggest tables in the ODX in terms of disk space.Even when I run the storage management task. It doesn’t clean the PK table’s. It always contains the primary key for every odx batch load.The cust

2479

11 months ago

Page 5 / 11

Badge winners

Dikshahas earned the badge Certified Solutions Architect
brett.taylorhas earned the badge Certified Solutions Architect
RobvdBrinkhas earned the badge Certified Solutions Architect
aftabphas earned the badge Ingest Expert
Leo Rogemanhas earned the badge Certified Solutions Architect

Show all badges

Terms and Conditions Cookie settings

Sign up

Already have an account? Login

Login with SSO

SSO login

Username *

E-mail address *

Full Name

(Private)

Only you and moderators can see this information

TX Partner / Customer *

Company

(Private)

Only you and moderators can see this information

Password *

I accept the terms & conditions

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Login with SSO

SSO login

Username or Email

Password

Remember me

Forgot password?

Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.

Enter your e-mail address

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.