Skip to main content
To homepage To homepage

Ingest

Ask questions about Ingest Instances in TimeXtender Data Integration

  • 288 Topics
  • 1,554 Replies
288 Topics
M
Participant
MMMParticipant
asked in Ingest
Snowflake Ingest instance functionality

Hi! We are currently setting up TimeXtender with Snowflake and have a bit of an issue when it comes TimeXtender not having the capability of managing the ODX in Snowflake (internal stage). As you know an external stage is needed via Azure Data Lake Gen2. The issue for us is that we would like to use private endpoint when creating this data lake in Azure but that requires business critical subscription to Snowflake which is 30 percent more expensive. Are there any plans/timeline for when TimeXtender will be capable of handling the Ingest instance in Snowflake and if so could you please give a rough estimate on when that might be available? We need to make some decisions on our end on how to proceed and would like to know the answer before we move forward.

224
Christian Hauggaard
Community Manager
Emiel van Schaik
Visitor
Emiel van SchaikVisitor
asked in Ingest
Does anyone have experience with Adnamics as a datasource?

As title suggests, I wonder if anyone in this community has experience with ingesting TX with the data source Adnamics and their API. Thanks in advance!

273
Christian Hauggaard
Community Manager
B
Contributor
bjornContributor
asked in Ingest
Avoid URL encoding in the Query

Hey, I use the REST connector to consume a CSV file. In my query parameters I got one for auth key. So when I try to load the file a get an error and the reason is that parameter get encoded (% → %25) . How can I avoid encoding in the query? /Bjørn A.

473
Christian Hauggaard
Community Manager
A
Starter
AmrStarter
asked in Ingest
Invalid column count Excel sheet on storage account

Suddenly all the excel sheets are not loading anymore however some of them no one touched 6 months ago. the error is: Executing table excel_sheet:failed with error:Exception Type: System.ExceptionMessage: Invalid column count Stack Trace: at DataStorageEngine.DataLakeGen2.DataLakeGen2Transfer.<>c__DisplayClass41_5.<UploadData>b__14(DataTable dataTable, Partition partition) at DataStorageEngine.DataLakeGen2.DataLakeGen2Transfer.UploadData(ADLSGen2APIManager api, DataSourceModel dataSourceModel, DataLakeGen2ExecutionMethod executionMethod, Action`2 writeExecutionLog) at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.<>c__DisplayClass35_2.<TransferDataAsync>b__7(DataLakeGen2ExecutionMethod executionMethod) at DataSourceEngine.Custom.CustomSourceEngine.InvokeDataSourceExecution(Action`1 action) at DataSourceEngine.Custom.CustomSourceEngine.RunExecution[M](DataExecutionDestination dataExecutionMethod, Action`1 action) at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.<>c

548
Thomas Lind
Community Manager
R
Participant
r.miltenburgParticipant
asked in Ingest
Schema already exists error when change data soruce connection

Hi, We've updatet our Data Integration version from 6766.1 to 6963.1. Which means we had to update our outdated providers. So we followed the steps mentioned in this post Change data source provider | Community (timextender.com). This results in an error when we try to full load changed data source. It keeps giving is the error: “The schema …. already exists on the SQL storage and cannot be used” Which actually means we can’t update our providers/ data sources. We have to create new ones en have to map all our tables in de odx. Anyone else facing this problem and have a solution for this?

716
R
Participant
T
Participant
thzmanParticipant
asked in Ingest
SOAP request with encryption

​Hi, I’m sorry for this somewhat lengthy post but I have a problem that I don’t really know how to solve and I’m just looking to see if anyone have run into the same issue, or maybe already solved it. We are going to fetch data from a time management system that have a SOAP API and all the information I find in the community refers to RSD files, that is not to be used anymore. I don’t know how I would solve the problem that I have even with an RSD file, but that’s somewhat beside the point here. 😊 The issue is that we must take the request XML, encrypt the whole thing using AES, into a fingerprint that in turn will need to be inserted into the actual request that is then sent to the API. I have received some code in C# that does the following steps: Generates a timestamp and an MD5 hash of the request XML Generates the fingerprint XML using the timestamp and the newly created MD5 hash with the schema like in the attached picture fingerprint_schema Encrypts the fingerprint XML with AES

3711
T
Participant
N
Contributor
nmoContributor
asked in Ingest
REST: replace '' with NULL

I am trying to replace the CDATA Dynamics365 adapter with the TX REST connector (8.0.0) and I am having some issues where empty values from the API is interpreted as empty strings rather than NULL and as a result creates tables with incorrect datatypes How do i set up a way to replace these empty strings with NULL values and keep the correct data types as well I have tried with various combination of “Search and replace" on the endpoint as well as the individual nodes in the table builder without any luck. I also have the “Empty fields as null” option enabled.

213
Thomas Lind
Community Manager
avanrijn
Contributor
avanrijnContributor
asked in Ingest
Data Source migration tool

Dear sir, Our client wants to upgrade to the latest and the greatest of TimeXtender.Right now they are on TX version 6766.1. So they have to migrate their CData datasources to the new TX datasources. Question: Is there a ‘datasource migration-tool’ to help them with that ? Or do they have to setup/configure the new TimeXtender datasources manually ? Hope you can shine a light on this. Regards,Arthur

533
Christian Hauggaard
Community Manager
A
Participant
annfossParticipant
asked in Ingest
CSV Connector Compatibility with Incremental or History Settings?

Hi,I'm using the TimeXtender CSV Data Source provider and was wondering if it's possible to combine this provider with either incremental load settings or history settings. I’ve tried enabling incremental load—both in the ODX and by setting the table as incremental in the first data area—but I receive the error: "The provider does not support incremental load." Interestingly, I don't get the same error when I set the table as a history table instead, but when I try to execute the table, the result is empty. I'm wondering if this could be due to incorrect natural key configuration from my side, or if the CSV provider simply doesn't support historical loading. Any clarification would be appreciated. Thank you!

344
Thomas Lind
Community Manager
B
Explorer
Boglarka BaranyExplorer
asked in Ingest
API REST Get Schema function not working

The Get Schema function is not working for the REST endpoints / table flattening. I am trying to use version: 8.0.0.0. I get the following message: Any ideas what is happening?

678
Christian Hauggaard
Community Manager
rvgfox
Problem Solver
rvgfoxProblem Solver
asked in Ingest
Data Source to Azure Blob Containers

I’ve an Azure storage account with a blob Container that has folders and files. How can I configure a data source to use it?

325
Christian Hauggaard
Community Manager
M
Starter
MiesStarter
asked in Ingest
Not all rows of CSV file are loaded during Full load ingest task

I use the CSV provider to load a folder with multiple CSV files. I know one of the files has 17471 rows. Only the ingest task only loads the first 14521 rows. I don't have any filtering in the ingest task. Does anyone know how I can load the full file?

6812
Christian Hauggaard
Community Manager
C
Explorer
cmynhardtExplorer
asked in Ingest
MS Graph API Group and Group Members do not always return the Group Id

I am extracting data from the MS Graph API using the REST connector in TimeXtender. The issue I am facing is that I can easily retrieve the members of the groups but unfortunately, the GroupId is not included in the Query and I do not see any way to include the {id} parameter into the saved results. If anyone has solved this, can you please help with some information on how to solve this?

261
N
Contributor
A
Starter
anders.bengtssonStarter
asked in Ingest
Connector for Active Directory - examples

Have anyone an example on a successful connector regarding a normal Active Directory - not Azure AD. Can’t find any information about it but I guess many have done a connection using TX for that. Regards Anders Bengtsson

18523
Christian Hauggaard
Community Manager
oscarcleve
Contributor
oscarcleveContributor
asked in Ingest
TimeXtender REST API override header

We have trouble with an endpoint in a TimeXtender REST API data source using the Override Headers functionality. The problem arose when we upgraded our TX software to the latest version of TDI/TIS and TX REST from 6848.1/6848.1 and 7.1.0.0. We get this error message: “Failed to execute request for endpoint 'NBABIItemWS'.HTTP Status Code 401 UnauthorizedResponse body:<error xmlns="http://docs.oasis-open.org/odata/ns/metadata"><code>Unauthorized</code><message>The credentials provided are incorrect</message></error>” We use the Override Headers functionality to limit page size for the particular endpoint. It seems that for some reason the authentication token is overwritten when using this feature, and there fore we can not authenticate. We have plenty of other endpoints in the same data source not using Override Headers, and those are working fine, so we suspect that the issue has to do with that setting. We use this combination of versions: Is there a way we can alter the call for the

604
oscarcleve
Contributor
A
Problem Solver
aftabpProblem Solver
asked in Ingest
Need some help with ingesting

This is my JSON { "projectNumber": "123456", "name": "TX", "addendumValues": { "addendumGroups": [ { "addendumGroupName": "G1", "rows": [ { "columns": { "revnr_G1": [ "Rev.1" ], "date_G1": [ "2023-06-21T00:00:00.0000000" ] } } ] }, { "addendumGroupName": "G2", "rows": [ { "columns": { "revnr_G2": [ "Rev.1" ], "date_G2": [ "2023-06-21T00:00:00.0000000" ] } }, { "columns": { "revnr_G2": [ "Rev.2" ], "date_G2": [ "2023-07-21T00:00:00.0000000" ] } }, { "columns": { "revnr_G2": [ "Rev.3" ], "date_G2": [ "2023-08-21T00:00:00.0000000" ] } } ] } ] }} xpath: /json/addendumGroups has multiple addendumGroupName, here i present only two.I am trying to extract values for revnr_G1, date_G1, revnr_G2 and date_G2. The code below only shows revnr_G1 and date_G1.I am wondering why not revnr_G2 and date_G2 (please look away from case sensitivty) <api:info title="addendumGroups" desc="Generated schema file for TimeXtender." xmlns:other="http://apiscript.com/ns?v1"> <!-- Attributes for the main JSON struct

9512
A
Problem Solver
M
Explorer
MvestExplorer
asked in Ingest
Full Load Transfer vs. On-Demand Execution

Hey there, We used to have the On-Demand setting enabled for one of our data sources. With this setting turned on, the package that executes the corresponding tables would take roughly one hour to complete. Recently, we switched to a Full Load Transfer Task, followed by an Execute Tables package. However, now the transfer/ingestion process takes almost 2.5 hours (and then another 20 minutes to execute the tables). We thoroughly checked to ensure that the tables included in the Full Load are the same as those processed with On-Demand, and they are. In the logs I can see a lot of these messages (every 5 minutes), so I am wondering how much time the full transfer is idly waiting.

351
Christian Hauggaard
Community Manager
P
Contributor
PieterQContributor
asked in Ingest
Fabric lakehouse ingest - error Import Metadata

Hi Everyone, We want to use Fabric lakehouse as ingest layer, and did the setup according to this article: We are running TimeXtender version 6898.1 We created the lakehouse form Fabric, not by using the ‘Create Storage; option from TDI. When we test the connection in TDI, everything looks fine: However, the metadata import task fails with following error: The execution failed with error:Exception Type: Azure.RequestFailedExceptionMessage: Service request failed. Status: 400 (BadRequest) ErrorCode: IncomingOperationUntrusted Headers: x-ms-error-code: IncomingOperationUntrusted Access-Control-Allow-Origin: * Access-Control-Allow-Headers: REDACTED Access-Control-Allow-Methods: REDACTED Access-Control-Expose-Headers: REDACTED Strict-Transport-Security: REDACTED X-Content-Type-Options: REDACTED x-ms-root-activity-id: REDACTED Content-Length: 0 Date: Thu, 06 Mar 2025 14:57:14 GMT Server: Microsoft-HTTPAPI/2.0 Stack Trace: at Azure.Core.HttpPipelineExtensions.ProcessMessage(HttpPipeline pipe

705
Christian Hauggaard
Community Manager
oscarcleve
Contributor
oscarcleveContributor
asked in Ingest
Limited functionality TimeXtender REST data source

Just recently the TimeXtender REST data source at a client was not able to be edited to the same extent anymore. When pressing “Get Schema” in the Table Flattening Table Builder, we get this error message: No changes or updates were made to the TDI, ODX, or the TX REST data connection as the error started to appear. We use this combination of versions: As suggested here: The exact TX REST version used is 7.1.0.0. If possible we would like a solution to this problem that does not require an upgrade of the TDI software.

631
Christian Hauggaard
Community Manager
A
Participant
annfossParticipant
asked in Ingest
TimeXtender Rest data source with CSV response

Hi I have a REST data source that provides the desired output in CSV format. However, I'm looking for guidance on formatting the XML output to match the expected structure. Currently, the XML output I receive looks like this: <?xml version="1.0" encoding="UTF-8"?><Name_of_table> <value> "name_of_column1","name_of_column2","name_of_column3","000: name_of_column4," "G name,with comma","2000M01","Volumindeks, kalenderjustert",. "G name,with comma","2000M02","Volumindeks, kalenderjustert",. "G name,with comma","2000M04","Volumindeks, kalenderjustert",3.2 "G name,with comma","2000M05","Volumindeks, kalenderjustert",50.2... </value></Name_of_table> Is there any support for CSV responses in the Rest Connector, or do you recommend creating a XSLT file for this purpose?

878
A
Participant
anders.e.jonsson
Contributor
anders.e.jonssonContributor
asked in Ingest
Dynamic Values for parameter in same endpoint REST data source

Hi, I am trying to set up a REST connection. Here are the prerequisites: The API delivers data for a maximum of one month, and no more than 10,000 rows at a time. I'm a bit of a beginner when it comes to the new Rest data source 🙄I wonder what the easiest way to do is to;1. Read all rows for the selected period using the fetchFromActivityId field. That field is in one of the API's tables.2. Loop through month by month until I arrive at today's date.The API works fine. It is no problem to fetch 10,000 rows from a selected period.All ideas are gratefully received 🙂 BRAnders

766
anders.e.jonsson
Contributor
K
Contributor
KajEmergoContributor
asked in Ingest
Connection issue ODX Server upgrade

When launching the newly installed ODX Server configuration (20.10.45), we get the following error: First, we thought this was a Firewall issue on our (Azure VM), because this test-netconnection did not work (see the second one failing): When I try to reach that second address from my local laptop, I get the same failure. Could you explain what is going on here? When using the ‘tracert’ to timextender-api.azurewebsites.net command in the command prompt on my local machine I see that it forwards to waws-prod-am2-061.cloudapp.net. When we try to do the same on the TX Azure VM, it times out on the first step. Do you have any advice on how to proceed here? Could it still be a Firewall issue on our VM?

22110
rory.smith
TimeXtender Xpert
avanrijn
Contributor
avanrijnContributor
asked in Ingest
Working with RSD file with TX version 6892

Dear sir, Is it still possible to work with RSD files in TX version 6892 ? I found this link: Is it possible to do something like this with the latest 8.0 TimeXtender REST provider ? I am trying to work with an API that is refreshing there Bearer Token every hour “expiresIn”: 3600 curl -X 'POST' \ 'https://openapi.binnenbeter.nl/api/v1/login' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "tenant": "horstaandemaas", "username": "*****", "password": "*****"}' { "token": "eyJwMmMi**********lRW5SWTZKaTFzb1EiLCJhbGciOiJQQkVTMi1IUzUxMitBMjU2S1ciLCJlbmMiOiJBMjU2R0NNIn0.J8cstvlbqRrWzsRulKtnBFv4eKG6r3fvudZI*********gvjw", "tokenType": "Bearer", "roles": "ADMIN, API", "expiresIn": 3600} So I have to come up with a solution that dynamically refreshes the token before it is fetching the data or else the API will deny access because the token is out of date. If you have any ideas please let me know. The plan is: What I am trying is to run a powershell script to fetch

818
avanrijn
Contributor
M
Visitor
MonikaVisitor
asked in Ingest
Missing 'Select tables'

Hi, I cannot find the option to ‘Select tables’ from the source: My TX version is 6848.1Where should I look for that?

422
rvgfox
Problem Solver
A
Participant
annfossParticipant
asked in Ingest
Rest data source - Json-stat format

Hi Does the rest data source support json-stat format? Is it able to interpret it and transform it into a structured table? I tried using the JSON data format, but I received an XML file that the ODX could not extract any columns from.

341
Thomas Lind
Community Manager

Most helpful members this week

Badge winners

  • Innovator
    oscarclevehas earned the badge Innovator
  • Certified Solutions Architect
    leo.noharethas earned the badge Certified Solutions Architect
  • Innovator
    Bernarddbhas earned the badge Innovator
  • Certified Solutions Architect
    JeroenStahas earned the badge Certified Solutions Architect
  • Ingest Expert
    nmohas earned the badge Ingest Expert
Show all badges

Popular tags

Terms and ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings