To homepage To homepage

- - Knowledge base overview
Product Updates
Training
Portal
For Partners
TimeXtender Classic

Home
Community overview
TimeXtender Desktop Q&A
Ingest

Ingest

Ask questions about Ingest Instances in TimeXtender Data Integration

237 Topics
1,274 Replies

When you subscribe we will email you when there is a new topic in this category

237 Topics

Recently active

Most replies Most views Newest first

sierd.zuiderveldContributor

asked in Ingest

TimeZone differences for DSA exexution logs vs ODX.

Our machine runs in UTC+2.00, whilst our Azure databases (by default) run on UTC time. Now when we schedule an ODX Transfer task, the interface let's us schedule these tasks in UTC time. The Execution logs of a transfer task are also in UTC time. However, schedules for execution packages are in local time, UTC 2.00, and therefore there logs are also in UTC 2.00. I guess my question is, why is this the case? Both the ODX & Execution service are services on the same machine with UTC 2.00. Both the ODX as well as the MDW and the project repo are azure db's in UTC. Why would one use the machine time and the other the azure db timezone? And is there a way to make them the same other than resetting the machine time to UTC? Version 20.10.37

903

1 year ago

rory.smithTimeXtender Xpert

asked in Ingest

ODX instance lineage vs DWH instance lineage

TX: 6284.1Is there a reason why Lineage generated from the Data Source Explorer in the ODX Server includes the downstream stuff in a DWH instance whereas generating Lineage from a field in a Data Area in the DWH instance does not show the ODX part of the lineage?

563

1 year ago

bas.hopstakenTimeXtender Xpert

asked in Ingest

Is it possible to clean the execution logs in ODX Server?

Hi all,I have some customers that are reloading data every x minutes in ODX Server (V20.10.x). This generates a lot of execution log entries. When I open the Execution Log in the ODX Server, it takes a long time. I think that removing old execution logs will improve the speed of opening this execution log overview window. I have no idea if it is possible to delete old execution log entries?Another option could be that the execution log view first shows the datetime range you like to see logs and then opens the overview.

ebayContributor

HelloI am wondering, how can I connect to Zip File? I searched and asked Xpilot and then I get this answer: To connect a Zip file in TimeXtender, you can use the "File" data source type. But I couldnt find File datasource. Has the name changed or is there another method?Best Regards

1082

1 year ago

jermaine.cotinoExplorer

asked in Ingest

Unable to see a post when logged in

I was interested in this link.Data directly from Data lake to MDW (without ODX) | Community (timextender.com) I am not able to access this link which was posted in the below of the message. https://support.timextender.com/timextender-product-focus-group-123/virtual-odx-capability-to-read-directly-from-existing-data-lake-container-no-second-copy-614 Please advice,

161

1 year ago

rogier.helmusProblem Solver

asked in Ingest

CData Parquet - Settings for Azure Blob Storage (multiple tables in seperate folders)

Hi all,I have a client that has a datalake with parquet files in an Azure Storage Account Blob container (Datalake Gen2). This datalake has not been created by ODX server, but has been created by Synapse (scripted, delta lake parquet files). Since we want to load multiple tables from multiple folders we use the CData Parquet connector v2023 (TX 20.10.41). The issue we have is that we can’t find the right settings for the connector to recognize that the folder we’re connecting to contains multiple tables and that the parquet files are in a subfolder of the table folder. For the connection we use a Shared Access Signature and the connection is working:The name of the container is ‘development’ which we configure in the Base URL. In the URI we point to the folder of the bronze layer, the system and the database: The database folder contains a folder for each table. In each table folder some tables have a a partition folder followed by the actual parquet file. Since the connector needs to

2383

1 year ago

p.ruijterContributor

asked in Ingest

Set up API basic auth with token request

I have below the API request for Power Shell. Now I want to set it up in TX. How to set up (Basic) Authentication for a request with a token retrieve? # Force the use of SSL [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12 ####################### # SETTINGS # ####################### # REST API SETTINGS # $API_key = "HIDDEN" $API_username = "HIDDEN" $API_password = "HIDDEN" $API_company = "HIDDEN" $API_URL_Login = "https://secure.xxxxxxxxxxxxx.eu/ta/rest/v1/login" ####################### # LOGIN AND RETRIEVE TOKEN FOR UKG Ready # ####################### Write-Host "### Retrieving token" $headers_login = @{ 'Content-Type' = 'application/json'; 'Accept' = 'application/json'; 'API-Key' = $API_key; } Write-Host "- Headers prepared" $request_login = '{"credentials": {"username": "'

3224

1 year ago

KeremOGParticipant

asked in Ingest

adding excel files from different sharepoint subsites

Hi, I've been trying to add new excel file from a new sharepoint group from the same company link to the same ODX server by using CData connector. Is there any setting to add both sharepoint links or make it visible? Now, I can only see the link with group1.sharepoint page examples;https://company.sharepoint.com/sites/group1https://company.sharepoint.com/sites/group2Best!

1053

1 year ago

juriaan.hensbroekContributor

asked in Ingest

Adding filter for one table in OData connector

Hi,I'm using the CData ADO.NET Provider for OData 2022 (22.0.8322.0) to extract Topdesk data.Due to performance reasons Topdesk made some changes in their feed. Resulting in that, since 19th june 2023, 2 larger tables (IncidentDurationBlocks and IncidentSnapshots) cannot longer be extracted without filtering on a date to make the dataset smaller.How can I add a filter for only one table in my extract?Can I create an OData RSD file?I get this error when I try to extract:System.Data.CData.OData.ODataException (0x80004005): [500] Could not execute the specified command: Unexpected response status code: 503. ---> rds220I.IC: Unexpected response status code: 503

39317

1 year ago

peter.jensenTimeXtender Xpert

asked in Ingest

What will happen in the ODX Server - Datalake when we reach the 10.000th incremental load ?

Hi,we have a customer running incremental loads multiple times a day in the ODX Server (using Data Lake Gen2). We’re reaching the 10.000th incremental run.I’m wondering how the ODX server will handle that 10.000th increment (since the naming of the parquet files allows a number up to 9999).BR.Peter

693

1 year ago

andrew.gebhardContributor

asked in Ingest

CDATA Provider for LinkedIn 2022 unable to retrieve OrganizationFollowerCount

Hi community, The CDATA Provider for LinkedIn 2022 contains a bug/error which causes it to call an invalid API URI for the connector’s table ‘OrganizationFollowerCount’. The appropriate URI for this table according to Organization Network Size - LinkedIn | Microsoft Learn is:https://api.linkedin.com/v2/networkSizes/urn:li:organization:1234567?edgeType=CompanyFollowedByMemberHowever, CDATA is calling the following:2023-06-12T11:31:21.793+02:00 2 [5|Q-Id] [HTTP|Req: 3] GET https://api.linkedin.com/rest/networkSizes/1234567?edgeType=CompanyFollowedByMemberWhich results in the following error message:2023-06-12T11:31:22.172+02:00 3 [5|Q-Id] [HTTP|Res: 3] {"message":"Key parameter value '1234567' is invalid","status":400}2023-06-12T11:31:22.172+02:00 2 [5|Q-Id] [HTTP|Res: 3] HTTP/1.1 400 Bad Request, 66 Bytes Transferred2023-06-12T11:31:22.194+02:00 1 [5|Q-Id] [EXEC|Messag] Executed query: [SELECT [FirstDegreeSize], 0 AS [odx_batch_number] FROM [LinkedIn].[Organiz

ekw9Contributor

Hi all,We’re trying to troubleshoot some thorny networking issues, and I need to know more about how ADO.net data transfer works behind the scenes - specifically for old-school business unit SQL Server data sources, if that makes a difference. I know for sure is that data is read from the data source and written into the destination table in the staging database, but I’m fuzzier on what happens between the two places. To start off with, I have two key questions: We’re using a tool server to run TimeXtender, and a different server to host SQL Server and the TimeXtender databases. Is data routed through the that tool server when it is read via the ADO.net package? I assume that it is, but we need to know for sure. The other question I have is what happens after the data is pulled from the data source? Is it pulled into memory somewhere? If so, where? The source server, the tool server, the destination data server, or some combination of the three? Is the data ever written to disk anywher

1928

1 year ago

andrew.gebhardContributor

asked in Ingest

How to recognize deleted records in ODX Server Storage

Hi team, In the business units and data warehouses, deletions in an incremental table are identified by system field IsTombstone = 1. You can use this field in a data selection rule to filter out deletions between the business unit and DSA. This field is not present in ODX Server tables with incremental and delete handling enabled. Is there a way to identify records that have been deleted from incremental tables in ODX Server? And if not, what is the purpose of enabling deletion handling in ODX Server? Kind regards,Andrew - E-mergo

1737

1 year ago

brian.wijningsExplorer

asked in Ingest

application Id isn't changing after new settings

I have made a new app registration in Azure for TimeXtender. Provide the rights exactlly as the old app registration but the secret is expired.Error in TX : AADSTS7000222: The provided client secret keys for app 'OLD_APP_ID' are expired.I filled in the datasource and global enviorment the NEW_APP_ID but the error still is there. Restarted the ODX server, removed the pipelines but still same issue. How can i force that it will use the new application ID becaused I changed it on the places it is used. (Also with the new secret ofcourse)

10711

1 year ago

william.sprangersExplorer

asked in Ingest

ADO.NET Transfer consistently slow on incremental table with no new rows

Hi! I am experiencing an issue with ADO.NET transfer times on incrementally loaded tables where the ADO.NET transfer takes the same amount of time regardless of the amount of new rows coming into the table. A full load of the table containing about 45 million rows takes about 25 minutes, on the next incremental load the load time is still the same with ADO.NET taking up around 24 of these minutes. Our current data flow is as follows: SQL server → TimeXtender → Azure elastic pool (this is where all of our TimeXtender databases resides) Full load - 45 million rows Incremental load - 11 thousand new rows in the _R table The amount of new rows coming in to the table Has anyone experienced a similar issue? My best guess is that the problem resides in the azure elastic pool where the ado.net transfer is being throttled. Even if I have 0 rows in the _R table the ADO.NET transfer time is the same. Thank you!

32418

1 year ago

bas.hopstakenTimeXtender Xpert

asked in Ingest

ODX Server filter on column in all tables

Hi community, I’m using version V6221.1. I’d like to filter all tables in the data source on column ‘dataAreaId’. I try to set this up using the following settings: Do I really need to set this filter on every table I load from this data source? Or is this a bug?

882

1 year ago

wouter.goslingaTimeXtender Xpert

asked in Ingest

HTTP 404 error on incremental load with update & delete detection

I am using ODX version 6143.1. When loading a table from a source system, incremental load on a datetime column (with an offset of 7 days) works fine. But when I set up the incremental load rule with Detect deletes and Detect updates selected, the first incremental load (which is the initial load) runs successful, but a second load (which should be the increment) gives me a 404 HTTP error:Response status code does not indicate success: 404 (The specified path does not exist.). Edit: problem only arises when setting the Delete detection option. When I remove this checkbox, the 404 error disappears (so the Detect primary key updates option does work correct).

17215

1 year ago

devin.tiemensTimeXtender Xpert

asked in Ingest

Retry steps ODX transfer task

Hi, Is it possible to setup retry steps for ODX transfer tasks in the >6024.1 version of TimeXtender? If yes, where? If not, please add this as an idea. Greets,Devin

612

1 year ago

danielTimeXtender Xpert

asked in Ingest

tableAccount is not contained. Parameter name: tableAccount

Dear Community,i'm working with timeXtender 20.10.22 and my source is a Business Unit with the NAV (Business Central) adaptor.I've switched in my project between the source being Production (BC) to Acceptance (BC).Now im getting this error and I cannot do any thing anymore. Also switching back does not work. Can anyone help me? Thanks! tableAccount is not contained.Parameter name: tableAccountDetails:tableAccount is not contained. ...Module: timeXtenderSystem.ArgumentException at TimeXtender.DataManager.TableUsageMatrix.GetTableUsage(Table_NAV tableAccount, Account account) at TimeXtender.DataManager.Adapter_NAV.GetExpectedDeployedObjectsPrivate(Table_NAV tableToDelpoy, Guid projectId, Guid tableId, List`1 sqlObjects, ProviderDestinationSql providerDestinationSql) at TimeXtender.DataManager.Adapter_NAV.GetExpectedDeployedObjects(IDataAdapterTable table, Guid projectId, Guid tableId, List`1 sqlObjects) at TimeXtender.DataManager.StepTableSimpleDeploy.GetExpectedDeployedObjects(L

583

1 year ago

Niels JonkerContributor

asked in Ingest

Transfering multiple tables from Theobald Error while copying content to stream

I am setting up TimeXtender to extact tables from SAP with Theobald. I have a selection of tables that I can succesfuly extract when I add just one table to a transfer task. However, when I add all relevant tables to a single transfer task I run into errors. The extraction seems to be succesful on the Theobald side but moving the data to the ODX storage gives the following error “Error while copying content to a stream”. I can figure out why the tranfer is working when I do it per table and gives an error when I do multiple tables.. Full error sample below: Executing table rest_mara_generalarticledata:failed with error:System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.ObjectDisposedException: Cannot access a closed Stream. at System.IO.__Error.StreamIsClosed() at System.IO.MemoryStream.get_Position() at System.Net.Http.StreamToStreamCopy.StartAsync() --- End of inner exce

24910

1 year ago

mads.nielsenExplorer

asked in Ingest

ODX "loses" transactions

When loading regularly (every hour) using incremental load, the ODX seems to “lose” transactions from source. Next run of the incremental load, ODX does not catch these missing transactions. The work around is to full load data, but the this takes long time. And it renders the incremental load useless as it is not to be trusted.We are running 20.10.34. Does any of the newer 20 versions have a fix to this?Regards Mads

22718

1 year ago

sguntupalliContributor

asked in Ingest

Duplicate tables in ODX Storage

I am working on Sandbox Environment.I on using ODX 20.10.36.64.I executed tables on 25th and today 27th in ODX DataSource.What I noticed is in my SQL server new tables were created.Do I need to update any settings to resolve this?SQL server

632

1 year ago

roland.peetersExplorer

asked in Ingest

CDATA Exchange Server Connector

Is there someone who has a quick reference card how to create the connection with an exchange server using the CDATA connector. We tried several settings with the server, but nothing works.

612

1 year ago

gerrit.degleeExplorer

asked in Ingest

Load csv from API

I've got an API that returns JSON data. One of the fields contains a UUE encoded csv file. Is there a way to load this to the ODX without some external pre processing?Thanks,Gerrit

854

1 year ago

ebayContributor

asked in Ingest

The bcp client received an invalid column length for column ID 1.

Hi all, I create a datatable from Json file with REST data source but got an error as below. ‘The bcp client received an invalid column length for column ID 1.’ Since the the column is zero based I look for the data in second column in my json file. I have ensured that the first and second columns are not read in the RSD file. However, the problem still persists. I believe the issue may be due to the data in this column being too large. This is API: https://archive-api.open-meteo.com/v1/archive?latitude=51.51&longitude=-0.13&start_date=2018-01-01&end_date=2023-04-15&models=best_match&daily=weathercode,temperature_2m_max,temperature_2m_min,temperature_2m_mean,apparent_temperature_max,apparent_temperature_min,sunrise,sunset,shortwave_radiation_sum,precipitation_sum,rain_sum,snowfall_sum,precipitation_hours,windspeed_10m_max,windgusts_10m_max,winddirection_10m_dominant,et0_fao_evapotranspiration&timezone=Europe%2FBerlin How can I solve this issue? Thank you

6709

1 year ago

Page 7 / 10

Badge winners

Emiel van Schaikhas earned the badge Certified Solutions Architect
pkaustubhahas earned the badge Certified Solutions Architect
avanrijnhas earned the badge Ingest Expert
danielhas earned the badge Deliver Expert
p.ruijterhas earned the badge Jobs & Executions Expert

Show all badges

Terms and Conditions Cookie settings

Sign up

Already have an account? Login

Login with SSO

SSO login

Username *

E-mail address *

Full Name

(Private)

Only you and moderators can see this information

TX Partner / Customer *

Company

(Private)

Only you and moderators can see this information

Password *

I accept the terms & conditions

loginBox.register.email_repeat

Login to the community

No account yet? Create an account

Login with SSO

SSO login

Username or Email

Password

Remember me

Forgot password?

Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.

Enter your e-mail address

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.