Ingest

Hi,I'm using the CData ADO.NET Provider for OData 2022 (22.0.8322.0) to extract Topdesk data.Due to performance reasons Topdesk made some changes in their feed. Resulting in that, since 19th june 2023, 2 larger tables (IncidentDurationBlocks and IncidentSnapshots) cannot longer be extracted without filtering on a date to make the dataset smaller.How can I add a filter for only one table in my extract?Can I create an OData RSD file?I get this error when I try to extract:System.Data.CData.OData.ODataException (0x80004005): [500] Could not execute the specified command: Unexpected response status code: 503.  ---> rds220I.IC: Unexpected response status code: 503

Hi,we have a customer running incremental loads multiple times a day in the ODX Server (using Data Lake Gen2).  We’re reaching the 10.000th incremental run.I’m wondering how the ODX server will handle that 10.000th increment (since the naming of the parquet files allows a number up to 9999).BR.Peter

Hi community, The CDATA Provider for LinkedIn 2022 contains a bug/error which causes it to call an invalid API URI for the connector’s table ‘OrganizationFollowerCount’. The appropriate URI for this table according to Organization Network Size - LinkedIn | Microsoft Learn is:https://api.linkedin.com/v2/networkSizes/urn:li:organization:1234567?edgeType=CompanyFollowedByMemberHowever, CDATA is calling the following:2023-06-12T11:31:21.793+02:00 2 [5|Q-Id] [HTTP|Req: 3] GET https://api.linkedin.com/rest/networkSizes/1234567?edgeType=CompanyFollowedByMemberWhich results in the following error message:2023-06-12T11:31:22.172+02

Hi all,We’re trying to troubleshoot some thorny networking issues, and I need to know more about how ADO.net data transfer works behind the scenes - specifically for old-school business unit SQL Server data sources, if that makes a difference. I know for sure is that data is read from the data source and written into the destination table in the staging database, but I’m fuzzier on what happens between the two places. To start off with, I have two key questions: We’re using a tool server to run TimeXtender, and a different server to host SQL Server and the TimeXtender databases. Is data routed through the that tool server when it is read via the ADO.net package? I assume that it is, but we need to know for sure. The other question I have is what happens after the data is pulled from the data source? Is it pulled into memory somewhere? If so, where? The source server, the tool server, the destination data server, or some combination of the three? Is the da

Hi team, In the business units and data warehouses, deletions in an incremental table are identified by system field IsTombstone = 1. You can use this field in a data selection rule to filter out deletions between the business unit and DSA. This field is not present in ODX Server tables with incremental and delete handling enabled. Is there a way to identify records that have been deleted from incremental tables in ODX Server? And if not, what is the purpose of enabling deletion handling in ODX Server? Kind regards,Andrew - E-mergo

I have made a new app registration in Azure for TimeXtender. Provide the rights exactlly as the old app registration but the secret is expired.Error in TX : AADSTS7000222: The provided client secret keys for app 'OLD_APP_ID' are expired.I filled in the datasource and global enviorment the NEW_APP_ID but the error still is there. Restarted the ODX server, removed the pipelines but still same issue. How can i force that it will use the new application ID becaused I changed it on the places it is used. (Also with the new secret ofcourse)

Hi! I am experiencing an issue with ADO.NET transfer times on incrementally loaded tables where the ADO.NET transfer takes the same amount of time regardless of the amount of new rows coming into the table. A full load of the table containing about 45 million rows takes about 25 minutes, on the next incremental load the load time is still the same with ADO.NET taking up around 24 of these minutes. Our current data flow is as follows: SQL server → TimeXtender → Azure elastic pool (this is where all of our TimeXtender databases resides)  Full load - 45 million rows Incremental load - 11 thousand new rows in the _R table

Hi community, I’m using version V6221.1. I’d like to filter all tables in the data source on column ‘dataAreaId’. I try to set this up using the following settings:  Do I really need to set this filter on every table I load from this data source? Or is this a bug?

I am using ODX version 6143.1. When loading a table from a source system, incremental load on a datetime column (with an offset of 7 days) works fine. But when I set up the incremental load rule with Detect deletes and Detect updates selected, the first incremental load (which is the initial load) runs successful, but a second load (which should be the increment) gives me a 404 HTTP error:Response status code does not indicate success: 404 (The specified path does not exist.). Edit: problem only arises when setting the Delete detection option. When I remove this checkbox, the 404 error disappears (so the Detect primary key updates option does work correct).

Hi, Is it possible to setup retry steps for ODX transfer tasks in the >6024.1 version of TimeXtender? If yes, where? If not, please add this as an idea. Greets,Devin

Dear Community,i'm working with timeXtender 20.10.22 and my source is a Business Unit with the NAV (Business Central) adaptor.I've switched in my project between the source being Production (BC) to Acceptance (BC).Now im getting this error and I cannot do any thing anymore. Also switching back does not work. Can anyone help me? Thanks! tableAccount is not contained.Parameter name: tableAccountDetails:tableAccount is not contained. ...Module: timeXtenderSystem.ArgumentException   at TimeXtender.DataManager.TableUsageMatrix.GetTableUsage(Table_NAV tableAccount, Account account)   at TimeXtender.DataManager.Adapter_NAV.GetExpectedDeployedObjectsPrivate(Table_NAV tableToDelpoy, Guid projectId, Guid tableId, List`1 sqlObjects, ProviderDestinationSql providerDestinationSql)   at TimeXtender.DataManager.Adapter_NAV.GetExpectedDeployedObjects(IDataAdapterTable table, Guid projectId, Gui

I am setting up TimeXtender to extact tables from SAP with Theobald. I have a selection of tables that I can succesfuly extract when I add just one table to a transfer task. However, when I add all relevant tables to a single transfer task I run into errors. The extraction seems to be succesful on the Theobald side but moving the data to the ODX storage gives the following error “Error while copying content to a stream”. I can figure out why the tranfer is working when I do it per table and gives an error when I do multiple tables.. Full error sample below: Executing table rest_mara_generalarticledata:failed with error:System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.ObjectDisposedException: Cannot access a closed Stream.   at System.IO.__Error.StreamIsClosed()   at System.IO.MemoryStream.get_Position()   at Syste

When loading regularly (every hour) using incremental load, the ODX seems to “lose” transactions from source. Next run of the incremental load, ODX does not catch these missing transactions. The work around is to full load data, but the this takes long time. And it renders the incremental load useless as it is not to be trusted.We are running 20.10.34. Does any of the newer 20 versions have a fix to this?Regards Mads

I am working on Sandbox Environment.I on using ODX 20.10.36.64.I executed tables on 25th and today 27th in ODX DataSource.What I noticed is in my SQL server new tables were created.Do I need to update any settings to resolve this?SQL server 

Is there someone who has a quick reference card how to create the connection with an exchange server using the CDATA connector.  We tried several settings with the server, but nothing works. 

I've got an API that returns JSON data. One of the fields contains a UUE encoded csv file. Is there a way to load this to the ODX without some external pre processing?Thanks,Gerrit

Hi all, I create a datatable from Json file with REST data source but got an error as below. ‘The bcp client received an invalid column length for column ID 1.’ Since the the column is zero based I look for the data in second column in my json file. I have ensured that the first and second columns are not read in the RSD file. However, the problem still persists. I believe the issue may be due to the data in this column being too large. This is API: https://archive-api.open-meteo.com/v1/archive?latitude=51.51&longitude=-0.13&start_date=2018-01-01&end_date=2023-04-15&models=best_match&daily=weathercode,temperature_2m_max,temperature_2m_min,temperature_2m_mean,apparent_temperature_max,apparent_temperature_min,sunrise,sunset,shortwave_radiation_sum,precipitation_sum,rain_sum,snowfall_sum,precipitation_hours,windspeed_10m_max,windgusts_10m_max,winddirection_10m_dominant,et0_fao_evapotranspiration&timezone=Eur

After connecting successfully to my SQL server and adding the tables & rows I am able to preview the data by Select Tables. The Synchronize task also completed successfully. However, when I execute the Transfer task it fails with 2 errors (see attachments). I can’t figure out the reason why it fails though but the error message suggests something that (The specified path does not exist). I tried to remove the File Container in Azure and sync+transfer again, but it does’nt make a difference.  

Hi,We are using the Excel Online connector which is authenticated with a service account and uses delegated permissions to access Excel files on Sharepoint. The idea is that all relevant files will be shared with this account and then loaded into our DWH. The required files are visible on Sharepoint when siging in with the user:But for some reason, they are not when using the Excel Online connector (with option Show shared documents = ‘True’). I know that the connector uses the /SharedWithMe (OneDrive) call to fetch shared items, since that is what I inferred from the logging.This call indeed retrieves no results through the Graph Explorer, but the files are visible through another, similar, call on the Graph Explorer (the one from ‘Insights’):

TX: 20.10.39ODX: 20.10.37ODX Storage: SQL Server DB on-premDSA/MDW: SQL Server DB on-prem We are successfully able to connect to a DB2 system and synchronize, we can also successfully use the Data Source Explorer to query tables. Every table we try to extract fails with the following error (schema/table/column names obfuscated): Executing table [schema].[table]:failed with error:edo220W.gm: The column [column] does not exist   at edo220W.ai.GetOrdinal(String )   at System.Data.CData.DB2.DB2DataReader.GetOrdinal(String name)   at System.Data.SqlClient.SqlBulkCopy.WriteRowSourceToServerCommon(Int32 columnCount)   at System.Data.SqlClient.SqlBulkCopy.WriteRowSourceToServerAsync(Int32 columnCount, CancellationToken ctoken)   at System.Data.SqlClient.SqlBulkCopy.WriteToServer(IDataReader reader)   at DataStorageEngine.SQL.SQLStorageEngine.TransferWithBulkCopy(SqlCommand destinationCommand, St

Is it possible to connect to an Amazon Timestream database (https://aws.amazon.com/timestream/) with the ODX?

I am trying to get data using REST API Data Source: ODXProvider: CData 2022Authentication: OAuthFormat: JSONOAuth Grant Type: ClientAuthorize OAuth SuccessfulTest Connection getting below error: RSSBus.RSBException.HTTP Protocol error. 400 bad requests    

Dear Support,I'm running into a weird error when I try to load customer data from the customer table of Navision. The error is: The numerical value is too large to fit into a 96 bit decimal.I use a query table to only get the columns I need. In this query table I use a where clause in which I exclude some customernumbers. But now I found out that I also need these numbers in my customer dimension. But when I clear the WHERE clause, the error comes up again.If I execute the same query in SQL Management studio it runs good and I don't see some strange values in one of the columns.Do you have any idea?

Hi I am just testing our V21. I have started with something simple, a 2 column CSV file with an integer key field and a text group code field. There are 18 rows.I can load it into the ODX fine (running on local SQL Server db) and I can preview the contents in the ODX “Select Tables” Pane.I can drag it to a Data Warehouse Data Area and it creates and maps the table fine with appropriate field settings.When I deploy the table in the Data Area that works fine.When Executing I get an error on the transfer which does not tell me what is happening. The error I get returned is as follows: Transfer table failedDetails:Transfer table failedModule: timeXtenderSystem.Exception   at TimeXtender.DataManager.StepTransferExecute.DoDataWarehouse(IStepSetup stepSetup, List`1 mergeODXTransferInfromation, VariableResolveObject dynamicResolveObject)   at TimeXtender.DataManager.StepTransferExecute.DoAbstractSte

Hi community,I have a client attempting to load a postgres database into ODX Server (TX version 20.10.x). The table contains a field with dates that are invalid for SQL but valid in Postgres, for example 20011/01/01. Upon transfer, the task completes with errors that it is unable to parse datetime value. All datetime fields have been converted to VARCHAR(MAX) using Override Datatypes and this is confirmed by looking in the ODX Storage in SQL Server Management studio. Since the error still presents itself, it seems likely that the error is coming a step earlier, during the CDATA conversion. Unfortunately, it seems that there is no way to generate RSD/schema files from the CDATA Postgres provider. Normally I would change the datatype there in these situations, since it seems to be going wrong in CDATA somewhere. How can I load this table? Kind regards,Andrew - E-mergo