Data Sources Q&A

Hey community,I am trying to set up pagination for my REST connector to ADP. I need to use the CData parameters $skip & $top.My first call should be: https://api.eu.adp.com/hr/v2/workers?$skip=0&$top=100Then I need to increment the $skip parameter by 100 so the next call would be: https://api.eu.adp.com/hr/v2/workers?$skip=100&$top=100In need to end when I receive an empty response. Anyone that has experience with this?

Hello,for one of our customers we are using the “Odbc Data Provider” so that we can a local ODBC with a specific driver called "Progress OpendeEdge” (Source: ProFashionALL). We need to add Additional Connection Properties for this datasource. In the older TX versions (< v20) we could add these Additional Properties. This is missing in the new TX version so we can't connect to this source at the moment. This is the main source and crucial in this project. Anyone ran into this issue or has a workaround?There is a workaround but this must done at the source where they store the credentials in the “_user table". I hoped there would be a solution at our (TX) side. Hope to hear.

(Using Timextender 20.10.35.64)I have 2 date fields on a table. I want to load the row if one OR the other fields have changed in the data source. How do I do that?I tried doing this (see under)  but that caused duplicates because data had already been loaded (when only the loggedat field was the incremental field)  If I do this (see under) will it only load if both of the fields have changed? 

Hi everybody,Our client uses a SOAP API from a supplier. While TX supports SOAP the extraction requires multiple steps and string manipulation. The complete process is:For each table in the list of tables: Call the first SOAP endpoint to retrieve the names of the fields in the table Call the second SOAP endpoint to retrieve a page of data for the table Split the data into individual rows by splitting the string on the ^ character Split the rows into individual colums by splitting the string on the | character Fetch until the number of rows returned is less than the page size Move to next table We created a custom data provider for the TX custom data connector. This works fine in TX 20.10, but is not supported in 6221, so we need to create an alternative. I’m aware that you can use RSD files for more advanced logic, but as far as I can see this doesn’t include an option to

Hi all,We are trying to load CSV files into the ODX using the CSV data source. However we are getting errors on fields it is trying to interpret as int (housenumber), but some are filled in as 12-18 and therefor failing. We have tried to use the override data type option, but to no avail. What can we do about this?Kind regards,Maarten

Good DayI am setting up a new Data Source in TimeXtender. This dataset is very large and would like to eventually have this running as an incremental, but I have the task of importing the last 5 years of data first.I have tried to set this up with an Incremental Table setup in TX, but I keep getting a timeout setting. Iam wondering if I could set this table up as Simple for the historical load and then swap to Incremental but am worried my data would be cleared on changing the table type. Does anyone have any experience with loading in data with a similar scenario?Kerry 

Hi,My client wants to connect to the App Store Connect API (https://developer.apple.com/documentation/appstoreconnectapi) but we are having trouble with getting the authorization to work.Calls to the API require JSON Web Tokens (JWT) for authorization, specifically using the “ES256” signature algorithm. We tried using the CData Rest connector but found in the documentation that “...the JWT signature algorithm cannot be set directly. Only the RS256 algorithm is supported”  https://cdn.cdata.com/help/DWH/ado/pg_oauth.htm#jwtHas anyone experience connecting to the App Store Connect API?

Hi FriendsI have used TX CDS to connect SQL Server because It was a best practice recommendation, but now I don’t know if CDS will be deprecated, What is the best connector for SQL Server? CDS will be deprecated? RegardsIgnacio

We are struggling with nested calls in a POST API with custom body. We don't change the URI on every input, but need to change the body on every request:We use Cursor based pagination using rows@next for pagination. And as it seems you can only use one pagination per rsd, we can only get one iteration using nested calls.Query slicers only seem to be able to slice URI's (https://cdn.cdata.com/help/DWG/jdbc/pg_queryslicer.htm). In our example, we are extracting ReservationId's from a Reservation request and need them as a body input for ReservationItems in batches of 1000.Any help would be appreciated, let me know if you

We’re facing issues with CData REST API connector when requesting an API for a large dataset. TX sends the request, but seems to never receive a response. The execution fails when it hits the timeout limit (currently set to 3600 seconds).In the logs (verbosity 4) I see that nothing happens between the time that the request was sent and the moment of timeout. I put the same URI, headers and parameters in Azure Data Factory. After 9 minutes it receives a response of 204MB and 176K rows. In Postman, I received a response after 8 minutes. While I agree that it might be better to somehow make smaller API request. In fact, when I limit the date range to only this year, TX get

We have an API in which we have to pass a POST request with a request body to obtain the data. In Qlik Sense and Postman we have a basic connection established. However, there you can easily specify that the call should be a POST and you can also easily fetch the request body. In the CDATA REST connector in TX, we do not know where to fetch this request body in the connection. The demo endpoint we are trying to connect is:https://api.mews-demo.com/api/connector/v1/configuration/getwith the request body:{"ClientToken": "E0D439EE522F44368DC78E1BFB03710C-D24FB11DBE31D4621C4817E028D9E1D","AccessToken": "7059D2C25BF64EA681ACAB3A00B859CC-D91BFF2B1E3047A3E0DEC1D57BE1382","Client": "NameOfYourCompanyOrApplication"}Any help would be appreciated.

Hi all,I have a customer at which the data & analytics team has to report which data (tables & fields) is loaded into the data lake with ODX server for a certain source. This is because the source contains GDPR data and the data & analytics team has to prove to the legal team that they aren’t loading GDPR data into the data lake and further upstream.Unfortunately the TX documentation feature documents from the DSA and onward. The DSA documentation isn’t feasible for this request since we’re renaming fields and adding transformations. Is there a workaround to get this overview for a ODX server connection?Kind regards, RogierTimeXtender version: 20.10.39

Hi,I have problem with pagination and recursive queryI have rest api endpoint:https://api.procountor.com/api/invoiceswhich returns invoice headers as"id": 6591273,"partnerId": 1208831,"type": "PURCHASE_INVOICE","status": "PAID","invoiceNumber": 16...etc.pagination works fine, I get every invoice headers (thousands) when pagination is set like this in RSD-file:<api:set attr="EnablePaging" value="true"/><api:set attr="pagenumberparam" value="page" /><api:set attr="pagesizeparam" value="size" /><api:set attr="pagesize" value="100" />But the actual details of invoice comes from endpoint like this:https://api.procountor.com/api/invoices/6591273So, I have to iterate thru one by one every invoice id that I get from https://api.procountor.com/api/invoices I have followed instructions from herehttps://legacysupport.timextender.com/hc/en-us/ar

Hi,I have a problem with data catching from JSON file. The file looks like this:And I would like to get values from data field but only for en_US locale.I was trying to add filter in RSD file like below but it did not help:Can that type of filtering be used there?Greets, Aleksei

Hello,To implement pagination in the API endpoint "https://domainname/api/v2/search/tickets?updated_since=2023-01-01T02:00:00Z&page{}", you need to provide the page parameter in a nested format. I prepared a custom RSD file specific to this API after reading the "Nested parameter" article. I have created a loop starting from 1 in the given code.  However, I'm not able to see any data. I'm not receiving any errors during the transfer and synchronization processes. You can find the RSD file attached.I would appreciate in resolving this issue.Best Regards

I am looking for a way to find all tables in a perspective in a way I can analyse the differences between perspectives. In SQL in the metadata repository I would like to join ProjectPerspectives to the Datatables to find all Perspectives with certain tables, find which tables are not in a perspective and are therefor missed while loading data etc. I do not see an identifier thats Unique between the two so I am hoping one of you does :).  Thanks in advance, Remco

Hi TimeXtenderI have an urgent request (!)I am trying to set up a Script action (which I will later use in a table as a post-script) that executes a POST CURL in a different tool.I have tried to execute the POST CURL in Postman and it is working.Now I want to do the same from TimeXtender.I have an URL and a Token.How can I set it up? Looking forward to hearing from you.Ismail

We were able to get this CSV file data source stored in an Azure Data Lake Gen2 storage container to work, but thought it may be useful to share, as we found that that the setup worked best when we were careful not to enter any other settings, but instead just entered those settings that were needed and described below. Source File: CSV.Storage Location:  Azure Data Lake Gen2 storage containerAuthentication Method: App Registration added to the storage container via a role assignment of "Storage Blob Data Contributor". Started by adding a new data source and chose the following:On the data source details page, entered the information for the 9 items outlined below, being careful not to enter information in any other boxes and just updating the n

Hello,I am trying to extract data from azure AD. I am at the point where I can get data using the microsoft Graph API, and have it loaded into Timextender.The issue I am facing has to do with pagination. Microsoft graph API returns an url containing information of the next 'page’ of data. According to documentation I should adjust my RSD file as follows:  <api:set attr="DataModel" value="DOCUMENT" />  <api:set attr="URI" value="https://graph.microsoft.com/v1.0/groups" />     <api:set attr="EnablePaging" value="true" />  <api:set attr="pageurlpath" value="/json/@odata.nextLink" />  <api:set attr="RepeatElement" value="/json/value/" />This does not however loop over the pages, but still only gives me a top 100 of groups. I have a feeling maybe I need to escape either the @ or the . in the pageurlpath. Or am I missing something more obvious? Kind regards,Rutger

Hi,  I am wondering if it is at all possible to use data stored in a TimeXtender table as variables in a REST API call. I have two use cases that hopefully explain what I am trying to achieve A combination of two REST API's where one requires the output of the other: The first one retrieves all possible classes that can be attended, the second one retrieves all people enrolled in such a class. But the class name is a parameter in the header. I'd like to store the class names in a table and then loop through them (not sure if that's the way to go) and request all enrolled students. A Machine Learning functionIn, for example, Azure Machine Learning Studio I am able to host a model using a REST API. Then, I can call the model and by inserting the parameter values I receive a predicted value. I'd like to do this in the data warehouse since it saves time compared to requesting it every time a dashboard reloa

Hi all,I need to pass multiple custom headers in a RSD file:Host: xxxxx.cloud.xxxxx.com Authorization: Bearer dfdflejlflefdjeljfeoj Content-Length: 26Is this the correct way to do this? <api:set attr="Header:Name#1" value="Authorization" /> <api:set attr="Header:Value#1" value="Bearer dfdflejlflefdjeljfeoj" /> <api:set attr="Header:Name#2" value="Host" /> <api:set attr="Header:Value#2" value="xxxxx.cloud.xxxxx.com" /> <api:set attr="Header:Name#3" value="Content-Length" /> <api:set attr="Header:Value#3" value="26" />Greets, Rogier

We have a project perspective called 'Reference' which contains ODX tables. When we are executing it through the execution package it is throwing an error  but when we try executing manually it is executing successfully.the file format which we are extracting is  XLSX. can you please help us to resolve this issue.

Is it possible to connect to ActiveMQ (https://activemq.apache.org) queue (as a source)?I use c# client app to fetch the data from the queue and was wondering if it is possible to use TimeXtender instead. 

I have a setup of a TimeXtender 20.10.40.64, ODX, ADL, ADF, Self hosted IR which I get to work just fine. I need to extract data from an IFS10 source, Oracle database, and it works until syncing then it seems to start an endless loop and never finish or gives an error. In TX I havesetup a new data source, Azure Data Factory - Oracle included all connection info information In Select tables I click “Search” and get all the tables available in the Oracle DB I Include a couple a tables Setup a transfer task Syncing starts running and I can see in ADF that is is running but it never receives any data from the data sourceI have tested from a SQL data source with ADF and it works just fine, both sync and transferI have in ADF tested to preview tables in the dataset created by TX and I can preview tables from Oracle DB just fine. I have even tried to create my own pipeline in ADF to copy dat

Hi,Is it possible to connect to several databases that are on the same SQL server with one data source connection?We have tried to connect to multiple databases on the same SQL Server using the provider “TimeXtender SQL Data Source 17.0.1.0”.We tried leaving the Database parameter blank, putting in *, and writing a comma-separated list of databases, but neither way works. Has anyone had any success doing this with this connector or another?We’re using version 6221.1.Thanks!