Ingest

I am banging my head against the wall for this one for some while now. I would appriciate some help.Here is the RSD-file<api:script xmlns:api="http://apiscript.com/ns?v1" xmlns:xs="http://www.w3.org/2001/XMLSchema"> <!-- See Column Definitions to specify column behavior and use XPaths to extract column values from JSON. --> <api:info title="data" desc="Generated schema file." xmlns:other="http://apiscript.com/ns?v1"> <!-- You can modify the name, type, and column size here. --> <attr name="bildeLinke" xs:type="string" readonly="false" other:xPath="/json/data/bildeLinke" /> <attr name="bygning" xs:type="string" readonly="false" other:xPath="/json/data/bygning" /> <attr name="bygningsId" xs:type="string" readonly="false" other:xPath="/json/data/bygningsId" /> <attr name="del"

Hello,TimeXtender: 20.10.31.64ODX:  20.10.31We have a case where a customer has to move from SQL database to a OneLake Parquet as the data extraction from one of their systems. We need to find a way to get this data into TimeXtender.I could find that there is the possibility for one lake parquet connector, but it seems tied to the new version of TX as I can’t find that connector in our version of tx/odx. Is there any possibility for us to do it another way while on our version? Thank you,Victor 

Hi, I'm using the TimeXtender REST Data Source (5.0) in combination with dynamic values.The API endpoint needs a filter (Period) in order to work. As you can see in the last screenshot the dynamic value query refers to a table in TX.This table contains all the yearmonth combinations of 2023: 202301, 202302, 202303 etc. When the datasource is executed in Timextender desktop it only returns the following months: 202301, 202302, 202306, 202312.It seems like the filter does not recognize all the yearmonth combinations or skips most of the months. Has anyone had the same issue or experience with this configuration?In the screenshots you can see the curent pagination and parameter configuration.I’m also using table flattening but that doesn’t seem to make a difference. Thank you. Kind regards, Joris de Leeuw 

Good morning Team,The database [ODX_COL_PROD] has 2.0 TB of data, and we need tu purge it.Can someone give or send me informacion/documentation about the purge data task.Thanks for your help.Ignacio 

We are trying to follow the setup guidelines so that we can get to grips with TimeXtender leveraging Fabric. We are having issues at the very beginning in trying to assign our App Registration ID to the Fabric Workspace. It just wont show as an option in the list when trying to grant membership access to the Fabric Workspace. If we specifically type in the name of the App Registration ID then we get an error below the dialog indicating that we “can’t use invalid or duplicate emails”. We are obviously missing something, anyone have any ideas?

Hi,In version 20, we have a setup with two environments, a dev and a prod, that ingests data via a business unit. We are limiting the data ingested into dev by using the “Environment name” project variable together with a date filter, much like what is described here:https://legacysupport.timextender.com/hc/en-us/articles/360032668012-Creating-Dynamic-Data-Selection-Rules-using-Project-VariablesWhat is the equivalent of doing this in the latest version of TimeXtender? The client’s setup has one ODX and two DW instances (dev and prod). Their need is to limit the amount of data ingested into the Dev DW instance. Is this possible with the features available in version 6436.1?Best,Pontus Berglund

Hi all,I’m using the CData Connector for Parquet files and take files from a SFTP server. The files always contain the data from the last day (with some overlap).I made the following setup:Primary key Incremental Load with updates The first load is always ok, but then there is no new folder in the DataLake created? What I’m doing wrong?ThanksMichael

Hi community,I have this strange thing I cannot explain. I have a data connection that has been working properly for months. Since the beginning of May the transfer task results in a time out error, where after 1 hour the command time out is reached. I have tried everything to resolve this error; nothing is working. I tried with one really small table and 1 thread (of course setting the time out to a few minutes 😉); same error. The strange thing is; I have cloned this exact data connection and tested the transfer task running this exact similar table over 1 thread. This works without any problems. I have also added several other tables, also without any problems I fetch the rows of all tables within a few seconds. Does anybody have ANY idea what might cause this extremely weird behaviour? 

Hello,My transaction journal tables unfortunately don't have a useful update/modified datetime column, only a posting date or similar. I'm looking to incrementally load these tables, since they are my biggest tables. I can do this partially based on the Posting Date column which won't change after creation, however I have some columns, e.g. BalanceDue which will mutate once an Invoice has been (partially) paid. Hence, I also can't use an offset incremental loading rule for posting date, since I won't catch these changes.Anyone got an idea to still minimize loading times on these tables?

Hello,I started seeing this error message multiple times a day in the event viewer.Does anyone know what may be causing it? Synchronize executions failed:System.Data.SqlClient.SqlException (0x80131904): Violation of PRIMARY KEY constraint 'PK_ODX.ExecutionLogs'. Cannot insert duplicate key in object 'ODX.ExecutionLogs'.(...) Thanks 

The people before me defined certain variables in TX. One of these is [vDatabase] which is futher used in a stored procedure in ODX:Select....FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_CATALOG = '[vDatabase]' AND RIGHT(TABLE_NAME,2) NOT IN ('_L','_M','_T', '_R', '_B') AND TABLE_SCHEMA NOT IN ('dbo','view', 'XLS') AND COLUMN_NAME IN ('modifyDate','last_update','Registrert_Dato') END The variable is also used in SQL snippets in ODX, DSA and MDW:DECLARE @sql NVARCHAR(MAX)SELECT @sql = COALESCE(@sql + ' UNION ALL', '') + ...... FROM %s'+ CHAR(10) , TABLE_SCHEMA, TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, TABLE_SCHEMA + '.' + TABLE_NAME, COLUMN_NAME, COLUMN_NAME, COLUMN_NAME, COLUMN_NAME, COLUMN_NAME, COLUMN_NAME, TABLE_SCHEMA + '.' + '[' + TABLE_NAME + ']' )FROM INFORMATION_SCHEMA.COLUMNS WHERETABLE_CATALOG = '[vDatabase]' AND RIGHT(TABLE_NAME,2) NOT IN ('_L','_M','_T', '_B') AND TABLE_SCHEMA NOT IN ('dbo','view', 'XLS')EXEC(@sql)Howev

Hi,I have set up a Storage Management Task with only the Rollup option ticked, and filled in the right ADF credentials. One of my source tables has been selected for this task.TX says the runs are successful and they finish in a fraction of a second, but nothing happens in the storage account. The version folders contain the full load file + all the incremental files, as before. I have tried with many different settings for min and max rollup file size. I have not specified “folder (optional)”. I have tried with and without specifying integration runtime.No pipeline run seems to show up in ADF after triggering the task.

I have a REST datasource that uses pagination with URL paramater page. Fairly simple.The next page (if there is one) is returned not as a header but as an element in the json result:  So, to enable paging I have to set up something like this: And:I have two questions on this: I get an error on the XPath Value saying ‘Expression must evaluate to a node-set.’ . How do I determine the correct XPath to the json element? And my second question is how do I make sure the complete URL is replaced instead of only the /api/… part?

hi, i have a exclude columns made like“ exclude if schema <all” and table <all> and column contains “ plusg”  for 1 table i want a specific include likeinclude if schema <all> and table equals “plusgmoc” and column equals “plusgtest” it seems that the column isn’t added, i have done a sync and a new transfer. I look in the generated parquet file and can’t find the column.Is this expecten behaviour? how to solve? 

Hello,Our team has recently started shifting from using Business Units as our data ingestion approach towards ODX server.Our project contains 3 environments (Dev, Test, Prod). All of our data providers give us 2 tables per dimension/fact (e.g. dim.user and dim.user_qa etc.) and we store them in 2 separate databases (Datastore and Datastore_qa respectively) with exactly same names (dim.user).We have a following setup in place implemented using “Environment properties”:Environment Source database Dev Datastore_QA Test Datastore_QA Production Datastore  This way contents of dim.user table are loaded from Datastore_QA database on Dev and Test environments, and from Datastore database on Production environment. Is there any way to achieve the same result utilizing ODX server as data ingestion approach? Hopefully this makes sense, thank you! 

Hi, we are loading a table from our source data base. A column which is part of the key, contains values that are similar but not the same. The difference is in a trailing space. Now, when TimeXtender reads this dataset, it claims a violation of uniqueness. The column is set to be VARCHAR(), which probably causes the system to trim the trailing space, resulting in a duplicate value.As the source system recognizes this as different entries, we want to keep both. How to solve this?  

One of our client's has a problem with the version mismatch between odx and desktop application. They are running on 6645.1 odx but some of the desktop versions are still on 6626.1 they get an error that the odx version is locking them on 6645.1 but it's expecting 6626.1. Is there a workaround for this or a fix? or is this a bug? I know the ‘easy’ solution is that all the desktops versions should be updated at the same time for everyone ones you also update the odx but that's not always possible.

Hi all,I need to extract data from a huge table in SAP (KONV) based on a subquery from another extracted table (VBRK). Any ideas on how to solve this using the XtractIS? I can’t use quey tables on this type of connector and I tried to put the subquery in the filter rows filter for “in list” which also didn’t work either. Any more ideas how to solve this? Many thanksKai

Hello,TimeXtender: 20.10.40.64ODX: 20.10.31We have an issue on certain source where transfer tasks result in “completed with error” or “failed” in the ODX.This lead to the question if there is anyway to capture this without going into the ODX and manually checking every transfer, as there is no way to set up any notificaiton.We need to someone capture this, as the loads within TX will run as usual without failing but not get new data from the sources in most cases. Has anyone found a solution to streamline this? Within Timextender or with a external solution?Thank you,Victor

I am working in a project and the execution is to mach slow,  It has take 35 minutes for 5 tables, that all of them has 10000 rows.    

Hello, I am using ODX server 20.10.47. I would like to limit the amount of records loaded by implementing following filtering rule on column [perakt] among others:LEFT ([perakt],2) > 20I am trying to achieve this in following way using “Filter Rows” options on data source but it does not work:Also if i click OK and load the data, the 2 upper filters are not applied, just because the 3rd filter does not work.Are there any ways of achieving this through “Filter rows” functionality? Or the only option is to create a custom query table?Thank you!

Hi, Since we upgraded to 6626.1 our TimeXtender SQL Data Source (18.0.1.0) type data source has been failing. We believe it is because 6626.1 expects a TimeXtender SQL Data Source (20.0.1.0). This either is a bug that needs to be fixed, or in the product update log / in the portal there should be some warning about this. If this was mentioned somewhere in a product update please share the link to the article.   

I'm using a FireBird database connection over ODBC.Syncing gives me the following error:System.ArgumentException: Column 'TABLE_SCHEMA' does not belong to table Tables.There is a column TABLE_SCHEM (no A). Can I override this somehow?Key Value Is Sensitive Schema Name TABLE_SCHEM false This does not work as additional option in the Data Source TimeXtender Desktop 6536.1 Kind regards,Thijs 

Hello,We are currently utilizing the new TimeXtender REST API 6.1 and leveraging its built-in paging feature to connect to an external REST API. We're not using an RSD configuration for this setup.I've encountered a challenge with pagination, specifically with the use of a parameter named "Cursor." This parameter is critical for the pagination logic in the API we're connecting to. Each row in the data returned by the API has a unique Cursor value, but for pagination purposes, only the first Cursor value from the current page should be used in the subsequent POST request to fetch the next page.Here’s how we've set up the Cursor parameter:Name: cursor Type: XPath Value: /TX_Autogenerated_Root/TX_Autogenerated_Element/data/arTransactions/edges/cursorThe issue arises because I need to ensure that only the first Cursor value from each fetched page is used in the pagination post body for the next request. I'm unsure how to configure TimeXtender to utilize only the first Cursor value for this

v20.10.35.64I just noticed there’s a bug in the current version of our ERP related to the UpdateDate & UpdateTS field which impacts incremental loading. Hence, I'm investigating the possibilities to set up a semi-incremental loading infrastructure. With my TX version I can also set up the following, which is limited:  My database has data ranging from 2013-today. I'm looking to find a way to load 2013-2023 just once and then only full load the data with a UpdateDate >= 01-01-2024. I'm thinking to achieve this the following way:Set up a Query Table for every table where UpdateDate >= 01-01-2024. Set up a Load task, LOAD_FULL which has the original tables Set up a Load task, LOAD_INCREMENTAL, which has the Query table with just data from 2024. Execute the LOAD_FULL task just once. Schedule & Execute LOAD_Incremental every 30 minutes. If I do it this way, it's not a classic incremental loading process.Can I still utilize TimeXtenders ODX/DSA incremental loading functionaliti