Skip to main content
To homepage To homepage

Prepare

Ask questions about Prepare Instances in TimeXtender Data Integration

  • 236 Topics
  • 1,059 Replies
236 Topics
A
Problem Solver
aftabpProblem Solver
asked in Prepare
Creating unique IDs with NEWID()

I have two tables with no unique identifer. There a two posts that mentions NEWID() as a way to create a unique identifier, so it seems that it can be used:How to add a default value to a unique identifier field – TimeXtender SupportHow to Add a Default Value to a Unique Identifier Field in TimeXtender Classic | CommunityI have created a filed in each of my table and made it Primary Key.My question is:What happens if we Deoply and Execute on DSA table with a column that creats uniqe id with NEWID()? Does it create new IDs that are different, or does the table keep the IDs generated before? 

41
rory.smith
TimeXtender Xpert
A
Visitor
ArniVisitor
asked in Prepare
Settings related to Incremental load

I’m planning to change a history table in DSA into an incremental table without removing the history and I want to make sure that I understand some of the settings correctly. The ideal outcome for me is the following:New data added incrementally without the history being maintained Regarding the old history: I want to keep all rows with [SCD Is Current]=1 I don’t want to keep rows with [SCD Is Current]=0 Here are some questions I have about options in Table Settings > Data extraction :Does it make sense for this scenario to have the boxes checked for “Truncate raw table before transfer” and “Empty raw table after data cleansing”? What is the best approach for me to take with the delete options (“Don’t handle deletes”, “Use soft deletes” and “Use hard deletes”)? If I’m fine with new rows being overwritten can I then create some kind of incremental rule to make sure that old history is changed (or at least not for [SCD Is Current]=1)?Thanks,Árni

241
andrew.gebhard
TimeXtender Xpert
T
Starter
Tim TurnerStarter
asked in Prepare
Prepare Instance: Read from an existing Lakehouse

Hello,Is it possible to Extract data from an existing Lakehouse into a Prepare instance without needing to move it into the TimeXtender Ingest instance? The use case is to avoid replicating the data again, if the customer has already ingested data into a data lake or Lakehouse.Thanks in advanceTim

80
T
Starter
sierd.zuiderveld
Contributor
sierd.zuiderveldContributor
asked in Prepare
SCD Surrogate Hash Key is non-deterministic.

Hi,I obtain a deployment error when I try to make my table historic. Computed column 'SCD Surrogate Hash Key' in table ' <mytable>' cannot be persisted because the column is non-deterministic. I think this has to do with the excessive amount of columns in the table: 695 columns, all varchar(2000). What confuses me is that when I set the hashing algorithms to debug, I still get the same error  I don't have any transformations on this table, its a straight copy from to ODX.  Is there a limit on the amount of fields a History table can have? 

1726
M
Contributor
M
TimeXtender Xpert
MuddiTimeXtender Xpert
asked in Prepare
Classic and statistics

Hello Timextender and communityI’m looking to understand how Timextender Classic deals with collumn statistics that aren’t tied to indexes. Our project is built on Azure SQL DB.Obviously TimeXtender automatically rebuilds indexes as part of its data cleansing procedures. However, rebuilding indexes only updates statistics on the indexed columns—not on the rest. Performance  relies heavily on accurate statistics for query optimization (especially when MAXDOP isn’t explicitly set), this leaves me wondering: Does TimeXtender have any built-in process or configuration to update statistics on non-indexed columns? If not, are there recommended best practices for managing these statistics in a TX solution—either as part of the deployment or as a post-process step? After dealing with a sudden drop in performance, we’ve observed performance improvements after manually updating statistics in the entire project. This takes 16 hours though, so I’d like to ensure we’re not missing any built-in ca

201
rory.smith
TimeXtender Xpert
Søren Sørensen
Contributor
Søren SørensenContributor
asked in Prepare
Max value of multiple columns in Lakehouse

I’m looking for a TimeXtender Fabric Lakehouse (pyspark.sql) expert 😀I’ve got a SQL transformation that selects the maximum value of 10 columns:(select max(val) from (values ([Column_1]), ([Column_2]), ..., ([Column_10])) as [Values](val))How do I achieve this in a Prepare Lakehouse custom transformation?I could build the mother/father of all massive case statements, but I’d prefer something simpler and more elegant … if possible!?

321
rory.smith
TimeXtender Xpert
Paul McLeod Bespoke Analytics
TimeXtender Xpert
Paul McLeod Bespoke AnalyticsTimeXtender Xpert
asked in Prepare
Create Sandbox Ingest in V21

I have a client that wants to explore a significant rework of their Prepare instance. They want to duplicate their existing Prepare instance in Dev to create a sandbox instance. Is there a recommended approach to achieve this? They are using version 6898.1

302
rory.smith
TimeXtender Xpert
rvgfox
Problem Solver
rvgfoxProblem Solver
asked in Prepare
Custom Editor

Somebody can explain how to use the custom editor feature?Is it possible to use SSMS to edit the t-sql scripts?How must be setup it? 

774
rvgfox
Problem Solver
P
Contributor
Paul BridgeContributor
asked in Prepare
How to Clone a Table From One Prepare Instance Into Another

HiLooking to quickly replicate an existing data pipeline in a new prepare instance. Is it possible to clone a table in one prepare instance into another, or somehow export a table from one prepare instance and import into another?I cannot see any option, so my guess is no, but just thought I’d ask.RegardsPaul

579
rory.smith
TimeXtender Xpert
R
Explorer
roland.peetersExplorer
asked in Prepare
Custom Data Selection in DSA layer in TX v20.x

I have a table X in the DSA layer from TX and I can't find the script to get the right output.  There is a field FCL that possible have multiple records with different YearFrom and YearEnd. FCL   YearFrom   YearEnd1        2010           20201        2021           20502        2010           2050  I need a data selection rules that only imports from the ODX layer the FCL with the highest YeardEnd group by FCLFCL   YearFrom   YearEnd1        2021           20502        2010           2050  Is there anyone who has a suggestion. The output from is table is userd as lookup table for another table in the same DSA layer. 

365
N
Contributor
A
Problem Solver
aftabpProblem Solver
asked in Prepare
Deploy running for over 20 minutes

This is a simple Deploy and its been running for almost 20 minutes.Whats wrong here?  

678
Christian Hauggaard
Community Manager
L
Starter
leo.noharetStarter
asked in Prepare
Help with Primary Keys and Incremental Load in Table Inserts

Hi, I am facing an issue with duplicate records when using incremental load on a table that receives data via a table insert from another table in the same data area. The incremental load selection rule is based on DW_TimeStamp, and there are three PK in the target table.The issue is that the primary key does not seem to be enforced in the target table, as I am getting duplicate rows in the table. I tested this with another table where data is inserted through a table insert (but without incremental load), and I still see duplicate values for the PK fields and despite setting the table option to "Use instance setting (error)", no error is thrown when duplicates occur.  I have tested this behavior in TDI 6935.1 and 6926.1, and the issue persists. My questions are:  How do primary keys work with table inserts? Should they prevent duplicates, or do they not function as expected when using table inserts? Can incremental load be achieved using table inserts? If so, what are the necessary

957
L
Starter
avanrijn
Contributor
avanrijnContributor
asked in Prepare
Direct Read option latest TX version

Dear Sir,In TX version 20 you had a configuration setting Direct Read.I cannot find this setting in the latest and the greatest version in TimeXtender.Question: Is Direct Read still available ? How can I set this up ?Thanx in advance for looking into this. The "Direct Read" option in TimeXtender is particularly useful in scenarios where you need to transfer data between different data areas within the same physical database but using different schemas. Here are some specific scenarios where you might use the Direct Read option:  Large Data Volumes:  When dealing with large tables, Direct Read can be more efficient as it allows for direct data transfer within the same database, reducing the overhead associated with external data movement tools like SSIS or ADO.NET.  Incremental Loads:  Direct Read is beneficial for incremental loads where only a subset of data needs to be transferred based on specific criteria (e.g., timestamp or modified date). This can help in optimizing performanc

472
Christian Hauggaard
Community Manager
J
Contributor
JogchumSiRContributor
asked in Prepare
Columns with source data type money behaving incorrect

Hi All, Hopefully you can help me with this one. We have a source table from our POS systems containing sales transactions. There are a few value columns we transfer that are stored as a money data type and we see some weird things happening with the values of the columns. Differences of 0.01 up to 0.0001 are appearing from nowhere in the transfer from ODX to DWH and i don't know how this i happening.The columns that have money data type are called ‘Bedrag', ’Korting’ and ‘NettoBedrag’ Take transaction 52. In the source the values are like this: After transfer with ODX, i looked in the parquet file using ParquetViewer and that looks like this: Still the same, nothing wrong here.I imported the table without any modifications in TimeXtender and it looks like this: After a transfer the data looks like this: If you look closely we lost 0.0001 on the first record ‘NettoBedrag’. Also wel lost 0.0001 on the Bedrag of the second record.I've tried a few things to mitigate this. Different numeri

22019
Christian Hauggaard
Community Manager
P
Starter
philippus.asefawStarter
asked in Prepare
Exclude DW_ID from Primary Key on Database

Hey everyone,I have the following question:Is it possible, at the database level, to set the primary key to the one selected in TX (e.g., "Umkod_ID") instead of the default "DW_ID"?The issue arises because we access our table via PowerApps, and since the app existed beforehand, references in the code were already predefined. Currently, the app's primary key is set to the "Umkod_ID" field of the table. However, in our database, the primary key is set to the system field "DW_ID." Since this table is only loaded once and no transformations occur, the system fields are not particularly important to us in this case.We resolved this issue by manually populating the table, but I am curious if anyone knows a workaround that could be applied to similar cases in the future.

622
P
Starter
P
Contributor
PieterQContributor
asked in Prepare
Join multiple timelines from DSA to dimension

Case:In the DSA I have two tables (Employee and Employee Addresses).The fist table has history in it from the source system (so no TX  history active).On the second table we have TX History active (Type II) . It is not privided by the source system. When joining the tables to a dimension, the timelines must be combined.In what way can this be achived in TX? Emp_ID RegistrationNumber ExpDate ValidFromDate ValidToDate SCD Is Current       109030 39935270430 4-7-2020 1-01-00 1-10-24 0       109030 39935270430 4-7-2029 2-10-24 31-12-99 1                                           Emp_ID ZIP City SCD From DateTime SCD To DateTime SCD Is Current       109030 1625 HV Hoorn 1-01-00 31-05-24 0       109030 1000 AB Amsterdam 1-06-24 31-12-99 1                         Result of join:                 DWH_ID Emp_ID RegistrationN

532
Christian Hauggaard
Community Manager
bas.hopstaken
TimeXtender Xpert
bas.hopstakenTimeXtender Xpert
posted in Prepare
Batch cleansing option - best practices, tips and tricks

Hi all,I'd like to start a discussion about the batch cleansing option. You can find this option in: Table Settings → Performance → Enable batch data cleansing. This option is disabled by default. I'm looking for best practices when you should use it and how to determine the best batch cleansing size.I normally enable it on big tables (5.000.000 + rows), this reduces the disk space used for logging. But in some cases it also improves performance. It can result in faster loading of a table. However, I'm always struggling determining the correct batch size. I know the answer will be ‘it depends’. But I'm trying to find some best practices for this option.For now I start at 250.000 rows for batch cleansing and increase with 250.000 rows. I write down the timings and pick the best setting. This setting depends on the number of rows and number of transformations in the table. I'm curious if there are any practices or tips and tricks for this?

7083
rory.smith
TimeXtender Xpert
B
Contributor
bjornContributor
asked in Prepare
Updated TX 6814.1 with strange behavior for incr tables in prepare (desktop)

Hi,I updated our TX version to the newest 6814.1 I have added many tables in TimeXtender Ingest Server with incremental load. Why do I have to change all incr. tables in "desktop app" to explicit incremental load, can't you use “Automatic” anymore? Error msg: when executing with automatic on a table with incr setup from the datasource I get this type of error “Invalid object name 'shema.table_I'.” Something I haven't done right... regards,Bjørn A.

1016
devin.tiemens
TimeXtender Xpert
Roy V
Problem Solver
Roy VProblem Solver
asked in Prepare
Object reference not set to an instance of an object.

I am deploying a project to Snowflake.On a table I get the following error:Object reference not set to an instance of an object.As read in another ticket I did a ‘Save and reload' but unfortunately that did not solve the issue.It;s on deploying the data cleansing script however it does not show me the script in the deployment log, I only see the error message. Is there an easy  way i can see what lookup column is causing the issue ? 

19211
Roy V
Problem Solver
T
Participant
thzmanParticipant
asked in Prepare
Object reference not set to an instance of an object.

Hi!Getting this error when I try to open the Dev instance. Test and Prod is OK.I could open an earlier version of the project but after I made some changes and tried to save and deploy, I first got a warning that said that this was not the latest version. When I chose to save anyway, I got the same error. Is there some way to “force save” to make this earlier version the latest one and get rid of the error?

661
Thomas Lind
Community Manager
D
Contributor
dsvartzmanContributor
asked in Prepare
Multiple ODX for different enviroments

Hi,I was wondering  if it's possible to use multiple ODX instances for different enviroments (DEV/ACC/PRD).I know that at 20.10 you could only use a single ODX as after multiple enviroment transfer all mapping to ODX would be lost when using multiple ODX instances Is this still the case? Is this issue solved in next-gen?  

11115
D
Contributor
rvgfox
Problem Solver
rvgfoxProblem Solver
asked in Prepare
Buid dependency Graph

This is the first time that I see it: What it is?

411
Christian Hauggaard
Community Manager
Roy V
Problem Solver
Roy VProblem Solver
asked in Prepare
snowflake depoyment message 'skipped - uses unsupported feautures' -> apparantly key stores

I am deploying a project to Snowflake and see a lof of tables are skipped.Then we get the messages below (as an example) Apparantly the tales are skipped as key stores / supernatural keys  are not supported .Is this a feature to become available on short notice as our entire datawarehouse uses these?Besides that, is there a way I can see what any other unsupported feautures we have?

525
Roy V
Problem Solver
P
Contributor
Paul BridgeContributor
asked in Prepare
Issue with Data Area When Attempting to View Report / Errors

HiI have an issue whereby I get the following message when running the ‘Report / Errors’ menu option in one of my data areas. However, when I click OK I do not get any further information to help me identify the issue. Please advise what I need to do to troubleshoot this further.TimeXtender Data Integration version 6745.1Thanks

10012
daniel
TimeXtender Xpert
F
Contributor
FenchContributor
asked in Prepare
Persistent mapping error message on syncing MDW with ODX

Dear all,When synchronizing the mdw with the odx through right mouse click on the odx and selecting “Synchronize (...)”, the error messages as attached always show up. “Synchronize with Remapping (...)” only leads to complete removal of the field mappings in the subject tables.As there are other ways to sync the mdw with the odx, it’s still possible to work with the subject tables in the mdw. Nevertheless i wonder what causes the error messages and would like to eliminate them along with the risk of having to redo all work on the tables in a later stage.

894
F
Contributor

Badge winners

Show all badges
Terms and ConditionsCookie settings