Skip to main content
To homepage To homepage

Prepare

Ask questions about Prepare Instances in TimeXtender Data Integration

  • 226 Topics
  • 1,023 Replies
226 Topics
J
Contributor
JogchumSiRContributor
asked in Prepare
Columns with source data type money behaving incorrect

Hi All, Hopefully you can help me with this one. We have a source table from our POS systems containing sales transactions. There are a few value columns we transfer that are stored as a money data type and we see some weird things happening with the values of the columns. Differences of 0.01 up to 0.0001 are appearing from nowhere in the transfer from ODX to DWH and i don't know how this i happening.The columns that have money data type are called ‘Bedrag', ’Korting’ and ‘NettoBedrag’ Take transaction 52. In the source the values are like this: After transfer with ODX, i looked in the parquet file using ParquetViewer and that looks like this: Still the same, nothing wrong here. I imported the table without any modifications in TimeXtender and it looks like this: After a transfer the data looks like this: If you look closely we lost 0.0001 on the first record ‘NettoBedrag’. Also wel lost 0.0001 on the Bedrag of the second record. I've tried a few things to mitigate this. Different nume

15719
Christian Hauggaard
Community Manager
P
Starter
philippus.asefawStarter
asked in Prepare
Exclude DW_ID from Primary Key on Database

Hey everyone, I have the following question: Is it possible, at the database level, to set the primary key to the one selected in TX (e.g., "Umkod_ID") instead of the default "DW_ID"? The issue arises because we access our table via PowerApps, and since the app existed beforehand, references in the code were already predefined. Currently, the app's primary key is set to the "Umkod_ID" field of the table. However, in our database, the primary key is set to the system field "DW_ID." Since this table is only loaded once and no transformations occur, the system fields are not particularly important to us in this case. We resolved this issue by manually populating the table, but I am curious if anyone knows a workaround that could be applied to similar cases in the future.

452
P
Starter
L
Starter
leo.noharetStarter
asked in Prepare
Help with Primary Keys and Incremental Load in Table Inserts

Hi, I am facing an issue with duplicate records when using incremental load on a table that receives data via a table insert from another table in the same data area. The incremental load selection rule is based on DW_TimeStamp, and there are three PK in the target table. The issue is that the primary key does not seem to be enforced in the target table, as I am getting duplicate rows in the table. I tested this with another table where data is inserted through a table insert (but without incremental load), and I still see duplicate values for the PK fields and despite setting the table option to "Use instance setting (error)", no error is thrown when duplicates occur. I have tested this behavior in TDI 6935.1 and 6926.1, and the issue persists. My questions are: How do primary keys work with table inserts? Should they prevent duplicates, or do they not function as expected when using table inserts? Can incremental load be achieved using table inserts? If so, what are the necessary con

353
L
Starter
P
Contributor
PieterQContributor
asked in Prepare
Join multiple timelines from DSA to dimension

Case: In the DSA I have two tables (Employee and Employee Addresses). The fist table has history in it from the source system (so no TX history active). On the second table we have TX History active (Type II) . It is not privided by the source system. When joining the tables to a dimension, the timelines must be combined. In what way can this be achived in TX? Emp_ID RegistrationNumber ExpDate ValidFromDate ValidToDate SCD Is Current 109030 39935270430 4-7-2020 1-01-00 1-10-24 0 109030 39935270430 4-7-2029 2-10-24 31-12-99 1 Emp_ID ZIP City SCD From DateTime SCD To DateTime SCD Is Current 109030 1625 HV Hoorn 1-01-00 31-05-24 0 109030 1000 AB Amsterdam 1-06-24 31-12-99 1 Result of join: DWH_ID Emp_ID RegistrationNumber ExpDate ZIP City SCD From DateTime SCD To DateTime SCD Is Current 1 109030 39935270430 4-7-2020 1625 HV Hoorn 1-01-00 31-05-24 0 2 109030 39935270430 4-7-2029 1000 AB Amsterdam 1-06-14 1-10-24 0 3 109030 39935270430 4-7-2029 1000 AB Amsterdam 2-10-24 31-12-99 1

362
Christian Hauggaard
Community Manager
bas.hopstaken
TimeXtender Xpert
bas.hopstakenTimeXtender Xpert
posted in Prepare
Batch cleansing option - best practices, tips and tricks

Hi all, I'd like to start a discussion about the batch cleansing option. You can find this option in: Table Settings → Performance → Enable batch data cleansing. This option is disabled by default. I'm looking for best practices when you should use it and how to determine the best batch cleansing size. I normally enable it on big tables (5.000.000 + rows), this reduces the disk space used for logging. But in some cases it also improves performance. It can result in faster loading of a table. However, I'm always struggling determining the correct batch size. I know the answer will be ‘it depends’. But I'm trying to find some best practices for this option. For now I start at 250.000 rows for batch cleansing and increase with 250.000 rows. I write down the timings and pick the best setting. This setting depends on the number of rows and number of transformations in the table. I'm curious if there are any practices or tips and tricks for this?

6513
rory.smith
TimeXtender Xpert
B
Contributor
bjornContributor
asked in Prepare
Updated TX 6814.1 with strange behavior for incr tables in prepare (desktop)

Hi, I updated our TX version to the newest 6814.1 I have added many tables in TimeXtender Ingest Server with incremental load. Why do I have to change all incr. tables in "desktop app" to explicit incremental load, can't you use “Automatic” anymore? Error msg: when executing with automatic on a table with incr setup from the datasource I get this type of error “Invalid object name 'shema.table_I'.” Something I haven't done right... regards, Bjørn A.

976
devin.tiemens
TimeXtender Xpert
Roy V
Problem Solver
Roy VProblem Solver
asked in Prepare
Object reference not set to an instance of an object.

I am deploying a project to Snowflake. On a table I get the following error: Object reference not set to an instance of an object. As read in another ticket I did a ‘Save and reload' but unfortunately that did not solve the issue. It;s on deploying the data cleansing script however it does not show me the script in the deployment log, I only see the error message. Is there an easy way i can see what lookup column is causing the issue ?

15211
Roy V
Problem Solver
T
Participant
thzmanParticipant
asked in Prepare
Object reference not set to an instance of an object.

Hi!Getting this error when I try to open the Dev instance. Test and Prod is OK. I could open an earlier version of the project but after I made some changes and tried to save and deploy, I first got a warning that said that this was not the latest version. When I chose to save anyway, I got the same error. Is there some way to “force save” to make this earlier version the latest one and get rid of the error?

561
Thomas Lind
Community Manager
D
Contributor
dsvartzmanContributor
asked in Prepare
Multiple ODX for different enviroments

Hi, I was wondering if it's possible to use multiple ODX instances for different enviroments (DEV/ACC/PRD).I know that at 20.10 you could only use a single ODX as after multiple enviroment transfer all mapping to ODX would be lost when using multiple ODX instances Is this still the case? Is this issue solved in next-gen?

9915
D
Contributor
rvgfox
Problem Solver
rvgfoxProblem Solver
asked in Prepare
Buid dependency Graph

This is the first time that I see it: What it is?

351
Christian Hauggaard
Community Manager
Roy V
Problem Solver
Roy VProblem Solver
asked in Prepare
snowflake depoyment message 'skipped - uses unsupported feautures' -> apparantly key stores

I am deploying a project to Snowflake and see a lof of tables are skipped. Then we get the messages below (as an example) Apparantly the tales are skipped as key stores / supernatural keys are not supported . Is this a feature to become available on short notice as our entire datawarehouse uses these? Besides that, is there a way I can see what any other unsupported feautures we have?

485
Roy V
Problem Solver
P
Contributor
Paul BridgeContributor
asked in Prepare
Issue with Data Area When Attempting to View Report / Errors

Hi I have an issue whereby I get the following message when running the ‘Report / Errors’ menu option in one of my data areas. However, when I click OK I do not get any further information to help me identify the issue. Please advise what I need to do to troubleshoot this further. TimeXtender Data Integration version 6745.1 Thanks

9012
daniel
TimeXtender Xpert
F
Contributor
FenchContributor
asked in Prepare
Persistent mapping error message on syncing MDW with ODX

Dear all, When synchronizing the mdw with the odx through right mouse click on the odx and selecting “Synchronize (...)”, the error messages as attached always show up. “Synchronize with Remapping (...)” only leads to complete removal of the field mappings in the subject tables. As there are other ways to sync the mdw with the odx, it’s still possible to work with the subject tables in the mdw. Nevertheless i wonder what causes the error messages and would like to eliminate them along with the risk of having to redo all work on the tables in a later stage.

794
F
Contributor
anders.e.jonsson
Contributor
anders.e.jonssonContributor
asked in Prepare
Using instance name or id in a viariable

Hi, We are migratring i customer from version 20 to 21. In version 20 they use a system variable to limit the amount of data loaded into certain DSA tables in the development environment.is there any way to accomplish this in V21 - setting a variable that only affects one instance? BRAnders

644
J
Explorer
Roy V
Problem Solver
Roy VProblem Solver
asked in Prepare
Simple mode in combination with incremental load

I have a table which loads incremental in the ODX In the datawarehouse the table is on Automatic and in simple mode.The setting to keep fields up to date is off. Will the incremental records from the ODX do inserts and updates, but update only on the fields with no transformations/lookups or will no updates be made at all when an updated existing record is retrieved from the source? As we don’t have any transformations in that part I was hoping we could skip the RAW table and that records get updated if newer versions come by.The documentation only mentions what the ‘keeping fields up to date’ does for lookup fields , transformation fields and so on, not what exactly happens when that box is not checked

933
daniel
TimeXtender Xpert
A
Problem Solver
aftabpProblem Solver
asked in Prepare
Saving ANSI in DVH without proper JSON

I am working with a solution that delivers 3 end points The first end-point https://data.mysystem.com/86fb92b5-ae9c Data from this end-point is aleady in the DVH id is the unique id for the attached document: { "Type": "images", "fileName": "some_file_name.pdf", "id": "3de1cef70101", "mimeType": "application/pdf" } The second end-point To reterive the document the endpoint is https://data.mysystem.com/attachment/ <id> This gives the output as ANSI %PDF-1.7%����1 0 obj<</Type/Catalog/Pages 2 0 R/>>endobj2 0 obj<</Type/Pages/Count 21/Kids[ 4 0 R 27 0 R 31 0 R 37 0 R 46 0 R 48 0 R 50 0 R 54 0 R 56 0 R 60 0 R 62] >>endobj3 0 obj<</MSIP_Label_7a2396b7-5846-48ff-8468-5f49f8ad722a_Enabled(true) /MSIP_Label_7a2396b7-5846-48ff-8468-5f49f8ad722a_SetDate(2022-10-05T08:45:49Z) /MSIP_Label_7a2396b7-5846-48ff-8468-5f49f8ad722a_Method(Standard) /MSIP_Label_7a2396b7-5846-48ff-8468-5f49f8ad722a_Name(Lav) /MSIP_Label_7a2396b7-5846-48ff-8468-5f49f8ad722a_SiteId(e6795081-6391-442e-9ab4-5e9ef74f18ea)(�� I

241
Christian Hauggaard
Community Manager
darrirafn
Starter
darrirafnStarter
asked in Prepare
Data Cleansing Rules hangs / times out

Hi there. I‘m having trouble with a three tables in my project. What they have in common is that they are tables that I created specifically (that is not with a mapping to ODX) and populate with a Custom Table Insert. The problem seems very similar to this one: Data Clensing results in time out | Community (timextender.com) For example, I deployed and executed one of my tables yesterday evening (not a large one, about 20.000 rows). And these are the results: I’m not sure what the problem is but the tables are able to finish execution if the Custom Table Insert is set to valid, however then all my lookup fields (and fixed value fields) are null. If it’s set to raw, then the execution of the data cleansing rules seems to go on forever. In relation to the answers to the other question from @Bernarddb, then yes Operator for the lookup fields is normally always none for me (but I also don’t get exactly the same error): Some of the settings might be wrong perhaps: - Tables are not guarded on

913
darrirafn
Starter
W
Explorer
woutersmikeExplorer
asked in Prepare
Add transformation on a field not visible when table is opened in new window

Hello Don’t know if this is a bug or a deliberate choice, but when you want to add a transformation on a field in a table that was opened in ‘new window (CTRL+W shortcut), then the section for Field Transformation is not available. I have included a picture. When your table is in the DSA/MDW node and you do right-mouse click, then you can choose Field Transformation. When your table is in a separate window, then this particular option is missing

393
Christian Hauggaard
Community Manager
O
Starter
o.beckeStarter
asked in Prepare
History table not inserting new records to valid table from raw table

Dear Support, I have encountered a strange phenomenon with the history settings and need urgent support. Situation: Records which were once flag as new records from the ERP system show up as SCD Is Current is 0. These orders were canceled in the ERP as a next step and flagged correctly in the history raw table table as a type II change (which is correct). However, they do not get inserted into the valid table as new record. Today we realized even more records are missing in the valid table. TX Version: 20.10.40.64. Please advise on how to proceed. This is causing us major issues in our month end closing reporting. It used to be fine and we only experienced it this week. Thank you. Oscar

785
O
Starter
R
Explorer
r.miltenburgExplorer
asked in Prepare
Unable to add job for Data Warehouse and Semantic model

Hi guys, Somehow I am unable to create a job for my semantic models or my datawarehouse/ data area. I can’t select the model when I’m trying to add a new job.

863
W
Explorer
Roy V
Problem Solver
Roy VProblem Solver
asked in Prepare
Error deploying historic table to Snowflake

Good morning I am deploying a specific table with history (mostly type2) to Snowflake. However in the generated script the semicolumn is not placed correctly (after the ; some additional ‘and’ clauses are added). This results in an error creating the data cleansing procedure. I added the script in the attachment.Can you please have a look.

403
Thomas Lind
Community Manager
A
Problem Solver
aftabpProblem Solver
asked in Prepare
Data Clensing results in time out

This problem arrived after the issue with SNK is resolved. I am not sure if the two cases are related in any way but that was the last change made in TX. I am trying to figure out why data clensing is taking so much time. It ultimately ends in time out: Error during Data Processing for Table: DSA.FACT_TABLE_A at: DSAExecution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.Operation cancelled by user.Warning: Null value is eliminated by an aggregate or other SET operation.The wait operation timed out Stored Procedure: [DSA].[usp_FACTABLE_Clean]Parameters:@version = 2177 Line number: 0 Details: SQL Server: 'TEST-SQL-SERVER'SQL Error Number: -2 The total number of rows in ODX are 9 664 965. It shouldn’t take more than a few minutes to transfer and data clens !

15316
Bernarddb
TimeXtender Xpert
D
Starter
DushyantStarter
asked in Prepare
How to add Query table for Mysql data source

Hi Team, We have a data provider for which we need to connect to MySQL. We want to incrementally bring data from the source, but the problem is that all the columns in the table are of the string data type. This prevents us from setting any incremental logic on the date column, which is currently stored as a VARCHAR(8). I tried casting the column to a date format, which works in the query tool, but when I try to add the query table, the option is disabled. I am using Mysql 5.1 client data source. Could you please suggest how I can achieve this?

538
D
Starter
A
Visitor
alozVisitor
asked in Prepare
CTE (Common Table Expression)

Is it possble to create a table based on CTE in timextender, or how would you do it?

975
Christian Hauggaard
Community Manager
D
Contributor
dpjatContributor
asked in Prepare
404 Path can't be found

Hi everyone, back again with another issue. We have now setup our ODX with storage account as storage layer. refreshing it sometimes breaks though. I'm getting 404 error, path can't be found. This happens with transfer all option. When I run the set of tables one by one, everything seems to be fine. I saw 1 more post like this for oracle, where the connector got updated. We're using an older version so I suppose can update the cdata connector and try again. However: the weird thing is that this also happens with ODX to DSA. Any clue how this is possible? Path does not exist, but I can find it in the storage account.

10711
Christian Hauggaard
Community Manager

Most helpful members this week

Badge winners

  • Innovator
    oscarclevehas earned the badge Innovator
  • Certified Solutions Architect
    leo.noharethas earned the badge Certified Solutions Architect
  • Innovator
    Bernarddbhas earned the badge Innovator
  • Certified Solutions Architect
    JeroenStahas earned the badge Certified Solutions Architect
  • Ingest Expert
    nmohas earned the badge Ingest Expert
Show all badges

Popular tags

Terms and ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings