Skip to main content
To homepage To homepage

Prepare

Ask questions about Prepare Instances in TimeXtender Data Integration

  • 230 Topics
  • 1,052 Replies
230 Topics
M
Explorer
mchinskyExplorer
asked in Prepare
Execution Row Count

After manually executing a package, where do I find a log that shows the # of rows processed per table?  I'm trying to verify my incremental loads are working properly

1363
Christian Hauggaard
Community Manager
Jan Stroeve
Contributor
Jan StroeveContributor
asked in Prepare
Is Custom Data Masking possible?

The following article describes several types of data masking: https://legacysupport.timextender.com/hc/en-us/articles/5463363539101-Dynamic-Data-Masking .Does TimeXtender allow custom data masking as well? The example table shown at the top of this article is exactly what I am trying to accomplish. However, the article doesn't describe this type of masking, does it?  

972
harishkm
Community Manager
A
Contributor
alain.thireContributor
asked in Prepare
Loading transactions

I have a transaction table in odx with the following fields (simplified)Key,Value, transactiondate Example100, V100, 2023-01-01100, V100A,2023-01-31 09100, V100B,2023-01-31 10 Each day there are multiple transactions added , of course with  another transaction date.Note in the example I  have only shown the date part (not hours, minutes, seconds) What I need is a table with the latest update in DSAI created the same table. Set primary key to KEY and added an incremental selection rule on transactiondate . Added history settings to update the Value and transaction date based on the keyWhere it goes wrong is if we get in one day multiple transactions 100, V100A,2023-01-31 09100, V100B,2023-01-31 10I traced down the cleansing procedure and Tx detects that there are twice the same key. So far so good.Next it puts the ID of  the latest transcation into <table>_L  and only processes the ID’s that ar not in <table>_LThe result is that I get  100, V100A,2023-01-31 09 which is incor

53211
A
Contributor
daniel
TimeXtender Xpert
danielTimeXtender Xpert
posted in Prepare
Supernatural keys & Data Cleansing performance

Dear Community,I like to build my data estates with supernatural keys but in lager datasets the data cleansing starts to take very, very long. Do you happen to have the same issues? Is there a way to make the supernatural keys load faster? Even with incremental loading it begins to be super slow:I've ran a test on 435,397 records. This is on a Azure SQL with 10 vCores1. is a full load on the table with 7 supernatural keys.2. is a full load on the same table without the supernatural keys.1 has data cleansing of 1 second. 2 has a data cleansing of 104 seconds!Second I've done a test on the same tables but now incremental loads:1. incremental load with 7 supernatural keys2. incremental load without supernatural keys1 has a data cleansing time 1,6 seconds and 2 a data cleansing of a whopping 129 seconds!I'm not so sure I want to keep using the supernatural keys. What do you guys do? Take care= Daniel

2762
rory.smith
TimeXtender Xpert
sierd.zuiderveld
Contributor
sierd.zuiderveldContributor
asked in Prepare
Force Supernatural key value

Hi, I have a list that maps product_codes to product_ID's.What I would like to create is a Key Store that, when fed a product_code from the list, produces the same product_ID and when fed a new code (so not in list) produces a supernatural_key as usual. Is it possible to force the keystore to create the ID's as shown above? 

1182
Christian Hauggaard
Community Manager
rvgfox
Problem Solver
rvgfoxProblem Solver
asked in Prepare
Setup Pre and Post script

The setup phases are confusing, can anyone explain where exactly they run? The Pre script runs previously to insert the records in the valid table, it’s correct in my case, but the post script that must be run after the records inset, doesn’t run.My script it’s: 

1133
Christian Hauggaard
Community Manager
B
Contributor
brandon.spencerContributor
asked in Prepare
Are "Over Ride Data Types" applied in order like a SQL case statement?

Does anyone know whether or not the rules you put in the over ride data types area are applied sequentially like they would be in a SQL case statement (i.e. once a field evaluates to TRUE it ignores the subsequent logic)?

501
Christian Hauggaard
Community Manager
rvgfox
Problem Solver
rvgfoxProblem Solver
asked in Prepare
How to use variables to usage conditions

I would like to run a job only if the number of records in the raw table it’s diferent of the number of records in the valid table wit a condition.How can I do it?

1084
rory.smith
TimeXtender Xpert
E
Starter
eric.vangemertStarter
asked in Prepare
Securing sensitive data

Data in the context of GDPR or sensitive to stock markets require attention in the datawarehouse with regard to securing the data, in some cases this also to the people developing the datawarehouse. We have been looking into several options, and now plan to do a proof of concept on transparent data encryption.   Anybody experience with securing sensitive data in the datawarehouse, either with this setting or in other ways?

591
S
Contributor
R
Visitor
rajneesh.mangotraVisitor
asked in Prepare
converted to type nvarchar

Hello!!Please help to figuer out this error "The given value of type String from the data source cannot be converted to type nvarchar of the specified target column" while pulling data from source to stage.Anyone Who Can Help!!!

2154
S
Contributor
rvgfox
Problem Solver
rvgfoxProblem Solver
asked in Prepare
Error. Indexes missing

The autoindexes doesn’t created in the physical table 

342
rvgfox
Problem Solver
S
Participant
satish.padarthiParticipant
asked in Prepare
TRIM/RTRIM/LTRIM

Did anyone notice that these functions do not seem to work directly on a field? I always have to create a new one to use trim function on the original filed. Am I missing something?I would like to avoid the repetitive process of manually creating additional fields when I have a table which needs TRIM.

12212
Christian Hauggaard
Community Manager
M
Starter
mikkel.hansenStarter
asked in Prepare
Normalizing table containing filed in json format

Hi.I have a question for you all:I have a table for example link thisfiled_name:FormatProduct_id: IntegerDelivery_metod: varchar(50)Inventory_id: IntegerInventory_Entries: JSONInventory_entries can contain one or several entries in JSON format like this:[{"SKU":"A","Price":"100"},{"SKU":"B","Price":""00"}]Instead off one line with the json entries nested into one single field i would like to Normalize the tabel into several lines, one for each SKU.Any ideas?

982
Christian Hauggaard
Community Manager
rvgfox
Problem Solver
rvgfoxProblem Solver
asked in Prepare
Error in SQL Snippet

Hi, with this snippet:  I get an error in deploy: The sippet it’s correct as you can see in: 

731
Stefan
Employee
M
Contributor
Michel van DrunenContributor
asked in Prepare
Error during filtering ODX Azure Data Factory-DSA

Hi,Our customer uses the new TimeXtender version with ODX (Azure Datalake). MDW (Azure SQL Database) and also uses the NAV connector. We experience issues when adding data selection rules in MDW layer on existing tables.  System.AggregateException: One or more errors occurred. ---> System.AggregateException: Pipeline execution failed ---> System.Exception: {  "errorCode": "ActionFailed",  "message": "Activity failed because an inner activity failed; Inner activity name: TIMEXTENDER COPY DATA, Error: The request failed with status code '\"BadRequest\"'.",  "failureType": "UserError",  "target": "TIMEXTENDER COPY DATA LOOP",  "details": ""}   --- End of inner exception stack trace ---   at ODX.DataFactory.DataFactoryResourceHelper.ValidatePipelineRunResult(PipelineRun pipelineRun, DataFactoryConnectionModel dataFactoryConnection)   at ODX.DataFactory.DataFactoryResourceHelper.ExecutePipeline(Dictionary`2 pipelineParameters, DataFactoryConnectionModel dataFactoryConnection, String p

1301
JTreadwell
Employee
rory.smith
TimeXtender Xpert
rory.smithTimeXtender Xpert
asked in Prepare
Transformations / changes not visible until forced full load

When you add tables to a DWH data area (from an Azure Data Lake backed ODX) and subsequently perform a Deploy & Execute, data is loaded normally as expected. In this example I am adding AW2014's Person.Person and Sales.SalesPerson tables. I subsequently want to add a field where I add a transformation to generate the full name for a person (Firstname + ' ’ + Lastname). When I add this field with the transformation and perform a Deploy & Execute, the field is not populated.The reason for this is probably the same reason under: https://legacysupport.timextender.com/hc/en-us/requests/25792 I would expect changes to ETL to trigger a full reload without needing to either change the Table Settings > Data Extraction > Full Load or perform a cumbersome full reload through D&E > Review tasks.Given that this issue seems generic and not only limited to Historiy tables, it is probably a good idea to solve this generally as well.

1357
JTreadwell
Employee
rory.smith
TimeXtender Xpert
rory.smithTimeXtender Xpert
asked in Prepare
Data Profile does not always give the correct highest/lowest occurence

When using the Data Profile function on the Person.Person table in the AdventureWorks2014 database I see strange results for the highest and lowest occurence in the LastName field: Diaz is the highest and Billstrom is the lowest. In reality Abbas is the lowest when sorting by LastName ASC where count(*) == 1 I am not entirely sure that 'Highest occurence’ is clear to everyone that is not a natvie speaker: you could also assume this should be the highest sorted value.

261
JTreadwell
Employee
ignacio
Contributor
ignacioContributor
asked in Prepare
Create Identity column

Hi all,  I need to create an Identity column, How could I create it in TX? Thanks Ignacio

1362
ignacio
Contributor
T
Visitor
tobias.schmockerVisitor
asked in Prepare
Change Data Type in Cleansing

Hi, Let's assume I am importing from CSV file. The Data is very dirty and somehow I have stings in my int filed "ID" so the filed is interpreted as NVARCHAR(MAX). If I change data type of the field (right click > Edit Field) from string back to integer I receive a runtime error during the execution. What I was expecting was to see was corresponding entries in the _M and _L tables. Is this not part of data cleansing after all? We have this problem all over and could not find a better solution than to build custom views with TRY_CAST etc. where we lose lineage etc. What is the best way to clean up such data (keep valid rows, write errors/warnings on invalid rows)? We are talking 100+ tables so I'm looking for a highly scalable solution here not a one time work around. Thanks and BR, Tobias

1492
fwagner
Employee
B
Contributor
brandon.spencerContributor
asked in Prepare
Solutions for different schemas in a data lake?

I am new to TX and data lakes.  I have a common problem that others have commented on and would like to know if: 1) The functionality now exists, or if it does not 2) How are others getting around the issue? The issue is that I need to be able to segregate data sources in the data lake.  I am pulling in data from multiple SQL sources where the source schema and table names are the same, and I need to differentiate those tables in the data lake.  By schema would be best, but if that is not possible, I need some other ideas.  Thanks!

1546
B
Contributor
G
Contributor
gediminas.juskenasContributor
asked in Prepare
Converting int/char to date

Hello, I have a problem that i am struggling with. I have a table that contains field called [Period] and it stores month dates in following int format: 2201, 2202, 2203 which i need to translate to 2022-01-01, 2022-02-01, 2022-03-01 etc. to date format (always 1st day of month) The problem is i can convert the source format to date format in the view, but when i am trying to generate a table from that view, i always get an error that [Period] cannot be converted to date format. I have different options and nothing has worked as timextender always fails to convert to date format. Everything that failed on timextender, worked perfectly on MSSQL so i am not sure what is wrong. Thanks for your help

2446
fwagner
Employee
A
Starter
alexander.hauensteinStarter
asked in Prepare
Location of Errors and Warnings Messages

Dear CommunityIf I'm clicking on Report->Errors or Warnings then I see a fulll list of them. Where are those records stored? In which table can I find them ?My problem is that once forgot to check the "Error" manually, therefore I want to create a script which regularly should check if errors exists.But wo far, I couldn't the table/view/place where this Infos are stored.Any ideas? RegardsAlex 

621
JTreadwell
Employee
bas.hopstaken
TimeXtender Xpert
bas.hopstakenTimeXtender Xpert
asked in Prepare
Incremental Table insert - Is it possible?

Hi,I'm searching for a way to get a table insert incremental. I have a table (about 30 million records). This has a row for each hour of the day for 10.000 machines, so this table is getting bigger very quickly.I do a lot of transformations in this table. This resulted in a load time of over 10 hours. I've discussed this with our TimeXtender Solution Specialist and we've decided to split the table into multiple smaller tables. Now I do have 1 table (loaded incrementally from ODX), this table has a lot of lookups and simple transformations. This table is loading quickly because it's incrementally loaded.The result of this table is inserted in a new table (Raw table), using a table insert. This table is converting cumulative numbers to noncumulative, using the SQL functions LAG, OVER, PARTITION BY and ORDER BY. Because it is a table insert I have no idea how to do this incremental. I've set NonClusted Indexes (also on Raw table) on all fields used in the PARTION BY function. And enabled

4692
bas.hopstaken
TimeXtender Xpert
E
Starter
erich.kallnikStarter
asked in Prepare
_L and _M Behavior for Field Validations

Hi all,I'll try to find out how to get rows in the _L table when I use a field validation. For field validation I can decide if I get a warning or an error information. But I only get a message in the _M like this:How can I get the rows that don't match the field validation criterion to get an entry in the _L table? 

332
E
Starter
L
Starter
larsingvar.nilssonStarter
asked in Prepare
Invalid column name when transfer from uneven sources

Hi all, I have not been able to find any similar posts. My problem is that I stumble over an execution error when trying to map two different sources with identical columns, except one, that only exist in one source. I except data from one of the sources into the column and expect Null from the other on the lines originating from the other. But however Timextender throws error "Invalid column name 'XXXXX' " on the column at execution. Have tried full deploy. Only if I remove the XXXXX column it will execute. Any ideas?

552
S
Contributor

Badge winners

  • Certified Solutions Architect
    hay71has earned the badge Certified Solutions Architect
  • Certified Solutions Architect
    Himanshu Chhas earned the badge Certified Solutions Architect
  • Innovator
    oscarclevehas earned the badge Innovator
  • Certified Solutions Architect
    leo.noharethas earned the badge Certified Solutions Architect
  • Innovator
    Bernarddbhas earned the badge Innovator
Show all badges
Terms and ConditionsCookie settings