Hi all,
I am looking for architectural advice on designing a future-proof multi-ERP data warehouse setup in TimeXtender.
We currently have multiple ERP systems feeding into one data warehouse. The goal is to create a scalable pattern that works now, but also remains maintainable when more ERP systems are added later.
The main question is around where and when to create keys, especially super natural keys / business keys, and where to perform the joins between facts and dimensions. I found some scattered articles on keys and stuff, but not really an proposed best practice in TimeXtender. I understand there are multiple ways to Rome depending on the setup/preferences/performance etc. but I reckon there would be an advices standard based on TimeXtender working and fundamentals.
Example simplified architecture:
DSA.ERP1.Customers
- CustomerID
- CustomerName
- DW_SourceCode
- SNK_Customer = DW_SourceCode + CustomerID
DSA.ERP1.Invoices
- InvoiceID
- CustomerID
- Amount
- DW_SourceCode
- SNK_Invoice = DW_SourceCode + InvoiceID
- SNK_Customer = DW_SourceCode + CustomerID
DSA.ERP2.Customers
- CustomerID
- CustomerName
- DW_SourceCode
- SNK_Customer = DW_SourceCode + CustomerID
DSA.ERP2.Invoices
- InvoiceID
- CustomerID
- Amount
- DW_SourceCode
- SNK_Invoice = DW_SourceCode + InvoiceID
- SNK_Customer = DW_SourceCode + CustomerID
Then in staging:
DSA.dim_customer
- DW_ID / surrogate key - system generated
- Customer_SNK
- Source
- Customer_ID
- Customer_Name
DSA.fact_turnover
- Invoice_SNK
- Customer_SNK
- Customer_FK
- Amount
The intended logic is:
Source invoice tables in all ERP’s
↓
Create SNK_Invoice and SNK_Customer in DSA in every ERP invoice table
↓
Union/insert into staging.fact_turnover
↓
Lookup Customer_FK by joining fact_turnover.Customer_SNK to dim_customer.Customer_SNK
↓
Use Customer_FK as the relationship to dim_customer.DW_ID
- Is this the recommended pattern in TimeXtender for a multi-ERP architecture?
- Should SNKs be created directly in the individual DSA source tables, or later in a consolidated staging layer?
- At which layer would you normally perform the lookup the fact to the dimension to get the dimension surrogate key, in this case DW_ID. In the DSA or in the MDW?
- Are there TimeXtender-specific features or best practices we should use for this, such as SCD for dimensions, hashing over SNK or just have multiple PK columns?
It would be great if there’s like a template already for a multi ERP environment on how to best build your dimension and facts and where to do which transformation in the knowledge base. If not, it might be an idea to create one.