Incremental Data Load Issue

  • 22 August 2023
  • 5 replies
  • 102 views

Hi Support,

 

We first raised this problem/bug back in Feb 2022.

 

It seems that the incremental rules are not working correctly. The client Xandor had removed these incremental rules from their Live system some time ago but re-introduced them this month, and have failed. The problem seems to be related to using multiple incremental rules e.g. 'MODIFIEDDATETIME > (Last Max Value) - 3600 seconds & TRANSDATE > (Last Max Value) - 15,552,000 seconds. 

 

Please see attached document for further information on this. The client is currently on version 20.10.30.64 and have TX configured to use SSiS for data transfers. 

 

Do you know if there is a solution to this issue please?

 

Thanks,

Kashif 


5 replies

Please see uploaded document - in particular, the SQL Profiler output from the incremental load. The predicate is only subtracting 1 hour from TRANSDATE rather than 180 Days as set in JDM.

Userlevel 4
Badge +5

Dear @Kashif ,

These bugs, if it is a bug, is always hard te prove. As I look a look at the first document I could not find the proof of this bug. I did not recalculate though.

To be honest i'm wondering why you use such a incremental rule. When I worked with AX or NAV the MODIFIEDDATE or timestamp is always enough.

Are you 100% sure that these new records fall into the rule?
When I take a look at the second document the AX server is handling a where with 3 rules and an OR. This makes it very hard to really check all possibilities.

What I would do is either: 
- See if I can find hard proof that these records should be loaded with this incremental selection rule

- Check if a ‘simple’ incremental rule gives me the desired results (just a rule on the MODIFIEDDATE)

Take care

= Daniel
 

Userlevel 6
Badge +5

@Kashif can you please confirm which dates are shown in your incremental table? You can check this by previewing the Business Unit table and selecting incremental in the instance dropdown

 

Hi Daniel, Christian,

 

Please see response below from the client on this:

 

1. This is just one example of using "such a rule" to assist in reducing the number of rows. It should work.

2. We are 100% sure that the records are falling into the rule - I have provided all of the evidence in the 2 documents uploaded.

3. Yes, there are 3 statements, one for each of the Companies extracted. They have parentheses which make it quite simple. We are looking at the extract for company 'pick'.

4. I've provided the hard poof in the ticket. The LEDGERTRANS transactions are not being picked up (point 3 in the first uploaded document - TRANSDATE and MODIFIEDDATETIME are 17/08/2023). Yes, they are 'pick' transactions.

5. Yes, the simple incremental rule works. This is what I had originally just based on MODIFIEDDATETIME, however, I need to improve performance with a selection on TRANSDATE, and in other cases, need to use a combination of rules 

6. I provided the dates in the incremental table (point 2 in the first uploaded document), here it is again from the JDM interface:

7. It is quite clear from the settings that I have set in JDM and the subsequent SQL Profiler trace that the incorrect settings are being submitted to SQL, as I pointed out in the second document I uploaded. This is why we believe this is a bug.

Userlevel 6
Badge +5

Hi @Kashif I have opened a support ticket for us to investigate further, and arrange a screen share

Reply