Skip to main content
To homepage To homepage
Solved

ADF pipeline run fails with empty configuration

K
Contributor
Forum|alt.badge.img+2

Hi,

We are receiving some weird error messages when executing our project, which uses Azure Data Factory to copy tables from our Data Lake storage to the Azure SQL db (ODX → DWH). The error can be found in the attachments. When inspecting the pipeline run on ADF, it seems to be missing the configured activities (see second attachment). When executing the table manually from TX, the pipeline runs succesfully. Also the pipeline seems correctly configured on ADF and the pipeline run that is created when executing the table manually shows the correct activities.

Any ideas as to what might be causing this?

 

Best regards,

Kaj

 

 

Best answer by Christian Hauggaard

Hi @KajEmergo, it seems that you opened a ticket regarding the same issue in November 2022, at which point a reboot of the server seemed to fix the issue. Have you tried rebooting the server recently?

@KeremOG can you please share the full error message?

Some follow-up questions to troubleshoot both issues further:

  1. How many threads are you running the package in? Have you tried reducing the number of threads?
  2. Have you setup retries in the execution package?
  3. Have you tried increasing the timeouts on the DSA (by right clicking on the DSA and selecting “edit data warehouse”)?
View original
2 Attachments
K
1 person likes this
  • K
    KeremOG
Did this topic help you find an answer to your question?

12 replies

Christian Hauggaard
Community Manager
Forum|alt.badge.img+5

Hi @KajEmergo which TimeXtender version are you using? 

K
Contributor
Forum|alt.badge.img+2
  • KajEmergo
  • Author
  • Contributor
  • 52 replies
  • February 23, 2023

Hi Christian, we are on version 20.10.38!

Christian Hauggaard
Community Manager
Forum|alt.badge.img+5

is it a particular table/data source that fails? is the issue intermittent (i.e. sometimes the package executes successfully, sometimes it fails)?

K
Contributor
Forum|alt.badge.img+2
  • KajEmergo
  • Author
  • Contributor
  • 52 replies
  • February 23, 2023

The issue is not related to a specific table but several ones (from Lake to DWH). It might be worth mentioning that it is mostly on DSA tables that have mappings from 3 different sources in ODX Server (the same ERP systems but from different sub-companies). It is indeed intermittent, the issue only happens occasionally.

K
Participant
  • KeremOG
  • Participant
  • 12 replies
  • February 28, 2023

Hi,

We are having the same issue on DSA tables that are merged from 3-4 different sources into one table. ADF gives 'Internval Server Error’ with no activity in the specific pipeline.  When runing them seperately, they run succesfully all the time, but not in the system execution packages. It became an quite urgent issue for the system that fails often.

TX version 20.10.38!

Best,

Kerem

Christian Hauggaard
Community Manager
Forum|alt.badge.img+5

Hi @KajEmergo, it seems that you opened a ticket regarding the same issue in November 2022, at which point a reboot of the server seemed to fix the issue. Have you tried rebooting the server recently?

@KeremOG can you please share the full error message?

Some follow-up questions to troubleshoot both issues further:

  1. How many threads are you running the package in? Have you tried reducing the number of threads?
  2. Have you setup retries in the execution package?
  3. Have you tried increasing the timeouts on the DSA (by right clicking on the DSA and selecting “edit data warehouse”)?
K
Participant
  • KeremOG
  • Participant
  • 12 replies
  • February 28, 2023

Hi @Christian Hauggaard ,

  1. Yes, we were using 4 then decreased to 3 threads.
  2. No, we setted up just. 
  3. No they are setted to Command timeout 1800s, connection timeout 60s.

Error message for the specific pipeline is:

"InternalServerError executing request”

 

Christian Hauggaard
Community Manager
Forum|alt.badge.img+5

@KeremOG ok thanks for confirming. What is the full error message returned within TimeXtender? Can you please try reducing to 2 threads to see if this helps? Also try increasing the connection timeout to 180s

K
Contributor
Forum|alt.badge.img+2
  • KajEmergo
  • Author
  • Contributor
  • 52 replies
  • February 28, 2023

@Christian Hauggaard, we will indeed adjust to 2 threads. Retries are configured now as well.

Please find 2 different error messages from TX attached. 

2 Attachments
Christian Hauggaard
Community Manager
Forum|alt.badge.img+5

@KajEmergo thanks for providing error messages. Did you reboot the server recently? Can you also increase the timeout settings on the DSA? Please let me know if the problem persists with these new settings

K
Participant
  • KeremOG
  • Participant
  • 12 replies
  • March 2, 2023

@Christian Hauggaard , yes we rebooted. We did the changes, it worked without failures for now. Thanks for the help.

For the future solutions, we would like to hear more about the cause. If you would have any tips, we would be glad to hear!

Christian Hauggaard
Community Manager
Forum|alt.badge.img+5

Glad to hear the issue has been resolved. Can you please help us by selecting and marking a best answer?

 

Reply


Most helpful members this week

Terms and ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings