Skip to main content
To homepage To homepage

Hi all,

Since about a week ago, we’re seeing random but frequent (almost daily) failures in ODX extract tasks from two sources:

  • TimeXtender SQL Data Source (version 22.0.2.1) (SQL database) – runs from 05:00 to ~05:45 (45 min) UTC. Error: Cannot access a closed Stream (HttpRequestException).

  • TimeXtender SQL Data Source (version 22.0.2.1) (Synapse Severless SQL pool)  – runs from 05:00 to ~05:00:30 (30 sec) UTC. Error: System.TimeoutException: The operation has timed out.

Manual retries later in the morning succeed without issue.

The Synapse serverless SQL pool is pre-triggered ~15 minutes in advance to avoid cold start, and both the ODX environment and the data sources have generous timeout settings.

We use Azure Data Lake Gen2 storage as our ODX storage.

Any idea what might be causing these recent failures?

Thanks,
sva

Hi ​@sva 

Do you see a difference if you schedule it to start at a different time? Meaning it is when it is set to start that is the issue, not how.

Hi ​@Thomas Lind ,

I tested this and it does indeed have an impact. I also added a new source, which means that the above two no longer crash, but only the new source does. It seems that the ODX storage cannot handle this workload at the same time?

Hi ​@sva 

Do the new data source share the same server or even the same database?

One data source can only run one task of this data source at a time, but it can run multiple tasks at the same time across data sources.

So it may not be able to handle two or more data sources running on the same database or server at the same time.

Hi ​@sva do you have an update on the above? thanks

Running the jobs at different times initially seemed to mitigate the issue, which may indicate that some of the data sources are connected to databases residing on the same server. It also suggests that an individual job may simply become too large, which links to a known memory limitation in certain ODX builds.

In older versions (e.g., 6848.1), Parquet files are fully built in memory before being uploaded to Azure Data Lake. With large datasets, this can easily exhaust RAM (our ODX server only has 4 GB), and if a network interruption occurs, the stream is closed prematurely, causing the “Cannot access a closed Stream” error.

From version 20.10.13 onwards, the ODX Ingest Service supports chunked uploads, which reduces memory usage and makes the process more resilient. For now, splitting the jobs so that smaller batches are processed has proven to be an effective workaround.

Terms and ConditionsCookie settings