Skip to main content
Solved

Direct Read option latest TX version


avanrijn
Contributor
Forum|alt.badge.img+2

Dear Sir,

In TX version 20 you had a configuration setting Direct Read.


I cannot find this setting in the latest and the greatest version in TimeXtender.

Question: Is Direct Read still available ? How can I set this up ?

Thanx in advance for looking into this.
 

The "Direct Read" option in TimeXtender is particularly useful in scenarios where you need to transfer data between different data areas within the same physical database but using different schemas. Here are some specific scenarios where you might use the Direct Read option: 

  1. Large Data Volumes: 

  • When dealing with large tables, Direct Read can be more efficient as it allows for direct data transfer within the same database, reducing the overhead associated with external data movement tools like SSIS or ADO.NET. 

  1. Incremental Loads: 

  • Direct Read is beneficial for incremental loads where only a subset of data needs to be transferred based on specific criteria (e.g., timestamp or modified date). This can help in optimizing performance and reducing load times. 

  1. Performance Optimization: 

  • If you are experiencing performance issues with other data transfer methods, such as ADO.NET or SSIS, switching to Direct Read can help. For example, adding the OPTION(RECOMPILE) to the stored procedure can significantly improve performance by utilizing the appropriate indexes. 

Best answer by rory.smith

Hi,

in TDI you will typically have your data areas inside your single database as separate schemas. This removes the need for direct read in most scenarios. As TDI does not support SSIS as a transfer method, that reason is also no longer an issue.

In an on-prem scenario where you would ingest into SQL and transform in SQL you could consider TX 20.10.x or the upcoming Classic if it hinges on Direct Read. Otherwise I would step away from the “separate database per data area / layer” paradigm as that costs performance. As you would likely store Ingest in ADLS Gen2 or Onelake, most implementations would not benefit.

The only place where I would see this be useful is in a future situation where Prepare → Prepare data movements are supported as those will likely be database to database moves that may benefit from Direct Read in Azure SQL MI and SQL Server 2025/2022.

View original
Did this topic help you find an answer to your question?

rory.smith
TimeXtender Xpert
Forum|alt.badge.img+7
  • TimeXtender Xpert
  • April 22, 2025

Hi,

in TDI you will typically have your data areas inside your single database as separate schemas. This removes the need for direct read in most scenarios. As TDI does not support SSIS as a transfer method, that reason is also no longer an issue.

In an on-prem scenario where you would ingest into SQL and transform in SQL you could consider TX 20.10.x or the upcoming Classic if it hinges on Direct Read. Otherwise I would step away from the “separate database per data area / layer” paradigm as that costs performance. As you would likely store Ingest in ADLS Gen2 or Onelake, most implementations would not benefit.

The only place where I would see this be useful is in a future situation where Prepare → Prepare data movements are supported as those will likely be database to database moves that may benefit from Direct Read in Azure SQL MI and SQL Server 2025/2022.


Christian Hauggaard
Community Manager
Forum|alt.badge.img+5

Hi ​@avanrijn 

As Rory mentioned this is not supported in TDI at the moment, as the approach is to use data areas with different schemas on the same database.

Would you like me to convert this to a product idea?


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings