Allow multiple jobs to be run concurrently

Hi,

this one is really important for many customers in multi-env situations that share VMs. Being able to add extra ExecutionEngines to a VM (one for each environment for instance) would alleviate this, but being able to use the concurrent Execution Package for situations where you have frequent jobs overlapping the runtime of longrunning jobs seems rather obvious.

Similarly, the ODX Server currently has a limit of one execution per source which adds overhead and unneeded complexity in the same situation.

Hi, I totally agree with this! For small projects, having 1 massive job for all DWH instances is acceptable, however, large projects will have continuous data flows across multiple instances that cannot feasibly be executed sequentially. A large project may have 10s of instances × 3 or 4 environments, and some will need to be simultaneously reloaded. 

Since multiple execution packages within a single job are also ran sequentially, and an execution package is restricted to an instance, this means that currently only one instance per VM can load data at a single moment!

Also, with this restriction, the options for ‘parallel’ and ‘sequence’ in eXmon do not make sense and this will undermine any complexity or triggers built in eXmon since the job engine will just execute everything simultaneously. 

Is there anyway to create another execution service under a different user on the same machine?

Kind regards,

Andrew

Hi, just want to emphasize the need and urge for this feature. We’re migrating a big project to TX21 and just came to the conclusion that the execution of our 27 SSLs cannot happen with 4 concurrent threads. This means that it will take 8+ hours to refresh/execute all Power BI datasets, which is unacceptable.

Manually splitting the executions over multiple VMs is a waste of resources, as the actual work is being done by Power BI/SQL Db. To my opinion this is also not in line with TX’s automation philosophy. I would expect TX to manage concurrent Jobs in a similar way as it works in TX20.

Besides, we have Jobs that need to execute every hour. These Jobs are now blocking other executions.

For me this issue it’s critical too.

Hi everyone,

I need to address a critical issue with TimeXtender and its integration with eXmon. We are encountering a significant limitation: the inability to execute multiple jobs simultaneously on the same execution server instance. This restriction severely impacts our ability to handle large-scale Data Warehouse (DWH) environments effectively.

For large projects with complex data flows across numerous instances, sequential execution is simply not viable. We require the capability to run multiple jobs in parallel to ensure continuous and efficient data processing. The current setup, which only allows one job per instance at a time, is insufficient and limits our processing capacity.

Furthermore, the 'parallel' and 'sequence' options in eXmon become meaningless under these constraints. The tool's inability to handle simultaneous executions undermines the complexity and triggers we have built into our processes.

We urgently need a solution that enables the parallel execution of multiple jobs on the same server instance. Without this capability, using eXmon to manage jobs created with TimeXtender becomes impractical and counterproductive.

I urge the TimeXtender and eXmon development teams to address this issue promptly. Implementing a solution that supports true parallel execution is essential for maintaining the efficacy and relevance of these tools in large-scale environments.

Thank you for bringing this up. We are aware of this limitation and the challenges it poses. We are currently working on a broader redesign of executions to address this limitation. However, in the mean time, in 6744.1 we have added the ability to execute multiple Deliver (SSL) Instances within the same job.  

Hi @JTreadwell ,

while I know the improvement is there, there is no documentation or mention of it in the Product Updates post - could that be added?

@rory.smith, it was added to the notes last week: