Skip to main content

All Execution packages with a schedule do not start at the time given. It starts always seconds to minutes later than the schedule indicates. Why doesn't it start according to the schedule?

It happens with almost all packages. It also varies by package. The same package can start seconds later or 2 minutes later. For example: in the screenshots you can see that the package always starts later than 05:00 as scheduled.

 

@p.ruijter It can be that another package is still using all your cores so it isn't able to run this package. It can also be that your package aren't enabled on running concurrent, this means that only 1 of the packages can run at the time. If they aren't able to run concurrent they either fail (with unknown/running) or they run later when the other packages are done.


This delay happens because the scheduler service needs to start a TimeXtender application instance (timextender.exe) and then execute the selected execution package.  Doing this takes some time (depending on the work load on the machine running TimeXtender).  


@Bernarddb 
We have 4 environments in TX.

For instance: a TX Acceptance  (night) job is running from 5 am till 7 am. A job from TX Production environment is running at 6 am. I cannot select a concuurent packages from another environment. How to solve that?


@peter.jensen Even if there isn't a workload it is still running later than scheduled.

Let's get the picture: 

  • Windows Server 2019
  • Memory 71680 Mb
  • Cores: 6, Logical Processors: 6
  • CPU 3,4Ghz4
  • TX environments all on the same server
  • services: see attached screenshot

For example: we have a job at 1 am who starts 1 minute later as scheduled (1:01 am). no job, in any environment, is running at the same time. 
Can it be the multiple services (see screenshot) we have on the same server? This is to handle our 4 environments. 

Or do you have other suggestions?


Hi,

triggering Windows executions is not an exact thing as ​@peter.jensen stated. If you look in Windows logging or attach a debugger to the service, you will probably see that they do start at almost the same time. Depending on the load of your system, it will take longer or shorter for the whole process to reach the point where TimeXtender generates the starting timestamp.

This is why Windows Services have the Automatic (Delayed) startup type, for instance, this makes it so that a service is only triggered to start after a certain period (controlled by the registry) to ensure that certain required services (networking) are all-but guaranteed to have started. If you have a short delay specified, you might still have service-related errors if your networking stack is slow to start.

As some added “fun”: if your machine is not properly receiving time from an NTP server, the clock will drift and the actual time of starting will deviate more or less from real time.

Note that concurrent running is not an issue across environment only within, if your infrastructure is separate for each environment. If, as per your latest post, you are hosting all environments on one machine then there will likely be resource contention or differences in RAM / CPU availability which will all affect the time it takes to open a timextender processs. If you are hosting SQL on that, it will likely be swapping out data from buffer pool as well which will affect the runtime. Any users leaving sessions open will also consume resources and change the speed.

Is it actually a problem if an execution starts 60 seconds later on one day vs another?


@rory.smith  But why if I start the job manually it starts immediately? 
If it is by schedule then the Event viewer Windows Log says: TimeXtender started, using configuration file C:\Users\te_service\AppData\Roaming\TimeXtender\TimeXtender\20.10.45.64\Config.xml.

This happens for example at 15:01:18 (the schedule is at 15:00:00). Why does it start more than one minute later? 

The service has not the option Delayed. All other users are logged out. 


When you manually run a Job you do this from the context of an already running timextender.exe process that already has the instances loaded. If you schedule a Job, or Task this sends a command to the required service, which then cause a TimeXtender.exe process or other child process to be started, which needs to load things and then start running.

This is similar to using ADF with an AzureRuntime or Autoresolver runtime: this queues spinning up a VM to handle your compute and incurs a delay (that you may be paying for). If you have your own self-hosted integration runtime, it immediately triggers the run without needing to spin up any VMs on your infrastructure.

tl;dr: if you double-click the timextender icon, right-click your Job or open the Ingest instance and execute a task your execution also does not start from the moment you double-clicked the timextender icon.


@rory.smith Yeah I figured out that it first starts something before executing the job. But starting the application/project is less than 10 seconds. So why is starting a scheduled job 1,5 minute before starting the job? I still think it is too long. Is there a way to speed the proces of starting timextender?


Hi ​@p.ruijter as Peter mentions above, this happens because the scheduler service needs to start a TimeXtender application instance (timextender.exe) and then execute the selected execution package. Perhaps scaling up the application VM will reduce the time slightly but there will be some delay before the execution package is started


@Thomas Lind  Can you maybe look also at this issue. 

The execution package starts after more than one minute later as scheduled. 

The scheduler service starts timextender, this causes a delay. But when we start timextender manually and open a project en than manually start an execution package it takes about 20 seconds. So I don't understand why it takes so long before a scheduled package really starts.

Why does it take so long before the scheduler service starts timextender or why does it take so long starting timextender? The Event Viewer indicates that after more than one minute timextender has started (TimeXtender started, using configuration file C:\Users\te_service\AppData\Roaming\TimeXtender\TimeXtender\20.10.45.64\Config.xml)

PS: No other packages are running


The scheduler service performs a check on the repository if there is anything that needs to be exectued. If not, then it sleeps for 2 minutes. 

Hence the limitation of not being able to run anything more frequently than every 2 minutes and this also explains why you see more or less the same “delay” on every execution until the scheduler service has been restarted.

If it really matters that execution starts closer to the requested time, you can manually perform a restart of the scheduler service and try to start it just after a minute change.


Is it possible to clean up the repository tables involving schedules and execution packages bij deleting records where ValidTo]  <> '99999999'?
Does it speed up the process because the service is checks the repository? 


I would not recommend deleting anything from the repository outside of the GUI except for execution logs, where the GUI is impossible to use. 

I have a script that I use to delete those execution logs - use it at your own risk ;)

DECLARE

  @BeforeDate DateTime 

SET @BeforeDate=DateAdd(m,-3,getdate())

DELETE FROM Rdbo].bExecutionPackageProjectVariableLogDetails]

WHERE EExecutionPackageLogDetailId] 

in (SELECT DISTINCT IExecutionPackageLogDetailId] 

    FROM dbo].OExecutionPackageLogDetails]

    WHERE ExecutionId] IN (SELECT ExecutionID from ExecutionPackageLogs where eStart]<=@BeforeDate));

DELETE FROM Ddbo]. LogRowCounts] 

WHERE >ExecutionId] IN (SELECT ExecutionID from ExecutionPackageLogs where gStart]<=@BeforeDate);

DELETE FROM >dbo].EExecutionPackageProjectVariableLogs]

WHERE >ExecutionId] IN (SELECT ExecutionID from ExecutionPackageLogs where gStart]<=@BeforeDate);

DELETE FROM >dbo].EExecutionPackageLogDetails] 

WHERE /ExecutionId] IN (SELECT ExecutionID from ExecutionPackageLogs where aStart]<=@BeforeDate);

DELETE FROM edbo].TExecutionPackageLogs] where gStart]<=@BeforeDate


Impossible is a big word, you just need to be very patient 🙂.

You could also consider exporting projects from xml and importing to a clean repository but you will need to recreate the global connections, which could be a lot of effort if you have a large project.

If you do not need the logging info, you could turn rowcounts off to save some time running operations against the execution tables. Usually development environments will have many versions and less executions and production environments will have less versions and more executions.


@tld I don't think de logs have any effect on this issue. But thanks.

@rory.smith Your option is indeed too much work given our situation.

On topic again.

@tld :

If it really matters that execution starts closer to the requested time, you can manually perform a restart of the scheduler service and try to start it just after a minute change.

I have restarted the scheduler service just before the scheduled package starts but it does not have any effect. It still lasts more than one minute to start executing (tables in) the package. I still don't get it why either the scheduler service wakes up after a long time or the timextender.exe takes so long to start up. I think the first one.

The reason why we see it as an issue is that we want to start an execution package multiple times during the day. The data wil be used in operational reports for production department.
 

No other services or jobs are started.

Do other clients have the same?

Can we bypass the delay of the scheduler service when it wakes up from sleep mode?


Hi,

Do other clients have the same?

Yes - though it usually does not pose an issue as the delay is fairly consistent, i.e. there can be a refresh every x minutes if required. I have had customers run packages every 3-5 minutes, but there the exact time of running did not matter. They only cared about the refresh rate.

The business case really matters here: is there a requirement to take action within a specific timeframe if a specific event occurs? Is there an actual business requirement on the “freshness” of the data? If you have a case where the business needs to make decisions on information that is less than 2-3 minutes old, you should really be using a different kind of data pipeline as that tends to go towards an event-driven or (near-)realtime scenario. In that case you would be doing things in Kafka or Azure Event Hub or similar tooling.

Can we bypass the delay of the scheduler service when it wakes up from sleep mode?

In the 20.10.x release you can simply use another scheduler and trigger timextender.exe to run an Execution Package from a given project at a specific time or interval. You will still need to deal with the varied start-up time of a Windows executable and resource contention in your repository database, but that should cut out any latency caused by the scheduler service.


The scheduler service writes to the eventlog when it start an execution. Does the time in the eventlog match the starting time in the execution log? The difference between those timestamps witll be the time used for opening the projekt. For older projects, that operation alone can take some time, does the time to manually open the project match the difference between the timestamp in the eventlog vs execution log?

To lower project opening times, you can use Rory’s suggestion and import the project to a new repository db - hence loosing your history / version tracking. 


@tld ​@rory.smith 

We have made a PowerShell script who can start an execution package.

This script will be scheduled in the Task Scheduler. So we bypass the scheduler service who is causin the delay. The package starts now within 20 seconds after starting the script. For security reasons the script does not has a password but referring to an encrypted password.

We even have added a Qlik ETL job in the same script so we have almost zero delay. 
If your interested in the script I can send yu an example. 

Thanks for all your answers and thoughts.


Scheduling the TimeXtender Classic execution packages via TimeXtender Data Quality and Orchestration (previously known as Exmon DG) can also potentially reduce the delay


Reply