All Execution packages with a schedule do not start at the time given. It starts always seconds to minutes later than the schedule indicates. Why doesn't it start according to the schedule?
It happens with almost all packages. It also varies by package. The same package can start seconds later or 2 minutes later. For example: in the screenshots you can see that the package always starts later than 05:00 as scheduled.
Best answer by p.ruijter
We have made a PowerShell script who can start an execution package.
This script will be scheduled in the Task Scheduler. So we bypass the scheduler service who is causin the delay. The package starts now within 20 seconds after starting the script. For security reasons the script does not has a password but referring to an encrypted password.
We even have added a Qlik ETL job in the same script so we have almost zero delay.
If your interested in the script I can send yu an example.
Thanks for all your answers and thoughts.
+4
+2This delay happens because the scheduler service needs to start a TimeXtender application instance (timextender.exe) and then execute the selected execution package. Doing this takes some time (depending on the work load on the machine running TimeXtender).
We have 4 environments in TX.
For instance: a TX Acceptance (night) job is running from 5 am till 7 am. A job from TX Production environment is running at 6 am. I cannot select a concuurent packages from another environment. How to solve that?
Let's get the picture:
For example: we have a job at 1 am who starts 1 minute later as scheduled (1:01 am). no job, in any environment, is running at the same time.
Can it be the multiple services (see screenshot) we have on the same server? This is to handle our 4 environments.
Or do you have other suggestions?
+7Hi,
triggering Windows executions is not an exact thing as
This is why Windows Services have the Automatic (Delayed) startup type, for instance, this makes it so that a service is only triggered to start after a certain period (controlled by the registry) to ensure that certain required services (networking) are all-but guaranteed to have started. If you have a short delay specified, you might still have service-related errors if your networking stack is slow to start.
As some added “fun”: if your machine is not properly receiving time from an NTP server, the clock will drift and the actual time of starting will deviate more or less from real time.
Note that concurrent running is not an issue across environment only within, if your infrastructure is separate for each environment. If, as per your latest post, you are hosting all environments on one machine then there will likely be resource contention or differences in RAM / CPU availability which will all affect the time it takes to open a timextender processs. If you are hosting SQL on that, it will likely be swapping out data from buffer pool as well which will affect the runtime. Any users leaving sessions open will also consume resources and change the speed.
Is it actually a problem if an execution starts 60 seconds later on one day vs another?
If it is by schedule then the Event viewer Windows Log says: TimeXtender started, using configuration file C:\Users\te_service\AppData\Roaming\TimeXtender\TimeXtender\20.10.45.64\Config.xml.
This happens for example at 15:01:18 (the schedule is at 15:00:00). Why does it start more than one minute later?
The service has not the option Delayed. All other users are logged out.
+7When you manually run a Job you do this from the context of an already running timextender.exe process that already has the instances loaded. If you schedule a Job, or Task this sends a command to the required service, which then cause a TimeXtender.exe process or other child process to be started, which needs to load things and then start running.
This is similar to using ADF with an AzureRuntime or Autoresolver runtime: this queues spinning up a VM to handle your compute and incurs a delay (that you may be paying for). If you have your own self-hosted integration runtime, it immediately triggers the run without needing to spin up any VMs on your infrastructure.
tl;dr: if you double-click the timextender icon, right-click your Job or open the Ingest instance and execute a task your execution also does not start from the moment you double-clicked the timextender icon.
+5Hi
The execution package starts after more than one minute later as scheduled.
The scheduler service starts timextender, this causes a delay. But when we start timextender manually and open a project en than manually start an execution package it takes about 20 seconds. So I don't understand why it takes so long before a scheduled package really starts.
Why does it take so long before the scheduler service starts timextender or why does it take so long starting timextender? The Event Viewer indicates that after more than one minute timextender has started (TimeXtender started, using configuration file C:\Users\te_service\AppData\Roaming\TimeXtender\TimeXtender\20.10.45.64\Config.xml)
PS: No other packages are running
+1The scheduler service performs a check on the repository if there is anything that needs to be exectued. If not, then it sleeps for 2 minutes.
Hence the limitation of not being able to run anything more frequently than every 2 minutes and this also explains why you see more or less the same “delay” on every execution until the scheduler service has been restarted.
If it really matters that execution starts closer to the requested time, you can manually perform a restart of the scheduler service and try to start it just after a minute change.
Is it possible to clean up the repository tables involving schedules and execution packages bij deleting records where [ValidTo] <> '99999999'?
Does it speed up the process because the service is checks the repository?
+1I would not recommend deleting anything from the repository outside of the GUI except for execution logs, where the GUI is impossible to use.
I have a script that I use to delete those execution logs - use it at your own risk ;)DECLARE
@BeforeDate DateTime
SET @BeforeDate=DateAdd(m,-3,getdate())
DELETE FROM [dbo].[ExecutionPackageProjectVariableLogDetails]
WHERE [ExecutionPackageLogDetailId]
in (SELECT DISTINCT [ExecutionPackageLogDetailId]
FROM [dbo].[ExecutionPackageLogDetails]
WHERE [ExecutionId] IN (SELECT ExecutionID from ExecutionPackageLogs where [Start]<=@BeforeDate));
DELETE FROM [dbo].[LogRowCounts]
WHERE [ExecutionId] IN (SELECT ExecutionID from ExecutionPackageLogs where [Start]<=@BeforeDate);
DELETE FROM [dbo].[ExecutionPackageProjectVariableLogs]
WHERE [ExecutionId] IN (SELECT ExecutionID from ExecutionPackageLogs where [Start]<=@BeforeDate);
DELETE FROM [dbo].[ExecutionPackageLogDetails]
WHERE [ExecutionId] IN (SELECT ExecutionID from ExecutionPackageLogs where [Start]<=@BeforeDate);
DELETE FROM [dbo].[ExecutionPackageLogs] where [Start]<=@BeforeDate
+7Impossible is a big word, you just need to be very patient :-).
You could also consider exporting projects from xml and importing to a clean repository but you will need to recreate the global connections, which could be a lot of effort if you have a large project.
If you do not need the logging info, you could turn rowcounts off to save some time running operations against the execution tables. Usually development environments will have many versions and less executions and production environments will have less versions and more executions.
On topic again.
If it really matters that execution starts closer to the requested time, you can manually perform a restart of the scheduler service and try to start it just after a minute change.
I have restarted the scheduler service just before the scheduled package starts but it does not have any effect. It still lasts more than one minute to start executing (tables in) the package. I still don't get it why either the scheduler service wakes up after a long time or the timextender.exe takes so long to start up. I think the first one.
The reason why we see it as an issue is that we want to start an execution package multiple times during the day. The data wil be used in operational reports for production department.
No other services or jobs are started.
Do other clients have the same?
Can we bypass the delay of the scheduler service when it wakes up from sleep mode?
+7Hi,
Do other clients have the same?
Yes - though it usually does not pose an issue as the delay is fairly consistent, i.e. there can be a refresh every x minutes if required. I have had customers run packages every 3-5 minutes, but there the exact time of running did not matter. They only cared about the refresh rate.
The business case really matters here: is there a requirement to take action within a specific timeframe if a specific event occurs? Is there an actual business requirement on the “freshness” of the data? If you have a case where the business needs to make decisions on information that is less than 2-3 minutes old, you should really be using a different kind of data pipeline as that tends to go towards an event-driven or (near-)realtime scenario. In that case you would be doing things in Kafka or Azure Event Hub or similar tooling.
Can we bypass the delay of the scheduler service when it wakes up from sleep mode?
In the 20.10.x release you can simply use another scheduler and trigger timextender.exe to run an Execution Package from a given project at a specific time or interval. You will still need to deal with the varied start-up time of a Windows executable and resource contention in your repository database, but that should cut out any latency caused by the scheduler service.
+1The scheduler service writes to the eventlog when it start an execution. Does the time in the eventlog match the starting time in the execution log? The difference between those timestamps witll be the time used for opening the projekt. For older projects, that operation alone can take some time, does the time to manually open the project match the difference between the timestamp in the eventlog vs execution log?
To lower project opening times, you can use Rory’s suggestion and import the project to a new repository db - hence loosing your history / version tracking.
We have made a PowerShell script who can start an execution package.
This script will be scheduled in the Task Scheduler. So we bypass the scheduler service who is causin the delay. The package starts now within 20 seconds after starting the script. For security reasons the script does not has a password but referring to an encrypted password.
We even have added a Qlik ETL job in the same script so we have almost zero delay.
If your interested in the script I can send yu an example.
Thanks for all your answers and thoughts.
+5Scheduling the TimeXtender Classic execution packages via TimeXtender Data Quality and Orchestration (previously known as Exmon DG) can also potentially reduce the delay
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
