Hi team,
An issue that I am running into relatively often is that a job fails to fail or complete, staying in the ‘running’ state indefinitely. Sometimes cloud services can experience transient issues, which may cause these anomalies. More importantly, however, there is no way to clear the queue, stop existing jobs, or force a job to restart when it's already running. Furthermore, there is no timeout on jobs, so they can continue to run forever.
Example: this job has been in the running state for about 30 hours. It runs a number of Ingest Transfer and Cleanup tasks:
In the ODX Execution Queue I can see that there are no ODX tasks waiting or running, so I cannot cancel anything from there:
Starting the job manually or via eXmon does not work, I get the message failed to run message from eXmon: “The TimeXtender job 'ODX - OneLake and FTP - Transfer + Cleanup' is already running”. If this was a prepare or deliver job, it would now be blocking the other prepare or delivery jobs from running.
--
The only way to reset the jobs is by stopping the ODX Service and restarting it. This causes all ODX tasks (also outbound) to fail instantly, which is far from ideal in an environment where data is constantly being loaded, and lengthy reloads may be running outside of work hours. It's also difficult to explain to a client that they may need to do this periodically to keep their environment running.
We should have more control over the how jobs run, fail, and cancel. Jobs waiting or running in the queue should be able to be removed without restarting a service and a large unplanned impact to the platform.
Kind regards,
Andrew
