Hello,
I have submitted the following question to TimeXtender support. I wanted to post the question here to see if there are other people that have build or are trying a similar solution to monitor the executions.
We are currently implementing a monitoring and orchestration framework based on the TimeXtender Meta Collection data.
According to the TimeXtender documentation, Meta Collection metadata is exported once every 24 hours to the data lake. We have also observed this behavior in our environment. The documentation states that metadata is exported once every 24 hours and the exported files overwrite the previous export.
Our use case
We are using the following Meta Collection entities to monitor the execution status of our data platform:
- Ingest_ExecutionTasks
- Ingest_ExecutionLogs
- Ingest_Tasks
- Ingest_TaskVersions
- Ingest_DataSourceVersions
The objective is to determine:
- Which tasks were executed
- Whether the execution was successful
- Whether warnings or errors occurred
- Which datasource and table were affected
- What error message was generated
This information is being used to build a centralized monitoring solution and to control downstream processes, including Qlik reloads. Our goal is to prevent analytical dashboards from being refreshed when upstream TimeXtender processes have failed.
Current limitation
The current Meta Collection refresh interval of approximately 24 hours is too infrequent for our monitoring and orchestration requirements.
A load failure may occur shortly after the daily export has been generated, resulting in a delay of nearly 24 hours before the failure becomes visible through Meta Collection.
Because of this delay, the metadata is suitable for historical reporting and troubleshooting, but difficult to use as a reliable operational monitoring source for near real-time process control. This limitation is also reflected in our internal evaluation of the solution.
Questions
- Is it possible to increase the Meta Collection export frequency?
- Is there a hidden configuration setting that controls the export interval?
- Is a higher refresh frequency currently on the TimeXtender roadmap?
- Are there alternative APIs, tables, endpoints, event streams, or other supported mechanisms that provide execution status information more frequently than the current Meta Collection export?
- What would TimeXtender recommend for customers that want to build operational monitoring and automated orchestration based on load execution status?
Requested enhancement
Ideally, we would like to see one of the following options:
- Configurable Meta Collection refresh frequency.
- Hourly exports.
- Execution metadata updates immediately after completion of a task or workflow.
- A supported API endpoint exposing execution status in near real-time.
- Event-based notifications/webhooks when executions complete or fail.
We believe this would significantly increase the value of Meta Collection for operational monitoring, automated data quality controls, process orchestration, and downstream reporting platforms.
