I have an incremental Rest Datasource that has been working correctly and we are testing it in our Dev environment.
This morning it did something strange.
Up until now it has been storing all of the Parquet files in a Single folder.
This morning it created a new Date Timestamped Folder and appeared to restart the incremental load process from scratch.
Why would this happen?
Below is a screen shot of the 2 Folders and their contents:
Why did it start the incremental load over?
Some of these data sources take hours to reload history, I need to understand what triggered the datasource to decide it had to restart the incremental load process in a new folder.
The down stream TDI Data Areas appear to only load from the new folder.
The only thing that I can potentially think may have changed this is I promoted the Ingest instance from the Dev Instance (Which is the one I am showing) to our production Instance.
Not sure if this could have had an effect.
Any assistance would be welcome.
