The TimeXtender REST Data Source has the ability to add an API endpoint with table flattening in multiple ways: either adding the endpoint once with multple tables or adding the same endpoint multiple times. I've done some tests to find out which is fastest.
The test was done on an API with an unstructured JSON data source, with root nodes on multiple levels where the XSLT Table Builder can iterate over. This means I needed to add 4 flattened tables on the same endpoint, each with different root nodes. Version numbers were REST provider version 9.0, TDI 6935.1, with Ingest running on and Azure VM and storage on ADLS Gen 2.
My hypothisis was that 1x endpoint with 4x tables would be quicker than 4x endpoints with easch 1x table, because TimeXtender would call the API once and the multiple XSLT tansformations can happen on the same result data, rather than the API being called multiple times which would add unnecessary tansfer of data.
To test this, first I added the endpoint with four flattened tables, and made sure the data source was synchronized. The load times were on average 19 sec. (1 thread) and 7 sec. (4 threads).
Then I added the same tables as four seperate endpoints, with exactly the same parameters. Then I did an Import and Synchronize Metadata in the Ingest and committed these changes. The changes were only ‘changed metadata’ for the tables because the source tables were mapped to the existing target tables from the previous sync that had the same table and field names. Now, the load times increased to 25 sec (1 thread) and 24 sec. (4 threads) on average.
My conclusion is that there is a big performance benefit in using 1 endpoint with multiple flattened tables, probably because the API is called once and the subsequent XSLT transformations can execute on multiple threads.
PS The table names come from the XSLT
When working with the XSLT table builder, I've noticed that the table name (that appears in Metadata Manager) comes from the XSLT code, not the "Name" field in the Table Flattening section in the portal. Once the XSLT is generated, (sometimes with ‘Default_Table_Name’), changing the Name field will neither update the XLST nor subsequently the table name in the Metdata Manager. I've reported this to TimeXtender as a suggestion for improvement.
