Hi team,
When connecting to an API it is normal to need to define SQL/Data Lake friendly datatypes. They may differ from the datatypes at the API endpoint, if those exist at all. In the TX data source, there are currently two ways to automatically define this metadata:
- Default: use one page of data
- Perform exhaustive metadata scan: uses all data in the endpoint to determine data/lake SQL data-types
Often API's document their datatypes which may be different from the inferred datatypes, certainly with the ‘default’ setting above, as there may be unexpected values in the field later on. This leads us to use costly exhaustive metadata scans, or define generalist ‘override datatypes’ rules which often lead to excessively large datatypes to cover the highest possibilities.
In the CDATA REST data source, we could use schema files to manipulate data-types on a per-field basis. It would be nice to have this option in the TX REST data source. The XSLT table builder seems like an appropriate place, since at that point we are defining the field metadata for the API.
Kind regards,
Andrew
