The current flow in Ingest Metadata Management is to ask the source what it contains, which will either get exact field types from DBMS-style sources or run a sample from many file- or API-based sources. In the latter variant, this is often a frustrating process: the actual type of a field may only be apparent in a record past the sampling threshold, or doing this might be slow.
In the CData connectors, you would specify types in .rsd. This was annoying, but would at least result in what you needed. The current functionality leads to issues.
Oftentimes we actually know what the type is beforehand and want to simply specify it instead of needing to load a lot of data. An example is an API that has a field that is actually a numeric(p,s) but mostly manifests as an integer. If there are more than one of these type of issues at the same time, it may be impossible to design a filter that allows for a one-shot metadata sync. Another example is a CSV file where Ingest detects nvarchar() fields where I know they are varchar(). As nvarchar() are a wider type, the resource consumption is much larger.
I believe the flow is currently something like:
source type → Ingest inferred type → (Override type) → Ingest storage type → (Implicit cast from Ingest to Prepare) → Prepare storage
I would like to see a few extra options with respect to metadata:
- let me change types as delivered from source in the Metadata Manager. I am not sure there is value in setting the Ingest inferred type and then overriding after, so I suggest directly setting the Ingest storage type
- turn off type evaluation entirely. I might want to do this once and then keep up manually or just input the types myself, depending on the source. This means a metadata sync would only need to ask for tables/views/fields and should therefore be faster
- allow adding a filter only for metadata sync so that I can limit the amount of rows to evaulate for typing separately from transfer task filtering
