Metadata Manager - Select Tables Functionality

​@Paul Bridge I think that it’s related to my comment to this: 

Hi ​@rvgfox 

Yeah - I did see your post in response to their metadata manager announcement but wasn’t sure if it was the same issue. However, having read your response it does sound the same.

Let me ping off the bottom of that post as well then.

Regards

Paul

Yes it’s. Previouslly it was possible to filter the tables based on rules, by name, schema, etc. And that rules allow us to reduce the quantity of data to read to build the metadata dictionary.

Now, in the new “Metadata Manager” I think that it read the full data source and after that, manually we must select table by table wich tables we will include.

Please ​@Christian Hauggaard give us some clarification.

Hi,

I think this should be an Idea. For some sources (SAP Table), you can prefilter the list of tables to avoid needing to query 600K tables but not in general.Being able to prefilter database / catalog, schema, and tables for any source would be very useful.

Similarly you may have SharePoint as a source with thousands of sites and many thousands of lists and tables.

​@rory.smith I think that there is a problem here. My data source it’s SAP (thousand of tables) but in the previous versions, with the “Select Tables” option, I’ve a rule to select ONLY the tables/views in the schema tx. Those views I’ve create it in my SQL Server pointing to SAP tables.

Now in the “metadata manager” it shows only that tables not all, for my it’s ok, but, What’s happen if I need to change it?

Please ​@Christian Hauggaard review it because I think that it must be fixed.

Hi,

for SAP I directly work against SAP using Theobald Xtract IS (or Universal in rare cases), no need to maintain views in between. The SAP Table connector works against Xtract IS and has a table filter that allows you to prefilter.

I would like to see this kind of functionality on all sources. If you have a swagger-based REST, there might be many endpoints you do not want to see as well.

I have converted this to a product idea, and forwarded the feedback to the product team

Hi Cristian

Thanks for converting into a product idea, however this causes me an immediate problem as I can no longer add any JDE source tables to one of our customers production environments. I will log a support ticket separately to see about how we can revert back to the prior version of TDI.

Hi ​@Paul Bridge 

Please submit a support ticket and we will look into this as soon as possible

Hi ​@Christian Hauggaard ,
Is the object filtering, as it was available in V20 ODX, going to be available in TDI Ingest any time soon?
Some of our customers are using data sources in which they don’t have access to all objects.  This makes the metadata manager fail with errors (due to ‘access denied’ on some tables) and log unnecessary errors.  If we would be able to make a table selection upfront on the data source that would be solved as well (and it would be much faster to sync the metadata).