Skip to main content
To homepage To homepage
SUBMITTED

Join the ibrainstorm: taking XPilot to the next level

Related products:TimeXtender Data Integration
  • January 21, 2024
  • 1 reply
  • 45 views
rogier.helmus
Problem Solver
Forum|alt.badge.img+1

Hi all,

I’d like to propose that we start boosting XPilot into an AI that can help us in developing our data warehouses by supporting questions and tasks on the meta data of our model.

I think this should be possible since the TimeXtender model can be represented in an XML file and recently ChatGPT has added an XML interpreter which allows you to ask questions about it. ChatGPT does this by translating the question to Python XML parsing code and then running it to get the results, So far it’s been a struggle to get it working but I think it should be possible to create an AI on the TimeXtender model it if it has enough knowledge about how the TimeXtender meta data is structured. 

I would like for XPilot to not only give me answers but also to make changes to the model. This can be a huge time saver in the case of repetitive tasks (e.g. updating multiple tables). 

Below is a list of categories and sample questions / actions that I’d like to be able to ask to XPilot. If you like to idea of getting a super boosted XPilot, please upvote my post. What types of questions or tasks would you like to see supported? Write them below in the comments! 

Greets,

Rogier

 

TimeXtender categories and questions / tasks

  • Data model analysis
    • List the tables that are missing a primary key definitions
    • List the DSA tables that I can join the DSA OrderLine table to based on the fields available?
    • List the tables are using a post or pre script and that don't have a dependency set
    • List the DSA views that don't have mapped fields
    • List the tables with property [….] enabled/disabled.
    • List the MDW tables that have custom records configured
    • List the DSA tables with object dependencies and list the dependant tables for each table
    • List the tables or fields with […] in the description
    • List the fields that have a TimeXtender or custom transformation but no description

 

  • Code generation / transformations
    • Create a tables called 'Sales' from the ODX table SalesLines from the AdventureWorks data source and add the following fields […]
    • Set a default value of [...] for the fields […]
    • Create a new varchar 50 field with a fixed value of 1 and a fixed value of 2.
    • Set the batch size to [...]  for the DSA tables […] when the current value is [...].
    • Convert the varchar fields in the Customer table to nvarchar
    • Can you suggest a custom SQL transformation that creates a running total of the Revenue field grouped by the CustomerKey and ordered by the SalesDataKey? (the current XPilot can already answer this)
    • Set the value of the [...] field in the DSA [...]  table to [...]  when the value is [...] .
    • Change the operator of the LastUpdate field to [...].

 

  • Model quality and integrity
    • What is the distribution of the CustomerType field in the DSA table Customers?
    • Does the  OrderLine table have duplicate records on the field OrderLineId?
    • Give me a list of all DSA tables that have no rows
    • Does the ProductDescription field in the DSA Product table have NULL values?
    • List the tables with duplicate values

 

  • Model governance and security
    • Tag all the fields in the data model with the name [...]  with the tag [...]
    • List the tables that have a field tagged name [...]

 

  • Data lineage
    • List the ODX data sources, tables and fields that are used to calculate the [...] field in the [….]  table in the [...] semantic layer
    • Which fields in the DSA [...] table aren't used?
    • List the MDW tables that have a [...] surrogate key field

 

  • Team development
    • Summarize the DWH Production instance changes since version […]
    • Warn me when a colleague starts working on a model I'm currently working on
    • Warn me when a colleague makes a change to a table that I currently also have changes open on

 

1 reply

rory.smith
TimeXtender Xpert
Forum|alt.badge.img+8
  • rory.smith
  • TimeXtender Xpert
  • January 22, 2024

This is a great suggestion! Evaluating how well LLMs can serve in this area is bound to lead to at least a few areas where it can be trusted enough to be incorporated.

I think it is good to split the proposed functionality into subtypes:

  • interrogative: the LLM scans code and delivers answers or statistics or marks objects. These answers should be easy to check and therefore pose less “risk” or require less deep knowledge from the human operator to evaluate
  • creative: the LLM is asked to produce new objects or change objects (and thereby generating new code). This would require a good way of visualising what is proposed to be done, as well as why the suggested solution was created that way. Until enough experience has been had, we will not know how to trust this and how much knowledge is required to evaluate. Perhaps the LLM can self-evaluate the solution and complexity of the problem to suggest.

My general experience with different LLM in the realm of producing new code has not been very positive yet. Partially this is due to general LLMs not being up to the task, and partially due to not being experienced in the kind of errors these models will make or how to use prompt guardrails to avoid them. What does work is see the general approach taken and using that for “inspiration”. As long as you have extensive knowledge on the topic, that helps.

 

From my perspective interrogative LLM functionality should be much easier to slot in and already be very helpful. The creative side will take more care to prevent it from causing more issues to the unaware.

 

    Terms and ConditionsAccessibility statement