Skip to main content
SUBMITTED

Notification on "logical" errors in Data warehouse load

Related products:TimeXtender Data Integration
  • March 9, 2023
  • 3 replies
  • 24 views

rvgfox
Problem Solver
Forum|alt.badge.img+4

There are tables like _L and _M to store the records when there are some errors in the load proccess.

It would be nice to get a notification in the load proccess if these types of errors exists.

Now the only way to know it it’s to see the content of the tables “manually”.

3 replies

rory.smith
TimeXtender Xpert
Forum|alt.badge.img+8
  • TimeXtender Xpert
  • March 15, 2023

This one is really important now that the TX Dictionary project is no longer functional. Instead of needing to build something to monitor the raw _L and _M tables, having notifications would be great.


devin.tiemens
TimeXtender Xpert
Forum|alt.badge.img+6
  • TimeXtender Xpert
  • March 17, 2023

I agree, It might also be nice to have the ability to add these tables to a semantic model rightaway in TimeXtender, to add them to an endpoint. Now this is only possible when connecting to the specific database with e.g. PowerBI or Qlik.


rvgfox
Problem Solver
Forum|alt.badge.img+4
  • Author
  • Problem Solver
  • March 17, 2023

I’ve created this code to control that tables:

-- Variable con consulta a ejecutar
DECLARE @Esquema NVARCHAR(4)='DSA'
DECLARE
@SQL NVARCHAR(MAX)
SELECT
@SQL='SELECT
''<TableName>'' AS TableName
, t1.ErrorText
, COUNT(*) AS Registros
FROM ' + @Esquema + '.<ErrorTable> AS t1
INNER JOIN ' + @Esquema + '.<ControlTable> AS t2
ON t1.ErrorNumber = t2.ErrorNumber
GROUP BY t1.ErrorNumber
, t1.Severity
, t1.ErrorText';
WITH TablasErrores
AS (SELECT
SUBSTRING(t.NAME, 1, LEN(t.NAME) - 2) AS TableName
, t.NAME AS ControlTable
, REPLACE(t.NAME, '_L', '_M') AS ErrorTable
, SUM(p.rows) AS RowCounts
FROM sys.tables AS t
INNER JOIN sys.indexes AS i
ON t.OBJECT_ID = i.object_id
INNER JOIN sys.partitions AS p
ON i.object_id = p.OBJECT_ID
AND i.index_id = p.index_id
WHERE SCHEMA_NAME(t.schema_id) = @Esquema
AND RIGHT(t.NAME, 2) = '_L'
AND i.OBJECT_ID > 255
AND i.index_id IN(0, 1)
GROUP BY
t.NAME
HAVING SUM(p.rows) > 0)
SELECT
@SQL=STRING_AGG(REPLACE(REPLACE(REPLACE(@SQL, '<TableName>', TableName), '<ErrorTable>', ErrorTable), '<ControlTable>', ControlTable), ' UNION ')
FROM TablasErrores
EXEC sp_executesql
@SQL