Multiple file import : skip last {n} lines or skip line starting with

Joeri De Valck

July 30, 2019 12:03

Completed

Hello,

While the skip header lines is great, I would hope to see the skip {n} last lines also.

I have a fixed length text file, where each record/line is 903 characters, but there is a "closure" line to reflect the number of records in the file. This line has a different length and causes the import to fail the file.

This line starts is like

*** last record after 12503 records ***

(yes, coming from an old mainframe ;-)

I have tried to specify the ** as header line, and use skip 1 header line, but it seems the import does not like this (seems to hang up the import process completely)

Raf Lemmens November 04, 2019 15:24

Additionally :

Check header lines (order of columns, name changes, ...)

work-around

If you have programming skills :

write a small cleanup script
add external executable
add execution package
call the script first to clean out the source file
add business units and dwh steps

If you don't have programming skills and only DiscoveryHub is an option:

read in the file as if it were 1 big column
DW_Id should match the rownumber of the file
Using TX standard functions (sql) you can delete the necessary lines.
eg delete from table where dw_id > (select max(dw_id) from table) - 2
Use the data export of TX to write to a flat file
Use this file to import into your project;

Multiple file import : skip last {n} lines or skip line starting with

1 comment