Skip to main content
Solved

Importing new data .. what to do ?

  • 15 August 2024
  • 6 replies
  • 50 views

hi there

I need a little advise form you guys .. 

In the near future I need a new datasource but I’m not quite sure what the best approach is .. 

I’m going to have 600 inidivudal xml files which all contains different product data .. these will be updated daily or at minmum weekly.

What would be the smartest way to import all these files into 1 table ?

Should I make a parser or should I do something else ?

Hi @truttafisker 

Is the files set up with the same “fields” and data types or are all different?

XML files will automatically be merged together if they all have the same setup but are stored in the same folder. If not you will have to add asterisks in the URI to limit the amount of files.

For example this URI

abfss://sharedfiles/XML/4th/*2014-GL33_Exp.XML

Will give only the one file from here.

But if I changed it to abfss://sharedfiles/XML/4th/ it would be both files as they do share the same setup but contain data from different periods.

Hi @truttafisker 

Is the files set up with the same “fields” and data types or are all different?

XML files will automatically be merged together if they all have the same setup but are stored in the same folder. If not you will have to add asterisks in the URI to limit the amount of files.

For example this URI

abfss://sharedfiles/XML/4th/*2014-GL33_Exp.XML

Will give only the one file from here.

But if I changed it to abfss://sharedfiles/XML/4th/ it would be both files as they do share the same setup but contain data from different periods.

Hi Thomas

 

I’m pretty sure that the individual files/product lines can contain different values .. and there aren’t null values for those parameters that doesn’t have any values .. 

Hi @truttafisker 

The values of fields can be different, it is just the names of the tags or whatever it is called that needs to be the same.

In any case I would first just try to connect to just the folder and see what happens. If it gives errors due to them being too different you can limit it by adding parts of names.

You can also use RSD files if you want to connect to many different kinds of files in the same data source.

Hi @truttafisker 

The values of fields can be different, it is just the names of the tags or whatever it is called that needs to be the same.

In any case I would first just try to connect to just the folder and see what happens. If it gives errors due to them being too different you can limit it by adding parts of names.

You can also use RSD files if you want to connect to many different kinds of files in the same data source.

aaah, now I think I understand .. I will look into it .. thanks in advance ..

Hi @truttafisker 

If you do have a solution that works for you, please share it here so others with similar issues can see how you did it.

that I will do as soon as I get to it .. thanks for all your help

Reply


Terms and ConditionsCookie settings