This data source can be used to connect to Parquet files in many locations. It is also able to find files across folders and subfolders. 

If your setup requires a specific setup to work there also is various methods available to handle this.

• Configuration manual
  • General notes on setting comma-separated parameters
  • Connection settings
    • Path
    • Include sub-folders
    • Included file types
    • File aggregation pattern:
    • Location specific fields
    • Location
      • Local file or folder
      • Azure Blob Storage settings
      • AWS S3 Bucket settings
      • SharePoint or OneDrive settings
      • Google Cloud Storage settings
      • SFTP settings
    • Culture setup
    • Culture and client culture
    • Delimiter
    • Line ending
    • Quote character
  • Header setup
    • Has header record
    • Include empty headers
    • Skip top
    • Skip comment rows
    • Comment start character(s)
  • Incomplete row handling
    • Ignore blank rows
    • Ignore incomplete rows
    • Trim spaces
    • Empty fields equal to null
    • Default value if null
  • Datatype handling
    • Metadata URI:
    • Max cell length
    • Data format
    • True values
    • False values
    • Parsed row limit

Configuration manual

The following settings can be used with the TimeXtender CSV Data Source connector.

General notes on setting comma-separated parameters

For every setting where a comma-separated list can be passed as a parameter, it is possible to wrap certain items in square brackets – within the brackets, any special character can be used.

Eg. the following date format list setting will be parsed as a 3-long list (notice the comma within the brackets):

yyyyMMdd,[dd,MM,yyyy],yyyy-MM-dd

Connection settings

In this section you can specify the location of the file(s) with the additional settings on how to access it.

Path

The actual path to the csv file(s). You can only specify one path per data connector. It can be a folder or a file. If your location isn’t a local one, it may have an alternate field that specifies where it is located like DriveId in the SharePoint location option. In that regard it can be a / to show that it is the root level.

Include sub-folders

Specify whether CSV files should be collected from subdirectories too.

Note that this setting is ignored for AWS, Azure and Google Cloud locations.

Included file types

Comma-separated list of file extensions to read. Any file with an extension not specified in this field will be ignored.

File aggregation pattern:

Comma-separated list of file pattern(s) that should be treated as one table. Supported wildcard characters are '*' and '?'.

When set, all CSV files belonging to a pattern will be treated as one table. Note that grouped files are expected to share the same schema, otherwise the process will break during full data load.

Location specific fields

The rest of the connection settings are differ based on the selected Location type, and only the relevant fields would be editable – but those fields will be mandatory.

Location

The following location types are supported:

• Local file or folder
• Azure Blob Storage
• AWS S3 Bucket
• SharePoint or OneDrive
• Google Cloud Storage
• SFTP

Local file or folder

• When the file is locally stored on some drive.

Azure Blob Storage settings

• Azure blob connection string: the complete connection string to the container. You can learn more about configuring connection strings here : Configure a connection string - Azure Storage | Microsoft Learn
• Azure tenant ID: The tenant id the storage account is located in. Is needed when doing OAuth 2.0 authentication using the client grant type.
• Azure client ID: The client id for the application that is used for doing OAuth 2.0 client authentication.
• Azure client secret: The secret value that is set up for the application
• Azure storage account name: The storage account name that hosts the container.
• Azure blob container: the container name

AWS S3 Bucket settings

Currently the connector only supports connecting with Access Key Id. You can learn more in AWS documentation.

• AWS region: the related region string (eg. eu-west-2)
• AWS access key ID: access key id to S3 bucket
• AWS secret access key: secret access key to the S3 bucket
• AWS bucket: the bucket name

SharePoint or OneDrive settings

• SharePoint/OneDrive client ID: The Client ID of the app you use for access. The app used needs the following granted Application permissions Files.Read.All and Sites.Read.All. To be sure you have enough rights you may also add Group.Read.All and User.Read.All.
• SharePoint/OneDrive client secret: The secret code that is set for the app used.
• SharePoint/OneDrive Tenant ID: The tenant id of the company that hosts the SharePoint site.
• SharePoint/OneDrive Drive ID: The drive id pointing to the Document Library you connect to. You can find these by using Postman and by looking at the Graph API collection Graph Fork. You can use the app to connect to this and locate the SharePoint folder under Application. In there you have two requests, https://graph.microsoft.com/v1.0/sites and https://graph.microsoft.com/v1.0/sites/{{SiteID}}/drives. Use sites to find the site ID and use the SiteID to find the Drive ID.

Google Cloud Storage settings

The connector supports GCM authentication with service account keys. You can learn more about them here: Create and delete service account keys  |  IAM Documentation  |  Google Cloud

• Google credential file: Path to the Service Account private key (JSON or P12)
• Google storage bucket name: the bucket name

SFTP settings

The connector supports SFTP authentication with password or a public key file. It is mandatory to use one of these methods.

• SFTP host
• SFTP port
• SFTP user name
• SFTP key path