Custom Data Provider alternative in latest TX version

Hi everybody,

Our client uses a SOAP API from a supplier. While TX supports SOAP the extraction requires multiple steps and string manipulation. The complete process is:

• For each table in the list of tables:
  • Call the first SOAP endpoint to retrieve the names of the fields in the table
  • Call the second SOAP endpoint to retrieve a page of data for the table
    • Split the data into individual rows by splitting the string on the ^ character
    • Split the rows into individual colums by splitting the string on the | character
    • Fetch until the number of rows returned is less than the page size
  • Move to next table

We created a custom data provider for the TX custom data connector. This works fine in TX 20.10, but is not supported in 6221, so we need to create an alternative. I’m aware that you can use RSD files for more advanced logic, but as far as I can see this doesn’t include an option to perform the string manipulation (splitting). For the time being, we’ve ported the logic to a Python script, but as this is external to TX it’s not a desirable solution.

What course of action do you suggest to implement this logic?