Hello,
We have our data warehouse setup on TXD Portal.
Our data source is with provider Azure Data Factory - SQL Server and the data loaded in our Ingest instances is in Azure data lake storage. Then the data is processed in our prepare instances also in Azure SQL database.
The question is now from the data source we have some columns being encrypted via AES_ENCRYPT pySpark function with 16bit key. Is there a possibility to decrypt the data anywhere from our end in ingest instances or prepare instances. For example to add a customized script somewhere (preferably in pySpark as well using AES_DECRYPT function) ? Otherwise, is there encryption method that can be applied from the data source and by that we can decrypt more easily in TXD in SQL Server?
Looking forward to your suggestion/answer. Thank you.