Skip to main content
To homepage To homepage

I’m looking for a TimeXtender Fabric Lakehouse (pyspark.sql) expert 😀

I’ve got a SQL transformation that selects the maximum value of 10 columns:

(select max(val) from (values (vColumn_1]), (nColumn_2]), ..., (,Column_10])) as 0Values](val))

How do I achieve this in a Prepare Lakehouse custom transformation?
I could build the mother/father of all massive case statements, but I’d prefer something simpler and more elegant … if possible!?

Hi ​@Søren Sørensen ,

 

I have not tried but theoretically you should be able to either use Spark SQL: https://spark.apache.org/docs/latest/api/sql/index.html#max

or use the post-script feature with pySpark, i.e. something along these lines: https://stackoverflow.com/questions/57531778/how-to-find-the-max-value-of-multiple-columns

Reply


Terms and ConditionsCookie settings