Spark SQL - JAVA syntax of CASE-THEN?

Question

I noticed I can use CASE-THEN with Spark if I use an SQLContext and the .sql() function. Is there a way to use this in a JAVA syntax, directly on dataframes, too? How? Now, I write:

SparkConf sparkConf = new SparkConf();
JavaSparkContext ctx = new JavaSparkContext(sparkConf);
SQLContext sqlContext = new SQLContext(ctx);
DataFrame df = //some imported data 
df.registerTempTable("df");
sqlContext.sql("SELECT *use case-then in here* FROM df");

I'm looking for something like

df.select(case("this").then("that"));

Yuan JI · Accepted Answer · 2016-07-18 10:46:35Z

8

Just import org.apache.spark.sql.functions then use when(Column col, Object obj).

import org.apache.spark.sql.functions;

df.select(functions.when(df.col("colName").equalTo("this"), "that").otherwise("something"));

edited Jul 18, 2016 at 10:46

answered Jul 18, 2016 at 10:31

Yuan JI

2,9952 gold badges23 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

lte__ Over a year ago

This is what I am currently doing, as I wrote in the question :) I left out the step of registering the temp table, sorry. Updated now. I'm interested whether this has a JAVA syntax equvalent. :)

lte__ Over a year ago

Can you also chain these whens, like:

df.select(functions.when(df.col("colName").equalTo("this"), "that"). when(df.col("colName").equalTo("somethingelse"), "dosomething").otherwise("something"));

?

Collectives™ on Stack Overflow

Spark SQL - JAVA syntax of CASE-THEN?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related