Apache spark java conditional replacement of column

Question

I am trying to replace a column in Dataset with value Y if it contains X.

import org.apache.spark.sql.functions.*;

public static void main(String[] args) {
    Dataset<Row> DS;//already created dataset
    DS = DS.withColumn(column,when(col(column) === "X", "Y"));
}

It is giving me compilation error of The method col(String) undefined in Eclipse. I dont know what I am doing wrong. Please help.

Alper t. Turker · Accepted Answer · 2018-01-29 20:10:08Z

1

functions is a Scala object, so if you import * it has to be static:

import static org.apache.spark.sql.functions.*;

answered Jan 29, 2018 at 20:10

Alper t. Turker

35.3k9 gold badges89 silver badges118 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Alper t. Turker Over a year ago

Like stated at the beginning - function is a Scala object, which translates to Java singleton. Simplifying, you import static methods from the object.

Collectives™ on Stack Overflow

Apache spark java conditional replacement of column

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related