Unable to parse back SQL expression string generated by spark itself

Question

I am running into a scenario where I need to convert spark expression to sql expression, and later need to parse sql expression back to spark expression. In most of the cases it work fine, but in some cases it throws error.

For example following works fine in spark

val sqlContext = spark.sqlContext
import sqlContext.implicits._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._

val df = Seq("Britain", "Germany", "USA", "Russia", "Japan").toDF("Country")

val japan = 'Country === "Japan"
df.filter(japan).show 
val sqlExp = japan.expr.sql
println(sqlExp) // output: (`Country` = 'Japan')
df.filter(expr(sqlExp)).show

But when I try same with following expression it fails:

val expression = 'Country.contains("S")
println(expression.expr.sql)
df.filter(expression).show
val parsedExpression = expr(expression.expr.sql) //output: contains(`Country`, 'S')
df.filter(parsedExpression).show

It seems like it works with only standard sql syntax. When I use expr("country LIKE '%S%'") it is able to parse.

Is there a way to parse back such an sql expression (that is generated by spark) to spark expression?

what is the error message you get? also, i see no such function contains in Spark-SQL documentation — Vamsi Prabhala
– Vamsi Prabhala, Commented Dec 21, 2018 at 15:49

user10821012 · Accepted Answer · 2018-12-21 16:17:33Z

1

The Expression.sql method:

Is not a part of officially public API (as stated many times by the developers code in o.a.s.sql.catalyst should be considered "weakly" private).
Is not explictly intended to generate valid SQL string and can be even an arbitrary string./

In fact contains(Country, 'S') is valid in neither sql (or spark-sql) nor expr.

answered Dec 21, 2018 at 16:17

user10821012

261 bronze badge

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Unable to parse back SQL expression string generated by spark itself

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related