Pass filters as parameter to Dataframe.filter function

Question

I have a Dataframe userdf as

val userdf = sparkSession.read.json(sparkContext.parallelize(Array("""[{"id" : 1,"name" : "user1"},{"id" : 2,"name" : "user2"}]""")))

scala> userdf.show
+---+-----+
| id| name|
+---+-----+
|  1|user1|
|  2|user2|
+---+-----+

I want to retrieve user with id === 1 and same I can achieve using code like

scala> userdf.filter($"id"===1).show
+---+-----+
| id| name|
+---+-----+
|  1|user1|
+---+-----+

What I want to achieve is like

val filter1 = $"id"===1
userdf.filter(filter1).show

These filters are fetch from configuration files and I am trying to achieve a more complex query using this building block, something like

userdf.filter(filter1 OR filter2).filter(filter3).show

where filter1, filter2, filter3, AND and OR condition are fetched from configurations

Thanks

lev · Accepted Answer · 2018-10-15 07:23:36Z

2

the filter method can also accept a string that it a sql expression.
this code should produce the same result

userdf.filter("id = 1").show

so you can just get that string from config

answered Oct 15, 2018 at 7:23

lev

4,1274 gold badges37 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

user811602 Over a year ago

this solution will not work with multiple "and" and "or" condition. ie, userdf.filter($"name"==="user1" || $"id" === 1) works fine, but userdf.filter("id=2" || "id=1") is not working. stackoverflow.com/questions/35881152/…

lev Over a year ago

as long as it is a valid sql statement it should work: userdf.filter("id=2 or id=1")

user811602 Over a year ago

thanks. the statement "as long as it is a valid sql statement" is very helpful.

JeffLL Over a year ago

@user811602 any luck on resolving this issue?

user811602 Over a year ago

@JellfLL I have created valid sql string statement as commented by lev.

Collectives™ on Stack Overflow

Pass filters as parameter to Dataframe.filter function

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related