Given a list of strings, how can I check if those strings are in a list in Scala?

Question

I have list of keywords

val mykeywords = Array("Glass", "Bone China", "Ceramic", "Clock")

I have dataframe

val df = Seq("Title1", "Title2", "glass", "CloCK").toDF("Title)

I want to generate dataframe

Title Flag
Title1 0
Title2 0
glass 1
CloCK 1

My Current Code

val mykeywords: Array[String] = Array("Glass", "Bone China", "Ceramic", "Clock").map(_.toLowerCase())
val df2 = df.withColumn("Flag", lower(col("Title")).rlike(mykeywords.mkString("|")).cast(IntegerType))

Which is currently not working properly for some string matches, Please point out if there is better way

Can you provide what is the error or the scenario that you code is not working ?? — Sarath Chandra Vema
– Sarath Chandra Vema, Commented Dec 4, 2019 at 6:56

Ivan Stanislavciuc · Accepted Answer · 2019-12-04 08:20:06Z

1

Convert mykeywords into a dataframe too and right join both data frames on values in case insensitive way. Replace null match with 0 and non-null match with 1.

df
.join(right = mykeywordsDf, joinType = "right", joinExprs = lower(df("Title")).equalTo(lower(mykeywordsDf("Title")))
.withColumn(
   "Flag", 
   when(mykeywordsDf("Title").isNull, 0)
    .when(mykeywordsDf("Title").isNotNull, 1)
)

edited Dec 4, 2019 at 8:20

answered Dec 4, 2019 at 7:22

Ivan Stanislavciuc

7,27517 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Rohan · Accepted Answer · 2019-12-04 09:03:47Z

0

Your approach is nearly correct just you have to put string literal in rlike like this

df.withColumn("Flag", col("Title").rlike("(?i)^"+ mykeywords.mkString("|")+ "$").cast("Integer")).show()

This above code will work.

answered Dec 4, 2019 at 9:03

Rohan

315 bronze badges

1 Comment

Rohan Over a year ago

If it solves your given problem then accept the answer so that other peeps might get help

blackbishop · Accepted Answer · 2019-12-04 19:00:44Z

0

You can also use inin function to check if column expression is contained in your mykeywords list:

df.withColumn("Flag", 
              lower(col("Title")).isin(mykeywords.map(_.toLowerCase): _*).cast("int")
             ).show()

Output:

+------+----+
| Title|Flag|
+------+----+
|Title1|   0|
|Title2|   0|
| glass|   1|
| CloCK|   1|
+------+----+

answered Dec 4, 2019 at 19:00

blackbishop

32.8k11 gold badges61 silver badges86 bronze badges

Collectives™ on Stack Overflow

Given a list of strings, how can I check if those strings are in a list in Scala?

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related