Filter an Array using another Array

Question

   val value = Array["id","sd","cd"]  -- List of columns 
   val cols_list = Array["cd","id","tm","no","in","ts","nm"] -  -- List of columns

i want a list with columns not in cols_list.

code i tried as below :

     val newcol = for (x <- cols_list if x.toString.toUpperCase() not in value )

it's throwing error as value not is not a member of String. is there a way that we can achieve? Please suggest.

val finallist = cols_list.filterNot(value.contains(_)) you can use — Ram Ghadiyaram
– Ram Ghadiyaram, Commented Apr 24, 2020 at 2:39
"not in " ... Scala is not Python. I'm also wondering why you added tags for "Apache Spark"? What is the relation to it? — UninformedUser
– UninformedUser, Commented Apr 24, 2020 at 3:06

Ram Ghadiyaram · Accepted Answer · 2020-04-25 02:15:50Z

2

simplest is filterNot method (Apart from diff) that returns all elements from a list for which your function returns false.

val value = Array("id","sd","cd")  // List of columns
     val cols_list = Array("cd","id","tm","no","in","ts","nm")

     val finallist = cols_list.filterNot(value.contains(_)) //cols_list.par.filterNot also you can use
     println(finallist.mkString(" "))
 }

Result : tm no in ts nm

How it works...

filter creates a collection with those elements that do not satisfy the predicate p and discarding the rest. This is collection level and will work for all Collections API in scala signature :

  def filterNot(p: (A) => Boolean): Collection[A]

edited Apr 25, 2020 at 2:15

answered Apr 24, 2020 at 2:39

Ram Ghadiyaram

29.4k16 gold badges102 silver badges133 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

sparkscala Over a year ago

@ Ram Ghadiyaram. Thanks for your help. I need to convert value,col_list to upper case and then get the final list. Also i don't want to use map operations. Since map operation will take longer time and write everything to a single partition.

notNull · Accepted Answer · 2020-04-24 13:35:24Z

0

use .diff to get list of columns not in cols_list

val value = Array("id","sd","cd")
val cols_list = Array("cd","id","tm","no","in","ts","nm")
value.diff(cols_list)
//Array[String] = Array(sd)

//case insensitive 
value.map(x => x.toUpperCase).diff(cols_list.map(x => x.toUpperCase)).map(x => x.toLowerCase)
//Array[String] = Array(sd)

UPDATE:

cols_list.diff(value)
//Array[String] = Array(tm, no, in, ts, nm)

cols_list.map(x => x.toUpperCase).diff(value.map(x => x.toUpperCase)).map(x => x.toLowerCase)
//Array[String] = Array(tm, no, in, ts, nm)

cols_list.map(x => x.toUpperCase).diff(value.map(x => x.toUpperCase)).map(x => "\"a." + x.toLowerCase + "\"").mkString(",")
//String = "a.tm","a.no","a.in","a.ts","a.nm"

edited Apr 24, 2020 at 13:35

answered Apr 24, 2020 at 0:26

notNull

31.8k4 gold badges41 silver badges58 bronze badges

5 Comments

sparkscala Over a year ago

Thank you Shu for your prompt response. I tried but i am not seeing the expected output. I am expecting the values not in value should be displayed as below. "tm","no","in","ts","nm"

s.polam Over a year ago

Change this - value.diff(cols_list) to cols_list.diff(value) you will get above values.

notNull Over a year ago

@sparkscala, Please check the updated answer, we just need to inter change cols_list and value..!

sparkscala Over a year ago

Actually i missed one thing. I want to add something like below for the output. "a.tm","a.no","a.in","a.ts","a.nm" .Any help is appreciated.

notNull Over a year ago

@sparkscala, use

cols_list.map(x => x.toUpperCase).diff(value.map(x => x.toUpperCase)).map(x => "\"a." + x.toLowerCase + "\"").mkString(",")

if the answer helped Click on Upvote and Accept!

Collectives™ on Stack Overflow

Filter an Array using another Array

2 Answers 2

1 Comment

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related