Removing empty strings from maps in scala

Question

val lines: RDD[String] = sc.textFile("/tmp/inputs/*")
val tokenizedLines = lines.map(Tokenizer.tokenize)

in the above code snippet, the tokenize function may return empty strings. How do i skip adding it to the map in that case? or remove empty entries post adding to map?

axmrnv · Accepted Answer · 2014-11-05 09:43:08Z

29

tokenizedLines.filter(_.nonEmpty)

answered Nov 5, 2014 at 9:43

axmrnv

1,03411 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Siva Over a year ago

does this filter the result or remove them? I am keen on removing them.

axmrnv Over a year ago

The filter returns a new collection with no empty strings.

Siva Over a year ago

when i print the tokenized lines post filtering, it still has empty strings in array buffer. Am i missing something additionally?

axmrnv Over a year ago

As in scaladoc, RDDs are immutable, so you can not modify them in place. And you should avoid mutable data structures as long as possible in Scala. So you may write val tokenizedLines = lines.map(Tokenizer.tokenize).filter(_.nonEmpty).

Kuldeep Singh Over a year ago

You can not remove anything from RDD, its immutable, solution is correct as you can collection filtered data for your purpose and use where ever you want to use map without empty string.

Daniel C. Sobral · Accepted Answer · 2014-11-05 22:01:39Z

11

The currently accepted answer, using filter and nonEmpty, incurs some performance penalty because nonEmpty is not a method on String, but, instead, it's added through implicit conversion. With value objects being used, I expect the difference to be almost imperceptible, but on versions of Scala where that is not the case, it is a substantial hit.

Instead, one could use this, which is assured to be faster:

tokenizedLines.filterNot(_.isEmpty)

answered Nov 5, 2014 at 22:01

Daniel C. Sobral

298k88 gold badges508 silver badges688 bronze badges

Comments

Álvaro Valencia · Accepted Answer · 2018-09-24 08:12:04Z

1

You could use flatMap with Option.

Something like that:

lines.flatMap{
     case "" => None 
     case s => Some(s)
}

edited Sep 24, 2018 at 8:12

Álvaro Valencia

1,2179 silver badges18 bronze badges

answered Nov 5, 2014 at 9:54

crak

1,6752 gold badges20 silver badges33 bronze badges

1 Comment

crak Over a year ago

ow lines.flatMap{case "" => Nil case s => Seq(s)} so ?

Álvaro Valencia · Accepted Answer · 2018-09-24 11:37:11Z

0

val tokenizedLines = (lines.map(Tokenizer.tokenize)).filter(_.nonEmpty)

edited Sep 24, 2018 at 11:37

Álvaro Valencia

1,2179 silver badges18 bronze badges

answered Nov 5, 2014 at 16:59

user1989252

797 bronze badges

Collectives™ on Stack Overflow

Removing empty strings from maps in scala

4 Answers 4

5 Comments

Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

5 Comments

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related