Using apache spark to iterate over string

Question

For example we have the string "abcdabcd"

And we want to count all the pairs (e.g: "ab" or "da") that are available in the string.

So how do we do that in apache spark?

I asked this cause it looks like that the RDD does not support sliding function:

rdd.sliding(2).toList
//Count number of pairs in list
//Returns syntax error on first line (sliding)

Community · Accepted Answer · 2017-05-23 11:44:37Z

5

Apparently it supports sliding via mllib as shown by zero323 here

import org.apache.spark.mllib.rdd.RDDFunctions._

val str = "abcdabcd"

val rdd = sc.parallelize(str)

rdd.sliding(2).map(_.mkString).toLocalIterator.forEach(println)

will show

ab
bc
cd
da
ab
bc
cd

edited May 23, 2017 at 11:44

CommunityBot

11 silver badge

answered Dec 11, 2015 at 17:07

Odomontois

16.4k2 gold badges39 silver badges74 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

lkn2993 Over a year ago

and how do we count these pairs? Well btw you look like someone who is hunting my scala questions here :)

Alberto Bonsanto Over a year ago

@lkn2993 using the classic word count approach in Apache Spark

Collectives™ on Stack Overflow

Using apache spark to iterate over string

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related