Convert Apache Spark Scala code to Python

Question

Can anyone convert this very simple scala code to python?

val words = Array("one", "two", "two", "three", "three", "three")
val wordPairsRDD = sc.parallelize(words).map(word => (word, 1))

val wordCountsWithGroup = wordPairsRDD
    .groupByKey()
    .map(t => (t._1, t._2.sum))
    .collect()

What do you expect the output of the code to be? I would guess that code is counting word occurrences, right? So is the expected output { "one" : 1, "two" : 2, "three" : 3} ? — eugenioy
– eugenioy, Commented Jun 12, 2015 at 20:34
import collections;words = ["one", "two", "two", "three", "three", "three"];collections.Counter(words) would work if { "one" : 1, "two" : 2, "three" : 3} is what you want. — Steven Rumbalski
– Steven Rumbalski, Commented Jun 12, 2015 at 20:38
yes i'm expecting output like this : [('one', 1), ('two', 3), ('three', 3)]. what will be the python code for .map(t => (t._1, t._2.sum)) line? — muktadiur
– muktadiur, Commented Jun 12, 2015 at 20:45

Soroosh Sarabadani · Accepted Answer · 2015-06-12 20:49:40Z

5

try this:

words = ["one", "two", "two", "three", "three", "three"]
wordPairsRDD = sc.parallelize(words).map(lambda word : (word, 1))

wordCountsWithGroup = wordPairsRDD
    .groupByKey()
    .map(lambda t: (t[0], sum(t[1])))
    .collect()

answered Jun 12, 2015 at 20:49

Soroosh Sarabadani

4395 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Junayy · Accepted Answer · 2015-06-12 20:53:52Z

2

Two translate in python :

from operator import add
wordsList = ["one", "two", "two", "three", "three", "three"]
words = sc.parallelize(wordsList ).map(lambda l :(l,1)).reduceByKey(add).collect()
print words
words = sc.parallelize(wordsList ).map(lambda l : (l,1)).groupByKey().map(lambda t: (t[0], sum(t[1]))).collect()
print words

answered Jun 12, 2015 at 20:53

Junayy

1,1407 silver badges13 bronze badges

Comments

Ahmad Ragab · Accepted Answer · 2015-06-12 20:58:05Z

2

Assuming you already have a Spark context defined and ready to go:

 from operator import add
 words = ["one", "two", "two", "three", "three", "three"]
 wordsPairRDD = sc.parallelize(words).map(lambda word: (word, 1))
      .reduceByKey(add)
      .collect()

Checkout the github examples repo: Python Examples

answered Jun 12, 2015 at 20:58

Ahmad Ragab

1,1171 gold badge12 silver badges26 bronze badges

Collectives™ on Stack Overflow

Convert Apache Spark Scala code to Python

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related