for loop into map method with Spark using Scala

Question

Hi I want to use a "for" into a map method in scala.

How can I do it?

For example here for each line read I want to generate a random word :

val rdd = file.map(line => (line,{
            val chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
            val word = new String;
            val res = new String;
            val rnd = new Random;
            val len = 4 + rnd.nextInt((6-4)+1);

            for(i <- 1 to len){
                val char = chars(rnd.nextInt(51));
                word.concat(char.toString);
            }

            word;
}))

My current output is :

Array[(String, String)] = Array((1,""), (2,""), (3,""), (4,""), (5,""), (6,""), (7,""), (8,""), (9,""), (10,""), (11,""), (12,""), (13,""), (14,""), (15,""), (16,""), (17,""), (18,""), (19,""), (20,""), (21,""), (22,""), (23,""), (24,""), (25,""), (26,""), (27,""), (28,""), (29,""), (30,""), (31,""), (32,""), (33,""), (34,""), (35,""), (36,""), (37,""), (38,""), (39,""), (40,""), (41,""), (42,""), (43,""), (44,""), (45,""), (46,""), (47,""), (48,""), (49,""), (50,""), (51,""), (52,""), (53,""), (54,""), (55,""), (56,""), (57,""), (58,""), (59,""), (60,""), (61,""), (62,""), (63,""), (64,""), (65,""), (66,""), (67,""), (68,""), (69,""), (70,""), (71,""), (72,""), (73,""), (74,""), (75,""), (76,""), (77,""), (78,""), (79,""), (80,""), (81,""), (82,""), (83,""), (84,""), (85,""), (86...

I don't know why the right side is empty.

I have the same problem but in my case i already declared them as var but still didn't work — AVIK DUTTA
– AVIK DUTTA, Commented Mar 13, 2018 at 9:07

The Archetypal Paul · Accepted Answer · 2017-01-13 11:47:57Z

2

There's no need for var here. It's a one liner

  Seq.fill(len)(chars(rnd.nextInt(51))).mkString

This will create a sequence of Char of length len by repeatedly calling chars(rnd.nextInt(51)), then makes it into a String.

Thus you'll get something like this :

import org.apache.spark.rdd.RDD
import scala.util.Random

val chars = ('a' to 'z') ++ ('A' to 'Z')

val rdd = file.map(line => {
  val randomWord = {
    val rnd = new Random
    val len = 4 + rnd.nextInt((6 - 4) + 1)
    Seq.fill(len)(chars(rnd.nextInt(chars.length-1))).mkString
  }
  (line, randomWord)
})

edited Jan 13, 2017 at 11:47

answered Jan 13, 2017 at 9:27

The Archetypal Paul

41.9k20 gold badges107 silver badges136 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

eliasah Over a year ago

I've took the liberty to update your answer if you don't mind. :)

The Archetypal Paul Over a year ago

If you like. I thought it was fairly clear already :) I tweaked it still further

akuiper · Accepted Answer · 2017-01-13 03:49:06Z

1

word.concat doesn't modify word but return a new String, you can make word a variable and add new string to it:

var word = new String
....
for {
    ...
    word += char
    ...
}

answered Jan 13, 2017 at 3:49

akuiper

216k33 gold badges362 silver badges379 bronze badges

2 Comments

user7375007 Over a year ago

when I try it i receive this error error: value += is not a member of String word += char.toString

akuiper Over a year ago

You need to declare word as a variable. Notice the var keyword in front of word instead of val which is immutable.

Collectives™ on Stack Overflow

for loop into map method with Spark using Scala

2 Answers 2

2 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related