1

Now, I am trying to create a Map[String, String] from the csv file where the word is the Key*, and the pronunciation is the Value. I have managed to do it myself using the code below.

def mapFile(filename: String): Map[String, String] = {
    var content: String = ""
    val file: BufferedSource = Source.fromFile(filename)

    for (line <- file.getLines()) {
      if (!line.contains("//")) {
        content = content + line + "//"
      }
    }

    content.split("//").map(_.split("  ")).map(arr => arr(0) -> arr(1)).toMap
}

So file reads the text file, and for every line in the the text file that is not //, it creates a string and then splits the string into key-value, key being split by " " and value being split by `"//"``.

However, it is too slow.
Is there a more efficient way i can create the map without it taking 5 minutes?

Improve this question
1
  • 1
    if (ok == false) is the same thing as if (!line.contains(";;;")). Choose the 2nd one, it makes more sense. Commented Feb 11, 2019 at 3:25

1 Answer 1

Reset to default
3

I believe your main problem is that you are reading all your file into a String to reprocess it after. Which means, you don't only allocate twice of required memory, but that you process your file twice too.

The first improvement you may made to your code is to do everything in just one iteration. 

import scala.io.Source

def mapFile(filename: String): Map[String, String] =
  (for {
    line <- Source.fromFile(filename).getLines
    if (line.nonEmpty && !line.startsWith(";;;"))
    Array(word, pronunciation) = line.split("  ")
  } yield word -> pronunciation).toMap

The above code is equivalent (and will be desugared to something very similar) to this:

import scala.io.Source

def mapFile(filename: String): Map[String, String] =
  Source
    .fromFile(filename)
    .getLines
    .filter(line => line.nonEmpty && !line.startsWith(";;;"))
    .map(line => line.split("  "))
    .map { case Array(word, pronunciation) => word -> pronunciation }
    .toMap

Additionally, if the input file is too big, you may give a look to FS2, or Akka-Streams, or any other kind of streaming to process the file by chunks.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

ur a genius, this worked amazing, went from 5 minutes to 5 seconds. thank you, what exactly is happening during after Array?
@fuckshy If you mean Array(word, pronunciations @ _*), it is just pattern matching, it means (in roughly human words): We know the output of split is an Array, and we are sure there will always be at least two elements (assuming all words in your file will have at least one syllabus, which kinda makes sense) so, we will take the first one out and give it the name word, and the rest will be take out on as a Sequence which we will call pronunciations, hope it is clearer now. Reference.
@user11039395 edited the answer: Fixed the error and added the "desugared" form of the code.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.