scala quirky in this while loop code

Question

Yesterday, this piece of code caused me a headache. I fixed it by reading the file line by line. Any ideas ?

The while loop never seems to get executed even though the no of lines in the file is greater than 1.

 val lines = Source.fromFile( new File("file.txt") ).getLines;

 println( "total lines:"+lines.size );

 var starti = 1;
 while( starti < lines.size ){
   val nexti = Math.min( starti + 10, lines.size  );

   println( "batch ("+starti+", "+nexti+") total:" + lines.size )
   val linesSub = lines.slice(starti, nexti)
   //do something with linesSub
   starti = nexti
 }

0__ · Accepted Answer · 2013-09-27 17:35:38Z

14

This is indeed tricky, and I would even say it's a bug in Iterator. getLines returns an Iterator which proceeds lazily. So what seems to happen is that if you ask for lines.size the iterator goes through the whole file to count the lines. Afterwards, it's "exhausted":

scala> val lines = io.Source.fromFile(new java.io.File("....txt")).getLines
lines: Iterator[String] = non-empty iterator

scala> lines.size
res4: Int = 15

scala> lines.size
res5: Int = 0

scala> lines.hasNext
res6: Boolean = false

You see, when you execute size twice, the result is zero.

There are two solutions, either you force the iterator into something 'stable', like lines.toSeq. Or you forget about size and do the "normal" iteration:

while(lines.hasNext) {
  val linesSub = lines.take(10)
  println("batch:" + linesSub.size)
  // do something with linesSub
}

edited Sep 27, 2013 at 17:35

answered Jul 21, 2011 at 0:36

0__

67.5k21 gold badges180 silver badges271 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Michael Dillon Over a year ago

+1 for pointing out the right way to do iteration with a while loop in Scala.

ziggystar Over a year ago

So why do you consider this to be a bug? Iterators usually don't know how many elements they contain. In Particular you can't find out the number of lines in a file without reading the entire file.

smartnut007 Over a year ago

It depends on what one calls a bug. It behaves as the authors intended and is documented. But, the naming is not intuitive. I most definitely did not expect it to return an iterator. getLines does not imply an iterator to me in any way. I most definitely like this feature though.

0__ Over a year ago

I think Iterator should not have a size method. If you provide one, it should not have side effects, because most sane minds wouldn't expect them. At best, the method name should be different. Maybe count()?

smartnut007 Over a year ago

I agree. count implies an active operation much more than size.

|

Kevin Wright · Accepted Answer · 2011-07-21 09:46:54Z

None of the above answers quite hits the nail on the head.

Theres a good reason why an Iterator is returned here. By being lazy, it takes pressure off the heap, and the String representing each line can then be garbage collected as soon as you've finished with it. In the case of large files, this can make all the difference for avoiding an OutOfMemoryException.

Ideally, you'd work directly with the iterator and not force it into a strict collection type.

Using grouped then, as per om-nom-nom's answer:

for (linesSub <- lines grouped 10) {
  //do something with linesSub
}

And if you wanted to retain the println counter, zip in an index:

for ( (linesSub, batchIdx) <- (lines grouped 10).zipWithIndex ) {
  println("batch " + batchIdx)
  //do something with linesSub
}

If you really need the total, invoke getLines twice. Once for the count, and a second time to actually process the lines.

larissa · Accepted Answer · 2011-07-21 00:37:27Z

4

The second time you call lines.size it returns 0. This is because lines is an iterator, not an array.

answered Jul 21, 2011 at 0:37

larissa

4934 silver badges16 bronze badges

Comments

0__ · Accepted Answer · 2012-07-18 16:44:05Z

4

I've rewritten your code in a Seq way, that was proposed in @0__ answer:

val batchSize = 10;
val lines = Source.fromFile("file.txt").getLines.toSeq;

 println( "total lines:"+lines.length);

 var processed = 0;
 lines.grouped(batchSize).foreach( batch => {
      println( "batch ("+processed+","+(processed+Math.min(lines.length-processed,batchSize))+")
               total:"+lines.length
      );
      processed = processed + batchSize;
      //do something with batch
   }
 )

edited Jul 18, 2012 at 16:44

0__

67.5k21 gold badges180 silver badges271 bronze badges

answered Jul 21, 2011 at 0:51

om-nom-nom

63k13 gold badges186 silver badges231 bronze badges

Collectives™ on Stack Overflow

scala quirky in this while loop code

4 Answers 4

6 Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

6 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related