EOFException at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)

Question

I was trying to run a matrix multiplication example presented by Mr. Norstadt under following link http://www.norstad.org/matrix-multiply/index.html. I can run it successfully with hadoop 0.20.2 but I tried to run it with hadoop 1.0.3 but I am getting following error. Is it the problem with my hadoop configuration or it is compatibility problem in the code which was written in hadoop 0.20 by author.Also please guide me that how can I fix this error in either case. Here is the error I am getting.

in thread "main" java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:180)
        at java.io.DataInputStream.readFully(DataInputStream.java:152)
        at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1486)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1470)
        at TestMatrixMultiply.fillMatrix(TestMatrixMultiply.java:60)
        at TestMatrixMultiply.readMatrix(TestMatrixMultiply.java:87)
        at TestMatrixMultiply.checkAnswer(TestMatrixMultiply.java:112)
        at TestMatrixMultiply.runOneTest(TestMatrixMultiply.java:150)
        at TestMatrixMultiply.testRandom(TestMatrixMultiply.java:278)
        at TestMatrixMultiply.main(TestMatrixMultiply.java:308)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

Thanks in advance

Regards, waqas

If it makes problems due it can't read the sequencefile header, why don't you regenerate the file? Or just use 20.2? — Thomas Jungblut
– Thomas Jungblut, Commented May 25, 2012 at 13:10
Thomas yes ofcourse 20.2 is an option but I wanted to use it with newer version(1.0.3). And if this is all working fine with 20.2 then it means that method for writing or reading sequence files is correct(at least for 20.2). Do you think that there might be difference in support of reading and writing sequence files in 20.2 and 1.0.3? — waqas
– waqas, Commented May 29, 2012 at 11:55
This error can only occur if the size of the input file is less than 4 bytes - can you provide the file size of the input file(s) in your original question — Chris White
– Chris White, Commented May 29, 2012 at 23:44
@Chris thanks.. I got the problem. Infact it reads _SUCCEESS file along with others while using FileSystem.listStatus method. Can you please tell me how to filter _SUCCESS files while using listStatus. Thanks — waqas
– waqas, Commented May 30, 2012 at 14:43
I see you've answered your own question - stackoverflow.com/questions/10817824/…, please post an answer linking to that question so as to mark this question as answered, and to help those that stumble across this question in future — Chris White
– Chris White, Commented May 30, 2012 at 15:21

AdrieanKhisbe · Accepted Answer · 2015-02-07 21:12:54Z

5

I also encountered same problem, in my case it was giving exception because I haven't closed SequenceFile.Writer object. Problem got resolved when I added sequenceFileWriter.close() statement in my code.

Also if input in MapReduce program is output of some previous MapReduce program then you have to explicitly write code to ignore _SUCCESS file.

PS: I am using CHD4 Cloudera Hadoop configuration

edited Feb 7, 2015 at 21:12

AdrieanKhisbe

4,0668 gold badges40 silver badges45 bronze badges

answered Apr 30, 2013 at 2:10

Neo

792 silver badges6 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Community · Accepted Answer · 2017-05-23 10:28:08Z

2

I got it right. Infact it was not filtering _SUCCESS file automatically and due to reading this log file exception mentioned in the question was occuring. I filtered the files and now its working fine. Here is the link how I filtered the files Filter log files(_success and _log) in FileSystem.liststatus

edited May 23, 2017 at 10:28

CommunityBot

11 silver badge

answered May 30, 2012 at 15:27

waqas

1,1255 gold badges19 silver badges33 bronze badges

Collectives™ on Stack Overflow

EOFException at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1508)

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related