Not able to write SequenceFile in Scala for Array[NullWritable, ByteWritable]

Question

I have a Byte Array in Scala: val nums = Array[Byte](1,2,3,4,5,6,7,8,9) or you can take any other Byte array.

I want to save it as a sequence file in HDFS. Below is the code, I am writing in scala console.

import org.apache.hadoop.io.compress.GzipCodec
nums.map( x => (NullWritable.get(), new ByteWritable(x)))).saveAsSequenceFile("/yourPath", classOf[GzipCodec])

But, it's giving following error:

error: values saveAsSequenceFile is not a member of Array[ (org.apache.hadoop.io.NullWritable), (org.apache.hadoop.io.ByteWritable)]

You require to import these classes as well (in scala console).

import org.apache.hadoop.io.NullWritable
import org.apache.hadoop.io.ByteWritable

I have seen above approach many time. that's why i went for it. But now I got it. — M_Gandhi
– M_Gandhi, Commented Mar 12, 2018 at 7:25

Knows Not Much · Accepted Answer · 2018-03-11 00:22:55Z

2

The method saveAsSequenceFile is available on an RDD not on an array. So first you need to lift your array into an RDD and then you will be able to call the method saveAsSequenceFile

val v = sc.parallelize(Array(("owl",3), ("gnu",4), ("dog",1), ("cat",2), ("ant",5)), 2)
v.saveAsSequenceFile("hd_seq_file")

http://homepage.cs.latrobe.edu.au/zhe/ZhenHeSparkRDDAPIExamples.html

answered Mar 11, 2018 at 0:22

Knows Not Much

31.8k66 gold badges208 silver badges395 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

M_Gandhi Over a year ago

Great ! It works. Even other examples in that link are excellent. Thanks for sharing.

Collectives™ on Stack Overflow

Not able to write SequenceFile in Scala for Array[NullWritable, ByteWritable]

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related