read binary file (.bin) as byte array in Pyspark

Question

I have a .bin file and want to partition it and get it as byte array. but using map() don't help me and when I get size of whole data, it isn't size of my file (it is bigger than file's size)

For test it, I was faced with other problem, when I use getNumPartitions() to get number of partition, 1 is printed in output but function that use in map(), is called more than one. Another problem is when sum the size of each partition, result isn't my file's size and bigger than it(I get size with sys.getsizeof() in map() function)

How read .bin file as byte array? and,
What's a way for partition file as fixed size? and,
Can I make partition with overlapping and set location of spliting?

How did you load your .bin file?

WoodChopper
– WoodChopper

2015-12-28 07:50:57 +00:00
Commented Dec 28, 2015 at 7:50 — WoodChopper
– WoodChopper, Commented Dec 28, 2015 at 7:50
@WoodChopper with textFile()

user3416282
– user3416282

2015-12-28 08:33:53 +00:00
Commented Dec 28, 2015 at 8:33 — user3416282
– user3416282, Commented Dec 28, 2015 at 8:33

Reactormonk · Accepted Answer · 2015-12-28 06:32:39Z

1

For fixed size, take a look at https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.SparkContext.binaryRecords

answered Dec 28, 2015 at 6:32

Reactormonk

21.8k16 gold badges78 silver badges128 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user3416282 Over a year ago

thanks alot, is it same as textFile()? can I replace it with textFile witout change in other code?what thing I must import? I use spark 1.4.4

Reactormonk Over a year ago

@user3416282 a) I hope so, and b) no idea, I don't use python.

Collectives™ on Stack Overflow

read binary file (.bin) as byte array in Pyspark

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related