writing bytes to file in python

Question

I am writing bittorrent client. It can download pieces from peers, but I can't make it write pieces to files correctly. The problem is the encoding. Because of the wrong encoding client is writing wrong bytes to file. I have found encoding called "unicode_internal". It seems to be correct one but the problem didn't go away. Despite the constant piece size(16384 bytes) sometimes the file size increases by 16386 or so. Here's how I write pieces to file. Nothing special.

with open(path, 'a', encoding='unicode_internal') as f:
    f.seek(offset, 0)
    f.write(data.decode('unicode_internal'))

I tryed to open file in 'rb' mode but it doesn't help. Part of the stdout from working client:

piece size: 16384
sum of pieces lengths: 49152
filesize: 49152

piece size: 16384
sum of pieces lengths: 65536
filesize: 65536

piece size: 16384
sum of pieces lengths: 81920
filesize: 81922 #Here it is. Size increased by 16386 bytes. The piece size is 16384

piece size: 16384
sum of pieces lengths: 98304
filesize: 98306

What am I doing wrong?

I am using windows 7. data is bytes, then it converts to str — Pavel Koshara
– Pavel Koshara, Commented Mar 7, 2015 at 19:36
Why are you even trying to use an encoding? You have bytes, write to a wb or ab binary file. No decoding, no encoding. You'll have problems with newline translation otherwise. — Martijn Pieters
– Martijn Pieters, Commented Mar 7, 2015 at 19:42

rutsky · Accepted Answer · 2015-03-07 19:42:02Z

5

You need to open file in binary mode at write bytes:

data = bytes(...) # some data in bytes type
with open(path, 'ab') as f:
    f.seek(offset, 0)
    f.write(data)

When opening in text mode, independently of used encoding, Python can do transformations with line-ending. E.g. on Windows it will convert single line-feed character \n (0x0A) to "Windows-style line-ending": \r\n (0x0D, 0x0A) — two characters.

answered Mar 7, 2015 at 19:42

rutsky

4,1602 gold badges33 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

writing bytes to file in python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related