Implement Custom Str or Buffer in Python

Question

I'm working with python-gnupg to decrypt a file and the decrypted file content is very large so loading the entire contents into memory is not feasible.

I would like to short-circuit the write method in order to to manipulate the decrypted contents as it is written.

Here are some failed attempts:

import gpg
from StringIO import StringIO

# works but not feasible due to memory limitations
decrypted_data = gpg_client.decrypt_file(decrypted_data)

# works but no access to the buffer write method
gpg_client.decrypt_file(decrypted_data, output=buffer())

# fails with TypeError: coercing to Unicode: need string or buffer, instance found
class TestBuffer:
    def __init__(self):
        self.buffer = StringIO()

    def write(self, data):
        print('writing')
        self.buffer.write(data)

gpg_client.decrypt_file(decrypted_data, output=TestBuffer())

Can anyone think of any other ideas that would allow me to create a file-like str or buffer object to output the data to?

I'm not sure what you're trying to do here. But the first problem is that you need a self in def write. — abarnert
– abarnert, Commented Jul 19, 2018 at 23:28
Meanwhile, are you using a stringio.StringIO or an io.StringIO there? If the former, I don't see where any coercing to Unicode is happening, so you'll need to give us a minimal reproducible example that shows the code and what happens. If the latter… well, you should still give us a minimal reproducible example demonstrating the error, but in that case you're mixing and matching 3.x-style io stack with the 2.x-style file-object stack, which is doable, but can get confusing. — abarnert
– abarnert, Commented Jul 19, 2018 at 23:29
I want to write the decrypted contents to a stream where I can manipulate the data being written to the stream as-it-is-written. — doremi
– doremi, Commented Jul 19, 2018 at 23:29
@abarnet I'm using stringio.StringIO(). The TypeError is being raised by the gpg library due to not passing a str or buffer obj to the output param. — doremi
– doremi, Commented Jul 19, 2018 at 23:30
Without looking at the docs or the source, it looks like simulating a file isn't going to do you any good here, because the library doesn't want a file object, it wants a string (presumably a filename?) or a buffer. Are there docs to look at somewhere, or is the only way to figure out how to use this library to grub through the source? — abarnert
– abarnert, Commented Jul 19, 2018 at 23:39

Florian Weimer · Accepted Answer · 2018-07-22 21:40:05Z

1

You can implement a subclass of one of the classes in the io module described in I/O Base Classes, presumably io.BufferedIOBase. The standard library contains an example of something quite similar in the form of the zipfile.ZipExtFile class. At least this way, you won't have to implement complex functions like readline yourself.

answered Jul 22, 2018 at 21:40

Florian Weimer

34.2k4 gold badges59 silver badges103 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

doremi Over a year ago

As mentioned in the comments above, the gpg lib won't accept an instance of io.BufferedIOBase - only a str or buffer. However, I wasn't aware of this class so I thank you. It may come in handy.

Collectives™ on Stack Overflow

Implement Custom Str or Buffer in Python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related