Compress data in Java and decompress in Python

Question

So I am trying to compress (gzip or similar format) a JSON object before I throw it in my MySQL database. I am currently storing the data as BLOB. I have tried to use the following Java method to compress the data:

public static byte[] compress(String str) throws Exception {
    if (str == null || str.length() == 0) {
        return null;
    }

    ByteArrayOutputStream obj = new ByteArrayOutputStream();
    GZIPOutputStream gzip = new GZIPOutputStream(obj);
    gzip.write(str.getBytes("UTF-8"));
    gzip.close();
    return obj.toByteArray();
}

and then store it in the database using setBytes() with a PreparedStatement and am having no issues with this. What I am having issues with is decrypting the data in Python 2.7 I have tried using zlib.decompress() to no avail. It can't seem to read the data that Java is storing. I also need to write a conversion script in Python to compress the old rows into this new format. So whatever format I needs to be readable by the Python decompress() whether it was compressed with Java or Python 2.7

I am happy to provide anymore information that can assist in helping to find a solution to my dilemma.

Thanks.

EDIT: Some of the Python Code:

class KitPvPMatch(Base):
    """ The SQLAlchemy declarative model class for a User object. """
    __tablename__ = 'kit_pvp_matches'
    __table_args__ = {
        'mysql_engine': 'InnoDB',
        'mysql_charset': 'utf8'
    }

    match_id = Column(INTEGER(11), autoincrement=True, primary_key=True, nullable=False)
    season = Column(Unicode(5), nullable=False)
    winner = Column(Unicode(16), nullable=False)
    loser = Column(Unicode(16), nullable=False)
    ladder_id = Column(TINYINT(4), nullable=False)
    data = Column(BLOB, nullable=False)

# The line in question
jsonData = json.loads(zlib.decompress(match.data))

# The error
error: Error -3 while decompressing data: incorrect header check

The gzip module seems to only be for files? I am using a database. — MasterGberry
– MasterGberry, Commented Dec 6, 2013 at 22:14
Ahh, right you are. Sorry. I clearly didn't read your post very carefully. — mgilson
– mgilson, Commented Dec 6, 2013 at 22:16

Community · Accepted Answer · 2017-05-23 11:56:56Z

1

Here is a post that goes over unzipping using zlib with a stream.

Otherwise, have you tried the gzip docs for gzip.py. You may need a temp file. The documentation for gzip is here. There is a fairly decent solution for this approach in the following post on decompression..

If you haven't already, ensure that you are getting bytes back from SQL. Python is flexible so it may be a string. Call bytearray(string) on your string if this is the case.

If that doesn't work:

What format is the data in when returned by your SQL command?
What error, if any are you getting?

edited May 23, 2017 at 11:56

CommunityBot

11 silver badge

answered Dec 6, 2013 at 22:09

Andrew Scott Evans

1,03315 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

MasterGberry Over a year ago

The gzip module seems to only be for files? I am using a database.

Andrew Scott Evans Over a year ago

you could try pumping to a temp file and then unzipping. I added some beef but without your python code (where the problem seems to be), I am a bit at a loss,

Andrew Scott Evans Over a year ago

try this post on unzipping gzipped data as well stackoverflow.com/questions/12571913/…

Andrew Scott Evans Over a year ago

i would have dropped this in the comment field fyi, but I don't have the rep yet.

MasterGberry Over a year ago

I added some of the Python code to help give some more information as to whats going on.

|

ahfx · Accepted Answer · 2023-10-31 00:46:13Z

1

I'm rehashing the answer from here: https://stackoverflow.com/a/12572031/7298096 because the question of this thread is exactly the topic I was looking for. In my case the Java code is compressing the content with DeflaterOutputStream and then encoding with Base64.encodeBase64String The error: Error -3 while decompressing data: incorrect header check is resolved if I provide a 32 signal offset to zlip decompres:

import base64
import zlib

data = "ENCODED_COMPRESSED_STRING_FROM_JAVA"

output_str = zlib.decompress(base64.b64decode(data), 32 + zlib.MAX_WBITS).decode('utf-8')

answered Oct 31, 2023 at 0:46

ahfx

3772 silver badges9 bronze badges

Collectives™ on Stack Overflow

Compress data in Java and decompress in Python

2 Answers 2

9 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

9 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related