How to snappy compress a file using a python script

Question

I am trying to compress in snappy format a csv file using a python script and the python-snappy module. This is my code so far:

import snappy
d = snappy.compress("C:\\Users\\my_user\\Desktop\\Test\\Test_file.csv")
with open("compressed_file.snappy", 'w') as snappy_data:
     snappy_data.write(d)
snappy_data.close()

This code actually creates a snappy file, but the snappy file created only contains a string: "C:\Users\my_user\Desktop\Test\Test_file.csv"

So I am a bit lost on getting my csv compressed. I got it done working on windows cmd with this command:

python -m snappy -c Test_file.csv compressed_file.snappy

But I need it to be done as a part of a python script, so working on cmd is not fine for me.

Thank you very much, Álvaro

GlynD · Accepted Answer · 2017-10-04 11:53:40Z

2

You are compressing the plain string, as the compress function takes raw data.

There are two ways to compress snappy data - as one block and the other as streaming (or framed) data

This function will compress a file using framed method

import snappy

def snappy_compress(path):
        path_to_store = path+'.snappy'

        with open(path, 'rb') as in_file:
          with open(path_to_store, 'w') as out_file:
            snappy.stream_compress(in_file, out_file)
            out_file.close()
            in_file.close()

        return path_to_store

snappy_compress('testfile.csv')

You can decompress from command line using:

python -m snappy -d testfile.csv.snappy testfile_decompressed.csv

It should be noted that the current framing used by python / snappy is not compatible with the framing used by Hadoop

edited Oct 4, 2017 at 11:53

answered Sep 26, 2017 at 11:21

GlynD

4424 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to snappy compress a file using a python script

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related