How do I upload a large SQLite file to Object Storage using OpenStack's Swift (via Python)

Question

I have an Object Storage instance on IBM's Bluemix, and I'm attempting to upload a ~32GB SQLite file. Here is my Python code which uses the OpenStack Swift API (with the credentials removed):

import swiftclient
conn = swiftclient.Connection(key="pw",authurl="url",auth_version='3',os_options={"project_id": "project_id","user_id": "user_id","region_name": "region"})
container_name = 'containerName'
file_name = 'file.sqlite'
with open(file_name, 'rb') as sqlite_file:
    conn.put_object(container_name,file_name,sqlite_file)

I tested this code with a small .html file and it uploaded without a problem. When I changed the file to the SQLite file, it ran for > 5 hours and eventually gave a "requests.exceptions.ConnectionError: [Errno 32] Broken pipe" error. What am I doing wrong?

Sanjay.Joshi · Accepted Answer · 2016-05-31 05:13:23Z

2

You'll need to read-up on Swift DLO/SLO support and manifests. Here's a blog post that might help with context on what manifests are and the differences between Static Large Object and Dynamic Large Object support.

Basically, I'd recommend the following approach:

Download/install the Python-SwiftClient binary
Use its upload command in conjunction with your object storage credentials from the Bluemix service. In the manifest article above, it discusses this approach here. Take note of the upload command, the use of the --use-slo flag and the ability to define the size of the concatenated segments generated. Roughly, the invocation will look like this:

$ swift --os-auth-url=https://identity.open.softlayer.com/v3 --os-user- id=some_hex_value --os-password="weird_characters" --os-project-id=another_hex_value --os-region-name=dallas -V 3 upload my_object_storage_container_name -S int_seg_size_in_bytes my_local_large_file_with_some_extension --use-slo

my_local_large_file_with_some_extension segment 3
my_local_large_file_with_some_extension segment 1
my_local_large_file_with_some_extension segment 2
my_local_large_file_with_some_extension segment 0
my_local_large_file_with_some_extension/1443450560.000000/160872806/52428800/00000002
my_local_large_file_with_some_extension/1443450560.000000/160872806/52428800/00000003
my_local_large_file_with_some_extension/1443450560.000000/160872806/52428800/00000001
my_local_large_file_with_some_extension/1443450560.000000/160872806/52428800/00000000
my_local_large_file_with_some_extension

Good luck.

edited May 31, 2016 at 5:13

answered May 31, 2016 at 4:58

Sanjay.Joshi

3041 silver badge12 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Ross Lewis Over a year ago

It's running now. Some segments are working and some are erroring out: put_object(u'RossL_segments', u'bigfile.sqlite/slo/1438471533.000000/31807655936/200000000/00000007', ...) failure and no ability to reset contents for reupload. Any advice on this?

Sanjay.Joshi Over a year ago

@RossLewis Per the swift docs here, you may be able to re-run the command using the --leave-segments argument to try and get the failures re-uploaded. I've encountered upload segment errors from time to time and usually succeed after retrying. You may also be able to tweak the segment sizing and number of threads to match your network characteristics better.

Ross Lewis Over a year ago

Thanks again for helping me out on this! Now that I have the file uploaded, can I get rid of all of the segments in the _segment container?

Sanjay.Joshi Over a year ago

@RossLewis . Nope. Basically, the _segment container contains the actual data in its respective pieces. The file that represents the db.sqlite asset is really a manifest file that is a single point virtual representation of the segment files which OpenStack understands as being concatenated together. In other words, it simply holds a list of the segments within it in a way that Openstack Object Storage understands as needing to be concatenated when the object is fetched (downloaded). Glad that you found success. If you're good, please accept the answer ;-)

Collectives™ on Stack Overflow

How do I upload a large SQLite file to Object Storage using OpenStack's Swift (via Python)

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related