Upload file to S3 folder using python boto

Question

I am trying to upload files from local directory to S3 folder. I am able to upload files to S3 bucket but I am unable to upload files to folder within S3 bucket.

Could any one help? What am i doing wrong here..

Here is the code:

import os
import sys 
import boto3
import fnmatch
import pprint
import re
import hashlib

SOURCE_DIR  = '/home/user/Downloads/tracks/'
BUCKET_NAME = 'mybucket'
S3_FOLDER = 'mybucket/folder1/'

client = boto3.client('s3')

s3 = boto3.resource('s3')

def get_md5(filename):
    f = open(filename, 'rb')
    m = hashlib.md5()

    while True:
        data = f.read(10240)
        if len(data) == 0:
            break
        m.update(data)

    return m.hexdigest()

def get_etag(filebase,filepath):
    for item in bucket.objects.all():
        keyfile = S3_FOLDER + filebase
        if(keyfile == item.key):
            md5  = get_md5(filepath)
            etag = item.e_tag.strip('"').strip("'")

            if etag != md5:
                print(filebase + ": " + md5 + " != " + etag)
                return(files_to_upload.append(filepath))
        else:
            return(files_to_upload.append(filepath))

files_to_upload = []
for root, dirnames, filenames in os.walk(SOURCE_DIR):
    for filename in filenames:
        filepath = os.path.join(root, filename)
        get_etag(filename,filepath)

for f in files_to_upload:
    client.put_object(Bucket=BUCKET_NAME, Key=f)

Mark B · Accepted Answer · 2017-11-11 17:35:49Z

2

Folders don't really exist in S3. You can prefix the file name (object key) with the something that looks like a folder path.

It's not entirely clear to me what your code is doing with the file paths, but your code needs to be changed to something like this:

for f in files_to_upload:
    key = "my/s3/folder/name/" + f
    client.put_object(Bucket=BUCKET_NAME, Key=key, Body=f)

Note: You weren't passing a Body parameter, so I think your code was just creating empty objects in S3.

answered Nov 11, 2017 at 17:35

Mark B

203k27 gold badges339 silver badges332 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

alan pter Over a year ago

Thanks Mark. I got the point now. It worked. I was trying to upload only modified or new files to S3. But the code uploads everything...

Collectives™ on Stack Overflow

Upload file to S3 folder using python boto

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related