lambda function fails with s3 key error. in amazon web services

Question

I have a lambda function that moves files from one s3 bucket to another :

import json
import boto3
from datetime import datetime, timedelta

def lambda_handler(event, context):
    # TODO implement

    SOURCE_BUCKET = 'source-bucket'
    DESTINATION_BUCKET = 'destination-bucket'

    s3_client = boto3.client('s3')

    # Create a reusable Paginator
    paginator = s3_client.get_paginator('list_objects_v2')

    # Create a PageIterator from the Paginator
    page_iterator = paginator.paginate(Bucket=SOURCE_BUCKET)

    # Loop through each object, looking for ones older than a given time period
    for page in page_iterator:
        for object in page['Contents']:
            if object['LastModified'] < datetime.now().astimezone() - timedelta(hours=1):   # <-- Change time period here
                print(f"Moving {object['Key']}")

                # Copy object
                s3_client.copy_object(
                    ACL='bucket-owner-full-control',
                    Bucket=DESTINATION_BUCKET,
                    Key=object['Key'],
                    CopySource={'Bucket':SOURCE_BUCKET, 'Key':object['Key']}
                )

                # Delete original object
                s3_client.delete_object(Bucket=SOURCE_BUCKET, Key=object['Key'])

I am getting error :

Response:
{
  "errorMessage": "'Contents'",
  "errorType": "KeyError",
  "stackTrace": [
    "  File \"/var/task/lambda_function.py\", line 21, in lambda_handler\n    for object in page['Contents']:\n"
  ]
}

Request ID:
"518e0f39-63e4-43df-842d-b73d56f83cd8"

Function Logs:
START RequestId: 518e0f39-63e4-43df-842d-b73d56f83cd8 Version: $LATEST
[ERROR] KeyError: 'Contents'
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 21, in lambda_handler
    for object in page['Contents']:END RequestId: 518e0f39-63e4-43df-842d-b73d56f83cd8
REPORT RequestId: 518e0f39-63e4-43df-842d-b73d56f83cd8  Duration: 1611.00 ms    Billed Duration: 1700 ms    Memory Size: 128 MB Max Memory Used: 76 MB  Init Duration: 248.12 ms

can someone help here. It has moved all the files but still giving me error.

Chris Williams · Accepted Answer · 2020-06-28 10:19:27Z

1

This is assuming that the key Contents is always returned. If there are not objects in the bucket this will not exist.

Add a simple if "Contents" in page to handle it not always existing.

So your function code might look like

import json
import boto3
from datetime import datetime, timedelta

def lambda_handler(event, context):
    # TODO implement

    SOURCE_BUCKET = 'source-bucket'
    DESTINATION_BUCKET = 'destination-bucket'

    s3_client = boto3.client('s3')

    # Create a reusable Paginator
    paginator = s3_client.get_paginator('list_objects_v2')

    # Create a PageIterator from the Paginator
    page_iterator = paginator.paginate(Bucket=SOURCE_BUCKET)

    # Loop through each object, looking for ones older than a given time period
    for page in page_iterator:
        if "Contents" in page:
            for object in page['Contents']:
                if object['LastModified'] < datetime.now().astimezone() - timedelta(hours=1):   # <-- Change time period here
                    print(f"Moving {object['Key']}")

                    # Copy object
                    s3_client.copy_object(
                        ACL='bucket-owner-full-control',
                        Bucket=DESTINATION_BUCKET,
                        Key=object['Key'],
                        CopySource={'Bucket':SOURCE_BUCKET, 'Key':object['Key']}
                    )

                    # Delete original object
                    s3_client.delete_object(Bucket=SOURCE_BUCKET, Key=object['Key'])
        else:
            print("No Contents key for page!")

edited Jun 28, 2020 at 10:19

answered Jun 28, 2020 at 10:07

Chris Williams

35.7k4 gold badges46 silver badges79 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

cloudbud Over a year ago

yes it worked, do you think I should have try catch or it is fine that way?

Chris Williams Over a year ago

I suppose it depends how you want to handle the failure, I personally prefer to validate through checks rather than have it fall to an exception. I would suggest adding an elif to do some logging perjaps though

cloudbud Over a year ago

what will elif will have? could you update it in answer?

Chris Williams Over a year ago

Oops meant else, updated answer, Basically just logging that it was skipped for debug purposes. Always helps to understand after running the script :)

Collectives™ on Stack Overflow

lambda function fails with s3 key error. in amazon web services

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related