Infinite loop in DynamoDB scan

Question

As explained in Complete scan of dynamoDb with boto3 I build a solution to full scan a DynamoDB table using some condition. This is my code:

def dynamo_scan(table_name, params_dict):
    dynamo_table = boto3.resource('dynamodb').Table(table_name)
    response = do_scan(dynamo_table, params_dict)
    results = response['Items']
    while response.get('LastEvaluatedKey'):
        print "Iterating"
        print response.get('LastEvaluatedKey')
        params_dict['ExclusiveStartKey'] = response['LastEvaluatedKey']
        response = do_scan(dynamo_table, params_dict)
        results.extend(response['Items'])
    return results

def do_scan(dynamo_table, params_dict):
    return dynamo_table.scan(**params_dict)

But when the code is executed it gets into an infinite loop. This is the output for the each iteration of LastEvaluatedKey:

{u'my_id': u'1698', u'identity': u'2017075002312'}
{u'my_id': u'1883', u'identity': u'85500397082900013318629'}
{u'my_id': u'1698', u'identity': u'2017075002312'}
{u'my_id': u'1883', u'identity': u'85500397082900013318629'}
{u'my_id': u'1698', u'identity': u'2017075002312'}
{u'my_id': u'1883', u'identity': u'85500397082900013318629'}

And it keeps looping trough those two pairs.

EDITED:

I added the method do_scan.

LexMulier · Accepted Answer · 2017-04-10 15:36:50Z

2

I'm not sure what your do_scan() does exactly, but this works:

def dynamo_scan(table_name, params_dict):
    dynamo_table = boto3.resource('dynamodb').Table(table_name)
    response = dynamo_table.scan(params_dict)
    results = response['Items']

    while 'LastEvaluatedKey' in response:
        response = dynamo_table.scan(ExclusiveStartKey=response['LastEvaluatedKey'])
        results += response['Items']

    return results

answered Apr 10, 2017 at 15:36

LexMulier

2734 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Enrique Dev Over a year ago

Yes, I added my method do_scan. I think is the same solution as yours, but i gets every time into an infinite loop.

LexMulier Over a year ago

How much data is in your database?

Collectives™ on Stack Overflow

Infinite loop in DynamoDB scan

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related