Query all rows from DynamoDB using python

Question

As far as I went, there is very little information on how to retrieve all rows of a DynamoDB.

I used table.scan() But when using this method you have a limit, so you will not get all the items.

I tried table.query() But it says: Either the KeyConditions or KeyConditionExpression parameter must be specified in the request.

Thanks a lot

Sasho Ristovski · Accepted Answer · 2020-02-06 14:31:58Z

1

The only way is to use table.scan() but you will need to add consistent read.

Scan uses eventually consistent reads when accessing the data in a table; therefore, the result set might not include the changes to data in the table immediately before the operation began. If you need a consistent copy of the data, as of the time that the Scan begins, you can set the ConsistentRead parameter to true . Here is a link to the documentation. https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Client.scan

If LastEvaluatedKey is empty, then the "last page" of results has been processed and there is no more data to be retrieved.

If LastEvaluatedKey is not empty, it does not necessarily mean that there is more data in the result set. The only way to know when you have reached the end of the result set is when LastEvaluatedKey is empty.

edited Feb 6, 2020 at 14:31

answered Feb 6, 2020 at 13:57

Sasho Ristovski

1471 silver badge7 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Miguel Santos Over a year ago

Can you just type the line of code or something? that is not clear for someone who knows little about Dynamo

Miguel Santos Over a year ago

response = table.scan( ConsistentRead=True ) this does not work as I still get limited data

Sasho Ristovski Over a year ago

if this didn't helped than you will try to get it with parallel scan.. You will need to add TotalSegments and Segments parameters also in your table scan.

Sasho Ristovski Over a year ago

Also try to add ReturnConsumedCapacity parameter also to check your capacity when scanning your table.

Miguel Santos Over a year ago

As far as I know, parallel scan also has limit capacity (1MB default like scan?) I am using the LastEvaluatedKey and do a loop of scans but this is very slow. So in the end I would also need a loop for the parallel scan using the LastEvaluatedKey right?

|

Collectives™ on Stack Overflow

Query all rows from DynamoDB using python

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related