Problem in counting elements in DynamoDB with Nodejs using scan

Question

I have a NodeJS function that scan a table in DynamoDB (without primary sort key) and return the number of elements of the column sync that are null. My table:

var params = {
    AttributeDefinitions: [
        {
        AttributeName: "barname",
        AttributeType: "S"
        },
        {
        AttributeName: "timestamp",
        AttributeType: "S"
        }
    ],
    KeySchema: [
        {
        AttributeName: "barname",
        KeyType: "HASH"
        },
        {
        AttributeName: "timestamp",
        KeyType: "RANGE"
        }
    ],
    ProvisionedThroughput: {
        ReadCapacityUnits: 1,
        WriteCapacityUnits: 1
    },
    TableName: tableName
};

The function that count when sync==false

var dynamodb = new AWS.DynamoDB({apiVersion: '2012-08-10'});
async function getCountNoSync(type){
    console.log(type)
    var params = {
        TableName: tableName,
        FilterExpression: 'sync = :sync and billing = :billing',
        ExpressionAttributeValues: {
            ':billing' : {S: type},
            ':sync' : {BOOL: false}
          },
    };
    
    var count = 0;
    await dynamodb.scan(params).promise()
        .then(function(data){
            count = data.Count;
        })
        .catch(function(err) {
            count = 0;
            console.log(err);
        });

    return count;
}

The function works fine If a have few elements in my table (eg. less than 150). If the number of elements are higher, the count variable is always 0. It loooks like the scan do not find all elements.

Any ideia? Best regards

Jens · Accepted Answer · 2021-07-05 18:56:04Z

The reason that you do not find all the items where attribute sync == null is that the scan operation is only reading part of your table.

As the documentation states:

If the total number of scanned items exceeds the maximum dataset size limit of 1 MB, the scan stops and results are returned to the user as a LastEvaluatedKey value to continue the scan in a subsequent operation.

So if your table is several hundred of megabytes big, you need to call scan() multiple times and provide the LastEvaluatedKey to read the next "page" of your table. This process is also called "pagination".

But this will take a lot of time and the time this needs will just increase with your table size. The proper way of doing this would be to create an index of the sync field and then do a query() on that index.

You can read more about that in the AWS documentation:

Collectives™ on Stack Overflow

Problem in counting elements in DynamoDB with Nodejs using scan

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related