DynamoDB scan returns multiple scan results

Question

So I've written the below function. This version is a bit abridged and I've anonymized the data but the critical components are there.

The function basically takes in a list of parameters from an API-Gateway call, queries a db for each of them then returns the results.

I'm finding that the scan runs perfectly with one parameter, but returns duplicate data when more than 1 are called . From the logs I can see that the scans are running multiple times when multiple params are passed

For example, with one param the function logs return

2020-03-19 20:27:42.974 Starting the 0 scan with 3 as the id 
2020-03-19 20:27:43.047 The 0 scan has completed successfully

With two params the logs are

2020-03-19 20:28:42.189 Starting the 0 scan with 2 as the id
2020-03-19 20:28:42.261 The 0 scan has completed successfully
2020-03-19 20:28:42.262 Starting the 1 scan with 3 as the id
2020-03-19 20:28:42.267 The 0 scan has completed successfully
2020-03-19 20:28:42.293 The 1 scan has completed successfully

And with 3 params the logs are

2020-03-19 20:29:49.209 Starting the 0 scan with 1 as the id
2020-03-19 20:29:49.323 The 0 scan has completed successfully
2020-03-19 20:29:49.325 Starting the 1 scan with 2 as the id
2020-03-19 20:29:49.329 The 0 scan has completed successfully
2020-03-19 20:29:49.380 The 1 scan has completed successfully
2020-03-19 20:29:49.381 Starting the 2 scan with 3 as the id
2020-03-19 20:29:49.385 The 1 scan has completed successfully
2020-03-19 20:29:49.437 The 2 scan has completed successfully

Here is the code that runs the for loop and the scan. I've hardcoded the parameters and excluded some non-pertinent stuff

     const params = ['1','2','3'];
     for (let i = 0; i < params.length; i++) {
      console.log("Starting the " + i + " scan with " + params[i] + " as the scan parameter")
      const scanParams = {
      TableName: "Dynamo_Table",
      FilterExpression: "Org = :Org",
      ExpressionAttributeValues: { ":Org": params[i] },
      ProjectionExpression: "User_ID, Org, first_name, last_name"
     };
     await dynamoClient.scan(scanParams, function(err, data) {
      if (err) {
        console.log("data retrival failed, error logged is :" + err);
        return err;
      }
      else {
        console.log("The " + i +" scan has completed successfully")
        //console.log("data retrival successful: " + JSON.stringify(data));
        userData = userData.concat(data.Items)
        //console.log("partial data structure is " + data)
      }
    }).promise();
  }
      responseData = JSON.stringify(userData)
      console.log("Complete response is " + responseData)
      console.log("data after execution scan is " + data)

I've tried to force the program to wait on the scan's competition by defining a wait and using AWS's .promise() function. However, these don't seem to be blocking the thread execution. I'm not sure exactly why its launching multiple scans though. The for loop isn't running more times than it should, so why is the search function getting called?

So you just want to query items with certain org values. Is Org the partition key? Also, you are misusing the SDK functions. You're awaiting something that is not a promise. You've mixed up promises and callbacks. You would use const data = await dynamoClient.scan(params).promise();, but scan is the wrong solution here. — jarmod
– jarmod, Commented Mar 19, 2020 at 23:30
Thanks jarmod, I realized that a few days ago and it totally slipped my mind. Would the code you shared also work with a query? I'd test it now but I'm away from my work comp — ronald mcdolittle
– ronald mcdolittle, Commented Mar 20, 2020 at 2:57
Yes, but you want multiple queries, one per org, assuming that Org is indeed the partition key(is it?) — jarmod
– jarmod, Commented Mar 20, 2020 at 12:02
Yeah it is, in this case I'm already running scan in the for loop so shouldn't need to modify anything besides that one command — ronald mcdolittle
– ronald mcdolittle, Commented Mar 20, 2020 at 13:16
Hey one other question. The code you shared, why does it work? Does the scan return a callback not a promise? I'm pretty fuzzy on node's async functionality — ronald mcdolittle
– ronald mcdolittle, Commented Mar 20, 2020 at 13:18

Phani Teja · Accepted Answer · 2020-03-19 22:29:46Z

2

Whenever you want to search something in your DynamoDB database it's recommended that you use the Query option instead of Scan

This is because the Scan reads each and every item of the database whereas Query only looks for the mentioned Hask key (primary key).

If you want to look for data with a particular "attribute" in your mind you can use Global Secondary Index wherein you can set the "attribute" as the Hash key and at the same time pick a Sort key of your choice. This might solve your problem wherein the table is returning the answer multiple times.

answered Mar 19, 2020 at 22:29

Phani Teja

17310 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

jarmod · Accepted Answer · 2020-03-20 19:49:23Z

Here is an example of how to use the DynamoDB DocumentClient to query multiple items by partition key and collect the results. This uses the promisified variant of the query() call, and waits for all query promises to be fulfilled using Promise.all().

var AWS = require('aws-sdk');
AWS.config.update({ region: 'us-east-1' });

const dc = new AWS.DynamoDB.DocumentClient();

// Array of organization IDs we want to query
const orgs = ['1', '2', '3'];

// Async function to query for one specific organization ID
const queryOrg = async org => {
  const params = {
    TableName: 'orgs',
    KeyConditionExpression: 'org = :o1',
    ExpressionAttributeValues: { ':o1': org, },
  };

  return dc.query(params).promise();
}

// Async IIFE because you cannot use await outside of an async function
(async () => {
  // Array of promises representing async organization queries made
  const promises = orgs.map(org => queryOrg(org));

  // Wait for all queries to complete and collect the results in an array
  const items = await Promise.all(promises);

  // Results are present in the same order that the queries were mapped
  for (const item of items) {
    console.log('Item:', item.Items[0]);
  }
})();

Collectives™ on Stack Overflow

DynamoDB scan returns multiple scan results

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related