3

My setup - AWS Lambda, AWS Dynamo DB, nodejs and serverless.

I am fetching a JSON object array from some API and trying to insert them in my Dynamo DB. The only solution which I got till now is to iterate over the array and make a DocumentClient() PUT call for every object.
My data has around 1000 objects and write throughput is set to 1. So even that solution fails for me after some 300 records insertion with this error - The level of configured provisioned throughput for the table was exceeded.

Can it be done in a single query?

1

1 Answer 1

7

Each item insertion will consume 1 WCU (per KB), there is no other way.

Instead of individual inserts you could use batchWrite which is more efficient. But still, you need to adjust your insertion rate to the table write throughput.

Also, be aware that your inserts are using 300 seconds of burst capacity (your first 300 insertions) at the beginning but after those are consumed, you can only go 1 write/s with 1 WCU.

Here is an example of batch inserting that will wait between chunks and reschedule failed inserts:

async function batchedAsync({list, callback, chunkSize=10, msDelayBetweenChunks=0}) {
  const emptyList = new Array(Math.ceil(list.length / chunkSize)).fill();
  const clonedList = list.slice(0);
  const chunks = emptyList.map(_ => clonedList.splice(0, chunkSize));
  for (let chunk of chunks) {
    if (msDelayBetweenChunks) {
      await new Promise(resolve => setTimeout(resolve, msDelayBetweenChunks));
    }
    await callback(chunk, chunks);
  }
}

async function writeItems(tableName, chunk, chunks) {
  const {UnprocessedItems} = await documentClient.batchWrite({
    RequestItems: {
      [tableName]: chunk.map(item => {
        return {PutRequest: {Item: item}};
      })
    }
  }).promise();
  if (UnprocessedItems.length) {
    chunks.push(UnprocessedItems);
  }
}

// Example
batchedAsync({
  list: itemsToInsert,
  callback: writeItems.bind(null, myTableName),
  chunkSize: 2, // adjust to provisioned throughput. Max 25 (batchWrite dynamodb limit)
  msDelayBetweenChunks: 1000
});
Sign up to request clarification or add additional context in comments.

3 Comments

Good solution. The fill() function requires a first argument in typescript though.
@nachoab : I am getting Resource Not Found error. Can you please tell me from this code, how we will provide TableName?
@Vishnu replace the "TableName" object key with the actual table name. Take a look at the batchWrite doc I looks like it should probably be a param for "writeItems"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.