AWS DynamoDB and Lambda: Scan optimizations / performance

Question

To store api-gateway websocket-connections, I use a dynamoDB-table. When posting to stored connections, I retrieve the connection in a lambda-function via:

const dynamodb = new DynamoDB.DocumentClient();
const { Items, Count } = await dynamodb.scan({ TableName: 'Websocket' }).promise();

// post to connections

This is not really fast; the query takes around 400 - 800ms which could be better in my opinion. Can I change something on my implementation or is there maybe another aws-service which is better for storing these tiny infos about the websocket-connection (its really just a small connection-id and a user-id)?

Do you retrieve the whole table without filtering? what happens if that table grows — Alex
– Alex, Commented Jan 12, 2020 at 15:46
Do you need to get the entire table? Isn't there a correlator that you could use from the API context as a lookup key? — jarmod
– jarmod, Commented Jan 12, 2020 at 16:05
I only have a maximum of ~2-20 users at the same time, so I want to retrieve complete contents of the table. For my tests there was only 1 entry. — misantronic
– misantronic, Commented Jan 12, 2020 at 16:27
you are hitting from local machine when it is giving 400-800 ms? — best wishes
– best wishes, Commented Jan 13, 2020 at 3:59

best wishes · Accepted Answer · 2020-01-12 16:35:16Z

It has nothing to do with dynamodb, if you do a scan on any database which reads from disk, it will take time and money from your pocket.

You can use any of below solution to achieve what you are doing.

Instead of storing all the websocket ids as separate row, consider having single record in which ids are stored, so that you can do a single query (not scan) and proceed.

Cons:

a. multiple writes to same row will result in race condition. and few reads might get lost, you can use conditional write to update record to solve this problem (have a always increasing version, and update the record only if version in db = version you read from db)

b. There is a limit on size of single document in dynamodb. As of now it is 400kb.
Store websocket id as separate row but group them by different keys, and create secondary index on these keys. Store the keys in a single row. While doing a fetch first get all relevant groups, and then query (not scan) all the items of that group. It will not exactly solve your problem but you can do interesting things like, let's say there are 10 groups, every second, messages for 1 groups are sent. this will make sure that load on your message sending infrastructure is also balanced. And you can keep incrementing number of groups as user increases.
Keep the ids in a cache like aws elastic cache and add/remove ids as new entries are made in dynamodb by using aws lambda and dyanmodb streams. It will make sure that you reads are fast. At the same time if cache goes down you can use dynamodb to populate it again by doing scan on dynamodb.

Cons:

a. Extra component to maintain.

Collectives™ on Stack Overflow

AWS DynamoDB and Lambda: Scan optimizations / performance

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related