Lambda + DynamoDB + API gateway - Loop and execute results before returning

Question

I am trying to create a lambda API Call that will go through the results of the DynamoDB table and execute different API calls to 3rd party systems.

The function for scanning is running correctly as it returns me the correct data back.

I have a problem with Iteration here as it wouldn't trigger an Axios call to Linkedin, however, the function is being called console.log bidIt would show in cloud watch.

The cloud watch is not displaying any error.

Ideally function below when finished and added one more API call + modification to dynamo DB will be something like a cron job that will fire every x amount of times within an hour.

If anyone can suggest how can I handle this issue / suggest a different approach I would greatly appreciate it.

const dynamodb = require("aws-sdk/clients/dynamodb");
const axios = require("axios");
const docClient = new dynamodb.DocumentClient();

const tableName = "**************";

exports.executeJobs = async(event) => {
    const { httpMethod, path } = event;
    if (httpMethod !== "GET") {
        throw new Error(
            `Method only accepts GET method, you tried: ${httpMethod} method.`
        );
    }

    console.log("received:", JSON.stringify(event));

    var params = {
        TableName: tableName,
        ProjectionExpression: "#timestamper, #userId,#campaignId, #type,#info,#token",
        FilterExpression: "#timestamper < :timestamper",
        ExpressionAttributeNames: {
            "#timestamper": "timestamper",
            "#type": "type",
            "#info": "info",
            "#userId": "userId",
            "#campaignId": "campaignId",
            "#token": "token",
        },
        ExpressionAttributeValues: {
            ":timestamper": Date.now(),
        },
    };

    console.log("Scanning Jobs table."); 
    
    
      let scanResults = [];
    let items;

    do {
        items = await docClient.scan(params).promise();
        items.Items.forEach(async function (item) { 
            scanResults.push(item)
            
            try { 
    const response = await bidIt(item.campaignId, item.info.currency, item.token, item.info.bid);
    console.log(response);
  } catch (error) {
    console.error(error);
  }
            
        });
        params.ExclusiveStartKey = items.LastEvaluatedKey;
    } while (typeof items.LastEvaluatedKey != "undefined");
     
 
      const response = {
            statusCode: 200,
            body: JSON.stringify(scanResults),
        };

        console.log(
            `response from: ${path} statusCode: ${response.statusCode} body: ${response.body}`
        );
        return response;
};
 
  
    async function bidIt(campaignId, currency, token, bid) {
        console.log("bidIT");
        try {
            axios
                .post(
                    "https://api.linkedin.com/v2/adCampaignsV2/" + campaignId, {
                        patch: {
                            $set: {
                                unitCost: {
                                    amount: bid,
                                    currencyCode: currency,
                                },
                            },
                        },
                    }, {
                        headers: {
                            "Content-Type": "application/json",
                            "X-RestLi-Method": "PARTIAL_UPDATE",
                            Authorization: "Bearer " + token,
                        },
                    }
                )
                .then((result) => {
                     
                    return UpdateCpc(campaignId, currency, token, bid);
                });
        } catch (error) {
            console.log("error", error);
            // appropriately handle the error
        }
    }

Update: Thanks, Kalev for your reply, I have modified code however that didn't still didn't wait for API call to be made and it closed query. Screenshot attached from cloud watch. (I changed bidIt to getMinBid name)

async function getMinBid(campaignId, currency, token, bid) {
    try {
        console.log("getMinBid");
        let result = await axios.post(
                "https://api.linkedin.com/v2/adCampaignsV2/" + campaignId, {
                    patch: {
                        $set: {
                            unitCost: {
                                amount: bid,
                                currencyCode: currency,
                            },
                        },
                    },
                }, {
                    headers: {
                        "Content-Type": "application/json",
                        "X-RestLi-Method": "PARTIAL_UPDATE",
                        Authorization: "Bearer " + token,
                    },
                }
            )
            .then((result) => {
                console.log("axios success");
                console.log(JSON.stringify(result));
                let minCost = result.data.split("lower than ").pop();
                minCost = parseFloat(minCost) + bid; 
            });
          return UpdateCpc(campaignId, currency, token, minCost);
    } catch (error) {
        console.log('test bid')
        console.log("error", error);
        // appropriately handle the error
    }
}

Kalev · Accepted Answer · 2020-06-29 12:18:00Z

3

Original answer

Your problem is with the async call to axios.post in bidIt. The call produces a promise, but you do not await on the promise, nor do you return the promise, so the function returns undefined.

Note that the return UpdateCpc is made within the then handler of the promise. It does not return from the bidIt call (which is what I assume you were trying to do there).

Using async/await, you can do the following:

async function bidIt(campaignId, currency, token, bid) {
    try {
        result = await axios.post(<parameters>));
        return UpdateCpc(campaignId, currency, token, bid);
    } catch (error) { ... }
}

Using promises, you can do the following: (this style is a little redundant, but some may prefer it)

async function bidIt(campaignId, currency, token, bid) {
    return axios.post(<parameters>))
                .then((result) => UpdateCpc(campaignId, currency, token, bid))
                .catch((err) => { ... });
}

Update

I had originally missed the problem in the part of the code that calls bidIt. There too, the asynchronous call is not handled correctly.

In your loop, the await is done within the promise handler, but the rest of the code continues and terminates.

Specifically, you call an async function from the foreach call. All of these function run asynchronously, but you do not await on them, and you do not return them as a promise.

This is your current do-while loop, simplified:

do {
    items = await docClient.scan(params).promise();
    items.Items
         .forEach(async function (item) { 
                      scanResults.push(item)
                      try { 
                          const response = await bidIt(<parameters>);
                      } catch (error) { ... }
                  });
    params.ExclusiveStartKey = items.LastEvaluatedKey;
} while (typeof items.LastEvaluatedKey != "undefined");

Note how the await bidIt call is in the scope of the async function called from items.Items.forEach. The call to items.Items.forEach itself creates a bunch of asynchronous calls, and continues the execution before the asynchronous calls get a chance to finish.

Fix

Using async/await, you can fix the loop in the following way:

do {
    items = await docClient.scan(params).promise();
    for (item of items.Items) {
        scanResults.push(item)
        try { 
            const response = await bidIt(<parameters>);
        } catch (error) { ... }
    }
    params.ExclusiveStartKey = items.LastEvaluatedKey;
} while (typeof items.LastEvaluatedKey != "undefined");

The same can also be achieved using promises, but will require collecting all produced promises, and then returning a combined promise, by doing Promise.all(...) on the set of promises that your code produces. It requires some nuance, and would necessitate changing more than just the do-while loop, so I'm omitting the code here. It's not complicated, but you do need to be careful not to lose any promises.

edited Jun 29, 2020 at 12:18

answered Jun 28, 2020 at 20:53

Kalev

1,1888 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Remik B Over a year ago

I have applied your solution but axios is still not firing.

Kalev Over a year ago

I have added the fix to the code in an update to my answer. Asynchronous programming can be tricky at times. You should pay careful attention to the scope of your calls, and to whether you await all the asynchronous calls. In general, when it seems like an async call is not running, this is usually the problem.

Remik B Over a year ago

That's Great Kalev you're a lifesaver. It is working I just go console.log output from inside of bidIt function. Thanks a lot.

Collectives™ on Stack Overflow

Lambda + DynamoDB + API gateway - Loop and execute results before returning

1 Answer 1

Original answer

Update

Fix

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Original answer

Update

Fix

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related