0

TL;DR: getting errors using AWS Lambda and API-Gateway

Background

I wrote a python script to help me track some index funds I am using. It basically scrapes (using requests and bs4) the price from different websites.

I wanted to turn this into an API so I can use it anywhere, so I used Flask and Connexion to make this (followed this https://realpython.com/flask-connexion-rest-api/). That's all working perfectly fine locally, but I'm having trouble implementing this on AWS.

I have zipped up my code and the corresponding app packages to AWS Lambda and using that in the API Gateway. However I am getting the following errors:

Endpoint response body before transformations: {"errorMessage": "module initialization error"} Method response body after transformations: {"errorMessage": "module initialization error"}

I've googled for this, followed the steps here (Amazon API Gateway : response body is not transformed when the API is called via Postman?), deployed the app etc - not sure why this is happening.

here is a snippet of the code:

import requests
from bs4 import BeautifulSoup
from datetime import date, timedelta
import datetime
import json


def CurrentPortfolioValue(event, context):
    initial_porfolio_price = float (100)
    initial_date = date(2018, 8, 19) #set the initial date to a date where the fortnightly payment has gone
    num_of_shares_vanguard = float(500)
    num_of_shares_AMPNZShares = float(300)
    num_of_shares_AMPAustralisianProperty = float(100)
    fortnightly_contrib = float(100)
    current_value = initial_porfolio_price
    #print(initial_date)

    #sets a user-agent    
    example_headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0'} #can make this better by having a list of random header


    #vanguard
    vanguard = requests.get("https://www.bloomberg.com/quote/VAN1579:AU",headers=example_headers) #need to have headers to avoid being blocked
    soup = BeautifulSoup(vanguard.content, 'html.parser')
    vanguard_price = soup.find_all("span", class_ = 'priceText__1853e8a5')[0].get_text()
    #print("Vanguard Price:\n" + vanguard_price+"AUD")

    #AMP Capital NZ Index 
    ampnzshares = requests.get("https://www.ampcapital.com/nz/en/investments/funds/index-funds/nz-shares-index-fund",headers=example_headers) #need to have headers to avoid being blocked
    soup1 = BeautifulSoup(ampnzshares.content, 'html.parser')
    ampnzshares_price = soup1.find_all("div", class_ = 'ht-module_title ht-highlight_color')[1].get_text()
    #print("AMP Capital NZ Index Price:\n" + ampnzshares_price+"NZD")

    #AMP Capital Australisian Property Index Fund
    ampAustralisianProperty = requests.get("https://www.ampcapital.com/nz/en/investments/funds/index-funds/australasian-property-index-fund",headers=example_headers) #need to have headers to avoid being blocked
    soup2 = BeautifulSoup(ampAustralisianProperty.content, 'html.parser')
    ampAustralianProperty_price = soup2.find_all("div", class_ = 'ht-module_title ht-highlight_color')[1].get_text()
    #print("AMP Capital Australisian Property Index Fund:\n" + ampAustralianProperty_price+"NZD")

    #get day difference between today and initial date
    current_date = date.today()
    days_difference = (current_date - initial_date).days #.days shows just the number of days
    #print(days_difference)

    count = days_difference
    while count > 14: #while difference is less than 14 days i.e forntightly payment hasnt been made
        current_value += initial_porfolio_price + (float(50) * float(vanguard_price)) + (float(50) * float(ampnzshares_price))
        count = count - 14
    print("Current Portfolio value is $" + str(current_value))
    return {
        "statusCode": 200,
        "body": json.dumps(current_value)}

CurrentPortfolioValue() #retrieves current portfolio value

my yaml file looks like ths:

swagger: "2.0"
info:
  description: This is the swagger file that goes with our server code
  version: "1.0.0"
  title: Swagger REST Article
consumes:
  - "application/json"
produces:
  - "application/json"

basePath: "/api"

paths:
  /value: #name of the endpoint
    get:
      operationId: "ShareScraper.CurrentPortfolioValue" #link to the module - this is called when you navigate to /value
      tags:
        - "Portfolio"
      summary: "The portfolio data structure supported by the server application"
      description: "Read the current portfolio value"
      responses:
        200:
          description: "Successful read current portfolio value"
          schema:
            type: "string"
            items:
              properties:
                portfolio_value:
                  type: "string"

Thanks in advance!

[Edit] This is my first time creating an api and using AWS so sorry if I am missing something fairly simple/obvious

Improve this question
11
  • Is your current_value defined? The function code is appending a number to it. Commented Aug 27, 2019 at 10:04
  • @MunavirChavody yes current_value is defined. I haven't included that bit of the code as it is just calculations. The code fully works as expected, it's just not working in aws Commented Aug 27, 2019 at 10:15
  • Can you try running the lambda function alone and see how the execution goes? You might get more detailed error there. Commented Aug 27, 2019 at 10:17
  • I'm not too sure how to run just the lambda function, is it from the Test button? (im using the online Console) Commented Aug 27, 2019 at 10:37
  • Yeah. Same. Use Test button. If you have any test object to be passed to the event variable, pass it to the pop up. Commented Aug 27, 2019 at 10:40

2 Answers 2

Reset to default
1

Your Lambda function doesn’t appear to be correctly formatted as a Lambda. As per the AWS documentation, here, your Lambda must be formatted as a handler and must accept the event and context arguments.

You will need to edit your code to look more like this in order to get it to run:

from botocore.vendored import requests
from bs4 import BeautifulSoup
import datetime
from datetime import date, timedelta
import json


def lambda_handler(event, context):

    initial_porfolio_price = float (100)
    initial_date = date(2018, 8, 19) #set the initial date to a date where the fortnightly payment has gone
    num_of_shares_vanguard = float(500)
    num_of_shares_AMPNZShares = float(300)
    num_of_shares_AMPAustralisianProperty = float(100)
    fortnightly_contrib = float(100)
    current_value = initial_porfolio_price
    #print(initial_date)

    #sets a user-agent
    example_headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0'} #can make this better by having a list of random header

    #vanguard
    vanguard = requests.get("https://www.bloomberg.com/quote/VAN1579:AU",headers=example_headers) #need to have headers to avoid being blocked
    soup = BeautifulSoup(vanguard.content, 'html.parser')
    print(soup)
    vanguard_price = soup.find_all("span", class_ = 'priceText__1853e8a5')[0].get_text()
    #print("Vanguard Price:\n" + vanguard_price+"AUD")

    #AMP Capital NZ Index
    ampnzshares = requests.get("https://www.ampcapital.com/nz/en/investments/funds/index-funds/nz-shares-index-fund",headers=example_headers) #need to have headers to avoid being blocked
    soup1 = BeautifulSoup(ampnzshares.content, 'html.parser')
    ampnzshares_price = soup1.find_all("div", class_ = 'ht-module_title ht-highlight_color')[1].get_text()
    #print("AMP Capital NZ Index Price:\n" + ampnzshares_price+"NZD")

    #AMP Capital Australisian Property Index Fund
    ampAustralisianProperty = requests.get("https://www.ampcapital.com/nz/en/investments/funds/index-funds/australasian-property-index-fund",headers=example_headers) #need to have headers to avoid being blocked
    soup2 = BeautifulSoup(ampAustralisianProperty.content, 'html.parser')
    ampAustralianProperty_price = soup2.find_all("div", class_ = 'ht-module_title ht-highlight_color')[1].get_text()
    #print("AMP Capital Australisian Property Index Fund:\n" + ampAustralianProperty_price+"NZD")

    #get day difference between today and initial date
    current_date = date.today()
    days_difference = (current_date - initial_date).days #.days shows just the number of days
    #print(days_difference)

    count = days_difference
    while count > 14: #while difference is less than 14 days i.e forntightly payment hasnt been made
        current_value += initial_porfolio_price + (float(50) * float(vanguard_price)) + (float(50) * float(ampnzshares_price))
    count = count - 14
    print("Current Portfolio value is $" + str(current_value))
    return {
       "statusCode": 200,
       "body": json.dumps(current_value)}

You didn’t post your import statements but at the very least you’ll need to import json.

Note that the call to the function is also removed. I’ve assumed that this function is running in isolation (ie nothing else running with it) and therefore you do not need to call it within the Lambda itself, as it will get executed in the Lambda environment.

EDIT:

I've updated the code block and run it. It is indeed running in Lambda, however, at line 27 it is trying to find "span", class_ = 'priceText__1853e8a5' which doesn't exist in the response to the vanguard = requests.get... request. Instead, as you can see in the screenshot below, we're receiving something else, and Lambda is giving us the error "errorMessage": "list index out of range".

To find out why, we have to look in the Log Output. When we look at the output we see that instead of a page with the pricing information that we're trying to scrape, what is being received is actually a Captcha page; Bloomberg recognize that you're scraping the data and they don't want people scraping their data, so to prevent this they want you to complete a Captcha, and because haven't built a way to handle this situation the Lambda is erroring out.

Lambda Image

Improve this answer
Sign up to request clarification or add additional context in comments.

10 Comments

ive edited the post to include what imports are happening. this function runs in isolation yes. I have tried adding the event and context, getting a different error now: module initialization error: list index out of range
You are missing a bunch of values... You need to assign values to initial_portfolio_value, count, vanguard_price, amznshares, so you can use them. Also, it should be = not += for the current_value definition. You could also pass these values in either via the body of the event or as query string parameters and they’ll be accessible in your code as part of the Event object.
Also worth noting that the Print statement will not return that to the caller, that print will only appear in the logs, the Lambda will only return what is in the return statement.
sorry i wasnt clear before, the code is working locally - i have omitted all of that code from the post. So all of those values are assigned
They have to be accessible inside the Lambda, somehow... Please include the complete function code (anonymized) so that I can update my answer.
|
1

Your lambda code doesn't have the event and context argument. Which are required arguments for a lambda python handler. You can modify your code as,

def CurrentPortfolioValue(event, context):

    current_value += initial_porfolio_price + (float(50) * float(vanguard_price)) + (float(50) * float(ampnzshares_price))
    count = count - 14
    print("Current Portfolio value is $" + str(current_value))
    return {
        "statusCode": 200,
        "body": json.dumps(current_value)}

CurrentPortfolioValue() #retrieves current portfolio value
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.