Python AWS Lambda Web scraper

Question

I need to scrape a URL to send content to my Lambda function. I am trying to achieve it by packaging BeautifulSoup with my Lambda function, but I am getting import errors like: cannot import name 'CharsetMetaAttributeValue' etc. I am not sure if bs4 should be used in AWS environment or not.

Any suggestions would be helpful.

@hjpotter92, checked Lambda docs, but seems aws doesn't provide any inbuilt thing for creating scraper. — sheenu85
– sheenu85, Commented Mar 28, 2018 at 4:26

sheenu85 · Accepted Answer · 2018-03-28 23:17:59Z

2

I am able to use compiled lxml package with Lambda function for web scraper. Compiled package is available at github this link: https://github.com/JFox/aws-lambda-lxml/tree/master/3.6.4

answered Mar 28, 2018 at 23:17

sheenu85

1682 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python AWS Lambda Web scraper

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related