Running selenium webdriver in amazon lambda python

Question

I want to run BeautifulSoup and selenium webdriver in amazon lambda and my running environment is python 3.6. Is it possible to run ? if so How. My intention is to scrap datas from a webpage using beautiful soup 4 and selenium(Since it has to scrap data dynamically generated by javascript).

Unable to import module 'lambda_function': No module named 'bs4', and Unable to import module 'lambda_function': No module named ' "Selenium"' @TarunLalwani — skysoft999
– skysoft999, Commented Apr 21, 2018 at 7:24
Did you package those modules as instructed by lambda? Also do remember that there will be no firefox or chrome inside your lambda. So you need to have a external grid which needs to be available to your lambda for it to work — Tarun Lalwani
– Tarun Lalwani, Commented Apr 21, 2018 at 7:29
you can scrap without using selenium also. if you need to click some button or scroll then check for the xhr request in network tab — nishant kumar
– nishant kumar, Commented Apr 21, 2018 at 7:32

robroc · Accepted Answer · 2018-05-01 20:42:23Z

40

Yes, it's possible. You need to package a headless Chrome binary and chromedriver along with all the Python packages you need. You'll also need to set several options in Selenium's Chrome web driver to make it work.

I wrote a step-by-step tutorial after spending several frustrating weeks trying to deploy it.

answered May 1, 2018 at 20:42

robroc

1,2121 gold badge12 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Nicholas Martinez · Accepted Answer · 2018-07-31 19:07:13Z

1

You will need to create a deployment package and upload it to Lambda if you are going to use dependancies outside of the standard library.

I have a write up about using BS4 and Lambda together. I did not use Selenium within Lambda but I do have extensive Selenium experience. You will not be able to execute commands within a browser using Lambda. You are going to need to have a remote server stood up, running Selenium Server. Download Selenium and the webdrivers on the machine that you wish to do the web scraping, start the .jar file, it will open a port on the machine Selenium will communicate with.

Considering that you will need a machine running probably windows to fire up a browser and scrape these pages, you probably don't need lambda in the end.

edited Jul 31, 2018 at 19:07

answered Apr 21, 2018 at 12:12

Nicholas Martinez

5864 silver badges16 bronze badges

2 Comments

Randy31 Over a year ago

You can definitely use selenium inside Lambda using this as a starting point: github.com/21Buttons/pychromeless

Pithikos Over a year ago

You can run any application headless pretty much.. as long as there is a way to control it without a mouse

Collectives™ on Stack Overflow

Running selenium webdriver in amazon lambda python

2 Answers 2

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related