I have a question. Is it possible to run a python script, which using selenium to scrape the data from dynamic website, on server(amazon ec2). I trying to find some info about this a lot of times, but this is without results.
Thanks everyone!
1 Answer
EDIT 2024: chrome_options= updated to options= to support latest version.
Servers don't use GUI/Windows and they even don't have monitors. It is called 'headless' server (because monitor looks like server's head)
Selenium runs web browser which needs GUI/Windows to display its window.
But some web browsers can also run without displaying window - they can run headless.
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(options=chrome_options) # `chrome_options=` converted to `options=`
For Firefox should be similar code.
There was also Selenium's webdrive PhantomJS which worked headless but it is not developed any more.
Linux has also program Xvfb which can create fake/virtual monitor on headless computer and it could run programs which can't run without GUI/XWindows.
I didn't check it but one of this method may work on Amazon EC2.
