Using PhantomJS for headless browser in Selenium - Python

Question

I'm currently using Python-Selenium to run some tests. It's supposed to log in the webpage, input username and password, and do some other things. When I execute it using the Firefox browser, it works fine, however when I use PhantonJS I'm getting the following error:

2016-01-29 16:18:29 - ERROR - An exception occurred Message: {"errorMessage":"Unable to find element with id 'user_email'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"91","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:57257","User-Agent":"Python-urllib/2.7"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"id\", \"sessionId\": \"fcd54080-c6e6-11e5-94bd-27954be890c7\", \"value\": \"user_email\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/fcd54080-c6e6-11e5-94bd-27954be890c7/element"}}
Screenshot: available via screen
Traceback (most recent call last):
  File "webcrawler.py", line 105, in login
    email = self.singleton.driver.find_element_by_id("user_email")
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 234, in find_element_by_id
    return self.find_element(by=By.ID, value=id_)
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 712, in find_element
    {'using': by, 'value': value})['value']
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 201, in execute
    self.error_handler.check_response(response)
  File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 181, in check_response
    raise exception_class(message, screen, stacktrace)
NoSuchElementException: Message: {"errorMessage":"Unable to find element with id 'user_email'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"91","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:57257","User-Agent":"Python-urllib/2.7"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"id\", \"sessionId\": \"fcd54080-c6e6-11e5-94bd-27954be890c7\", \"value\": \"user_email\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/fcd54080-c6e6-11e5-94bd-27954be890c7/element"}}
Screenshot: available via screen

Debugging the issue a little more, I find the page content is:

DEBUG - <html><head></head><body></body></html>

This is I start the Selenium driver:

def __init__(self, browser="phantomjs"):
        if browser == "phantomjs":
            self.driver = webdriver.PhantomJS()
            self.driver.set_window_size(1120, 550)
        elif browser == "firefox":
            self.driver = webdriver.Firefox()
        elif browser == "chrome":
            self.driver = webdriver.Chrome()
        elif browser == "remote":
            self.driver = webdriver.Remote()
        elif browser == "ie":
            self.driver = webdriver.Ie()
        else:
            raise ValueError("Invalid browser value:  %s"  % browser)

Could someone tell me how to fix that? It works perfectly using Firefox, but I'm going to deploy this in a node in AWS so I need to use some some of headless browser.

cybertextron · Accepted Answer · 2016-01-30 00:29:12Z

2

The option given by @alecxe is valid too. In my particular case, the webpage uses SSL, so the solution is to create the PhantomJS as:

self.driver = webdriver.PhantomJS(service_args=['--ignore-ssl-errors=true', '--ssl-protocol=any'])

Also make sure to set the window size, in order to create a dummy browser:

self.driver.set_window_size(1120, 550)

answered Jan 30, 2016 at 0:29

cybertextron

11k32 gold badges117 silver badges220 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

alecxe Over a year ago

Yeah, I remember using that --ignore-ssl-errors flag to tackle different PhantomJS-specific issues too. Thanks.

Community · Accepted Answer · 2017-05-23 12:19:29Z

1

First of all, you can still use Firefox on AWS, just run it under a virtual display:

How do I run Selenium in Xvfb?

As for your PhantomJS specific issue, you need a wait - wait for the presence of the element:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebdriverWait(driver, 10)

email_element = wait.until(EC.presence_of_element_located((By.ID, "user_email")))
email_element.send_keys(email)

edited May 23, 2017 at 12:19

CommunityBot

11 silver badge

answered Jan 30, 2016 at 0:27

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Collectives™ on Stack Overflow

Using PhantomJS for headless browser in Selenium - Python

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related