Pycharm: Selenium with Python: Unable to find web elements using headless chrome

Question

When I run the below code with headless chrome I'm facing issues while identifying the elements. The same code runs just runs fine by commenting the below lines with head full mode.

# chrome_options.add_argument('--headless')

# chrome_options.add_argument('--disable-gpu')

Test Details:

Operating system: Windows10

ChromeDriver: 75.0.3770.8

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
driver = webdriver.Chrome(ChromeDriverManager().install(), options=chrome_options)
wait = WebDriverWait(driver,30)
driver.maximize_window()
driver.get('https://learn.letskodeit.com/')
print(driver.title)
wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Login'))).click()
wait.until(EC.visibility_of_element_located((By.ID, 'user_email'))).send_keys("[email protected]")
wait.until(EC.visibility_of_element_located((By.ID, 'user_password'))).send_keys("abcabc")
wait.until(EC.visibility_of_element_located((By.NAME, 'commit'))).click()
print(driver.title)
driver.close()
driver.quit()

Output:

"C:\Program Files (x86)\Python37-32\python.exe" C:/PycharmProjects/seleniumwd2/basics/RunHeadlessChromeTests.py

Checking for win32 chromedriver:75.0.3770.8 in cache
Driver found in C:\Users\vishr\.wdm\chromedriver\75.0.3770.8\win32/chromedriver.exe
Home | Let's Kode It
Traceback (most recent call last):
  File "C:/PycharmProjects/seleniumwd2/basics/RunHeadlessChromeTests.py", line 15, in <module>
    wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Login'))).click()
  File "C:\Users\vishr\AppData\Roaming\Python\Python37\site-packages\selenium\webdriver\support\wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 


Process finished with exit code 1

Your stacktrace suggests an issue locating an element with an id attribute of user_email. Is there such an element on the page-under-test? — orde
– orde, Commented Jun 12, 2019 at 5:12

KunduK · Accepted Answer · 2019-06-12 08:52:52Z

for headless browser you have to set the window size to fire on event.Because headless browser can't recognise where to click without window size.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('window-size=1920x1080')

driver = webdriver.Chrome(executable_path='path/to/chrome driver',options=chrome_options)
wait = WebDriverWait(driver,30)
driver.maximize_window()
driver.get('https://learn.letskodeit.com/')
print(driver.title)
wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Login'))).click()
wait.until(EC.visibility_of_element_located((By.ID, 'user_email'))).send_keys("[email protected]")
wait.until(EC.visibility_of_element_located((By.ID, 'user_password'))).send_keys("abcabc")
wait.until(EC.visibility_of_element_located((By.NAME, 'commit'))).click()
print(driver.title)
driver.close()
driver.quit()

Printed output on console on headless mode.

Home | Let's Kode It
Let's Kode It

Eugene K · Accepted Answer · 2019-06-12 05:40:38Z

0

You're clicking a button that loads another page. You have to wait for the other page to load. This is tricky to get perfect.

The easiest solution:

sleep(3)

The better solution using implicit waiting:

driver.implicitly_wait(3)

An even better solution using explicit waiting:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

WebDriverWait(driver, 3).until(EC.presence_of_element_located((By.ID, "user_email")))

An even better better solution would be to catch exceptions, and layer all these solutions.

answered Jun 12, 2019 at 5:40

Eugene K

3,4672 gold badges25 silver badges36 bronze badges

Comments

undetected Selenium · Accepted Answer · 2019-06-12 06:09:54Z

0

To click() on the element with text as Login you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following solutions:

Using PARTIAL_LINK_TEXT:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "Login"))).click()

Using CSS_SELECTOR:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.navbar-link.fedora-navbar-link[href='/sign_in']"))).click()

Using XPATH:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@class='navbar-link fedora-navbar-link' and @href='/sign_in']"))).click()

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

answered Jun 12, 2019 at 6:09

undetected Selenium

194k44 gold badges304 silver badges387 bronze badges

7 Comments

VISHVAMBRUTH J T Over a year ago

Can you take a look at my code and help me to run it via headless chrome.

undetected Selenium Over a year ago

There is absolutely nothing wrong in your code. You simply need to replace the EC visibility_of_element_located() & ID to element_to_be_clickable & Xpath / Css

VISHVAMBRUTH J T Over a year ago

Changed to still no luck. wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Login'))).click() wait.until(EC.element_to_be_clickable((By.ID, 'user_email'))).send_keys("[email protected]") wait.until(EC.element_to_be_clickable((By.ID, 'user_password'))).send_keys("abcabc") wait.until(EC.element_to_be_clickable((By.NAME, 'commit'))).click()

undetected Selenium Over a year ago

Where did you get LINK_TEXT from? In my answer it's PARTIAL_LINK_TEXT

VISHVAMBRUTH J T Over a year ago

Changing to PARTIAL_LINK_TEXT also dint work: wait.until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, 'Login'))).click()

|

cruisepandey · Accepted Answer · 2019-06-12 07:08:50Z

0

VISHVAMBRUTHJAVAGALTHIMMEGOWDA,

I tried your code and was getting the same exception, First I thought that username was in Frame or iframe but it is not.

Then I tried to introduce webdriver wait and it worked just fine :

Code :

wait = WebDriverWait(driver,10)

driver.maximize_window()

driver.get("https://learn.letskodeit.com/")

print(driver.title)

driver.find_element_by_xpath("//a[contains(text(),'Login')]").click()

wait.until(EC.visibility_of_element_located((By.ID, 'user_email'))).send_keys("[email protected]")

#driver.find_element_by_id("user_email").

driver.find_element_by_id("user_password").send_keys("abcabc")

driver.find_element_by_name("commit").click()

print(driver.title)

Remember to imports these :

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

EDIT 1 :

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu')  # Last I checked this was necessary.
driver = webdriver.Chrome("C:/Users/XXXX/Downloads/BrowserDriver/chromedriver_win32/chromedriver.exe", chrome_options=options)
wait = WebDriverWait(driver,10)

driver.get("https://learn.letskodeit.com/")

print(driver.title)


wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Login'))).click()

wait.until(EC.visibility_of_element_located((By.ID, 'user_email'))).send_keys("[email protected]")

wait.until(EC.visibility_of_element_located((By.ID, 'user_password'))).send_keys("abcabc")

wait.until(EC.visibility_of_element_located((By.NAME, 'commit'))).click()

print(driver.title)

edited Jun 12, 2019 at 7:08

answered Jun 12, 2019 at 5:45

cruisepandey

29.5k6 gold badges23 silver badges43 bronze badges

4 Comments

VISHVAMBRUTH J T Over a year ago

I changed the code as you suggested. But it works just fine with the head full browser but not with head less browser. Please let me know how to make the code works with the headless chrome.

cruisepandey Over a year ago

with headless browser you can write the same code with waits. Let me give you code.

cruisepandey Over a year ago

Updated under edit1 section, please check . I haven't tested it tho.

VISHVAMBRUTH J T Over a year ago

Tried the code under edit1 section but still unable to run it using headless chrome. The code works fine with the head full mode. Please help.

Collectives™ on Stack Overflow

Pycharm: Selenium with Python: Unable to find web elements using headless chrome

4 Answers 4

Comments

Comments

7 Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

7 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related