Script can't find Element but Console can [Selenium] [Python]

Question

I'm trying to write script to automate the process of downloading the instagram stories but I'm failing already when trying to log in.

I'm writing the code inside Pycharm. I just tried my usual approach to any problem. First, solve it with typing the commands out in the console and if it works writing the commands which worked inside the console down in a script. But here is the issue. The function which worked perfectly fine inside the python console fails inside the script.

I've noticed that my selenium was outdated but upgrading it didn't help ether. I also made a new project to test weather that made difference, which it didn't.

I've also tried skipping the first step inside the script and just opening the url to which I'm redirected. But the second commands failed as well.

When I create a new variable to store the output of the driver.find_element_by_link_text() in, it returns an empty list. This leads me to belive that somehow selenium is unable to search the contetns of the page.

I've also tried the same on Chrome and Safari. This also didn't work.

Here is the code:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Firefox()
driver.get("https://instagram.com/")
#next command fails
driver.find_element_by_link_text("Melde dich an.").click()

#if the first command is skipped by entering in the url 
#in driver.get(https://www.instagram.com/accounts/login/?source=auth_switcher)
#the following command fails as well.
driver.find_element_by_name("username").send_keys("HereIsTheUsername")
driver.find_element_by_name("password").send_keys("HereIsThePassword")
driver.find_element_by_name("password").send_keys(Keys.RETURN)
driver.close()

In the console these commands worked as mentioned, Here is what I've entered into the console:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("https://instagram.com/")
driver.find_element_by_link_text("Melde dich an.").click()
#if it failed here would be an error message
element = driver.find_element_by_name("username")

With the script the error message is this:

Traceback (most recent call last): File "/Users/alisot2000/PycharmProjects/Instagram downloader/venv/Main.py", line 6, in driver.find_element_by_link_text("Melde dich an.").click() File "/Users/alisot2000/PycharmProjects/Instagram downloader/venv/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 428, in find_element_by_link_text return self.find_element(by=By.LINK_TEXT, value=link_text) File "/Users/alisot2000/PycharmProjects/Instagram downloader/venv/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element 'value': value})['value'] File "/Users/alisot2000/PycharmProjects/Instagram downloader/venv/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute self.error_handler.check_response(response) File "/Users/alisot2000/PycharmProjects/Instagram downloader/venv/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: Melde dich an.

how are you sure that console found the element? did you use that to verify? — faruk13
– faruk13, Commented Jul 17, 2019 at 5:54
From the last error message it seems clear that the script didn't find the element. It generally happens in sites which aren't straight forward to use selenium to. — faruk13
– faruk13, Commented Jul 17, 2019 at 5:58
Looks like a synchronization problem to me. Add an explicit wait before you try to find the username element. — GPT14
– GPT14, Commented Jul 17, 2019 at 6:15

sudoxx · Accepted Answer · 2019-07-17 09:36:37Z

Issues you might be experiencing:

1. Synchronization Issue

For most automation tasks, there will be different loading times of web pages based on the processing power of the machine and how strong your internet connection is.

To solve this there are library import Waits from selenium that we can use. Here is a sample below:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
try:
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "myDynamicElement"))
    )
finally:
    driver.quit()

2. Wrong language set in selenium profile

Selenium will use your locale in most cases when running automation scripts but in the case that you might want another language here is a sample code for FireFox.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary

profile = webdriver.FirefoxProfile()
# switch out 'de' with another two character language code
profile.set_preference("intl.accept_languages",'de')
driver = webdriver.Firefox(firefox_profile=profile, executable_path='<insert_your_gecko_driver_path_here>')

driver.get("https://instagram.com/")
driver.close()

3. Working Code(Tested on Mojave 10.14.5)

Here is a diff of your code and the altered code: https://www.diffchecker.com/G0WWB4Ry

setup a virtualenv
pip install selenium
download geckodriver
set path to gecko driver in code
run script with success result

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
# these two imports are for setting up firefox driver and options 
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
# import these three lines below if you are having synchronization issues
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

profile = webdriver.FirefoxProfile()
# here is where you need to set your language explicitly if its defaulting to an undesired language
# just replace the second parameter with your 2 character language code
# this line is not needed if your desired language is locale
profile.set_preference("intl.accept_languages",'de')
# throw in your path here <insert_your_gecko_driver_path_here>
driver = webdriver.Firefox(firefox_profile=profile, executable_path='<insert_your_gecko_driver_path_here>')

driver.get("https://instagram.com/")

# added these two lines below to solve synchronization issue 
# element wasnt clickable until page finished loading
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.LINK_TEXT, "Melde dich an.")))
#next command fails 
driver.find_element_by_link_text("Melde dich an.").click()

#if the first command is skipped by entering in the url 
#in driver.get(https://www.instagram.com/accounts/login/?source=auth_switcher)
#the following command fails as well.
driver.find_element_by_name("username").send_keys("HereIsTheUsername")
driver.find_element_by_name("password").send_keys("HereIsThePassword")
driver.find_element_by_name("password").send_keys(Keys.RETURN)
driver.close()

Thank you very much for this detailed response. The synchronization was indeed the issue. Waiting one second solved the problem.

HaniPham · Accepted Answer · 2019-07-17 06:12:13Z

0

def ClickElementByName(name,driver): 
    while True:
        try:
            driver.find_element_by_name(name).click()
            break
        except:
            sleep(1)
            pass
 Too long to wait the website run.
 Replace ClickElementByName("username", driver)

answered Jul 17, 2019 at 6:12

HaniPham

133 bronze badges

Comments

Serjojo Serjojo · Accepted Answer · 2019-07-19 04:59:13Z

0

driver.find_element_by_xpath('//input[@name="username"]').send_keys("HereIsTheUsername") driver.find_element_by_xpath('//input[@name="password"]').send_keys("HereIsTheUsername") driver.find_element_by_xpath('//div[text()="Log In"]').click()

answered Jul 19, 2019 at 4:59

Serjojo Serjojo

11 silver badge3 bronze badges

Collectives™ on Stack Overflow

Script can't find Element but Console can [Selenium] [Python]

3 Answers 3

Issues you might be experiencing:

1. Synchronization Issue

2. Wrong language set in selenium profile

3. Working Code(Tested on Mojave 10.14.5)

1 Comment

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Issues you might be experiencing:

1. Synchronization Issue

2. Wrong language set in selenium profile

3. Working Code(Tested on Mojave 10.14.5)

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related