0

You can skip the introduction (my steps after searching for the "main question" string in the browser)

I am new to Python and I have followed instructions online to install python3. I also used the following link to install the other tools needed for website scraping.

http://www.pyladies.com/blog/Get-Your-Mac-Ready-for-Python-Programming/

My first problem was How to install pip on python 3.6, not the default python 2.7.

pip3 install numpy
python3
>>> import numpy

I used the commands above in the terminal. Then I continued installing the tools mentioned in the first link. All good. I then installed selenium using the command:

pip3 install -U selenium

Then I tried to test whether I could run a simple py program in the terminal. I only had an issue with the file permissions so I used the following command to change that:

sudo chmod +x test.py

and all good :)

Then I tried installing a webdriver: https://chromedriver.storage.googleapis.com/index.html?path=2.32/

I tried running a py script using the command: python3 ./test-scrape.py that contained the following:

import time
from selenium import webdriver

driver = webdriver.Chrome('/Users/radomer/chromedriver')  # Optional argument, if not specified will search path.
driver.get('http://www.google.com/xhtml');
time.sleep(5) # Let the user actually see something!
search_box = driver.find_element_by_name('q')
search_box.send_keys('ChromeDriver')
search_box.submit()
time.sleep(5) # Let the user actually see something!
driver.quit()

main question

Now, I am trying to scrape some fields of a website.

I visit the login page, put the name and the password, login, paste a value in a specific box and then click enter. It seems that the ENTER part does not work.

import time
from selenium import webdriver

driver = webdriver.Chrome('/Users/radomer/chromedriver')  # Optional argument, if not specified will search path.
driver.get('######');
time.sleep(1) # Let the user actually see something!
inputElement = driver.find_element_by_id("email")
inputElement.send_keys("###@###.com")
time.sleep(1)
inputElement = driver.find_element_by_id("password")
inputElement.send_keys("#####")
time.sleep(1)
element = driver.find_element_by_xpath("//a[@id='loginSubmit']")
element.click()
time.sleep(5)
element = driver.find_element_by_xpath("//li[@class='#####']")
element.click()
time.sleep(5)
Key = driver.find_element_by_id("#####")
time.sleep(3)
Key.send_keys("155289")
time.sleep(3)

from selenium.webdriver.common.keys import Keys

visitPage = driver.find_element_by_class_name("nxm-td name campaign-title clickable")
time.sleep(1)
visitPage.click()

driver.quit()

The problems hinters in the last lines of my code. After clicking ENTER above, I am getting a list with only one value. Its HTML is given below:

<div class="nxm-td name lalala-title clickable" data-lalala-id="155289">
        <div class="text-ellipsis tooltip" data-rel="######">####</div>
    </div>

I tried changing my python script after getting the Xpath from the site:

visitcampaignPage = driver.find_element_by_xpath("//*[@id="campaigns-all"]/div[1]/div[1]/div[2]/div/div/div/div/div[3]")

or an alternative way to get it using the class name:

visitPage = driver.find_element_by_class_name("nxm-td.name.campaign-title clickable")

and it only worked with the below:

visitPage = driver.find_element_by_class_name("clickable")

but I might have multiple lines with this class, so it will get confused. How do I get it working with multiple class elements?

1 Answer 1

1

Based on uniqueness, you can use one of the compound class names as you have done in your last example but may be other class names which might be unique.

Another way is to use css_selector as shown below:

visitPage = driver.find_element_by_css_selector(".nxm-td.name.lalala-title.clickable")
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.