You can skip the introduction (my steps after searching for the "main question" string in the browser)
I am new to Python and I have followed instructions online to install python3. I also used the following link to install the other tools needed for website scraping.
http://www.pyladies.com/blog/Get-Your-Mac-Ready-for-Python-Programming/
My first problem was How to install pip on python 3.6, not the default python 2.7.
pip3 install numpy
python3
>>> import numpy
I used the commands above in the terminal. Then I continued installing the tools mentioned in the first link. All good. I then installed selenium using the command:
pip3 install -U selenium
Then I tried to test whether I could run a simple py program in the terminal. I only had an issue with the file permissions so I used the following command to change that:
sudo chmod +x test.py
and all good :)
Then I tried installing a webdriver: https://chromedriver.storage.googleapis.com/index.html?path=2.32/
I tried running a py script using the command: python3 ./test-scrape.py that contained the following:
import time
from selenium import webdriver
driver = webdriver.Chrome('/Users/radomer/chromedriver') # Optional argument, if not specified will search path.
driver.get('http://www.google.com/xhtml');
time.sleep(5) # Let the user actually see something!
search_box = driver.find_element_by_name('q')
search_box.send_keys('ChromeDriver')
search_box.submit()
time.sleep(5) # Let the user actually see something!
driver.quit()
main question
Now, I am trying to scrape some fields of a website.
I visit the login page, put the name and the password, login, paste a value in a specific box and then click enter. It seems that the ENTER part does not work.
import time
from selenium import webdriver
driver = webdriver.Chrome('/Users/radomer/chromedriver') # Optional argument, if not specified will search path.
driver.get('######');
time.sleep(1) # Let the user actually see something!
inputElement = driver.find_element_by_id("email")
inputElement.send_keys("###@###.com")
time.sleep(1)
inputElement = driver.find_element_by_id("password")
inputElement.send_keys("#####")
time.sleep(1)
element = driver.find_element_by_xpath("//a[@id='loginSubmit']")
element.click()
time.sleep(5)
element = driver.find_element_by_xpath("//li[@class='#####']")
element.click()
time.sleep(5)
Key = driver.find_element_by_id("#####")
time.sleep(3)
Key.send_keys("155289")
time.sleep(3)
from selenium.webdriver.common.keys import Keys
visitPage = driver.find_element_by_class_name("nxm-td name campaign-title clickable")
time.sleep(1)
visitPage.click()
driver.quit()
The problems hinters in the last lines of my code. After clicking ENTER above, I am getting a list with only one value. Its HTML is given below:
<div class="nxm-td name lalala-title clickable" data-lalala-id="155289">
<div class="text-ellipsis tooltip" data-rel="######">####</div>
</div>
I tried changing my python script after getting the Xpath from the site:
visitcampaignPage = driver.find_element_by_xpath("//*[@id="campaigns-all"]/div[1]/div[1]/div[2]/div/div/div/div/div[3]")
or an alternative way to get it using the class name:
visitPage = driver.find_element_by_class_name("nxm-td.name.campaign-title clickable")
and it only worked with the below:
visitPage = driver.find_element_by_class_name("clickable")
but I might have multiple lines with this class, so it will get confused. How do I get it working with multiple class elements?