Python > Selenium + CSV: How to scrape from a website and write 2 columns onto CSV?

Question

In need of some troubleshooting for some code that does the following:

1) Scrape links from a webpage
2) Scrape text for the links, from the same page

Had some success in extracting links and writing as a single column:

elements = driver.find_elements_by_xpath("//a[@href]")
with open('csvfile01.csv', "w", newline='') as output:
    writer = csv.writer(output)
    for element in elements:
        writer.writerow([element.get_attribute("href")])

Unfortunately, was stuck when it came to:
1) getting the "text" for the links, and
2) exporting it as a separate column...
3) scraping a specific part of the webpage for links, e.g. in a table ("td") or a div section

The code as it stands now:

from selenium import webdriver
import time
import csv

driver = webdriver.Chrome()


driver.get("https://en.wikipedia.org/wiki/Main_Page")
time.sleep(5)

columns = ['text', 'link']

e1 = driver.find_element_by_css_selector("a")
e2 = driver.find_elements_by_xpath("//a[@href]")
elements = zip(e1,e2)


time.sleep(5)

with open('csvfile01.csv', "w", newline='') as output:

    writer = csv.writer(output)

    for element in elements:
        writer.writerow(columns)
        writer.writerows(elements)

driver.quit()

Any suggestions would be much appreciated. Thanks!

Noah · Accepted Answer · 2019-12-28 16:32:29Z

1

As far as the getting the text goes , you can do .text , also your css selector dosent seem right considering it is only “a”, to get an xpath/css selector just inspect the element and right click it then click copy then you get a list of things to copy, I do not use selenium much but when I did use it I noticed in the xpath that only 1 number would change (like if it’s a table of proxies) so I just defined a counter and incremented it in a loop

edited Dec 28, 2019 at 16:32

answered Dec 28, 2019 at 16:27

Noah

17414 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python > Selenium + CSV: How to scrape from a website and write 2 columns onto CSV?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related