0

I have been searching similar questions but none seem to answer what my problem is. I have also tried searching google and can't find any answers. I am pretty new to programming so please let me know if my code is not the best way to do this.

I am using Selenium to pull some data from a website and I want to put it all in a CSV file. I have it working except my code only puts the last item into the csv file. I think it is putting all the results on cell A1 and just overwriting it every time it gets a new result. I attached my code below but I might be going about it completely wrong.

The first part of the code is just defining variables and running selenium. The find_by_xpath() function is where I am running into trouble. When I run the print(sel_test) line everything seems fine and it prints out all of the results. However when I try to write sel_test to a csv it only gives the last result. Any ideas?

Thanks!

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from datetime import datetime
import csv, time, os, datetime, re

# Current date time in local system

today = datetime.date.today()
day = today.strftime("%d")
month = today.strftime("%m")
year = today.strftime("%y")
year_for_scrape = today.strftime("20" + "%y")
yesterday = today - datetime.timedelta(days=1)
yesterday_day_global = yesterday.strftime("%d")
print("month number is", month)
print("year is", year_for_scrape)

# Call chromedriver and navigate to a page
driver = webdriver.Chrome(chromedriver_location)
driver.get('http://pipeline.wyo.gov/ApdCompCompMenu.cfm')

# Define the x paths from using inspect element in chrome and copy x path
beginning_month = '//*[@id="CalendarCompletiom"]/table/tbody/tr[3]/td[1]/input[1]'
beginning_day = '//*[@id="CalendarCompletiom"]/table/tbody/tr[3]/td[1]/input[2]'
beginning_yr = '//*[@id="CalendarCompletiom"]/table/tbody/tr[3]/td[1]/input[3]'
end_month = '//*[@id="CalendarCompletiom"]/table/tbody/tr[3]/td[3]/input[1]'
end_day ='//*[@id="CalendarCompletiom"]/table/tbody/tr[3]/td[3]/input[2]'
end_yr = '//*[@id="CalendarCompletiom"]/table/tbody/tr[3]/td[3]/input[3]'
go_find = '//*[@id="SubmitClassDate"]'

# Use selenium webdriver
driver.find_element_by_xpath(beginning_month).send_keys("01")
driver.find_element_by_xpath(beginning_day).send_keys("01")
driver.find_element_by_xpath(beginning_yr).send_keys(year_for_scrape)
driver.find_element_by_xpath(end_month).send_keys(month)
driver.find_element_by_xpath(end_day).send_keys(day)
driver.find_element_by_xpath(end_yr).send_keys("2020")
driver.find_element_by_xpath(go_find).click()
print("Done")


#Number of records wanted
num_rec = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]

def find_by_xpath():
    #csv
    csv_file = open('Permit_scrape.csv','w', newline='')
    csv_writer=csv.writer(csv_file)
    csv_writer.writerow(['API'])

    
    for i in range(len(num_rec)):
        sel_test = driver.find_element_by_xpath('/html/body/table[2]/tbody/tr[' +str(i+6) + ']/td[2]').text
        print(sel_test)
            
    #Write to CSV
    csv_writer.writerow([sel_test])
    csv_file.close()
    print('excel_done')

find_by_xpath()

2 Answers 2

2

I believe your issue is stemming from csv_writer.writerow([sel_test]) not being inside the for loop

Try this:

def find_by_xpath():
    #csv
    csv_file = open('Permit_scrape.csv','w', newline='')
    csv_writer=csv.writer(csv_file)
    csv_writer.writerow(['API'])

    
    for i in range(len(num_rec)):
        sel_test = driver.find_element_by_xpath('/html/body/table[2]/tbody/tr[' +str(i+6) + ']/td[2]').text
        #Write to CSV
        csv_writer.writerow([sel_test])
    csv_file.close()
    print('excel_done')
Sign up to request clarification or add additional context in comments.

2 Comments

Wow thank you! I should have realized that. I appreciate the help!
No problem, happens all the time!
0

In order to append a new line to the existing file, open the file in append mode, by using either 'a' or 'a+' as the access mode.

def find_by_xpath(): csv_file = open('Permit_scrape.csv','a', newline='')

In this case, you should consider that new empty line may be added when new line is add

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.