Appending data in csv using Python

Question

Why is the data on my existing csv will not stack up but instead it will rewrite the data of the previous page.

for x in range(1,5):
    r = 'https://www.nda-toys.com/29/games-wholesale?page='
    url = r+ str(x)
    driver.get(url)
    
    elements = driver.find_elements_by_class_name('catConttop')
    links = [] 
    datalist =[]

    for ele in elements:
        links += (ele.find_element_by_tag_name('a').get_attribute('href'),) 
      
        
        for link in links:
            driver.get(link)
        
            name = driver.find_element_by_xpath('/html/body/div[1]/div[2]/div[1]/span[2]/h3/strong').text
            code = driver.find_element_by_xpath('/html/body/div[1]/div[2]/div[1]/span[2]/table/tbody/tr[2]/td[2]').text
            price = driver.find_element_by_xpath('/html/body/div[1]/div[2]/div[1]/span[3]/span[1]').text
            avail = driver.find_element_by_xpath('/html/body/div[1]/div[2]/div[1]/span[3]/span[3]/strong').text
        
            df_text =[name,code,price,avail,link] 
            datalist.append((name,code,price,avail,link))
            ColName=['carName','Loc','price','avail','link']
        
        df = pd.DataFrame(datalist,columns=['name','code','price','avail','link'])
        df.to_csv(r'C:\Users\Ryzen 5\Desktop\work\hannah.csv',index=True,encoding='utf-8')

MisterO · Accepted Answer · 2021-02-11 15:38:27Z

1

You will need to create your main dataframe outside of your first for loop and append to that at the end of each loop. Finally you will export that main df to csv at the end outside the loop. Otherwise you are rewriting it and exporting it with each page loop.

# make main df
# for loop on each page:
    # logic to format each page
    # append to main df
# export csv

answered Feb 11, 2021 at 15:38

MisterO

1396 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Appending data in csv using Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related