How to create a pandas data frame from web scraping using python

Question

I'm trying to create a pandas data frame using data that I'm collecting from the web.So every time there is a login ,the data should be stored in a data frame that i will save as an excel file. I have a code that creates the data frame and writes the data, but it does not append the data, so whenever there's a new entry it overwrites the existing information that was in the excel file.

Python Code:

count = 0
data = []
df = pd.DataFrame(columns=['Date','Country','Name','Age'])
df.loc[count, 'Date'] = datetime.datetime.now().strftime('%d %B %Y')
df.loc[count, 'Country'] = driver.find_element_by_name('country').get_attribute('value')
df.loc[count, 'Name'] = driver.find_element_by_name('customername').get_attribute('value')
df.loc[count, 'Age'] = driver.find_element_by_name('acli.age').get_attribute('value')
df.to_excel("C:\Web Data\MIS REPORT.xlsx")

Instead of the code overwriting the existing rows in the excel file it should append it.

user12758604 · Accepted Answer · 2020-03-10 14:01:03Z

1

You did not increment your count every time there is a new entry, hence it's always overwriting the first entry of the dataframe!

Also, with each entry, the following lines:

count = 0
data = []
df = pd.DataFrame(columns=['Date','Country','Name','Age'])

should not be executed again so as to not recreate the dataframe/reset the count back to 0.

Hope that helps! :)

answered Mar 10, 2020 at 14:01

user12758604

3751 silver badge9 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to create a pandas data frame from web scraping using python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related