How to make python loop through array of urls and write data per row in csv?

Question

I have a set of urls (stock data) for which I want certain data to be put into a csv. Per row I need to have:

name price recrat opinion

A csv appears but has no data, and I get the error:

ValueError: too many values to unpack

How should I go about this? Here is my code so far:

# -*- coding: utf-8 -*-
import urllib2
from bs4 import BeautifulSoup
import csv
from datetime import datetime

quote_page = ['http://uk.mobile.reuters.com/business/quotes/overview/AALB.AS',
   'http://uk.mobile.reuters.com/business/stocks/overview/ABNd.AS',
   'http ://uk.mobile.reuters.com/business/stocks/overview/ACCG.AS', 
   'http ://uk.mobile.reuters.com/business/stocks/overview/AD.AS']


for link in quote_page:
    try:
        page = urllib2.urlopen(link)
        soup = BeautifulSoup(page, 'html.parser')

        name_box = soup.find('span', attrs={'class': 'company-name'})
        name = name_box.text.strip()
        print name

        price_box = soup.find('span', attrs={'class':'price'})
        price = price_box.text.strip()
        print price

        recrating_box = soup.find('div', attrs={'class':'recommendation-rating'})
        recrat = recrating_box.text.strip()
        print recrat

        opinion = soup.find('div', attrs={'class':'recommendation-marker'})['style']
        print opinion
    except TypeError:
        continue

quote_page.append((name, price, recrat, opinion))   
    # open a csv file with append, so old data will not be erased
with open('index.csv', 'a') as csv_file:
    writer = csv.writer(csv_file)
    for name, price in quote_page:
        writer.writerows([name, price, recrat, opinion, datetime.now()])

If quote_page is the list with all those links in it (as shown at the top of your code), what values are name and price supposed to take as you iterate through them? — khelwood
– khelwood, Commented Oct 1, 2017 at 8:35

Adders · Accepted Answer · 2017-10-01 09:10:47Z

1

Tested and working:

# -*- coding: utf-8 -*-
import urllib2
from bs4 import BeautifulSoup
import csv
from datetime import datetime

quote_page = ['http://uk.mobile.reuters.com/business/quotes/overview/AALB.AS',
   'http://uk.mobile.reuters.com/business/stocks/overview/ABNd.AS',
   'http://uk.mobile.reuters.com/business/stocks/overview/ACCG.AS', 
   'http://uk.mobile.reuters.com/business/stocks/overview/AD.AS']

results = []

for link in quote_page:
    try:
        page = urllib2.urlopen(link)
        soup = BeautifulSoup(page, 'html.parser')

        name_box = soup.find('span', attrs={'class': 'company-name'})
        name = name_box.text.strip()
        print name

        price_box = soup.find('span', attrs={'class':'price'})
        price = price_box.text.strip()
        print price

        recrating_box = soup.find('div', attrs={'class':'recommendation-rating'})
        recrat = recrating_box.text.strip()
        print recrat

        opinion = soup.find('div', attrs={'class':'recommendation-marker'})['style']
        print opinion
    except TypeError:
        continue

    results.append((name, price, recrat, opinion))   

# open a csv file with append, so old data will not be erased
with open('index.csv', 'w') as csv_file:
    writer = csv.writer(csv_file)
    for item in results:
        writer.writerow([item[0], item[1], item[2], item[3], datetime.now()])

There were 3 issues, first, you were overwriting an active list - Not a good idea: I renamed this to results.

Second, you were trying to iterate over the list but accessing only 2 of the 4 items. I've done these as indexed.

Finally, as you were iterating, you'd want to do it line by line so writerows needs to be changed to writerow.

edited Oct 1, 2017 at 9:10

answered Oct 1, 2017 at 8:59

Adders

6658 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Henk ter Kuile Over a year ago

This creates a csv, but puts every character of the first array item in a separate cell. and gives an error: Traceback (most recent call last): File "45.py", line 36, in <module> writer.writerows([name, price, recrat, opinion, datetime.now()]) _csv.Error: sequence expected

Adders Over a year ago

updated my answer and tested it this time - works like a charm :)

Adders Over a year ago

I'll add further detail now

Henk ter Kuile Over a year ago

I'm new to python but love it already. I've got plans to expand the script and try to make it compare the newly crawled data to the old data and add columns in csv that show % of change.... Will be quite a job... :)

Adders Over a year ago

Good luck - Enjoy Python - it's awesome :)

Collectives™ on Stack Overflow

How to make python loop through array of urls and write data per row in csv?

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related