Python requests url link which contains a variable part

Question

I want to request a list of urls which have the following formats for example:

http://en.wikipedia.org/wiki/Monty_Python/001/

http://en.wikipedia.org/wiki/Monty_Python/002/

....

These codes 001,... I saved in a csv file.

import csv
import requests

inputfile = open('trycode.csv','r')
for row in inputfile:
    url = 'http://en.wikipedia.org/wiki/Monty_Python/'
    source = requests.get(url + row/)

And obviously it doesn't work... How can I correctly call these urls with the last part coming from the csv file?

Many thanks!

Apart from the extra '/' at the end, which is a syntax error, why does this "obviously" not work? — Daniel Roseman
– Daniel Roseman, Commented Jul 30, 2015 at 15:39
Maybe you mean url + row + '/' ? Or '%s%s/' % (url, row)? — larsks
– larsks, Commented Jul 30, 2015 at 15:40
@DanielRoseman The problem is the url in the end contains a / , so I don't know how to put it. without the / , the url add is not correct. — ximu
– ximu, Commented Jul 30, 2015 at 16:02
@larsks yes, you are right, I tried to put instead now url + row +'/', but when I print the url link, en.wikipedia.org/wiki/Monty_Python/00%A2, not en.wikipedia.org/wiki/Monty_Python/002, — ximu
– ximu, Commented Jul 30, 2015 at 16:07

CLaFarge · Accepted Answer · 2015-07-30 17:18:20Z

1

Are you actually using a Comma Separated Value format for your file? Seems if just a short string, I'd use a txt file, but that's just me.

It's always a good idea to build your string URL before trying to call it... this way you can inspect it first... also important to close your file:

import csv
import requests

inputfile = open('trycode.csv','r')
for row in inputfile:
    url = 'http://en.wikipedia.org/wiki/Monty_Python/' + row
    source = requests.get(url)
inputfile .close()

Hope this helps.

answered Jul 30, 2015 at 17:18

CLaFarge

1,38511 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

ximu Over a year ago

Thanks! But when I tried your code, for example using a row entry "01020" I always get en.wikipedia.org/wiki/Monty_Python/01020%0A, ideally I want en.wikipedia.org/wiki/Monty_Python/010200, maybe something wrong with my file format?

CLaFarge Over a year ago

Looks like &0A is a URL-encoded newline character... check your source file to ensure that you have clean data, or Replace the %0A before using the value from each line.

ximu Over a year ago

Thanks! I will take a look at the file to see if its a clean txt file.

Collectives™ on Stack Overflow

Python requests url link which contains a variable part

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related