HTML data grabbing in python?

Question

I'm fairly new to programming and I am trying to take data from a webpage and use it in my python code. Basically, I'm trying to take the price of an item for a game by having python grab the data whenever I run my code, if that makes sense. Here's what I'm struggling with in particular:

The HTML page I'm using is for runescape, namely

http://services.runescape.com/m=itemdb_oldschool/api/catalogue/detail.json?item=4151

This page provides me with a bunch of dictionaries from which I am trying to extract the price of the item in question. All I really want to do it get all of this data into python so I can then manipulate it. My current code is:

import urllib2

response =urllib2.urlopen('http://services.runescape.com/m=itemdb_oldschool/api/catalogue/detail.json?item=4151')

print response

And it outputs:

addinfourl at 49631760 whose fp = socket._fileobject object at 0x02F4B2F0

whereas I just want it to display exactly what is on the URL in question.

Any ideas? I'm sorry if my formatting is terrible. And if it sounds like I have no idea what I'm talking about, it's because I don't.

You should look at a web scraping library like beautiful soup. — Paul Rooney
– Paul Rooney, Commented Jun 15, 2016 at 4:58

lightalchemist · Accepted Answer · 2016-06-15 04:54:28Z

2

If the webpage returns a json-encoded data, then do something like this:

import urllib2
import json

response = urllib2.urlopen("http://services.runescape.com/m=itemdb_oldschool/api/catalogue/detail.json?item=4151")
data = json.load(response)   

print(data)

Extract the relevant keys in the data variable to get the values you want.

answered Jun 15, 2016 at 4:54

lightalchemist

10.2k4 gold badges49 silver badges55 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

lightalchemist Over a year ago

@bethu Remember to do your error checking. For e.g., check for potential exceptions, and check that returned values are valid before accessing the content of the response object. Also, using the requests library might make things easier for you.

Collectives™ on Stack Overflow

HTML data grabbing in python?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related