3

One of the arguments I make to my (Microbiology and Genetics) students is that "data" is/are messy, and Python can help with that (of course other languages can too). So here is a practical kind of web-based data-gathering exercise.

I notice that there a few people who answer Python-related questions among the users with the highest rep. Among the questions which naturally arise:

I wish to recover the current rep and rate of rep increase for (top rated) Pythonistas on Stack Overflow, so as to predict whether or when Alex Martelli will overtake Steven Lott or Greg Hewgill? what about Konrad Rudolph? Is it trivial because the increases for these guys are all pegged to the limit?

More generally, in the absence of an API for querying (which I don't think there is) is there any alternative to looking through the URLs of pages for patterns, loading those pages with Python and then scraping the html? I realize there is probably not a general approach, but I'm interested in how people would approach this problem.

Edit: @fitzgeraldsteele: Generally. SO is really just a (contrived) example.

Improve this question
5
  • 1
    Not sure what the question is here...Are you looking for how to screen scrape in general, or for stackoverflow specifically? Commented Feb 2, 2010 at 3:30
  • Apparently, there /was/ an API for this exact task (stackoverflow.uservoice.com/pages/general/suggestions/…), but it seems to be broken now Commented Feb 2, 2010 at 3:33
  • Simply scraping and parsing with BeautifulSoup or regex would allow the whole project to be finished in a few hours. Commented Feb 2, 2010 at 3:34
  • @Max. S: Please post that as an answer so we can upvote it. Commented Feb 2, 2010 at 11:14
  • 1
    If you're asking about HTML scraping, this is an absolute duplicate. All of these: stackoverflow.com/search?q=%5Bpython%5D+%5Bhtml%5D+soup And no, Martelli may be thinner, smarter and better looking, but he'll never catch me. I have long-running unit tests that give me too much "free" time while refactoring. Commented Feb 2, 2010 at 11:21

1 Answer 1

Reset to default
3

There's a perfectly usable monthly "data dump" of Stack Overflow under Creative Commons license, see for example here (just the first one "under my thumb" of the many links about this -- at least one per month). For such analysis as my average weekly rep relative to some other poster's, such monthly dollops of data are much more usable than screen-scrapes.

If you do want to screen-scrape some (other;-) site, and that doesn't violate their policies or their robots.txt files, Python is one of several excellent choices -- start with scrapy, and you won't have all that much extra work to do, for example.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Thanks, Alex. Both scrapy and (even just the idea of) a monthly "data dump" are new to me.
They're both excellent ideas, especially the (Creative Commons license) data dump (there are reasons why SO became the leading programming Q&A site so rapidly -- great ideas behind it!-).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.