1

Yes that sounds overly complicated.

I am trying to mine data from pages on our intranet. The pages are secure. The connection is refused when I try to get the contents with urllib.urlopen().

So I would like to use python to open a web browser to open the site then click some links that trigger javascript pop ups containing tables of info that I want to collect.

Any suggestions on where to begin?

I know the format of the page. It is something like this:

<div id="list">
    <ul id="list item">
        <li><a onclict="Openpopup('1');">blah</a></li>
    </ul>
    <ul></ul>
    etc

Then a hidden frame becomes visible and the fields in the table within are filled.

<div>
    <table>
       <tr><td><span id="info_i_want">...
Improve this question

4 Answers 4

Reset to default
5

First off, I suggest that it's better to figure out what the page needs that JS is providing, and fake that - you'll have an easier time scraping the page if a browser isn't involved.

If it's just Javascript making an XMLHttpRequest, you can find the page from which the Javascript fetches the iframe data and connect directly to that.

But in spite of that you may need a library that does Javascript execution (if the reverse-engineering is too hard or it uses challenge tokens). A web-rendering framework like Gecko or WebKit might be appropriate.

Take a good look at Selenium if you insist on using a true web browser or cannot get the programmatic methods to work.

Once you've gotten the page contents via whatever method, you need an HTML parser (such as sgmllib or [almost] xml.dom). I suggest a DOM library. Parse the DOM and extract the contents from the appropriate node in the resulting tree.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

Thanks. The is the smart approach. I will try to determine where the javascript is fetching the info then querry that source directly. If not I'll look into selenium.
I see the javascript function that makes the request. It sends a json object to a given url. I can build the json object, it is very simple, {'ID': #}, but I am not sure how to pass the json object in python. The javascript uses the jquery function .ajax and has fields for url, data, etc. Do you have any recommendations how I could send the json object and retrieve the results with python?
@sequoia Python since version 2.6 has a JSON library (module name json). Use json.dumps to get a string representing the object you build as a Python dictionary ({'ID': '#'}), then send it just like any other HTTP request (it's probably a POST).
the HTTP part is what I am vague on. Is the json string appended to the end of the url? I have only every used jquery and .ajax function to make these requests so I don't know how to send the information otherwise. If you would like to deffer me to a place were I could read up on these basics that would be helpful too.
@sequoia No, the JSON is the body of the request. I don't have a handy-dandy reference for you to look at, unfortunately. You just do what the JS is doing - probably a POST.
3

The connection is refused when I try to get the contents with urllib.urlopen(). probably means you have to make a post request using python urllib module.I would suggest you use urllib2.You may also need to handle cookies, referrer,user-agent from your python code.

To see all the post request fired from your browser use firefox's live-http-headers.

For the javascript part,

Your best bet is to run a headless browser e.g phantomjs which understands all the intricacies of JavaScript, DOM etc but you will have to write your code in Javascript, benefit is that you can do whatever you want.

As, @phihag mentioned selenium is also a good option.

Improve this answer

3 Comments

+1 for phantomjs. I recently wrote a tool with it to test my JSON REST api from a command line, while first performing django authentication
Thanks. great suggestions. I will look into urllib2. I would like to be able to achieve this task with python. I have limited knowledge of python, but even less of javascript. How would I know if I have to handle cookies?
for that you have to use live http header[LHH] addon I mentioned. When you will try to login into your website take a look at LHH's log. If there is any header named like PHPSESSID or cookie-set. than you have to handle cookie.
1

First of all, you should really find out why the connection is refused when you access the page with Python. Most likely, you'll have to perform HTTP authentication or specify a different User-Agent.

Firing up a browser, navigating, and getting the HTML back is a complex task. Luckily, you can implement it using selenium.

Improve this answer

1 Comment

Thanks for your suggestion. I will look into selenium. This is a daunting task and I am new to these programming endeavors, but the threat of repeated manual data mining is even more frightening.
1

Consider taking a look at splinter which is a simpler webdriver API than Selenium.

Improve this answer

4 Comments

This looks promising. Do you know if I can use splinter to access site that require username and password authentication?
Yes, you can either use the api to fill in the form elements and submit the page, or you can do that by hand and then automate the rest. It can be used interactively, as well as full-scripted.
I'm having some issues getting splinter working on windows, but I don't see anything in the documentation which suggests that this is not possible. When I attempt to go through the splinter turtorial, from splinter.browser import Browser fails because there is no module named lxml.cssselect is present. I installed splinter again and saw that there were errors. Certain files were missing 'lxml.etree.c' and some others. I think this due to not having the xmllib2-devel and xsltlib-devel libraries. I cannot find them for windows. Are you very familiar with splinter? have recommendations? thanks.
Sounds like you need to install the missing lxml module: pip install lxml. Unfortunately I don't use Windows, so I'm not sure what else you need.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.