Using Python mechanize on specific form with javascript

Question

I'm trying to use the Python mechanize module to retrieve data through this form: http://archive.stsci.edu/kepler/data_search/search.php?form=fuf

The thing I'm having trouble with is "Output Columns" area in the lower left, which uses javascript to specify the output format. I'd like to be able to select a specific list of items, which would normally be selected with the surrounding javascript buttons.

I'm new to mechanize and haven't looked at javascript in ages. I've been looking through the responsible javascript but am unsure of what to do. Any tips?

More specifically: Is there a way to use mechanize to modify the elements in a list?

ChrisP · Accepted Answer · 2012-03-18 01:38:50Z

2

I recommend making the POST call directly to that form, rather than interacting with the form somehow through mechanize. Here is how I would do this with mechanize:

import mechanize
from urllib import urlencode
opener = mechanize.build_opener()
data = {"resolver":"NED","radius":"0.02","equinox":"J2000","ktc_target_type[]":"LC","ktc_target_type[]":"SC","extra_column_name_1":"ktc_kepler_id","extra_column_value_1":"","extra_column_name_2":"ktc_kepler_id","extra_column_value_2":"","extra_column_name_3":"ktc_kepler_id","extra_column_value_3":"","extra_column_name_4":"ktc_kepler_id","extra_column_value_4":"","selectedColumnsCsv":"Mark,ktc_kepler_id,ktc_investigation_id,sci_data_set_name,ktc_target_type","selectedColumnsList[]":"ktc_target_type","availableColumns":"Mark","ordercolumn1":"ang_sep","ordercolumn2":"ktc_kepler_id","ordercolumn3":"","coordformat":"sex","outputformat":"HTML_Table","max_records":"1001","max_rpp":"100","action":"Search"}
d = urlencode(data)
url = 'http://archive.stsci.edu/kepler/data_search/search.php'
stuff = opener.open(url,d)
info_i_want = stuff.read()

Basically, put the contents of your search in a dictionary, make a POST call to the server, and read your response.

answered Mar 18, 2012 at 1:38

ChrisP

5,9721 gold badge36 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

StrangeQuirk Over a year ago

Thanks for the answer. I'd been thinking of doing it through POST, but hadn't had any knowledge of that so had been reluctant to try; your code works great, though. What about file upload, though? How do I do that through urlencode?

StrangeQuirk Over a year ago

Also, I've figured out that with mechanize I can simply edit the contents of the hidden selectedColumnsCsv element to choose the output columns; I had assumed that it would be wiped clean by the javascript preprocess() function. In this case, is there any advantage to using POST over mechanize?

Collectives™ on Stack Overflow

Using Python mechanize on specific form with javascript

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related