Python read http: html page as seen in browser - with javascripts results

Question

Can I get http page as seen in browser - fully formed after javascripts are run? No need to submit data and press buttons. IMHO it's standard task, where can I see example to run all scripts and get result?

Via:

u = urllib.request.urlopen('https://www.*')
data = u.read()

I get page as seen if I choose view source in browser. However, when I inspect elements on page, I see how code expands, e.g.:

<div class="js-events-container"></div>

expands to:

<div class="js-events-container">    <table class="zebra noBorderTbl" style="width: 100%;">
        <tbody><tr>
            <th>1</th>
            <th>2</th>
            <th>3</th>
        </tr>
...
        </tr>
            </tbody></table>
</div>

bertus wisman · Accepted Answer · 2016-05-23 19:18:49Z

1

i see js in the class name, it is probably javascript, i think there isn't a way to get the full page with urllib. you need to pull the site after the javascript starts. you will need selenium or phantomjs to do the job.

answered May 23, 2016 at 19:18

bertus wisman

511 silver badge4 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Alex Martian Over a year ago

Thank you. Isn't my task rather standard? Maybe there are examples how to do that - just load page without providing inputs (could not find now)?

Rajesh Yogeshwar · Accepted Answer · 2016-05-24 06:22:15Z

0

You can also give a look to this particular library dryscape. It is javascript aware as per documentation.

answered May 24, 2016 at 6:22

Rajesh Yogeshwar

2,1792 gold badges22 silver badges39 bronze badges

Comments

Alex Martian · Accepted Answer · 2016-07-19 17:30:03Z

0

I now load page using selenium, then get page_source. In spite of name, page_source give not page source, but page result after java scripts are run.

answered Jul 19, 2016 at 17:30

Alex Martian

3,9207 gold badges44 silver badges76 bronze badges

Collectives™ on Stack Overflow

Python read http: html page as seen in browser - with javascripts results

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related