Python to read Javascript Date

Question

I am using BeautifulSoup to get data from a webpage. The webpage provides a date, which I see when I open the page in Firefox. However, when I view page source there is no date, just some javascript that generates the date. I see there are some related questions on here, I see references to ajax and json, I am just an amaeteur programmer though and remain confused here. Here is some of the HTML code which has the javascript code in it with the date I need.

<div class="match-details">
  <p class="floatleft">
    BARCLAYS PREMIER LEAGUE 

    <span>
      <script type="text/javascript">
        (function(){
        var d = new Date(1345489200000);

        var year = d.getFullYear();
        var month = d.getMonth() + 1;
        var day = d.getDate();
        var minutes = d.getMinutes();
        var hours = d.getHours();                                        

        if (minutes < 10) { minutes = '0' + minutes; }
        var dmy = [day, month, year];
        var hm = [hours, minutes];
        if (SITE_EDITION == 'us/en') {
            var dmy = [month, day, year];    
        }
        var matches_local = dmy.join('/') + " " + hm.join(':'); 
        matches_local += "<span class='live-red'>*</span>";

        document.write(matches_local);
        })();                                                       
      </script>
    </span>

  </p>
</div>

Could you outdent the code a bit? There's no need for pushing it off the page... — Tim Pietzcker
– Tim Pietzcker, Commented Mar 1, 2013 at 20:28
@BurhanKhalid that code will output a date to the page when run in a browser. He wants to know how to get that programatically if he's screen-scraping with python — entropy
– entropy, Commented Mar 1, 2013 at 20:30
@TimPietzcker I edited it to fix indentation but we'd have to wait for people to review the edit and accept it — entropy
– entropy, Commented Mar 1, 2013 at 20:30
Are you just trying to find the string new Date(1345489200000); and turn that into a Python datetime object? Or are you trying to read the page rendered by this JavaScript and extract a date from the resulting HTML? — abarnert
– abarnert, Commented Mar 1, 2013 at 20:58

Community · Accepted Answer · 2017-05-23 11:57:40Z

2

BeautifulSoup is an HTML processing library. You need a HTML + Javascript processing library.

Read up on this Question : Programmatic Python Browser with JavaScript

As that QA states...you basically either need to use a real browser -- via Selenium -- or use a python browser that supports javascript -- like Spynner.

edited May 23, 2017 at 11:57

CommunityBot

11 silver badge

answered Mar 1, 2013 at 20:32

Jonathan Vanasco

15.8k11 gold badges53 silver badges74 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

appleLover Over a year ago

thanks for the response. I am looking at pyv8 unfortunately i am having a hard time getting it set up on ubuntu. the people maintaining the pyv8 site recommend to use the prebuilt version but there is no prebuilt version for linux. i am going to open a new thread specifically asking how javascript parses the the line above. i think that will be simpler.

Jonathan Vanasco Over a year ago

Sorry, I was not clear & edited my repsonse. You need a HTML + Javascript processing library. PyV8 will only let you run javascript. It won't parse the page & tell you which javascript to run. You need to have a javascript-supporting HTML browser to trigger the correct events and allow for the DOM to be manipulated.

Collectives™ on Stack Overflow

Python to read Javascript Date

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related