6

I am looking to extract email, phone and name value from the below code in SCRIPT tag(not in Body) using Beautiful soup(Python). I see Beautiful soup can be used for extracting.

I tried getting page using the following code -

fileDetails = BeautifulSoup(urllib2.urlopen('http://www.example.com').read())
results = fileDetails.find(email:")

This Ajax request code is not repeating in the page again. Can we also write try and catch so that if it doesn't found it in the page, it won't throw any error.

<script type="text/javascript" language='javascript'> 
$(document).ready( function (){
   
   $('#message').click(function(){
       alert();
   });

    $('#addmessage').click(function(){
        $.ajax({ 
            type: "POST",
            url: 'http://www.example.com',
            data: { 
                email: '[email protected]', 
                phone: '9999999999', 
                name: 'XYZ'
            }
        });
    });
});

Once I get this, I also want to store in an excel file.

Thanks in anticipation.

Improve this question
4
  • Either provide an actual url, or the relevant script tag contents (better url, if possible). Commented Aug 4, 2014 at 4:28
  • This is the example. I think, it's easy to find the text from HTML file. Commented Aug 4, 2014 at 4:34
  • I'm sorry for bothering, but this is not clear. How can we help if we cannot see where exactly the desired data is inside the HTML? Please be more specific. The better you ask, the more chances of a good and fast answer you have at the topic. Commented Aug 4, 2014 at 4:35
  • I am sorry for that. It's in ajax script. Script is posted in the question. I am looking to extract '[email protected]','9999999999' and 'XYZ'. Commented Aug 4, 2014 at 4:38

2 Answers 2

Reset to default
8

Alternatively to the regex-based approach, you can parse the javascript code using slimit module, that builds an Abstract Syntax Tree and gives you a way of getting all assignments and putting them into the dictionary:

from bs4 import BeautifulSoup
from slimit import ast
from slimit.parser import Parser
from slimit.visitors import nodevisitor


data = """
<html>
    <head>
        <title>My Sample Page</title>
        <script>
        $.ajax({
            type: "POST",
            url: 'http://www.example.com',
            data: {
                email: '[email protected]',
                phone: '9999999999',
                name: 'XYZ'
            }
        });
        </script>
    </head>
    <body>
        <h1>What a wonderful world</h1>
    </body>
</html>
"""

# get the script tag contents from the html
soup = BeautifulSoup(data)
script = soup.find('script')

# parse js
parser = Parser()
tree = parser.parse(script.text)
fields = {getattr(node.left, 'value', ''): getattr(node.right, 'value', '')
          for node in nodevisitor.visit(tree)
          if isinstance(node, ast.Assign)}

print fields

Prints:

{u'name': u"'XYZ'", u'url': u"'http://www.example.com'", u'type': u'"POST"', u'phone': u"'9999999999'", u'data': '', u'email': u"'[email protected]'"}

Among other fields, there are email, name and phone that you are interested in.

Hope that helps.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

2

You can get the script tag contents via BeautifulSoup and then apply a regex to get the desired data.

Working example (based on what you've described in the question):

import re
from bs4 import BeautifulSoup

data = """
<html>
    <head>
        <title>My Sample Page</title>
        <script>
        $.ajax({
            type: "POST",
            url: 'http://www.example.com',
            data: {
                email: '[email protected]',
                phone: '9999999999',
                name: 'XYZ'
            }
        });
        </script>
    </head>
    <body>
        <h1>What a wonderful world</h1>
    </body>
</html>
"""

soup = BeautifulSoup(data)
script = soup.find('script')

pattern = re.compile("(\w+): '(.*?)'")
fields = dict(re.findall(pattern, script.text))
print fields['email'], fields['phone'], fields['name']

Prints:

[email protected] 9999999999 XYZ

I don't really like the solution, since that regex approach is really fragile. All sorts of things can happen that would break it. I still think there is a better solution and we are missing a bigger picture here. Providing a link to that specific site would help a lot, but it is what it is.

UPD (fixing the code OP provided):

soup = BeautifulSoup(data, 'html.parser')
script = soup.html.find_next_sibling('script', text=re.compile(r"\$\(document\)\.ready"))

pattern = re.compile("(\w+): '(.*?)'")
fields = dict(re.findall(pattern, script.text))
print fields['email'], fields['phone'], fields['name']

prints:

[email protected] 9999999999 Shamita Shetty
Improve this answer

10 Comments

The code is working like a charm. I got a syntax error. This small x is creating a problem in the following code: <button type="button" class="close" data-dismiss="modal" aria-hidden="true">×</button> It says, SyntaxError: Non-ASCII character '\xc3' in file import.py on line 17, but no encoding declared; see python.org/peps/pep-0263.html for details
Added this # -- coding: utf-8 --import os and it's working fine.
hey @alecxe, I have updated the script. I am getting an error now. It is because <script type="text/javascript" language='javascript'> ?
@Chopra what error? I've pasted that script code you've provided into my example and it works as before.
I just added <script type="text/javascript" language='javascript'> instead of <script>. It says, KeyError: 'name. I think, it's not able to find script tag now.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.