Get data from javascript to python using scrapy python

Question

How to get the data from javascript content using scrapy python ? The javascript look like this

<script type="text/javascript">
  var ad_reply_url = "http://www2.mudah.my/ar/send/0?ca=3_s&id=49825097&l=0";
  var mcvl = "";
  var images = [
     'http://img.rnudah.com/images/13/133608119523265.jpg', 
     'http://img.rnudah.com/images/13/135608116569903.jpg', 
     'http://img.rnudah.com/images/13/137608113616541.jpg', 
     'http://img.rnudah.com/images/13/139608119186498.jpg'
  ];
 var thumbnails = [
    'http://img.rnudah.com/thumbs/13/133608119523265.jpg',
    'http://img.rnudah.com/thumbs/13/135608116569903.jpg',
    'http://img.rnudah.com/thumbs/13/137608113616541.jpg',
    'http://img.rnudah.com/thumbs/13/139608119186498.jpg'
 ];</script>

So, what I want is. I want the data from var images and print that data like this

['http://img.rnudah.com/images/13/133608119523265.jpg','http://img.rnudah.com/images/13/135608116569903.jpg', 'http://img.rnudah.com/images/13/137608113616541.jpg','http://img.rnudah.com/images/13/139608119186498.jpg' ];

Can anyone help me ? thanks.

Community · Accepted Answer · 2017-05-23 10:32:41Z

1

I'm not using Scrapy Python, just regular Python. It is pretty straightforward though:

Code Sample:

import ast
import re

page_source = '''
<script type="text/javascript">
  var ad_reply_url = "http://www2.mudah.my/ar/send/0?ca=3_s&id=49825097&l=0";
  var mcvl = "";
  var images = [
     'http://img.rnudah.com/images/13/133608119523265.jpg',
     'http://img.rnudah.com/images/13/135608116569903.jpg',
     'http://img.rnudah.com/images/13/137608113616541.jpg',
     'http://img.rnudah.com/images/13/139608119186498.jpg'
  ];
 var thumbnails = [
    'http://img.rnudah.com/thumbs/13/133608119523265.jpg',
    'http://img.rnudah.com/thumbs/13/135608116569903.jpg',
    'http://img.rnudah.com/thumbs/13/137608113616541.jpg',
    'http://img.rnudah.com/thumbs/13/139608119186498.jpg'
 ];</script>
'''

variables = re.findall('(?si)var(.*?);', page_source)

var_collection = {}
for var in variables:
    var = var.strip()
    var_key = var.split(' = ')[0]
    var_value = ast.literal_eval(var.split(' = ')[1])
    var_collection.update({var_key: var_value})

print(var_collection['images'])

Output:

['http://img.rnudah.com/images/13/133608119523265.jpg', 'http://img.rnudah.com/images/13/135608116569903.jpg', 'http://img.rnudah.com/images/13/137608113616541.jpg', 'http://img.rnudah.com/images/13/139608119186498.jpg']

Related: https://stackoverflow.com/a/18108644/295246

edited May 23, 2017 at 10:32

CommunityBot

11 silver badge

answered Nov 8, 2016 at 17:29

HEADLESS_0NE

3,5564 gold badges36 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

shahril Over a year ago

okay .. thanks give me a hint .. just now I was trying to manipulate your code and now I got what I want ... thanks man ! :)

HEADLESS_0NE Over a year ago

@shahril Glad it helped. Feel free to upvote or accept this answer as your solution, to your discretion. Thanks!

HEADLESS_0NE Over a year ago

@Ale Pleasure is mine! Seems this question doesn't come up too often :)

Collectives™ on Stack Overflow

Get data from javascript to python using scrapy python

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related