How to scrape text from span with ng-if attribute using python?

Question

I am trying to scrape the data from paytm web page using python BeautifulSoup https://paytm.com/shop/p/masha-mauve-satin-nighty-WNIGHW000NT14_45BBPPFR?src=search-grid&tracker=autosuggest%7Cundefined%7Cmasha%20nighty%7Cgrid%7CSearch%7C1. I am able to scrap all those fields which are using direct class names but some fields are using angularjs attributes and I don't have any idea how to do that.

I know how to scrape data of a span with defined class:-

mrp = link_soup.find_all("span" , class_="price")[0].string.strip()

But don't know how to do the same for the mentioned code.

Code from which I want to scrape data:-

 <span ng-if="!product.product.isOnlyCarCategory">Buy for Rs 329</span>

I want to scrape number 329 from this span.

Whole Code:-

<div itemprop="offers" itemscope="itemscope" itemtype="https://schema.org/Offer" class="buy-bar">
              <button class="md-raised fl md-button md-default-theme" ng-transclude="" type="button" ng-show="!product.product.notSellable &amp;&amp; !product.product.preorder" ng-click="product.buyProduct();_gaq.push(['_addTrans','3523347','Bboy Body Wears', '329','', '0', '', '', 'India'])" ng-class="{'dealBtn' : product.product.vertical_id == 5 ,'cartupdating' : product.isCartUpdating }" ng-disabled="product.isCartUpdating || !product.product.instock" tabindex="0" aria-hidden="false" aria-disabled="false"><span ng-show="!product.product.add_qty_using_price &amp;&amp; product.product.vertical_id !=5" aria-hidden="false">
 <span ng-if="!product.product.isOnlyCarCategory">Buy for Rs 329</span>
                  <meta itemprop="price" content="">
                  <div>
<span ng-show="product.product.actual_price != product.product.offer_price" class="strike" aria-hidden="false">Rs. 749</span><span ng-show="product.product.actual_price != product.product.offer_price" aria-hidden="false" class=""> | 56%  Discount</span></div>
                  <meta itemprop="pricecurrency" content="INR"></span><span ng-show="!product.product.add_qty_using_price &amp;&amp; product.product.vertical_id ==5" aria-hidden="true" class="ng-hide"><span> Buy for Rs.329</span>
                  <meta itemprop="price" content="">
                  <div><span ng-show="product.product.attributes.deal_mrp" class="strike ng-hide" aria-hidden="true">Rs. </span><span ng-show="product.product.attributes.discounted_MRP" aria-hidden="true" class="ng-hide"> | Pay at store ~ Rs. </span><span ng-show="product.product.attributes.discount_perc" aria-hidden="true" class="ng-hide"> | Saving upto  %</span></div>
                  <meta itemprop="pricecurrency" content="INR"></span><span ng-show="product.add_qty_using_price" aria-hidden="true" class="ng-hide"><span class="text"></span></span></button>
              <button class="md-raised fl md-button md-default-theme ng-hide" ng-transclude="" type="button" ng-show="!product.product.notSellable &amp;&amp; product.product.preorder" ng-click="product.buyProduct();_gaq.push(['_addTrans','3523347','Bboy Body Wears', '329','', '0', '', '', 'India'])" ng-class="product.isCartUpdating ? 'cartupdating': ''" ng-disabled="product.isCartUpdating || !product.product.instock ? &quot;disabled&quot;: &quot;&quot;" tabindex="0" aria-hidden="true" aria-disabled=""><span>Preorder Now For Rs. 329
                  <meta itemprop="price" content="329">
                  <div><span ng-show="product.product.actual_price != product.product.offer_price" class="strike" aria-hidden="false">Rs. 749</span><span ng-show="product.product.actual_price != product.product.offer_price" aria-hidden="false" class=""> | 56%  Discount</span></div>
                  <meta itemprop="pricecurrency" content="INR"></span></button>
              <div ng-show="product.product.notSellable" class="md-button md-raised fr ng-hide" aria-hidden="true"><a ng-href="" target="_blank"><span class="text"></span></a></div>

            </div>

Is there any solution ?

try to look at the page without js enabled, you will see what you can parse — Whitefret
– Whitefret, Commented May 9, 2016 at 7:54
@Whitefret I have done that part but now I have to scrape this page only — User0706
– User0706, Commented May 9, 2016 at 8:06
I don't think you understood, without js, you can't get to the page so you can't scrap it. you need something like selenium to act like a browser, then you will be able to parse your page — Whitefret
– Whitefret, Commented May 9, 2016 at 8:15
@Whitefret okay. So anyhow I've to use selenium or something for parsing that page? — User0706
– User0706, Commented May 9, 2016 at 8:18
yeah you have to. The javascript in this page will only show when it is interpreted by something. Python can't do it alone. Selenium is a librairy that acts like a browser, so it will handle the js part for you and you will be able to scrap the page like a plain html page — Whitefret
– Whitefret, Commented May 9, 2016 at 8:20

Community · Accepted Answer · 2017-05-23 10:28:08Z

0

Use this link to get a better overview.
To answer your question more precisely, Paytm gets data in js files about the product. Following link gives data about any product listed at Paytm:
https://catalog.paytm.com/v1/p/product-url&callback=angular.callbacks._0&channel=web&version=2
The link given by you:
https://paytm.com/shop/p/masha-mauve-satin-nighty-WNIGHW000NT14_45BBPPFR?src=search-grid&tracker=autosuggest%7Cundefined%7Cmasha%20nighty%7Cgrid%7CSearch%7C1.
product-url :
masha-mauve-satin-nighty-WNIGHW000NT14_45BBPPFR?src=search-grid&tracker=autosuggest%7Cundefined%7Cmasha%20nighty%7Cgrid%7CSearch%7C1
Hope it helps you.

edited May 23, 2017 at 10:28

CommunityBot

11 silver badge

answered May 9, 2016 at 11:02

sbk23

131 silver badge8 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to scrape text from span with ng-if attribute using python?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related