fetch text from web with Angular JS tags such as ng-view

Question

I'm trying to fetch all the visible text from a website, I'm using python-scrapy for this work. However what i observe scrapy only works with HTML tags such as div,body,head etc. and not with angular js tags such as ng-view, if there is any element within ng-view tags and when I do a right-click on the page and do view source then the content inside the tag doesn't appear and it displays like <ng-view> </ng-view>, So how can I use python to scrap the elements within this ng-view tags.Thanks in advance..

The (manual) approach to find JS calls to server (e.g. using fiddler) and then call them from your web scraper described in stackoverflow.com/questions/35050746/… — Michael Freidgeim
– Michael Freidgeim, Commented Jul 28, 2018 at 14:00

Ales Maticic · Accepted Answer · 2015-06-05 18:45:26Z

1

To answer your question

how can I use python to scrap the elements within this ng-view tags

You can't.

The content you want to scrape renders on the client side(browser), what scrapy get's you is just static content from server, your browser than interprets the HTML code and renders the JS code. And JS code than fetches different content from server again and makes some stuff with it.

Can it be done?

Yes!

One of the ways is to use some sort oh headless browser like http://phantomjs.org/ to fetch all the content. Once you have the content you can save it and scrape it as you wish. The thing is that this kind of web scraping is not as easy and straight forward as just scraping regular HTML. There is a reason why Google still doesn't scrape web pages that render their content via JS.

answered Jun 5, 2015 at 18:45

Ales Maticic

1,9753 gold badges13 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

min2bro Over a year ago

Is there any way I can do it using python, I don't have any experience working on JS.

Ales Maticic Over a year ago

you can save the content rendered in the headless browser, save it to your system, and than scrape the content with Python.

Shubham Goyal Over a year ago

@min2bro if you want to use Python, you can use Selenium. It has a Python client driver - seleniumhq.org/download

ElJeffe Over a year ago

@AlesMaticic Is there a way to do it within the client browser? I have a chrome extension that reads a page on the domain via XHR, but I'm running into the same issue. When I pass the result source to jquery, it doesn't contain the dynamic <div ng-view> code from Angular. Can I $compile it or something to get the full page code?

Collectives™ on Stack Overflow

fetch text from web with Angular JS tags such as ng-view

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related