Web Scraping: Problems with the HTML source code [duplicate]

Question

I try to web scrape data from this site: https://price.metal.com/Rare-Earth (rare earth material prices) with Python/BeautifulSoup. The question is not about Python, but the HTML source code of the website. When I am using the "Inspect Element" function of Firefox I am getting the needed data/HTML tag I am looking for:

<div class="td-border-item frame-text-over-flow" title="294,000 ~ 299,000"> 294,000 ~ 299,000

But the problem is that when I view the Page Source of https://price.metal.com/Rare-Earth I cannot find the above element with

So I am curious if there is another HTML file which contains the information that I am looking for?

Yes. There are AJAX calls to https://price.metal.com/api/get_product_lists?t=1582288095974&second_level_name=rare+earth which has "newest_price":{"Average":296500,"Highs":299000,"Low":294000, — mplungjan
– mplungjan, Commented Feb 21, 2020 at 12:29
The data has likely been loaded and inserted via JavaScript. What resources actually get loaded, should be easy enough to figure out using your browser dev tools, network panel. — C3roe
– C3roe, Commented Feb 21, 2020 at 12:29

Duc Nguyen · Accepted Answer · 2020-02-21 17:14:15Z

0

This element does not contain the value because it was loaded by asynchronization mechanism (Ajax appending HTML value after that page loaded or binding data for example).

I suggest you combine it with Selenium Python parallelly, inspect the tag and get value by Selenium. You will get it.

https://selenium-python.readthedocs.io/

edited Feb 21, 2020 at 17:14

answered Feb 21, 2020 at 12:30

Duc Nguyen

1682 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Web Scraping: Problems with the HTML source code [duplicate]

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related