I am trying to scrape the data contained in a table on https://www.bop.gov/coronavirus/. However, when one first visits the page the table is hidden behind a link (https://www.bop.gov/coronavirus/#) that leads to the same page but expands the hidden table on the page. However, I cannot find within this link within the webpage's source code or using selenium in order to expand the table and scrape its data. How can I go about accessing the data in this table using python?
1
-
Code the actions that a user would do to show this table... click a link, click a button, etc. I can't see which table you are referring to because you haven't included that step in your description.JeffC– JeffC2020-04-27 21:13:18 +00:00Commented Apr 27, 2020 at 21:13
Add a comment
|
2 Answers
The endpoint from which the data is loaded on the page is available under the network tab of the developer tools. The data you need is loaded from
You might also want to take a look at
as the first link only contains the short codes for the names.
Comments
The table data is readily available under the div with id="totals_breakdown".
You can directly call the page_source and parse the data for that element with BeautifulSoup without needing to "show" the element.
If you MUST show the element for some reason, you simply have to remove the class closed from the div with id="totals_breakdown"
