Python Selenium Web Scraping Hidden Div

Question

Well, as the title suggests I'm trying to scrape some data from a website (example) using Selenium, however I'm having trouble getting the data hidden in each row from the Pro Results table, the one that shows when you click the Show Details button (+).

This is my code:

from bs4 import BeautifulSoup

from selenium import webdriver

# Set some Selenium Options
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')

# Webdriver
wd = webdriver.Chrome('chromedriver',options=options)

# URL
url = 'https://www.tapology.com/fightcenter/fighters/30449-sultan-aliev'

# Load URL
wd.get(url)

# Get HTML
soup = BeautifulSoup(wd.page_source, 'html.parser')

# All rows of the Pro Record table 
rows = soup.findAll('div', {'class': 'result'})

print(len(rows)) 

# [Out] 18

# Try to find all hidden data
hidden = soup.findAll('div', {'class': 'detail tall'})

print(hidden)

# [Out] []

As you can see I can get easily the rows of the table but when I try to get the hidden data I just can't find a way to get it.

I'm not very familiar to Selenium either, so any guidance will be welcome.

Pedro Lobito · Accepted Answer · 2020-04-06 03:06:01Z

1

The json that contains the information you need is being fetched from tapology api using a js request.
To retrieve this information, install seleniumwire and use:

from seleniumwire import webdriver
import requests
# ...
driver = webdriver.Firefox()
driver.scopes = [ 'api.tapology.com'] # filter api.tapology.com requests only 
driver.get('https://www.tapology.com/fightcenter/fighters/30449-sultan-aliev')

for request in driver.requests:
    print(request.path)
    r = requests.get(request.path, headers=request.headers)
    print(r.json())  # the info you need is here

https://api.tapology.com/v1/internal_ranking_items/47211352261 # ranking data https://api.tapology.com/v1/internal_fighters/472130449 # fighter data

edited Apr 6, 2020 at 3:06

answered Apr 4, 2020 at 21:19

Pedro Lobito

99.8k36 gold badges274 silver badges278 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Xhoan Over a year ago

Thanks! this solution is so simple and returned way more data than I expected.

Hunter Tran · Accepted Answer · 2020-04-04 20:59:16Z

1

Maybe you don't need to extract data from HTML. A quick inspect in Chrome's developer tools shows me that this site has API to query for data, but you need to use the exact same request header.

internal_fighters in JSON format

internal_ranking_items in JSON format

Another alternative for the problem is just to simulate the "click" action on the button.

The problem with your "hidden" div is that the div tag is dynamically added when the user clicks on the (+) button.

# click submit button
submit_button = wd.find_elements_by_xpath('//*[@id="fighterRecord"]/section[1]/ul/li[1]/div/div[4]/i')[0]
submit_button.click()

edited Apr 4, 2020 at 20:59

answered Apr 4, 2020 at 20:53

Hunter Tran

16k2 gold badges16 silver badges27 bronze badges

2 Comments

Xhoan Over a year ago

Silly question, ¿how did you find the xpath?, tried this and it worked, however, I only get the first row to show the hidden div. How can I do the same for the rest of the rows?

Hunter Tran Over a year ago

right click on the element in chrome developer tool > Copy > Copy XPath

KunduK · Accepted Answer · 2020-04-04 22:52:23Z

If you want use selenium only try following code.You need to click on each expand button to get the next table info.Then use element.get_attribute("textContent")

Code:

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

driver=webdriver.Chrome()
driver.get("https://www.tapology.com/fightcenter/fighters/30449-sultan-aliev")
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"span.closebutton_closeButton--3abym"))).click()
tablerecords=WebDriverWait(driver,20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"div.result")))
print(len(tablerecords))
for row in range(len(tablerecords)):
    tablerecords = WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.result")))
    try:
        expand_btn=tablerecords[row].find_element_by_xpath(".//div[@class='more']/i")
        driver.execute_script("arguments[0].click();", expand_btn)
        time.sleep(2)
        hiddenelements=tablerecords[row].find_element_by_xpath("./following-sibling::div[1]").get_attribute('textContent')
        print(hiddenelements)
    except:
        continue

Output:

25
Billing:Preliminary CardDuration:3 x 5 Minute RoundsWeight:Welterweight · 170 lbs (77.1 kg) · Weigh-In 170.0 lbs (77.1 kgs)Odds:-120 · Near EvenReferee:Leon Roberts
UFC on ESPN+ 7
UFC 230: Cormier vs. Lewis· Aliev Injury
Billing:Preliminary CardDuration:3 x 5 Minute RoundsWeight:Welterweight · 170 lbs (77.1 kg) · Weigh-In 171.0 lbs (77.6 kgs)Odds:+250 · Moderate UnderdogReferee:Osiris Maia
UFC on FOX 26· Aliev Injury
Billing:Preliminary CardDuration:3 x 5 Minute RoundsWeight:Welterweight · 170 lbs (77.1 kg) · Weigh-In 171.0 lbs (77.6 kgs)Odds:+135 · Slight UnderdogReferee:Ed CollantesDisclosed Pay:$20,000 ($10K Base, $10K Bonus)
UFC 202: Diaz vs. McGregor 2· Aliev Injury
Billing:Preliminary CardDuration:3 x 5 Minute RoundsWeight:Welterweight · 170 lbs (77.1 kg) · Weigh-In 170.0 lbs (77.1 kgs)Odds:-180 · Slight FavoriteReferee:Bobby RehmanUFC on FOX 14 Performance of the Night
Billing:Main CardDuration:3 x 5 Minute RoundsWeight:Light Heavyweight · 205 lbs (93.0 kg)
Billing:Main CardDuration:3 x 5 Minute RoundsWeight:Middleweight · 185 lbs (84.0 kg) · Weigh-In 185.4 lbs (84.1 kgs)Referee:Valentin Tarasov
Billing:Main CardDuration:3 x 5 Minute RoundsWeight:Middleweight · 185 lbs (83.9 kg) · Weigh-In 185.8 lbs (84.3 kgs)Odds:-350 · Moderate FavoriteReferee:Herb Dean
Billing:Preliminary CardDuration:3 x 5 Minute RoundsWeight:Middleweight · 185 lbs (83.9 kg) · Weigh-In 185.5 lbs (84.1 kgs)Odds:+145 · Slight UnderdogReferee:Joseph Hawes
Billing:Main CardDuration:3 x 5 Minute RoundsWeight:Light Heavyweight · 205 lbs (93.0 kg)
Billing:Main CardDuration:2 x 5 Minute Rounds
Billing:Main CardDuration:3 x 5 Minute Rounds
Title Bout:Tournament ChampionshipBilling:Main CardDuration:3 x 5 Minute RoundsWeight:Light Heavyweight · 205 lbs (93.0 kg)
ProFC 39: Global Grand Prix (Stage 6)· Omari Akhmedov injury
Title Bout:Tournament ChampionshipBilling:Main CardDuration:2 x 5 Minute RoundsWeight:Light Heavyweight · 205 lbs (93.0 kg)
Billing:Preliminary CardDuration:3 x 5 Minute RoundsWeight:Light Heavyweight · 205 lbs (93.0 kg)
Billing:Main CardDuration:2 x 5 Minute RoundsWeight:Light Heavyweight · 205 lbs (93.0 kg)
Title Bout:Tournament ChampionshipBilling:Main EventDuration:2 x 5 Minute RoundsWeight:Light Heavyweight · 205 lbs (93.0 kg)

Collectives™ on Stack Overflow

Python Selenium Web Scraping Hidden Div

3 Answers 3

1 Comment

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related