Scraping an 'onclick' table with Selenium in Python

Question

I am attempting to scrape the following webpage, using Selenium in Python (with Chrome Web Driver).

https://www.betexplorer.com/soccer/argentina/superliga/argentinos-jrs-talleres-cordoba/ptSIK7kB/#ah 1

I only wish to collect the rows of data in which the bookmaker is Bet365.

I have been able to obtain all the rows where this is the case. However, I am struggling to scrape the information within the 'onclick' table that appears when the values are clicked:

The image above shows the table ARCHIVE ODDS, which appears when the 5.90 is clicked.

The aim is to collect the information from each table in all the rows where Bet365 is the bookmaker.

My attempt so far has been to locate all the 'onclick' links using a CSS-selector:

table_links = browser.find_elements_by_css_selector("span[onclick*='16);']")

And then to loop through each of the table_links, click each one, and scrape the data which appears using the xpath:

bet365table = []
for i in table_links:
    i.click()
    xx = browser.find_element_by_xpath("//TBODY[@id='aodds-tbody']")
    bet365table.append(xx)

However, this fails each time with the error stating the element is not clickable.

Update the question with the relevant HTML and your code trials — undetected Selenium
– undetected Selenium, Commented Nov 23, 2018 at 11:26
@DebanjanB i've updated the post with my attempts so far - from my end the link in the post works, but please let me know if you're having difficulties reaching it! — bobman
– bobman, Commented Nov 23, 2018 at 11:41

QHarr · Accepted Answer · 2018-11-23 12:21:16Z

1

You could also mimic the XHR requests and get the JSON responses. Bet365 has id of 16. You can test for qualifying rows with CSS selector

import requests
import pandas as pd
import json
from pandas.io.json import json_normalize
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

d = webdriver.Chrome()
d.get("https://www.betexplorer.com/soccer/argentina/superliga/argentinos-jrs-talleres-cordoba/ptSIK7kB/#ah")
WebDriverWait(d,10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, ".in-bookmaker-logo-link.in-bookmaker-logo-link--primary.l16")))

base = 'https://www.betexplorer.com/archive-odds/'
links = d.find_elements_by_css_selector("[onclick$=', 16);']")
extracted_links = [link.get_attribute("onclick").strip("load_odds_archive(this, '").strip("', 16);") for link in links]
json_links = [base + link + '/16/?_=1' for link in extracted_links]

for link in json_links:
    res = requests.get(link)
    data= json.loads(res.content)
    data = json_normalize(data)
    print(data)

d.quit()

answered Nov 23, 2018 at 12:21

QHarr

84.5k14 gold badges58 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

bobman Over a year ago

Did not know this was a possible method! Can I please ask, how did you find out this is a possibility for this website?

QHarr Over a year ago

I used dev tools (F12) to monitor the traffic when pressing to display the tables for odds. I noticed that 16 was associated with Bet365 and that a JSON response was returned. I also saw that the start of the request URL was constant, the middle part came from the existing 365 href and that the end was a constant plus an incrementing number which bore no real significance AFAICS, so I just plugged the number 1 onto the end of the request URL. I suspect they may even offer an API so worth looking into that.

Collectives™ on Stack Overflow

Scraping an 'onclick' table with Selenium in Python

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related