Python web scrape - can't display js content

Question

The old script was working until yesterday. In page source it used to display the web content, e.g. "IssuesTraded 3,037 3,025 3,030", but not any more.. can't figure out what changed..

browser = webdriver.Chrome(executable_path=r"C:/Windows/chromedriver.exe")

urlpage = 'https://www.wsj.com/market-data/stocks/marketsdiary'
browser.get(urlpage)
results = browser.find_elements_by_xpath("//*[@id='root']//*[@class='style--grid--3gzjbqou ']")

# print('Number of results', len(results))
data = []
for result in results:
    data.append(result.text)

data = [x.split("\n") for x in data]
data = [y for x in data for y in x]

Summer · Accepted Answer · 2020-04-09 03:52:44Z

1

It looks like you have linked to a class name that can change dynamically:

results = browser.find_elements_by_xpath("//*[@id='root']//*[@class='style--grid--3gzjbqou ']")

They probably updated the app, so 3gzjbqou is no longer there. Instead of linking to the full class name, you can make this locator more stable by removing 3gzjbqou piece and only link to the part of that class.

So try this line instead of yours for searching results:

results = browser.find_elements_by_xpath("//*[@id='root']//*[contains(@class, 'style--grid')]")

I hope this helps, good luck!

answered Apr 9, 2020 at 3:52

Summer

1,5643 gold badges11 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python web scrape - can't display js content

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related