from selenium import webdriver
driver = webdriver.Chrome()
driver.get("url_goes_here")
p_id = driver.find_elements_by_tag_name("script")
This procures me the script I need. I don't need to execute it, as it's already executed and running upon initial page load. It contains a variable named "task". How do I access its value with Selenium?
The regex module re can help you with that:
import re
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("url_goes_here")
p_id = driver.find_elements_by_tag_name("script")
for script in p_id:
innerHTML=script.get_property('innerHTML')
task=re.search('var task = (.*);',innerHTML)
if task is not None:
print(task.group(1))
What this does is look through the innerHTML of each script and, from the defined search pattern ('var task = (.*);'), capture the matching string group ((.*)). Print out the group if a match is found.
script.get_property('innerHTML') It returns an empty string! :(
.get_attribute instead of .get_property do you still get an empty string?.get_attribute, but I eventually solved my problem by extracting the appropriate element via the .find_elements_by_xpath rather than via by tag name as depicted in the original question and THEN using your .get_property('innerHTML') suggestion. My issue is solved now, and since you were the only one who at least put me on the right path to its solution, I will mark your answer as correct! Thank you!!! :)you can access value of tag or any element of html via .text or .getText()
Since you are using find_elements_by_tag_name() which returns list of elements.
Iterate that list and check element.text contains task then print text of that element.
p_id = driver.find_elements_by_tag_name("script")
for id in p_id:
if 'task' in id.text:
print(id.text)
Since you are using find_elements_by_tag_name() which returns list of elements.. Any better suggestion for scraping for scripts specifically? Would searching by... let's say, XPath, produce a better formatted output or something? Second, this is the actual content of an element of the output list that gets produced by the execution of the find_elements_by_tag_name(): <selenium.webdriver.remote.webelement.WebElement (session="53d2976784532dd4717abff68170b22a", element="8d9f432d-7a2e-47e0-8023-d6c092ee9620")> As you can see, it's not very legible.
task is that right?#Use Xpath instead:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("url_goes_here")
p_id = driver.find_element(By.XPATH,"ADDXPATH")
p_id.get_attribute('outerHTML')
