Using selenium with python, how can I get Var from HTML where it's declared in a JS <script> element

Question

I want to get var declared inside a JS in the htm;. but there are no ids, elements. How can I get this data?

Because there is no address, but only var name, I don't know how to do it

Website HTML:

<script type="text/javascript">
var imgInfoData = 'data which i want to crawl'

</script>

My python Code:

#set url
HOMEPAGE = "https://land.naver.com/info/complexGallery.nhn?newComplex=Y&startImage=Y&rletNo=102235"


#open web
driver = webdriver.Firefox()
driver.wait = WebDriverWait(driver, 2)
driver.get(HOMEPAGE)

#try to get text from html
time.sleep(1)
WebDriverWait(driver, 3).until(EC.presence_of_element_located((By.XPATH, '//script["var"]'))).text

Cant you use something like beautifulsoup? crummy.com/software/BeautifulSoup/bs4/doc Common library to scrap the web — user1462442
– user1462442, Commented Mar 29, 2019 at 1:12
I could get data from normal HTML. but in this case, I don't know how to crawl data and I couldn't find the way in beautifulsoup either. T.T — SEUNGIL LEE
– SEUNGIL LEE, Commented Mar 29, 2019 at 1:36
stackoverflow.com/questions/38547569/… I believe you should be able to find script tags in beautiful soup then run all those code through any type of javascript parser to find those variables. Parsing should be a common routine — user1462442
– user1462442, Commented Mar 29, 2019 at 1:42
I would not recommend using regex to parse turning languages. Please use something else. commandcenter.blogspot.com/2011/08/… softwareengineering.stackexchange.com/questions/223634/… — user1462442
– user1462442, Commented Mar 29, 2019 at 1:48

Mark SuckSinceBirth · Accepted Answer · 2019-03-29 02:17:26Z

I check the site you are scraping and it seems the scripts was already included in the html page, so i think you don't need to use webdriver and you can just use requests and beautifulsoup.

get the html data using requests:

res = requests.get(url, headers=headers, params=params)

Then Soup the html text to get the script tags and find which tags has the var imgInfoData:

soup = BeautifulSoup(res.text, "html5lib")
    scripts = soup.findAll('script', attrs={'type':'text/javascript'})
    for script in scripts:
        if "var imgInfoData" in script.text: #script with imgInfoData captured
            return script.text.replace("var imgInfoData =","").strip()[:-1]

just remove the

var imgInfoData =

and

;

of the text to get the string value or you could use regex to get the json string inside a text.

Full Code:

import requests
from bs4 import BeautifulSoup

def getimgInfoData():
    url = "https://land.naver.com/info/complexGallery.nhn"
    headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
    params = {"newComplex":"Y",
              "startImage":"Y",
              "rletNo":"102235"}
    res = requests.get(url, headers=headers, params=params)

    soup = BeautifulSoup(res.text, "html5lib")
    scripts = soup.findAll('script', attrs={'type':'text/javascript'})
    for script in scripts:
        if "var imgInfoData" in script.text: #script with imgInfoData captured
            return script.text.replace("var imgInfoData =","").strip()[:-1]
    return None

print(getimgInfoData())

then just convert the result from getimgInfoData() to json if you want.

Collectives™ on Stack Overflow

Using selenium with python, how can I get Var from HTML where it's declared in a JS <script> element

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related