How to parse an html webpage that executes javascript

Question

I am trying to write a program that scraps for the IUPACcondensed on this webpage .

Here G03307GF is the ID. I need this:

HexNAc(b1-?)[Fuc(a1-?)]GlcNAc(b1-2)Man(a1-3)[HexNAc(b1-?)[Fuc(a1-?)]GlcNAc(b1-2)Man(a1-6)]Man(b1-4)GlcNAc(b1-4)[Fuc(a1-6)]GlcNAc

I tried to use selenium for this.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.add_argument('--headless')
driver = webdriver.Chrome('', options = chrome_options)

# takes accession number and returns IUPAC
def getIUPAC(acc_no):

    url = 'https://glytoucan.org/Structures/Glycans/' + acc_no

    driver.get(url)
    IUPAC = driver.find_element_by_xpath('//*[@id="descriptors"]/togostanza-iupaccondensed//main/div/pre/code/text()')
    driver.close()

    return IUPAC

IUPAC = getIUPAC('G37498VS')

print(IUPAC)

It says the the element does not exist.

Possible duplicate of Can Xpath expressions access shadow-root elements? — Smart Manoj
– Smart Manoj, Commented May 25, 2019 at 3:55

VeNoMouS · Accepted Answer · 2019-05-25 03:55:36Z

2

import re
import requests

def getIUPAC(acc_no):
    ret = requests.get('https://glytoucan.org/Structures/Glycans/{}'.format(acc_no))
    z = re.search('<meta name="description".*?The IUPAC representation is (.+)\.\s+The', ret.content, re.DOTALL | re.MULTILINE)
    return z if z else 'Unknown'


print('IUPAC is {}'.format(getIUPAC('G03307GF')))

Our result is...

IUPAC is HexNAc(b1-?)[Fuc(a1-?)]GlcNAc(b1-2)Man(a1-3)[HexNAc(b1-?)[Fuc(a1-?)]GlcNAc(b1-2)Man(a1-6)]Man(b1-4)GlcNAc(b1-4)[Fuc(a1-6)]GlcNAc

answered May 25, 2019 at 3:55

VeNoMouS

3102 silver badges5 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Shaurya Over a year ago

That worked! Thanks. I did not use 're' because there was some issue with bytes but the 'requests' thing worked.

Alan · Accepted Answer · 2019-05-25 19:45:12Z

0

Better use requests as shown by VeNoMouS. Just wanted to add that you're getting element does not exist because the driver was closed before you printed it.

answered May 25, 2019 at 19:45

Alan

813 bronze badges

Collectives™ on Stack Overflow

How to parse an html webpage that executes javascript

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related