How to extract multiple text using selenium python

Question

I am scrolling google job page to extract multiple company names, but getting only 2 records.

Can anyone suggest me how to tweak the below code to get all the companies name present next to word 'via' as showing in the below image.

driver.get("https://www.google.com/search?q=bank+jobs+in+india&rlz=1C1CHBF_enIN869IN869&oq=upsc+jo&aqs=chrome.1.69i57j0i433i512j0i131i433i512j0i512l3j0i131i433i512l2j0i512j0i433i512&sourceid=chrome&ie=UTF-8&ibp=htl;jobs&sa=X&sqi=2&ved=2ahUKEwjR27GN_qPzAhX4ppUCHb_0B_QQkd0GegQIORAB#fpstate=tldetail&sxsrf=AOaemvIxuJXh3if0tw7ezZfjkXRe5DSxsA:1632911697417&htivrt=jobs&htidocid=hr3yUBTZAssve05hAAAAAA%3D%3D")

name = []
cnt = 0
try:
    while True:
        element = driver.find_elements_by_xpath("//div[@role='treeitem']")
        driver.execute_script("arguments[0].scrollIntoView(true);", element[cnt])
        time.sleep(2)
        try:
            nam = driver.find_element_by_xpath("//div[contains(@class, 'oNwCmf')]").text
            nam1 = nam.split("\nvia ")[1]
            name.append(nam1.split("\n")[0])
        except:
            name.append("")
        cnt=cnt+1
except:
    pass

pmadhu · Accepted Answer · 2021-09-30 15:00:24Z

1

Try like this:

Get the name nam using WebElement element(instead of finding with driver). Since we are finding element within elements now, add a dot in the xpath. This will get the name of that particular Element.

try:
    while True:
        element = driver.find_elements_by_xpath("//div[@role='treeitem']")
        driver.execute_script("arguments[0].scrollIntoView(true);", element[cnt])
        time.sleep(2)
        try:
            nam = element[cnt].find_element_by_xpath(".//div[contains(@class, 'oNwCmf')]").text # finds the name of that particular element[cnt], add a dot to find element within element.
            print(nam)
            nam1 = nam.split("\nvia ")[1]
            name.append(nam1.split("\n")[0])
        except:
            name.append("")
        cnt=cnt+1
except:
    pass

answered Sep 30, 2021 at 15:00

pmadhu

3,4332 gold badges14 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to extract multiple text using selenium python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related