Get multiple elements by tag with Python and Selenium

Question

My code goes into a website and scrapes rows of information (title and time).

However, there is one tag ('p') that I am not sure how to get using 'get element by'.

On the website, it is the information under each title.

Here is my code so far:

import time

from selenium import webdriver
from bs4 import BeautifulSoup
import requests

driver = webdriver.Chrome()
driver.get('https://www.nutritioncare.org/ASPEN21Schedule/#tab03_19')
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
eachRow = driver.find_elements_by_class_name('timeline__item')
time.sleep(1)
for item in eachRow:
    time.sleep(1)
    title = item.find_element_by_class_name('timeline__item-title')
    tim = item.find_element_by_class_name('timeline__item-time')
    tex = item.find_element_by_tag_name('p') # This is the part I don’t know how to scrape
    print(title.text, tim.text, tex.text)

Miguel Almonte · Accepted Answer · 2021-02-14 23:30:52Z

I checked the page and there are several p tags, I suggest to use find_elements_by_tag_name instead of find_element_by_tag_name (to get all the p tags including the p tag that you want) and iterate over all the p tags elements and then join the text content and do strip on it.

from selenium import webdriver
from bs4 import BeautifulSoup
import time
import requests
driver = webdriver.Chrome()

driver.get('https://www.nutritioncare.org/ASPEN21Schedule/#tab03_19')
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
eachRow = driver.find_elements_by_class_name('timeline__item')
time.sleep(1)
for item in eachRow:
    time.sleep(1)
    title=item.find_element_by_class_name('timeline__item-title')
    tim=item.find_element_by_class_name('timeline__item-time')
    tex=item.find_elements_by_tag_name('p')
    text = " ".join([i.text for i in tex]).strip()
    print(title.text,tim.text, text)

VirtualScooter · Accepted Answer · 2021-02-14 23:22:50Z

1

Since the webpage has several p tags, it would be better to use the .find_elements_by_class() method. Replace the print call in the code with the following:

    print(title.text,tim.text)
    for t in tex:
        if t.text == '':
            continue
        print(t.text)

answered Feb 14, 2021 at 23:22

VirtualScooter

1,9083 gold badges21 silver badges31 bronze badges

1 Comment

Peter Mortensen Over a year ago

From a comment: "find_element_by_* and find_elements_by_* are removed in Selenium 4.3.0. Use find_element instead.". Though it doesn't really answer the question what can be done if the number of elements is different from exactly one. There may be a canonical Stack Overflow answer somewhere.

Peter Mortensen · Accepted Answer · 2022-11-10 17:48:23Z

0

Maybe try using different find_elements_by_class... I don't use Python that much, but try this unless you already have.

edited Nov 10, 2022 at 17:48

Peter Mortensen

31.4k22 gold badges110 silver badges134 bronze badges

answered Feb 14, 2021 at 23:04

Wyatt

216 bronze badges

5 Comments

Void S Over a year ago

The p tag does not have a class name unfortunately

Wyatt Over a year ago

what does 'p' represent?

Void S Over a year ago

paragraph, not sure if its considered tag or css selector etc

Wyatt Over a year ago

id know then because tag name shouldwork but if it doesn't i guess i can't help sorry

Wyatt Over a year ago

unless xpath. (//p[text() = 'JBL']) works

Collectives™ on Stack Overflow

Get multiple elements by tag with Python and Selenium

3 Answers 3

Comments

1 Comment

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related