python selenium search element by class name

Question

I want to collect the detailed recommendation description paragraphs that a person received on his/her LinkedIn profile, such as this link:

https://www.linkedin.com/in/teddunning/details/recommendations/

(This link can be viewed after logging in any LinkedIn account)

Here is my best try:

for index, row in df2.iterrows():

linkedin = row['LinkedIn Website']
current_url = f'{linkedin}/details/recommendations/'
driver.get(current_url)
time.sleep(random.uniform(2,3))
descriptions=driver.find_elements_by_xpath("//*[@class='display-flex align-items-center  t-14 t-normal t-black']")
s=0

for description in descriptions:
    s+=1
    print(description.text)
    df2.loc[index, f'RecDescription_{str(s)}'] = description.text

The urls I scraped in df2 are all similar to the example link above. The code find nothing in the "descriptions" variable.

My question is: What element I should use to find the detailed recommendation content under "received tab"? Thank you very much!

Teddy · Accepted Answer · 2022-12-18 23:47:34Z

1

Well you would first get the direct parent of the paragraphs. You can do that with XPath, class or id whatever fits best. After that you can do Your_Parent.find_elements(by=By.XPATH, value='./child::*') you can then loop over the result of that to get all paragraphs.

Edit

This selects all the paragraphs i have not yet looked into seperating them by post but here is what i got so far:

parents_of_paragraphs = driver.find_elements(By.CSS_SELECTOR, "div.display-flex.align-items-center.t-14.t-normal.t-black")

text_total = ""

for element in parents_of_paragraphs:
    paragraph = element.find_element(by=By.XPATH, value='./child::*')
    text_total += f"{paragraph.text}\n"

print(text_total)

edited Dec 18, 2022 at 23:47

answered Dec 18, 2022 at 22:35

Teddy

5945 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

KDWB Over a year ago

Thank you very much. The thing is that I didn't find an xpath/class/id that can loop on different user profiles. The xpath I found is unique in different user's profile. And class names I tried never let me find the corresponding element. Do you have any suggested element to find these praragraphs? Thanks again.

Teddy Over a year ago

ok, i am realizing this will not be as easy as just taking a parent and its children. i'm going to take a deeper look at this.

Collectives™ on Stack Overflow

python selenium search element by class name

1 Answer 1

Edit

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Edit

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related