Trying to loop through profile lists using Selenium

Question

I'm trying to loop through all profiles and store the name of the person, the job profile and the location in a list. Here is the screenshot of the screen LinkedIn screen I am on:

Here is the li html tag that I'll have to loop over:

<li class="reusable-search__result-container ">
          
          <div class="entity-result  ">
          
            <div class="entity-result__item">
  <div class="entity-result__image">
    <div class="display-flex align-items-center">
      
      <a class="app-aware-link" aria-hidden="true" href="https://www.linkedin.com/search/results/people/headless?geoUrn=%5B103644278%5D&amp;origin=FACETED_SEARCH&amp;keywords=python%20developer">
    <div id="ember522" class="ivm-image-view-model ember-view">  <div class="
    ivm-view-attr__img-wrapper ivm-view-attr__img-wrapper--use-img-tag display-flex
    
  ">
    <div class="EntityPhoto-circle-3-ghost-person ivm-view-attr__ghost-entity ">
<!---->    </div>
</div>

</div>
</a>

    </div>
  </div>
  <div class="entity-result__content entity-result__divider pt3 pb3 t-12 t-black--light">
    <div class="mb1">
      
    <div class="linked-area flex-1 cursor-pointer">
  
      
  <div class="t-roman t-sans">
    <span class="entity-result__title">
      <div class="display-flex">
  <span class="entity-result__title-line flex-shrink-1 entity-result__title-text--black ">
    <span class="entity-result__title-text  t-16">
      <a class="app-aware-link" href="https://www.linkedin.com/search/results/people/headless?geoUrn=%5B103644278%5D&amp;origin=FACETED_SEARCH&amp;keywords=python%20developer">
        <!---->LinkedIn Member<!---->
      </a>
<!---->    </span>
  </span>
<!----></div>

    </span>
  </div>

    <div>
      <div class="entity-result__primary-subtitle t-14 t-black">
        <!---->Software Developer<!---->
      </div>
        <div class="entity-result__secondary-subtitle t-14">
          <!---->United States<!---->
        </div>
    </div>

  

    
</div>


    </div>

      <div class="linked-area flex-1 cursor-pointer">
  
        <p class="entity-result__summary entity-result__summary--2-lines t-12 t-black--light ">
          <!---->Current: Full Stack Software<span class="white-space-pre"> </span><strong><!---->Developer<!----></strong><span class="white-space-pre"> </span>at GE Healthcare<!---->
        </p>
      
</div>


<!---->  </div>
  <div class="entity-result__actions entity-result__divider entity-result__actions--empty">
<!---->    <!---->
  </div>
</div>

          
</div>

        
        </li>

Currently, I'm able to get the profile names using this code:

profile_names = []
linkedin_members = browser.find_elements_by_xpath('//span[@class="entity-result__title"]')

for linkedin_member in linkedin_members:
    name = linkedin_member.find_element_by_xpath('.//a[@class="app-aware-link"]').get_attribute('text').strip()
    profile_names.append(name)

But I'm unable to get the job locations and job profiles. Can anyone guide me on the code for that?

I tried something like this but it threw an error:

profile_names = []
job_profiles = []

linkedin_members = browser.find_elements_by_xpath('//div[@class="linked-area flex-1 cursor-pointer"]')

for linkedin_member in linkedin_members:
    name = linkedin_member.find_element_by_xpath('.//a[@class="app-aware-link"]').get_attribute('text').strip()
    job_profile = linkedin_member.find_element_by_xpath('.//div[@class="entity-result__primary-subtitle"]').text
    profile_names.append(name)
    job_profiles.append(job_profiles)

Could you pls specify which error is thrown when you run the code? — Rafael Setton
– Rafael Setton, Commented Jun 4, 2021 at 11:24
It was throwing this error: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":".//a[@class="app-aware-link"]"} — ShridharK
– ShridharK, Commented Jun 4, 2021 at 11:26

Prophet · Accepted Answer · 2021-06-04 12:38:38Z

2

Another way to do this is:

members_serach_results_xpath = '//div[@class="entity-result__item"]'
member_name_xpath = '//span[contains(@class,"entity-result__title-text")]//span[@dir]'
member_location_xpath = '//div[contains(@class,"entity-result__secondary-subtitle")]'
member_job_title_xpath = '//div[@class="entity-result__item"]//div[contains(@class,"entity-result__primary-subtitle")]'

profile_names = []
profile_addresses = []
profile_job_titles = []
linkedin_members = browser.find_elements_by_xpath(members_serach_results_xpath)

for linkedin_member in linkedin_members:
    name = linkedin_member.find_element_by_xpath('.' + member_name_xpath).get_attribute('text').strip()
    profile_names.append(name)
    address = linkedin_member.find_element_by_xpath('.' + member_location_xpath).get_attribute('text').strip()
    profile_addresses.append(address)
    job_title = linkedin_member.find_element_by_xpath('.' + member_job_title_xpath).get_attribute('text').strip()
    profile_job_titles.append(job_title)

Here I put the locators as parameters out of the code.
It's one of best practices not to put locators hardcoded inside the methods using it.

edited Jun 4, 2021 at 12:38

answered Jun 4, 2021 at 11:38

Prophet

33.5k28 gold badges58 silver badges90 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

ShridharK Over a year ago

I'll definitely look into this code later. Thanks for posting :)

C. Peck Over a year ago

I like the approach! Just pointing out your xpaths on lines 2 and 3 use single quotes inside single quotes, the class names like entity-result__title-text should be surrounded by double quotes

Prophet Over a year ago

@C.Peck thanks for corrections! It's all because of Java :) There we always use " for external strings and ' for internal strings. Opposite of Python

ShridharK Over a year ago

Have asked a new question: stackoverflow.com/questions/67870236/…

C. Peck · Accepted Answer · 2021-06-04 11:20:40Z

1

You just have to identify those elements (and I think you can do so using the class with a css selector), then loop through the elements and append the text to the appropriate array.

profile_names = []
linkedin_members = browser.find_elements_by_xpath('//span[@class="entity-result__title"]')

for linkedin_member in linkedin_members:
    name = linkedin_member.find_element_by_xpath('.//a[@class="app-aware-link"]').get_attribute('text').strip()
    profile_names.append(name)

user_positions = []

positions = browser.find_elements_by_css_selector('div.entity-result__primary-subtitle')

for position in positions:
    user_positions.append(position.text.strip())
    
user_locations = []

locations = browser.find_elements_by_css_selector('div.entity-result__secondary-subtitle')

for location in locations:
    user_locations.append(location.text.strip())

answered Jun 4, 2021 at 11:20

C. Peck

3,7193 gold badges21 silver badges38 bronze badges

9 Comments

ShridharK Over a year ago

Thanks it worked! Can you let me know why the code that I had tried didn't work?

ShridharK Over a year ago

I had got this error: It was throwing this error: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":".//a[@class="app-aware-link"]"}

ShridharK Over a year ago

Essentially, I wanted to just use 1 for loop to get the data for all the 3 things. Your solution works completely fine and I'll go ahead with that but just wanted to understand where I was going wrong.

C. Peck Over a year ago

The problem for your code is with //div[@class="linked-area flex-1 cursor-pointer"]. The div that this finds doesn't have any descendants with tag a, so './/a[@class="app-aware-link"]' doesn't locate anything.

C. Peck Over a year ago

I can put it all in one loop if you prefer

|

Collectives™ on Stack Overflow

Trying to loop through profile lists using Selenium

2 Answers 2

4 Comments

9 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

9 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related