How to scrape the nested data from Linkedin page using Selenium and Python

Thank you everyone!

I ended up with this code under that worked.

get_education_school = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//ul[@class='pvs-list ']/li//span[contains(@class, 'hoverable-link-text')]//span[1]")))]

get_education_years = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//ul[@class='pvs-list ']/li//span[contains(@class, 't-14 t-normal t-black--light')]//span[1]")))]

results_education_school = []
results_education_years = []
for i,j in zip(get_education_school, get_education_years):
    results_education_school.append(i)
    results_education_years.append(j)

print(results_education_school)
print(results_education_years)

I would first get the list for the education section.

education_list = driver.find_element(By.CSS_SELECTOR, 'ul.pvs-list')
# loop through education_list for place and years
# would recommend relative locators for this task.
# find the image and get the first and second span with text inside of them.

I am adding further details to the code now. Please hold.

You can use below properties to identify the school name list:

ancestorClass="optional-action-target-wrapper display-flex flex-column full-width" class="display-flex align-items-center" tag="DIV"

Use these properties to identify the year list:

ancestorClass="optional-action-target-wrapper display-flex flex-column full-width" class="t-14 t-normal t-black--light" tag="SPAN"

You may use above info to compose an XPath to locate the list, or if you don't mind using other python libraries, there is a sample code in GitHub to scrape the school and year.

@Nadia S. you can try the following code. I have provided comments inline inside the code.

    @Test
    public void linkedInTest() {
        driver.get("https://www.linkedin.com");

        // You need to enter the credentials for your linkedin below for login
        driver.findElement(By.id("session_key")).sendKeys("");
        driver.findElement(By.id("session_password")).sendKeys("");
        driver.findElement(By.className("sign-in-form__submit-button")).click();
        driver.get("https://www.linkedin.com/in/williamhgates/details/education/");

        //Wait for the Education details to get populated. 
        WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(7));
        wait.until(ExpectedConditions.visibilityOfElementLocated(
                By.xpath("//div[@class = 'pvs-list__container']//div[@class = 'scaffold-finite-scroll__content']/ul")));
        
        //Take all elements showing education details in a list 
        List<WebElement> allEducation = driver.findElements(By
                .xpath("//div[@class = 'pvs-list__container']//div[@class = 'scaffold-finite-scroll__content']/ul/li"));
        //Extract details of each education item in the list. 
        //Below the details are directed to console. You can use a collection to store them.
        for (WebElement oneEducation : allEducation) {
            WebElement education = oneEducation.findElement(
                    By.xpath(".//*[contains(@class,\"mr1 hoverable-link-text\")]/span[@aria-hidden='true']"));
            System.out.print("Education - " + education.getText());
            try {
                WebElement educationType = oneEducation
                        .findElement(By.cssSelector(".t-14.t-normal span[aria-hidden='true']"));
                System.out.print("      Education Type - " + educationType.getText());
            } catch (NoSuchElementException e) {
                System.out.print("      Education Type - " + "is Not Specified");
            }
            try {
                WebElement educationYear = oneEducation
                        .findElement(By.cssSelector(".t-14.t-normal.t-black--light span[aria-hidden='true']"));
                System.out.println("        Education Year - " + educationYear.getText());
            } catch (NoSuchElementException e) {
                System.out.println("        Education Year - " + "is Not Specified");
            }
        }

    }