Webscraping a dynamic page which requires scrolling using Selenium in Java

Question

I want to find and print all the names of the backgrounds found on this page:

https://store.steampowered.com/points/shop/c/backgrounds/cluster/2

And the problem in my program is that I can't scroll to the bottom of page. I tried

Actions
Selecting a random element to perform a space key press on it
A bunch of Javascript code (none of which has worked on this site)
The Robot method (with which I send a pagedown keystroke and it does work, yet I can't use the pc to do anything else while it's scrolling)

This is my program so far:


public static void steamScraper() {
    
    System.setProperty("webdriver.chrome.driver", "C:\\Users\\user\\Documents\\Selenium\\chromedriver_win32\\chromedriver.exe");
    
    ChromeOptions options = new ChromeOptions();
    options.setBinary("C:\\Users\\user\\Downloads\\chrome-win\\chrome.exe");
    WebDriver driver = new ChromeDriver(options);
    
    driver.get("https://store.steampowered.com/points/shop/c/backgrounds/cluster/2");
    
    driver.manage().window().maximize();
    
    // let the page load for some time
    try {
        Thread.sleep(5000);
    } catch (InterruptedException e1) {
        // TODO Auto-generated catch block
        e1.printStackTrace();
    }
    
    JavascriptExecutor js = (JavascriptExecutor) driver;
    
    // i = 2200, because there are ~44000 items and 20 items load each scroll, thus 44000 / 20 = 2200
    
    for (int i = 0; i < 10; i++) {
        
        try {
            Thread.sleep(1000);
        } catch (InterruptedException e1) {
            e1.printStackTrace();
        }
        
        System.out.println(i + 1 + ". " + "scrolling...");
        
    //  here is where I would do the scrolling, and I tried a lot of javascript methods to scroll to the bottom of the page
    //  yet none that I have tried yet would actually perform a scroll (though the method below worked on another site than Steam)
    //
    //  js.executeScript("window.scrollTo(0, document.body.scrollHeight)");

    }
    
    System.out.println("");
    
    // get all div classes containing an item (background)
    
    List<WebElement> nameOfGames = driver.findElements(By.xpath(".//div[@class='rewarditem_AppIconContainer_3Oyyi']//img"));
    
    for (WebElement webElement : nameOfGames) {

        // print the item name (title of img)
        
        System.out.println(webElement.getAttribute("title"));
        
    }
    
    
} ```

The following link may answer your question: stackoverflow.com/questions/48850974/… I usually use python for my webscraping, but lower down the code is translated to java — TomM
– TomM, Commented Apr 29, 2021 at 18:54
Is it crashing, or just not doing what you want? Try printing some attribute of the element you are using to anchor the scroll each time and see if that is the issue — TomM
– TomM, Commented Apr 29, 2021 at 19:14
@TomM I'm not sure I understood you correctly, but when I tried Zenab Gorach's approach it just didn't scroll to the bottom of the page, and when I print an attribute inside the while loop it prints only once. That means it only tries to scroll once I suppose. — JustAnOwOAccount
– JustAnOwOAccount, Commented Apr 29, 2021 at 19:32
My best guess is that you find the elements on the page with the List<WebElement> nameOfGames = driver.findElements... line, but when you actually load the page the additional elements aren't rendered. So you can only move to the end of the list of elements that were there when the page originally loaded. You would need to add the new elements to that list after the page loads the next set, then scroll to the lowest element you added. I'm not as familiar with calling js scripts from selenium so I'm not sure how to help there. — TomM
– TomM, Commented Apr 29, 2021 at 20:45

JustAnOwOAccount · Accepted Answer · 2021-04-30 00:39:21Z

I think I just came up with the solution:

I made a new List<WebElement> gamesAfterFirstScroll = null; list. I use this list in a for-loop by finding all elements that are of interest to me, then I used action.moveToElement(gamesAfterFirstScroll.get(lastGameItem)) and action.perform() to scroll to the last item. This is how it looks:

        List<WebElement> nameOfGames = driver.findElements(By.xpath(".//div[@class='rewarditem_AppIconContainer_3Oyyi']//img"));
        
        Actions action = new Actions(driver);
    
        List<WebElement> gamesAfterFirstScroll = null;
        
        System.out.println("Scrolling.\r\n");
        
        for(int i = 0; i < 50; i++) {

            try {
                Thread.sleep(2000);
            } catch (InterruptedException e1) {
                // TODO Auto-generated catch block
                e1.printStackTrace();
            }
            
            gamesAfterFirstScroll = driver.findElements(By.xpath(".//div[@class='rewarditem_AppIconContainer_3Oyyi']//img"));
                
            int lastGameItem = gamesAfterFirstScroll.size() - 1;
            
            action.moveToElement(gamesAfterFirstScroll.get(lastGameItem));
            action.perform();
            
            System.out.println(i+1 + ". scroll");
            
        }
            
        for (WebElement e : gamesAfterFirstScroll) {
            
            System.out.println(e.getAttribute("title"));
            nameOfGames.add(e);
        }

Collectives™ on Stack Overflow

Webscraping a dynamic page which requires scrolling using Selenium in Java

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related