Scroll Page in Slack chat with Selenium using Python

Question

My script is scraping data from a Slack chat for which i don't have an API so i use Selenium for Webscraping, and when i refresh the page i need to scroll at the bottom of the messages page.

I managed to locate effectively the scroll bar with the selector and I'm trying different ways to move the page down, but it doesn't move by an inch.

Thanks for any suggestion!

scroll_selector = 'body > div.p-client_container > div > div > div.p-client_workspace_wrapper > div.p-client_workspace > div.p-client_workspace__layout > div:nth-child(2) > div:nth-child(2) > div > div.p-file_drag_drop__container > div.p-workspace__primary_view_body > div > div:nth-child(3) > div > div > div.c-scrollbar__track > div'
            try:
                WebDriverWait(driver, 3).until(EC.presence_of_element_located((By.CSS_SELECTOR, scroll_selector)))
                scrollbar_track = driver.find_element(By.CSS_SELECTOR, scroll_selector)
                print("HTML Element located")
            except TimeoutException:
                print("HTML Element not located")    
            
            # Here different options tried to move the sidebar
            driver.execute_script("arguments[0].click();", scrollbar_track)
            logger.debug("Scroll bar Clicked")

            driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", scrollbar_track)
            driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

            scrollbar_track.send_keys(Keys.PAGE_DOWN)

            scrollbar_track.send_keys(Keys.DOWN)
            time.sleep(2)
            scrollbar_track.send_keys(Keys.DOWN)

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") -- this line should do it... can you open the page and run window.scrollTo(0, document.body.scrollHeight) in your browser console to verify it works? — ryyyn
– ryyyn, Commented Jun 20, 2024 at 2:02
Thanks for the reply - i just tried and it returns an "undefined" comment — Virrrrrr
– Virrrrrr, Commented Jun 20, 2024 at 2:07
Adding to the above, it works on this website for example but not on the webapp of Slack — Virrrrrr
– Virrrrrr, Commented Jun 20, 2024 at 2:13

ryyyn · Accepted Answer · 2024-06-20 04:37:05Z

0

I remembered I had Slack and checked out the HTML. So when an HTML document is larger than the viewport, you can use window.scrollTo() to scroll up and down. But in this case, the scrolling is happening within an HTML element (the messages area). You were on the right path trying to locate the scrollbar, but we need to find the scollable element itself and call scrollTo() on it.

It was very not obvious which element this was. I had to use the following code in dev console to bind an event listener to all elements and then scroll the messages area so I could find it:

document.querySelectorAll("*").forEach(function(div){
  // Bind each to a scroll event listener
  div.addEventListener("scroll", function(){
      console.log(this);
  });
});

(credit to Scott Marcus)

The element located:

<div data-qa="slack_kit_scrollbar" role="presentation" class="c-scrollbar__hider" tabindex="-1">

Then I constructed an XPATH to find this element. There are two scrolling panels with the same HTML signature (opening tag): one for channels/DMs and one for messages. So to get the second one, the XPATH is (//div[@data-qa="slack_kit_scrollbar"])[2].

Putting it together:

messages_scroller = driver.find_element(By.XPATH, 
                    '(//div[@data-qa="slack_kit_scrollbar"])[2]')
driver.execute_script("arguments[0].scrollTo(0, arguments[0].scrollHeight);",
                      messages_scroller)

edited Jun 20, 2024 at 4:37

answered Jun 20, 2024 at 4:28

ryyyn

6293 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Virrrrrr Over a year ago

That's perfect - it works thank you!

Collectives™ on Stack Overflow

Scroll Page in Slack chat with Selenium using Python

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related