Web scraping using Click button

Question

Team, I am trying to click on load more button, I'm able to click and run the macro with no issues for only one click. that is for one time. I need help for the below points

I'm trying to automate the code to repeat the click button until the page loads all the data for web scraping.
Also, i need a code to check the load more button in the web page exists or not before web scraping the data to excel. If "Load more" button not found proceed with the next code. (FYI Load more exists at the bottom of my web page).

Thanks and please reply me if my question is not clear.

below is the Html code before clicking load more button

<button type="button" class="btn primary btn-primary modal-button-print add-notes" data-bind="click: getNotes, visible: isLoadMoreButtonEnable() &amp;&amp; !$root.providerShouldAcceptDecline()">
  <i class="fa fa-refresh" aria-hidden="true"></i>Load More
</button>

below is the Html code after clicking load more button multiple times until it loads full data

<button class="btn primary btn-primary modal-button-print add-notes" style="display: none;" type="button" data-bind="click: getWoNotes, visible: isLoadMoreNotesButtonEnable() &amp;&amp; !$root.providerShouldAcceptDecline()">
                <i class="fa fa-refresh" aria-hidden="true"></i>Load More Work Order Notes
            </button>

The difference i see from the above html code is style="display: none;" got added after i click button multiple times till it loads full data in web page.

I have a sample web site that looks similar to my webpage. I took this link here to just show how the page loads in my website.

Sub abc()

  
Set IE = New InternetExplorer
Link = my url
.
.
.
.
For L = 2 To Lr1
    IE.navigate Link 
    Set Html = New MSHTML.HTMLDocument
    Set Ws = Scraping
    Do
    DoEvents: Loop Until IE.readyState = READYSTATE_COMPLETE
    Application.Wait (Now + TimeValue("00:00:05"))
    IE.document.querySelector("button[type=button]").Click
    Do
    DoEvents: Loop Until IE.readyState = READYSTATE_COMPLETE
    Application.Wait (Now + TimeValue("00:00:05"))
    IE.document.querySelector("button[type=button]").Click
    Do
    DoEvents: Loop Until IE.readyState = READYSTATE_COMPLETE
    Application.Wait (Now + TimeValue("00:00:05"))
    IE.document.querySelector("button[type=button]").Click
    Do
    DoEvents: Loop Until IE.readyState = READYSTATE_COMPLETE
    Application.Wait (Now + TimeValue("00:00:05"))

    Html.body.innerHTML = IE.document.querySelectorAll(".list").Item(1).outerHTML
    Set Tariku = Html.querySelectorAll(".columns")
    Set data = Html.querySelectorAll(".datalist")
        With Ws

        ' Do all the stuff  

        End With
        IE.document.querySelector("#Logout").Click
        IE.Quit
       Exit Sub
    
  Next L

End Sub

Two questions: 1. Did you check if the page works in IE when you click the button? I recently wanted to automate a load more button, but the script of the page that was stored for the button did not work in IE. 2. Is the button the only one on the page or at least the first one in the HTML source code? You use querySelector(), which addresses exactly the first button found. — Zwenn
– Zwenn, Commented Jun 1, 2020 at 17:49
@Zwenn, 1. Yes currently I'm using IE and it's working fine when I click the button... 2. That is the only button on the page. — TCritical
– TCritical, Commented Jun 1, 2020 at 18:58
This button will be available when the data is more in the particular page. Some cases you will not find the load more button since it is having limited data which fits exactly in the page... — TCritical
– TCritical, Commented Jun 1, 2020 at 19:05

Zwenn · Accepted Answer · 2020-06-02 09:08:21Z

1

You can try this. Is it possible for you to post the URL if it don't work?

Sub Abc()

Dim browser As Object
Dim url As String
Dim nodeButton As Object
Dim noButtonFound As Boolean

  url = "Your URL here"

  'Initialize Internet Explorer, set visibility,
  'call URL and wait until page is fully loaded
  Set browser = CreateObject("internetexplorer.application")
  browser.Visible = False
  browser.navigate url
  Do Until browser.ReadyState = 4: DoEvents: Loop

  'Click button as often as found
  Do
    'Try to catch button
    Set nodeButton = browser.document.getElementsByTagName("button")(0)

    'Check if button was found
    If Not nodeButton Is Nothing Then
      'Check if it has an style attribute
      If nodeButton.hasAttribute("style") Then
        'Check if button is visible
        If nodeButton.getAttribute("style") <> "display: none;" Then
          'Click button
          nodeButton.Click

          'Wait for load more data
          Application.Wait (Now + TimeSerial(0, 0, 5))
        End If
          'No visible button found, leave loop
          noButtonFound = True
        End If
      Else
        'No visible button found, leave loop
        noButtonFound = True
      End If
    Else
      'No visible button found, leave loop
      noButtonFound = True
    End If
  Loop Until noButtonFound

  'All dynamic data was load
  'Do here what ever you want
  'But I think you don't need a new html document
End Sub

edited Jun 2, 2020 at 9:08

answered Jun 1, 2020 at 20:22

Zwenn

2,2672 gold badges10 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

TCritical Over a year ago

Zwenn, thanks for your reply.. code works fine, but it is not looping until load button not found. Sorry, that i cant provide url. I have mentioned the html code after the load more button is clicked until it loads the full data in web page. I think this will help to complete the code.

TCritical Over a year ago

this is the code (style="display: none; ") got added after clicking the button multiple times to load the page full.

Zwenn Over a year ago

@RAJSHA If I understood it correctly, the button is always there. It is only hidden when all data has been loaded. Therefore the loop is never left. I have extended the loop in my answer. It now queries for the style attribute and its value.

TCritical Over a year ago

Zwenn, The lines ("If nodeButton.getAttribute("style") <> "display: none;" Then nodeButton.Click") is not executing. it is getting skipped and the if condition is going infinite loop..

Zwenn Over a year ago

@RAJSHA I saw it and expanded it again a few minutes ago.

|

Collectives™ on Stack Overflow

Web scraping using Click button

1 Answer 1

9 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Related