Need to scrape a table which is loaded through ajax using python(selenium)

Question

I have a page that has a table (table id= "ctl00_ContentPlaceHolder_ctl00_ctl00_GV" class="GridListings" )i need to scrape. I usually use BeautifulSoup & urllib for it,but in this case the problem is that the table takes some time to load ,so it isnt captured when i try to fetch it using BS. I cannot use PyQt4,drysracpe or windmill because of some installation issues,so the only possible way is to use Selenium/PhantomJS I tried the following,still no success:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.PhantomJS()
driver.get(url)
wait = WebDriverWait(driver, 10)
table = wait.until(EC.presence_of_element_located(By.CSS_SELECTOR, 'table#ctl00_ContentPlaceHolder_ctl00_ctl00_GV'))

The above code doesnt give me the desired contents of the table. How do i go about achieveing this???

What do you mean doesnt give me the desired contents?? Is there any exception?? And Could you share your table HTML here as well?? — Saurabh Gaur
– Saurabh Gaur, Commented Aug 4, 2016 at 17:23
Hi,link the bottom table is what i want(table with class="GridListings") — user3425403
– user3425403, Commented Aug 4, 2016 at 17:27
I'm sorry but I'm not able to view table this time that's why I'm telling you share here the table HTML.. — Saurabh Gaur
– Saurabh Gaur, Commented Aug 4, 2016 at 17:30

Padraic Cunningham · Accepted Answer · 2016-08-04 19:21:25Z

2

You can get the data using requests and bs4,, with almost if not all asp sites there are a few post params that always need to be provided like __EVENTTARGET, __EVENTVALIDATION etc.. :

from bs4 import BeautifulSoup
import requests

data = {"__EVENTTARGET": "ctl00$ContentPlaceHolder$ctl00$ctl00$RadAjaxPanel_GV",
    "__EVENTARGUMENT": "LISTINGS;0",
    "ctl00$ContentPlaceHolder$ctl00$ctl00$ctl00$hdnProductID": "139",
    "ctl00$ContentPlaceHolder$ctl00$ctl00$hdnProductID": "139",
    "ctl00$ContentPlaceHolder$ctl00$ctl00$drpSortField": "Listing Number",
    "ctl00$ContentPlaceHolder$ctl00$ctl00$drpSortDirection": "A-Z, Low-High",
    "__ASYNCPOST": "true"}

And for the actual post, we need to add a few more values to out post data:

post = "https://seahawks.strmarketplace.com/Charter-Seat-Licenses/Charter-Seat-Licenses.aspx"
with requests.Session() as s:
    s.headers.update({"User-Agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:47.0) Gecko/20100101 Firefox/47.0"})
    soup = BeautifulSoup(s.get(post).content)

    data["__VIEWSTATEGENERATOR"] = soup.select_one("#__VIEWSTATEGENERATOR")["value"]
    data["__EVENTVALIDATION"] = soup.select_one("#__EVENTVALIDATION")["value"]
    data["__VIEWSTATE"] = soup.select_one("#__VIEWSTATE")["value"]

    r = s.post(post, data=data)
    soup2 = BeautifulSoup(r.content)
    table = soup2.select_one("div.GridListings")
    print(table)

You will see the table printed when you run the code.

edited Aug 4, 2016 at 19:21

answered Aug 4, 2016 at 19:12

Padraic Cunningham

181k30 gold badges264 silver badges327 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user3425403 Over a year ago

You are absolutely right!!!!! Ican see the table now.Thanks a lot!!!I have a doubt though,since we are statically posting few params to the page,there is no chance of the same page being downloaded right?I mean the table is going to be the updated one every time we run this right?

Padraic Cunningham Over a year ago

@user3425403, yes, we are only really doing the required validation, there are also lot more params you can pass, I have not looked through them all but you should be able to influence how the data is returned by adding more.

user3425403 Over a year ago

,is there a way to identify which are the required form data's?

A. STEFANI · Accepted Answer · 2016-08-04 19:26:28Z

0

If you want to scrap something, it will be nice first to install a web debugger ( Firebug for Mozilla Firefox for example) to watch how the website you want to scrap is working.

Next, you need to copy the process of how the website is connecting to backoffice

As you said, the content that you want to scrap is being loaded asynchronously (only when the document is ready)

Assuming the debugger is running and also you have refreshed the page, you will see on the network tab the following request:

POST https://seahawks.strmarketplace.com/Charter-Seat-Licenses/Charter-Seat-Licenses.aspx

The final process flow to reach your goal will be:

1/ Use requests python module
2/ Open a requests session to the index page website site (with cookies handling)
3/ Scrap all the input for the specific POST form request
4/ Build a POST parameter DICT containing all inputs & value fields scrapped in the previous step + adding some specific fixed params.
5/ POST the request (with required data)
6/ Use finally BS4 module (as usual) to soup the answered html to scrap your data

Please see bellow a working code:

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

from bs4 import BeautifulSoup
import requests

base_url="https://seahawks.strmarketplace.com/Charter-Seat-Licenses/Charter-Seat-Licenses.aspx"

#create requests session
s = requests.session()

#get index page
r=s.get(base_url)

#soup page
bs=BeautifulSoup(r.text)

#extract FORM html
form_soup= bs.find('form',{'name':'aspnetForm'})

#extracting all inputs
input_div = form_soup.findAll("input")

#build the data parameters for POST request
#we add some required <fixed> data parameters for post
data={
    '__EVENTARGUMENT':'LISTINGS;0',
    '__EVENTTARGET':'ctl00$ContentPlaceHolder$ctl00$ctl00$RadAjaxPanel_GV',
    '__EVENTVALIDATION':'/wEWGwKis6fzCQLDnJnSDwLq4+CbDwK9jryHBQLrmcucCgL56enHAwLRrPHhCgKDk6P+CwL1/aWtDQLm0q+gCALRvI2QDAKch7HjBAKWqJHWBAKil5XsDQK58IbPAwLO3dKwCwL6uJOtBgLYnd3qBgKyp7zmBAKQyTBQK9qYAXAoieq54JAuG/rDkC1djKyQMC1qnUtgoC0OjaygUCv4b7sAhfkEODRvsa3noPfz2kMsxhAwlX3Q=='
}
#we add some <dynamic> data parameters
for input_d in input_div:
    try:
        data[ input_d['name'] ] =input_d['value'] 
    except:
        pass #skip unused input field

#post request
r2=s.post(base_url,data=data)

#write the result
with open("post_result.html","w") as f:
    f.write(r2.text.encode('utf8'))

Now, please get a look at "post_result.html" content and you will find the data !

Regards

edited Aug 4, 2016 at 19:26

answered Aug 4, 2016 at 17:26

A. STEFANI

6,7711 gold badge25 silver badges49 bronze badges

7 Comments

user3425403 Over a year ago

Did u check the post paramaters to the page??I dont think thats gonna work.You will understand if u had looked at the post request

A. STEFANI Over a year ago

Year, that is why I added step 3 & 4 in the process, I will update my answer with a working code in the next 5 min.

user3425403 Over a year ago

Thanks for the effort,but still that table is not captured : look at the output file,the table would come up inside the innerdiv below,which is still empty <div class="GridListings"> <div> </div> </div>

A. STEFANI Over a year ago

Yes, you are right that the table div is empty :-) But there is more data visible in the document (we can see the section for example). We maybe need to add fixed headers an/or some data params to the POST request

A. STEFANI Over a year ago

Code was updated : adding __EVENTARGUMENT, __EVENTTARGET, __EVENTVALIDATION are required data params

|

Collectives™ on Stack Overflow

Need to scrape a table which is loaded through ajax using python(selenium)

2 Answers 2

3 Comments

7 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

7 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related