Get Table after Button Click BeautifulSoup Python

Question

I am trying to scrape a table from the following site :

    http://www.boursakuwait.com.kw/Stock/StkHData.aspx?STK=103

This is the following code that am using

    ap = argparse.ArgumentParser()

    ap.add_argument("--id",required = True, help = "enter the id")
    ap.add_argument("--from",required = True, help = "From date")
    ap.add_argument("--to",required = True, help = "To date")

    args = vars(ap.parse_args())

    id_num = args["id"]
    from_date = args["from"]
    to_date = args["to"]

    browser = webdriver.Firefox()
    url1 = "http://www.boursakuwait.com.kw/Stock/StkHData.aspx?STK=" + id_num

    browser.get(url1)

    elem = browser.find_element_by_id("ContentMatter_txtFrom")
    elem.send_keys(from_date)
    elem = browser.find_element_by_id("ContentMatter_txtTo")
    elem.send_keys(to_date)
    elem = browser.find_element_by_id("ContentMatter_Button1").click()

    time.sleep(5)

    parsed = requests.get(url1)

    soup = BeautifulSoup(parsed.text,'html.parser')
    tble = soup.findAll("table",attrs={"class":"hoverTable"})

    print(tble)

After entering the from and to date, and simulating the button click, my goal is obtain the resulting table. However, the code does not recognize the table. I have tried other forms of the code as well (looping through rows and cols), but they haven't worked either.

My guess is that the url doesn't get updated, since it is a hoverTable. Is there a way to obtain the table in this case?

P.S: the id_num are integers, 101,102,103 etc. displaying the respective stocks of the companies. The link pasted comes with id 103. STK=103

Murthi · Accepted Answer · 2017-06-15 11:16:41Z

1

you try with the following code.

soup.select("table.hoverTable") //to select the table

answered Jun 15, 2017 at 11:16

Murthi

5,3471 gold badge12 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Lakshmi Narayanan Over a year ago

this worked as well. the real issue however, was with the requests. I was able to solve it by using "browser.page_source". Since the browser object carried the updated table, and the requests did not.

Lakshmi Narayanan · Accepted Answer · 2017-06-15 11:26:30Z

1

I was able to solve it, by using the browser object. Replacing

    parsed = requests.get(url1)
    soup = BeautifulSoup(parsed.text,'html.parser')

with

    parsed = browser.page_source
    soup = BeautifulSoup(parsed,'html.parser')

since the browser object carried the state of the url from the beginning instead of the requests used, in between.

A easier way to get the table was suggested by Murthy

    tble = soup.select("table.hoverTable")

The original code works as well.

answered Jun 15, 2017 at 11:26

Lakshmi Narayanan

5,36213 gold badges55 silver badges97 bronze badges

1 Comment

JeffC Over a year ago

Please don't forget to come back and accept this as the answer when you are allowed. Thanks!

Collectives™ on Stack Overflow

Get Table after Button Click BeautifulSoup Python

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related