Scraping a webpage in python with a multiple inputs

Question

I need to get data from a table in this website https://www.cashbackforex.com/en-US/tools/economic-impacts.aspx using python. The code that I wrote so far is

from bs4 import BeautifulSoup
import requests

url = 'https://www.cashbackforex.com/en-US/tools/economic-impacts.aspx'

with requests.Session() as session:
    session.headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36'}

    # parsing parameters
    response = session.get(url)
    soup = BeautifulSoup(response.content, "lxml")
    print(soup.select('input[type="button"]'))
    data = {
        'dnn$ctr1601$Chart$ddlCurrencies': 'USD',
        'dnn$ctr1601$Chart$ddlReports': 'US Change in NonFarm Payrolls',
        'dnn$ctr1601$Chart$ddlTimeZone': '(UTC) Coordinated Universal Time',
        '__EVENTTARGET': soup.find('input', {'name': '__EVENTTARGET'}).get('value', ''),
        '__EVENTARGUMENT': soup.find('input', {'name': '__EVENTARGUMENT'}).get('value', ''),
        '__VIEWSTATE': soup.find('input', {'name': '__VIEWSTATE'}).get('value', ''),
        '__VIEWSTATEGENERATOR': soup.find('input', {'name': '__VIEWSTATEGENERATOR'}).get('value', ''),
        'btnApplyTools': soup.find('input', {'id': 'btnApplyTools'}).get('value', '')
    }

    # parsing data
    response = session.post(url, data=data)

    soup = BeautifulSoup(response.content, "lxml")
    print(soup)

but every time I run the program I can't find the values in table. I think the program doesn't send the input values to server, but I'm not sure.

The following table:

What request are you trying to mimic? When I click the red "Apply" button I don't see a POST to economic-impacts.aspx in the browser developer tools.. — alecxe
– alecxe, Commented Jan 3, 2017 at 5:36
When I change a variable, i.e. USD, US Change in Nonfarm Payroll and hit the apply bottom. The Table called "Event History" below it, changes. I need to extract that table. But When I read that table with BeautifulSoup, non of those rows are present. — Milad Resketi
– Milad Resketi, Commented Jan 3, 2017 at 7:29

Andersson · Accepted Answer · 2017-01-03 09:14:39Z

3

I checked provided page and found that there is no need in Session() and sending multiple parameters to get required table. All you need is to specify inst parameter (something like identifier of filter) and timezone. For example, for USD/US Change in NonFarm Payrolls inst parameter value is 10332295, timezone for (UTC) Coordinated Universal Time is 3.

So your request should be something like

params = {'inst': '10332295', 'timezone': '3'}
response = requests.get('https://www.cashbackforex.com/DesktopModules/Chart/HistoricalEventFigures.ashx', params=params)

And then you can parse response in convenient way, e.g.:

from xml.dom import minidom

xml = minidom.parseString(response.text)
print([i.childNodes[0].wholeText for i in xml.getElementsByTagName("Date")])
print([i.childNodes[0].wholeText for i in xml.getElementsByTagName("ReportName")])
...

The output:

['2 Dec 2016', '4 Nov 2016', '7 Oct 2016', '2 Sep 2016', '5 Aug 2016', '8 Jul 2016', '3 Jun 2016',...]
['US Change in NonFarm Payrolls', 'US Change in NonFarm Payrolls', 'US Change in NonFarm Payrolls', 'US Change in NonFarm Payrolls',...]

edited Jan 3, 2017 at 9:14

answered Jan 3, 2017 at 8:59

Andersson

52.8k18 gold badges83 silver badges132 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Milad Resketi Over a year ago

Thanks for your answer. Your code partially works, But I need the program to go through all the dropdown lists. so i can get all of them. For example, it needs to start from AUD find all the events starting from AU Bank Holiday - ANZAC Day to the end. How did you find the "cashbackforex.com/DesktopModules/Chart/…" ?

Andersson Over a year ago

When you click Apply button, you send 3 GET requests: https://www.cashbackforex.com/DesktopModules/Chart/Candles.ashx for data displayed on chart, https://www.cashbackforex.com/DesktopModules/Chart/SameTimeReports.ashx for first table and https://www.cashbackforex.com/DesktopModules/Chart/HistoricalEventFigures.ashx for second (Event History) table.

Andersson Over a year ago

Each pair of Event finder like AUD-AU Bank Holiday - ANZAC has its own inst parameter, but I don't know what exact range of inst values you should apply for AUD pairs as they seem to be not sequential...

Collectives™ on Stack Overflow

Scraping a webpage in python with a multiple inputs

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related