how to extract a chart that gets it data from an API using python

Question

I am trying to get the data from this website, https://en.macromicro.me/charts/947/commodity-ccfi-scfi ,

I understand that the data is called from an API, how do I find out how the call is made and how do I extract it using python?

I am new to python and html in general so I have no idea where to start.

I tried,

import requests
from bs4 import BeautifulSoup
import json
import pandas as pd
urlheader = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36",
    "X-Requested-With": "XMLHttpRequest",
    "Accept": "application/json, text/javascript, */*; q=0.01",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-US,en;q=0.9",
    "Authorization": "Bearer 640eabc473294fbac27930ef08d28ab4",
    "Connection": "keep-alive",
    "Cookie": "PHPSESSID=5q99iuiarvf1ba2lafh6je5hr5; _ga=GA1.2.628840989.1624431403; _gid=GA1.2.146418174.1624431403; _fbp=fb.1.1624431403269.1337227854; _hjTLDTest=1; _hjid=89fd1c1b-93a7-4da9-bf90-46efcb6aae15; _hjFirstSeen=1",
    "DNT": "1",
    "Host": "en.macromicro.me",
    "Referer": "https://en.macromicro.me/charts/947/commodity-ccfi-scfi",
}

url = "https://en.macromicro.me/charts/data/947"



req = requests.post(url,headers=urlheader)
soup = BeautifulSoup(req.content, "lxml")
print(soup)

But I get the following error

<html><body><p>{"status":"Method Not Allowed","code":405,"text":"HTTP 405 (POST \/charts\/data\/947)","level":0}</p></body></html>

Ram · Accepted Answer · 2021-06-23 07:17:46Z

1

You can make a GET request to the API using requests and convert the response to json.

resp has the data in json format and you can easily extract the info that you need.

import requests

url = "https://en.macromicro.me/charts/data/947"

resp = requests.get(url)
resp = resp.json()

edited Jun 23, 2021 at 7:17

answered Jun 23, 2021 at 7:12

Ram

4,7892 gold badges17 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

anarchy Over a year ago

hey it doesnt work, i get an error, i updated my question

anarchy Over a year ago

your code gives this error {'status': 'Method Not Allowed', 'code': 405, 'text': 'HTTP 405 (POST /charts/data/947)', 'level': 0}

Ram Over a year ago

Why are you making a POST request ? My code and your code are completely different. How would you expect it to work ?

anarchy Over a year ago

hey @Ram I dont get it, sometimes i get this {'success': 0, 'data': [], 'msg': 'error #244'} sometimes it works, sometimes i get <bound method Response.json of <Response [200]>>

anarchy Over a year ago

is the website blocking me?

Collectives™ on Stack Overflow

how to extract a chart that gets it data from an API using python

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related