API calls return JSON on my local machine but return HTML on Heroku

Question

I have a Discord bot that makes some API calls. When I run the bot from my computer, it all works fine, however when I push my code to Heroku (where I am hosting my bot) the API call returns this error:

UnhandledPromiseRejectionWarning: FetchError: invalid json response body at https://sky.shiiyu.moe/api/v2/profile/SirArchibald97 reason: Unexpected token < in JSON at position 0
2021-01-07T11:45:02.931529+00:00 app[worker.1]:     at /app/node_modules/node-fetch/lib/index.js:272:32
2021-01-07T11:45:02.931532+00:00 app[worker.1]:     at processTicksAndRejections (internal/process/task_queues.js:97:5)
2021-01-07T11:45:02.931532+00:00 app[worker.1]:     at async Object.execute (/app/methods/link.js:93:20)
2021-01-07T11:45:02.931532+00:00 app[worker.1]:     at async Client.<anonymous> (/app/index.js:47:28)
2021-01-07T11:45:02.931832+00:00 app[worker.1]: (node:4) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
2021-01-07T11:45:02.931970+00:00 app[worker.1]: (node:4) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Edit: I am requesting two different APIs in my code, using the node-fetch npm module. This is the one which returns the error:

let res = await fetch(`https://sky.shiiyu.moe/api/v2/profile/${player.displayname}`);
let data = await res.json();

And this is the one which works fine:

let p_response = await fetch(`https://api.hypixel.net/player?key=${api_key}&name=${args[1]}`);
let { player } = await p_response.json();

Edit 2: I have switched from node-fetch to axios in the hope that it might work, and I am now recieving a 503 status error instead. Once again, it works fine on my machine, but returns the error when I deploy to Heroku. Here is the new code:

const axios = require("axios");
let res = await axios.get(`https://sky.shiiyu.moe/api/v2/profile/${player.displayname}`);
let data = res.data;

invalid json response body at https://sky.shiiyu.moe/api/v2/profile/SirArchibald97 reason: Unexpected token < in JSON at position 0—it looks like you're getting HTML or XML or something instead of JSON. Take a look at the response without trying to parse it as JSON. It may be an error page or something. Also, check the HTTP response code you get. — Chris
– Chris, Commented Jan 7, 2021 at 20:59
I know it’s returning HTML, but I’m not sure why, as the request returns JSON when I run the bot from my system. I have logged the response to the console and it returns the same thing as the JSON but as one long string of text rather than an object. — SirArchibald
– SirArchibald, Commented Jan 8, 2021 at 8:35
Can you provide a minimal reproducible example of an API request? Are you setting an accept header? — Chris
– Chris, Commented Jan 8, 2021 at 13:27
@Chris I have just edited my question to include the examples you wanted. And no, I am not sending an accept header as I am not entirely sure how. — SirArchibald
– SirArchibald, Commented Jan 8, 2021 at 13:38
have you checked which request headers are being set in both instances? — QHarr
– QHarr, Commented Jan 9, 2021 at 23:45

Abkarino · Accepted Answer · 2021-01-16 13:23:39Z

1

+150

I checked the API you are using, using CURL.

curl "https://sky.shiiyu.moe/api/v2/profile/SirArchibald97"

HTTP/1.1 503 Service Temporarily Unavailable
Date: Sat, 16 Jan 2021 13:27:30 GMT
Content-Type: text/html; charset=UTF-8
Connection: close
Set-Cookie: __cfduid=dfef31ed9a3326383a42315e646837f531610803650; expires=Mon, 15-Feb-21 13:27:30 GMT; path=/; domain=.shiiyu.moe; HttpOnly; SameSite=Lax; Secure
X-Frame-Options: SAMEORIGIN
Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Thu, 01 Jan 1970 00:00:01 GMT
cf-request-id: 07acfa3991000041f79a8d8000000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=DfiyBY1Zc4DC%2Bxzh2Ug61Brj2X5zQ4Ae5VlhYj8Ci7bRKlaJGGTO9PFZuXqXYyOEIiWKdM2n8ykJlwf3Y79zCUxIire2LUy%2BzAtDGAxqcg%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 61282ca2893341f7-MRS

I got an HTML response. It was CloudFlare protection. It is probably working in your local machine due to CloudFlare cookies.

This is a issue with the API configuration. You need to reach them and ask to whitelist you IP or you would have to use some sort of browser emulator like selenium to bypass CloudFlare firewall.

answered Jan 16, 2021 at 13:23

Abkarino

1,4461 gold badge12 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

SirArchibald Over a year ago

Ahh ok, thank you for figuring it out. I have been stuck with this issue for a while.

Abkarino Over a year ago

Great, glad to help.

AnupamChugh Over a year ago

Thanks @Abkarino for the direction. I’m facing a similar Cloudflare. It shows a captcha page when i do a GET reques to a website from Zapier. Quick question: By “you need to reach them” you mean whom? Cloudflare?

Abkarino Over a year ago

@AnupamChugh you need to contact whom setup cloudflare on the host. If it is automatic, they you should find a place to setup whitelist in your Heroku dashboard or contact Heroku customer support. Zapier has an ip range which you are told to use for whitelisting.

Alexandr Tovmach · Accepted Answer · 2021-01-10 09:28:45Z

0

Sending request with default axios config works well, so your issue related to:

axios config
heroku config

const player = { displayname: 'voice' };

const sendReq = async () => {
  let res = await axios.get(`https://sky.shiiyu.moe/api/v2/profile/${player.displayname}`);
  console.log(res.data)
}

<script src="https://cdnjs.cloudflare.com/ajax/libs/axios/0.21.1/axios.min.js"></script>
<button onclick="sendReq()">Send request</button>

answered Jan 10, 2021 at 9:28

Alexandr Tovmach

3,1791 gold badge18 silver badges34 bronze badges

Comments

LastM4N · Accepted Answer · 2021-01-13 12:06:16Z

0

I hope the workaround here solves the issue GET request returns index.html doc instead of json data

Let me know!

edited Jan 13, 2021 at 12:06

answered Jan 13, 2021 at 11:53

LastM4N

2,2502 gold badges19 silver badges24 bronze badges

Collectives™ on Stack Overflow

API calls return JSON on my local machine but return HTML on Heroku

3 Answers 3

4 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related