But when I am running my code it Hangs and there is no response. Here is my code. Thanks for your help.
import json, requests, time
url="http://www.indiegogo.com/private_api/campaigns/the-middleman-the-crowd-funded-franchise-resurrection/pledges"
r=requests.get(url)
print(r.text)
solved by adding User-Agent to headers
import requests
url = ("https://www.indiegogo.com/private_api/campaigns/"
"the-middleman-the-crowd-funded-franchise-resurrection/pledges")
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) '
'AppleWebKit/537.36 (KHTML, like Gecko) '
'Chrome/39.0.2171.95 Safari/537.36'}
r = requests.get(url, headers=headers)
print(r.text)
gives us
<!DOCTYPE html>
<head>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
<meta http-equiv="cache-control" content="max-age=0" />
<meta http-equiv="cache-control" content="no-cache" />
<meta http-equiv="expires" content="0" />
<meta http-equiv="expires" content="Tue, 01 Jan 1980 1:00:00 GMT" />
<meta http-equiv="pragma" content="no-cache" />
<meta http-equiv="refresh" content="10; url=/distil_r_blocked.html?requestId=c556d45a-825c-463b-ad6a-dace0f6ab5c9&httpReferrer=%2Fprivate_api%2Fcampaigns%2Fthe-middleman-the-crowd-funded-franchise-resurrection%2Fpledges" />
<script type="text/javascript">
(function(window){
try {
if (typeof sessionStorage !== 'undefined'){
sessionStorage.setItem('distil_referrer', document.referrer);
}
} catch (e){}
})(window);
</script>
<script type="text/javascript" src="/gogodstlywuvxyesewzaxdveubcvzbbwtuvuxv.js" defer></script><style type="text/css">#d__fFH{position:absolute;top:-5000px;left:-5000px}#d__fF{font-family:serif;font-size:200px;visibility:hidden}#zxtwystcxcbebz{display:none!important}</style></head>
<body>
<div id="distilIdentificationBlock"> </div>
</body>
</html>
or it is not what you was looking for?
It looks like this site rejects requests from bots and creates infinite loops.
A possible workaround is to use selenium package with ChromeDriver:
import os
from contextlib import closing
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions
from selenium.webdriver.support.wait import WebDriverWait
url = ("https://www.indiegogo.com/private_api/campaigns/"
"the-middleman-the-crowd-funded-franchise-resurrection/pledges")
# here we are assuming that driver binary located in current directory
# (path to driver should be absolute),
# change if it differs
chrome_driver_path = os.path.abspath('chromedriver')
xpath = '/html/body/pre'
with closing(webdriver.Chrome(chrome_driver_path)) as web_driver:
web_driver.get(url)
web_driver_wait = WebDriverWait(web_driver, timeout=5)
locator = (By.XPATH, xpath)
web_driver_wait.until(expected_conditions
.presence_of_element_located(locator))
text = web_driver.find_element_by_xpath(xpath).text
print(text)
gives us
{"response": [{"pledger_display_name": "bobreturns",
"pledger_profile_url": "/individuals/5355846",
"time_ago": "over 3 years ago",
"pledger_image_url": "https://g1.iggcdn.com/assets/individuals/missing/cubepeep-3c70b9b8bef12ca70a38d32e7c635576ae8a66bc7cfff10612ba6876fd7c68c7.png",
"display_amount": "$20", "display_amount_iso_code": "USD",
"pledge_details_url": null, "order_id": null},
{"pledger_display_name": "Russell Auer",
"pledger_profile_url": "/individuals/1167704",
"time_ago": "over 3 years ago",
"pledger_image_url": "https://g2.iggcdn.com/assets/individuals/missing/cubepeep-3c70b9b8bef12ca70a38d32e7c635576ae8a66bc7cfff10612ba6876fd7c68c7.png",
"display_amount": "$15", "display_amount_iso_code": "USD",
"pledge_details_url": null, "order_id": null},
{"pledger_display_name": "Christopher Kindred",
"pledger_profile_url": "/individuals/318158",
"time_ago": "over 3 years ago",
"pledger_image_url": "https://g0.iggcdn.com/assets/individuals/missing/cubepeep-3c70b9b8bef12ca70a38d32e7c635576ae8a66bc7cfff10612ba6876fd7c68c7.png",
"display_amount": "$50", "display_amount_iso_code": "USD",
"pledge_details_url": null, "order_id": null},
{"pledger_display_name": "lukemjohnsonusd",
"pledger_profile_url": null, "time_ago": "over 3 years ago",
"pledger_image_url": "https://g1.iggcdn.com/assets/individuals/missing/cubepeep-3c70b9b8bef12ca70a38d32e7c635576ae8a66bc7cfff10612ba6876fd7c68c7.png",
"display_amount": "$15", "display_amount_iso_code": "USD",
"pledge_details_url": null, "order_id": null},
{"pledger_display_name": "Kristian Bjørkelo",
"pledger_profile_url": "/individuals/1330533",
"time_ago": "over 3 years ago",
"pledger_image_url": "https://c1.iggcdn.com/indiegogo-media-prod-cld/image/upload/c_fill,f_auto,h_50,w_50/v1432588431/yj6ruqvreu3hmbpqdvej.jpg",
"display_amount": "$15", "display_amount_iso_code": "USD",
"pledge_details_url": null, "order_id": null},
{"pledger_display_name": "grandmoffdavid",
"pledger_profile_url": null, "time_ago": "over 3 years ago",
"pledger_image_url": "https://g2.iggcdn.com/assets/individuals/missing/cubepeep-3c70b9b8bef12ca70a38d32e7c635576ae8a66bc7cfff10612ba6876fd7c68c7.png",
"display_amount": "$15", "display_amount_iso_code": "USD",
"pledge_details_url": null, "order_id": null},
{"pledger_display_name": "Anonymous",
"pledger_profile_url": null, "time_ago": "over 3 years ago",
"pledger_image_url": "https://g0.iggcdn.com/assets/individuals/missing/cubepeep-3c70b9b8bef12ca70a38d32e7c635576ae8a66bc7cfff10612ba6876fd7c68c7.png",
"display_amount": "$30", "display_amount_iso_code": "USD",
"pledge_details_url": null, "order_id": null},
{"pledger_display_name": "Craig Shemin",
"pledger_profile_url": "/individuals/4550201",
"time_ago": "over 3 years ago",
"pledger_image_url": "https://g1.iggcdn.com/assets/individuals/missing/cubepeep-3c70b9b8bef12ca70a38d32e7c635576ae8a66bc7cfff10612ba6876fd7c68c7.png",
"display_amount": "$20", "display_amount_iso_code": "USD",
"pledge_details_url": null, "order_id": null},
{"pledger_display_name": "@Mech_Turtle",
"pledger_profile_url": null, "time_ago": "over 3 years ago",
"pledger_image_url": "https://g2.iggcdn.com/assets/individuals/missing/cubepeep-3c70b9b8bef12ca70a38d32e7c635576ae8a66bc7cfff10612ba6876fd7c68c7.png",
"display_amount": null, "display_amount_iso_code": null,
"pledge_details_url": null, "order_id": null},
{"pledger_display_name": "venusdemolly",
"pledger_profile_url": null, "time_ago": "over 3 years ago",
"pledger_image_url": "https://g0.iggcdn.com/assets/individuals/missing/cubepeep-3c70b9b8bef12ca70a38d32e7c635576ae8a66bc7cfff10612ba6876fd7c68c7.png",
"display_amount": "$50", "display_amount_iso_code": "USD",
"pledge_details_url": null, "order_id": null}], "count": 10,
"pagination": {"previous": null, "next": 2, "current": 1, "per_page": 10,
"count": 1340, "pages": 134}}
It didn't work for PhantomJS driver and ChromeDriver instantiated with --headless option
closing function itself you can read at the docs