php curl data scraping

Question

I have a CURL code to fetch data from a site it is working fine for last few months but suddenly stop working for me it says

HTTP/1.0 302 Moved Temporarily

my code is:

$ch = curl_init();
curl_setopt($ch, CURLOPT_REFERER, $baseUrl);
curl_setopt($ch, CURLOPT_PROXY, $proxy[0]);
curl_setopt($ch, CURLOPT_PROXYPORT, $proxy[1]);
//curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIE         , $phpSId);
curl_setopt($ch, CURLOPT_COOKIEJAR      , $cookie);
curl_setopt($ch, CURLOPT_COOKIEFILE     , $cookie);
curl_setopt($ch, CURLOPT_USERAGENT      , "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:7.0.1) Gecko/20100101 Firefox/7.0.1");
curl_setopt($ch, CURLOPT_TIMEOUT        , 40);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST , 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER , 0);
curl_setopt($ch, CURLOPT_URL            , $url);
curl_setopt($ch, CURLOPT_HEADER         , 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION , 1);
curl_setopt($ch, CURLOPT_POST           , 1);
curl_setopt($ch, CURLOPT_POSTFIELDS     , $data);
$result = curl_exec($ch);
curl_close ($ch);
unset($ch); 
die($result);

Please help, thanks in advance

What is your question? How are we supposed to know what changed on the server you're scraping? Can you give us the URL you're polling? — middus
– middus, Commented Oct 24, 2011 at 19:59
Why don't you give us an example URL so we can check for ourselves? — phihag
– phihag, Commented Oct 24, 2011 at 20:01
yeah, that's a redirect to another URL, so, follow it in your browser, and replace the URL you were trying with the new one — Robot Woods
– Robot Woods, Commented Oct 24, 2011 at 20:01
they have blocked my ip and i don't know how they blocked proxy servers, when i am using dynamic proxy server they redirect me to a page with error "group not allowed", how we can block proxies on a php server? — GauravSTomar
– GauravSTomar, Commented Oct 27, 2011 at 5:02

phihag · Accepted Answer · 2011-10-24 20:04:17Z

1

The specified options already make curl follow redirects. However, in the case of a long redirect chain, you may want to increase CURLOPT_MAXREDIRS.

You can use a packet dumper such as wireshark to check which requests are sent by curl. It may be simply a bug in the scraped website which causes it to redirect infinitely.

answered Oct 24, 2011 at 20:04

phihag

289k75 gold badges475 silver badges489 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

steampowered Over a year ago

@gauravstomar you can analyze the redirects a little faster with a web browser compared to wireshark if you are not familiar with wireshark. Just look at the http headers to see how many redirects are occuring. Then set CURLOPT_MAXREDIRS appropriatly

phihag Over a year ago

@JM. There may be some specific anti-curl or JavaScript-required trickery involved. Setting CURLOPT_MAXREDIRS to, say, 1000, should be more than any browser accepts.

GauravSTomar Over a year ago

actually they have blocked proxy access to there website, how it is possible?

phihag Over a year ago

@gauravstomar Some proxies set an X-Forwarded-For header, and there are public proxy datbases. Are you certain that the proxy is the problem, and not curl? Oftentimes, changing the User-Agent header is enough to evade such silly blockades or misconfigurations.

phihag Over a year ago

@gauravstomar php is not really the best way to set up a proxy, although there are existing implementations. Basically, you use curl to set up a new HTTP request. However, it's way easier not to use php and use a regular proxy such as squid or apache with mod_proxy.

|

Collectives™ on Stack Overflow

php curl data scraping

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related