receive csv from server and parse using php

Question

this was working up until quite recently and i cannot seem to crack the case.

if you manually visit the url hit against in the script, the results are there..but if i do it in the code, i am having an issue.

you can see in my output test that i am no longer getting any output...

any ideas?

                <?
                //$ticker=urldecode($_GET["ticker"]);
                $ticker='HYG~FBT~';
                echo $ticker;
                $tickerArray=preg_split("/\~/",$ticker);
                 // create curl resource
                        $ch = curl_init();


                        // set urlm
                        curl_setopt($ch, CURLOPT_URL, "http://www.batstrading.com/market_data/symbol_data/csv/");




                        //return the transfer as a string

                        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

                        // $output contains the output string
                        $a='';

                        $output = curl_exec($ch);

                            echo "<br><br>OUTPUT TEST: ".($output);


                $lineCt=0;
                $spaceCt=0;
                $splitOutput=preg_split("[\n|\r]",$output);




                for($ii=0;$ii<sizeof($tickerArray);$ii++){
                $i=0;
                $matchSplit[$ii]=-1;
                         while($i<sizeof($splitOutput) && $matchSplit[$ii]==-1){



                           $splitOutput2=preg_split("/\,/",$splitOutput[$i]);
                               if($i>0){

                                    if(strcasecmp($splitOutput2[0],strtoupper($tickerArray[$ii]))==0){
                                        $matchSplit[$ii]=$splitOutput[$i]."@";
                                    }

                               }
                        $i++;
                        }
                        if($matchSplit[$ii]==-1){
                            echo "notFound@";
                        }else{
                            echo $matchSplit[$ii];
                        }
                }


                //echo ($output);


                        curl_close($ch);  


                ?>

What have you done to debug? Does the resource permit downloading by 3rd parties (i.e. you)? Are they blocking requests from things like cURL? — Michael Berkowski
– Michael Berkowski, Commented Feb 16, 2012 at 2:03
this was working for over a year. how would i further debug this outside of checking for output? — jason m
– jason m, Commented Feb 16, 2012 at 2:06
You could try setting a user agent so that the request looks like it's coming from a browser. It is possible they have tried to block "non-browser" user agents. — F21
– F21, Commented Feb 16, 2012 at 2:11

F21 · Accepted Answer · 2012-02-16 22:14:11Z

1

I added a user agent to your script and it seems to work fine here:

$ch = curl_init(); 

curl_setopt($ch, CURLOPT_URL, "http://www.batstrading.com/market_data/symbol_data/csv/"); 
$agent= 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)'; 
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); 
curl_setopt($ch, CURLOPT_VERBOSE, true); 
curl_setopt($ch, CURLOPT_USERAGENT, $agent); 
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 15); //time out of 15 seconds
$output = curl_exec($ch); 
curl_close($ch);

//Then your output paring code

The output I get:

HYG~FBT~

OUTPUT TEST: Name,Volume,Ask Size,Ask Price,Bid Size,Bid Price,Last Price,Shares Matched,Shares Routed SPY,35641091,0,0.0,0,0.0,134.38,34256509,1384582 BAC,22100508,0,0.0,0,0.0,7.78,20407265,1693243 QQQ,12085707,0,0.0,0,0.0,62.65,11703725,381982 XLF,11642347,0,0.0,0,0.0,14.47,11429581,212766 VXX,9838310,0,0.0,0,0.0,28.2,9525266,313044 EEM,9711498,0,0.0,0,0.0,43.28,9240820,470678 IWM,8272528,0,0.0,0,0.0,81.19,7930349,342179 AAPL,6145951,0,0.0,0,0.0,498.24,4792854,1353097

It is also good practice to close the CURL connection once you are done. I believe that might also play a part in your issues.

If you are still getting issues, check to see if the server the script runs on can access that site.

Update

Upon further investigation, here's what I believe is the root of the problem.

The problem lies with the provider of the CSV file. Perhaps due to some issues on their end, the CSV are generated, but only contain the headers. There were instances where there were indeed data in there.

The data is only avaliable during set hours of the day.

In any case, fetching the empty file = the parser would print @notFound, leading us to assume that there was an issue with CURL.

So my suggestion is to add some further checking to the script to check whether the CSV file actually contains any data at all and is not a file containing just the headings.

Finally, setting a timeout for CURL should fix it as the CSV takes a while to be generated by the provider.

edited Feb 16, 2012 at 22:14

answered Feb 16, 2012 at 2:17

F21

33.6k27 gold badges105 silver badges179 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

jason m Over a year ago

were can i look up this agent info?

F21 Over a year ago

You should be able to get the user agent string of the browser YOU are using to visit the php page by this: $_SERVER['HTTP_USER_AGENT']; Or here for an extensive list: zytrax.com/tech/web/browser_ids.htm

jason m Over a year ago

is there any logical reason it would work readily for you and not me?

F21 Over a year ago

I have updated my answer and setted more options. Have you tried clearing your browser cache?

F21 Over a year ago

I can think of 2 reasons off the top of my head. 1. Your browser is sending you a cached version of the page. Clear your cache to work around this. 2. The remote server implements some sort of rate limiting and you have hit the server too many times.

|

Collectives™ on Stack Overflow

receive csv from server and parse using php

1 Answer 1

8 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Related