I am very new to coding in PHP/MySQL, for fun I've been downloading some World of Worldcraft data from their API to test what I've learnt about PHP.
Sorry for the long post, I'm trying to give as much info as needed, whilst being concise :)
The short version is "my code takes 25 minutes to run. Can I do anything (things) better to speed this up?
LONG VERSION BELOW
Blizzard have a lot of data held about their servers, in a JSON format & there's a JSON file per server (I've a list 123 servers in total). I've written a basic JSON file that holds the name & URL of the 123 servers (serv_gb.json).
The code should run as follows:-
- Read serv_gb.json to count the number of rows to use in a For loop.
- Record a start time for my own check.
- The For loop starts, and it reads & accesses the URL's in serv_gb.json.
- The URL holds a JSON file for that server, it's parsed and held in an array. The URL when opened in a web browser takes on average 20 seconds to load & holds approx 100k rows.
- It then uses the foreach to read & send the data to my database.
- The end time is recorded & a duration calculated.
There is more data in the JSON file than I need. I've tried two ways to filter this.
a) Just try to send all of the data in the JSON & if it fails to & would normally error, hide it with "error_reporting(0);". (Lazy, yes, because this is a pet project just for me, I thought I'd try it!).
b) Adding the following If loop around the SQL to only send the correct data to the database.
foreach($auctions as $val)
if ($val['item'] == 82800) {
{
$sql = "INSERT INTO `petshop`.`ah` (datedatacollected, seller, petSpeciesId, server, buyout, region, realrealm) VALUES('".$cTime."','".$val['owner']."','".$val['petSpeciesId']."', '".$servername."', '".$val['buyout']."','gb','".$val['ownerRealm']."')";
$mysqli->query($sql);
}
}
Idea A is about 15% slower than Idea B. Idea B takes the 25 minutes to run through all 123 JSON's.
I run this on a local machine (my own desktop pc) running Apache2.4, MySQL57 & PHP Version 5.6.32
Any advice is appreciated, I think the following are good questions that I need help learning more about.
- Is there any ways to making the loops or reads faster (or am I at the mercy of Blizzard's servers)?
- Is it the writing to the DB that's bottle necking the process?
- Would splitting the URLs up & triggering multiple JSON reads at the same time speed this up?
If you've got this far in the post. A huge THANK YOU for taking the time to read it, I hope it was clear and made sense!
MY PHP CODE
$servjson = '../servers/serv_gb.json';
$servcontents = file_get_contents($servjson);
$servdata = json_decode($servcontents,true);
$jrc = count($servdata['servers']);
echo "Json Row Count: ".$jrc."<p>";
$sTime = (new \DateTime())->format('Y-m-d H:i:s');
for($i=0; $i<$jrc; $i++){
$json = $servdata['servers'][$i]['url'];
$contents = file_get_contents($json);
$data = json_decode($contents,true);
$servername = $data['realms'][0]['slug'];
$auctions = $data['auctions'];
foreach($auctions as $val)
{
$sql = "INSERT INTO `petshop`.`ah` (datedatacollected, seller, petSpeciesId, server, buyout, region, realrealm) VALUES('".$cTime."','".$val['owner']."','".$val['petSpeciesId']."', '".$servername."', '".$val['buyout']."','gb','".$val['ownerRealm']."')";
$mysqli->query($sql);
}
}
$eTime = (new \DateTime())->format('Y-m-d H:i:s');
$dTime = abs(strtotime($eTime) - strtotime($sTime));