2

I'm looking for a simple workaround to import an existing nested JSON data into several MySQL tables. JSON does not have bidirectional relations so they should be generated automatically, I think.

Here is a data sample:

[
    {
        "targetgroup": "Staff",
        "plan": "this field just exists and should be ignored in database",
        "budgetlevel": "Government",
        "spots": 5,
        "edutype": "Bachelor",
        "qualilevel": "Specialist",
        "speciality": "Mathematician",
        "qualification": "Finished",
        "faculty": "Applied mathematics",
        "institute": "this field is sometimes empty in input data",
        "eduform": "Full-time",
        "profiles": [
            "Jr. Arithmetic manager"
        ],
        "entrancetests": [
            {
                "subject": "math",
                "typeoftest": "GOV",
                "minscore": "37",
                "ratingtype": "out of 100"
            },
            {
                "subject": "language",
                "typeoftest": "GOV",
                "minscore": "27",
                "ratingtype": "out of 100"
            },
            {
                "subject": "physics",
                "typeoftest": "GOV",
                "minscore": "40",
                "ratingtype": "out of 100"
            }
        ]
    },
  {
        "targetgroup": "Educational workers",
        "plan": "fridge",
        "budgetlevel": "Legacy",
        "spots": 26,
        "edutype": "Bachelor",
        "qualilevel": "Master",
        "speciality": "Data analysis",
        "qualification": "Finished",
        "faculty": "Machine learning mathematics",
        "institute": "",
        "eduform": "Full-time",
        "profiles": [
            "Head counting manager"
        ],
        "entrancetests": [
            {
                "subject": "Discrete mathematics",
                "typeoftest": "GOV",
                "minscore": "32",
                "ratingtype": "Out of 100"
            },
            {
                "subject": "Algorythm theory",
                "typeoftest": "GOV",
                "minscore": "51",
                "ratingtype": "Out of 100"
            },
            {
                "subject": "Advanced exception catching",
                "typeoftest": "GOV",
                "minscore": "56",
                "ratingtype": "Out of 100"
            }
        ]
    }
]

Database structure:

table "dep:"

id(auto increment) | targetgroup | budgetlevel | spots | edutype ... etc, same as JSON field names

table "profiles"

id (relative to corresponding parent block) | name

table "entrancetests":

id (relative to corresponding parent block) | subject | typeoftest | minscore | ratingtype

I have a general idea about how to import non-nested JSON but I have a hard time figuring out how do I add relations, how do I define parent block inside the loop?

1
  • json_decode and start looping through it. Some databases can store the entire json blob and create relations from it as well. Commented Mar 31, 2017 at 3:00

1 Answer 1

2

Below is a fully functional transactional pdo process for your exact case. I've tested on my server with 3 MyISAM tables that match your structure. Now, it is possible that you are not into pdo at the moment; if not, I hope my offering will compel you to try something new and modernize the way you query your database.

$json='[
    {
        "targetgroup": "Staff",
        "plan": "this field just exists and should be ignored in database",
        "budgetlevel": "Government",
        "spots": 5,
        "edutype": "Bachelor",
        "qualilevel": "Specialist",
        "speciality": "Mathematician",
        "qualification": "Finished",
        "faculty": "Applied mathematics",
        "institute": "this field is sometimes empty in input data",
        "eduform": "Full-time",
        "profiles": [
            "Jr. Arithmetic manager"
        ],
        "entrancetests": [
            {
                "subject": "math",
                "typeoftest": "GOV",
                "minscore": "37",
                "ratingtype": "out of 100"
            },
            {
                "subject": "language",
                "typeoftest": "GOV",
                "minscore": "27",
                "ratingtype": "out of 100"
            },
            {
                "subject": "physics",
                "typeoftest": "GOV",
                "minscore": "40",
                "ratingtype": "out of 100"
            }
        ]
    },
  {
        "targetgroup": "Educational workers",
        "plan": "fridge",
        "budgetlevel": "Legacy",
        "spots": 26,
        "edutype": "Bachelor",
        "qualilevel": "Master",
        "speciality": "Data analysis",
        "qualification": "Finished",
        "faculty": "Machine learning mathematics",
        "institute": "",
        "eduform": "Full-time",
        "profiles": [
            "Head counting manager"
        ],
        "entrancetests": [
            {
                "subject": "Discrete mathematics",
                "typeoftest": "GOV",
                "minscore": "32",
                "ratingtype": "Out of 100"
            },
            {
                "subject": "Algorythm theory",
                "typeoftest": "GOV",
                "minscore": "51",
                "ratingtype": "Out of 100"
            },
            {
                "subject": "Advanced exception catching",
                "typeoftest": "GOV",
                "minscore": "56",
                "ratingtype": "Out of 100"
            }
        ]
    }
]';


$db=new PDO("mysql:host=yourhost;dbname=yourdbname;charset=utf8","username","password");

try{
    $db->setAttribute(PDO::ATTR_ERRMODE,PDO::ERRMODE_EXCEPTION);  // this will deny subsequent queries from being executed if there is an error and permit exception handle at the bottom
    $db->beginTransaction();

    // dep
    $dep_cols=array("targetgroup","budgetlevel","spots",
        "edutype","qualilevel","speciality","qualification",
        "faculty","institute","eduform");  // declare columns
    $dep_keys=array_map(function($v){return ":$v";},$dep_cols);  // build :keys    
    $dep_cols=array_combine($dep_keys,$dep_cols);   // assign :keys
    var_export($dep_cols);
    $dep_query="INSERT INTO `dep` (`".implode('`,`',$dep_cols)."`)"; // list columns as csv
    $dep_query.=" VALUES (".implode(',',array_keys($dep_cols)).");";
    echo "<div>$dep_query</div>";
    $stmt_add_dep=$db->prepare($dep_query);

    // profile
    $profile_cols=array('name');
    $profile_query="INSERT INTO `profile` (`id`,`".implode('`,`',$profile_cols)."`)"; // list columns as csv
    $profile_query.=" VALUES (LAST_INSERT_ID(),".implode(',',array_fill(0,sizeof($profile_cols),"?")).");";
    echo "<div>$profile_query</div>";
    
    // entrancetests
    $entrance_cols=array('subject','typeoftest','minscore','ratingtype');  // declare columns
    $entrance_keys=array_map(function($v){return ":$v";},$entrance_cols);  // build :keys
    $entrance_cols=array_combine($entrance_keys,$entrance_cols);  // assign :keys    
    var_export($entrance_cols);
    $entrance_query="INSERT INTO `entrancetests` (`id`,`".implode('`,`',$entrance_cols)."`)"; // list columns as csv
    $entrance_query.=" VALUES (LAST_INSERT_ID(),".implode(',',array_keys($entrance_cols)).");";
    echo "<div>$entrance_query</div>";
    $stmt_add_entrance=$db->prepare($entrance_query);
        
    foreach(json_decode($json) as $d){
        foreach($dep_cols as $k=>$v){
            $stmt_add_dep->bindValue($k,(property_exists($d,$v)?$d->$v:""));
            echo "<div>$k => {$d->$v}</div>";
        }
        $stmt_add_dep->execute();
        echo "<div>Dep Affected Rows: ",$stmt_add_dep->rowCount(),"</div><br>";
        
        $stmt_add_profile=$db->prepare($profile_query);
        foreach($d->profiles as $k=>$v){
            $stmt_add_profile->bindValue($k+1,$v);
            echo "<div>",$k+1," => $v</div>";
        }
        $stmt_add_profile->execute();
        echo "<div>Profile Affected Rows: ",$stmt_add_profile->rowCount(),"</div><br>";
        
        foreach($d->entrancetests as $o){
            foreach($entrance_cols as $k=>$v){
                $stmt_add_entrance->bindValue($k,(property_exists($o,$v)?$o->$v:""));
                echo "<div>$k => {$o->$v}</div>";
            }
        }
        $stmt_add_entrance->execute();
        echo "<div>Entrance Affected Rows: ",$stmt_add_entrance->rowCount(),"</div><br>";
    }
    
    // $db->commit();  // Only use with InnoDB tables.  MyISAM is auto-commit
    
}
catch(PDOException $e){
    // $db->rollBack();  // Only works if InnoDB table.  If MyISAM table, it doesn't rollback.
    echo "Error message: {$e->getMessage()}. File: {$e->getFile()}. Line: {$e->getLine()}";
    // do not show these error messages to users when you go live
}

As is, this script will stop executing subsequent queries if a query has an error.

Sign up to request clarification or add additional context in comments.

6 Comments

Thank you so much for your help. I'm a bit new to SO, so I shold definitely have posted the code I had. It works like a charm, however, it returns "unexpected '['" if I'm using it like it is. Removing [0] from json_decode makes it work, but I'm only getting the first entry of "entrancetests" in MyISAM table.
Figured it out, looks like declaring arrays with [] starts with PHP 5.4, I'm using 5.3 on my local server.
@mrkl Good catch. Well done for correcting that hiccup yourself. (Other users sometimes just lump it back on me to fix these types of errors.)
Looks like it slill parses only the first entry for "entrancetests" and also only the first parent entry in the entire JSON. Generated page shows the looping through all the "entrancetests", I see all the tests in browser, but only one gets in the table. I know your code isn't intended to parse multiple entries, but from what I see it should work with all the "entrancetests" at least. Pretty werid.
@mrkl Okay, I've updated (and tested) my code to process multiple batches of queries. I hope all goes well for you.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.