Converting nested JSON to data frame

Question

I am trying to convert a heavily nested json into a proper data frame (proper by tidy standards). An MWE of the json is copied at the end of the question, as I wanted to make sure it captured every element of the json.

I've tried:

library(jsonlite)
library(tidyverse)
dat <- jsonlite::fromJSON('data_toy.json') %>% pluck(1) %>% imap_dfr(~mutate(.x, department = .y))

but this returns:

Error: Columns `time_spent`, `school_breakdown`, `reason_for_taking_course`, `student_years`, `interest_before` must be 1d atomic vectors or lists

I've also tried:

dat <- jsonlite::fromJSON('data_toy.json', simplifyVector = FALSE, 
                          simplifyDataFrame = FALSE, flatten=FALSE)
dat.df <- map_df(dat, ~{
  flatten_df(.x[[1]]) %>%
    dplyr::mutate(department = names(.x)[1])
})

but this returns:

Error in bind_rows_(x, .id) : Argument 3 must be length 1, not 0

How can I convert this to a data frame?

Data file (data_toy.json):

{
    "department": {
        "BME": [
            {
                "course_name": "BMD_ENG_250-0_20: Thermodynamics",
                "instructor": "Neha Kamat",
                "time_spent": {},
                "school_breakdown": {
                    "Education & SP": 0,
                    "Communication": 0,
                    "Graduate School": 0,
                    "KGSM": 0
                },
                "reason_for_taking_course": {
                    "Distribution requirement": 0,
                    "Major/Minor requirement": 53
                },
                "student_years": {
                    "Freshman": 5,
                    "Sophomore": 37
                },
                "interest_before": {
                    "1-Not interested at all": 1,
                    "2": 5
                },
                "comments": [
                    "is amazing and you will love her!",
                    "Prof. is so nice"
                ],
                "instructor_gender": "F"
            },
            {
                "course_name": "BMD_ENG_250-0_20: Thermodynamics",
                "instructor": "Neha Kamat",
                "time_spent": {},
                "school_breakdown": {
                    "Education & SP": 0,
                    "Communication": 0,
                    "Graduate School": 0,
                    "KGSM": 0
                },
                "reason_for_taking_course": {
                    "Distribution requirement": 0,
                    "Major/Minor requirement": 53
                },
                "student_years": {
                    "Freshman": 5,
                    "Sophomore": 37
                },
                "interest_before": {
                    "1-Not interested at all": 1,
                    "2": 5
                },
                "comments": [
                    "is amazing and you will love her!",
                    "Prof. is so nice"
                ],
                "instructor_gender": "F"
            }
        ],
        "LING": [
            {
                "course_name": "BMD_ENG_250-0_20: Thermodynamics",
                "instructor": "Neha Kamat",
                "time_spent": {},
                "school_breakdown": {
                    "Education & SP": 0,
                    "Communication": 0,
                    "Graduate School": 0,
                    "KGSM": 0
                },
                "reason_for_taking_course": {
                    "Distribution requirement": 0,
                    "Major/Minor requirement": 53
                },
                "student_years": {
                    "Freshman": 5,
                    "Sophomore": 37
                },
                "interest_before": {
                    "1-Not interested at all": 1,
                    "2": 5
                },
                "comments": [
                    "is amazing and you will love her!",
                    "Prof. is so nice"
                ],
                "instructor_gender": "F"
            },
            {
                "course_name": "BMD_ENG_250-0_20: Thermodynamics",
                "instructor": "Neha Kamat",
                "time_spent": {},
                "school_breakdown": {
                    "Education & SP": 0,
                    "Communication": 0,
                    "Graduate School": 0,
                    "KGSM": 0
                },
                "reason_for_taking_course": {
                    "Distribution requirement": 0,
                    "Major/Minor requirement": 53
                },
                "student_years": {
                    "Freshman": 5,
                    "Sophomore": 37
                },
                "interest_before": {
                    "1-Not interested at all": 1,
                    "2": 5
                },
                "comments": [
                    "is amazing and you will love her!",
                    "Prof. is so nice"
                ],
                "instructor_gender": "F"
            }
        ]
    }
}

Julius Vainora · Accepted Answer · 2018-12-02 00:54:29Z

3

Using flatten = TRUE seems to be the key here:

dat <- jsonlite::fromJSON('data_toy.json', flatten = TRUE)[[1]]
dat %>% bind_rows() %>% mutate(department = rep(names(dat), map_dbl(dat, nrow)))

answered Dec 2, 2018 at 0:54

Julius Vainora

48.4k9 gold badges95 silver badges108 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Adam_G Over a year ago

That did it. Thank you!

Collectives™ on Stack Overflow

Converting nested JSON to data frame

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related