How to convert nested lists with data frames inside to one data frame in R

Question

How can I convert a quite complicated nested lists with data frames inside to one consolidated data frame?

Here are first 3 records in my raw input in JSON.

{
    "abc": [[{"variant": "enabled", "allocation": 100}], [{"variant": "control", "allocation": 100}]],
    "def": [[{"variant": "enable", "allocation": 100}]], 
    "hahaha": [[{"variant": "superman", "allocation": 5}, {"variant": "superhero", "allocation": 95}]]
}

Then I loaded this JSON file into R.

library(jsonlite)
myList <- fromJSON("myjsonfile.json")

str(myList)

List of 8988
 $  abc                                                                        :List of 2
  ..$ :'data.frame':    1 obs. of  2 variables:
  .. ..$ variant   : chr "enabled"
  .. ..$ allocation: int 100
  ..$ :'data.frame':    1 obs. of  2 variables:
  .. ..$ variant   : chr "control"
  .. ..$ allocation: int 100
$ def                                                                          :List of 1
  ..$ :'data.frame':    1 obs. of  2 variables:
  .. ..$ variant   : chr "enable"
  .. ..$ allocation: int 100
$ hahaha                                                                       :List of 1
  ..$ :'data.frame':    2 obs. of  2 variables:
  .. ..$ variant   : chr [1:2] "superman" "superhero"
  .. ..$ allocation: int [1:2] 5 95

As you can see in each list, there could be different number of data frames and each data frame may contain different number of obs.

Ideally I want to get one dataframe as below:

test_name, segments, variant, allocation
abc, 1, enabled, 100
abc, 2, control, 100
def, 1, enable, 100,
hahaha, 1, superman, 5
hahaha, 1, superhero, 95

What is a scalable approach for all 8988 records here? Appreciate your helps here.

Can you create a short example list which is similar to your data and share a reproducible example using dput ? — Ronak Shah
– Ronak Shah, Commented Sep 5, 2020 at 2:15
The data you have shared is not reproducible. We cannot copy/paste that into R and use it. See stackoverflow.com/questions/5963269 — Ronak Shah
– Ronak Shah, Commented Sep 5, 2020 at 5:05

Joris C. · Accepted Answer · 2020-09-05 09:31:39Z

1

Here is an approach that:

Melts the nested list to a data.frame with rrapply() (in the rrapply-package).
Reshapes the data.frame using tidyr's pivot_wider() and unnest().

library(rrapply)
library(tidyverse)

## melt to data.frame
mydf <- rrapply(myList, how = "melt")

## reshape data.frame
mydf_reshaped <- pivot_wider(mydf, names_from = "L3") %>%
  unnest(c(variant, allocation)) %>%
  rename(test_name = L1, segments = L2)

mydf_reshaped
#> # A tibble: 5 x 4
#>   test_name segments variant   allocation
#>   <chr>     <chr>    <chr>          <int>
#> 1 abc       ..1      enabled          100
#> 2 abc       ..2      control          100
#> 3 def       ..1      enable           100
#> 4 hahaha    ..1      superman           5
#> 5 hahaha    ..1      superhero         95

This should directly generalize to the complete json-file as well.

Data

myList <- list(abc = list(structure(list(variant = "enabled", allocation = 100L), class = "data.frame", row.names = 1L), 
                          structure(list(variant = "control", allocation = 100L), class = "data.frame", row.names = 1L)), 
               def = list(structure(list(variant = "enable", allocation = 100L), class = "data.frame", row.names = 1L)), 
               hahaha = list(structure(list(variant = c("superman", "superhero"
               ), allocation = c(5L, 95L)), class = "data.frame", row.names = 1:2)))

answered Sep 5, 2020 at 9:31

Joris C.

6,2943 gold badges14 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

yuckeye Over a year ago

Brilliant! Thank you for this great solution!

David · Accepted Answer · 2020-09-05 02:25:30Z

0

Easiest way is to use dplyr bind_rows().

library(dplyr)

df_list <- list(iris[1:5,], iris[6:10,])

bind_rows(df_list)

answered Sep 5, 2020 at 2:25

David

2,7471 gold badge15 silver badges24 bronze badges

2 Comments

yuckeye Over a year ago

This would not work. I have many lists inside one list. And each list includes different number of data frames with different number of rows.

David Over a year ago

bind rows will still work with nested list. but it wont add the test_name column. Possibly use @Joris C solution

Collectives™ on Stack Overflow

How to convert nested lists with data frames inside to one data frame in R

2 Answers 2

1 Comment

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related