I need to transform following data frame with json values in column into dataframe columnar structure so that it will be taking less space and easy to compute.
Sample DataFrame:
| obs_id | date | obs |
|---|---|---|
| I2213 | 2021-12-31 23:20:02.761008 | "[{'type': 'air', 'results': {'bat': {'F1': 0.1, 'F2': 0.2}}, {'type': 'water', 'results': {'neo': {'F1': 0.3}}]" |
| I2213 | 2022-01-01 23:20:02.761008 | "[{'type': 'earth', 'results': {'cat': {'F1': 0.4}}]" |
| I2213 | 2022-01-02 23:20:02.761008 | "[{'type': 'air', 'results': {'bat': {'F1': 0.2, 'F2': 0.1}}]" |
Required Transformation format:
| obs_id | date | obs.air.bat.F1 | obs.air.bat.F2 | obs.water.neo.F1 | obs.earth.cat.F1 |
|---|
not sure if multi-level columns will suit better here.
I tried to create a separate dataframe from obs column like:
df1 = pd.DataFrame(df['obs'].values.tolist())
but since it contains list instead of dictionary, it doesn't work. Is it possible to achieve the require format?