Uploading a Json object array to Firestore

Question

I have this nicely ordered yet large Json object Array that follows this structure:

{
   "object1":[
      {
         "data1":19.77,
         "data2":-0.953125,
         "data3":-0.265625,
         "id":17231,
         "date":"2021-05-14/08:38:47.471"
      },
      {
         "data1":19.77,
         "data2":-0.953125,
         "data3":-0.265625,
         "id":17231,
         "date":"2021-05-14/08:38:47.596"
      },
      {
         "data1":19.77,
         "data2":-0.9609375,
         "data3":-0.2734375,
         "id":17231,
         "date":"2021-05-14/08:38:47.721"
      }
   ],
   "object2":[
      {
         "data1":19.91,
         "data2":-0.9765625,
         "data3":-0.109375,
         "id":18996,
         "date":"2021-05-14/08:38:47.681"
      },
      {
         "data1":19.91,
         "data2":-0.9765625,
         "data3":-0.109375,
         "id":18996,
         "date":"2021-05-14/08:38:47.806"
      },
      {
         "data1":19.91,
         "data2":-1,
         "data3":-0.1171875,
         "id":18996,
         "date":"2021-05-14/08:38:47.931"
      },
      {
         "data1":19.91,
         "data2":-0.96875,
         "data3":-0.1015625,
         "id":18997,
         "date":"2021-05-14/08:38:48.051"
      },
      {
         "data1":19.91,
         "data2":-0.96875,
         "data3":-0.1015625,
         "id":18997,
         "date":"2021-05-14/08:38:48.059"
      },
      {
         "data1":19.91,
         "data2":-0.9765625,
         "data3":-0.109375,
         "id":18997,
         "date":"2021-05-14/08:38:48.176"
      }
   ]
}

And I've been trying to upload it to firebase with no success, trying to follow this schema in Firestore:

Objects
    ├───Object1
    │   ├───[Upload Timestamp]
    │   │   └───Data Array 1
    │   ├───[Upload Timestamp]
    │   │   └───Data Array 2
    │   └───[Upload Timestamp]
    │       └───Data Array 3
    └───Object2
        ├───[Upload Timestamp]
        │   └───Data Array 1
        ├───[Upload Timestamp]
        │   └───Data Array 2
        ├───[Upload Timestamp]
        │   └───Data Array 3
        ├───[Upload Timestamp]
        │   └───Data Array 4
        ├───[Upload Timestamp]
        │   └───Data Array 5
        └───[Upload Timestamp]
            └───Data Array 6

I tried to do this with no luck using nested For Loops, Dictionaries and whatnot. So far, I've been only capable of uploading the last Data Array of each object to Firestore.

Is there an way I can properly upload this to Firestore ?

EDIT: Adding more info about my code

What I've got:

import firebase_admin
from firebase_admin import credentials
from firebase_admin import firestore
import json
from itertools import groupby
import datetime
from datetime import date

today = date.today()
d1 = today.strftime("%d.%m.%Y")

json_file_path = "./myfile.json" 

with open(json_file_path, 'r') as j:
    contents = json.loads(j.read())
data = {}
#Sorting my json file
for key, items in groupby(sorted(contents, key = lambda x: (x['id'], x['date'])), key=lambda x: x['id']):  
    data[key] = list(items)

cred = credentials.Certificate("myAppCreds.json")

firebase_admin.initialize_app(cred, {
'projectId': "MyProjectID",
})

db = firestore.client()

for keys in data:
    doc_ref = db.collection('Objects').document(keys).collection(d1).document(str(datetime.datetime.now()))
    for info in data[keys]:
        doc_ref.set(keys)

What happens:

The following entry is created in the Firestore:

Objects/object1/19.05.2021/2021-05-19 13:26:03.933868

Where the document "2021-05-19 13:26:03.933868" is constantly updated with new values, instead of a new one being created for each new value. The same happens with object2

Your timestamp is unique to every record inside your object array, so you would have one leaf only under each [Upload Timestamp]. — Niloct
– Niloct, Commented May 19, 2021 at 14:43
I'm not familiar with the interface, but I've managed to create an object on the admin with your object schema. How would I test upload ? — Niloct
– Niloct, Commented May 19, 2021 at 15:26
Basically you need to define an array of maps, inside each object node. — Niloct
– Niloct, Commented May 19, 2021 at 15:32
Please edit your question with your upload code that isn't working. — Niloct
– Niloct, Commented May 19, 2021 at 15:36

Niloct · Accepted Answer · 2021-05-19 20:02:50Z

Here's an working draft, with some remarks:

import firebase_admin
from firebase_admin import credentials
from firebase_admin import firestore
import json
from datetime import date

today = date.today()
d1 = today.strftime("%d.%m.%Y")

json_file_path = "./myfile.json" 

with open(json_file_path, 'r') as j:
    contents = json.loads(j.read())

# 1. This will be the criteria to sort each inner list
# first by ID, then by date, ascending
def _sort(item):
    return (item['id'], item['date'])

# 2. Sorting my json file lists, in place
for lista in contents.values():
    lista.sort(key=_sort)

cred = credentials.Certificate("myAppCreds.json")

firebase_admin.initialize_app(cred, {
'projectId': "MyProjectID",
})    

db = firestore.client()

# Inside the collection Objects, I create a document for each object key,
# Then set the inner list as an inner document, with value equals to the list.
for key, lista in contents.items():
    doc_ref = db.document('Objects/' + key).set({'lista': lista})

This won't sort absolutely per id/date, because the objects are sorted individually. You may want to adapt this.

In firebase, I've successfully created the docs running this code.

Collectives™ on Stack Overflow

Uploading a Json object array to Firestore

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related