diff --git a/docs/models/images/app.png b/docs/models/images/app.png new file mode 100644 index 00000000..90367ead Binary files /dev/null and b/docs/models/images/app.png differ diff --git a/docs/models/images/brick.png b/docs/models/images/brick.png new file mode 100644 index 00000000..8eff8306 Binary files /dev/null and b/docs/models/images/brick.png differ diff --git a/docs/models/images/brick_conf_b.png b/docs/models/images/brick_conf_b.png new file mode 100644 index 00000000..6a4bfc92 Binary files /dev/null and b/docs/models/images/brick_conf_b.png differ diff --git a/docs/models/images/brick_conf_v.png b/docs/models/images/brick_conf_v.png new file mode 100644 index 00000000..1283d6cb Binary files /dev/null and b/docs/models/images/brick_conf_v.png differ diff --git a/docs/models/images/ei_error.png b/docs/models/images/ei_error.png new file mode 100644 index 00000000..91400a3c Binary files /dev/null and b/docs/models/images/ei_error.png differ diff --git a/docs/models/images/model.png b/docs/models/images/model.png new file mode 100644 index 00000000..2dae4162 Binary files /dev/null and b/docs/models/images/model.png differ diff --git a/docs/models/images/obj_detection_v.png b/docs/models/images/obj_detection_v.png new file mode 100644 index 00000000..90ba659e Binary files /dev/null and b/docs/models/images/obj_detection_v.png differ diff --git a/docs/models/models.md b/docs/models/models.md new file mode 100644 index 00000000..61f4d14e --- /dev/null +++ b/docs/models/models.md @@ -0,0 +1,493 @@ +This is a possible implementation of AI model support in AppLab. + +# Models +In AI, a model is a trained system able to recognize patterns and/or make predictions. +In AppLab, a model is an autonomous entity that can exist independently of brick utilization. + +The approach proposed for the storage of custom models is to mirror the library architecture. +Custom models will reside in a tree structure on the board, where associated configuration variables are defined within a YAML configuration file. + +A user can create a new model by creating the model folder structure. Custom metadata can be added to the model's YAML file. + +## Example of models definition +Models are defined by the following folder structure. This could be a package to be installed on the board. +``` +models/ + model_name/ + README.md + conf.yaml + metadata.yaml + builds/ + build#1 + build#2 +``` + +*NOTE:* We need to eventually discuss how to handle different builds of the same model. In this case, we must ensure we reference the specific build to be run and manage the model state, model definition state, and deletion accordingly. + +*NOTE:* To be discussed how to map Edge Impulse in an AppLab model. See Edge Impulse section below. + +The ```conf.yaml``` file contains generic model information. The application expects this data to be available for all models. The ```model_categories``` is a list of the categories of the model. +``` + runner: brick + name : "Glass breaking classifier" + model_categories: [audio, video] + model_labels: + - "Background" + - "Glass_Breaking" +``` + +The ```metadata.yaml``` file contains model-specific metadata. +``` + source: "edgeimpulse" + ei-project-id: 749446 + source-model-id: "glass-breaking" + source-model-url: "https://studio.edgeimpulse.com/studio/749446" +``` +NOTE: Should we have a metadata section for each category? + +## Category +### Category definition +In AI, a category is a common way to classify AI models, indicating the type of data they process or generate. Examples include audio, image, video, and text model categories. + +In AppLab, a category is defined as a keyword used to connect models to bricks. It defines which models can be attached to which bricks. Both a model and a brick have a list of categories. + +### Linking Models to Bricks +In the brick configuration, we can choose the model to be used with the brick. The provided list is a filtered list of all models where the intersection of the model's category list and the brick's category list is not empty. + +**We assume here that every model with a given category is able to work with any provider.** If there is a special model that is only able to work with a specific brick, we can always add a custom category to associate the model with the working brick. + +*NOTE:* We need to clarify how to configure this when more than one container (and thus, more than one model) is defined within a brick. + +## Brick +A brick is defined in the brick-list.yaml +``` +- id: arduino:audio_classification + name: Audio Classification + categories: audio, other_cat +``` + +*NOTE:* categories is a model specific variable, what to do for a brick that does not involve a model? +Somewhere is null, somewhere is "storage" + +## App.yaml +This file contains the app's information and configuration. +``` +name: Copy of Detect objects on images +description: Object detection in the browser +ports: [] +bricks: +- arduino:web_ui: {} +- arduino:object_detection: + model: dog-detector +- arduino:mood_detector: {} +icon: 🏞️ +``` +Here we added the model variable to be used for each brick. +*Note:* We assume here there is one model for each brick. + +# This section will address Edge Impulse specific case +An Edge Impulse project is a set of input data and one or more impulses. + +For each project there is the project type (audio/video...) that should be set accordingly to input data. +Changing input data type is recognized, for an audio project and an image upload, the user will be prompted with a message: +![Change project type](images/ei_error.png) + +An impulse is a set of configurations defined on input data that can generate a model according to a classification method, a processing block, and hardware parameters. + +For each project, you can define one or more impulses. A model is the output of an impulse build, and a build can be executed more than one time, with different parameters. +Edge Impulse provides information about the last build present on the system. + +*NOTE:* We need to discuss how to map the Edge Impulse structure to an AppLab model. Here are some notes for discussion: +* AppLab model is an EI project; AppLab build is the impulse. (We always have the last build of a specific impulse) +* AppLab model is an impulse; AppLab build is an Ei build. +* AppLab model is an impulse; AppLab build is a specific build with specific build parameters. + +# Use Cases: +The following section explores how models and bricks looks like using this architecture. + +## Case #1 Simple app +We want to create an application with two bricks with no models, one of them require some variables. E.g. arduino_cloud +### Brick definition (brick-list.yaml) +``` +- id: arduino:arduino_cloud + name: Arduino Cloud + description: Connects to Arduino Cloud + require_container: false + require_model: false + require_devices: false + mount_devices_into_container: false + ports: [] + category: [audio] + variables: + - name: ARDUINO_DEVICE_ID + description: Arduino Cloud Device ID + - name: ARDUINO_SECRET + description: Arduino Cloud Secret +- id: arduino:dbstorage_sqlstore + name: Database - SQL + description: Simplified database storage layer for Arduino sensor data using SQLite + local database. + require_container: false + require_model: false + require_devices: false + mount_devices_into_container: false + ports: [] + category: storage +``` +*Note:* Is the category keyword here connected to the model concept? + +The brick contains some variables, visible to the user that should/could be configured by the user from the AppLab brick configuration. +![Brick configuration_button](images/brick_conf_b.png) +![Brick configuration_vars](images/brick_conf_v.png) + +### App.yaml definition +The configured variables will be stored in the related brick section in the App.yaml file, as follow: +``` +name: Copy of Blinking LED from Arduino Cloud +description: Control the LED from the Arduino IoT +ports: [] +bricks: +- arduino:arduino_cloud: + variables: + ARDUINO_DEVICE_ID: my_device_id + ARDUINO_SECRET: my_secret +- arduino:dbstorage_tsstore: {} +icon: ☁️ +``` + +## Case #2 Same brick, two compatible models +We have one brick able to support the ``audio`` category and we have two models, both working with the ``audio`` category, generated by different AI engine. + +The brick is always the same, it is supposed the input/env variables will not change. +### Model definition +``` +models/ + hey_arduino_ei/ + conf.yaml + metadata.yaml + builds/edge_impulse_hey_arduino.eim + hey_arduino_qlc/ + conf.yaml + metadata.yaml + builds/ + qualcomm_hey_arduino.model +``` +### hey_arduino_ei/conf.yaml +``` +name : "Edge Impulse Hey Arduino model" +model_category: [audio] +model_labels: + - "hey_arduino" + - "other_labels_here" +``` +*Note:* The following section was removed: +``` + model_configuration: + "EI_KEYWORD_SPOTTING_MODEL": "/models/ootb/ei/keyword-spotting-hey-arduino.eim" + ``` +### hey_arduino_ei/metadata.yaml +``` +source: "edgeimpulse" +ei-project-id: 757509 +ei-impulse-id: 30 +source-model-id: "hey-arduino" +source-model-url: "https://studio.edgeimpulse.com/studio/757509" +``` +### hey_arduino_qlc/conf.yaml +``` +name : "Qualcomm Hey Arduino model" +model_category: [audio] +model_labels: + - "hey_arduino" + - "other_labels_here" +``` +### hey_arduino_qlc/metadata.yaml +``` +source: "qualcomm-ai-hub" +ei-gpu-mode: false +source-model-id: "hey_arduino_spotting-lite" +source-model-url: "https://aihub.qualcomm.com/models/hey_ardino_spotting_lite" +``` +### Brick definition (brick-list.yaml) +``` +- id: arduino:keyword_spotting + name: Keyword Spotting + description: Brick for keyword spotting + require_container: true + require_model: true + require_devices: false + mount_devices_into_container: false + ports: [] + model_category: [audio] + required_devices: + - microphone +``` +*Note:* The following variables are defined in the current implementation but should be removed because they must be hidden from the final user. +``` + variables: + - name: CUSTOM_MODEL_PATH + default_value: /home/arduino/.arduino-bricks/ei-models + description: path to the custom model directory + - name: EI_KEYWORD_SPOTTING_MODEL + default_value: /models/ootb/ei/keyword-spotting-hey-arduino.eim + description: path to the model file +``` +### App.yaml definition, when using EI model +``` +name: Hey Arduino, ei model +description: When "Hey Arduino!" keyword is detected by the microphone, the led matrix will react +ports: [] +bricks: +- arduino:keyword_spotting: + EI_KEYWORD_SPOTTING_MODEL: "/models/hei_arduino_ei/builds/edge_impulse_hey_arduino.eim" +icon: ❤️ +``` +### App.yaml definition, when using QLC model +Just change the brick section as follow: +``` +bricks: +- arduino:keyword_spotting: +EI_KEYWORD_SPOTTING_MODEL: "/models/hei_arduino_qlc/builds/qualcomm_impulse_hey_arduino.model" +``` + +## Case #3 Two distinct bricks using the same compatible model +There is one model and two bricks with a different runner using the same model. + +In this case, the worst thing that could happen is that different running engines could require a different configuration. + +In case the shared model is exposed to the container via a filesystem overlay, we need to guard against unintended model updates by the instance container. +### Model definition +``` +models/ + yolo_x_nano/ + conf.yaml + metadata.yaml + builds/yolo-x-nano.eim +``` +### yolo_x_nano/conf.yaml +name : Lightweight-Face-Detection +model_category: [video] +model_labels: + - "face" + - "other_labels_here" +### yolo_x_nano/metadata.yaml +``` +source: "edgeimpulse" +ei-project-id: 717280 +source-model-id: "YOLOX-Nano" +source-model-url: "https://github.com/Megvii-BaseDetection/YOLOX" +``` +### Brick definition (brick-list.yaml) +``` +- id: arduino:object_detection + name: Object Detection + description: "Brick for object detection" + require_container: true + require_model: true + require_devices: false + mount_devices_into_container: false + ports: [] + model_category: [video] +- id: arduino:fast_runner_object_detection + name: Object Detection + description: "Brick for faster object detection" + require_container: true + require_model: true + require_devices: false + mount_devices_into_container: false + ports: [] + model_category: [video] +``` +### App.yaml definition +``` +name: Detect objects on images using two webcam, compare slow and fast engine over the same model +ports: [] +bricks: +- arduino:object_detection: {} + variables: + CUSTOM_MODEL_PATH: /home/arduino/.arduino-bricks/ei-models + EI_OBJ_DETECTION_MODEL: /models/yolo_x_nano/builds/yolo-x-nano.eim +- arduino:fast_object_detection: {} + variables: + CUSTOM_MODEL_PATH: /home/arduino/.arduino-bricks/ei-models + QLC_OBJ_DET_ENGINE_MODEL: /models/yolo_x_nano/builds/yolo-x-nano.eim + SET_CUSTOM_PARAMETER: 5 +icon: 🏞️ +``` +*NOTE:* Should these variables be exposed or hardcoded? In any case they should be not visible to the user. + +## Case #4 Same brick, two custom compatible models +### Model definition +``` +models/ + hey_arduino_ei/ + conf.yaml + metadata.yaml + builds/edge_impulse_hey_arduino.eim + builds/edge_impulse_hey_arduino_MFCC.eim + hey_arduino_qlc/ + conf.yaml + metadata.yaml + builds/ + qualcomm_hey_arduino_quant8.model + qualcomm_hey_arduino_quant32.model + qualcomm_hey_arduino_1k_samples.model + qualcomm_hey_arduino_1M_samples.model + +``` +The conf and metadata YAML files are the same as in Case #2. This custom model scenario, however, changes the way we handle model lifecycle management and model provision to the container. (Could a filesystem overlay be an option?) + +*NOTE:* There is another case here: the container could decide to autonomously download the model from a remote site using its metadata. If this happens, we only need to expose the metadata. + +### Brick definition (brick-list.yaml) +``` +- id: arduino:keyword_spotting + name: Keyword Spotting + description: Brick for keyword spotting + require_container: true + require_model: true + require_devices: false + mount_devices_into_container: false + ports: [] + model_category: [video] + required_devices: + - microphone + variables: + - name: EI_KEYWORD_SPOTTING_MODEL +``` +*NOTE:* The EI_KEYWORD_SPOTTING_MODEL is defined in the App.yaml, but we need to discuss where this should be declared +### App.yaml definition, when using EI model +``` +name: Hey Arduino, ei model +description: When "Hey Arduino!" keyword is detected by the microphone, the led matrix will react +ports: [] +bricks: +- arduino:keyword_spotting: + EI_KEYWORD_SPOTTING_MODEL: "/models/hei_arduino_ei/builds/edge_impulse_hey_arduino.eim" +icon: ❤️ +``` +### App.yaml definition, when using QLC model +Just change the brick section as follow: +``` +bricks: +- arduino:keyword_spotting: +EI_KEYWORD_SPOTTING_MODEL: "/models/hei_arduino_qlc/builds/qualcomm_impulse_hey_arduino.model" +``` +## Case #5 Two distinct bricks using the same custom model +This case is the same as Case #3, where the variables used to customize the model are configurable by the user, with an optional default value. + +The following section explores whether this model is suitable for specific use cases. + +## Case #6 The user chooses a model for a brick (link Figma here) +In the brick configuration, there is a list of compatible models and the user selects the ```dog-detector``` model. +In this case, the model will be stored in ```App.yaml```. The brick section will look like: +``` +bricks: +- arduino:web_ui: {} +- arduino:object_detection: + model: dog-detector +- arduino:mood_detector: {} +``` +## Case #7 We need to know if a model build or a model definition can be deleted (link the Figma here): +To delete a build oir a model we need to know the state of the model in the system: +* **Downloaded** (present on the board and available for a brick to be used) +* **Installed** At least one brick is referencing the model. +* **In use** A brick instance is currently running with the model. + +The Downloaded state can be checked by verifying the model's presence on the board's file system (FS). +The Installed state can be inferred by checking all application ```App.yaml``` files for references to the model. + +*NOTE:* How is the "In Use" state determined? + +We can provide a list of models, their states, and all the builds on the File System (FS) (if we implement different versions/builds). +We can allow the deletion of only specific builds or the entire model structure. + +## Case #8 We want to support a different model/executor +We aim to support a learning model, where the model is defined by a set of weights and generated by a framework like TensorFlow. We intend to deploy and run this model within a web browser using JavaScript (TensorFlow.js). + +In this scenario, we can declare the model using its folder structure (API requests can be developed if required) and the necessary YAML files. + +The model (which is the set of weights in this case) can either reside directly on the board or be available via a URL defined in its metadata. Since we assume the model runs in a browser, we need a brick where a container is configured to retrieve the model and execute the container's JavaScript. + +The question here is how to make the model variables available to the container. There are several options: + +1. Folder Mounting: In the brick configuration, we can specify the model identifier, and this ID can be passed to the container to mount the related folder. The container can then access all the model variables and proceed to deploy or download the model. + +*NOTE:* Concurrency for different containers needs to be addressed here, but this is feasible. + +2. Environment Variables: Alternatively, we can make the entire model configuration available to the container as environment variables. + +# Questions to be discussed: +1. Should we keep "audio" as a generic category instead of using something more specific, such as ```ei-audio``` or ```qualcomm-audio```? +At this level, this has no relevance. + +Using the generic ```audio``` category allows us to make the same audio model compatible with bricks that use different AI execution engines. + +If this becomes technically impossible in the future, we can then add a more specific category to map the model to its required brick. + +2. What the "runner" variable represents? It is defined in the model. + ```runner: brick``` + +3. Why there is the +```require model``` in the brick definition? + +4. Variable Configuration: Should variables related to a model be placed in the model or in the brick? + +Since different bricks may use different variables to enable the container engine to start the AI engine, it is better to do not declare these variables in the model definition. + +When the user provides custom values, the variable definitions will be placed in the App.yaml file. See Use Case #3 App.yaml below, where one brick requires: +```EI_OBJ_DETECTION_MODEL: /models/yolo_x_nano/builds/yolo-x-nano.eim``` +and the other one requires: +```QLC_OBJ_DET_ENGINE_MODEL: /models/yolo_x_nano/builds/yolo-x-nano.eim +SET_CUSTOM_PARAMETER: 5 +``` +**App.yaml definition** +``` +name: Detect objects on images using two webcam, compare slow and fast engine over the same model +ports: [] +bricks: +- arduino:object_detection: {} + variables: + CUSTOM_MODEL_PATH: /home/arduino/.arduino-bricks/ei-models + EI_OBJ_DETECTION_MODEL: /models/yolo_x_nano/builds/yolo-x-nano.eim +- arduino:fast_object_detection: {} + variables: + CUSTOM_MODEL_PATH: /home/arduino/.arduino-bricks/ei-models + QLC_OBJ_DET_ENGINE_MODEL: /models/yolo_x_nano/builds/yolo-x-nano.eim + SET_CUSTOM_PARAMETER: 5 +icon: 🏞️ +``` + +5. Check the use case of the Figma where we go to EI and come back with a model + +# Proposed Changes to Bricks and Models Configuration +A Model definition contains a list of categories. + +A Brick definition containes a list of supported categories and a default_model_name. + + # Variables + This is a cross-cutting issue, not strictly related to models, but it's worth spending a moment here because it affects how models and bricks are managed. + +In the current implementation, all variables originating from the following sources are collected into a single map: +- Variables declared in the brick with a default model. +- Variables defined in the model. +- Variables defined or overridden by the user in the App.yaml file. +This map is then passed directly to the ```environment:``` section of the Compose file. + +If a variable can be overridden by the user, it must be declared in the brick definition, it is not required or mandatory to declare +the other variables that can not be visible/overrided by the user. + +*NOTE:* Due to a recent issue where using an empty string as a default value is possible, the proposal is to add the is_required flag to the brick definition file, as follows: +``` +- name: MY_VARIABLE + description: Variable with empty string value as default + default: "" + required: true +``` + +*NOTE:* Another proposal is to add all required brick variables to the brick definition file. The drawback is that the brick file could potentially become large. The advantage is that all variables are visible, which reduces the risk of conflicts when adding new variables to existing projects. +``` +- name: MY_VARIABLE + description: Variable is not user-overridable + private: true +```