Nutrition-AI Test Methodology

Testing and QA Processes and Procedures

We are releasing updated versions of the Nutrition-AI SDK on a regular basis. Before we release, we conduct a number of tests and validations to maintain high level of performance.

Here is a general overview of steps of our release and testing process for Nutrition-AI:

• Models for the Nutrition-AI SDK are trained every 4 weeks.

• After training we test the models on a testset of ~50,000 images of food examples that is close to our training data. This testset is coming from the same pool of data as our training and evaluation sets and is generated using a standard train/eval/test split approach. We typically achieve high accuracy (e.g., over 97%) on this testset as would be expected. This test is primary aimed at making sure there are no issues with the training run and our data pipelines.

• We also maintain a hold-out testset of ~70,000 images of much less canonical data that is more challenging for the models and represents more of a real-world use case. We use this to:

  • Test the our models independently, making sure that the average metrics are consistent between training runs. Actual accuracy of these tests is less important than consistency and directionality (i.e., accuracy should not be going down).

  • Test the our models, object detection and filters as an ensemble as they would be used in the SDK. This ensures that after models are integrated into the SDK the performance is likely to be consistent and good.

• Once these tests are passed, the models are handed over to the mobile team for integration into the SDKs.

• The mobile team carries out a number of unit tests are with a smaller number of images to ensure the models are correctly integrated:

  • images that the model has always recognized are used to verify consistency of results over time.

  • images of the first and last labels are tested to ensure consistency with the SQLite.

  • images of newly added foods are used to verify that the model can recognize new classes.

• If all of these tests pass then we have an internal SDK release and we test the release for about 1 week before releasing the SDK externally.

• The amount estimation algorithms are not a trainable algorithm. They stay consistent over time. The only trainable input is the bounding box from object detection, which is tested in the procedures outlined above.

• When amount estimation was first released we did tests on a range of common food items to verify they were measured within our target accuracy of 20%. We plan to extend this testing over time.

• We typically do not test on any subset of foods, but we can set up a dedicated accuracy reports on the foods specifically used by our clients if our clients request such reports.

• We are also able to add QA steps to test volume on foods requested by our clients.

If at any point you need help from the Passio team, please reach out to us at [e-mail]