Nutrition-AI Developer FAQ

Frequently Asked Questions

SDK Configuration

Can I bundle models and files with the app instead of letting the SDK to download them?

You can. When implementing the Passio sdk configuration, use the sdkDownloadModels = false. This would make the size of the app fairly high, but would remove the need for models to downloaded later. You still need wifi or LTE connectivity to use the models because models require valid license key checked regularly against Passio's license key API.

What is the size of the SDK?

The core SDK is fairly light-weight coming in under 10MB on both Android and iOS. The SDK does not come bundled with models and data files. Instead the SDK is configured to pull models from Passio's cloud servers when the SDK is activated with a valid key. The models are loaded in the background like levels in mobile games. The models and data files are typically between 40MB - 60MB.

What are the minimum iOS and Android environments you can support?

On iOS we are supporting iOS 13 and above. We recommend you build the SDK with the latest Xcode, but may be able to support earlier versions if needed. Please contact us at [e-mail]

On Android we are supporting Android 26 and above. We recommend you use the latest version of Kotlin, but we can support older versions (e.g. we recently did a deployment for 1.4.21).

How often do you release the SDK?

We typically release our updated SDK every two to three months. In some cases we shift to bi-weekly releases to address any critical updates. The update of our SDK is typically very straightforward and a simple process, it just requires a drag and drop replacement of the model. Some cases you may need to change a few lines of code, but this is not frequent and our technical team is always available to support that.

What are the implications of having an expired/revoked token for SDKs already published in the app?

There are two scenarios: (a) if you're not current with the payments and the tokens expire, the SDK will stop working, (b) when you get an updated SDK and associated keys, the old SDK keys will continue to work as long as the contract is current and so there will be no impact on the users who don't update to a new SDK.

Where does the image recognition happens ? Does it happen on the client-side SDK or, through an API call to the Passio server ?

Computer vision is performed on the client side (on user phones). We also have a database of foods and nutrition references bundled as a part of the client-side SDK so the entire process of visual logging can happen without API calls to the cloud. We provide around 40k common foods as a part of the client-side SDK. We also have a database of over 1.5M foods available via API calls to the Passio's Server. That database is used to log foods via UPC / Barcodes and can be used to augment manual search.

The Github repositories provided by Passio are only for mobile targets. If we plan to offload the computationally complex matter of categorizing all Meal objects via recognition net to a Cloud hosting provider such as AWS or Linode. Is this possible?

We provide mobile SDKs for integration; we don’t support cloud-side analysis of images at this time.

Are there any other API calls the SDK is making to the Passio server regarding analytics/usage statistics?

Yes. We are typically collecting limited performance statistics to improve the SDK, but before we do that we will be sharing the specifics with you. This provision is included in the contract. One metric we are monitoring is scan/log ratio - essentially the average performance of the model across the population. Of course, no personally identifiable information(PII) is captured with us.

Another API call is invoked from the UPC functionality. If you're using our UPC feature, the packaged foods data is stored on the cloud and requires an API call.

What is the threshold about response time for the APIs?

Since computer vision runs on devices the response times are nearly instantaneous. We process image input at 2 frames per second. Response times for packaged foods depend on the network, but are typically sub second.

How does the learning and correlations work with client’s data and business logic ?

The way to approach learning, correlations and all downstream activities from your side would be by using the actual user logs. The SDK generates detailed user logs that are stored in your applications and are fully managed and controlled by you on the client side (logs or any user data are not sent to Passio). You would use those logs to build models, create recommendations and conduct studies to correlate nutrition logs with the metabolic and other data. Of course, you can send the logs from the client to your servers or third party servers as you would with any other information in your app.

Is there a way for the app to save the image as well? Being able to see the food will give more context for the HCP as well as add another way to log in the event that the AI does not scan things well enough.

• Passio won’t be collecting the user’s scanned images of the food items. We only provide the SDK.The customer's application has to handle the image capture and store it themselves. Hence, the application has to design the image capture/storing feature accordingly.

How does the data model look like? What is the role of parent, sibling and child relationships?

The data model is based upon the difference of 2 types of objects - nutrition references, and visual labels.

Nutrition references are objects that have nutrition data such as nutrients, measurements. There are 2 different types of nutrition references - references with a product code, hence a Branded Product, or average generic nutrition references that are referring to such things e.g. “raw spinach” (in general).

Visual Labels are labels that are in our visual hierarchy. They refer to visual objects, and are too ambiguous to have nutrition data themselves.

Below the leaf nodes of our visual hierarchy, there is a mapping to the nutrition references, in order to make sure that every visual label has nutrition data attached to it. It’s a many-to-many mapping. A visual label can have several nutrition references attached to it (e.g. the label “almond milk” can have different nutrition references attached to it, representing different branded almond milks). Also a branded almond milk nutrition reference can be attached to both the visual label “ almond milk (in glass)” and “boxed milk” (inside of packaging box). In addition to a mapping to nutrition references, some visual labels have references to recipes (which are essentially a set of nutrition references and specific measurements that define the composition of the recipe).

The different paths of recognition can recognize different types of objects.

e.g. OCR scanning or barcode scanning will recognize a nutrition reference with product code. In this case, parent, sibling and child relationships are not relevant.

Visual scanning will recognize a visual label inside of the visual hierarchy. The parent of it represents the parent object one visual level higher in the hierarchy. E.g. if you recognize “raw spinach leaves” - the parent might be “raw greens”. Siblings of the visual label might be alternative visual labels that can look visually very similar, such as “baby kale”. Children of the visual label are going to be either a) another set of visual labels, hence alternatives, in case the recognition can only recognize a high level visual label, or b) already nutrition references. In the case of “raw spinach leaves”, you will find nutrition references as children. In the case of the visual label “milk”, children will be “almond milk”, “soy milk” etc, and then the children of those labels will be the nutrition references.

If you text searching for a nutrition reference, then there will be no children, because nutrition references are the last layer of information. The parent of the nutrition reference will be the visual label that the nutrition reference is mapped to. And the siblings of the nutrition reference will represent other nutrition references that are attached to the same visual label, and can hence be considered alternative nutrition references.

How do we identify healthy food items?

We have a team of Registered Dietitians who create lists for us, but this has not been our focus until recently. We typically prefer our customers to make decisions on what is and is not healthy based on their models. For example, many dietitians consider keto to be unhealthy. Healthy diets are different for patients with different conditions and allergies.

Can you provide a sample of meals and recipes?

In the Nutrition AI SDK, we have the concept of a recipe. Those are set of nutrition references and specific measurements that define the composition of the recipe. Recipes can be attached to visual labels. This is specifically helpful, if for a given visual label, e.g. "bread with cream cheese" - there is no accurate branded product or no generic average reference available.

Meal Plans are a set of specific nutrition references and or recipes put together for a specific dietary purpose.

Can we get alternatives for specific items in meals as part of the API?

Yes.You would send an internal client-side API call with the name of the food item and get an object back including alternatives. You can drill down through the tree this way.

How can we interact with the API and learn user preferences?

You would learn user preferences by analyzing user logs and user behavior of using the client-side logging experience of the users that are controlled by your application. You do not need to interact with the API for this.

With respect to supplements, would a user be able to search for multiple nutrients to populate a multi-vitamin option?

The search is only searching for food names - this means, if the name of the supplement includes those nutrients, it would show up in the search results, e.g. “Brand X - Vitamin C, Iron and Calcium supplement”, but if the name of the food does not include those nutrients, then it would not show up, because we are not including nutrients right now into the search at all.

Would it be possible for Passio to provide a list of all the parameters associated with supplements?

Currently supplements are not separated from other packaged foods, this means you can expect the exact same information / parameters for supplements as for any packaged food you scan. In the future, if relevant for customers, we might surface additional nutrients for foods and supplements than the ones which are currently provided by the SDK.

Is the supplement information stored in the OpenFoodFacts database that is supplied with the SDK or is it in a separate database?

We are surfacing information from our own database, which is the result of importing, merging and curating information from USDA FDC database, Open Food Facts and data that our internal data team put together (e.g. by research on manufacturers websites etc). However, this does not mean that the data the SDK is surfacing is exactly the same as on FDC or Open Food Facts, because we do apply internal curation logic and might add additional resources, such as measurement names, etc.

If you are scanning a food, the SDK surfaces whether the original imported information was from Open Food Facts or from USDA FDC database.

If a user scans a multivitamin nutrition label, would he/she be able to see all the nutrients (and quantity of) that make up that multi-vitamin?

The SDK currently surfaces a specific set of about 20-30 nutrients, only those are provided in the current SDK, if present on the given food/supplement - they are provided together with their quantities. We are storing all nutrients in our backend DB and in the future, could provide all of them.

How can we use our local Nutrient File for nutrient analysis. Please let us know what we need to tweak in the Passio SDK for this?

Passio nutrition references are:

• Barcodes - Not sure if your local Nutrient File is as extensive as e.g. OpenFood Facts (we use OpenFood Facts), so you would probably not be able to map all of the barcodes to your Database.

• USDA FDC - We think that there will be foods where mapping is not possible, same as with barcodes.

• Passio Created Foods and Recipes - The only thing that could help with mapping is the name of the food.

Passio SDK returns product code/barcode and FDC ID of a food, if available.

The mapping would need to be done outside of the Passio SDK and inside of your application. Also, your team would need to investigate how to store and present the information for a food in the local Nutrient File. The data itself might be similar to the Passio Nutrition info, but possibly in a different format, so you would need to build your own UI on top of the mapping.

Would it be possible for Passio to provide the high-level food groups for customers to use?

Yes, we do populate the high-level food groups on the backend as tags. Here is the current list of food group tags:

alcoholic beverages

baby foods

baked goods

beverages

breakfast cereals

candies and sweets

cereal grains and pasta

cooking and baking ingredients

dairies and products

dairy substitutes

desserts

eggs and products

fats and oils

food additives

fried foods

fruits and products

legumes and products

meals

meat and products

nuts and seeds products

pickles

sauces, dips, spreads, gravies

seafood and products

seaweeds and products

snack and protein bars

snacks

soups

spices and condiments

stuffing

supplement

sweeteners

vegetables and products

Does the Passio SDK provide tags for NOVA scores related to food?

Yes, Passio SDK offers NOVA scores related to food. Here is the list of tags for Nova Scores:

NOVA1 - These are whole foods that have undergone little or no processing, such as fruits, vegetables, nuts, and seeds.

NOVA2 - This group includes substances extracted from foods, like oils, fats, sugar, and salt.

NOVA3 - Processed foods are products made by combining unprocessed or minimally processed foods with Group 2 ingredients. Examples include canned vegetables and cheese.

NOVA4- These are highly processed foods that often contain additives, preservatives, and flavorings. Examples include many packaged snacks, soft drinks, and fast food items.

Reference: https://world.openfoodfacts.org/nova

How can we have the small food icons in a rounded corner square instead of a circle in our app?

The SDK is responsible for getting the original image in three different resolutions ( 90, 180 and 360 pixels). Adding rounded corners or circle to the image is the responsibility of the app, and can be easily done with the original image. You can do it from your side in your app. Here is an example of a code that makes it easy in Swift: https://medium.com/swlh/create-a-circular-profile-image-swift-uikit-19ae4eaa8bb1

If at any point you need help from the Passio team, please reach out to us at [e-mail]