Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Welcome to Passio Nutrition-AI iOS SDK! When integrated into your app the SDK provides you with food recognition and nutrition assistant technology. The SDK creates a video preview layer and outputs foods recognized by our computer vision technology in the video feed of your live camera along with nutrition data related to the recognized foods.
As the developer, you have complete control of when to turn on/off the SDK and to configure the outputs which includes:
food names (e.g. banana, hamburger, fruit salad, quest chocolate bar)
lists of alternatives for recognized foods (e.g., soy milk would be an alternative of a visually recognized class milk)
barcodes detected on food packages
packaged foods recognized by the text detected on food packages
nutrition information detected on food packages via Passio's Nutrition Facts reader which returns information written in Nutrition Facts labels
nutrition information associated with the foods
food weight and volume for certain foods
By default the SDK does not record/store any photos or videos. Instead, as the end user hovers over a food item with his/her camera phone, the SDK recognizes and identifies food items in real time. This hovering action is only transitory/temporary while the end user is pointing the camera at a particular item and is not recorded or stored within the SDK. As a developer, you can configure the SDK to capture images or videos and store them in your app.
This guide will provide documentation for the latest version of the SDK. If the documentation is needed for an older version, be sure to check the VERSIONS section.
Passio Nutrition-AI SDK added data from Open Food Facts (https://en.openfoodfacts.org/). Each food that contains data from Open Food Facts will have the value set in its food origin list. In case you choose to display the these food items you agree to abide by the terms of the Open Food Facts license agreement (https://opendatacommons.org/licenses/odbl/1-0) and their terms of use (https://world.openfoodfacts.org/terms-of-use) and you will have to add to the UI the following license copy:
"This record contains information from Open Food Facts (https://en.openfoodfacts.org), which is made available here under the Open Database License (https://opendatacommons.org/licenses/odbl/1-0)"
When integrated into your app the SDK provides you with food recognition and nutrition assistant technology. The SDK creates a video preview layer and outputs foods recognized by Passio's computer vision technology in the video feed of your camera along with nutrition data related to the recognized foods.
As the developer, you have complete control of when to turn on/off the SDK and to configure the outputs which includes:
food names (e.g. banana, hamburger, fruit salad, Quest chocolate bar)
lists of alternatives for recognized foods (e.g., soy milk would be an alternative of a visually recognized class milk)
barcodes detected on food packages
packaged foods recognized by the text detected on food packages
nutrition information detected on food packages via Passio's Nutrition Facts reader which returns information written in Nutrition Facts labels
nutrition information associated with the foods
food weight and volume for certain foods
By default the SDK does not record/store any photos or videos. Instead, as the end user hovers over a food item with his/her camera phone, the SDK recognizes and identifies food items in real time. This hovering action is only transitory/temporary while the end user is pointing the camera at a particular item and is not recorded or stored within the SDK. As a developer, you can configure the SDK to capture images or videos and store them in your app.
Hit the big '+' button in your sidebar and select 'New Project' from the menu that pops up. Give your project a name, and you're good to go!
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
To use the SDK through license key, please make sure you receive your SDK license key from Passio.
To obtain an SDK key, visit https://www.passio.ai/nutrition-ai
If you want a layer of security, where you can authenticate the license and fetch the token on your backend, you don't need the SDK key.
In that case, the SDK will communicate with your backend, and your backend will proxy the requests to Passio's servers, adding the access token to the header. This way, the license keys and tokens are never shared in the app code.
You need to provide following things when configuring the SDK:
Proxy URL - Base url of the target proxy endpoint
Proxy headers - Required headers to all of the requests
Responisilities of proxy backend:
License authentication - the license will be authenticated on the proxy backend
Token fetching - token will be fetched on the proxy backend and appended to every call
The configuration for the proxy URL will be covered in the "Configure the SDK" section.
These are the minimum requirements for every supported platform
The SDK will only run or compile on iOS 13 or newer.
Passio SDK can only be used on a device and will not run on a simulator
The SDK requires access to iPhone's camera
Weight/Volume estimation will run only on iPhones with Dual Wide Camera (not on DualCamera). It will not run on the models below.
iPhone 11 Pro & Pro Max
iPhone 12 mini, Pro & Pro Max
iPhone 13 Pro & Pro Max
iPhone 14, Pro, Pro Max & Plus
Built with Kotlin version 1.6.10
Minimum Android SDK version is 26
The SDK requires access to the device's camera
The SDK is built upon CameraX, TensorFlow and Firebase's ML Vision so these dependencies will need to be added to the project manually
The SDK will automatically download the models according to the SDK version, by default the SDK will download compressed files. The download is faster and lighter, and it will take several seconds to decompress on the device. To download uncompressed files and shorter processing on the device you can set a flag below to false.
After obtaining special approval the developer can also host the models on an internal server.
In that case, the models will be served directly to the SDK. To find out more about this special configuration please contact your primary point of contact at Passio or reach us via support@passiolife.com
At the top of your view controller import the PassioNutritionAISDK and AVFoundation
Add the following properties to your view controller.
Create a function to configure the SDK and call it from viewDidLoad. Use the SDK key you received from Passio. We'll add the function for checkCameraAuthorizationAndStartDetection in the next step, so ignore the error for now.
You will receive the PassioStatus back from the SDK.
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
The SDK can detect barcodes located on the packaging of packaged foods. To implement real-time barcode scanning, the camera setup for continuous image recognition is needed.
Specifically for barcode scanning, these steps are required:
When defining a FoodDetectionConfiguration
object, enable the detectBarcode
flag.
In the recognitionResults
callback, check the property candidate.barcodeCandidates
. If this list is empty, no barcodes have been recognised.
If there is a single or multiple barcode candidates, use their value property and invoke this function to fetch the full nutrition data.
On smaller packages, barcodes can be very small. When trying to scan these barcodes, the user usually brings to package to close to the camera creating focus issues. To avoid this, but the camera zoom level to 2.
To recognise a barcode in an image, refer to the recognizeImageRemote function. The result of the barcode scanning process will come in the packagedFoodItem
attribute, and the resultType will be Barcode.
Set up the camera preview and detection session to recognise barcodes
Create a result view with two states: search and result
If the candidate.barcodeCandidates is empty, show the search state. Else, fetch the PassioFoodItem, and use it's name and icon to show the result
Example of an image that produces multiple BarcodeCandidates:
Starting from the SDK version 3.1.4 the NutritionAdvisor is only accessible using the NutritionAI Hub key. A key bought for the previous NutritionAI product won't work with this feature.
The Nutrition Advisor is an intelligent chatbot designed to offer personalised guidance on nutrition, diets, and healthy eating habits. Whether you're looking to explore a new diet, understand the nutritional value of certain foods, or find delicious and balanced recipes, the Nutrition Advisor is here to assist. Users can interact with the Nutrition Advisor through text messages, engaging in a conversational experience where they can ask questions about various diets like keto, vegan, Mediterranean, and more. The chatbot provides detailed insights into the benefits, potential drawbacks, and suitability of these diets based on individual goals and preferences.
As opposed to the rest of the SDK, the Nutrition Advisor has a different API entry point:
NutritionAdvisor
Use these steps to initialise the conversation and enable users to send messages:
To start a conversation initConversation
has to be invoked. The response will either indicate a success state, which means message can be sent to the advisor. In case of an error, an error message will detail the cause.
Text messages are sent using the sendMessage
API. The reply (if successful) will contain a PassioAdvisorResponse
response. This object consist of:
threadId - identifies the conversation
messageId - identifies the specific message
markupContent - text response from the advisor denoted with markup tags. This can be used to style the text when rendering
rawConent - text response without the markup tags
tools - list of actions that can be executed by the advisor. Only should be used by the SDK
extracedIngredients - will contain a list of foods if the request to the advisor was an image recognition request or extract ingredients from text message request
Image are sent to recognition using sendImage
. As mentioned above, in the PassioAdvisorResponse object, the extractedIngredients
property will hold the recognised foods.
In the case that a PassioAdvisorResponse contains a tool called searchMatchIngredient
, the Advisor has the ability to parse the text response and return the list of foods. If a message that has this tool is passed to the fetchIngredients
API, the returned response will contain foods in the extracedIngredients field.
This feature is particularly useful when trying to log foods when directly communicating with the chatbot. A common use case is when user ask for a specific recipe. They can then initiate the extraction of foods from the message, and see their nutrition value and finally log the entire meal in one action.
Create a screen where the user can type messages and send images to the NutritionAdvisor. Upon initiating the screen, set up the conversation by calling initConversation
.
When the user types a message in the input text field and taps "Send", invoke the sendMessage
function. Because this is an asynchronous function, add an animation to indicate this behaviour.
If the response from the advisor contains a tool called searchMatchIngredient
show an action button "Find foods" to initiate extraction of the foods. Invoke the fetchIngredients
function using the response from the Advisor.
Create a view where the user can see the extracted foods, their serving size and have the ability to log them.
Add the ability for the user to select an image from the library of the phone. Once the image is obtained, invoke the sendImage
to get the list of foods. If foods are recognised, the response will contain a list of foods, use the same result view to show them.
The Passio SDK has the ability the recognise anything from simple ingredients like blueberries and almonds to complex cooked dishes like a beef burrito with salad and fries.
There are two types of food recognition, each of them with their own strengths and weaknesses.
No, the image is sent to the backend for recognition
Very precise with all types of foods
On average 4-7 seconds
Yes, the recognition is done on the device
Good with single food items, struggles with complex cooked foods
Depending on the hardware of the device, ranges between 50-300ms
The Remote image recognition approach is good when the accuracy of the results is top priority, and waiting for the response is not an issue. This use case is implemented by taking static images from the camera or the library of the device and sending them for recognition in an asynchronous fashion.
The Local model approach is good when speed is of the essence. This use case is implemented using continuous frame recognition from the camera. A callback is registered to capture the results as they are coming from the camera stream.
This API sends an image as base64 format to an LLM on Passio's backend, and returns a list of recognised items.
The API can recognise foods raw or prepared, barcodes or nutrition facts tables. The type of recognition will be shown in the resultType
enum.
The default behaviour of this function is to resize the image to 512 pixels (longer dimension is resized, the other is calculated to keep aspect ratio). Using the PassioImageResolution
enum, the image can be either resized to 512, 1080 or keep the original resolution.
The response, presented as a list ofPassioAdvisorFoodInfo
objects, contains the name, portion and weight in grams recognised by the LLM. These attributes can be used for debugging, but the data from the nutritional database is contained either in the foodDataInfo
if the result type is a Food Item, or packagedFoodItem
if it's a Barcode or Nutrition Facts. To fetch the nutritional data for the PassioFoodDataInfo object, use the fetchFoodItemForDataInfo
function.
Create a screen where the user can snap one or multiple images using the camera of the device
Upon clicking next, the recognizeImageRemote
is invoked on each of the images in the list
Wait for all of the responses to come, add each results list to a final list of results. When the last asynchronous function is executed, present the final list to the user.
To set up the local model and the continuous scanning mode, the camera preview and the recognition session need to be defined.
To start using camera detection the app must first acquire the permission to open the camera from the user. This permission is not handled by the SDK.
Add the UI element that is responsible for rendering the camera frames:
To start using the camera in your Activity/Fragment, implement the PassioCameraViewProvider interface. By implementing this interface the SDK will use that component as the lifecycle owner of the camera (when that component calls onPause() the camera will stop) and also will provide the Context in order for the camera to start. The component implementing the interface must have a PreviewView in its view hierarchy.
Start by adding the PreviewView to your view hierarchy. Go to your layout.xml and add the following.
This approach is more manual but gives you more flexibility. You need to implement the PassioCameraViewProvider interface and supply the needed LifecycleOwner and the PreviewView added in the initial step.
After the user has granted permission to use the camera, start the SDK camera
PassioCameraFragment is an abstract class that handles Camera permission at runtime as well as starting the Camera process of the SDK. To use the PassioCameraFragment simply extend it in your own fragment and supply the PreviewView that has been added to the view hierarchy in the previous step.
To show the live camera preview, add the DetectionCameraView
to your view
The SDK can detect 3 different categories: VISUAL, BARCODE and PACKAGED. The VISUAL recognition is powered by Passio's neural network and is used to recognize over 4000 food classes. BARCODE, as the name suggests, can be used to scan a barcode located on a branded food. Finally, PACKAGED can detect the name of a branded food. To choose one or more types of detection, a FoodDetectionConfiguration object is defined and the corresponding fields are set. The VISUAL recognition works automatically.
The type of food detection is defined by the FoodDetectionConfiguration
object. To start the Food Recognition process a FoodRecognitionListener also has to be defined. The listener serves as a callback for all the different food detection processes defined by the FoodDetectionConfiguration. When the app is done with food detection, it should clear out the listener to avoid any unwanted UI updates.
Implement the delegate FoodRecognitionDelegate
:
Add the method startFoodDetection()
In viewWillAppear
request authorisation to use the camera and start the recognition:
Stop Food Detection in viewWillDisappear
:
Using the listener and the detection options start the food detection by calling the startFoodDetection method of the SDK.
Stop the food recognition in the onStop() lifecycle callback.
Add the method startFoodDetection()
and register a FoodRecognitionListener
Stop Food Detection on widget dispose:
The FoodCandidates
object that is returned in the recognition callbacks contains three lists:
detectedCandidates
detailing the result of VISUAL detection
barcodeCandidates
detailing the result of BARCODE detection
packagedFoodCandidates
detailing the result of PACKAGED detection
Only the corresponding candidate lists will be populated (e.g. if you define detection types VISUAL and BARCODE, you will never receive a packagedFoodCandidates list in this callback).
A DetectedCandidate represents the result from running Passio's neural network, specialized for detecting foods like apples, salads, burgers etc. The properties of a detected candidate are:
name
passioID (unique identifier used to query the nutritional databse)
confidence (measure of how accurate is the candidate, ranges from 0 to 1)
boundingBox (a rectangle detailing the bounds of the recognised item within the image dimensions)
alternatives (list of alternative foods that are visually or contextually similar to the recognised food)
croppedImage (the image that the recognition was ran on)
To fetch the full nutrition data of a detected candidate use:
Implement the camera screen using the steps above
Create a result view that can have two states: scanning and result
If the callback returns an empty list, show the scanning state. If it returns the result, display the name from the detectedCandidate.name
Example of an image that produces a DetectedCandidate:
The main object containing the nutritional information is called PassioFoodItem. This object is fetched from Passio's backend using one of several fetchFoodItemFor...
functions.
The PassioFoodItem is a top level object that can represent a single food like a banana, or a complex recipe like caesar salad.
The name
attribute is used to display the name of the food, while the details
contains additional information like the brand of the food or the name of the food which was used to populate the nutritional data
iconId
is used with the SDK function fetchIconFor
to fetch a small image of the food
refCode
is a parameter that can be used later on to retrieve the full food item again
amount
details the available serving size as well as the currently selected serving quantity and unit, used when calculating nutrients
There are three functions that can be used to fetch nutrient information for the PassioFoodItem:
The weight
function is used to fetch the sum mass of all of the ingredients.
This class holds the information about possible serving sizes and serving units, as well as the currently selected unit and quantity. Example of values for food item "apples"
serving units: small (2-3/4" dia), medium, large (3-1/4" dia), cup, oz, gram, ...
serving sizes: 1 small (2-3/4" dia), 2 cups, 100 grams, ...
selected quantity: 1
selected unit: medium
Serving units holds the list of all of the units that the system has the correct weight for. Serving sizes holds the list of predefines matches of a quantity and a serving unit. The selected quantity and unit are used to calculate nutrients on the PassioFoodItem level. While these values can be changed, the initial values are set by the SDK as a default portion.
Every PassioFoodItem has a list of ingredients. A food ingredient holds the nutritional information along with the portion size of each ingredient.
But, the most important part of the ingredient class is the PassioNutrients
and it's methods. The referenceNutrients attribute holds the referent (in most cases 100 grams) macronutrients and micronutrients, and are used to calculate the resulting nutrient values when changing serving sizes of the whole PassioFoodItem.
Also, an ingredient has PassioMetadata field, storing the information on the origin of the nutritional data, barcode value, possible ingredients (if the PassioFoodItem is a packaged food), and a list of tags.
The PassioFoodItem class encompasses both single food items like an avocado
as well as items with multiple ingredients such as homemade caprese salad
. The difference between these two items is how is the PassioFoodAmount calculated from the top-level PassioFoodItem.
Homemade caprese salad:
weight(): 176.5 grams
amount->selectedQuantity = 1
amount->selectedUnit = "serving"
ingredients: 1. fresh sliced cheese, 84 grams, amount->selectedQuantity = 3, amount-> selectedUnit = "oz" 2. balsamic vinegar, 16 grams, amount->selectedQuantity = 1, amount-> selectedUnit = "tbsp" 3. fresh basil, 3 grams, amount->selectedQuantity = 6, amount-> selectedUnit = "leaves" 4. tomatoes, 60 grams, amount->selectedQuantity = 3, amount-> selectedUnit = "medium slice" 5. olive oil, 13.5 grams, amount->selectedQuantity = 1, amount-> selectedUnit = "tbsp"
Invoking the function foodItem.nutrientsServingSize() will fetch the nutrients for the whole recipe, calculated by the weight of the serving size "1 serving".
Avocado:
weight(): 201 grams
amount->selectedQuantity = 1
amount->selectedUnit = "avocado"
ingredients:
1. avodaco, raw
, 201 grmas, amount->selectedQuantity = 1, amount->selectedUnit = "avocado"
A single food item will always have only one ingredient. Also, the amount object of the PassioFoodItem will be the same as the amount of the first ingredient. The ingredient is the item in the database used for nutritional data. For example, If the SDK recognises "milk" during the visual detection process, the default food item for "milk" is "milk, whole, 3.25% milkfat, without added vitamin a and vitamin d".
Fetch the PassioFoodItem
Use the nutrientsSelectedSize->calories, carbs, protein and fat to show the marcos
Use the amount class to create the serving size view
Give the user the option to log the food
The Passio Nutrition AI SDK search functionality gives access to over 2.2 million food records in our nutritional database. The search api takes in a text query and returns a list of PassioFoodDataInfo results as well as a list of suggested searches.
There are two different search APIs available in the SDK:
searchForFood
- tries to give best string matches based on the input parameter, often providing the same food from different brands. Example: Input parameter "eggs" will provide results like "Trader Joe's eggs", "Eggland's Best eggs", "Eggs in carton, Lucerne", ...
searchForFoodSemantic
- will try to contextually match the most similar foods to the input parameter. Example: Iinput parameter "eggs" will provide results like "Cooked eggs", "Scrambled eggs", "Eggs benedict", ...
Both searchForFood
and searchForFoodSemantic
have the same input parameters and result structures.
The PassioFoodDataInfo is used a reference object to fetch the full nutritional data. It has all of the attributes needed for displaying on a typical search screen: the name of the food, name of the brand (if possible), iconId, and a nutritional preview detailing the macro nutrients for a specified serving size and weight.
To fetch the PassioFoodItem using a search result, invoke the fetchFoodItemForDataInfo
.
The search will return 50 items that best match the input term, as well as a list of suggested searches that are semantically or contextually close to the input term.
Example: searching for term "coffee"
search results: "Coffee, Brewed", "Coffee, Breakfast Blend", Coffee - Brewed From Grounds", "Coffee, Brewed, Decaffeinated", "Coffee, Instant, Reconstituted Decaffeinated", ...
suggested searches: "coffee with whole milk", "coffee with creamer", "iced coffee"
Create an input text field to collect the user query
Create a horizontal list scroll to display the suggestions
Create a vertical list to display the results of the search
If the user clicks on a suggestion, pass that string to the search API and reload the UI with the new results
The SDK gives the ability to fetch food icons for a specific passioId
. There are two functions that can be used: lookupIconsFor
and fetchIconsFor
.
lookupIconsFor does a local search of the icons without doing any networking calls. It returns two icons. One is a placeholder icon depending on the type of food associated with the passioId. The second one is cached icon from a previous fetchIconsFor call, it it exists.
fetchIconsFor does a networking call to fetch an icon associated with the passioId, but it will check the local cache and return a previously fetched icon.
If displaying a PassioFoodItem, PassioIngredient or a PassioFoodDataInfo icon, pass the iconId
to the appropriate icon function.
Example of how to use both of these functions to first display the placeholder icon, and then fetch the remote icon:
A component for displaying food icons from the Passio SDK.
There are several functions that are used to retrieve the PassioFoodItem object, depending on the origin of the data. If the food item is being scanned using an image, the food item is fetched using either fetchFoodItemForPassioID
or fetchFoodItemForProductCode
. If search is being used to access to food item, the appropriate call is the fetchFoodItemForDataInfo
.
But if there is a use case that requires to save just one attribute to a database, that will later on be used to retrieve the full PassioFoodItem object, then the refCode
attribute is the answer. The refCode in combination with the fetchFoodItemForRefCode
will retrieve the original food item object from Passio's nutritional database.
refCode is a string hash of a complex object and can reach lengths above 180 characters.
If the implementing app is transitioning from generation 2
to generation 3
of the SDK, and has already stored generation 2 PassioID
s, invoking the fetchFoodItemLegacy
will retrieve the PassioFoodItem that was previously structured as PassioIDAttributes.
Follow these steps to include the SDK into your mobile application project.
Open your Xcode project.
Go to File > Swift Packages > Add Package Dependency.
In the "Add Package Dependency" dialog box, paste the URL: https://github.com/Passiolife/Passio-Nutrition-AI-iOS-SDK-Distribution
Click "Next". Xcode will validate the package and its dependencies.
In the next dialog box, you'll be asked to specify the version or branch you want to use. You can choose main for the latest version or specify a specific version or branch.
After you've made your selection, click "Next".
You'll then be prompted to select the targets in your project that should include the package. Check the boxes for the targets you want to include.
Click "Finish" to add the package to your project.
Xcode will download and add the PassioNutritionAISDK to your project. You can now import and start using the PassioNutritionAISDK.
Download the "PassioNutritionAISDK.xcframework.zip" file from https://github.com/Passiolife/Passio-Nutrition-AI-iOS-SDK-Distribution/blob/main/PassioNutritionAISDK.xcframework.zip
Unzip it and drag and drop it into your project. Make sure to select "Copy items if needed".
In project "General" -> "Frameworks, Libraries and Embedded Content" Change to "Embed & Sign"
If opening from Xcode, right click and select 'open as source code'
To allow camera usage add:
The Passio Android SDK is shipped in the form of an .aar file.
In the build.gradle file add the PassioSDK dependency
Sync Project with Gradle Files and the PassioSDK interface should be available for usage.
Visit the releases page of the Android-Passio-SDK-Distribution GitHub repository. Download the passiolib-release.aar file from the latest release.
To include the .aar file into your project go to the module that will contain the food recognition feature. Place the .aar file in the libs folder of the module. If the libs folder is missing, create one. Reference the .aar file in the build.gradle file of the implementing module.
Sync Project with Gradle Files and be sure that you can reference the PassioSDK class within your code.
Passio Android SDK is powered by TensorFlow and FirebaseVision with the camera being managed by CameraX. Add the dependencies to these three projects by adding these lines to the module's build.gradle file.
In order for the SDK to work add the following lines to the android section of the module's build.gradle file.
Running Sync Project with Gradle Files will enable access to the SDK.
1. Create a Github Personal Access Token (classic) with read:packages
access selected.
2. Create an .npmrc
file in the root of your project with the following lines replacing GITHUB_ACCESS_TOKEN
with the token you've created.
3. Add the Passio SDK dependency to your package.json
and run npm install
or yarn
.
or
4. Ensure the native dependencies are linked to your app.
For iOS, run pod install.
If pod install is failed with error message Specs satisfying the ReactNativePassioSDK (from ../node_modules/@passiolife/nutritionai-react-native-sdk-v3) dependency were found, but they required a higher minimum deployment target.
, ensure Podfile is configured with minimum ios version '13.0'
For Android, add this implementation line to the dependencies section on app/build.gradle
file.
Also for Android, make sure the android project is configured with minSdkVersion 26 or above (check the project's build.gradle
file).
The Flutter SDK is available as a package on pub.dev
This will add a line like this to your package's pubspec.yaml (and run an implicit flutter pub get
):
Alternatively, your editor might support flutter pub get
. Check the docs for your editor to learn more.
Now in your Dart code, you can use:
Add to top build.gradle file (Project: android)
The nutrition_ai build.gradle file dependency section should look like (Module: nutrition_ai)
Nutrition-AI SDK Overview
Passio Nutrition-AI SDK is an industry leading food recognition and nutrition tracking SDK designed to enable app developers to rapidly add AI-powered food logging capabilities and AI-powered nutrition coaching to their mobile applications.
The SDK is a part fo product and relies heavily to a wide range of APIs and functionality provided in the .
Our Nutrition-AI SDK offers powerful capabilities that are rapidly evolving, so some features may not appear in the overview video below.
A key shift from our legacy SDK is the move from real-time on-device food logging to photo logging analyzed via cloud-side LLMs. While real-time scanning is still available as of Q4 2024, it’s no longer the primary mode. Passio now recommends photo logging, voice logging, and text search as the primary modes.
Follow our documentation guides to get started and build your AI-powered applications. Please select your platform of choice to get to the documentation.
Open your Xcode project.
Go to File > Swift Packages > Add Package Dependency.
In the "Add Package Dependency" dialog box, paste the URL:
Click "Next". Xcode will validate the package and its dependencies.
In the next dialog box, you'll be asked to specify the version or branch you want to use. You can choose main for the latest version or specify a specific version or branch.
After you've made your selection, click "Next".
You'll then be prompted to select the targets in your project that should include the package. Check the boxes for the targets you want to include.
Click "Finish" to add the package to your project.
Xcode will download and add the PassioNutritionAISDK to your project. You can now import and start using the PassioNutritionAISDK.
Download the "PassioNutritionAISDK.xcframework.zip" file from
Unzip it and drag and drop it into your project. Make sure to select "Copy items if needed".
In project "General" -> "Frameworks, Libraries and Embedded Content" Change to "Embed & Sign"
If opening from Xcode, right click and select 'open as source code'
To allow camera usage add:
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
The SDK provides the ability to fetch a predefined set of suggested foods for logging, depending on the current time of day
Example of fetching suggestions for snack meal time:
banana, coffee, black tea, coke, cheddar cheese, chocolate chip cookies, potato chips, blueberry muffin, plain yogurt, strawberries, ...
UX Tip: Combine these predefined suggestions with user meal logs to provide a list of "quick suggestion" logs. This will enable users to log their meals even faster.
Meal Plans provide users suggestions which foods to consume depending on their dietary preference and time of day. The Meal Plan api is divided into two functions:
fetchMealPlans
that is used to retrieve all of the possible meal plans. These meal plans usually correlate with a specific diet like "Keto Diet".
fetchMealPlanForDay
is an api that is used to fetch specific foods recommended for a certain day of a target meal plan.
Some of the available meal plans:
Heart Healthy Diet, Ketogenic Diet, Managing Obesity, Managing Type 2 Diabetes, Low FODMAP, Healthy Kidney Diet, Balanced Diet, DASH Diet, PCOS Diet, Mediterranean Diet
The configuration process of the SDK has severable responsibilities when configured through SDK key:
Validation of the license key
Validating the currently present files
Downloading or updating the files required by the SDK to do recognition
Preparing the files to be used for inference
The SDK is configured using a PassioConfiguration
object. This object is used to define the location of the files that the SDK requires.
When the SDK is configured without SDK key (using Proxy URL), the SDK will be remoteOnly
. More information on this flag is described in below section.
The default behaviour (or by setting the sdkDownloadsModels field to true) of the SDK is to download the required files for . If the current files that are present on the device are older than the version of the SDK library, the SDK will download the newer version in the background, but still run the session with the older version of the files. Once the download is finished, the new files will be applied on the next session. Don't forget to add the import statement for the nutrition sdk.
The SDK will not download required files for when configured without SDK key. The SDK will be in remoteOnly
mode.
The models required for the SDK to run local recognition can take some time for the first download. If the local recognition is not needed, setting the remoteOnly
to true on the PassioConfiguration process will skip the download of the models. Remote recognition functions like recognizeImageRemote or searchForFood will still work.
The result of the configuration process is a PassioStatus
object, containing a PassioMode attribute along with the missingFiles, debugMessage, error object and the active models number. The attribute that needs to looked at is the mode.
PassioMode
is an enum describing the current state of the Passio SDK configuration process:
notReady -> The configuration process hasn't started yet.
failedToConfigure -> There was an error during the configuration process.
isBeingConfigured -> The SDK is still in the configuration process. Normally, you shouldn't receive this mode as a callback to the configure method. If you do please contact our support team.
isDownloadingModels -> The files required by the SDK to work are not present and are currently being downloaded.
isReadyForDetection -> The SDK is configured with the set of files defined by the activeModels attribute.
The missingFiles variable contains a list of all the files missing for the SDK to work optimally. When this list is not null, this doesn't mean that the SDK is not operational. The SDK may be initialized with older files, but it will advocate the download of the newest files through this list.
The debugMessage usually gives more verbose and human-readable information about the configuration process. It will always give an error message if the mode is FAILED_TO_CONFIGURE.
The activeModels will give the version of the files installed and ran by the SDK if the configuration process ran successfully i.e. if the mode is IS_READY_FOR_DETECTION.
There are cases when the implementing app wants to redirect all of the networking calls from the SDK to their backend first, before relaying the requests to Passio's backend. In this case, the app's backend acts like a proxy between the SDK and it's backend. To achieve this behaviour, the SDK offers two parameters in the PassioConfiguration object: proxyUrl
and proxyHeaders
.
If supplied, the SDK will use the proxyUrl
value as a base url when executing networking calls, and will append the provided proxyHeaders
to the request itself. When proxyUrl is provided, the SDK operates in remoteOnly state.
Make sure there is internet connection for the configuration process. The SDK needs to download the license file and the required files. If there is no internet connection the configuration status mode will be failedToConfigure.
The PassioStatus mode is notReady -> check debugMessage for the error, usually the wrong license key has been supplied or the files are missing.
The PassioStatus mode is isDownloadingModels -> register a PassioStatusListener
through the setPassioStatusListener method as it will enable tracking of the download progress.
The PassioStatus mode is isReadyForDetection -> still check the missingFiles because the SDK might be running with an older version of the files and newer need to be downloaded.
There could be cases when a certain food is not present in Passio's Nutritional Database. The standard practice in that case would be to guide the users through a creation of user created food items. These items are usually only accessible for that particular user, but there is an API that can be used to notify Passio of such items.
Using the fun submitUserCreatedFoodItem
, the app can notify Passio of user created foods.
PassioFoodItems submitted using this API go into our verification queue, and after their data has been verified by our nutrition data experts, they become available in the Nutritional Database.
On the other hand, there could be food items that have incorrect or outdated data. For these cases, the SDK provides an API to report such items, so that our data team can update them in the Nutritional Database.
To report incorrect data in a PassioFoodItem, reportFoodItem
can be used. The refCode of the food item must by supplied.
Add a button when displaying the food item to report incorrect data.
Show a form where users can add their feedback as well as provide the product code (barcode) of the item.
Upon clicking the "Report" button, invoke the reportFoodItem
function.
This Quick Start Guide will help you integrate each functionality and API of the Passio SDK using your custom UI. The guide outlines essential steps and provides examples for a smooth implementation.
The quick start demo project for iOS can be found here:
In this Quick Start Guide, we will cover the following functionalities:
Installation
Configure the SDK
Recognize Image Remote
Food Detail
In order to use the PassioSDK, your app needs to meet the following minimal requirements:
The SDK will only run or compile on iOS 13 or newer.
Passio SDK can only be used on a device and will not run on a simulator
The SDK requires access to iPhone's camera
Weight/Volume estimation will run only on iPhones with Dual Wide Camera (not on DualCamera). It will not run on the models below.
iPhone 11 Pro & Pro Max
iPhone 12 mini, Pro & Pro Max
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
Similarly to , the ingredient also has a name
, iconId
, refCode
, and amount
.
Use the name, details and to create the food header view
If remoteOnly
is used when configuring the SDK, calling with visual or packaged food detection enabled will have no effect.
Passio Nutrition-AI SDK added data from Open Food Facts (https://en.openfoodfacts.org/). Each food that contains data from Open Food Facts will be marked by public var isOpenFood: Bool. In case you choose to use foods that are marked by isOpenFood = true
you agree to abide by the terms of the Open Food Facts license agreement (https://opendatacommons.org/licenses/odbl/1-0) and their terms of use (https://world.openfoodfacts.org/terms-of-use) and you will have to add to the UI the following license copy:
"This record contains information from Open Food Facts (https://en.openfoodfacts.org), which is made available here under the Open Database License (https://opendatacommons.org/licenses/odbl/1-0)"
If you don't wish to display the Open Food Facts license somewhere in your app, it is your responsibility to ignore results where isOpenFood = true
.
To use the SDK please make sure you receive your SDK license key from Passio. The SDK WILL NOT WORK without a valid SDK key.
If you want a layer of security, where you can authenticate the license and fetch the token on your backend, you don't need the SDK key.
In that case, the SDK will communicate with your backend, and your backend will proxy the requests to Passio's servers, adding the access token to the header. This way, the license keys and tokens are never shared in the app code.
You need to provide following things when configuring the SDK:
Proxy URL - Base url of the target proxy endpoint
Proxy headers - Required headers to all of the requests
You will have to download the latest releases from the link below. The command "git clone" WILL NOT download the PassioNutritionAISDK.xcframework. https://github.com/Passiolife/Passio-Nutrition-AI-iOS-SDK-Distribution/releases. Download the PassioSDKQuickStart.zip
or the PassioSDKFullDemo.zip
and either install the framework using the Swift Package Manager directions or copy the PassioNutritionAISDK.xcframework
to your project. Make sure you have followed the directions in the README files.
In order to use the PassioSDK your app needs to meet the following minimal requirements:
The SDK will only run or compile on iOS 13 or newer.
Passio SDK can only be used on a device and will not run on a simulator
The SDK requires access to iPhone's camera
Weight/Volume estimation will run only on iPhones with Dual Wide Camera (not on DualCamera). It will not run on the models below.
iPhone 11 Pro & Pro Max
iPhone 12 mini, Pro & Pro Max
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
We'll now resolve the error introduced earlier by implementing FoodRecognitionDelegate
:
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
Add a function to setup the camera preview layer:
Add the method startFoodDetection()
This code will throw a missing delegate conformance error that we will resolve in the next step, ignore it for now.
Add a function to request authorization to use the camera and start recognition. Call the function from viewWillAppear
:
Create a function to stop Food Detection and call it from viewWillDisappear
:
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
Passio Nutrition AI Android SDK Documentation
The Passio Android SDK is shipped in the form of an .aar file.
Visit the releases page of the Android-Passio-SDK-Distribution GitHub repository. Download the passiolib-release.aar file from the latest release.
If you don't have access to the repository please contact our support team at support@passiolife.com
To include the .aar file into your project go to the module that will contain the food recognition feature. Place the .aar file in the libs folder of the module. If the libs folder is missing, create one. Reference the .aar file in the build.gradle file of the implementing module.
Sync Project with Gradle Files and be sure that you can reference the PassioSDK class within your code.
Passio Android SDK is powered by TensorFlow and FirebaseVision with the camera being managed by CameraX. Add the dependencies to these three projects by adding these lines to the module's build.gradle file.
In order for the SDK to work add the following lines to the android section of the module's build.gradle file.
Running Sync Project with Gradle Files will enable access to the SDK.
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
To configure the SDK using SDK key, we need to provide a SDK license key. Obtain your Passio key from the Passio developer portal.
To configure the SDK using Proxy URL, the SDK key is not required. The SDK will be on remoteOnly
mode in that case. You need to provide following things when configuring the SDK using Proxy URL:
Proxy URL - Base url of the target proxy endpoint
Proxy headers - Required headers to all of the requests
Create a new or open your existing project in Xcode and import the framework inside the View Controller
Define reference of SDK inside your View Controller
Configure the SDK using PassioConfiguration
shown in below code. Call configurePassioSDK()
when you need to configure SDK e.g. inside viewDidLoad()
function.
The models required for the SDK to run local recognition can take some time for the first download. If the local recognition is not needed, setting the remoteOnly
to true on the PassioConfiguration process will skip the download of the models. Remote recognition functions like recognizeImageRemote
or searchForFood
will still work.
When the SDK is configured without SDK key (using Proxy URL), the SDK will be remoteOnly
.
Implement the PassioStatusDelegate
protocol to receive configuration status updates. While the files are being downloaded, we can show in UI how many files are remanning for the download.
Update the UI based on the configuration status. The isReadyForDetection
status indicated that the SDK is setup successfully and it is ready to use. The implementation of self.askForCapturePermission()
method will be covered in the next section
The recognizeSpeechRemote
function is primarily used to fetch food items using a voice command, although it can be used to extract food items from a recipe in a text form as well.
The input to this function is just a string in free format, so the implementing side needs to do speech-to-text before using the API.
This function will be able to extract several pieces of data:
meal action, is this food being added or removed from the logs
meal time (breakfast, dinner, lunch or snack)
date of log
recognised name from the LLM
portion and weight from the LLM
nutritional data reference as PassioFoodDataInfo
Example of logging a morning breakfast:
The SDK doesn't have the functionality to record the voice session, that has to be handled by the app.
Create a screen where the user can record the voice logging command. Make sure to add the appropriate permissions.
The UI should enable the user to start/stop voice recording on a tap of a button. When the voice recording is done collect the string and use the SDK to extract food.
Once the SDK returns the results, show the list to the user with the option to deselect incorrect predictions.
The SDK can detect 3 different categories: VISUAL, BARCODE and PACKAGED. The VISUAL recognition is powered by Passio's neural network and is used to recognize over 3000 food classes. BARCODE, as the name suggests, can be used to scan a barcode located on a branded food. Finally, PACKAGED can detect the name of a branded food. To choose one or more types of detection, a FoodDetectionConfiguration object is defined and the corresponding fields are set. The VISUAL recognition works automatically.
To start the Food Recognition process a FoodRecognitionListener also has to be defined. The listener serves as a callback for all the different food detection processes defined by the FoodDetectionConfiguration.
Only the corresponding candidate lists will be populated (e.g. if you define detection types VISUAL and BARCODE, you will never receive a packagedFoodCandidates list in this callback).
Using the listener and the detection options start the food detection by calling the startFoodDetection method of the SDK.
Stop the food recognition in the onStop() lifecycle callback.
Try to run the code containing the foodListener defined above. Point the phone at the image below and see if you are getting the correct food printed on the screen (it should be red apples, but if it isn't don't worry, we'll cover the data structure later).
When starting the food detection with the FoodRecognitionListener as the callback, on every frame analyzed you will receive a FoodCandidates object. This is the structure of that data class:
You can see the structure of all the detection classes in the file called PassioAPI.kt
Depending on how you structured the FoodDetectionConfiguration object, some fields of the FoodCandidates object might be null. If the field is null it means that that type of detection has not been run. On the other hand, if the detection process ran to completion and the list is empty, then that type of detection could not recognize anything from the given frame.
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
A fast and easy way to get started with the SDK is to test it inside of PassioSDKQuickStart Demo App included in this package. Here are the steps:
Open the project in Xcode:
Replace the SDK Key in the PassioQuickStartViewController.swift file with the license key you get from Passio
Connect your iPhone and run
Modify the app bundle from "com.PassioDemoApp.demo" to "com.yourcompany...."
Run the demo app on your iPhone.
For support, please contact support@passiolife.com
You can test the SDK throughout your integration process and even before the integration. To test the SDK before the integration, we encourage you to use Passio's Reference App bundled with the SDK.
After you build the reference app or get the recognition to work inside of your app, you can test that the technology is working properly by pointing the phone at at parts of the image below to test recognition. You can test the following elements:
Food (non packaged)
Barcode
Packaged food
Nutrition facts
You can find a more extensive test of packaged and unpackaged foods below.
When testing the reference App, don't forget to add your key to the PassioExternalConnector class
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
Copyright 2022 Passio Inc
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
Every call to the SDK is made through the singleton object obtained with PassioSDK.instance call. The SDK is designed to be called from the main thread, and it will deliver results to the main thread. Its internal tasks will be run by Passio threads, so you don't need to manage the threading execution.
Two things are required for the SDK to work: a proper license key and the neural network files that the SDK uses to run the recognition process.
The license key can be requsted at support@passiolife.com
The files are shipped inside the PassioSDKSandbox project on the releases page of the Android-Passio-SDK-Distribution repository. Unzip the PassioSDKSandbox project and locate the files in the assets folder: /PassioSDKSandbox/app/src/main/assets. Files that the SDK needs have an extension .passiosecure and are encrypted
There are several ways to initialize the SDK depending on the location of the needed files:
Files are in the assets folder of your Android project
Let the SDK manage file download and update
Host the files on your own server, download the files, store them on the device's memory and point the SDK to the URIs of the needed files
For a first time integration the first approach is recommended
If the assets folder is not already present in your Android project, create one in the module where the SDK dependencies are located under the main directory (e.g. app/src/main/assets)
Transfer the .passiosecure files to the assets folder
Create a PassioConfiguration object with the application context and the license key
Call the configure method of the PassioSDK without changing the default PassioConfiguration object
By setting the sdkDownloadsModels field to true the SDK will do the initial download of the files. If the current files that are present on the device are older than the version of the SDK library, the SDK will download the newer version in the background, but still run the session with the older version of the files. Once the download is finished, the new files will be applied on the next session.
To track the download process you can implement a PassioStatusListener (recommended to define after the PassioStatus mode returns IS_DOWNLOADING_MODELS)
This approach is recommended if you are responsible for the download of the files. In this case the SDK does not care where the files are hosted and how they are downloaded, just where they are stored after the download has completed. Use the PassioConfiguration object to modify the initialization process with local files and pass the URIs of the files to the localFiles field.
After the configuration process is successful, delete the files from local device storage. The SDK will copy them over to its internal folder.
When you call the configure method of the Passio SDK with the PassioConfiguration object, you will need to define a callback to handle the result of the configuration process. This result is comprised in the PassioStatus object.
The mode of PassioStatus defines what the current status of the configuration process is. There are 5 different modes, and they all should be handled by the implementing side.
NOT_READY -> The configuration process hasn't started yet.
FAILED_TO_CONFIGURE -> The configuration process resulted in an error. Be sure to check the debugMessage of the PassioStatus object to get more insight into the nature of the error.
IS_BEING_CONFIGURED -> The SDK is still in the configuration process. Normally, you shouldn't receive this mode as a callback to the configure method.
IS_DOWNLOADING_MODELS -> The files required for the SDK to work are not present and are currently being downloaded.
IS_READY_FOR_DETECTION -> The configuration process is over and all the SDKs functionalities are available.
The missingFiles variable contains a list of all the files missing for the SDK to work optimally. When this list is not null, this doesn't mean that the SDK is not operational. The SDK may be initialized with older files, but it will advocate the download of the newest files through this list.
The debugMessage usually gives more verbose and human-readable information about the configuration process. It will always give an error message if the mode is FAILED_TO_CONFIGURE.
The activeModels will give the version of the files installed and ran by the SDK if the configuration process ran successfully i.e. if the mode is IS_READY_FOR_DETECTION.
The PassioStatus mode is NOT_READY -> check debugMessage for the error, usually the wrong license key has been supplied or the files are missing.
The PassioStatus mode is IS_DOWNLOADING_MODELS -> register a PassioStatusListener through the PassioSDK.instance.setPassioStatusListener() method as it will enable tracking of the download progress.
The PassioStatus mode is IS_READY_FOR_DETECTION -> still check the missingFiles because the SDK might be running with an older version of the files and newer need to be downloaded.
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
Once the food detection process has returned a PassioID, it is time to query the database and see what information Passio has on the detected food.
All the calls to the database are synchronous and are executed on the caller thread. For better performance execute these calls on a background thread.
The most detailed information can be found in the PassioIDAttributes object. To retrieve this object from a given PassioID, simply call
This call may return null if the given PassioID can't be found in the database. Be sure to check the nullability of the return object before proceeding.
Here we will cover the fields that are not self-explanatory:
imageName - An identifier for the image of the food. To retrieve the Drawable for the given image name use
The image for a food can also be retrieved using the PassioID and the method
parents, children, siblings - All the foods in Passio's Nutritional Database are organized in a tree-like structure with each PassioID being a node in that tree. Let's say we scanned a "mandarin". A child of a "mandarin" is a "tangerine", and its parent is a "whole citrus fruit". We can offer these alternatives to the user in case the user is pointing to a similar food to a mandarin.
passioFoodItemDataForDefault, passioFoodRecipe - The SDK differentiates single food items and food recipes. A single item represents just one particular food like an apple or peas. A food recipe consists of more than one single food item. For example a recipe is a Caprese salad which consists of mozzarella, tomatoes, basil, salt, black pepper and olive oil.
PassioFoodItemData holds the nutritional information for a food item like calories, carbs, protein, fat etc. It also has a reference weight on which these values are calculated and contains serving sizes and serving units. Serving sizes offer you quick options to select the size of the food with a given quantity and unit.
Recipes should be viewed as a collection of PassioFoodItemData objects.
If you have a PassioID returned from the detection process and you are just interested in the name of the corresponding food, use
To query the names of the foods from the database by a filter use
The returned list contains a PassioID paired with the name of the food.
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
To start using the camera in your Activity/Fragment, implement the PassioCameraViewProvider interface. By implementing this interface the SDK will use that component as the lifecycle owner of the camera (when that component calls onPause() the camera will stop) and also will provide the Context in order for the camera to start. The component implementing the interface must have a PreviewView in its view hierarchy.
Start by adding the PreviewView to your view hierarchy. Go to your layout.xml and add the following.
The SDK requires the device's Camera to work. To ensure that the Camera is working and sending frames to analyze, you need the user's permission to access the camera. As of now, only the back camera is supported by the SDK. There are two ways to implement Camera usage along with the permission request. Review both of these strategies and choose what best suits your code.
PassioCameraFragment is an abstract class that handles Camera permission at runtime as well as starting the Camera process of the SDK. To use the PassioCameraFragment simply extend it in your own fragment and supply the PreviewView that has been added to the view hierarchy in the previous step.
This approach is more manual but gives you more flexibility. You need to implement the PassioCameraViewProvider interface and supply the needed LifecycleOwner and the PreviewView added in the initial step.
Acquire the user's permission to use the Camera with these steps
Finally, after the user has granted permission to use the camera, start the SDK camera
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
With every shipped version of the SDK there will also be a PassioSDKSandbox containing the latest files and showcasing the basic setup of an app using the SDK. In this section we will cover the architecture of the small app and all the calls to the SDK keeping in mind where they are located and when they are called.
The app consists of one MainActivity that holds 2 fragments: RecognizerFragment and PermissionFragment. The latter will be displayed to the user if they deny access to the device camera. The RecognitionFragment is used to display the camera preview, start the recognition session and display the results through a view called FoodResultView.
The .aar file has already been imported as a separate module called passiolib-release. If you want to see how Android Studio structures the module, you can right-click on the passiolib-release module in the project viewer and choose the option "Open in". The passiolib-release module has been added as a dependency to the app modules, where we will use the SDK.
As mentioned above, the camera preview will be used in the RecognizerFragment. If you follow that fragment's onCreateView method you will see that the fragment inflates the view hierarchy found in the file fragment_recognizer.xml. Located in that view hierarchy we have the CameraX's PreviewView with an ID of "recognizerPreviewView". Taking a look back to the RecognizerFragment we can see that this component implements the PassioCameraViewProvider interface. Also, in the fragment's onStart lifecycle callback we are calling the
to start the preview and link the camera to the SDK. We don't need to stop the camera on onStop because we are passing the LifecycleObserver reference to the startCamera method. The camera will stop automatically when it needs to.
The SDK is initialized and configured in the onCreate method in the MainActivity.
This configure method means that we are using the files located in the assets folder of the app module. Verify that there are 4 files in the assets folder with the extension .passiosecure.
If you take a closer look, the second parameter of the PassioConfiguration object is your SDK license key. Be sure to edit this method and supply your own license key. If you run the app without providing the key, the onPassioSDKError callback will be called with a message about an invalid key.
The session is defined in the onStart and onStop lifecycle callbacks of the RecognizerFragment.
As described in the section Food detection session, we define a FoodDetectionConfiguration object and through it opt in for VISUAL and BARCODE detection. We then pass this object to startFoodDetection to receive results through the foodListener. When this fragment's onStop is called we are stopping the food detection process so that any results generated in the SDK after this point won't be delivered causing leaks.
The foodListener above only takes into account the candidates from the detectedCandidates and barcodeCandidate lists because we only opted in for VISUAL and BARCODE detection. The other files of the candidates object received in the onRecognitionResults method will be null. Depending on the candidates within these two lists we will change the state of the FoodResultView denoted as recognizerFoodResultView.
Taking a closer look at the FoodResultView we can see how to get food details such as the name or the icon depending on the source of the detection. If the passioID comes from the FOOD detection we can obtain the PassioIDAttributes through the
On the other hand, if the results come from the BARCODE detection, we can get the PassioIDAttributes from the networking call
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
Before you continue, please make sure you have a new key for version 2.x. The 1.4.x will not work with 2.x SDK!
Changes to the build.gradle file:
PassioMode.IS_AUTO_UPDATING
renamed to PassioMode.IS_DOWNLOADING_MODELS
PassioMode.IS_READY_FOR_NUTRITION
was removed.
PassioSDK.instance.setDownloadListener
was removed. Instead, the download process is traced through the PassioStatusListener
FoodDetectionConfiguration.detectOCR
renamed to detectPackagedFood
FoodDetectionConfiguration.detectLogos
was removed.
PassioSDK.instance.lookupImageForFilename, lookupPassioIDFor, lookupAlternativesFor, lookupAllSiblingsFor, lookupParentFor, lookupAllFoodItemsInDatabase, lookupAllPassioIDs, lookupAllVisuallyRecognizableFoodItems
were removed
PassioSDK.instance.searchForFood
now returns a list of Pair<passioID, food name>
PassioSDK.instance.lookupImageFor
was separated into two functions:
1. Local image lookup: lookupIconFor
2. Network image fetch: fetchIconFor
Nutrients in the PassioFoodItemData
have been refactored from Double
to UnitMass
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
This Quick Start Guide will help you integrate image recognition of the Passio SDK using your custom UI. The guide outlines essential steps and provides examples for a smooth implementation.
Check out the full code here. FULL CODE
.0
Note: Some of these detections might not be accessible to you if your license does not include that module
A DetectedCandidate represents the result from running Passio's neural network, specialized for detecting foods like apples, salads, burgers etc.
The PassioID is a unique identifier for food that is used to query our Nutritional Database. See section Nutritional Database for more details.
Try to opt in for BARCODE by enabling the flag detectBarcodes and point your device at the image of the barcodes below. The SDK's barcode detector will recognize the barcode and it will return the barcode number inside the BarcodeCandidate object.
Use the barcode field to query Passio's backend and if the scanned food can be found in Passio's database, a PassioIDAttributes object will be returned.
Passio uses OCR to recognize names of branded foods. Opt in with FoodDetectionConfiguration.detectPackagedFood and if the SDK has recognized a branded food that is present in Passio's Nutritional Database it will return the PackagedFoodCandidate of that food item. To retrieve full information about the item call:
The Visual Detection process can detect up to 5 objects per frame. Depending on your UI/UX you can display all the results to the end user, or just choose one of them. The criteria to choose the best candidate from a list is up to you, but the two most common strategies are the biggest bounding box or the highest confidence. This biggest bounding box represents the biggest object on the frame and the highest confidence points to the object that the neural network is most sure about.
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
To include the Passio SDK in your project, you’ll need to:
Add the SDK dependency using Maven: Ensure you have the correct repository and dependency in your build.gradle
file.
In the build.gradle
file add the PassioSDK dependency
Dependencies
Set up SDK Key: Obtain your SDK key from Passio’s Developer Portal. Add it to your BuildConfig
or securely store it as an environment variable in your project.
Check out the full code here.
1. Create a Github Personal Access Token (classic) with read:packages
access selected.
2. Create an .npmrc
file in the root of your project with the following lines replacing GITHUB_ACCESS_TOKEN
with the token you've created.
3. Add the Passio SDK dependency to your package.json
and run npm install
or yarn
.
or
4. Ensure the native dependencies are linked to your app.
For iOS, run pod install.
If pod install is failed with error message Specs satisfying the ReactNativePassioSDK (from ../node_modules/@passiolife/nutritionai-react-native-sdk-v3) dependency were found, but they required a higher minimum deployment target.
, ensure Podfile is configured with minimum ios version '13.0'
Also for Android, make sure the android project is configured with minSdkVersion 26 or above (check the project's build.gradle
file).
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
Create an ImageRecognitionsFragment
to handle image capture, gallery selection, and food item recognition.
Step 1: Select or Capture Image
For selecting or capturing an image, use PhotoPicker and PhotoCapture functions in ImageRecognitionsFragment
. Refer to Android's official documentation for detailed instructions:
Step 2: Recognize Food Item
The fetchResult()
method sends the selected image to the Passio SDK for recognition.
PassioSDK.instance.recognizeImageRemote
The recognizeImageRemote
function takes an image in Bitmap
format, analyzes it remotely using Passio's servers, and returns a list of food items recognized within the image. This function is asynchronous and utilizes a callback to return the result once the analysis is complete.
Output
The function returns a list of PassioAdvisorFoodInfo
objects through a callback
The structure of PassioAdvisorFoodInfo
is as follows:
recognisedName (String
): The name of the recognized food item.
portionSize (String
): Suggested portion size for the recognized food.
weightGrams (Double
): Estimated weight of the food item in grams.
foodDataInfo (PassioFoodDataInfo?
): An optional field containing more detailed nutritional information if available.
packagedFoodItem (PassioFoodItem?
): An optional field containing packaged food item data if the recognized item is a packaged product.
resultType (PassioFoodResultType
): The type of result, indicating if the food is freshly prepared, packaged, or a general item.
Step 3: Display Food Details
Navigate to a details screen upon item selection.
PassioSDK.instance.fetchFoodItemForDataInfo
The fetchFoodItemForDataInfo
function retrieves detailed food information from Passio's database based on provided data fields. This function is also asynchronous and accepts a callback that returns a PassioFoodItem
object with the full nutritional profile of the requested food item.
Input Parameters
foodDataInfo (PassioFoodDataInfo
): A data object containing metadata about the food item, including its ID and a preview of its nutrition details.
servingQuantity (Double
): The quantity or portion size of the food.
servingUnit (String
): The unit of measurement for the serving size, such as grams, ounces, or pieces.
Output
The function returns a PassioFoodItem
through the callback. This object contains the full nutritional breakdown of the food item, which can include:
Nutritional values (e.g., calories, protein, fat, carbohydrates)
Serving size details
Icons and images associated with the item
Check out the full code here. FULL CODE
1. Create a Github Personal Access Token (classic) with read:packages
access selected.
2. Create an .npmrc
file in the root of your project with the following lines replacing GITHUB_ACCESS_TOKEN
with the token you've created.
3. Add the Passio SDK dependency to your package.json
and run npm install
or yarn
.
or
4. Ensure the native dependencies are linked to your app.
If your project uses Expo, follow the steps below to work with a rebuild. Ignore the following command if your project is not based on Expo.
For iOS, run pod install.
If pod install is failed with error message Specs satisfying the ReactNativePassioSDK (from ../node_modules/@passiolife/nutritionai-react-native-sdk-v3) dependency were found, but they required a higher minimum deployment target.
, ensure Podfile is configured with minimum ios version '13.0'
Also for Android, make sure the android project is configured with minSdkVersion 26 or above (check the project's build.gradle
file).
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
If your project uses Expo, follow the steps below to work with a rebuild:
get nutrients value using PassioFoodItemNutrients
To retrieve all macro and micronutrient values using fetchNutrientsForPassioFoodItem
, by passing a passioFoodItem
, you would typically follow these steps:
You will receive PassioNutrients
Added support for localized content
Initialize Passio configuration at the entry point to enable localized content.
Once the SDK is ready, you can start food detection.
Android Platform: You need the user's permission to access the camera.
IOS Platform: Enter a value for NSCameraUsageDescription
in your Info.plist so the camera may be utilized.
To show the live camera preview, add the DetectionCameraView
to your view
Try to run the above code in component and Point the phone to the image below and see if you are getting the correct food in console log Got food detection event
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
Fuel Business Growth with Passio's AI Advisor Modules in Nutrition
Before you continue, please make sure you have a new key for version 2.x. The 1.4.x will not work with 2.x SDK!
Open project and delete the PassioSDKiOS framework
Drag and Drop the new "PassioNutritionAISDK.xcframework" to the project (select copy if needed)
In Projects > Targets > Your App > Frameworks, Libraries, and Embedded Content. Set the "PassioNutritionAISDK.xcframework" to "Embed & Sign"
import PassioSDKiOS
with import PassioNutritionAISDK
PassioSDK.shared
with PassioNutritionAI.shared
detectOCR
with detectPackagedFood
ocrCandidates
with packagedFoodCandidates
fetchPassioIDAttributesFor(ocrcode: $0
with fetchPassioIDAttributesFor(packagedFoodCode: $0.packagedFoodCode)
autoUpdate
with sdkDownloadsModels
isAutoUpdating
with isDownloadingModels
isReadyForNutrition
was removed use isReadyForDetection
The servingSizeQuantity was modified from
to:
The function func searchForFood(byText: String) -> [String]
return value was modified to -> [PassioIDAndName]
.
Removed func lookupPassioIDAttributesFor(name: String) -> PassioIDAttributes?
use instead func searchForFood(byText: String) -> [PassioIDAndName]
and then use the PassioID to lookup the PassioIDAttributes using func lookupPassioIDAttributesFor(passioID: PassioID) -> PassioIDAttributes?
.
the function
was renamed to
where the Bool Returns: UIImage and a bool, The boolean is true if the icons is food icon or false if it's a placeholder icon. If you get false you can use the asycronous funcion to "fetchIconFor" the icons from
Added UIImageView extension please check demo app to check how to use it.
References to logo detection
status.mode == .isReadyForNutrition
use only .isReadyForDetection
all the images from PassioIDAttributes and PassioFoodItemData where removed.
Copyright 2022 Passio Inc
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
Search the database of foods with a given search term.
searchForFood
now returns PassioSearchResult and a list of search options. The PassioSearchResult represent a specific food item associated with the search term, while search options provide a list of possible suggested search terms connected to the input term.
You can utilize the searchForFood
API from the SDK to retrieve food records based on a search string. This API returns a PassioSearchResult
object containing both results
and alternatives
.
The results property holds an array of food search results, while the alternatives property provides additional suggestions or alternative options if available.
How to get PassioFoodItem
using SerchFood Response?
There are cases when the barcode scanning item isn't present in the nutritional database. In the cases when the fetchFoodItemForProductCode doesn't yield a response, the user can scan the nutrition facts label on the packaging of the food and extract the data to create a new food item.
Similar to the real-time barcode scanning, the camera preview must be set up to enable continuous nutrition facts scanning. To start the Nutrition Facts scanning process a NutritionFactsRecognitionListener has to be defined. When the app is done with the nutrition fact scanning process, it should clear out the listener to avoid any unwanted UI updates.
The PassioNutritionFacts
contains all of the data that can be scanned from a nutrition facts label including:
Serving size (quantity and unit)
Weight
Calories
Carbs
Protein
Fat
Sugars
Other micronutrients like fiber, cholesterol...
To recognise nutrition facts in an image, refer to the recognizeImageRemote function. The result of the nutrition facts scanning process will come in the packagedFoodItem
attribute, and the resultType will be Nutrition Facts. The nutrients will be stored in the PassioNutrients object of the first ingredient, and the serving size info will be stored in the PassioFoodAmount.
There is a separate API called recognizeNutritionFactsRemote
that is specifically taylored for this task. The recognizeImageRemote has the ability to recognise multiple different aspects of food scanning, but if there is a use case where the only action is to extract the data from the nutrition facts label, it's advised to use the recognizeNutritionFactsRemote.
The barcode scanning process is always the most precise way to get nutritional information about a packaged food. If the barcode is not present in the nutritional database, give the user the option to scan the nutrition facts label.
If the user proceeds with the Nutrition Facts scan, register a listener to start receiving the results from the SDK. When a PassioNutritionFacts is returned from the callback, display the data so that the user can verify it's accuracy.
Enable to user to manually enter all of the other data that the Nutrition Facts scanner did not recognise.
Once a user saves this food item, be sure to include it in the search or barcode scanning results.
it's returns
The Quick start project demonstrate recognising food items from an image, either by taking a photo using the camera or selecting an image from the gallery.
Include the NSCameraUsageDescription
and NSPhotoLibraryUsageDescription keys in your app’s Info.plist
file. It tells the user why the app is requesting access to the device’s camera and photo library.
Once the Passio SDK status is isReadyForDetection
, we can start capturing image or picking it from gallery. Implement following code for asking user for Camera permission. Once user grants the permission, we can configure the camera for capturing images.
Above the viewDidLoad
method, where you create variables you want to be accessible anywhere in the ViewController file, create the following Instance Variables
Set up the Camera session and configure Input and Output
Configure the Live Preview and start the Session on the background thread
On click of capture button, provide a setting and a deleget to deliver the capturedPhoto to. This delegate will be this ViewController so we also need to conform to the protocol AVCapturePhotoCaptureDelegate
The AVCapturePhotoOutput
will deliver the captured photo to the assigned delegate which is our current ViewController by a delegate method
Ask for Photo Gallery permission:
Present the PHPickerViewController
with configuration
Implement the delegate for getting user picked images
The following function is responsible for sending a captured image to the Passio SDK for remote image recognition . It sends image to a remote server for recognition, and after receiving the response it updates the table view which will present a list of recognised food.
For each food recognised, we can create a table view cell e.g. FoodListCell
and pass the PassioAdvisorFoodInfo
object
PassioFoodItem
from PassioAdvisorFoodInfo
To display the food details, PassioFoodItem
is required.
Fetch PassioFoodItem
from PassioFoodDataInfo
We build a UI component to display macro-nutrients like calories, carbohydrates, protein, and fat. We get these macro nutrients for the selected size of the food.
The setSelectedQuantity()
function handles changes to the serving quantity. It calculates the corresponding weight based on the selected serving unit and updates the passioFoodItem
object accordingly.
For example, once user changes the quantity from a textfield or from a slider, we can call setSelectedQuantity()
function on passioFoodItem
.
The setSelectedUnit()
function handles changes to the serving unit.
For example, once user changes the unit (e.g. Packet
, Cup
, Gram
) from a list of units, we can call setSelectedQuantity()
function on passioFoodItem
.
To get list of all units for a food item:
To display the food details, passioFoodItem
is required.
We start by defining a FoodNutrient interface, which will represent individual nutrient information:
This defines the structure of each nutrient, with a title
for its name, a value
for its numerical data, and a unit
to specify the measurement unit (e.g., grams, calories).
We create a state variable, foodNutrients
, using React's useState
hook to hold an array of FoodNutrient
objects:
useEffect
We use useEffect
to trigger the extraction of nutrient information when the passioFoodItem
changes. The PassioSDK.getNutrientsOfPassioFoodItem()
method is used to get the nutrients based on the food item and its weight:
he extractFoodNutrients()
is a utility function that processes the raw data returned from the SDK and formats it into the FoodNutrient
structure.
Next, we build a UI component to display macro-nutrients like calories, carbohydrates, protein, and fat. In the renderMacro
function, we map through a predefined array of macro-nutrients, search for the corresponding nutrient in the foodNutrients
array, and display its value:
The renderFoodDetailCard
function is a React Native component that generates a card UI displaying detailed information about a food item. It features an icon, the food item's name, and additional information based on the food item's attributes.
Icon Display:
The icon of the food item is displayed within a circular View
. The PassioIconView
component is used to render the icon based on the iconId
and iconSize
properties.
Food Name:
The food item's name is displayed in a Text
component, styled for clarity with padding, font size, weight, and text transformation.
Additional Information:
The renderMacro()
function is called to render macronutrient information.
If passioFoodItem.isOpenFood
is true, the renderOpenFood()
function is called to display additional details relevant to open food items.
React Native component that generates a user interface for editing serving sizes of a food item. It displays the current serving size, provides an input field for updating the quantity, and offers selectable serving units.
The structure of passioFoodItem
is determined by the properties of prop.passioFoodItem
, which could include fields such as:
name
: The name of the food item.
iconId
: An identifier for the food item's icon.
amount
: An object containing serving sizes and units.
isOpenFood
: A boolean indicating whether the food item is an open food.
The onServingQuantityChange
function is a callback that handles changes to the serving quantity input. It updates the state of the input and recalculates the weight based on the quantity entered by the user. This function is optimized with useCallback
to prevent unnecessary re-renders.
To manage the input for serving quantities, ensuring that the food item's state is updated dynamically based on user input. It calculates the corresponding weight based on the selected serving unit and updates the passioFoodItem
state accordingly.
Serving Size Display:
A Text
component displays the serving sizes, formatted to show the calculated weight and its unit. The text is styled for emphasis.
Serving Quantity Input:
A TextInput
component allows users to input the serving quantity. It is styled with borders, padding, and a placeholder that indicates the expected input.
Selectable Serving Units:
A FlatList
displays the available serving units horizontally. Each item in the list is rendered as a Pressable
, allowing users to select a unit.
The background color of the selected unit changes to blue, and the text color changes to white for clarity.
Ingredients
The renderIngredient
function is a React Native component that displays a list of ingredients for a food item. It checks if the passioFoodItem
has ingredients and renders a list of them, including each ingredient's name, amount, and icon.
FlatList:
A FlatList
is used to display each ingredient.
Each ingredient is represented by a row, consisting of:
An icon (PassioIconView
) displaying the ingredient’s iconId
.
The name of the ingredient.
The selected unit or amount of the ingredient.
Render Item:
For each ingredient, a row is rendered, showing:
The ingredient’s icon.
The ingredient’s name.
The selected unit of the ingredient, such as its amount.
In this example, we'll use a Fragment to set up the SDK configuration. Add a new Fragment class named ConfigFragment.kt
to handle the configuration process.
Here is the ConfigFragment
that configures the Passio SDK:
PassioConfiguration:
Set up PassioConfiguration
with your context and SDK key.
Enable sdkDownloadsModels
to allow the SDK to download necessary models.
remoteOnly = true
enables the SDK to access remote resources if required.
PassioMode States: The SDK configuration callback provides different statuses:
NOT_READY: SDK is not ready, usually due to an issue.
FAILED_TO_CONFIGURE: Configuration failed.
IS_READY_FOR_DETECTION: SDK is configured and ready.
IS_BEING_CONFIGURED: SDK is currently being configured.
IS_DOWNLOADING_MODELS: SDK is downloading required models.
Navigation on Ready:
On successful configuration, the app navigates to ImageRecognitionsFragment
.
To display the configuration status, add a TextView
(e.g., tvSDKStatus
) in the fragment’s XML layout file.
The SDK setup can fail due to configuration or network issues. The onSDKError
function in the ConfigFragment
updates the UI with the relevant error message to help troubleshoot.
Check out the full code here. FULL CODE
A component that displays the camera feed and sends camera frames
The configuration process of the SDK has severable responsibilities:
Validation of the license key
Validating the currently present files
Downloading or updating the files required by the SDK to do recognition
Preparing the files to be used for inference
useEffect
: Runs the configure
function once when the component mounts (because of the empty dependency array []
).
PassioSDK.configure
: Calls the Passio SDK configuration method with a key and options (debugMode
, autoUpdate
).
If the SDK is ready (status.mode === 'isReadyForDetection'
), loadingState
is updated to 'ready'
.
If there’s an error (status.mode === 'error'
), the state is set to 'error'
.
If the status is 'notReady'
, nothing is done.
Error handling: If any error occurs during the configuration, the catch
block sets the loadingState
to 'error'
.
This is a method (from the Passio SDK) that registers callbacks for events related to model downloading: completedDownloadingFile: This callback is triggered when a file finishes downloading. The object passed to it contains filesLeft, which represents how many files are still left to download.
You can use this hook in any functional component to check whether the app has camera authorization. For example:
In this component, PassioConfigurationView
, you're conditionally rendering different UI elements based on the sdkStatus
and leftFile
values. Here's a breakdown of how this works:
Switch on sdkStatus
:
The component switches between different UI layouts based on the current value of sdkStatus
:
Case 'ready'
: When sdkStatus
is "ready"
, it displays a UI element that says SDK is Ready.
Case 'error'
: If sdkStatus
is "error"
, it shows an error message SDK is Error.
Default case: If sdkStatus
is anything else (including when it's 'init'
or 'downloading'
), it shows an ActivityIndicator
(a loading spinner) and, if leftFile
is not null
, it displays how many files are left to download.
Refer to this to create an Item.
To add a Passio SDK as package dependency to your Xcode project, select File
> Add Package Dependency
and enter following repository URL
https://github.com/Passiolife/Passio-Nutrition-AI-iOS-SDK-Distribution
Click "Next". Xcode will validate the package and its dependencies.
In the next dialog box, you'll be asked to specify the version or branch you want to use. Decide whether your project accepts updates to a package dependency "Up to the next major version" or "Up to the next minor version". To be more restrictive, select a specific version range or an exact version. Major versions tend to have more significant changes than minor versions, and may require you to modify your code when they update.
Xcode will download and add the PassioNutritionAISDK
to your project. You can now import and start using the PassioNutritionAISDK
.
Download the PassioNutritionAISDK.xcframework.zip
file from https://github.com/Passiolife/Passio-Nutrition-AI-iOS-SDK-Distribution/blob/main/PassioNutritionAISDK.xcframework.zip
Unzip it and drag and drop it into your project. Make sure to select "Copy items if needed".
In project "General" -> "Frameworks, Libraries and Embedded Content" Change to "Embed & Sign"
The FoodDetailsFragment
is a core component designed to display detailed information about a selected food item, including its name, icon, calories, macronutrients, and a list of micronutrients. Users can adjust serving sizes and quantities to view nutrient values accordingly.
The primary components in FoodDetailsFragment
include:
Food Item Information: Displays the name, icon, and serving size details.
Serving Size Selector: Allows users to select a preferred serving size or unit.
Nutritional Information: Shows calorie count and macronutrient breakdown (Carbs, Protein, Fat).
Micronutrient List: Provides detailed nutrient information for vitamins, minerals, and other nutrients.
The PassioFoodItem
model is the foundation of food details and includes the following properties:
id
String
Unique identifier for the food item.
refCode
String
Reference code for the item.
name
String
Name of the food item.
details
String
Description/details of the food.
iconId
String
ID for loading the food icon.
amount
PassioFoodAmount
Serving size and unit info.
ingredients
List<PassioIngredient>
List of ingredients in the food.
Example:
The food item name, icon, and details are rendered using PassioFoodItem
's properties.
PassioFoodAmount
This model defines possible serving sizes and units. PassioFoodAmount
includes:
servingSizes
List<PassioServingSize>
Available serving sizes for the food
servingUnits
List<PassioServingUnit>
Units available for the food item
Example:
The servingSizes
property is used to populate the RecyclerView adapter ServingSizeAdapter
, allowing users to select different serving sizes.
PassioServingSize
and PassioServingUnit
These classes are components of PassioFoodAmount
that define the quantity and units for serving sizes:
PassioServingSize
quantity
Double
Quantity of the unit
unitName
String
Name of the serving unit
PassioServingUnit
unitName
String
Linked to the PassioServingSize
weight
UnitMass
Weight in grams or other units
The setQuantityAndUnit
function allows users to specify a desired serving quantity and unit, updating the displayed nutrient values dynamically.
FoodDetailsFragment.kt
Check out the full code here. FULL CODE
PassioSDK provide support to below api's
Once the SDK is ready, you can start food detection.
Android Platform: You need the user's permission to access the camera.
IOS Platform: Enter a value for NSCameraUsageDescription
in your Info.plist so the camera may be utilized.
To show the live camera preview, add the DetectionCameraView
to your view
Try to run the above code in component and Point the phone to the image below and see if you are getting the correct food in console log Got food detection event
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
For the SDK to work, it requires a proper license key.
The license key can be requested at support@passiolife.com
When your component mounts, configure the SDK using your Passio provided developer key and start food detection.
autoUpdate: true
: Set to true
to enable the download of AI models from Passio's servers.
`remoteOnly:true` With this flag enabled, the SDK won't download the files needed for local recognition. In this case only remote recognition is possible
The possible states of the SDK after calling configure. Switch on sdkStatus.mode
to access the data associated with each state.
MODE: notReady
, isReadyForDetection
isBeingConfigured
and error
notReady
: SDK is not ready due to missing model files. Please download the specified file and call configure
again, passing in the localFileURLs of the downloaded files.
isReadyForDetection
: SDK configured successfully. This status much be reached before calling startFoodDetection
error
: SDK failed to configure in an unrecoverable way. Please read the error message for more information
The SDK uses the proxyUrl as the base URL for all network calls and appends the specified proxyHeaders to each request. When proxyUrl is set, the SDK functions exclusively in a remoteOnly mode.
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
This project provides React Native bindings for the Passio SDK.
React Native v0.68.0 or higher
Xcode 14.3 or higher
iOS 13 or higher
Android API level 26 or higher
Cocoapods 1.10.1 or higher
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
fetch list of possible hidden ingredients for a given food name.
Once the SDK is ready, you can start nutrition label detection.
Android Platform: You need the user's permission to access the camera.
IOS Platform: Enter a value for NSCameraUsageDescription
in your Info.plist so the camera may be utilized.
To show the live camera preview, add the DetectionCameraView
to your view
Try to run the above code in the component and Point the phone to the image below and see if you are getting the correct nutrients lable in the console log
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
When starting food detection with PassioSDK.startFoodDetection
you will receive a FoodDetectionEvent
. This is the structure of that data class:
If at any point you need help from the Passio team, please reach out to us at support@passiolife.com
fetch list of possible ingredients if a more complex food for a given food name.
fetch list of possible visual alternatives for a given food name.
This Quick Start Guide will help you integrate each functionality and API of the Passio SDK using your custom UI. The guide outlines essential steps and provides examples for a smooth implementation.
In this Quick Start Guide, we will cover the following functionalities:
Installation
Configure the SDK
Recognize Image Remote
Food Detail
Mentioned steps to install a private npm package and configure it for Android and IOS
React Native v0.68.0 or higher
Xcode 14.0 or higher
iOS 13 or higher
Android API level 26 or higher
Cocoapods 1.10.1 or higher
In this section, we explore how to configure the Passio SDK using your Passio key. Proper configuration is essential to ensure seamless communication between your application and the SDK.
Steps to Configure the SDK:
Obtain your Passio key from the Passio developer portal.
Follow the integration steps to initialize the SDK within your application.
This section guides you through using the Recognize Image Remote API. You'll learn how to capture images via the camera or select images from the gallery, and display the results in a bottom sheet.
Steps to Use Recognize Image Remote:
Implement image capture functionality.
Call the Recognize Image Remote API with the captured or selected image.
Handle and display the results effectively in a bottom sheet.
In this section, we demonstrate how to display food information, including the name, macronutrients, micronutrients, and how to change the serving size.
Steps to Show Food Detail:
Fetch food details using the appropriate API.
Render the food item's name, macronutrients, and micronutrients in your custom UI.
Provide functionality to adjust the serving size dynamically.
The RecognizeImageRemote
component is responsible for recognizing food items from an image, either by taking a photo
using the camera or selecting an image from the gallery
. It integrates with the Passio Nutrition SDK to fetch food details and displays the results in a bottom sheet.
useCameraPermissions
: It seems like you're using a custom hook to handle camera permissions, but it's not a native hook from libraries like expo-camera
Ensure that the hook is correctly implemented or imported.
cameraViewRef
: You need to ensure that this ref is properly created and assigned to the camera view. If you're using expo-camera
, ensure you're following the right method for capturing images.
setPictureUri
: This is the state updater function. Whenever a new picture is captured, you'll call setPictureUri
with the URI of the picture to update the state. This triggers a re-render of the component with the updated picture URI.
The onScanImage
function is responsible for sending a captured image to the Passio SDK for remote image recognition . It processes the image, sends it to a remote server for recognition, and updates the local state based on the response.
This part sends the image to PassioSDK
for remote recognition.
pictureUri?.replace("file://", "")
: This removes the file://
prefix from the pictureUri
, ensuring it's in a valid format for the SDK to process.
The third argument, "RES_512"
, likely specifies the resolution or processing mode for the image.
In this UI component using a BottomSheet
with dynamic sizing that contains a list of recognized items (from passioSpeechRecognitionModel
) and a close button.
refCode
unique identifier for the food item
RefCode
name
Name of the food item
string
details
Additional details or description of the food item
string
iconId
Icon identifier for the food item
PassioID | string
amount
Amount of the food item (e.g., quantity, volume)
PassioFoodAmount
ingredients
List of ingredients used in the food item
PassioIngredient[]
ingredientWeight
Weight of the food item, measured in units of mass
UnitMass
isOpenFood
Boolean indicating whether the data is sourced from openfood.org
boolean
id
string
A component for displaying food icons from the Passio SDK.
selectedUnit
The selected unit for the food amount.
selectedQuantity
The quantity of the selected unit.
weightGrams
The weight of the food amount in grams.
servingUnits
An array of serving units available for the food.
servingSizes
An array of serving sizes available for the food.
weight
The weight of the food amount using a specified unit of mass.