Visual food detection

A DetectedCandidate represents the result from running Passio's neural network, specialized for detecting foods like apples, salads, burgers etc.

class DetectedCandidate(
    val passioID: PassioID,
    val confidence: Float,
    val boundingBox: RectF,
    val croppedImage: Bitmap?
)

Barcode detection

Try to opt in for BARCODE by enabling the flag detectBarcodes and point your device at the image of the barcodes below. The SDK's barcode detector will recognize the barcode and it will return the barcode number inside the BarcodeCandidate object.

data class BarcodeCandidate(val barcode: Barcode, val boundingBox: RectF)

Use the barcode field to query Passio's backend and if the scanned food can be found in Passio's database, a PassioIDAttributes object will be returned.

PassioSDK.instance.fetchPassioIDAttributesForBarcode(barcode) { passioIDAttribute ->
    if (passioIDAttribute == null) {
        return@fetchPassioIDAttributesForBarcode
    }
    Toast.makeText(requestContext(), passioIDAttribute.name, Toast.LENGTH_SHORT).show()
}

Packaged food detection

Passio uses OCR to recognize names of branded foods. Opt in with FoodDetectionConfiguration.detectPackagedFood and if the SDK has recognized a branded food that is present in Passio's Nutritional Database it will return the PackagedFoodCandidate of that food item. To retrieve full information about the item call:

PassioSDK.instance.fetchPassioIDAttributesForPackagedFood(
    packagedFoodCandidate.packagedFoodCode
) { passioIDAttributes ->    
    if (passioIDAttributes == null) {        
        return@fetchPassioIDAttributesForPackagedFood    
    }    
    Toast.makeText(requestContext(), passioIDAttribute.name, Toast.LENGTH_SHORT).show()
}

The FoodCandidate returns a list of VisualCandidates, which object should I choose to display to the user?

The Visual Detection process can detect up to 5 objects per frame. Depending on your UI/UX you can display all the results to the end user, or just choose one of them. The criteria to choose the best candidate from a list is up to you, but the two most common strategies are the biggest bounding box or the highest confidence. This biggest bounding box represents the biggest object on the frame and the highest confidence points to the object that the neural network is most sure about.

If at any point you need help from the Passio team, please reach out to us at support@passiolife.com

Introduction

Passio Nutrition-AI SDK is an industry leading food recognition and nutrition tracking SDK designed to enable app developers to rapidly add AI-powered food logging capabilities and AI-powered nutrition coaching to their mobile applications.

The SDK is a part fo product and relies heavily to a wide range of APIs and functionality provided in the .

Video Overview

Our Nutrition-AI SDK offers powerful capabilities that are rapidly evolving, so some features may not appear in the overview video below.

Latest SDK vs. Legacy

A key shift from our legacy SDK is the move from real-time on-device food logging to photo logging analyzed via cloud-side LLMs. While real-time scanning is still available as of Q4 2024, it’s no longer the primary mode. Passio now recommends photo logging, voice logging, and text search as the primary modes.

Getting Started

Platform Specific SDKs

Follow our documentation guides to get started and build your AI-powered applications. Please select your platform of choice to get to the documentation.

In Case you're looking for APIs, Hub or Advisor

Welcome to Passio Nutrition-AI iOS SDK! When integrated into your app the SDK provides you with food recognition and nutrition assistant technology. The SDK creates a video preview layer and outputs foods recognized by our computer vision technology in the video feed of your live camera along with nutrition data related to the recognized foods.

As the developer, you have complete control of when to turn on/off the SDK and to configure the outputs which includes:

• food names (e.g. banana, hamburger, fruit salad, quest chocolate bar)

• lists of alternatives for recognized foods (e.g., soy milk would be an alternative of a visually recognized class milk)

• barcodes detected on food packages

• packaged foods recognized by the text detected on food packages

• nutrition information detected on food packages via Passio's Nutrition Facts reader which returns information written in Nutrition Facts labels

• nutrition information associated with the foods

• food weight and volume for certain foods

By default the SDK does not record/store any photos or videos. Instead, as the end user hovers over a food item with his/her camera phone, the SDK recognizes and identifies food items in real time. This hovering action is only transitory/temporary while the end user is pointing the camera at a particular item and is not recorded or stored within the SDK. As a developer, you can configure the SDK to capture images or videos and store them in your app.

Versions

Open Food Facts and Terms of Use

Passio Nutrition-AI SDK added data from Open Food Facts (https://en.openfoodfacts.org/). Each food that contains data from Open Food Facts will have the value set in its food origin list. In case you choose to display the these food items you agree to abide by the terms of the Open Food Facts license agreement (https://opendatacommons.org/licenses/odbl/1-0) and their terms of use (https://world.openfoodfacts.org/terms-of-use) and you will have to add to the UI the following license copy:

"This record contains information from Open Food Facts (https://en.openfoodfacts.org), which is made available here under the Open Database License (https://opendatacommons.org/licenses/odbl/1-0)"

The configuration process of the SDK has severable responsibilities when configured through SDK key:

• Validation of the license key

• Validating the currently present files

• Downloading or updating the files required by the SDK to do recognition

• Preparing the files to be used for inference

The SDK is configured using a PassioConfiguration object. This object is used to define the location of the files that the SDK requires.

SDK downloads the models

The default behaviour (or by setting the sdkDownloadsModels field to true) of the SDK is to download the required files for local recognition. If the current files that are present on the device are older than the version of the SDK library, the SDK will download the newer version in the background, but still run the session with the older version of the files. Once the download is finished, the new files will be applied on the next session. Don't forget to add the import statement for the nutrition sdk.

The SDK will not download required files for local recognition when configured without SDK key. The SDK will be in remoteOnly mode.

import PassioNutritionAISDK
import AVFoundation

...

let passioSDK = PassioNutritionAI.shared

override func viewDidLoad() {
    super.viewDidLoad()
    configureSDK()
}

// Configuring SDK using SDK key

func configureSDK() {
    let key = "YOUR_PASSIO_KEY"
    let passioConfig = PassioConfiguration(key: key)
    
    passioSDK.configure(passioConfiguration: passioConfig) { (status) in
        print("Mode = \(status.mode)\n missingfiles = \(String(describing: status.missingFiles))" )
    }
}

// Configuring SDK using Proxy

func configureSDK() {
    var passioConfig = PassioConfiguration()
    // Set the base URL of the target proxy endpoint
    passioConfig.proxyUrl = "https://yourdomain.com/nutrition/"
    // Set headers as per your requirements
    passioConfig.proxyHeaders = ["header" : "value"] 
    
    passioSDK.configure(passioConfiguration: passioConfig) { (status) in
        print("Mode = \(status.mode)\n missingfiles = \(String(describing: status.missingFiles))" )
    }
}

val passioConfiguration = PassioConfiguration(
    this.applicationContext,
    "Your license key here"
).apply {
    sdkDownloadsModels = true // This can be ommited
}

PassioSDK.instance.configure(passioConfiguration) { passioStatus ->
    // Handle PassioStatus states
} 

const [isReady, setIsReady] = useState(false);

// Effect to configure the SDK and request camera permission
// We are adding our developer key in LoadingContainerView
useEffect(() => {
  Promise.all([
    PassioSDK.configure({
      key: 'your-developer-key',
      autoUpdate: true,
    }),
    PassioSDK.requestCameraAuthorization(),
  ]).then(([sdkStatus, cameraAuthorized]) => {
    setIsReady(sdkStatus.mode === 'isReadyForDetection' && cameraAuthorized);
  });
}, []);

import 'package:nutrition_ai/nutrition_ai_sdk.dart';

void configureSDK() async {
   String passioKey = "Your_PassioSDK_Key";
   var configuration = PassioConfiguration(passioKey);
   passioStatus = await NutritionAI.instance.configureSDK(configuration);
   // Handle result of the configuration process.
}

Remote Only configuration

The models required for the SDK to run local recognition can take some time for the first download. If the local recognition is not needed, setting the remoteOnly to true on the PassioConfiguration process will skip the download of the models. Remote recognition functions like recognizeImageRemote or searchForFood will still work.

Handling the configure status object

The result of the configuration process is a PassioStatus object, containing a PassioMode attribute along with the missingFiles, debugMessage, error object and the active models number. The attribute that needs to looked at is the mode.

PassioMode is an enum describing the current state of the Passio SDK configuration process:

• notReady -> The configuration process hasn't started yet.

• failedToConfigure -> There was an error during the configuration process.

• isBeingConfigured -> The SDK is still in the configuration process. Normally, you shouldn't receive this mode as a callback to the configure method. If you do please contact our support team.

• isDownloadingModels -> The files required by the SDK to work are not present and are currently being downloaded.

• isReadyForDetection -> The SDK is configured with the set of files defined by the activeModels attribute.

The missingFiles variable contains a list of all the files missing for the SDK to work optimally. When this list is not null, this doesn't mean that the SDK is not operational. The SDK may be initialized with older files, but it will advocate the download of the newest files through this list.

The debugMessage usually gives more verbose and human-readable information about the configuration process. It will always give an error message if the mode is FAILED_TO_CONFIGURE.

The activeModels will give the version of the files installed and ran by the SDK if the configuration process ran successfully i.e. if the mode is IS_READY_FOR_DETECTION.

Proxy url and headers

There are cases when the implementing app wants to redirect all of the networking calls from the SDK to their backend first, before relaying the requests to Passio's backend. In this case, the app's backend acts like a proxy between the SDK and it's backend. To achieve this behaviour, the SDK offers two parameters in the PassioConfiguration object: proxyUrl and proxyHeaders.

Best practices

1. Make sure there is internet connection for the configuration process. The SDK needs to download the license file and the required files. If there is no internet connection the configuration status mode will be failedToConfigure.

2. The PassioStatus mode is notReady -> check debugMessage for the error, usually the wrong license key has been supplied or the files are missing.

3. The PassioStatus mode is isDownloadingModels -> register a PassioStatusListener through the setPassioStatusListener method as it will enable tracking of the download progress.

4. The PassioStatus mode is isReadyForDetection -> still check the missingFiles because the SDK might be running with an older version of the files and newer need to be downloaded.

Using SDK Key

To use the SDK through license key, please make sure you receive your SDK license key from Passio.

To obtain an SDK key, visit https://www.passio.ai/nutrition-ai

Without using SDK Key

If you want a layer of security, where you can authenticate the license and fetch the token on your backend, you don't need the SDK key.

In that case, the SDK will communicate with your backend, and your backend will proxy the requests to Passio's servers, adding the access token to the header. This way, the license keys and tokens are never shared in the app code.

You need to provide following things when configuring the SDK:

• Proxy URL - Base url of the target proxy endpoint

• Proxy headers - Required headers to all of the requests

Responisilities of proxy backend:

• License authentication - the license will be authenticated on the proxy backend

• Token fetching - token will be fetched on the proxy backend and appended to every call

The configuration for the proxy URL will be covered in the "Configure the SDK" section.

Minimum requirements

These are the minimum requirements for every supported platform

Welcome to Passio Nutrition-AI iOS SDK

When integrated into your app the SDK provides you with food recognition and nutrition assistant technology. The SDK creates a video preview layer and outputs foods recognized by Passio's computer vision technology in the video feed of your camera along with nutrition data related to the recognized foods.

As the developer, you have complete control of when to turn on/off the SDK and to configure the outputs which includes:

• food names (e.g. banana, hamburger, fruit salad, Quest chocolate bar)

• lists of alternatives for recognized foods (e.g., soy milk would be an alternative of a visually recognized class milk)

• barcodes detected on food packages

• packaged foods recognized by the text detected on food packages

• nutrition information detected on food packages via Passio's Nutrition Facts reader which returns information written in Nutrition Facts labels

• nutrition information associated with the foods

• food weight and volume for certain foods

By default the SDK does not record/store any photos or videos. Instead, as the end user hovers over a food item with his/her camera phone, the SDK recognizes and identifies food items in real time. This hovering action is only transitory/temporary while the end user is pointing the camera at a particular item and is not recorded or stored within the SDK. As a developer, you can configure the SDK to capture images or videos and store them in your app.

Creating a project

Hit the big '+' button in your sidebar and select 'New Project' from the menu that pops up. Give your project a name, and you're good to go!

If at any point you need help from the Passio team, please reach out to us at support@passiolife.com

Real-time barcode scanning

The SDK can detect barcodes located on the packaging of packaged foods. To implement real-time barcode scanning, the camera setup for continuous image recognition is needed.

Specifically for barcode scanning, these steps are required:

1. When defining a FoodDetectionConfiguration object, enable the detectBarcode flag.

2. In the recognitionResults callback, check the property candidate.barcodeCandidates. If this list is empty, no barcodes have been recognised.

3. If there is a single or multiple barcode candidates, use their value property and invoke this function to fetch the full nutrition data.

public func fetchFoodItemFor(productCode: String, completion: @escaping ((PassioNutritionAISDK.PassioFoodItem?) -> Void))

fun fetchFoodItemForProductCode(
    productCode: String,
    onResult: (foodItem: PassioFoodItem?) -> Unit
)

fetchFoodItemForProductCode(code: Barcode | PackagedFoodCode): Promise<PassioFoodItem | null>

Future<PassioFoodItem?> fetchFoodItemForProductCode(String productCode)

Barcode scanning from an image

To recognise a barcode in an image, refer to the recognizeImageRemote function. The result of the barcode scanning process will come in the packagedFoodItem attribute, and the resultType will be Barcode.

UI Example

1. Set up the camera preview and detection session to recognise barcodes

2. Create a result view with two states: search and result

3. If the candidate.barcodeCandidates is empty, show the search state. Else, fetch the PassioFoodItem, and use it's name and icon to show the result

Example of an image that produces multiple BarcodeCandidates:

There are cases when the barcode scanning item isn't present in the nutritional database. In the cases when the fetchFoodItemForProductCode doesn't yield a response, the user can scan the nutrition facts label on the packaging of the food and extract the data to create a new food item.

Real-time Nutrition Facts scanning

Similar to the real-time barcode scanning, the camera preview must be set up to enable continuous nutrition facts scanning. To start the Nutrition Facts scanning process a NutritionFactsRecognitionListener has to be defined. When the app is done with the nutrition fact scanning process, it should clear out the listener to avoid any unwanted UI updates.

private val listener = object : NutritionFactsRecognitionListener {
    override fun onRecognitionResult(nutritionFacts: PassioNutritionFacts?, text: String) {
        // Display the result
    }
}

override fun onResume() {
    super.onResume()
    PassioSDK.instance.startNutritionFactsDetection(listener)
}

override fun onPause() {
    PassioSDK.instance.stopNutritionFactsDetection()
    super.onPause()
}

The PassioNutritionFacts contains all of the data that can be scanned from a nutrition facts label including:

• Serving size (quantity and unit)

• Weight

• Calories

• Carbs

• Protein

• Fat

• Sugars

• Other micronutrients like fiber, cholesterol...

Nutrition Facts scanning from an image

To recognise nutrition facts in an image, refer to the recognizeImageRemote function. The result of the nutrition facts scanning process will come in the packagedFoodItem attribute, and the resultType will be Nutrition Facts. The nutrients will be stored in the PassioNutrients object of the first ingredient, and the serving size info will be stored in the PassioFoodAmount.

UI Example

1. The barcode scanning process is always the most precise way to get nutritional information about a packaged food. If the barcode is not present in the nutritional database, give the user the option to scan the nutrition facts label.

1. If the user proceeds with the Nutrition Facts scan, register a listener to start receiving the results from the SDK. When a PassioNutritionFacts is returned from the callback, display the data so that the user can verify it's accuracy.

1. Enable to user to manually enter all of the other data that the Nutrition Facts scanner did not recognise.

1. Once a user saves this food item, be sure to include it in the search or barcode scanning results.

Search

The Passio Nutrition AI SDK search functionality gives access to over 2.2 million food records in our nutritional database. The search api takes in a text query and returns a list of PassioFoodDataInfo results as well as a list of suggested searches.

There are two different search APIs available in the SDK:

1. searchForFood - tries to give best string matches based on the input parameter, often providing the same food from different brands. Example: Input parameter "eggs" will provide results like "Trader Joe's eggs", "Eggland's Best eggs", "Eggs in carton, Lucerne", ...

2. searchForFoodSemantic - will try to contextually match the most similar foods to the input parameter. Example: Iinput parameter "eggs" will provide results like "Cooked eggs", "Scrambled eggs", "Eggs benedict", ...

The PassioFoodDataInfo is used a reference object to fetch the full nutritional data. It has all of the attributes needed for displaying on a typical search screen: the name of the food, name of the brand (if possible), iconId, and a nutritional preview detailing the macro nutrients for a specified serving size and weight.

The search will return 50 items that best match the input term, as well as a list of suggested searches that are semantically or contextually close to the input term.

Example: searching for term "coffee"

• search results: "Coffee, Brewed", "Coffee, Breakfast Blend", Coffee - Brewed From Grounds", "Coffee, Brewed, Decaffeinated", "Coffee, Instant, Reconstituted Decaffeinated", ...

• suggested searches: "coffee with whole milk", "coffee with creamer", "iced coffee"

UI Example

1. Create an input text field to collect the user query

2. Create a horizontal list scroll to display the suggestions

3. Create a vertical list to display the results of the search

4. If the user clicks on a suggestion, pass that string to the search API and reload the UI with the new results

Food Icons

The SDK gives the ability to fetch food icons for a specific passioId. There are two functions that can be used: lookupIconsFor and fetchIconsFor.

1. lookupIconsFor does a local search of the icons without doing any networking calls. It returns two icons. One is a placeholder icon depending on the type of food associated with the passioId. The second one is cached icon from a previous fetchIconsFor call, it it exists.

2. fetchIconsFor does a networking call to fetch an icon associated with the passioId, but it will check the local cache and return a previously fetched icon.

Example of how to use both of these functions to first display the placeholder icon, and then fetch the remote icon:

RefCode

There are several functions that are used to retrieve the PassioFoodItem object, depending on the origin of the data. If the food item is being scanned using an image, the food item is fetched using either fetchFoodItemForPassioID or fetchFoodItemForProductCode. If search is being used to access to food item, the appropriate call is the fetchFoodItemForDataInfo.

But if there is a use case that requires to save just one attribute to a database, that will later on be used to retrieve the full PassioFoodItem object, then the refCode attribute is the answer. The refCode in combination with the fetchFoodItemForRefCode will retrieve the original food item object from Passio's nutritional database.

Using v2 PassioID in the v3 SDK

If the implementing app is transitioning from generation 2 to generation 3 of the SDK, and has already stored generation 2 PassioIDs, invoking the fetchFoodItemLegacy will retrieve the PassioFoodItem that was previously structured as PassioIDAttributes.

Follow these steps to include the SDK into your mobile application project.

The Passio SDK has the ability the recognise anything from simple ingredients like blueberries and almonds to complex cooked dishes like a beef burrito with salad and fries.

There are two types of food recognition, each of them with their own strengths and weaknesses.

The Remote image recognition approach is good when the accuracy of the results is top priority, and waiting for the response is not an issue. This use case is implemented by taking static images from the camera or the library of the device and sending them for recognition in an asynchronous fashion.

The Local model approach is good when speed is of the essence. This use case is implemented using continuous frame recognition from the camera. A callback is registered to capture the results as they are coming from the camera stream.

Remote image recognition

This API sends an image as base64 format to an LLM on Passio's backend, and returns a list of recognised items.

The default behaviour of this function is to resize the image to 512 pixels (longer dimension is resized, the other is calculated to keep aspect ratio). Using the PassioImageResolution enum, the image can be either resized to 512, 1080 or keep the original resolution.

The response, presented as a list ofPassioAdvisorFoodInfo objects, contains the name, portion and weight in grams recognised by the LLM. These attributes can be used for debugging, but the data from the nutritional database is contained either in the foodDataInfo if the result type is a Food Item, or packagedFoodItem if it's a Barcode or Nutrition Facts. To fetch the for the PassioFoodDataInfo object, use the fetchFoodItemForDataInfo function.

UI Example

1. Create a screen where the user can snap one or multiple images using the camera of the device

2. Upon clicking next, the recognizeImageRemote is invoked on each of the images in the list

3. Wait for all of the responses to come, add each results list to a final list of results. When the last asynchronous function is executed, present the final list to the user.

Local neural network model

To set up the local model and the continuous scanning mode, the camera preview and the recognition session need to be defined.

Camera preview

Add the UI element that is responsible for rendering the camera frames:

Start food detection

The type of food detection is defined by the FoodDetectionConfiguration object. To start the Food Recognition process a FoodRecognitionListener also has to be defined. The listener serves as a callback for all the different food detection processes defined by the FoodDetectionConfiguration. When the app is done with food detection, it should clear out the listener to avoid any unwanted UI updates.

The FoodCandidates object that is returned in the recognition callbacks contains three lists:

• detectedCandidates detailing the result of VISUAL detection

• barcodeCandidates detailing the result of BARCODE detection

• packagedFoodCandidates detailing the result of PACKAGED detection

Only the corresponding candidate lists will be populated (e.g. if you define detection types VISUAL and BARCODE, you will never receive a packagedFoodCandidates list in this callback).

Visual detection

A DetectedCandidate represents the result from running Passio's neural network, specialized for detecting foods like apples, salads, burgers etc. The properties of a detected candidate are:

• name

• passioID (unique identifier used to query the nutritional databse)

• confidence (measure of how accurate is the candidate, ranges from 0 to 1)

• boundingBox (a rectangle detailing the bounds of the recognised item within the image dimensions)

• alternatives (list of alternative foods that are visually or contextually similar to the recognised food)

• croppedImage (the image that the recognition was ran on)

To fetch the of a detected candidate use:

UI Example

1. Implement the camera screen using the steps above

2. Create a result view that can have two states: scanning and result

3. If the callback returns an empty list, show the scanning state. If it returns the result, display the name from the detectedCandidate.name

Example of an image that produces a DetectedCandidate:

The recognizeSpeechRemote function is primarily used to fetch food items using a voice command, although it can be used to extract food items from a recipe in a text form as well.

The input to this function is just a string in free format, so the implementing side needs to do speech-to-text before using the API.

This function will be able to extract several pieces of data:

• meal action, is this food being added or removed from the logs

• meal time (breakfast, dinner, lunch or snack)

• date of log

• recognised name from the LLM

• portion and weight from the LLM

• nutritional data reference as PassioFoodDataInfo

Example of logging a morning breakfast:

UI Example

1. Create a screen where the user can record the voice logging command. Make sure to add the appropriate permissions.

2. The UI should enable the user to start/stop voice recording on a tap of a button. When the voice recording is done collect the string and use the SDK to extract food.

3. Once the SDK returns the results, show the list to the user with the option to deselect incorrect predictions.

The SDK will automatically download the models according to the SDK version, by default the SDK will download compressed files. The download is faster and lighter, and it will take several seconds to decompress on the device. To download uncompressed files and shorter processing on the device you can set a flag below to false.

PassioNutritionAI.shared.requestCompressedFiles = false

After obtaining special approval the developer can also host the models on an internal server.

var passioConfig = PassioConfiguration(key: "your_key")
passioConfig.sdkDownloadsModels = false

In that case, the models will be served directly to the SDK. To find out more about this special configuration please contact your primary point of contact at Passio or reach us via support@passiolife.com

OpenFoodFacts Terms of use

Passio Nutrition-AI SDK added data from Open Food Facts (https://en.openfoodfacts.org/). Each food that contains data from Open Food Facts will be marked by public var isOpenFood: Bool. In case you choose to use foods that are marked by isOpenFood = true you agree to abide by the terms of the Open Food Facts license agreement (https://opendatacommons.org/licenses/odbl/1-0) and their terms of use (https://world.openfoodfacts.org/terms-of-use) and you will have to add to the UI the following license copy:

"This record contains information from Open Food Facts (https://en.openfoodfacts.org), which is made available here under the Open Database License (https://opendatacommons.org/licenses/odbl/1-0)"

If you don't wish to display the Open Food Facts license somewhere in your app, it is your responsibility to ignore results where isOpenFood = true.

SDK key

To use the SDK please make sure you receive your SDK license key from Passio. The SDK WILL NOT WORK without a valid SDK key.

Proxy URL

If you want a layer of security, where you can authenticate the license and fetch the token on your backend, you don't need the SDK key.

In that case, the SDK will communicate with your backend, and your backend will proxy the requests to Passio's servers, adding the access token to the header. This way, the license keys and tokens are never shared in the app code.

You need to provide following things when configuring the SDK:

• Proxy URL - Base url of the target proxy endpoint

• Proxy headers - Required headers to all of the requests

Request Access to the GitHub repository

You will have to download the latest releases from the link below. The command "git clone" WILL NOT download the PassioNutritionAISDK.xcframework. https://github.com/Passiolife/Passio-Nutrition-AI-iOS-SDK-Distribution/releases. Download the PassioSDKQuickStart.zip or the PassioSDKFullDemo.zip and either install the framework using the Swift Package Manager directions or copy the PassioNutritionAISDK.xcframework to your project. Make sure you have followed the directions in the README files.

Minimum Requirements

In order to use the PassioSDK your app needs to meet the following minimal requirements:

• The SDK will only run or compile on iOS 13 or newer.

• Passio SDK can only be used on a device and will not run on a simulator

• The SDK requires access to iPhone's camera

• Weight/Volume estimation will run only on iPhones with Dual Wide Camera (not on DualCamera). It will not run on the models below.

  • iPhone 11 Pro & Pro Max

  • iPhone 12 mini, Pro & Pro Max

If at any point you need help from the Passio team, please reach out to us at support@passiolife.com

At the top of your view controller import the PassioNutritionAISDK and AVFoundation

import AVFoundation
import PassioNutritionAISDK

Add the following properties to your view controller.

let passioSDK = PassioNutritionAI.shared
var videoLayer: AVCaptureVideoPreviewLayer?
var isSDKConfigured = false
var isProcessingResult = false

Create a function to configure the SDK and call it from viewDidLoad. Use the SDK key you received from Passio. We'll add the function for checkCameraAuthorizationAndStartDetection in the next step, so ignore the error for now.

func configureSDK() {
    // If you are using SDK key
    let key = "Your_SDK_Key"
    let passioConfig = PassioConfiguration(key: key)
    
    // If you are using Proxy URL
    var passioConfig = PassioConfiguration()
    passioConfig.proxyUrl = "https://yourdomain.com/nutrition/"
    passioConfig.proxyHeaders = ["header" : "value"] // Set required headers
    
    passioSDK.configure(passioConfiguration: passioConfig) { [weak self] status in
        guard let self else { return }
        
        print("Mode = \(status.mode)\n missingfiles = \(String(describing: status.missingFiles))" )
        self.isSDKConfigured = (status.mode == .isReadyForDetection)
        if self.isSDKConfigured {
            DispatchQueue.main.async {
                self.checkCameraAuthorizationAndStartDetection()
            }
        } else {
            print("SDK configuration failed. Mode: \(status.mode)")
        }
    }
}

override func viewDidLoad() {
    super.viewDidLoad()
    configureSDK()
}

You will receive the PassioStatus back from the SDK.

public struct PassioStatus {
    public internal(set) var mode: PassioSDK.PassioMode { get }
    public internal(set) var missingFiles: [PassioSDK.FileName]? { get }
    public internal(set) var debugMessage: String? { get }
    public internal(set) var activeModels: Int? { get }
}

public enum PassioMode {
    case notReady
    case isBeingConfigured
    case isDownloadingModels
    case isReadyForDetection
    case failedToConfigure
}

If at any point you need help from the Passio team, please reach out to us at support@passiolife.com

Add a function to setup the camera preview layer:

func setupPreviewLayer() {
    guard videoLayer == nil else { return }
    if let videoLayer = passioSDK.getPreviewLayer() {
        self.videoLayer = videoLayer
        videoLayer.frame = view.bounds
        DispatchQueue.main.async { [weak self] in
            self?.view.layer.insertSublayer(videoLayer, at: 0)
        }
    }
}

Add the method startFoodDetection() This code will throw a missing delegate conformance error that we will resolve in the next step, ignore it for now.

func startFoodDetection() {
    setupPreviewLayer()
    DispatchQueue.global(qos: .userInitiated).async { [weak self] in
        guard let self else { return }
        
        let config = FoodDetectionConfiguration(detectVisual: true,
                                                volumeDetectionMode: .none,
                                                detectBarcodes: true,
                                                detectPackagedFood: true)
        passioSDK.startFoodDetection(detectionConfig: config,
                                     foodRecognitionDelegate: self) { ready in
                print("SDK was not configured correctly")
        }
    }
}

Add a function to request authorization to use the camera and start recognition. Call the function from viewWillAppear:

func checkCameraAuthorizationAndStartDetection() {
    if AVCaptureDevice.authorizationStatus(for: .video) == .authorized {
        startFoodDetection()
    } else {
        AVCaptureDevice.requestAccess(for: .video) { granted in
            if granted {
                DispatchQueue.main.async { [weak self] in
                    self?.startFoodDetection()
                }
            } else {
                print("The user didn't grant access to use camera")
            }
        }
    }
}

override func viewWillAppear(_ animated: Bool) {
    super.viewWillAppear(animated)
    checkCameraAuthorizationAndStartDetection()
}

Create a function to stop Food Detection and call it from viewWillDisappear:

func stopFoodDetection() {
    passioSDK.stopFoodDetection()
    videoLayer?.removeFromSuperlayer()
    videoLayer = nil
}

override func viewWillDisappear(_ animated: Bool) {
    super.viewWillDisappear(animated)
    stopFoodDetection()
}

If at any point you need help from the Passio team, please reach out to us at support@passiolife.com

The quick start demo project for iOS can be found here: https://github.com/Passiolife/Passio-iOS-QuickStart/tree/main/PassioQuickStart

Functionality Overview

In this Quick Start Guide, we will cover the following functionalities:

1. Installation

2. Configure the SDK

3. Recognize Image Remote

4. Food Detail

Minimum Requirements:

In order to use the PassioSDK, your app needs to meet the following minimal requirements:

• The SDK will only run or compile on iOS 13 or newer.

• Passio SDK can only be used on a device and will not run on a simulator

• The SDK requires access to iPhone's camera

• Weight/Volume estimation will run only on iPhones with Dual Wide Camera (not on DualCamera). It will not run on the models below.

  • iPhone 11 Pro & Pro Max

  • iPhone 12 mini, Pro & Pro Max

If at any point you need help from the Passio team, please reach out to us at support@passiolife.com

To get started, please proceed to Installation

Before you continue, please make sure you have a new key for version 2.x. The 1.4.x will not work with 2.x SDK!

Remove SDK 1.4.x

1. Open project and delete the PassioSDKiOS framework

Add the new SDK 2.x

1. Drag and Drop the new "PassioNutritionAISDK.xcframework" to the project (select copy if needed)

2. In Projects > Targets > Your App > Frameworks, Libraries, and Embedded Content. Set the "PassioNutritionAISDK.xcframework" to "Embed & Sign"

Find and Replace all

1. import PassioSDKiOS with import PassioNutritionAISDK

2. PassioSDK.shared with PassioNutritionAI.shared

3. detectOCR with detectPackagedFood

4. ocrCandidates with packagedFoodCandidates

5. fetchPassioIDAttributesFor(ocrcode: $0 with fetchPassioIDAttributesFor(packagedFoodCode: $0.packagedFoodCode)

6. autoUpdate with sdkDownloadsModels

7. isAutoUpdating with isDownloadingModels

8. isReadyForNutrition was removed use isReadyForDetection

The servingSizeQuantity was modified from

public var servingSizeQuantity: Int

to:

public var servingSizeQuantity: Double

Changes to the functions

1. The function func searchForFood(byText: String) -> [String] return value was modified to -> [PassioIDAndName].

2. Removed func lookupPassioIDAttributesFor(name: String) -> PassioIDAttributes? use instead func searchForFood(byText: String) -> [PassioIDAndName] and then use the PassioID to lookup the PassioIDAttributes using func lookupPassioIDAttributesFor(passioID: PassioID) -> PassioIDAttributes?.

the function

public func lookupNameForPassioID(passioID: PassioNutritionAISDK.PassioID) -> String?

was renamed to

public func lookupIconFor(passioID: PassioNutritionAISDK.PassioID, size: PassioNutritionAISDK.IconSize = IconSize.px90, entityType: PassioNutritionAISDK.PassioIDEntityType = .item) -> (UIImage, Bool)

where the Bool Returns: UIImage and a bool, The boolean is true if the icons is food icon or false if it's a placeholder icon. If you get false you can use the asycronous funcion to "fetchIconFor" the icons from

public func fetchIconFor(passioID: PassioNutritionAISDK.PassioID, size: PassioNutritionAISDK.IconSize = IconSize.px90, entityType: PassioNutritionAISDK.PassioIDEntityType = .item, completion: @escaping (UIImage?) -> Void)

Added UIImageView extension please check demo app to check how to use it.

extension UIImageView {
    public func loadPassioIconBy(passioID: PassioNutritionAISDK.PassioID, entityType: PassioNutritionAISDK.PassioIDEntityType, size: PassioNutritionAISDK.IconSize = .px90, completion: @escaping (PassioNutritionAISDK.PassioID, UIImage) -> Void)
}

Removed from the SDK

1. References to logo detection

2. status.mode == .isReadyForNutrition use only .isReadyForDetection

all the images from PassioIDAttributes and PassioFoodItemData where removed.

public var image: UIImage? { get }
public var imageName: String { get }

PassioDownloadDelegate was merged into the PassioStatusDelegate

public protocol PassioStatusDelegate : AnyObject {
    func passioStatusChanged(status: PassioNutritionAISDK.PassioStatus)
    func passioProcessing (filesLeft: Int)
    func completedDownloadingAllFiles(filesLocalURLs: [PassioNutritionAISDK.FileLocalURL])
    func completedDownloadingFile(fileLocalURL: PassioNutritionAISDK.FileLocalURL, filesLeft: Int)
    func downloadingError(message: String)
}

Copyright 2022 Passio Inc

If at any point you need help from the Passio team, please reach out to us at support@passiolife.com

We'll now resolve the error introduced earlier by implementing FoodRecognitionDelegate:

extension ViewController: FoodRecognitionDelegate {
    func recognitionResults(candidates: FoodCandidates?, image: UIImage?) {
        guard !isProcessingResult else { return }
        
        if let candidates = candidates?.barcodeCandidates,
           let candidate = candidates.first {
            print("Found barcode: \(candidate.value)")
        }
        
        if let candidates = candidates?.packagedFoodCandidates,
           let candidate = candidates.first {
            print("Found packaged food: \(candidate.packagedFoodCode)")
        }
        
        if let candidates = candidates?.detectedCandidates,
           let candidate = candidates.first {
            isProcessingResult = true
            PassioNutritionAI.shared.fetchFoodItemFor(passioID: candidate.passioID) { [weak self] foodItem in
                if let foodItem {
                    let nutrients250g = foodItem.nutrients(weight: Measurement<UnitMass>(value: 250.0, unit: .grams))
                    
                    if let calories = nutrients250g.calories()?.converted(to: .kilocalories) {
                        print("Food: \(foodItem.foodItemName): \(calories.value) calories")
                    }
                }
                self?.isProcessingResult = false
            }
        }
    }
}

If at any point you need help from the Passio team, please reach out to us at support@passiolife.com

There could be cases when a certain food is not present in Passio's Nutritional Database. The standard practice in that case would be to guide the users through a creation of user created food items. These items are usually only accessible for that particular user, but there is an API that can be used to notify Passio of such items.

PassioFoodItems submitted using this API go into our verification queue, and after their data has been verified by our nutrition data experts, they become available in the Nutritional Database.

On the other hand, there could be food items that have incorrect or outdated data. For these cases, the SDK provides an API to report such items, so that our data team can update them in the Nutritional Database.

UI Example

1. Add a button when displaying the food item to report incorrect data.

2. Show a form where users can add their feedback as well as provide the product code (barcode) of the item.

3. Upon clicking the "Report" button, invoke the reportFoodItem function.

Quick Start Demo

A fast and easy way to get started with the SDK is to test it inside of PassioSDKQuickStart Demo App included in this package. Here are the steps:

1. Open the project in Xcode:

2. Replace the SDK Key in the PassioQuickStartViewController.swift file with the license key you get from Passio

3. Connect your iPhone and run

4. Modify the app bundle from "com.PassioDemoApp.demo" to "com.yourcompany...."

5. Run the demo app on your iPhone.

6. For support, please contact support@passiolife.com

Testing the SDK During the Integration

You can test the SDK throughout your integration process and even before the integration. To test the SDK before the integration, we encourage you to use Passio's Reference App bundled with the SDK.

After you build the reference app or get the recognition to work inside of your app, you can test that the technology is working properly by pointing the phone at at parts of the image below to test recognition. You can test the following elements:

• Food (non packaged)

• Barcode

• Packaged food

• Nutrition facts

class PassioExternalConnector 

var passioKeyForSDK: String {
    "YourPassioSDKKey"
}

If at any point you need help from the Passio team, please reach out to us at support@passiolife.com

The Nutrition Advisor is an intelligent chatbot designed to offer personalised guidance on nutrition, diets, and healthy eating habits. Whether you're looking to explore a new diet, understand the nutritional value of certain foods, or find delicious and balanced recipes, the Nutrition Advisor is here to assist. Users can interact with the Nutrition Advisor through text messages, engaging in a conversational experience where they can ask questions about various diets like keto, vegan, Mediterranean, and more. The chatbot provides detailed insights into the benefits, potential drawbacks, and suitability of these diets based on individual goals and preferences.

Conversation flow

Use these steps to initialise the conversation and enable users to send messages:

1. To start a conversation initConversation has to be invoked. The response will either indicate a success state, which means message can be sent to the advisor. In case of an error, an error message will detail the cause.

2. Text messages are sent using the sendMessage API. The reply (if successful) will contain a PassioAdvisorResponse response. This object consist of:

• threadId - identifies the conversation

• messageId - identifies the specific message

• markupContent - text response from the advisor denoted with markup tags. This can be used to style the text when rendering

• rawConent - text response without the markup tags

• tools - list of actions that can be executed by the advisor. Only should be used by the SDK

• extracedIngredients - will contain a list of foods if the request to the advisor was an image recognition request or extract ingredients from text message request

1. Image are sent to recognition using sendImage. As mentioned above, in the PassioAdvisorResponse object, the extractedIngredients property will hold the recognised foods.

NutritionAdvisor.shared.initConversation { advisorResult in
    switch advisorResult {
        case .success:
            print("The advisor can start acceping messages")
        case .failure(let error):
            print("The advisor can not start acceping messages. Error:- \(error.errorMessage)")
    }
}
let message = "Hi! I would like a recipe for a healthy burger"
NutritionAdvisor.shared.sendMessage(message: message) { advisorResponse in
    switch advisorResponse {
        case .success(let response): // Show text in a chat bubble
            print("Response:- \(response.rawContent)")
        case .failure(let error): // Handle error
            print("Chat Error:- \(error.errorMessage)")
    }
}
let image = Bundle.main.path(forResource: "image1", ofType: "png")
NutritionAdvisor.shared.sendImage(image: image) { advisorResponse in
    switch advisorResponse {
        case .success(let response): // Show list of foods
            print("Image Response:- \(response.extractedIngredients)")
        case .failure(let error): // Handle error
            print("Image Chat Error:- \(error.errorMessage)")
    }
}

NutritionAdvisor.instance.initConversation { result ->
    when (result) {
        is PassioResult.Success -> /* the advisor can start acceping messages */
        is PassioResult.Error -> /* handle conversation init error */
    }
}

NutritionAdvisor.instance.sendMessage("Hi! I would like a recipe for a healthy burger") { result ->
    when (result) {
        is PassioResult.Error -> {
            val errorMessage = result.message
            // Handle error
        }
        is PassioResult.Success -> {
            val text = result.value.rawContent
            // Show text in a chat bubble
        }
    }
}

val bitmap = loadBitmap()
NutritionAdvisor.instance.sendImage(bitmap) { result ->
    when (result) {
        is PassioResult.Error -> {
            val errorMessage = result.message
            // Handle error
        }
        is PassioResult.Success -> {
            val foods = result.value.extractedIngredients
            // Show list of foods
        }
    }
}

import { NutritionAdvisor } from '@passiolife/nutritionai-react-native-sdk-v3'

const [loadingState, setLoadingState] = useState<SDKStatus>('Init')
  useEffect(() => {
    async function configure() {
      try {
            NutritionAdvisor.initConversation()
              .then((item) => {
                setLoadingState(
                  item?.status === 'Success' ? 'Success' : 'Error'
                )
              })
              .catch((error) => {
                setLoadingState(error)
              })
      } catch (err) {
        console.error(`PassioSDK Error ${err}`)
        setLoadingState('Error')
      }
    }
    configure()
  }, [])


 const response = await NutritionAdvisor.sendMessage(message)
 switch (response?.status) {
          case 'Success':
            // get your response here {response.response}
            return
          case 'Error':
          // get your erorr response here {response.message}
            return
        }

const response = await NutritionAdvisor.sendImage(imageURI)
 switch (response?.status) {
          case 'Success':
            // get your response here {response.response}
            return
          case 'Error':
          // get your erorr response here {response.message}
            return
        }

final result = await NutritionAdvisor.instance.initConversation();
switch (result) {
  case Success():
    // Initialization succeeded, the advisor can start accepting messages
    break;
  case Error():
    // Handle the init error here
    break;
}

final result = await NutritionAdvisor.instance.sendMessage('Hi! I would like a recipe for a healthy burger');
switch (result) {
  case Success():
    // Parse the response from result.value
    break;
  case Error():
    // Handle the error that occurred while sending the message
    break;
}

Uint8List bytes = loadImage();
final result = await NutritionAdvisor.instance.sendImage(bytes);
switch (result) {
  case Success():
    // The food items are found in imageResult.value.extractedIngredients
    break;
  case Error():
    // Handle the error that occurred while sending the message
    break;
}

Extract ingredients

In the case that a PassioAdvisorResponse contains a tool called searchMatchIngredient, the Advisor has the ability to parse the text response and return the list of foods. If a message that has this tool is passed to the fetchIngredients API, the returned response will contain foods in the extracedIngredients field.

This feature is particularly useful when trying to log foods when directly communicating with the chatbot. A common use case is when user ask for a specific recipe. They can then initiate the extraction of foods from the message, and see their nutrition value and finally log the entire meal in one action.

UI Example

1. Create a screen where the user can type messages and send images to the NutritionAdvisor. Upon initiating the screen, set up the conversation by calling initConversation.

2. When the user types a message in the input text field and taps "Send", invoke the sendMessage function. Because this is an asynchronous function, add an animation to indicate this behaviour.

3. If the response from the advisor contains a tool called searchMatchIngredient show an action button "Find foods" to initiate extraction of the foods. Invoke the fetchIngredients function using the response from the Advisor.

4. Create a view where the user can see the extracted foods, their serving size and have the ability to log them.

1. Add the ability for the user to select an image from the library of the phone. Once the image is obtained, invoke the sendImage to get the list of foods. If foods are recognised, the response will contain a list of foods, use the same result view to show them.

Install Swift Package for Xcode 14.3 or newer

1. Open your Xcode project.

2. Go to File > Swift Packages > Add Package Dependency.

3. In the "Add Package Dependency" dialog box, paste the URL: https://github.com/Passiolife/Passio-Nutrition-AI-iOS-SDK-Distribution

4. Click "Next". Xcode will validate the package and its dependencies.

5. In the next dialog box, you'll be asked to specify the version or branch you want to use. You can choose main for the latest version or specify a specific version or branch.

6. After you've made your selection, click "Next".

7. You'll then be prompted to select the targets in your project that should include the package. Check the boxes for the targets you want to include.

8. Click "Finish" to add the package to your project.

9. Xcode will download and add the PassioNutritionAISDK to your project. You can now import and start using the PassioNutritionAISDK.

Manual Installation For Xcode lower than 14.3

• Download the "PassioNutritionAISDK.xcframework.zip" file from https://github.com/Passiolife/Passio-Nutrition-AI-iOS-SDK-Distribution/blob/main/PassioNutritionAISDK.xcframework.zip

• Unzip it and drag and drop it into your project. Make sure to select "Copy items if needed".

• In project "General" -> "Frameworks, Libraries and Embedded Content" Change to "Embed & Sign"

Edit your Info.plist

• If opening from Xcode, right click and select 'open as source code'

• To allow camera usage add:

`<key>>NSCameraUsageDescription</key><string>For real-time food recognition</string>`.

If at any point you need help from the Passio team, please reach out to us at support@passiolife.com

Using Swift Package Manager (Recommended)

• To add a Passio SDK as package dependency to your Xcode project, select File > Add Package Dependency and enter following repository URL https://github.com/Passiolife/Passio-Nutrition-AI-iOS-SDK-Distribution

• Click "Next". Xcode will validate the package and its dependencies.

• In the next dialog box, you'll be asked to specify the version or branch you want to use. Decide whether your project accepts updates to a package dependency "Up to the next major version" or "Up to the next minor version". To be more restrictive, select a specific version range or an exact version. Major versions tend to have more significant changes than minor versions, and may require you to modify your code when they update.

• Xcode will download and add the PassioNutritionAISDK to your project. You can now import and start using the PassioNutritionAISDK.

Manual Installation (For Xcode lower than version 14.3)

• Download the PassioNutritionAISDK.xcframework.zip file from https://github.com/Passiolife/Passio-Nutrition-AI-iOS-SDK-Distribution/blob/main/PassioNutritionAISDK.xcframework.zip

• Unzip it and drag and drop it into your project. Make sure to select "Copy items if needed".

• In project "General" -> "Frameworks, Libraries and Embedded Content" Change to "Embed & Sign"

The main object containing the nutritional information is called PassioFoodItem. This object is fetched from Passio's backend using one of several fetchFoodItemFor... functions.

PassioFoodItem

The PassioFoodItem is a top level object that can represent a single food like a banana, or a complex recipe like caesar salad.

public struct PassioFoodItem: Codable {
    public let id: String
    public let name: String
    public let details: String
    public let iconId: String
    public let licenseCopy: String
    public let amount: PassioNutritionAISDK.PassioFoodAmount
    public let ingredients: [PassioNutritionAISDK.PassioIngredient]
    public let refCode: String?
}

data class PassioFoodItem(
    val id: String,
    val refCode: String,
    val name: String,
    val details: String,
    val iconId: String,
    val amount: PassioFoodAmount,
    val ingredients: List<PassioIngredient>,
)

export interface PassioFoodItem {
  refCode?: RefCode
  name: string
  details?: strin
  iconId: PassioID | string
  amount: PassioFoodAmount
  ingredients?: PassioIngredient[]
  weight: UnitMass
  isOpenFood?: boolean
  openFoodLicense?: string
  id: string
}

class PassioFoodItem {
  final PassioFoodAmount amount;
  final String details;
  final String iconId;
  final String id;
  final List<PassioIngredient> ingredients;
  final String name;
  final String refCode;
  final PassioID? scannedId;
 }

• The name attribute is used to display the name of the food, while the details contains additional information like the brand of the food or the name of the food which was used to populate the nutritional data

• iconId is used with the SDK function fetchIconFor to fetch a small image of the food

• refCode is a parameter that can be used later on to retrieve the full food item again

• amount details the available serving size as well as the currently selected serving quantity and unit, used when calculating nutrients

There are three functions that can be used to fetch nutrient information for the PassioFoodItem:

PassioNutritionAI.shared.fetchFoodItemFor(passioID: "VEG0025") { (foodItem) in
        
  if let foodItem {
  
      // Getting nutrients for the referent 100 grams
      let nutrientsRef = foodItem.nutrientsReference()
            
      // Getting nutrients for a specific weight
      let nutrients250g = foodItem.nutrients(weight: Measurement<UnitMass>(value: 250.0, unit: .grams))
      
      // Getting nutrients using the amount->selectedQuantity and
      // amount->selectedUnit
      let nutrientsServing = foodItem.nutrientsSelectedSize()
            
      // Weight of the ingredients
      let weight = foodItem.weight()
  }
}

PassioSDK.instance.fetchFoodItemForPassioID("VEG0025") { foodItem ->
    // Getting nutrients for the referent 100 grams
    val nutrientsRef = foodItem!!.nutrientsReference()
    // Getting nutrients for a specific weight
    val nutrients250g = foodItem.nutrients(UnitMass(Grams, 250.0))
    // Getting nutrients using the amount->selectedQuantity and
    // amount->selectedUnit
    val nutrientsServing = foodItem.nutrientsSelectedSize()
    // Weight of the ingredients
val weight = foodItem.weight()
}

   const passioFoodItem = await PassioSDK.fetchFoodItemForPassioID("VEG0025")
  
   // Getting nutrients for the referent 100 grams
   const nutrients =  PassioSDK.getNutrientsReferenceOfPassioFoodItem(passioFoodItem)
   
   // Getting nutrients for a specific weight
   const nutrients =  PassioSDK.getNutrientsOfPassioFoodItem(passioFoodItem,{unit:'g',value:250})
  
   // Getting nutrients using the amount->selectedQuantity and
   // amount->selectedUnit
   const nutrients =  PassioSDK.getNutrientsSelectedSizeOfPassioFoodItem(passioFoodItem)
 
   // Weight of the ingredients
   const val weight = passioFoodItem.weight()

    final foodItem = await NutritionAI.instance.fetchFoodItemForPassioID('VEG0025');
    // Getting nutrients for the referent 100 grams
    final nutrientsRef = foodItem!.nutrientsReference();
    // Getting nutrients for a specific weight
    final nutrients250g =
        foodItem.nutrients(UnitMass(250.0, UnitMassType.grams));
    // Getting nutrients using the amount->selectedQuantity and
    // amount->selectedUnit
    final nutrientsServing = foodItem.nutrientsSelectedSize();
    // Weight of the ingredients
    final weight = foodItem.weight();

The weight function is used to fetch the sum mass of all of the ingredients.

PassioFoodAmount

This class holds the information about possible serving sizes and serving units, as well as the currently selected unit and quantity. Example of values for food item "apples"

• serving units: small (2-3/4" dia), medium, large (3-1/4" dia), cup, oz, gram, ...

• serving sizes: 1 small (2-3/4" dia), 2 cups, 100 grams, ...

• selected quantity: 1

• selected unit: medium

Serving units holds the list of all of the units that the system has the correct weight for. Serving sizes holds the list of predefines matches of a quantity and a serving unit. The selected quantity and unit are used to calculate nutrients on the PassioFoodItem level. While these values can be changed, the initial values are set by the SDK as a default portion.

PassioIngredient

Every PassioFoodItem has a list of ingredients. A food ingredient holds the nutritional information along with the portion size of each ingredient.

public struct PassioIngredient: Codable {
    public let id: String
    public let name: String
    public let iconId: String
    public let amount: PassioFoodAmount
    public let referenceNutrients: PassioNutrients
    public let metadata: PassioFoodMetadata
    public let refCode: String?
}

data class PassioIngredient(
    val id: String,
    val refCode: String,
    val name: String,
    val iconId: String,
    val amount: PassioFoodAmount,
    val referenceNutrients: PassioNutrients,
    val metadata: PassioFoodMetadata,
)

export interface PassioIngredient {
  name: string
  id: PassioID
  iconId: PassioID | string
  weight?: UnitMass
  referenceNutrients?: PassioNutrients
  nutrients?: PassioNutrients
  metadata?: PassioFoodMetadata
  amount?: PassioFoodAmount
}

class PassioIngredient {
  final PassioFoodAmount amount;
  final String iconId;
  final String id;
  final PassioFoodMetadata metadata;
  final String name;
  final String refCode;
  final PassioNutrients referenceNutrients;
}

Similarly to PassioFoodItem, the ingredient also has a name, iconId, refCode, and amount.

But, the most important part of the ingredient class is the PassioNutrients and it's methods. The referenceNutrients attribute holds the referent (in most cases 100 grams) macronutrients and micronutrients, and are used to calculate the resulting nutrient values when changing serving sizes of the whole PassioFoodItem.

Also, an ingredient has PassioMetadata field, storing the information on the origin of the nutritional data, barcode value, possible ingredients (if the PassioFoodItem is a packaged food), and a list of tags.

Fetching nutrients

PassioNutritionAI.shared.fetchFoodItemFor(passioID: "VEG0025") { (foodItem) in
  if let foodItem {
     let nutrients = foodItem.nutrientsReference()
     // Calories for 100 grams, values is in kCal
     let calories = nutrients.calories()?.value
     // Carbs for 100 grams, value is in grams
     let carbs = nutrients.carbs()?.value
     // Potassium for 100 grams, value is in milligrams
     let potassiumMilli = nutrients.potassium()?.value
     // Potassium for 100 grams, value is in grams
     let potassiumGrams = nutrients.potassium()?.gramsValue
  }
}

PassioSDK.instance.fetchFoodItemForPassioID("VEG0025") { foodItem ->
    val nutrients = foodItem!!.nutrientsReference()
    // Calories for 100 grams, values is in kCal
    val calories = nutrients.calories()?.value
    // Carbs for 100 grams, value is in grams
    val carbs = nutrients.carbs()?.value
    // Potassium for 100 grams, value is in milligrams
    val potassiumMilli = nutrients.potassium()?.value
    // Potassium for 100 grams, value is in grams
    val potassiumGrams = nutrients.potassium()?.gramsValue()
}

const passioFoodItem = await PassioSDKBridge.fetchFoodItemForPassioID("VEG0025")

   const nutrients =  PassioSDK.getNutrientsReferenceOfPassioFoodItem(passioFoodItem)
    // Calories for 100 grams, values is in kCal
     let calories = passioNutrients.calories?.value
     // Carbs for 100 grams, value is in grams
     let carbs = passioNutrients.carbs?.value
     // Potassium for 100 grams, value is in milligrams
     let potassiumMilli = passioNutrients.potassium?.value

    final foodItem = await NutritionAI.instance.fetchFoodItemForPassioID('VEG0025');
    final nutrients = foodItem!.nutrientsReference();
    // Calories for 100 grams, values is in kCal
    final calories = nutrients.calories?.value;
    // Carbs for 100 grams, value is in grams
    final carbs = nutrients.carbs?.value;
    // Potassium for 100 grams, value is in milligrams
    final potassiumMilli = nutrients.potassium?.value;
    // Potassium for 100 grams, value is in grams
    final potassiumGrams = nutrients.potassium?.gramsValue();

Single food item vs recipe

The PassioFoodItem class encompasses both single food items like an avocado as well as items with multiple ingredients such as homemade caprese salad. The difference between these two items is how is the PassioFoodAmount calculated from the top-level PassioFoodItem.

Homemade caprese salad:

• weight(): 176.5 grams

• amount->selectedQuantity = 1

• amount->selectedUnit = "serving"

• ingredients: 1. fresh sliced cheese, 84 grams, amount->selectedQuantity = 3, amount-> selectedUnit = "oz" 2. balsamic vinegar, 16 grams, amount->selectedQuantity = 1, amount-> selectedUnit = "tbsp" 3. fresh basil, 3 grams, amount->selectedQuantity = 6, amount-> selectedUnit = "leaves" 4. tomatoes, 60 grams, amount->selectedQuantity = 3, amount-> selectedUnit = "medium slice" 5. olive oil, 13.5 grams, amount->selectedQuantity = 1, amount-> selectedUnit = "tbsp"

Invoking the function foodItem.nutrientsServingSize() will fetch the nutrients for the whole recipe, calculated by the weight of the serving size "1 serving".

Avocado:

• weight(): 201 grams

• amount->selectedQuantity = 1

• amount->selectedUnit = "avocado"

• ingredients: 1. avodaco, raw, 201 grmas, amount->selectedQuantity = 1, amount->selectedUnit = "avocado"

A single food item will always have only one ingredient. Also, the amount object of the PassioFoodItem will be the same as the amount of the first ingredient. The ingredient is the item in the database used for nutritional data. For example, If the SDK recognises "milk" during the visual detection process, the default food item for "milk" is "milk, whole, 3.25% milkfat, without added vitamin a and vitamin d".

UI Example

1. Fetch the PassioFoodItem

2. Use the name, details and iconId to create the food header view

3. Use the nutrientsSelectedSize->calories, carbs, protein and fat to show the marcos

4. Use the amount class to create the serving size view

5. Give the user the option to log the food

Suggestions

The SDK provides the ability to fetch a predefined set of suggested foods for logging, depending on the current time of day

public enum PassioMealTime: String, Codable {
    case breakfast
    case lunch
    case dinner
    case snack
}

public func fetchSuggestions(mealTime: PassioMealTime,  completion: @escaping ([PassioFoodDataInfo]) -> Void)

enum class PassioMealTime(val mealName: String) {
    BREAKFAST("breakfast"),
    LUNCH("lunch"),
    DINNER("dinner"),
    SNACK("snack")
}

fun fetchSuggestions(
    mealTime: PassioMealTime,
    callback: (results: List<PassioFoodDataInfo>) -> Unit
)

export type PassioMealTime = 'breakfast' | 'lunch' | 'dinner' | 'snack'  

fetchSuggestions(
  mealTime: PassioMealTime
): Promise<PassioFoodDataInfo[] | null>

enum PassioMealTime {
  breakfast,
  lunch,
  dinner,
  snack,
}

Future<List<PassioFoodDataInfo>> fetchSuggestions(PassioMealTime mealTime)

Example of fetching suggestions for snack meal time:

banana, coffee, black tea, coke, cheddar cheese, chocolate chip cookies, potato chips, blueberry muffin, plain yogurt, strawberries, ...

Meal Plans

Meal Plans provide users suggestions which foods to consume depending on their dietary preference and time of day. The Meal Plan api is divided into two functions:

• fetchMealPlans that is used to retrieve all of the possible meal plans. These meal plans usually correlate with a specific diet like "Keto Diet".

• fetchMealPlanForDay is an api that is used to fetch specific foods recommended for a certain day of a target meal plan.

public func fetchMealPlans(completion: @escaping ([PassioMealPlan]) -> Void)

public func fetchMealPlanForDay(mealPlanLabel: String,
                                day: Int,
                                completion: @escaping ([PassioMealPlanItem]) -> Void)

public struct PassioMealPlan: Codable, Equatable {
    public var mealPlanLabel: String?
    public var mealPlanTitle: String?
    public var carbsTarget: Int?
    public var proteinTarget: Int?
    public var fatTarget: Int?
}

public struct PassioMealPlanItem {
    public var dayNumber: Int?
    public var dayTitle: String?
    public var mealTime: PassioMealTime?
    public var meal: PassioFoodDataInfo?
}

fun fetchMealPlans(callback: (result: List<PassioMealPlan>) -> Unit)

fun fetchMealPlanForDay(
    mealPlanLabel: String,
    day: Int,
    callback: (result: List<PassioMealPlanItem>) -> Unit
)

data class PassioMealPlan(
    val mealPlanTitle: String,
    val mealPlanLabel: String,
    val carbTarget: Int,
    val proteinTarget: Int,
    val fatTarget: Int
)

data class PassioMealPlanItem(
    val dayNumber: Int,
    val dayTitle: String,
    val mealTime: PassioMealTime,
    val meal: PassioFoodDataInfo
)

fetchMealPlans(): Promise<PassioMealPlan[] | null>

fetchMealPlanForDay(
  mealPlanLabel: string,
  day: number
): Promise<PassioMealPlanItem[] | null>

export interface PassioMealPlan {
  carbsTarget?: number
  fatTarget?: number
  mealPlanLabel: string
  mealPlanTitle: string
  proteinTarget?: number
}

export interface PassioMealPlanItem {
  dayNumber: number
  dayTitle: string
  mealTime: PassioMealTime
  meal: PassioFoodDataInfo
}

Future<List<PassioMealPlan>> fetchMealPlans()

Future<List<PassioMealPlanItem>> fetchMealPlanForDay(String mealPlanLabel, int day)

class PassioMealPlan {
  final int carbTarget;
  final int fatTarget;
  final String mealPlanLabel;
  final String mealPlanTitle;
  final int proteinTarget;
}

class PassioMealPlanItem {
  final int dayNumber;
  final String dayTitle;
  final PassioFoodDataInfo meal;
  final PassioMealTime mealTime;
}