The main object containing the nutritional information is called PassioFoodItem. This object is fetched from Passio's backend using one of several fetchFoodItemFor... functions.

PassioFoodItem

The PassioFoodItem is a top level object that can represent a single food like a banana, or a complex recipe like caesar salad.

public struct PassioFoodItem: Codable {
    public let id: String
    public let name: String
    public let details: String
    public let iconId: String
    public let licenseCopy: String
    public let amount: PassioNutritionAISDK.PassioFoodAmount
    public let ingredients: [PassioNutritionAISDK.PassioIngredient]
    public let refCode: String?
}

data class PassioFoodItem(
    val id: String,
    val refCode: String,
    val name: String,
    val details: String,
    val iconId: String,
    val amount: PassioFoodAmount,
    val ingredients: List<PassioIngredient>,
)

export interface PassioFoodItem {
  refCode?: RefCode
  name: string
  details?: strin
  iconId: PassioID | string
  amount: PassioFoodAmount
  ingredients?: PassioIngredient[]
  weight: UnitMass
  isOpenFood?: boolean
  openFoodLicense?: string
  id: string
}

class PassioFoodItem {
  final PassioFoodAmount amount;
  final String details;
  final String iconId;
  final String id;
  final List<PassioIngredient> ingredients;
  final String name;
  final String refCode;
  final PassioID? scannedId;
 }

• The name attribute is used to display the name of the food, while the details contains additional information like the brand of the food or the name of the food which was used to populate the nutritional data

• iconId is used with the SDK function fetchIconFor to fetch a small image of the food

• refCode is a parameter that can be used later on to retrieve the full food item again

• amount details the available serving size as well as the currently selected serving quantity and unit, used when calculating nutrients

There are three functions that can be used to fetch nutrient information for the PassioFoodItem:

PassioNutritionAI.shared.fetchFoodItemFor(passioID: "VEG0025") { (foodItem) in
        
  if let foodItem {
  
      // Getting nutrients for the referent 100 grams
      let nutrientsRef = foodItem.nutrientsReference()
            
      // Getting nutrients for a specific weight
      let nutrients250g = foodItem.nutrients(weight: Measurement<UnitMass>(value: 250.0, unit: .grams))
      
      // Getting nutrients using the amount->selectedQuantity and
      // amount->selectedUnit
      let nutrientsServing = foodItem.nutrientsSelectedSize()
            
      // Weight of the ingredients
      let weight = foodItem.weight()
  }
}

   const passioFoodItem = await PassioSDK.fetchFoodItemForPassioID("VEG0025")
  
   // Getting nutrients for the referent 100 grams
   const nutrients =  PassioSDK.getNutrientsReferenceOfPassioFoodItem(passioFoodItem)
   
   // Getting nutrients for a specific weight
   const nutrients =  PassioSDK.getNutrientsOfPassioFoodItem(passioFoodItem,{unit:'g',value:250})
  
   // Getting nutrients using the amount->selectedQuantity and
   // amount->selectedUnit
   const nutrients =  PassioSDK.getNutrientsSelectedSizeOfPassioFoodItem(passioFoodItem)
 
   // Weight of the ingredients
   const val weight = passioFoodItem.weight()

    final foodItem = await NutritionAI.instance.fetchFoodItemForPassioID('VEG0025');
    // Getting nutrients for the referent 100 grams
    final nutrientsRef = foodItem!.nutrientsReference();
    // Getting nutrients for a specific weight
    final nutrients250g =
        foodItem.nutrients(UnitMass(250.0, UnitMassType.grams));
    // Getting nutrients using the amount->selectedQuantity and
    // amount->selectedUnit
    final nutrientsServing = foodItem.nutrientsSelectedSize();
    // Weight of the ingredients
    final weight = foodItem.weight();

The weight function is used to fetch the sum mass of all of the ingredients.

PassioFoodAmount

This class holds the information about possible serving sizes and serving units, as well as the currently selected unit and quantity. Example of values for food item "apples"

• serving units: small (2-3/4" dia), medium, large (3-1/4" dia), cup, oz, gram, ...

• serving sizes: 1 small (2-3/4" dia), 2 cups, 100 grams, ...

• selected quantity: 1

• selected unit: medium

Serving units holds the list of all of the units that the system has the correct weight for. Serving sizes holds the list of predefines matches of a quantity and a serving unit. The selected quantity and unit are used to calculate nutrients on the PassioFoodItem level. While these values can be changed, the initial values are set by the SDK as a default portion.

PassioIngredient

Every PassioFoodItem has a list of ingredients. A food ingredient holds the nutritional information along with the portion size of each ingredient.

public struct PassioIngredient: Codable {
    public let id: String
    public let name: String
    public let iconId: String
    public let amount: PassioFoodAmount
    public let referenceNutrients: PassioNutrients
    public let metadata: PassioFoodMetadata
    public let refCode: String?
}

data class PassioIngredient(
    val id: String,
    val refCode: String,
    val name: String,
    val iconId: String,
    val amount: PassioFoodAmount,
    val referenceNutrients: PassioNutrients,
    val metadata: PassioFoodMetadata,
)

export interface PassioIngredient {
  name: string
  id: PassioID
  iconId: PassioID | string
  weight?: UnitMass
  referenceNutrients?: PassioNutrients
  nutrients?: PassioNutrients
  metadata?: PassioFoodMetadata
  amount?: PassioFoodAmount
}

class PassioIngredient {
  final PassioFoodAmount amount;
  final String iconId;
  final String id;
  final PassioFoodMetadata metadata;
  final String name;
  final String refCode;
  final PassioNutrients referenceNutrients;
}

Similarly to #passiofooditem, the ingredient also has a name, iconId, refCode, and amount.

But, the most important part of the ingredient class is the PassioNutrients and it's methods. The referenceNutrients attribute holds the referent (in most cases 100 grams) macronutrients and micronutrients, and are used to calculate the resulting nutrient values when changing serving sizes of the whole PassioFoodItem.

Also, an ingredient has PassioMetadata field, storing the information on the origin of the nutritional data, barcode value, possible ingredients (if the PassioFoodItem is a packaged food), and a list of tags.

Fetching nutrients

PassioNutritionAI.shared.fetchFoodItemFor(passioID: "VEG0025") { (foodItem) in
  if let foodItem {
     let nutrients = foodItem.nutrientsReference()
     // Calories for 100 grams, values is in kCal
     let calories = nutrients.calories()?.value
     // Carbs for 100 grams, value is in grams
     let carbs = nutrients.carbs()?.value
     // Potassium for 100 grams, value is in milligrams
     let potassiumMilli = nutrients.potassium()?.value
     // Potassium for 100 grams, value is in grams
     let potassiumGrams = nutrients.potassium()?.gramsValue
  }
}

PassioSDK.instance.fetchFoodItemForPassioID("VEG0025") { foodItem ->
    val nutrients = foodItem!!.nutrientsReference()
    // Calories for 100 grams, values is in kCal
    val calories = nutrients.calories()?.value
    // Carbs for 100 grams, value is in grams
    val carbs = nutrients.carbs()?.value
    // Potassium for 100 grams, value is in milligrams
    val potassiumMilli = nutrients.potassium()?.value
    // Potassium for 100 grams, value is in grams
    val potassiumGrams = nutrients.potassium()?.gramsValue()
}

const passioFoodItem = await PassioSDKBridge.fetchFoodItemForPassioID("VEG0025")

   const nutrients =  PassioSDK.getNutrientsReferenceOfPassioFoodItem(passioFoodItem)
    // Calories for 100 grams, values is in kCal
     let calories = passioNutrients.calories?.value
     // Carbs for 100 grams, value is in grams
     let carbs = passioNutrients.carbs?.value
     // Potassium for 100 grams, value is in milligrams
     let potassiumMilli = passioNutrients.potassium?.value

    final foodItem = await NutritionAI.instance.fetchFoodItemForPassioID('VEG0025');
    final nutrients = foodItem!.nutrientsReference();
    // Calories for 100 grams, values is in kCal
    final calories = nutrients.calories?.value;
    // Carbs for 100 grams, value is in grams
    final carbs = nutrients.carbs?.value;
    // Potassium for 100 grams, value is in milligrams
    final potassiumMilli = nutrients.potassium?.value;
    // Potassium for 100 grams, value is in grams
    final potassiumGrams = nutrients.potassium?.gramsValue();

Single food item vs recipe

The PassioFoodItem class encompasses both single food items like an avocado as well as items with multiple ingredients such as homemade caprese salad. The difference between these two items is how is the PassioFoodAmount calculated from the top-level PassioFoodItem.

Homemade caprese salad:

• weight(): 176.5 grams

• amount->selectedQuantity = 1

• amount->selectedUnit = "serving"

• ingredients: 1. fresh sliced cheese, 84 grams, amount->selectedQuantity = 3, amount-> selectedUnit = "oz" 2. balsamic vinegar, 16 grams, amount->selectedQuantity = 1, amount-> selectedUnit = "tbsp" 3. fresh basil, 3 grams, amount->selectedQuantity = 6, amount-> selectedUnit = "leaves" 4. tomatoes, 60 grams, amount->selectedQuantity = 3, amount-> selectedUnit = "medium slice" 5. olive oil, 13.5 grams, amount->selectedQuantity = 1, amount-> selectedUnit = "tbsp"

Invoking the function foodItem.nutrientsServingSize() will fetch the nutrients for the whole recipe, calculated by the weight of the serving size "1 serving".

Avocado:

• weight(): 201 grams

• amount->selectedQuantity = 1

• amount->selectedUnit = "avocado"

• ingredients: 1. avodaco, raw, 201 grmas, amount->selectedQuantity = 1, amount->selectedUnit = "avocado"

A single food item will always have only one ingredient. Also, the amount object of the PassioFoodItem will be the same as the amount of the first ingredient. The ingredient is the item in the database used for nutritional data. For example, If the SDK recognises "milk" during the visual detection process, the default food item for "milk" is "milk, whole, 3.25% milkfat, without added vitamin a and vitamin d".

UI Example

1. Fetch the PassioFoodItem

2. Use the name, details and iconId to create the food header view

3. Use the nutrientsSelectedSize->calories, carbs, protein and fat to show the marcos

4. Use the amount class to create the serving size view

5. Give the user the option to log the food

SDK Key

To use the SDK please make sure you receive your SDK license key from Passio. The SDK WILL NOT WORK without a valid SDK key.

To obtain an SDK key visit https://www.passio.ai/nutrition-ai

Minimum requirements

These are the minimum requirements for every supported platform

The configuration process of the SDK has severable responsibilities:

• Validation of the license key

• Validating the currently present files

• Downloading or updating the files required by the SDK to do recognition

• Preparing the files to be used for inference

The SDK is configured using a PassioConfiguration object. This object is used to define the location of the files that the SDK requires.

SDK downloads the models

The default behaviour (or by setting the sdkDownloadsModels field to true) of the SDK is to download the required files for local recognition. If the current files that are present on the device are older than the version of the SDK library, the SDK will download the newer version in the background, but still run the session with the older version of the files. Once the download is finished, the new files will be applied on the next session. Don't forget to add the import statement for the nutrition sdk.

import PassioNutritionAISDK
import AVFoundation

...

let passioSDK = PassioNutritionAI.shared

override func viewDidLoad() {
    super.viewDidLoad()
    let key = "Your_PassioSDK_Key"
    let passioConfig = PassioConfiguration(key: key)
    passioSDK.configure(passioConfiguration: passioConfig) { (status) in
        print("Mode = \(status.mode)\nmissingfiles = \(String(describing: status.missingFiles))" )
    }
}

val passioConfiguration = PassioConfiguration(
    this.applicationContext,
    "Your license key here"
).apply {
    sdkDownloadsModels = true // This can be ommited
}

PassioSDK.instance.configure(passioConfiguration) { passioStatus ->
    // Handle PassioStatus states
} 

const [isReady, setIsReady] = useState(false);

// Effect to configure the SDK and request camera permission
// We are adding our developer key in LoadingContainerView
useEffect(() => {
  Promise.all([
    PassioSDK.configure({
      key: 'your-developer-key',
      autoUpdate: true,
    }),
    PassioSDK.requestCameraAuthorization(),
  ]).then(([sdkStatus, cameraAuthorized]) => {
    setIsReady(sdkStatus.mode === 'isReadyForDetection' && cameraAuthorized);
  });
}, []);

import 'package:nutrition_ai/nutrition_ai_sdk.dart';

void configureSDK() async {
   String passioKey = "Your_PassioSDK_Key";
   var configuration = PassioConfiguration(passioKey);
   passioStatus = await NutritionAI.instance.configureSDK(configuration);
   // Handle result of the configuration process.
}

Remote Only configuration

The models required for the SDK to run local recognition can take some time for the first download. If the local recognition is not needed, setting the remoteOnly to true on the PassioConfiguration process will skip the download of the models. Remote recognition functions like recognizeImageRemote or searchForFood will still work.

Handling the configure status object

The result of the configuration process is a PassioStatus object, containing a PassioMode attribute along with the missingFiles, debugMessage, error object and the active models number. The attribute that needs to looked at is the mode.

PassioMode is an enum describing the current state of the Passio SDK configuration process:

• notReady -> The configuration process hasn't started yet.

• failedToConfigure -> There was an error during the configuration process.

• isBeingConfigured -> The SDK is still in the configuration process. Normally, you shouldn't receive this mode as a callback to the configure method. If you do please contact our support team.

• isDownloadingModels -> The files required by the SDK to work are not present and are currently being downloaded.

• isReadyForDetection -> The SDK is configured with the set of files defined by the activeModels attribute.

The missingFiles variable contains a list of all the files missing for the SDK to work optimally. When this list is not null, this doesn't mean that the SDK is not operational. The SDK may be initialized with older files, but it will advocate the download of the newest files through this list.

The debugMessage usually gives more verbose and human-readable information about the configuration process. It will always give an error message if the mode is FAILED_TO_CONFIGURE.

The activeModels will give the version of the files installed and ran by the SDK if the configuration process ran successfully i.e. if the mode is IS_READY_FOR_DETECTION.

Best practices

1. Make sure there is internet connection for the configuration process. The SDK needs to download the license file and the required files. If there is no internet connection the configuration status mode will be failedToConfigure.

2. The PassioStatus mode is notReady -> check debugMessage for the error, usually the wrong license key has been supplied or the files are missing.

3. The PassioStatus mode is isDownloadingModels -> register a PassioStatusListener through the setPassioStatusListener method as it will enable tracking of the download progress.

4. The PassioStatus mode is isReadyForDetection -> still check the missingFiles because the SDK might be running with an older version of the files and newer need to be downloaded.

Welcome to Passio Nutrition-AI iOS SDK! When integrated into your app the SDK provides you with food recognition and nutrition assistant technology. The SDK creates a video preview layer and outputs foods recognized by our computer vision technology in the video feed of your live camera along with nutrition data related to the recognized foods.

As the developer, you have complete control of when to turn on/off the SDK and to configure the outputs which includes:

• food names (e.g. banana, hamburger, fruit salad, quest chocolate bar)

• lists of alternatives for recognized foods (e.g., soy milk would be an alternative of a visually recognized class milk)

• barcodes detected on food packages

• packaged foods recognized by the text detected on food packages

• nutrition information detected on food packages via Passio's Nutrition Facts reader which returns information written in Nutrition Facts labels

• nutrition information associated with the foods

• food weight and volume for certain foods

By default the SDK does not record/store any photos or videos. Instead, as the end user hovers over a food item with his/her camera phone, the SDK recognizes and identifies food items in real time. This hovering action is only transitory/temporary while the end user is pointing the camera at a particular item and is not recorded or stored within the SDK. As a developer, you can configure the SDK to capture images or videos and store them in your app.

Open Food Facts and Terms of Use

Passio Nutrition-AI SDK added data from Open Food Facts (https://en.openfoodfacts.org/). Each food that contains data from Open Food Facts will have the value set in its food origin list. In case you choose to display the these food items you agree to abide by the terms of the Open Food Facts license agreement (https://opendatacommons.org/licenses/odbl/1-0) and their terms of use (https://world.openfoodfacts.org/terms-of-use) and you will have to add to the UI the following license copy:

"This record contains information from Open Food Facts (https://en.openfoodfacts.org), which is made available here under the Open Database License (https://opendatacommons.org/licenses/odbl/1-0)"

Real-time barcode scanning

The SDK can detect barcodes located on the packaging of packaged foods. To implement real-time barcode scanning, the camera setup for continuous image recognition is needed.

Specifically for barcode scanning, these steps are required:

1. When defining a FoodDetectionConfiguration object, enable the detectBarcode flag.

2. In the recognitionResults callback, check the property candidate.barcodeCandidates. If this list is empty, no barcodes have been recognised.

3. If there is a single or multiple barcode candidates, use their value property and invoke this function to fetch the full nutrition data.

public func fetchFoodItemFor(productCode: String, completion: @escaping ((PassioNutritionAISDK.PassioFoodItem?) -> Void))

fun fetchFoodItemForProductCode(
    productCode: String,
    onResult: (foodItem: PassioFoodItem?) -> Unit
)

fetchFoodItemForProductCode(code: Barcode | PackagedFoodCode): Promise<PassioFoodItem | null>

Future<PassioFoodItem?> fetchFoodItemForProductCode(String productCode)

Barcode scanning from an image

To recognise a barcode in an image, refer to the recognizeImageRemote function. The result of the barcode scanning process will come in the packagedFoodItem attribute, and the resultType will be Barcode.

UI Example

1. Set up the camera preview and detection session to recognise barcodes

2. Create a result view with two states: search and result

3. If the candidate.barcodeCandidates is empty, show the search state. Else, fetch the PassioFoodItem, and use it's name and icon to show the result

Example of an image that produces multiple BarcodeCandidates:

The Nutrition Advisor is an intelligent chatbot designed to offer personalised guidance on nutrition, diets, and healthy eating habits. Whether you're looking to explore a new diet, understand the nutritional value of certain foods, or find delicious and balanced recipes, the Nutrition Advisor is here to assist. Users can interact with the Nutrition Advisor through text messages, engaging in a conversational experience where they can ask questions about various diets like keto, vegan, Mediterranean, and more. The chatbot provides detailed insights into the benefits, potential drawbacks, and suitability of these diets based on individual goals and preferences.

Conversation flow

Use these steps to initialise the conversation and enable users to send messages:

1. To start a conversation initConversation has to be invoked. The response will either indicate a success state, which means message can be sent to the advisor. In case of an error, an error message will detail the cause.

2. Text messages are sent using the sendMessage API. The reply (if successful) will contain a PassioAdvisorResponse response. This object consist of:

• threadId - identifies the conversation

• messageId - identifies the specific message

• markupContent - text response from the advisor denoted with markup tags. This can be used to style the text when rendering

• rawConent - text response without the markup tags

• tools - list of actions that can be executed by the advisor. Only should be used by the SDK

• extracedIngredients - will contain a list of foods if the request to the advisor was an image recognition request or extract ingredients from text message request

1. Image are sent to recognition using sendImage. As mentioned above, in the PassioAdvisorResponse object, the extractedIngredients property will hold the recognised foods.

NutritionAdvisor.shared.initConversation { advisorResult in
    switch advisorResult {
        case .success:
            print("The advisor can start acceping messages")
        case .failure(let error):
            print("The advisor can not start acceping messages. Error:- \(error.errorMessage)")
    }
}
let message = "Hi! I would like a recipe for a healthy burger"
NutritionAdvisor.shared.sendMessage(message: message) { advisorResponse in
    switch advisorResponse {
        case .success(let response): // Show text in a chat bubble
            print("Response:- \(response.rawContent)")
        case .failure(let error): // Handle error
            print("Chat Error:- \(error.errorMessage)")
    }
}
let image = Bundle.main.path(forResource: "image1", ofType: "png")
NutritionAdvisor.shared.sendImage(image: image) { advisorResponse in
    switch advisorResponse {
        case .success(let response): // Show list of foods
            print("Image Response:- \(response.extractedIngredients)")
        case .failure(let error): // Handle error
            print("Image Chat Error:- \(error.errorMessage)")
    }
}

NutritionAdvisor.instance.initConversation { result ->
    when (result) {
        is PassioResult.Success -> /* the advisor can start acceping messages */
        is PassioResult.Error -> /* handle conversation init error */
    }
}

NutritionAdvisor.instance.sendMessage("Hi! I would like a recipe for a healthy burger") { result ->
    when (result) {
        is PassioResult.Error -> {
            val errorMessage = result.message
            // Handle error
        }
        is PassioResult.Success -> {
            val text = result.value.rawContent
            // Show text in a chat bubble
        }
    }
}

val bitmap = loadBitmap()
NutritionAdvisor.instance.sendImage(bitmap) { result ->
    when (result) {
        is PassioResult.Error -> {
            val errorMessage = result.message
            // Handle error
        }
        is PassioResult.Success -> {
            val foods = result.value.extractedIngredients
            // Show list of foods
        }
    }
}

import { NutritionAdvisor } from '@passiolife/nutritionai-react-native-sdk-v3'

const [loadingState, setLoadingState] = useState<SDKStatus>('Init')
  useEffect(() => {
    async function configure() {
      try {
            NutritionAdvisor.initConversation()
              .then((item) => {
                setLoadingState(
                  item?.status === 'Success' ? 'Success' : 'Error'
                )
              })
              .catch((error) => {
                setLoadingState(error)
              })
      } catch (err) {
        console.error(`PassioSDK Error ${err}`)
        setLoadingState('Error')
      }
    }
    configure()
  }, [])


 const response = await NutritionAdvisor.sendMessage(message)
 switch (response?.status) {
          case 'Success':
            // get your response here {response.response}
            return
          case 'Error':
          // get your erorr response here {response.message}
            return
        }

const response = await NutritionAdvisor.sendImage(imageURI)
 switch (response?.status) {
          case 'Success':
            // get your response here {response.response}
            return
          case 'Error':
          // get your erorr response here {response.message}
            return
        }

final result = await NutritionAdvisor.instance.initConversation();
switch (result) {
  case Success():
    // Initialization succeeded, the advisor can start accepting messages
    break;
  case Error():
    // Handle the init error here
    break;
}

final result = await NutritionAdvisor.instance.sendMessage('Hi! I would like a recipe for a healthy burger');
switch (result) {
  case Success():
    // Parse the response from result.value
    break;
  case Error():
    // Handle the error that occurred while sending the message
    break;
}

Uint8List bytes = loadImage();
final result = await NutritionAdvisor.instance.sendImage(bytes);
switch (result) {
  case Success():
    // The food items are found in imageResult.value.extractedIngredients
    break;
  case Error():
    // Handle the error that occurred while sending the message
    break;
}

Extract ingredients

In the case that a PassioAdvisorResponse contains a tool called searchMatchIngredient, the Advisor has the ability to parse the text response and return the list of foods. If a message that has this tool is passed to the fetchIngredients API, the returned response will contain foods in the extracedIngredients field.

This feature is particularly useful when trying to log foods when directly communicating with the chatbot. A common use case is when user ask for a specific recipe. They can then initiate the extraction of foods from the message, and see their nutrition value and finally log the entire meal in one action.

UI Example

1. Create a screen where the user can type messages and send images to the NutritionAdvisor. Upon initiating the screen, set up the conversation by calling initConversation.

2. When the user types a message in the input text field and taps "Send", invoke the sendMessage function. Because this is an asynchronous function, add an animation to indicate this behaviour.

3. If the response from the advisor contains a tool called searchMatchIngredient show an action button "Find foods" to initiate extraction of the foods. Invoke the fetchIngredients function using the response from the Advisor.

4. Create a view where the user can see the extracted foods, their serving size and have the ability to log them.

1. Add the ability for the user to select an image from the library of the phone. Once the image is obtained, invoke the sendImage to get the list of foods. If foods are recognised, the response will contain a list of foods, use the same result view to show them.

There are cases when the barcode scanning item isn't present in the nutritional database. In the cases when the fetchFoodItemForProductCode doesn't yield a response, the user can scan the nutrition facts label on the packaging of the food and extract the data to create a new food item.

Real-time Nutrition Facts scanning

Similar to the real-time barcode scanning, the camera preview must be set up to enable continuous nutrition facts scanning. To start the Nutrition Facts scanning process a NutritionFactsRecognitionListener has to be defined. When the app is done with the nutrition fact scanning process, it should clear out the listener to avoid any unwanted UI updates.

private val listener = object : NutritionFactsRecognitionListener {
    override fun onRecognitionResult(nutritionFacts: PassioNutritionFacts?, text: String) {
        // Display the result
    }
}

override fun onResume() {
    super.onResume()
    PassioSDK.instance.startNutritionFactsDetection(listener)
}

override fun onPause() {
    PassioSDK.instance.stopNutritionFactsDetection()
    super.onPause()
}

The PassioNutritionFacts contains all of the data that can be scanned from a nutrition facts label including:

• Serving size (quantity and unit)

• Weight

• Calories

• Carbs

• Protein

• Fat

• Sugars

• Other micronutrients like fiber, cholesterol...

Nutrition Facts scanning from an image

To recognise nutrition facts in an image, refer to the recognizeImageRemote function. The result of the nutrition facts scanning process will come in the packagedFoodItem attribute, and the resultType will be Nutrition Facts. The nutrients will be stored in the PassioNutrients object of the first ingredient, and the serving size info will be stored in the PassioFoodAmount.

UI Example

1. The barcode scanning process is always the most precise way to get nutritional information about a packaged food. If the barcode is not present in the nutritional database, give the user the option to scan the nutrition facts label.

1. If the user proceeds with the Nutrition Facts scan, register a listener to start receiving the results from the SDK. When a PassioNutritionFacts is returned from the callback, display the data so that the user can verify it's accuracy.

1. Enable to user to manually enter all of the other data that the Nutrition Facts scanner did not recognise.

1. Once a user saves this food item, be sure to include it in the search or barcode scanning results.

Search

The Passio Nutrition AI SDK search functionality gives access to over 2.2 million food records in our nutritional database. The search api takes in a text query and returns a list of PassioFoodDataInfo results as well as a list of suggested searches

public func searchForFood(byText: String, completion: @escaping (SearchResponse?) -> Void)

public struct PassioFoodDataInfo: Codable {
    public let brandName: String
    public let foodName: String
    public let iconID: PassioID
    public let labelId: String
    public let resultId: String
    public let score: Double
    public let scoredName: String
    public let type: String
    public let nutritionPreview: PassioSearchNutritionPreview?
    public let isShortName: Bool
}

public struct PassioSearchNutritionPreview: Codable {
    public var calories: Int
    public let carbs: Double
    public let fat: Double
    public let protein: Double
    public var servingUnit: String
    public var servingQuantity: Double
    public var weightUnit: String
    public var weightQuantity: Double
}

fun searchForFood(
    term: String,
    callback: (result: List<PassioFoodDataInfo>, searchOptions: List<String>) -> Unit
)

data class PassioFoodDataInfo(
    val foodName: String,
    val brandName: String,
    val iconID: PassioID,
    val score: Double,
    val scoredName: String,
    val labelId: String,
    val type: String,
    val resultId: String,
    val isShortName: Boolean,
    val nutritionPreview: PassioSearchNutritionPreview
)

data class PassioSearchNutritionPreview(
    val calories: Int,
    val carbs: Double,
    val protein: Double,
    val fat: Double,
    val servingUnit: String,
    val servingQuantity: Double,
    val weightUnit: String,
    val weightQuantity: Double
)

searchForFood(searchQuery: string): Promise<PassioSearchResult | null>

export interface PassioSearchResult {
  results: PassioFoodDataInfo[] | null
  alternatives?: string[] | null
}

export interface PassioFoodDataInfo {
  brandName?: string
  foodName: string
  iconID: PassioID | string
  labelId: string
  nutritionPreview?: PassioSearchNutritionPreview
  resultId: string
  score?: number
  scoredName?: string
  type?: string
  isShortName?: boolean
}

export interface PassioNutritionPreview {
  calories: number
  servingQuantity?: number
  servingUnit?: string
  weightQuantity: number
  weightUnit: string
  fat: number
  protein: number
  carbs: number
}

Future<PassioSearchResponse> searchForFood(String byText) {
    return NutritionAIPlatform.instance.searchForFood(byText);
}

class PassioSearchResponse {
  final List<PassioFoodDataInfo> results;
  final List<String> alternateNames;
}

class PassioFoodDataInfo {
  final String brandName;
  final String foodName;
  final PassioID iconID;
  final String labelId;
  final PassioSearchNutritionPreview nutritionPreview;
  final String resultId;
  final double score;
  final String scoredName;
  final String type;
  final bool useShortName;
}

class PassioSearchNutritionPreview {
  final int calories;
  final double carbs;
  final double fat;
  final double protein;
  final double servingQuantity;
  final String servingUnit;
  final double weightQuantity;
  final String weightUnit;
}

The PassioFoodDataInfo is used a reference object to fetch the full nutritional data. It has all of the attributes needed for displaying on a typical search screen: the name of the food, name of the brand (if possible), iconId, and a nutritional preview detailing the macro nutrients for a specified serving size and weight.

The search will return 50 items that best match the input term, as well as a list of suggested searches that are semantically or contextually close to the input term.

Example: searching for term "coffee"

• search results: "Coffee, Brewed", "Coffee, Breakfast Blend", Coffee - Brewed From Grounds", "Coffee, Brewed, Decaffeinated", "Coffee, Instant, Reconstituted Decaffeinated", ...

• suggested searches: "coffee with whole milk", "coffee with creamer", "iced coffee"

UI Example

1. Create an input text field to collect the user query

2. Create a horizontal list scroll to display the suggestions

3. Create a vertical list to display the results of the search

4. If the user clicks on a suggestion, pass that string to the search API and reload the UI with the new results

Food Icons

The SDK gives the ability to fetch food icons for a specific passioId. There are two functions that can be used: lookupIconsFor and fetchIconsFor.

1. lookupIconsFor does a local search of the icons without doing any networking calls. It returns two icons. One is a placeholder icon depending on the type of food associated with the passioId. The second one is cached icon from a previous fetchIconsFor call, it it exists.

2. fetchIconsFor does a networking call to fetch an icon associated with the passioId, but it will check the local cache and return a previously fetched icon.

Example of how to use both of these functions to first display the placeholder icon, and then fetch the remote icon:

public extension UIImageView {
    // for tableView Cell.

    func loadPassioIconBy(passioID: PassioID,
                          entityType: PassioIDEntityType,
                          size: IconSize = .px90,
                          completion: @escaping (PassioID, UIImage) -> Void) {
        let (placeHolderIcon, icon) = PassioNutritionAI.shared.lookupIconsFor(passioID: passioID,
                                                                     size: size,
                                                                     entityType: entityType)
        if let icon = icon {
            self.image = icon
        } else {
            self.image = placeHolderIcon
            PassioNutritionAI.shared.fetchIconFor(passioID: passioID) { image in
                if let image = image {
                    completion(passioID, image)
                }
            }
        }
    }
}

// call with your UIImageView
imageFoodIcon.loadPassioIconBy(passioID: passioID, entityType: .item) { id, image in
     DispatchQueue.main.async {
         self.imageFoodIcon.image = image
     }
}

fun ImageView.loadPassioIcon(
    passioID: PassioID,
    type: PassioIDEntityType = PassioIDEntityType.item,
    iconSize: IconSize = IconSize.PX90
) {
    this.tag = passioID
    val localImageResult = PassioSDK.instance.lookupIconsFor(context, passioID, iconSize, type)

    if (localImageResult.second != null) {
        setImageDrawable(localImageResult.second)
        return
    }

    setImageDrawable(localImageResult.first)

    PassioSDK.instance.fetchIconFor(context, passioID, iconSize) { drawable ->
        if (drawable != null && this.tag == passioID) {
            setImageDrawable(drawable)
        }
    }
}

import {
  PassioIconView,
  type IconSize
} from '@passiolife/nutritionai-react-native-sdk-v3'

         <View
            style={{
              height: 24,
              width: 24,
              overflow: 'hidden',
              borderRadius: 12,
            }}
          >
            <PassioIconView
              style={{
                height: 24,
                width: 24,
              }}
              config={{
                passioID: item.passioID,
                iconSize: IconSize.PX180,
              }}
            />
       </View>

class PassioImageWidget extends StatefulWidget {
  const PassioImageWidget({
    required this.iconId,
    this.type = PassioIDEntityType.item,
    this.iconSize = IconSize.px90,
    this.radius = 30,
    this.heroTag,
    super.key,
  });

  final String iconId;
  final PassioIDEntityType type;
  final IconSize iconSize;
  final double radius;
  final Object? heroTag;

  @override
  State<PassioImageWidget> createState() => _PassioImageWidgetState();
}

class _PassioImageWidgetState extends State<PassioImageWidget> {
  final ValueNotifier<PlatformImage?> _image = ValueNotifier(null);

  @override
  void initState() {
    _fetchImage();
    super.initState();
  }

  @override
  void dispose() {
    _image.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return ValueListenableBuilder<PlatformImage?>(
      valueListenable: _image,
      builder: (context, value, child) => value != null
          ? Hero(
              tag: widget.heroTag ?? UniqueKey(),
              child: CircleAvatar(
                radius: widget.radius,
                backgroundImage: MemoryImage(value.pixels),
              ),
            )
          : const CircularProgressIndicator(),
    );
  }

  Future<void> _fetchImage() async {
    try {
      final result = await NutritionAI.instance.lookupIconsFor(
        widget.iconId,
        iconSize: widget.iconSize,
        type: widget.type,
      );

      setImage(result.cachedIcon ?? result.defaultIcon);
      if (result.cachedIcon != null) {
        return;
      }

      if (result.cachedIcon == null && widget.iconId.isNotEmpty) {
        final fetchedImage = await NutritionAI.instance.fetchIconFor(
          widget.iconId,
          iconSize: widget.iconSize,
        );
        if (fetchedImage != null) {
          setImage(fetchedImage);
        }
      }
    } catch (error) {
      // Handle potential errors during image fetching
      log("Error fetching image: $error");
    }
  }

  void setImage(PlatformImage? image) {
    if (mounted) {
      _image.value = image;
    }
  }
}

RefCode

There are several functions that are used to retrieve the PassioFoodItem object, depending on the origin of the data. If the food item is being scanned using an image, the food item is fetched using either fetchFoodItemForPassioID or fetchFoodItemForProductCode. If search is being used to access to food item, the appropriate call is the fetchFoodItemForDataInfo.

But if there is a use case that requires to save just one attribute to a database, that will later on be used to retrieve the full PassioFoodItem object, then the refCode attribute is the answer. The refCode in combination with the fetchFoodItemForRefCode will retrieve the original food item object from Passio's nutritional database.

PassioNutritionAI.shared.fetchFoodItemFor(passioID: "VEG0025") { (foodItem) in
        if let refCode = foodItem?.refCode {
            PassioNutritionAI.shared.fetchFoodItemFor(refCode: refCode) { refFoodItem in
            // foodItem == refFoodItem
        }
    }
}

PassioSDK.instance.fetchFoodItemForPassioID("VEG0025") { foodItem ->
    // foodItem is the original object from the database
    val refCode = foodItem!!.refCode
    // refCode is stored and used in another session
    PassioSDK.instance.fetchFoodItemForRefCode(refCode) { refFoodItem ->
        // foodItem == refFoodItem
    }
}

const foodItem = await PassioSDK.fetchFoodItemForPassioID('VEG0025')
// foodItem is the original object from the database
const refCode = foodItem?.refCode
// refCode is stored and used in another session
const refCodeFoodItem = await PassioSDK.fetchFoodItemForRefCode(refCode)

final foodItem = await NutritionAI.instance.fetchFoodItemForPassioID('VEG0025');
// foodItem is the original object from the database
final refCode = foodItem!.refCode;
// refCode is stored and used in another session
final refFoodItem = await NutritionAI.instance.fetchFoodItemForRefCode(refCode);
// foodItem == refFoodItem

Using v2 PassioID in the v3 SDK

If the implementing app is transitioning from generation 2 to generation 3 of the SDK, and has already stored generation 2 PassioIDs, invoking the fetchFoodItemLegacy will retrieve the PassioFoodItem that was previously structured as PassioIDAttributes.

Follow these steps to include the SDK into your mobile application project.

`<key>>NSCameraUsageDescription</key><string>For real-time food recognition</string>`.

implementation files('libs/passiolib-release.aar')

dependencies {
    // TensorFlow
    implementation 'org.tensorflow:tensorflow-lite:2.8.0'
    implementation 'org.tensorflow:tensorflow-lite-metadata:0.4.0'

    // CameraX
    def camerax_version = "1.3.0-alpha12"
    implementation "androidx.camera:camera-core:$camerax_version"
    implementation "androidx.camera:camera-camera2:$camerax_version"
    implementation "androidx.camera:camera-lifecycle:$camerax_version"
    implementation 'androidx.camera:camera-view:1.3.0-alpha02'
    implementation 'androidx.camera:camera-extensions:1.3.0-alpha02'

    // Barcode and OCR
    implementation 'com.google.android.gms:play-services-mlkit-text-recognition:19.0.0'
    implementation 'com.google.android.gms:play-services-mlkit-barcode-scanning:18.3.0'
}

android {
    compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }
}

//npm.pkg.github.com/:_authToken=GITHUB_ACCESS_TOKEN
@passiolife:registry=https://npm.pkg.github.com

npm install @passiolife/nutritionai-react-native-sdk-v3

yarn add @passiolife/nutritionai-react-native-sdk-v3

cd ios; pod install

platform :ios, '13.0'

dependencies {
    // Add this line below for Passio SDK library
    implementation files("$rootDir/../node_modules/@passiolife/nutritionai-react-native-sdk-v3/android/libs/passiolib-release.aar")
    ...
}

buildscript {
    ext {
        ...
        minSdkVersion = 26
        ...
    }
}

 $ flutter pub add nutrition_ai

dependencies:
  nutrition_ai: ^3.0.3

import 'package:nutrition_ai/nutrition_ai.dart';

allprojects {
   repositories {
      ...
      flatDir {
         dirs project(':nutrition_ai').file('libs')
      }
   }
}

allprojects {
   repositories {
      ...
      flatDir {
         implementation (name: 'passiolib-release', ext: 'aar')
      }
   }
}

The Passio SDK has the ability the recognise anything from simple ingredients like blueberries and almonds to complex cooked dishes like a beef burrito with salad and fries.

There are two types of food recognition, each of them with their own strengths and weaknesses.

The Remote image recognition approach is good when the accuracy of the results is top priority, and waiting for the response is not an issue. This use case is implemented by taking static images from the camera or the library of the device and sending them for recognition in an asynchronous fashion.

The Local model approach is good when speed is of the essence. This use case is implemented using continuous frame recognition from the camera. A callback is registered to capture the results as they are coming from the camera stream.

Remote image recognition

This API sends an image as base64 format to an LLM on Passio's backend, and returns a list of recognised items.

The default behaviour of this function is to resize the image to 512 pixels (longer dimension is resized, the other is calculated to keep aspect ratio). Using the PassioImageResolution enum, the image can be either resized to 512, 1080 or keep the original resolution.

UI Example

1. Create a screen where the user can snap one or multiple images using the camera of the device

2. Upon clicking next, the recognizeImageRemote is invoked on each of the images in the list

3. Wait for all of the responses to come, add each results list to a final list of results. When the last asynchronous function is executed, present the final list to the user.

Local neural network model

To set up the local model and the continuous scanning mode, the camera preview and the recognition session need to be defined.

Camera preview

Add the UI element that is responsible for rendering the camera frames:

Start food detection

The type of food detection is defined by the FoodDetectionConfiguration object. To start the Food Recognition process a FoodRecognitionListener also has to be defined. The listener serves as a callback for all the different food detection processes defined by the FoodDetectionConfiguration. When the app is done with food detection, it should clear out the listener to avoid any unwanted UI updates.

The FoodCandidates object that is returned in the recognition callbacks contains three lists:

• detectedCandidates detailing the result of VISUAL detection

• barcodeCandidates detailing the result of BARCODE detection

• packagedFoodCandidates detailing the result of PACKAGED detection

Only the corresponding candidate lists will be populated (e.g. if you define detection types VISUAL and BARCODE, you will never receive a packagedFoodCandidates list in this callback).

Visual detection

A DetectedCandidate represents the result from running Passio's neural network, specialized for detecting foods like apples, salads, burgers etc. The properties of a detected candidate are:

• name

• passioID (unique identifier used to query the nutritional databse)

• confidence (measure of how accurate is the candidate, ranges from 0 to 1)

• boundingBox (a rectangle detailing the bounds of the recognised item within the image dimensions)

• alternatives (list of alternative foods that are visually or contextually similar to the recognised food)

• croppedImage (the image that the recognition was ran on)

UI Example

1. Implement the camera screen using the steps above

2. Create a result view that can have two states: scanning and result

3. If the callback returns an empty list, show the scanning state. If it returns the result, display the name from the detectedCandidate.name

Example of an image that produces a DetectedCandidate:

Suggestions

The SDK provides the ability to fetch a predefined set of suggested foods for logging, depending on the current time of day

Example of fetching suggestions for snack meal time:

banana, coffee, black tea, coke, cheddar cheese, chocolate chip cookies, potato chips, blueberry muffin, plain yogurt, strawberries, ...

Meal Plans

Meal Plans provide users suggestions which foods to consume depending on their dietary preference and time of day. The Meal Plan api is divided into two functions:

• fetchMealPlans that is used to retrieve all of the possible meal plans. These meal plans usually correlate with a specific diet like "Keto Diet".

• fetchMealPlanForDay is an api that is used to fetch specific foods recommended for a certain day of a target meal plan.

Some of the available meal plans:

Heart Healthy Diet, Ketogenic Diet, Managing Obesity, Managing Type 2 Diabetes, Low FODMAP, Healthy Kidney Diet, Balanced Diet, DASH Diet, PCOS Diet, Mediterranean Diet