Passio Nutrition-AI
  • Nutrition-AI SDK Overview
  • Guides
    • Nutrition AI SDK
      • SDK Key and minimum requirements
      • Installation
      • Configure the SDK
      • Use Cases
        • Food recognition
        • Nutrition data
        • Barcode scanning
        • Nutrition Facts scanning
        • Search, Food Icons, RefCode
        • Speech recognition
        • Nutrition Advisor
        • Suggestions and Meal Plans
        • User created foods and reports
    • iOS SDK Docs
      • Before getting started
      • Getting the ml models to the device
      • Run the demos first
      • Adding Passio SDK into your project
      • Initialize and configure the SDK
      • Start/Stop food detection
      • Food Recognition Delegate
      • Migration from SDK 1.4.X to 2.x
      • SDK API
      • Quick Start Guide
        • Installation
        • Configure the SDK (UIKit)
        • Recognise food using image
        • Food Details
    • Android SDK Docs
      • Getting started
        • Include the library
        • Camera
        • SDK Initialization and Configuration
        • Food detection session
        • Visual, Barcode and Packaged Food detection
        • Nutritional Database
        • Sandbox app
      • Troubleshooting on Android
      • Migration from SDK version 1.4.x to 2.x
      • Quick Start Guide
        • Importing the Android SDK to a project
        • Configure SDK and handle the result
        • RecognizeImageRemote
        • Food Details
    • React Native SDK Docs
      • Getting Started
        • Installation
        • SDK Initialization and Configuration
      • API Reference
        • configure & requestCameraAuthorization
        • recognizeImageRemote
        • searchForFood & searchForFoodSemantic
        • startFoodDetection
          • FoodDetectionEvent
        • fetchFoodItemForProductCode
        • fetchFoodItemForRefCode
        • fetchFoodItemForDataInfo
        • startNutritionFactsDetection
        • fetchFoodItemForPassioID
        • recognizeSpeechRemote
        • Fetch Nutrients
        • onDowloadingPassioModelCallBacks
        • detectFoodFromImageURI
        • addToPersonalization
        • updateLanguage
        • fetchHiddenIngredients
        • fetchVisualAlternatives
        • fetchPossibleIngredients
      • Properties
        • PassioFoodItem
        • PassioFoodDataInfo
        • PassioFoodAmount
        • PassioIngredient
        • PassioNutrients
        • UnitMass
        • ServingUnit
        • ServingSize
        • PassioSearchResult
        • NutritionFacts
        • PassioNutritionPreview
        • PassioSpeechRecognitionModel
        • PassioLogAction
        • PassioAdvisorFoodInfo
        • FoodCandidates
        • FoodDetectionEvent
        • NutritionDetectionEvent
        • DetectedCandidate
        • BarcodeCandidate
        • PackagedFoodCode
        • AmountEstimate
        • ImagesInfo
        • PassioStatus
        • FoodDetectionEvent
        • NutritionFacts
        • PassioMealPlan
        • PassioMealPlanItem
      • Nutriton Advisor
        • initConversation
        • sendMessage
        • sendImage
        • Guide
          • useNutritionAdvisor
        • Properties
          • PassioAdvisorResponse
          • PassioAdvisorMessageResultStatus
          • PassioAdvisorResultStatus
      • Quick Start Guide
        • Installation
        • Configure the SDK
        • RecognizeImageRemote
        • Food Detail
      • Guide
        • Integrate SDK Setup
        • Integrate Quick Scan
        • Integrate Food Search
        • Integrate Food Editor with Recipe
        • Integrate MealPlan
        • Integrate Suggestions
        • Integrate recognizeImageRemote
      • Components
        • DetectionCameraView
        • PassioIconView
      • Changelog
      • More
        • Display Macro Chart
          • MockFood
          • Utils PassioNutrient
        • Display Micro Progress
        • Getting nutrition value for a FoodItem
        • Getting nutrition value for a Recipe
        • Recalculate the nutrition value based on serving size options.
      • Migrations
        • Migrating from SDK 2.X to version 3.X
        • Structure Migrations From SDK 3.X to SDK 2.X
        • [Deprecated] Migrating from SDK 1.X to version 2.X
        • [Deprecated] Getting Started (v1)
          • Installation
          • SDK Initialization and Configuration
          • Start food detection
          • FoodDetectionEvent
          • Nutritional Database
      • Troubleshooting on RN
      • V2
        • Getting Started
          • Installation
          • SDK Initialization and Configuration
          • Start food detection
          • FoodDetectionEvent
        • RN SDK API
          • Properties
            • PersonalizedAlternative
            • FoodSearchResult
            • PassioNutrient
            • FoodDetectionEvent
            • DownloadModelCallBack
            • UPCProduct
            • ServingUnit
            • PassioStatus
            • PassioIDAttributes
            • PassioIDEntityType
            • PassioFoodItem
            • PassioRecipe
            • ServingSize
            • Measurement
            • UnitMass
            • NutritionFacts
        • Food Image
        • Search Food Item
        • Quick Scan
        • Multi Scan
        • MealPlan
        • Recipe
      • How do I Integrate a Passio SDK in EXPO?
    • Flutter SDK Docs
      • Getting Started
    • Before You Continue
    • Setup For Android
    • Initialize and configure the SDK
  • Fundamentals
    • Nutrition-AI Developer FAQ
      • Nutrition Data
      • Supported Phones
      • Security
      • Testing Volume Estimation
      • Testing Nutrition-AI SDK
      • Nutrition-AI Test Methodology
    • Nutrition API - Mobile SDK interoperability
      • JSON Response parsing
  • Versions
    • 3.2.4
      • SDK Key and minimum requirements
      • Installation
      • Configure the SDK
      • Use Cases
        • Food recognition
        • Nutrition data
        • Barcode scanning
        • Nutrition Facts scanning
        • Search, Food Icons, RefCode
        • Speech recognition
        • Nutrition Advisor
        • Suggestions and Meal Plans
        • User created foods and reports
    • 3.2.2
      • SDK Key and minimum requirements
      • Installation
      • Configure the SDK
      • Use Cases
        • Food recognition
        • Nutrition data
        • Barcode scanning
        • Nutrition Facts scanning
        • Search, Food Icons, RefCode
        • Speech recognition
        • Nutrition Advisor
        • Suggestions and Meal Plans
        • User created foods and reports
    • 3.2.0
      • SDK Key and minimum requirements
      • Installation
      • Configure the SDK
      • Use Cases
        • Food recognition
        • Nutrition data
        • Barcode scanning
        • Nutrition Facts scanning
        • Search, Food Icons, RefCode
        • Speech recognition
        • Nutrition Advisor
        • Suggestions and Meal Plans
    • 3.1.4
      • SDK Key and minimum requirements
      • Installation
      • Configure the SDK
      • Use Cases
        • Food recognition
        • Nutrition data
        • Barcode scanning
        • Search, Food Icons, RefCode
        • Speech recognition
        • Nutrition Advisor
        • Suggestions and Meal Plans
Powered by GitBook
On this page
  • Remote image recognition
  • UI Example
  • Local neural network model
  • Camera preview
  • Start food detection
  • Visual detection
  • UI Example
Export as PDF
  1. Versions
  2. 3.2.4
  3. Use Cases

Food recognition

The Passio SDK has the ability the recognise anything from simple ingredients like blueberries and almonds to complex cooked dishes like a beef burrito with salad and fries.

There are two types of food recognition, each of them with their own strengths and weaknesses.

Type
Works offline
Precision
Response time

No, the image is sent to the backend for recognition

Very precise with all types of foods

On average 4-7 seconds

Yes, the recognition is done on the device

Good with single food items, struggles with complex cooked foods

Depending on the hardware of the device, ranges between 50-300ms

The Remote image recognition approach is good when the accuracy of the results is top priority, and waiting for the response is not an issue. This use case is implemented by taking static images from the camera or the library of the device and sending them for recognition in an asynchronous fashion.

The Local model approach is good when speed is of the essence. This use case is implemented using continuous frame recognition from the camera. A callback is registered to capture the results as they are coming from the camera stream.

Remote image recognition

This API sends an image as base64 format to an LLM on Passio's backend, and returns a list of recognised items.

The API can recognise foods raw or prepared, barcodes or nutrition facts tables. The type of recognition will be shown in the resultType enum.

The default behaviour of this function is to resize the image to 512 pixels (longer dimension is resized, the other is calculated to keep aspect ratio). Using the PassioImageResolution enum, the image can be either resized to 512, 1080 or keep the original resolution.

let image = Bundle.main.path(forResource: "image1", ofType: "png")
PassioNutritionAI.shared.recognizeImageRemote(image: image) { passioAdvisorFoodInfo in
    print("Food Info:- \(passioAdvisorFoodInfo)")
}
public struct PassioAdvisorFoodInfo: Codable {
    public let recognisedName: String
    public let portionSize: String
    public let weightGrams: Double
    public let foodDataInfo: PassioFoodDataInfo
}
val bitmap = loadBitmapFromAssets(assets, "image1.png")
PassioSDK.instance.recognizeImageRemote(bitmap) { result ->
    // display the list
}

data class PassioAdvisorFoodInfo(
    val recognisedName: String,
    val portionSize: String,
    val weightGrams: Double,
    val foodDataInfo: PassioFoodDataInfo? = null,
    val packagedFoodItem: PassioFoodItem? = null,
    val resultType: PassioFoodResultType,
)

enum class PassioFoodResultType {
    FOOD_ITEM,
    BARCODE,
    NUTRITION_FACTS
}
import {
  PassioSDK,
  type PassioAdvisorFoodInfo,
  type PassioFoodDataInfo,
} from '@passiolife/nutritionai-react-native-sdk-v3'

import { launchImageLibrary } from 'react-native-image-picker'

const onScanImage = useCallback(async () => {
    try {
      const { assets } = await launchImageLibrary({ mediaType: 'photo' })
      if (assets) {
        setLoading(true)
        setPassioSpeechRecognitionModel(null)
        PassioSDK.recognizeImageRemote(
          assets?.[0].uri?.replace('file://', '') ?? ''
        )
          .then(async (candidates) => {
            setPassioSpeechRecognitionModel(candidates)
          })
          .catch(() => {
            Alert.alert('Unable to recognized this image')
          })
          .finally(() => {
            setLoading(false)
          })
      }
    } catch (err) {
      setLoading(false)
    }
  }, [])
// Create a File object from the image path
var file = File(imagePath);
// Read the file's bytes asynchronously
var bytes = await file.readAsBytes();
// Send the byte array to the NutritionAI for remote image recognition
final list = await NutritionAI.instance.recognizeImageRemote(bytes);

class PassioAdvisorFoodInfo {
  final PassioFoodDataInfo? foodDataInfo;
  final PassioFoodItem? packagedFoodItem;
  final String portionSize;
  final PassioFoodResultType resultType;
  final String recognisedName;
  final double weightGrams;
}

UI Example

  1. Create a screen where the user can snap one or multiple images using the camera of the device

  2. Upon clicking next, the recognizeImageRemote is invoked on each of the images in the list

  3. Wait for all of the responses to come, add each results list to a final list of results. When the last asynchronous function is executed, present the final list to the user.

Local neural network model

To set up the local model and the continuous scanning mode, the camera preview and the recognition session need to be defined.

Camera preview

To start using camera detection the app must first acquire the permission to open the camera from the user. This permission is not handled by the SDK.

Add the UI element that is responsible for rendering the camera frames:

var videoLayer: AVCaptureVideoPreviewLayer?

func setupPreviewLayer() {
    guard videoLayer == nil else { return }
    if let videoLayer = passioSDK.getPreviewLayer() {
        self.videoLayer = videoLayer
        videoLayer.frame = view.bounds
        view.layer.insertSublayer(videoLayer, at: 0)
    }
}

Start by adding the PreviewView to your view hierarchy. Go to your layout.xml and add the following.

<androidx.camera.view.PreviewView
    android:id="@+id/myPreviewView"
    android:layout_width="match_parent"
    android:layout_height="match_parent" />

Using the PassioCameraViewProvider

This approach is more manual but gives you more flexibility. You need to implement the PassioCameraViewProvider interface and supply the needed LifecycleOwner and the PreviewView added in the initial step.

class MainActivity : AppCompatActivity(), PassioCameraViewProvider {
	
    override fun requestCameraLifecycleOwner(): LifecycleOwner {
        return this
    }

    override fun requestPreviewView(): PreviewView {
        return myPreviewView
    }
}

After the user has granted permission to use the camera, start the SDK camera

override fun onStart() {
    super.onStart()
    if (!hasPermissions()) {
        ActivityCompat.requestPermissions(
            this,
            REQUIRED_PERMISSIONS,
            REQUEST_CODE_PERMISSIONS
        )
        return
    } else {
        PassioSDK.instance.startCamera(this /*reference to the PassioCameraViewProvider*/)
    }
}

Using the PassioCameraFragment

PassioCameraFragment is an abstract class that handles Camera permission at runtime as well as starting the Camera process of the SDK. To use the PassioCameraFragment simply extend it in your own fragment and supply the PreviewView that has been added to the view hierarchy in the previous step.

class MyFragment : PassioCameraFragment() {
	
    override fun getPreviewView(): PreviewView {
	return myPreviewView
    }

    override fun onCameraReady() { 
    	// Proceed with initializing the recognition session
    }

    override fun onCameraPermissionDenied() { 
    	// Explain to the user that the camera is needed for this feature to
        // work and ask for permission again
    }
}
import {
  PassioSDK,
  DetectionCameraView,
} from '@passiolife/nutritionai-react-native-sdk-v2';

To show the live camera preview, add the DetectionCameraView to your view

<DetectionCameraView style={{flex: 1, width: '100%'}} />
@override
Widget build(BuildContext context) {
  return Scaffold(
    body: Stack(
      children: [
        const PassioPreview(),
        ...
      ],
    ),
  );
}

Start food detection

The SDK can detect 3 different categories: VISUAL, BARCODE and PACKAGED. The VISUAL recognition is powered by Passio's neural network and is used to recognize over 4000 food classes. BARCODE, as the name suggests, can be used to scan a barcode located on a branded food. Finally, PACKAGED can detect the name of a branded food. To choose one or more types of detection, a FoodDetectionConfiguration object is defined and the corresponding fields are set. The VISUAL recognition works automatically.

The type of food detection is defined by the FoodDetectionConfiguration object. To start the Food Recognition process a FoodRecognitionListener also has to be defined. The listener serves as a callback for all the different food detection processes defined by the FoodDetectionConfiguration. When the app is done with food detection, it should clear out the listener to avoid any unwanted UI updates.

Implement the delegate FoodRecognitionDelegate:

extension PassioQuickStartViewController: FoodRecognitionDelegate {
  func recognitionResults(candidates: FoodCandidates?,
                          image: UIImage?) {
        if let candidates = candidates?.barcodeCandidates,
           let candidate = candidates.first {
            print("Found barcode: \(candidate.value)")
        }
        
        if let candidates = candidates?.packagedFoodCandidates,
           let candidate = candidates.first {
            print("Found packaged food: \(candidate.packagedFoodCode)")
        }
        
        if let candidates = candidates?.detectedCandidates,
           let candidate = candidates.first {
            print("Found detected food: \(candidate.name)")
        }
  }
}

Add the method startFoodDetection()

func startFoodDetection() {
    setupPreviewLayer()
                
    let config = FoodDetectionConfiguration(detectVisual: true,
                                            volumeDetectionMode: .none,
                                            detectBarcodes: true,
                                            detectPackagedFood: true)
    passioSDK.startFoodDetection(detectionConfig: config,
                                 foodRecognitionDelegate: self) { ready in
        if !ready {
            print("SDK was not configured correctly")
        }
    }
}

In viewWillAppear request authorisation to use the camera and start the recognition:

override func viewWillAppear(_ animated: Bool) {
    super.viewWillAppear(animated)
    if AVCaptureDevice.authorizationStatus(for: .video) == .authorized {
        startFoodDetection()
    } else {
        AVCaptureDevice.requestAccess(for: .video) { (granted) in
            if granted {
                DispatchQueue.main.async {
                    self.startFoodDetection()
                }
            } else {
                print("The user didn't grant access to use camera")
            }
        }
    }
}

Stop Food Detection in viewWillDisappear:

override func viewWillDisappear(_ animated: Bool) {
    super.viewWillDisappear(animated)
    passioSDK.stopFoodDetection()
    videoLayer?.removeFromSuperlayer()
    videoLayer = nil
}
private val foodRecognitionListener = object : FoodRecognitionListener {
    override fun onRecognitionResults(
        candidates: FoodCandidates,
        image: Bitmap?,
    ) {
        // Handle result
    }
}

Using the listener and the detection options start the food detection by calling the startFoodDetection method of the SDK.

override fun onStart() {
    super.onStart()
    val options = FoodDetectionConfiguration().apply {
        detectBarcodes = true
    }
    PassioSDK.instance.startFoodDetection(foodListener, options)
}

Stop the food recognition in the onStop() lifecycle callback.

override fun onStop() {
    PassioSDK.instance.stopFoodDetection()
    super.onStop()
}
const config: FoodDetectionConfig = {
   /**
   * Detect barcodes on packaged food products. Results will be returned
   * as `BarcodeCandidates` in the `FoodCandidates` property of `FoodDetectionEvent`
   */
  detectBarcodes: true,
 
  /**
   * Results will be returned as DetectedCandidate in the `FoodCandidates`and 
   * property of `FoodDetectionEvent`
   */

  detectPackagedFood: true,
};
    
useEffect(() => {
  if (!isReady) {
    return;
  }
  const subscription = PassioSDK.startFoodDetection(
    config,
    async (detection: FoodDetectionEvent) => {

      const { candidates, nutritionFacts } = detection

      if (candidates?.barcodeCandidates?.length) {
         
         // show barcode candidates to the user

      } else if (candidates?.packagedFoodCode?.length) {
        
        // show OCR candidates to the user

      } else if (candidates?.detectedCandidates?.length) {
        
        // show visually recognized candidates to the user

      }
    },
  );
  // stop food detection when component unmounts
  return () => subscription.remove(); 

}, [isReady]);

Add the method startFoodDetection() and register a FoodRecognitionListener

void _startFoodDetection() {
  var detectionConfig = FoodDetectionConfiguration();
  detectionConfig.detectBarcodes = true;
  NutritionAI.instance.startFoodDetection(detectionConfig, this);
}

@override
void recognitionResults(FoodCandidates? foodCandidates, PlatformImage? image) {
  // Handle result
}

Stop Food Detection on widget dispose:

@override
void dispose() {
  NutritionAI.instance.stopFoodDetection();
  super.dispose();
}

The FoodCandidates object that is returned in the recognition callbacks contains three lists:

  • detectedCandidates detailing the result of VISUAL detection

  • barcodeCandidates detailing the result of BARCODE detection

  • packagedFoodCandidates detailing the result of PACKAGED detection

Only the corresponding candidate lists will be populated (e.g. if you define detection types VISUAL and BARCODE, you will never receive a packagedFoodCandidates list in this callback).

Visual detection

A DetectedCandidate represents the result from running Passio's neural network, specialized for detecting foods like apples, salads, burgers etc. The properties of a detected candidate are:

  • name

  • passioID (unique identifier used to query the nutritional databse)

  • confidence (measure of how accurate is the candidate, ranges from 0 to 1)

  • boundingBox (a rectangle detailing the bounds of the recognised item within the image dimensions)

  • alternatives (list of alternative foods that are visually or contextually similar to the recognised food)

  • croppedImage (the image that the recognition was ran on)

public func fetchFoodItemFor(passioID: PassioNutritionAISDK.PassioID, completion: @escaping (PassioNutritionAISDK.PassioFoodItem?) -> Void)
fun fetchFoodItemForPassioID(
    passioID: PassioID,
    onResult: (foodItem: PassioFoodItem?) -> Unit
)
fetchFoodItemForPassioID(passioID: PassioID): Promise<PassioFoodItem | null>
Future<PassioFoodItem?> fetchFoodItemForPassioID(PassioID passioID)

UI Example

  1. Implement the camera screen using the steps above

  2. Create a result view that can have two states: scanning and result

  3. If the callback returns an empty list, show the scanning state. If it returns the result, display the name from the detectedCandidate.name

Example of an image that produces a DetectedCandidate:

PreviousUse CasesNextNutrition data

Last updated 7 months ago

The response, presented as a list ofPassioAdvisorFoodInfo objects, contains the name, portion and weight in grams recognised by the LLM. These attributes can be used for debugging, but the data from the nutritional database is contained either in the foodDataInfo if the result type is a Food Item, or packagedFoodItem if it's a Barcode or Nutrition Facts. To fetch the for the PassioFoodDataInfo object, use the fetchFoodItemForDataInfo function.

To start using the camera in your Activity/Fragment, implement the PassioCameraViewProvider interface. By implementing this interface the SDK will use that component as the lifecycle owner of the camera (when that component calls onPause() the camera will stop) and also will provide the Context in order for the camera to start. The component implementing the interface must have a in its view hierarchy.

To fetch the of a detected candidate use:

nutritional data
PreviewView
full nutrition data
Remote image recognition
Local neural network model