Food recognition

The Passio SDK has the ability the recognise anything from simple ingredients like blueberries and almonds to complex cooked dishes like a beef burrito with salad and fries.

Remote image recognition

This API sends an image as base64 format to an LLM on Passio's backend, and returns a list of recognised items.

The API can recognise foods raw or prepared, barcodes or nutrition facts tables. The type of recognition will be shown in the resultType enum.

The default behaviour of this function is to resize the image to 512 pixels (longer dimension is resized, the other is calculated to keep aspect ratio). Using the PassioImageResolution enum, the image can be either resized to 512, 1080 or keep the original resolution.

let image = Bundle.main.path(forResource: "image1", ofType: "png")
PassioNutritionAI.shared.recognizeImageRemote(image: image) { passioAdvisorFoodInfo in
    print("Food Info:- \(passioAdvisorFoodInfo)")
}
public struct PassioAdvisorFoodInfo: Codable {
    public let recognisedName: String
    public let portionSize: String
    public let weightGrams: Double
    public let foodDataInfo: PassioFoodDataInfo
}

val bitmap = loadBitmapFromAssets(assets, "image1.png")
PassioSDK.instance.recognizeImageRemote(bitmap) { result ->
    // display the list
}

data class PassioAdvisorFoodInfo(
    val recognisedName: String,
    val portionSize: String,
    val weightGrams: Double,
    val foodDataInfo: PassioFoodDataInfo? = null,
    val packagedFoodItem: PassioFoodItem? = null,
    val resultType: PassioFoodResultType,
)

enum class PassioFoodResultType {
    FOOD_ITEM,
    BARCODE,
    NUTRITION_FACTS
}

import {
  PassioSDK,
  type PassioAdvisorFoodInfo,
  type PassioFoodDataInfo,
} from '@passiolife/nutritionai-react-native-sdk-v3'

import { launchImageLibrary } from 'react-native-image-picker'

const onScanImage = useCallback(async () => {
    try {
      const { assets } = await launchImageLibrary({ mediaType: 'photo' })
      if (assets) {
        setLoading(true)
        setPassioSpeechRecognitionModel(null)
        PassioSDK.recognizeImageRemote(
          assets?.[0].uri?.replace('file://', '') ?? ''
        )
          .then(async (candidates) => {
            setPassioSpeechRecognitionModel(candidates)
          })
          .catch(() => {
            Alert.alert('Unable to recognized this image')
          })
          .finally(() => {
            setLoading(false)
          })
      }
    } catch (err) {
      setLoading(false)
    }
  }, [])

// Create a File object from the image path
var file = File(imagePath);
// Read the file's bytes asynchronously
var bytes = await file.readAsBytes();
// Send the byte array to the NutritionAI for remote image recognition
final list = await NutritionAI.instance.recognizeImageRemote(bytes);

class PassioAdvisorFoodInfo {
  final PassioFoodDataInfo? foodDataInfo;
  final PassioFoodItem? packagedFoodItem;
  final String portionSize;
  final PassioFoodResultType resultType;
  final String recognisedName;
  final double weightGrams;
}

The response, presented as a list ofPassioAdvisorFoodInfo objects, contains the name, portion and weight in grams recognised by the LLM. These attributes can be used for debugging, but the data from the nutritional database is contained either in the foodDataInfo if the result type is a Food Item, or packagedFoodItem if it's a Barcode or Nutrition Facts. To fetch the nutritional data for the PassioFoodDataInfo object, use the fetchFoodItemForDataInfo function.

UI Example

Create a screen where the user can snap one or multiple images using the camera of the device
Upon clicking next, the recognizeImageRemote is invoked on each of the images in the list
Wait for all of the responses to come, add each results list to a final list of results. When the last asynchronous function is executed, present the final list to the user.

Example of an image that produces a PassioAdvisorFoodInfo:

PreviousUse Cases NextNutrition data

Last updated 3 months ago

hashtagRemote image recognition

hashtagUI Example

Remote image recognition

UI Example