Two things are required for the SDK to work: a proper license key and the neural network files that the SDK uses to run the recognition process.

• The license key can be requsted at support@passiolife.com

• The files are shipped inside the PassioSDKSandbox project on the releases page of the Android-Passio-SDK-Distribution repository. Unzip the PassioSDKSandbox project and locate the files in the assets folder: /PassioSDKSandbox/app/src/main/assets. Files that the SDK needs have an extension .passiosecure and are encrypted

There are several ways to initialize the SDK depending on the location of the needed files:

1. Files are in the assets folder of your Android project

2. Let the SDK manage file download and update

3. Host the files on your own server, download the files, store them on the device's memory and point the SDK to the URIs of the needed files

1. Files are located in the Assets folder

• If the assets folder is not already present in your Android project, create one in the module where the SDK dependencies are located under the main directory (e.g. app/src/main/assets)

• Transfer the .passiosecure files to the assets folder

• Create a PassioConfiguration object with the application context and the license key

• Call the configure method of the PassioSDK without changing the default PassioConfiguration object

val passioConfiguration = PassioConfiguration(
    this.applicationContext,
    "Your license key here"
)

PassioSDK.instance.configure(passioConfiguration) { passioStatus ->
    // Handle PassioStatus states
}

2. The SDK will manage the download and updates

val passioConfiguration = PassioConfiguration(
    this.applicationContext,
    "Your license key here"
).apply {
    sdkDownloadsModels = true
}

PassioSDK.instance.configure(passioConfiguration) { passioStatus ->
    // Handle PassioStatus states
} 

To track the download process you can implement a PassioStatusListener (recommended to define after the PassioStatus mode returns IS_DOWNLOADING_MODELS)

3. Files are located in local device storage

This approach is recommended if you are responsible for the download of the files. In this case the SDK does not care where the files are hosted and how they are downloaded, just where they are stored after the download has completed. Use the PassioConfiguration object to modify the initialization process with local files and pass the URIs of the files to the localFiles field.

val passioConfiguration = PassioConfiguration(
    this.applicationContext,
    "Your license key here"
).apply {
    localFiles = /*List of files URIs*/
}

PassioSDK.instance.configure(passioConfiguration) { passioStatus ->
    // Handle PassioStatus states
} 

Handle the configuration response

When you call the configure method of the Passio SDK with the PassioConfiguration object, you will need to define a callback to handle the result of the configuration process. This result is comprised in the PassioStatus object.

class PassioStatus {
    var mode: PassioMode
    var missingFiles: List<FileName>?
    var debugMessage: String?
    var activeModels: Int?
}

The mode of PassioStatus defines what the current status of the configuration process is. There are 5 different modes, and they all should be handled by the implementing side.

enum class PassioMode {
    NOT_READY,
    FAILED_TO_CONFIGURE,
    IS_BEING_CONFIGURED,
    IS_DOWNLOADING_MODELS,
    IS_READY_FOR_DETECTION
}

• NOT_READY -> The configuration process hasn't started yet.

• FAILED_TO_CONFIGURE -> The configuration process resulted in an error. Be sure to check the debugMessage of the PassioStatus object to get more insight into the nature of the error.

• IS_BEING_CONFIGURED -> The SDK is still in the configuration process. Normally, you shouldn't receive this mode as a callback to the configure method.

• IS_DOWNLOADING_MODELS -> The files required for the SDK to work are not present and are currently being downloaded.

• IS_READY_FOR_DETECTION -> The configuration process is over and all the SDKs functionalities are available.

The missingFiles variable contains a list of all the files missing for the SDK to work optimally. When this list is not null, this doesn't mean that the SDK is not operational. The SDK may be initialized with older files, but it will advocate the download of the newest files through this list.

The debugMessage usually gives more verbose and human-readable information about the configuration process. It will always give an error message if the mode is FAILED_TO_CONFIGURE.

The activeModels will give the version of the files installed and ran by the SDK if the configuration process ran successfully i.e. if the mode is IS_READY_FOR_DETECTION.

Best practices

1. The PassioStatus mode is NOT_READY -> check debugMessage for the error, usually the wrong license key has been supplied or the files are missing.

2. The PassioStatus mode is IS_DOWNLOADING_MODELS -> register a PassioStatusListener through the PassioSDK.instance.setPassioStatusListener() method as it will enable tracking of the download progress.

3. The PassioStatus mode is IS_READY_FOR_DETECTION -> still check the missingFiles because the SDK might be running with an older version of the files and newer need to be downloaded.

If at any point you need help from the Passio team, please reach out to us at support@passiolife.com

val options = FoodDetectionConfiguration().apply {
    detectBarcodes = true
}

To start the Food Recognition process a FoodRecognitionListener also has to be defined. The listener serves as a callback for all the different food detection processes defined by the FoodDetectionConfiguration.

private val foodRecognitionListener = object : FoodRecognitionListener {
    override fun onRecognitionResults(
        candidates: FoodCandidates,
        image: Bitmap?,
        nutritionFacts: PassioNutritionFacts?
    ) {
        val detectedCandidates = candidates.detectedCandidates!!
        val barcodeCandidates = candidates.barcodeCandidates!!
        val passioID = detectedCandidates.first().passioID
        val foodName = PassioSDK.instance.lookupNameFor(passioID)
        Toast.makeText(requestContext(), foodName, Toast.LENGTH_SHORT).show()
    }
}

Using the listener and the detection options start the food detection by calling the startFoodDetection method of the SDK.

override fun onStart() {
    super.onStart()
    PassioSDK.instance.startFoodDetection(foodRecognitionListener)
}

Stop the food recognition in the onStop() lifecycle callback.

override fun onStop() {
    PassioSDK.instance.stopFoodDetection()
    super.onStop()
}

Try to run the code containing the foodListener defined above. Point the phone at the image below and see if you are getting the correct food printed on the screen (it should be red apples, but if it isn't don't worry, we'll cover the data structure later).

When starting the food detection with the FoodRecognitionListener as the callback, on every frame analyzed you will receive a FoodCandidates object. This is the structure of that data class:

data class FoodCandidates(
    val detectedCandidates: List<DetectedCandidate>? = null,
    val barcodeCandidates: List<BarcodeCandidate>? = null,
    val packagedFoodCandidates: List<PackagedFoodCandidate>? = null
)

Depending on how you structured the FoodDetectionConfiguration object, some fields of the FoodCandidates object might be null. If the field is null it means that that type of detection has not been run. On the other hand, if the detection process ran to completion and the list is empty, then that type of detection could not recognize anything from the given frame.

If at any point you need help from the Passio team, please reach out to us at support@passiolife.com

Welcome to the Passio Nutrition-AI Android SDK!

To start using the camera in your Activity/Fragment, implement the PassioCameraViewProvider interface. By implementing this interface the SDK will use that component as the lifecycle owner of the camera (when that component calls onPause() the camera will stop) and also will provide the Context in order for the camera to start. The component implementing the interface must have a in its view hierarchy.

Start by adding the PreviewView to your view hierarchy. Go to your layout.xml and add the following.

The SDK requires the device's Camera to work. To ensure that the Camera is working and sending frames to analyze, you need the user's permission to access the camera. As of now, only the back camera is supported by the SDK. There are two ways to implement Camera usage along with the permission request. Review both of these strategies and choose what best suits your code.

Using the PassioCameraFragment

PassioCameraFragment is an abstract class that handles Camera permission at runtime as well as starting the Camera process of the SDK. To use the PassioCameraFragment simply extend it in your own fragment and supply the PreviewView that has been added to the view hierarchy in the previous step.

Using the PassioCameraViewProvider

This approach is more manual but gives you more flexibility. You need to implement the PassioCameraViewProvider interface and supply the needed LifecycleOwner and the PreviewView added in the initial step.

Finally, after the user has granted permission to use the camera, start the SDK camera

Run the code to make sure that the preview is working

If at any point you need help from the Passio team, please reach out to us at support@passiolife.com

Once the food detection process has returned a PassioID, it is time to query the database and see what information Passio has on the detected food.

The most detailed information can be found in the PassioIDAttributes object. To retrieve this object from a given PassioID, simply call

PassioIDAttributes

Here we will cover the fields that are not self-explanatory:

The image for a food can also be retrieved using the PassioID and the method

1. parents, children, siblings - All the foods in Passio's Nutritional Database are organized in a tree-like structure with each PassioID being a node in that tree. Let's say we scanned a "mandarin". A child of a "mandarin" is a "tangerine", and its parent is a "whole citrus fruit". We can offer these alternatives to the user in case the user is pointing to a similar food to a mandarin.

2. passioFoodItemDataForDefault, passioFoodRecipe - The SDK differentiates single food items and food recipes. A single item represents just one particular food like an apple or peas. A food recipe consists of more than one single food item. For example a recipe is a Caprese salad which consists of mozzarella, tomatoes, basil, salt, black pepper and olive oil.

PassioFoodItemData

PassioFoodItemData holds the nutritional information for a food item like calories, carbs, protein, fat etc. It also has a reference weight on which these values are calculated and contains serving sizes and serving units. Serving sizes offer you quick options to select the size of the food with a given quantity and unit.

PassioFoodRecipe

Recipes should be viewed as a collection of PassioFoodItemData objects.

Other useful queries

If you have a PassioID returned from the detection process and you are just interested in the name of the corresponding food, use

To query the names of the foods from the database by a filter use

The returned list contains a PassioID paired with the name of the food.

If at any point you need help from the Passio team, please reach out to us at support@passiolife.com

The Passio Android SDK is shipped in the form of an file.

Download the .aar from our distribution repository

• Visit the of the Android-Passio-SDK-Distribution GitHub repository. Download the passiolib-release.aar file from the latest release.

• If you don't have access to the repository please contact our support team at support@passiolife.com

Include the .aar in your Android Studio project

To include the .aar file into your project go to the module that will contain the food recognition feature. Place the .aar file in the libs folder of the module. If the libs folder is missing, create one. Reference the .aar file in the build.gradle file of the implementing module.

Sync Project with Gradle Files and be sure that you can reference the PassioSDK class within your code.

Dependencies

Passio Android SDK is powered by and with the camera being managed by . Add the dependencies to these three projects by adding these lines to the module's build.gradle file.

In order for the SDK to work add the following lines to the android section of the module's build.gradle file.

Running Sync Project with Gradle Files will enable access to the SDK.

If at any point you need help from the Passio team, please reach out to us at support@passiolife.com

Visual food detection

A DetectedCandidate represents the result from running Passio's neural network, specialized for detecting foods like apples, salads, burgers etc.

Barcode detection

Try to opt in for BARCODE by enabling the flag detectBarcodes and point your device at the image of the barcodes below. The SDK's barcode detector will recognize the barcode and it will return the barcode number inside the BarcodeCandidate object.

Use the barcode field to query Passio's backend and if the scanned food can be found in Passio's database, a PassioIDAttributes object will be returned.

Packaged food detection

The FoodCandidate returns a list of VisualCandidates, which object should I choose to display to the user?

The Visual Detection process can detect up to 5 objects per frame. Depending on your UI/UX you can display all the results to the end user, or just choose one of them. The criteria to choose the best candidate from a list is up to you, but the two most common strategies are the biggest bounding box or the highest confidence. This biggest bounding box represents the biggest object on the frame and the highest confidence points to the object that the neural network is most sure about.

If at any point you need help from the Passio team, please reach out to us at support@passiolife.com