Sandbox app

With every shipped version of the SDK there will also be a PassioSDKSandbox containing the latest files and showcasing the basic setup of an app using the SDK. In this section we will cover the architecture of the small app and all the calls to the SDK keeping in mind where they are located and when they are called.

The app consists of one MainActivity that holds 2 fragments: RecognizerFragment and PermissionFragment. The latter will be displayed to the user if they deny access to the device camera. The RecognitionFragment is used to display the camera preview, start the recognition session and display the results through a view called FoodResultView.

PassioSDK library

The .aar file has already been imported as a separate module called passiolib-release. If you want to see how Android Studio structures the module, you can right-click on the passiolib-release module in the project viewer and choose the option "Open in". The passiolib-release module has been added as a dependency to the app modules, where we will use the SDK.

Camera

As mentioned above, the camera preview will be used in the RecognizerFragment. If you follow that fragment's onCreateView method you will see that the fragment inflates the view hierarchy found in the file fragment_recognizer.xml. Located in that view hierarchy we have the CameraX's PreviewView with an ID of "recognizerPreviewView". Taking a look back to the RecognizerFragment we can see that this component implements the PassioCameraViewProvider interface. Also, in the fragment's onStart lifecycle callback we are calling the

PassioSDK.instance.startCamera(this)

to start the preview and link the camera to the SDK. We don't need to stop the camera on onStop because we are passing the LifecycleObserver reference to the startCamera method. The camera will stop automatically when it needs to.

SDK Initialization

The SDK is initialized and configured in the onCreate method in the MainActivity.

val config = PassioConfiguration(this.applicationContext, "your_key")
PassioSDK.instance.configure(config) { passioStatus ->
    when (passioStatus.mode) {
        PassioMode.NOT_READY -> onPassioSDKError(passioStatus.debugMessage)
        PassioMode.IS_BEING_CONFIGURED -> onPassioSDKError(passioStatus.debugMessage)
        PassioMode.FAILED_TO_CONFIGURE -> onPassioSDKError(passioStatus.debugMessage)
        PassioMode.IS_DOWNLOADING_MODELS -> onPassioSDKError("Auto updating")
        PassioMode.IS_READY_FOR_DETECTION -> onPassioSDKReady()
    }
}

This configure method means that we are using the files located in the assets folder of the app module. Verify that there are 4 files in the assets folder with the extension .passiosecure.

If you take a closer look, the second parameter of the PassioConfiguration object is your SDK license key. Be sure to edit this method and supply your own license key. If you run the app without providing the key, the onPassioSDKError callback will be called with a message about an invalid key.

Detection session

The session is defined in the onStart and onStop lifecycle callbacks of the RecognizerFragment.

override fun onStart() {
    super.onStart()
    ...
    PassioSDK.instance.startCamera(this)
    val detectionConfig = FoodDetectionConfiguration().apply {
        detectBarcodes = true
    }
    PassioSDK.instance.startFoodDetection(foodListener, detectionConfig)
}

override fun onStop() {
    PassioSDK.instance.stopFoodDetection()
    super.onStop()
}

As described in the section Food detection session, we define a FoodDetectionConfiguration object and through it opt in for VISUAL and BARCODE detection. We then pass this object to startFoodDetection to receive results through the foodListener. When this fragment's onStop is called we are stopping the food detection process so that any results generated in the SDK after this point won't be delivered causing leaks.

private val foodListener = object : FoodRecognitionListener {
    override fun onRecognitionResults(candidates: FoodCandidates, image: Bitmap?, nutritionFacts: PassioNutritionFacts?) {
        val visualCandidates = candidates.detectedCandidates!!
        val barcodeCandidates = candidates.barcodeCandidates!!
        if (visualCandidates.isEmpty() && barcodeCandidates.isEmpty()) {
            recognizerFoodResultView.setSearching()
        } else if (barcodeCandidates.isNotEmpty()) {
            val bestBarcodeResult = barcodeCandidates.maxByOrNull { it.boundingBox.width() * it.boundingBox.height() } ?: return
            recognizerFoodResultView.setBarcodeResult(bestBarcodeResult.barcode)
        } else {
            val bestVisualCandidate = visualCandidates.maxByOrNull { it.confidence } ?: return
            recognizerFoodResultView.setFoodResult(bestVisualCandidate.passioID)
        }
    }

The foodListener above only takes into account the candidates from the detectedCandidates and barcodeCandidate lists because we only opted in for VISUAL and BARCODE detection. The other files of the candidates object received in the onRecognitionResults method will be null. Depending on the candidates within these two lists we will change the state of the FoodResultView denoted as recognizerFoodResultView.

Taking a closer look at the FoodResultView we can see how to get food details such as the name or the icon depending on the source of the detection. If the passioID comes from the FOOD detection we can obtain the PassioIDAttributes through the

PassioSDK.instance.lookupPassioAttributesFor(passioID)

On the other hand, if the results come from the BARCODE detection, we can get the PassioIDAttributes from the networking call

PassioSDK.instance.fetchPassioIDAttributesForBarcode(barcode) { passioIDAttributes ->
    ...
}

If at any point you need help from the Passio team, please reach out to us at [e-mail]