Image Recognition

Extract foods and ingredients from an uploaded image. This endpoint can recognize images of barcodes as well.

We recommend resizing images to 500px (maintaining aspect ratio) for this endpoint.

Extract Foods from Image (Auto-Typed) — Recommended

post

The recommended camera-to-food endpoint. Takes one image, figures out what kind of food content is in the frame — a packaged product, a barcode / UPC, a raw / cooked dish, or a combination — and runs the right extraction path automatically. Returns Passio-linked ingredients in a single round-trip.

Use this when

• A user wants to point their camera at food and log it: "I want to send an image and get back all the food items in it for logging" — this is the route to call.

• You don't know in advance whether the photo is a packaged product, a barcode, or a plate of food.

• You want one call instead of the legacy "detect → link" two-step (see deprecated detectIngredientsFromImages).

For specialised cases, use extractPackagedProduct (full nutrition-label / barcode parse) or extractFoodAndIngredients (when you specifically want the meal-structured response).

Input / output

• Body: AdvisorIncomingVisionRequest with a base64 data URI in image, or multipart/form-data with a binary image under video_file / image / image_file / video / file.

• Response: array of FC_IngredientExtractOutput (Passio-linked).

Authorizations

BearerAuth

AuthorizationstringRequired

Bearer token obtained from POST /v2/token-cache/unified/oauth/token/{YourAPIKey}

Body

application/jsonmultipart/form-data

imagestringRequired

Base64 encoded image as a data URI. Expected format: 'data:;base64,'. For example: 'data:image/jpeg;base64,/9j/4AAQSkZJRgABA...'. You can generate this from an image file using ImageMagick or similar tools: base64_data=$(base64 <image.jpg) && echo "data:image/jpeg;base64,${base64_data}". Supported formats: JPEG, PNG, WebP, and GIF.

messageobjectOptional

Optional context message or metadata about the image to improve analysis (e.g., {"context": "lunch meal", "dietary_preference": "vegetarian"}).

Responses

200

Successful response

application/json

400

Bad Request - Invalid input.

application/json

500

Internal Server Error.

application/json

post

/napi/tools/vision/extractIngredientsAutoTyped

HTTPcURLJavaScriptPython

POST /v2/products/napi/tools/vision/extractIngredientsAutoTyped HTTP/1.1
Host: api.passiolife.com
Authorization: Bearer YOUR_SECRET_TOKEN
Content-Type: application/json
Accept: */*
Content-Length: 29

{
  "image": "text",
  "message": {}
}

application/jsonmultipart/form-data

[
  {
    "ingredientName": "text",
    "portionSize": "text",
    "portionQuantity": 1,
    "weightGrams": 1,
    "weightIsExplicit": true,
    "mealName": "text",
    "componentName": "text",
    "childOf": "text",
    "barcode": "text",
    "packageType": "text",
    "scanType": "text",
    "isBrandOnly": true,
    "isSpecificBrandedProduct": true,
    "isUnknownBrand": true,
    "isSupplement": true,
    "isSeveralFoodsCombined": true,
    "isRecipe": true,
    "isSingleIngredient": true,
    "correctedTerm": "text",
    "canonicalQuery": "text",
    "descriptors": [
      "text"
    ],
    "preparationMethods": [
      "text"
    ],
    "cutsOrParts": [
      "text"
    ],
    "potentialBrands": [
      "text"
    ],
    "alternateNames": [
      "text"
    ],
    "semanticBrand": "text",
    "preferBranded": true,
    "preferRecipes": true,
    "preferGeneric": true,
    "preferIngredients": true,
    "dietaryTags": [
      "text"
    ],
    "baseFood": "text",
    "type": "text",
    "displayName": "text",
    "shortName": "text",
    "longName": "text",
    "scoredName": "text",
    "score": 1,
    "brandName": "text",
    "iconId": "text",
    "labelId": "text",
    "synonymId": "text",
    "recipeId": "text",
    "referenceId": "text",
    "resultId": "text",
    "nutritionPreview": {
      "portion": {
        "quantity": 1,
        "weight": {
          "unit": "text",
          "value": 1
        },
        "name": "text",
        "suggestedQuantity": [
          1
        ]
      },
      "calories": 1,
      "carbs": 1,
      "fat": 1,
      "protein": 1,
      "fiber": 1
    },
    "refCode": "text",
    "tags": [
      "text"
    ],
    "concerns": [
      1
    ],
    "itemType": "text",
    "productCode": "text"
  }
]

Example: Sending an image via JSON (recommended)

Convert your image to a base64 data URI and send it in the image field:

# 1. Convert image to base64 data URI
BASE64_DATA=$(base64 < food_photo.jpg)
DATA_URI="data:image/jpeg;base64,${BASE64_DATA}"

# 2. Send the request
curl --location --request POST 'https://api.passiolife.com/v2/products/napi/tools/vision/extractIngredientsAutoTyped' \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer <YOUR_TOKEN>' \
  --data "{\"image\": \"${DATA_URI}\"}"

Example: Sending an image via multipart form-data

Upload a binary file using the field name image:

curl --location --request POST 'https://api.passiolife.com/v2/products/napi/tools/vision/extractIngredientsAutoTyped' \
  --header 'Authorization: Bearer <YOUR_TOKEN>' \
  --header 'Content-Type: multipart/form-data' \
  --form 'image=@"food_photo.jpg"'

Use image as the form field name for image uploads and video_file for video uploads.

Understanding the Response

Each recognized item in the response includes:

• ingredientName, portionSize, weightGrams - the AI's raw recognition output (what the LLM identified and estimated).

• mealName - the name of the overall meal or dish (e.g. "Fruit Salad", "Chicken Stir Fry"). All items that belong to the same meal share this value.

• componentName - the sub-group within a meal this item belongs to (e.g. "salad", "dressing"). For simple dishes, all items typically share the same componentName.

• nutritionPreview - pre-calculated calories, macros, and serving size, mapped to Passio's nutrition database.

• refCode - a reference code to fetch full nutritional details for the item.

Displaying meals vs. individual ingredients: If you want to show the meal as a whole (e.g. "Fruit Salad") rather than listing each ingredient separately, use the mealName field. You can group the returned items by componentName to build a structured breakdown, or simply display mealName as a single entry. The nutritional totals for the meal can be calculated by summing the nutritionPreview values across all items that share the same mealName.

Which fields should I use? The weightGrams field is the AI's raw weight estimate. Passio uses this internally to map the recognition result to the best matching serving size in the nutrition database. The nutritionPreview already reflects this mapping with properly calculated calories and macros for the matched portion.

Unless you are connecting to your own external nutrition database, use nutritionPreview for display and refCode to fetch the full nutritional data model with all available portions and nutrients.