Best Practices

Your api key should never be visible to customers. Your backend service which you are going to enhance with the advisor should be the only server ever requesting a token.

Keep your API Key Safe

It is your responsibility to keep your API key safe.

If you make this request on your client side, they will be able to see your API key in the headers, and abuse it.

Auth Route

Requesting the Token

In order to obtain your token for the API, you will need to send a request with your API key to our token service.

For example:

https://api.passiolife.com/v2/token-cache/nutrition-advisor/oauth/token/1i5eXnFpiRfiBGgLibonnBg10Ct14nALTAK5Jb6B4V4o

The key values to note in the response are the access_token , customer_id , and the expires_in fields.

{
    "access_token": "eyJhbGciOiJSUzI1NiIs...",
    "expires_in": 86155.034762118,
    "token_type": "Bearer",
    "customer_id": "0d8bb889-36e1-11ee-b9e9-92e504c243a4"
}

Implement Refreshing

The token you receive from our service will have a time-stamp of its expiry. This is the seconds until this token expires from the time you obtained it.

Tokens can refresh before the expiration for various reasons, so token-reloads on 403 responses should be handled in your API if necessary.

Recommended Implementation:

• Put your license key into your applications env using injection, and ideally a secret manager

• Create a wrapper, or preflight check when sending requests to Nutrition Advisor such that:

  • If there is no current token, obtain one as documented below

  • Anytime you collect a token, note the time it was gathered. The sum of the timestamp of receiving the response, and the expires_in duration is when your token will expire.

  • Send new requests to refresh token as needed

  • apply the required headers as noted below

Required Headers

All incoming requests require that you have your Passio identifier in the request headers. Your customer id is returned with the token request once your API key has been validated, and can be pulled from the response to apply to your headers.

This is used to track usage and identify leaks. Any requests coming in with no identifier, or an invalid identifier will be rejected.

• Authorization Header: you must provide your access_token in an authorization header with the prefix Bearer for example: {"Authorization": "Bearer <access_token here>"}

• Passio Identification Header: you must provide your passio customer_id (either as returned with the token or as seen on your product dashboard) in a Passio-ID header

{
  "Authorization": "Bearer <access_token here>",
  "Passio-ID": "0d8bb889-36e1-11ee-b9e9-92e504c243a4"
}

Continue to API Request Setup to see code examples on making requests to Nutrition Advisor

Code Examples

These examples assume you have followed the steps from Requesting the Token and have received a response from that route.

Python

import requests

response = {
    "access_token": "eyJhbGciOiJSUzI1NiIsI...",
    "expires_in": 86155.034762118,
    "token_type": "Bearer",
    "customer_id": "0d8bb889-36e1-11ee-b9e9-92e504c243a4"
}

headers = {
    "Authorization": f"Bearer {response['access_token']}",
    "Passio-ID": response['customer_id']
}

url = "https://api.passiolife.com/v2/products/nutrition-advisor/..."

r = requests.get(url, headers=headers)
print(r.json())

Go

package main

import (
    "io/ioutil"
    "net/http"
)

func main() {
    response := map[string]interface{}{
        "access_token": "eyJhbGciOiJSUzI1NiIsI...",
        "expires_in": 86155.034762118,
        "token_type": "Bearer",
        "customer_id": "0d8bb889-36e1-11ee-b9e9-92e504c243a4",
    }

    client := &http.Client{}
    req, _ := http.NewRequest("GET", "https://api.passiolife.com/v2/products/nutrition-advisor/...", nil)
    req.Header.Add("Authorization", "Bearer "+response["access_token"].(string))
    req.Header.Add("Passio-ID", response["customer_id"].(string))

    resp, _ := client.Do(req)
    defer resp.Body.Close()
    body, _ := ioutil.ReadAll(resp.Body)
    println(string(body))
}

React/JS

import axios from 'axios';
import React, { useEffect } from 'react';

const response = {
    "access_token": "eyJhbGciOiJSUzI1NiIsI...",
    "expires_in": 86155.034762118,
    "token_type": "Bearer",
    "customer_id": "0d8bb889-36e1-11ee-b9e9-92e504c243a4"
};

const fetchData = async () => {
    const headers = {
        Authorization: `Bearer ${response.access_token}`,
        'Passio-ID': response.customer_id
    };
    try {
        const result = await axios.get('https://api.passiolife.com/v2/products/nutrition-advisor/...', { headers });
        console.log(result.data);
    } catch (error) {
        console.error(error);
    }
};

function App() {
    useEffect(() => {
        fetchData();
    }, []);

    return <div>App Component</div>;
}

export default App;

Swift

import Foundation

let response = [
    "access_token": "eyJhbGciOiJSUzI1NiIsI...",
    "expires_in": 86155.034762118,
    "token_type": "Bearer",
    "customer_id": "0d8bb889-36e1-11ee-b9e9-92e504c243a4"
]

guard let url = URL(string: "https://api.passiolife.com/v2/products/nutrition-advisor/...") else { return }

var request = URLRequest(url: url)
request.addValue("Bearer \(response["access_token"]!)", forHTTPHeaderField: "Authorization")
request.addValue(response["customer_id"]!, forHTTPHeaderField: "Passio-ID")

let task = URLSession.shared.dataTask(with: request) { data, response, error in
    guard let data = data, error == nil else {
        print(error?.localizedDescription ?? "No data")
        return
    }
    if let httpStatus = response as? HTTPURLResponse, httpStatus.statusCode != 200 {
        print("HTTP Status Code: \(httpStatus.statusCode)")
        return
    }
    if let responseString = String(data: data, encoding: .utf8) {
        print(responseString)
    }
}
task.resume()

cURL

curl -H "Authorization: Bearer eyJhbGciOi..." \
     -H "Passio-ID: 0d8bb889-36e1-11ee-b9e9-92e504c243a4" \
     "https://api.passiolife.com/v2/products/nutrition-advisor/..."

Introduction

This guide provides a concise overview of how to engage in a conversation with the Nutrition Advisor using the API. It includes code examples in Python to help you get started quickly.

This guide assumes you have read and executed the Authorization Process and Request Setup. All requests in code examples pass an undefined headers which you can quickly build following the steps in the above guides.

The Response Object

All responses from the Nutrition Advisor will be returned in the AdvisorResponse object format:

List Available Tools

To list the available tools, make a GET request to /v2/products/nutrition-advisor/tools. The response will include an array of tool objects with the following fields:

Code Example

import requests

url = "https://api.passiolife.com/v2/products/nutrition-advisor/tools"
response = requests.get(url, headers=headers)
tools = response.json()

# Store tools for future reference
cached_tools = {tool["name"]: tool for tool in tools}

Start A Conversation Thread

To start a new conversation, make a POST request to /v2/products/nutrition-advisor/threads. You can optionally include the ?plainText=true query parameter to receive plain text responses without markdown formatting.

Code Example

import requests

url = "https://api.passiolife.com/v2/products/nutrition-advisor/threads?plainText=false"
response = requests.post(url, headers=headers)
advisor_response = response.json()

thread_id = advisor_response["threadId"]

Interraction Flow

When you receive an AdvisorResponse back from the Nutrition Advisor, follow these steps:

1. Check if content is non-empty (Lines 2-17)

• If so, parse contentToolHints.

• If any tools are found, match them to the name field of a cached available tool.

• Use a switch statement based on the tool name to handle specific cases.

• For example, if the tool name is "SearchIngredientMatches", make a call to the target tool type using the message.

2. Check if dataRequest is not null (Lines 19-40)

• If not null, parse the name field and use a switch statement to handle specific cases.

• For example, if the tool name is "DetectMealLogsRequired" and the parameters field contains daysBack: 1, respond to the data request with the appropriate meal log data.

3. Check if actionResponse is not null (Lines 41-48)

• If not null, parse the name field and use a switch statement to handle specific cases.

• For example, if the tool name is "SearchIngredientMatches", parse the data field as JSON to obtain an array of ingredient data.

Code Example

def handle_advisor_response(advisor_response, thread_id, cached_tools):
    # check the advisor for `content`. This is its text-response to the chat
    # If a value exists here, no value will exist in either
    # `dataRequest` or `actionResponse`
    if advisor_response["content"]:
        content = json.loads(advisor_response["content"])
        tool_hints = advisor_response["contentToolHints"]
        
        for tool_name in tool_hints:
            if tool_name in cached_tools:
                if tool_name == "SearchIngredientMatches":
                    tool_url = f"https://api.passiolife.com/v2/products/nutrition-advisor/threads/{thread_id}/messages/tools/target/{tool_name}"
                    tool_response = requests.post(tool_url, json={"messageId": advisor_response["messageId"]}, headers=headers)
                    
                    # Recursively handle the response from the tool request
                    handle_advisor_response(tool_response.json(), thread_id, cached_tools)
                    return
                    
    # If we sent a message with InputSense tools enabled, the response might
    # contain `content`, if no input sense tools were detected as necessary.
    # otherwise, `content` is empty and `dataRequest` is filled, containing
    # info on the data you need to proivde
    if advisor_response["dataRequest"]:
        data_request = advisor_response["dataRequest"]
        
        if data_request["name"] == "DetectMealLogsRequired":
            parameters = json.loads(data_request["respondParameters"])
            days_back = parameters["daysBack"]
            
            respond_url = f"https://api.passiolife.com/v2/products/nutrition-advisor/threads/{thread_id}/messages/{advisor_response['messageId']}/respond"
            respond_data = {
                "data": "...",
                "runId": data_request["runId"],
                "toolCallId": data_request["toolCallId"]
            }
            respond_response = requests.post(respond_url, json=respond_data, headers=headers)
            
            # Recursively handle the response from fulfilling the data request
            handle_advisor_response(respond_response.json(), thread_id, cached_tools)
            return

    # If we sent a tool request, parse actionResponse to retrieve its data
    if advisor_response["actionResponse"]:
        action_response = advisor_response["actionResponse"]
        
        if action_response["name"] == "SearchIngredientMatches":
            ingredient_data = json.loads(action_response["data"])
            # ... do something with the ingredient data
            

Our Tools allow integrated actions between our AI Nutritionist and our database behind our and our services.

Targeted Tools

Target tools can execute some action on an existing message in the conversation history, and return some data related to its content.

See overview of Executing Target Tools

Extract Ingredients (SearchIngredientMatches)

When executed on a message, this tool will find the names of all individual ingredients mentioned within the message, and infer amounts in grams for each.

Each of these food items are then linked to an item from our Nutrition Database and returned in same format as results from a text search with our , with an additional few fields added by a wrapper to link back to your message content.

Return Type

After execution, the return value can be fetched via the actionResponse object, in the data field. The value will be a JSON encoded string.

[
  {
    "ingredientName": "banana", // this will be the string from the message
    "portionSize": "1 medium",
    "weightGrams": 300,
    
    "brandName": "the brand name of the food item if available. brand name can sometimes appear in food name",
    "displayName": "display name of the food item",
    "iconId": "the id to use for icon lookup",
    "labelId": "the labelid from the result, if avaialbe",
    "longName": "the food name",
    "nutritionPreview": {
      "calories": 0,
      "carbs": 0,
      "fat": 0,
      "portion": {
        "name": "string",
        "quantity": 0,
        "suggestedQuantity": [
          0
        ],
        "weight": {
          "unit": "string",
          "value": 0
        }
      },
      "protein": 0
    },
    "recipeId": "the recipeid from the result, if avaialbe",
    "referenceId": "the referenceid from the result",
    "resultId": "the resultid from search",
    "score": 0,
    "scoredName": "the name search used to score the item",
    "shortName": "the synonym or recipe matched; otherwise, the food name",
    "synonymId": "the synonymid from the result, if avaialbe",
    "type": "type of match"
  }
]

You can use the ingredientName to relate to your original message, as this will be the string from the message that was used to locate the ingredient information.

More detailed information on the food item can be retrieved using

InputSense Tools

InputSense tools can be enabled when sending user messages. These help reduce cost by preventing you from needing to provide all data at the start of a conversation.

These tools automatically sense if they will need any additional data based on the content of the users message. If they are necessary, your response will contain a dataRequest object. You then Respond To The DataRequest, supplying the data requested, and the following response will answer the users question, with the added context of the data supplied.

See Sending User Messages to see how to enable one in a request

Detect Meal Logs Required (DetectMealLogsRequired)

If enabled, when a user sends a message, Nutrition Advisor will read their message, and determine if the user has sad anything like review my last 2 days of eating or what can i improve in my diet.

These questions imply that the Advisor should have some history on the users eating habits. When detected, rather than receiving an answer immediately, the advisor will respond with a DataRequest

Note if the tool is not enabled, or it didn't think more data was needed, it will respond as usual, and leave dataRequest null.

DataRequest Respond Parameters

If necessary, the dataRequest.respondParameters field will be a JSON encoded string containing

{
    "daysBack": 0.0
}

The provided daysBack float value gives you an idea of how much data the user was asking to be reviewed, so you can send the appropriate amount.

Responding

You can respond with data of any format.

You could send back a json dump, a csv, some plain text saying "i ate an apple", and Advisor will sort out the rest!

The POST body should contain a JSON encoded string of your data (even if the original content is just a string itself).

Enabling This Tool

This tool can be enabled via a query parameter on the route when sending a user message by passing its tool nmae DetectMealLogsRequired in the inputSensors of your message request.

Vision Tools

Vision tools require an image to be sent with the tool request; either by URL or base64 encoded string.

Depending on the tool run, the Advisor will view your image, infer the requested information, and provide an actionResponse object containing the resulting data.

See Vision Tools Overview for more information

Visual Food Extraction(VisualFoodExtraction)

Nutrition Advisor will look at your image, find all food items/ingredients/drinks etc it can locate, and link them to our nutritional database, returning summary info for each food-item in the image.

Advisor will also supply its best-guess of amount (weight in grams) of each item.

Further information, nutrients, etc can be collected for this item via

The advisorResponse.data field will contain a JSON encoded string of an array like the following:

[
  {
    "ingredientName": "banana", 
    "portionSize": "1 medium",
    "weightGrams": 300,
    
    "brandName": "the brand name of the food item if available. brand name can sometimes appear in food name",
    "displayName": "display name of the food item",
    "iconId": "the id to use for icon lookup",
    "labelId": "the labelid from the result, if avaialbe",
    "longName": "the food name",
    "nutritionPreview": {
      "calories": 0,
      "carbs": 0,
      "fat": 0,
      "portion": {
        "name": "string",
        "quantity": 0,
        "suggestedQuantity": [
          0
        ],
        "weight": {
          "unit": "string",
          "value": 0
        }
      },
      "protein": 0
    },
    "recipeId": "the recipeid from the result, if avaialbe",
    "referenceId": "the referenceid from the result",
    "resultId": "the resultid from search",
    "score": 0,
    "scoredName": "the name search used to score the item",
    "shortName": "the synonym or recipe matched; otherwise, the food name",
    "synonymId": "the synonymid from the result, if avaialbe",
    "type": "type of match"
  }
]

After following authorization steps to receive your token, you can begin to interact with the advisor.

Start a Conversation Thread

Once you have your API client authorized, you will need to begin a conversation.

By default, responses will come with markdown style formatting. You can pass the query param ?plainText=true to this route in order to receive plain text, with no markdown formatting.

The React Demo Project for this uses ReactMarkdown package to display the messages with markdown.

Send A Message & Receive Response

Now you can begin the conversation. The content of message can be any serializable type.

Advisor IntelliSense Features

The advisor has 2 types of input sensing intelligence. These are designed to help you reduce token usage, by only suggesting actions or requesting data added to the conversation when it is deemed necessary.

Content Tool Hints (Auto, always enabled)

The AdvisorResponse model contains a field called contentToolHints.

Whenever you receive a message (ie, non-tool related) AdvisorResponse the content field will contain the message. the contentToolHints will provide an array of tool names that could be useful to run on this message, at your discretion.

For Example User sends: generate me a recipe using salmon and asparagus for dinner

The advisor will respond with a recipe for the user. Logically, this recipe is going to contain references to some fruits, veggies, meats, herbs, etc. Because there is reference to food items the response will contain: contentToolHints: ['SearchIngredientMatches'] This informs you that this particular tool may be worth running on the message.

In this case, it lets you know there are food items in the message that could be linked to Passio's Nutrition Database. In the Nutrition Mobile SDK Integration, this hint might prompt the UI displaying an action to "Log this meal", which would then run the extraction, link ingredients, and create a meal log for the user.

InputSense (Optionally enabled on Message Creation)

You can pass the name of Tools of type inputsense (currently limit of 1 per message) to run certain actions if they are detected as necessary. For instance, if DetectMealLogsRequired is given as an inputSensors and the users message was something like:

check my last 3 days of eating and tell me what I can improve

In this case you would receive a response containing a DataRequest with some info you should provide. In this case it would tell you it needs 3 days of the users meal log data.

This design prevents having to include all the users meal logs at all times in the conversation history, to save you token usage costs (as every token in a given conversation contributes to the cost of all future messages in that conversation).

AdvisorResponse Object

The AdvisorResponse object represents the response received from the Nutrition Advisor API. It contains information about the conversation thread, the advisor's message, and any additional data or actions related to the response.

Fields

• threadId (string): The unique identifier of the conversation thread.

• messageId (string): The unique identifier of the advisor's response message.

• content (string): The main content of the advisor's response. Empty if message has a dataRequest or actionResponse. Should be JSON Decoded from string when received.

• contentToolHints ([]string): An array of tool names (or empty) that the advisor has sensed could be useful to run on this response. See Content Tool Hints

• dataRequest (AdvisorDataRequest, optional): If the advisor requires additional information, this field contains the details of the data request.

• actionResponse (AdvisorActionResponse, optional): If the advisor performs an action based on the message, this field contains the details of the action response.

• usage (AdvisorResponseUsage, optional): Information about the token usage for the response.

AdvisorDataRequest Object

The AdvisorDataRequest object represents a data request from the advisor when additional information is needed to provide a complete response. This can be triggered on Sending Messages, if related parameters are enabled.

To respond to a DataRequest

RespondParameters will return a JSON encoded string of a key/value object containing information on the data being requested. For instance, when detecting meal logs you might receive:

{"daysBack": 3}

To give you some idea of how many logs to supply.

AdvisorActionResponse Object

The AdvisorActionResponse object represents an action performed by the advisor based on the message. This will be received after submitting a Tool Run.

The data field will contain a JSON encoded string of the objects returned by the tool.

// AdvisorResponse object
{
    // ActionResponse object
    "actionResponse": {
        "data": "json-encoded string of the data result of the tool execution. see tool definitions for details https://passio.gitbook.io/nutrition-advisor-api/interaction-guide/advisor-tools",
        "messageId": "the id of the message to execute the tool on",
        "name": "the tool name/key that was run"
    },
    "content": "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null.",
    "dataRequest": {...},
    "messageId": "string",
    "threadId": "string",
    "usage": {...}
}

AdvisorResponseUsage Object

The AdvisorResponseUsage object provides information about the token usage for the advisor's response.

{
    "actionResponse": {...},
    "content": "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null.",
    "dataRequest": {...},
    "messageId": "string",
    "threadId": "string",
    // Usage Object
    "usage": {
        "model": "the base model used under passios tools (for billing clarity)",
        "tokensUsed": 0
    }
}

Note: The InputTokens and OutputTokens fields are not included in the JSON representation of the AdvisorResponseUsage object.

Get Available Tools

This route will list the tools available for you to run, and differentiate which tools can be enabled for messages going to the advisor (resulting in possible Data Requests) or if they can be executed on messages from the advisor (Tool Executions).

Executing Tools

Using the tool name and type from the Get Tools route, you can execute a tool on a message.

The different tool types have different methods of execution described below.

Type - inputsense

Tools of type inputsense can be used with outgoing user messages. Pass the tool name in the inputSenseors field when sending a user message to enabled this tool. See Sending Messages

Type - target

Tools of type target are run on specific messages in the thread history.

These are often used in a Context Menu sense, for instance after asking the Advisor to generate a recipe, once you are happy with the recipe you get, you can set up a right-click action that runs the SearchIngredientMatches tool on the messageId to link all of the ingredients specified to passios food database.

Type - vision

Tools of type vision are actions that can be run on images. The image must be sent with the vision tool request, either by URL or a base64-encoded string in the requests image field.

To begin, you will need your API key. Once you have subscribed for api access, your product dashboard in our accounts portal (https://accounts.passiolife.com) will contain your license key, as well as your customer ID.

The customer ID is provided in the portal for convenience, though it is also returned with your token when requesting access. You can choose to hardcode it somewhere in your app, or just pull it out from the token response.