Overview

An overview of interacting with the Advisor

After following authorization steps to receive your token, you can begin to interact with the advisor.

Start a Conversation Thread

Once you have your API client authorized, you will need to begin a conversation.

By default, responses will come with markdown style formatting. You can pass the query param ?plainText=true to this route in order to receive plain text, with no markdown formatting.

The React Demo Project for this uses ReactMarkdown package to display the messages with markdown.

Create a new thread with the Nutrition Advisor

Create a new thread with the Nutrition Advisor

POSThttps://api.passiolife.com/v2/products/nutrition-advisor/threads

Response

200400500

OK

Body

application/json

actionResponseobject

contentstring

Example: "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null."

dataRequestobject

messageIdstring

threadIdstring

usageobject

Request

JavaScriptCurlPythonHTTP

const response = await fetch('https://api.passiolife.com/v2/products/nutrition-advisor/threads', {
    method: 'POST',
    headers: {},
});
const data = await response.json();

Response

200400500

{
  "actionResponse": {
    "data": "json-encoded string of the data result of the tool execution. see tool definitions for details https://passio.gitbook.io/nutrition-advisor-api/interaction-guide/advisor-tools",
    "messageId": "the id of the message to execute the tool on",
    "name": "the tool name/key that was run"
  },
  "content": "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null.",
  "dataRequest": {
    "name": "the name (tool key for the route) of the tool or function used. ex: SearchIngredientMatches",
    "respondParameters": "json-encoded string of the data to respond with",
    "runId": "the run id of the execution (required for responding)",
    "toolCallId": "the id of this tool in the execution (required for responding)"
  },
  "messageId": "text",
  "threadId": "text",
  "usage": {
    "model": "the base model used under passios tools (for billing clarity)",
    "tokensUsed": 0
  }
}

Send A Message & Receive Response

Now you can begin the conversation. The content of message can be any serializable type.

Advisor IntelliSense Features

The advisor has 2 types of input sensing intelligence. These are designed to help you reduce token usage, by only suggesting actions or requesting data added to the conversation when it is deemed necessary.

Content Tool Hints (Auto, always enabled)

The AdvisorResponse model contains a field called contentToolHints.

Whenever you receive a message (ie, non-tool related) AdvisorResponse the content field will contain the message. the contentToolHints will provide an array of tool names that could be useful to run on this message, at your discretion.

For Example User sends: generate me a recipe using salmon and asparagus for dinner

The advisor will respond with a recipe for the user. Logically, this recipe is going to contain references to some fruits, veggies, meats, herbs, etc. Because there is reference to food items the response will contain: contentToolHints: ['SearchIngredientMatches'] This informs you that this particular tool may be worth running on the message.

In this case, it lets you know there are food items in the message that could be linked to Passio's Nutrition Database. In the Nutrition Mobile SDK Integration, this hint might prompt the UI displaying an action to "Log this meal", which would then run the extraction, link ingredients, and create a meal log for the user.

InputSense (Optionally enabled on Message Creation)

You can pass the name of Tools of type inputsense (currently limit of 1 per message) to run certain actions if they are detected as necessary. For instance, if DetectMealLogsRequired is given as an inputSensors and the users message was something like:

check my last 3 days of eating and tell me what I can improve

In this case you would receive a response containing a DataRequest with some info you should provide. In this case it would tell you it needs 3 days of the users meal log data.

This design prevents having to include all the users meal logs at all times in the conversation history, to save you token usage costs (as every token in a given conversation contributes to the cost of all future messages in that conversation).

Add a message to the Nutrition Advisor thread

Add a message to the Nutrition Advisor thread

POSThttps://api.passiolife.com/v2/products/nutrition-advisor/threads/:threadId/messages

Path parameters

threadId*string

The ID of the thread thread

Body

application/json

The message to add to the thread

inputSensorsarray of string

messageany

Response

200400500

OK

Body

application/json

actionResponseobject

contentstring

Example: "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null."

dataRequestobject

messageIdstring

threadIdstring

usageobject

Request

JavaScriptCurlPythonHTTP

const response = await fetch('https://api.passiolife.com/v2/products/nutrition-advisor/threads/:threadId/messages', {
    method: 'POST',
    headers: {
      "Content-Type": "application/json"
    },
    body: JSON.stringify({}),
});
const data = await response.json();

Response

200400500

{
  "actionResponse": {
    "data": "json-encoded string of the data result of the tool execution. see tool definitions for details https://passio.gitbook.io/nutrition-advisor-api/interaction-guide/advisor-tools",
    "messageId": "the id of the message to execute the tool on",
    "name": "the tool name/key that was run"
  },
  "content": "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null.",
  "dataRequest": {
    "name": "the name (tool key for the route) of the tool or function used. ex: SearchIngredientMatches",
    "respondParameters": "json-encoded string of the data to respond with",
    "runId": "the run id of the execution (required for responding)",
    "toolCallId": "the id of this tool in the execution (required for responding)"
  },
  "messageId": "text",
  "threadId": "text",
  "usage": {
    "model": "the base model used under passios tools (for billing clarity)",
    "tokensUsed": 0
  }
}

AdvisorResponse Object

The AdvisorResponse object represents the response received from the Nutrition Advisor API. It contains information about the conversation thread, the advisor's message, and any additional data or actions related to the response.

Fields

• threadId (string): The unique identifier of the conversation thread.

• messageId (string): The unique identifier of the advisor's response message.

• content (string): The main content of the advisor's response. Empty if message has a dataRequest or actionResponse. Should be JSON Decoded from string when received.

• contentToolHints ([]string): An array of tool names (or empty) that the advisor has sensed could be useful to run on this response. See Content Tool Hints

• dataRequest (AdvisorDataRequest, optional): If the advisor requires additional information, this field contains the details of the data request.

• actionResponse (AdvisorActionResponse, optional): If the advisor performs an action based on the message, this field contains the details of the action response.

• usage (AdvisorResponseUsage, optional): Information about the token usage for the response.

AdvisorDataRequest Object

The AdvisorDataRequest object represents a data request from the advisor when additional information is needed to provide a complete response. This can be triggered on Sending Messages, if related parameters are enabled.

To respond to a DataRequest

Fulfill a data request

Fulfill a data request

POSThttps://api.passiolife.com/v2/products/nutrition-advisor/threads/:threadId/messages/:messageId/respond

Path parameters

threadId*string

Thread ID

messageId*string

Message ID

Body

application/json

The data response to fulfill the request

dataany

runIdstring

Example: "the run id of the execution (required for responding)"

toolCallIdstring

Example: "the id of this tool in the execution (required for responding)"

Response

200400500

OK

Body

application/json

actionResponseobject

contentstring

Example: "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null."

dataRequestobject

messageIdstring

threadIdstring

usageobject

Request

JavaScriptCurlPythonHTTP

const response = await fetch('https://api.passiolife.com/v2/products/nutrition-advisor/threads/:threadId/messages/:messageId/respond', {
    method: 'POST',
    headers: {
      "Content-Type": "application/json"
    },
    body: JSON.stringify({}),
});
const data = await response.json();

Response

200400500

{
  "actionResponse": {
    "data": "json-encoded string of the data result of the tool execution. see tool definitions for details https://passio.gitbook.io/nutrition-advisor-api/interaction-guide/advisor-tools",
    "messageId": "the id of the message to execute the tool on",
    "name": "the tool name/key that was run"
  },
  "content": "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null.",
  "dataRequest": {
    "name": "the name (tool key for the route) of the tool or function used. ex: SearchIngredientMatches",
    "respondParameters": "json-encoded string of the data to respond with",
    "runId": "the run id of the execution (required for responding)",
    "toolCallId": "the id of this tool in the execution (required for responding)"
  },
  "messageId": "text",
  "threadId": "text",
  "usage": {
    "model": "the base model used under passios tools (for billing clarity)",
    "tokensUsed": 0
  }
}

RespondParameters will return a JSON encoded string of a key/value object containing information on the data being requested. For instance, when detecting meal logs you might receive:

{"daysBack": 3}

To give you some idea of how many logs to supply.

AdvisorActionResponse Object

The AdvisorActionResponse object represents an action performed by the advisor based on the message. This will be received after submitting a Tool Run.

The data field will contain a JSON encoded string of the objects returned by the tool.

// AdvisorResponse object
{
    // ActionResponse object
    "actionResponse": {
        "data": "json-encoded string of the data result of the tool execution. see tool definitions for details https://passio.gitbook.io/nutrition-advisor-api/interaction-guide/advisor-tools",
        "messageId": "the id of the message to execute the tool on",
        "name": "the tool name/key that was run"
    },
    "content": "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null.",
    "dataRequest": {...},
    "messageId": "string",
    "threadId": "string",
    "usage": {...}
}

AdvisorResponseUsage Object

The AdvisorResponseUsage object provides information about the token usage for the advisor's response.

{
    "actionResponse": {...},
    "content": "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null.",
    "dataRequest": {...},
    "messageId": "string",
    "threadId": "string",
    // Usage Object
    "usage": {
        "model": "the base model used under passios tools (for billing clarity)",
        "tokensUsed": 0
    }
}

Note: The InputTokens and OutputTokens fields are not included in the JSON representation of the AdvisorResponseUsage object.

Get Available Tools

This route will list the tools available for you to run, and differentiate which tools can be enabled for messages going to the advisor (resulting in possible Data Requests) or if they can be executed on messages from the advisor (Tool Executions).

List available tools

Show the tool actions available to use. See tool documentation at: https://passio.gitbook.io/nutrition-advisor-api/interaction-guide/overview

GEThttps://api.passiolife.com/v2/products/nutrition-advisor/tools

Response

200400500

OK

Body

application/json

descriptionstring

displayNamestring

namestring

typeenum

targetinputsensevision

Request

JavaScriptCurlPythonHTTP

const response = await fetch('https://api.passiolife.com/v2/products/nutrition-advisor/tools', {
    method: 'GET',
    headers: {},
});
const data = await response.json();

Response

200400500

[
  {
    "description": "text",
    "displayName": "text",
    "name": "text",
    "type": "target"
  }
]

Executing Tools

Using the tool name and type from the Get Tools route, you can execute a tool on a message.

The different tool types have different methods of execution described below.

Type - inputsense

Tools of type inputsense can be used with outgoing user messages. Pass the tool name in the inputSenseors field when sending a user message to enabled this tool. See Sending Messages

Type - target

Tools of type target are run on specific messages in the thread history.

These are often used in a Context Menu sense, for instance after asking the Advisor to generate a recipe, once you are happy with the recipe you get, you can set up a right-click action that runs the SearchIngredientMatches tool on the messageId to link all of the ingredients specified to passios food database.

Execute a Tool

Execute an advisor interraction tool on a message. See available tools and documentation at: https://passio.gitbook.io/nutrition-advisor-api/interaction-guide/advisor-tools

POSThttps://api.passiolife.com/v2/products/nutrition-advisor/threads/:threadId/messages/tools/target/:toolName

Path parameters

threadId*string

Thread ID

toolName*string

Tool name to execute (only tools of type 'target' are valid)

Body

application/json

details of the target request

messageId*string

Example: "the id of the message to execute the tool on"

Response

200400500

OK

Body

application/json

actionResponseobject

contentstring

Example: "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null."

dataRequestobject

messageIdstring

threadIdstring

usageobject

Request

JavaScriptCurlPythonHTTP

const response = await fetch('https://api.passiolife.com/v2/products/nutrition-advisor/threads/:threadId/messages/tools/target/:toolName', {
    method: 'POST',
    headers: {
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      "messageId": "the id of the message to execute the tool on"
    }),
});
const data = await response.json();

Response

200400500

{
  "actionResponse": {
    "data": "json-encoded string of the data result of the tool execution. see tool definitions for details https://passio.gitbook.io/nutrition-advisor-api/interaction-guide/advisor-tools",
    "messageId": "the id of the message to execute the tool on",
    "name": "the tool name/key that was run"
  },
  "content": "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null.",
  "dataRequest": {
    "name": "the name (tool key for the route) of the tool or function used. ex: SearchIngredientMatches",
    "respondParameters": "json-encoded string of the data to respond with",
    "runId": "the run id of the execution (required for responding)",
    "toolCallId": "the id of this tool in the execution (required for responding)"
  },
  "messageId": "text",
  "threadId": "text",
  "usage": {
    "model": "the base model used under passios tools (for billing clarity)",
    "tokensUsed": 0
  }
}

Type - vision

Tools of type vision are actions that can be run on images. The image must be sent with the vision tool request, either by URL or a base64-encoded string in the requests image field.

Execute a Vision Tool

Execute a 'vision' type tool on an image. Inputs and outputs are not stored in conversation history to avoid excessive token usage

POSThttps://api.passiolife.com/v2/products/nutrition-advisor/threads/:threadId/messages/tools/vision/:toolName

Path parameters

threadId*string

Thread ID

toolName*string

Name of the tool to execute (as seen in name field from GET tools)

Body

application/json

details of the vision request

imagestring

Example: "url or base64 encoded image. https://example.com/image.jpg || data:image/jpeg;base64,<Base64ImageData>"

messageany

Response

200400500

OK

Body

application/json

actionResponseobject

contentstring

Example: "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null."

dataRequestobject

messageIdstring

threadIdstring

usageobject

Request

JavaScriptCurlPythonHTTP

const response = await fetch('https://api.passiolife.com/v2/products/nutrition-advisor/threads/:threadId/messages/tools/vision/:toolName', {
    method: 'POST',
    headers: {
      "Content-Type": "application/json"
    },
    body: JSON.stringify({}),
});
const data = await response.json();

Response

200400500

{
  "actionResponse": {
    "data": "json-encoded string of the data result of the tool execution. see tool definitions for details https://passio.gitbook.io/nutrition-advisor-api/interaction-guide/advisor-tools",
    "messageId": "the id of the message to execute the tool on",
    "name": "the tool name/key that was run"
  },
  "content": "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null.",
  "dataRequest": {
    "name": "the name (tool key for the route) of the tool or function used. ex: SearchIngredientMatches",
    "respondParameters": "json-encoded string of the data to respond with",
    "runId": "the run id of the execution (required for responding)",
    "toolCallId": "the id of this tool in the execution (required for responding)"
  },
  "messageId": "text",
  "threadId": "text",
  "usage": {
    "model": "the base model used under passios tools (for billing clarity)",
    "tokensUsed": 0
  }
}