Overview
An overview of interacting with the Advisor
After following authorization steps to receive your token, you can begin to interact with the advisor.
Start a Conversation Thread
Once you have your API client authorized, you will need to begin a conversation.
By default, responses will come with markdown style formatting. You can pass the query param
?plainText=true to this route in order to receive plain text, with no markdown formatting.
The React Demo Project for this uses ReactMarkdown package to display the messages with markdown.
Create a new thread with the Nutrition Advisor
POST /v2/products/nutrition-advisor/threads HTTP/1.1
Host: api.passiolife.com
Accept: */*
{
"actionResponse": {
"data": "json-encoded string of the data result of the tool execution. see tool definitions for details https://passio.gitbook.io/nutrition-advisor-api/interaction-guide/advisor-tools",
"messageId": "the id of the message to execute the tool on",
"name": "the tool name/key that was run"
},
"content": "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null.",
"dataRequest": {
"name": "the name (tool key for the route) of the tool or function used. ex: SearchIngredientMatches",
"respondParameters": "json-encoded string of the data to respond with",
"runId": "the run id of the execution (required for responding)",
"toolCallId": "the id of this tool in the execution (required for responding)"
},
"messageId": "text",
"threadId": "text",
"usage": {
"model": "the base model used under passios tools (for billing clarity)",
"tokensUsed": 1
}
}Send A Message & Receive Response
Now you can begin the conversation. The content of message can be any serializable type.
Advisor IntelliSense Features
The advisor has 2 types of input sensing intelligence. These are designed to help you reduce token usage, by only suggesting actions or requesting data added to the conversation when it is deemed necessary.
Content Tool Hints (Auto, always enabled)
The AdvisorResponse model contains a field called contentToolHints.
Whenever you receive a message (ie, non-tool related) AdvisorResponse the content field will contain the message. the contentToolHints will provide an array of tool names that could be useful to run on this message, at your discretion.
For Example
User sends: generate me a recipe using salmon and asparagus for dinner
The advisor will respond with a recipe for the user. Logically, this recipe is going to contain references to some fruits, veggies, meats, herbs, etc. Because there is reference to food items the response will contain: contentToolHints: ['SearchIngredientMatches'] This informs you that this particular tool may be worth running on the message.
In this case, it lets you know there are food items in the message that could be linked to Passio's Nutrition Database. In the Nutrition Mobile SDK Integration, this hint might prompt the UI displaying an action to "Log this meal", which would then run the extraction, link ingredients, and create a meal log for the user.
InputSense (Optionally enabled on Message Creation)
You can pass the name of Tools of type inputsense (currently limit of 1 per message) to run certain actions if they are detected as necessary. For instance, if DetectMealLogsRequired is given as an inputSensors and the users message was something like:
check my last 3 days of eating and tell me what I can improve
In this case you would receive a response containing a DataRequest with some info you should provide. In this case it would tell you it needs 3 days of the users meal log data.
This design prevents having to include all the users meal logs at all times in the conversation history, to save you token usage costs (as every token in a given conversation contributes to the cost of all future messages in that conversation).
Add a message to the Nutrition Advisor thread
The ID of the thread thread
["tool names to enable (only tools of type 'inputsense' are valid)"]POST /v2/products/nutrition-advisor/threads/:threadId/messages HTTP/1.1
Host: api.passiolife.com
Content-Type: application/json
Accept: */*
Content-Length: 100
{
"inputSensors": [
"tool names to enable (only tools of type 'inputsense' are valid)"
],
"message": null
}{
"actionResponse": {
"data": "json-encoded string of the data result of the tool execution. see tool definitions for details https://passio.gitbook.io/nutrition-advisor-api/interaction-guide/advisor-tools",
"messageId": "the id of the message to execute the tool on",
"name": "the tool name/key that was run"
},
"content": "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null.",
"dataRequest": {
"name": "the name (tool key for the route) of the tool or function used. ex: SearchIngredientMatches",
"respondParameters": "json-encoded string of the data to respond with",
"runId": "the run id of the execution (required for responding)",
"toolCallId": "the id of this tool in the execution (required for responding)"
},
"messageId": "text",
"threadId": "text",
"usage": {
"model": "the base model used under passios tools (for billing clarity)",
"tokensUsed": 1
}
}AdvisorResponse Object
The AdvisorResponse object represents the response received from the Nutrition Advisor API. It contains information about the conversation thread, the advisor's message, and any additional data or actions related to the response.
Fields
threadId(string): The unique identifier of the conversation thread.messageId(string): The unique identifier of the advisor's response message.content(string): The main content of the advisor's response. Empty if message has adataRequestoractionResponse. Should be JSON Decoded from string when received.contentToolHints([]string): An array of tool names (or empty) that the advisor has sensed could be useful to run on this response. See Content Tool HintsdataRequest(AdvisorDataRequest, optional): If the advisor requires additional information, this field contains the details of the data request.actionResponse(AdvisorActionResponse, optional): If the advisor performs an action based on the message, this field contains the details of the action response.usage(AdvisorResponseUsage, optional): Information about the token usage for the response.
AdvisorDataRequest Object
The AdvisorDataRequest object represents a data request from the advisor when additional information is needed to provide a complete response. This can be triggered on Sending Messages, if related parameters are enabled.
To respond to a DataRequest
Fulfill a data request
Thread ID
Message ID
the run id of the execution (required for responding)the id of this tool in the execution (required for responding)POST /v2/products/nutrition-advisor/threads/:threadId/messages/:messageId/respond HTTP/1.1
Host: api.passiolife.com
Content-Type: application/json
Accept: */*
Content-Length: 155
{
"data": null,
"runId": "the run id of the execution (required for responding)",
"toolCallId": "the id of this tool in the execution (required for responding)"
}{
"actionResponse": {
"data": "json-encoded string of the data result of the tool execution. see tool definitions for details https://passio.gitbook.io/nutrition-advisor-api/interaction-guide/advisor-tools",
"messageId": "the id of the message to execute the tool on",
"name": "the tool name/key that was run"
},
"content": "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null.",
"dataRequest": {
"name": "the name (tool key for the route) of the tool or function used. ex: SearchIngredientMatches",
"respondParameters": "json-encoded string of the data to respond with",
"runId": "the run id of the execution (required for responding)",
"toolCallId": "the id of this tool in the execution (required for responding)"
},
"messageId": "text",
"threadId": "text",
"usage": {
"model": "the base model used under passios tools (for billing clarity)",
"tokensUsed": 1
}
}RespondParameters will return a JSON encoded string of a key/value object containing information on the data being requested. For instance, when detecting meal logs you might receive:
{"daysBack": 3}
To give you some idea of how many logs to supply.
AdvisorActionResponse Object
The AdvisorActionResponse object represents an action performed by the advisor based on the message. This will be received after submitting a Tool Run.
The data field will contain a JSON encoded string of the objects returned by the tool.
// AdvisorResponse object
{
// ActionResponse object
"actionResponse": {
"data": "json-encoded string of the data result of the tool execution. see tool definitions for details https://passio.gitbook.io/nutrition-advisor-api/interaction-guide/advisor-tools",
"messageId": "the id of the message to execute the tool on",
"name": "the tool name/key that was run"
},
"content": "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null.",
"dataRequest": {...},
"messageId": "string",
"threadId": "string",
"usage": {...}
}AdvisorResponseUsage Object
The AdvisorResponseUsage object provides information about the token usage for the advisor's response.
{
"actionResponse": {...},
"content": "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null.",
"dataRequest": {...},
"messageId": "string",
"threadId": "string",
// Usage Object
"usage": {
"model": "the base model used under passios tools (for billing clarity)",
"tokensUsed": 0
}
}Note: The InputTokens and OutputTokens fields are not included in the JSON representation of the AdvisorResponseUsage object.
Get Available Tools
This route will list the tools available for you to run, and differentiate which tools can be enabled for messages going to the advisor (resulting in possible Data Requests) or if they can be executed on messages from the advisor (Tool Executions).
Show the tool actions available to use. See tool documentation at: https://passio.gitbook.io/nutrition-advisor-api/interaction-guide/overview
GET /v2/products/nutrition-advisor/tools HTTP/1.1
Host: api.passiolife.com
Accept: */*
[
{
"description": "text",
"displayName": "text",
"name": "text",
"type": "target"
}
]Executing Tools
Using the tool name and type from the Get Tools route, you can execute a tool on a message.
The different tool types have different methods of execution described below.
Type - inputsense
Tools of type inputsense can be used with outgoing user messages. Pass the tool name in the inputSenseors field when sending a user message to enabled this tool. See Sending Messages
Type - target
Tools of type target are run on specific messages in the thread history.
These are often used in a Context Menu sense, for instance after asking the Advisor to generate a recipe, once you are happy with the recipe you get, you can set up a right-click action that runs the SearchIngredientMatches tool on the messageId to link all of the ingredients specified to passios food database.
Execute an advisor interraction tool on a message. See available tools and documentation at: https://passio.gitbook.io/nutrition-advisor-api/interaction-guide/advisor-tools
Thread ID
Tool name to execute (only tools of type 'target' are valid)
the id of the message to execute the tool onPOST /v2/products/nutrition-advisor/threads/:threadId/messages/tools/target/:toolName HTTP/1.1
Host: api.passiolife.com
Content-Type: application/json
Accept: */*
Content-Length: 60
{
"messageId": "the id of the message to execute the tool on"
}{
"actionResponse": {
"data": "json-encoded string of the data result of the tool execution. see tool definitions for details https://passio.gitbook.io/nutrition-advisor-api/interaction-guide/advisor-tools",
"messageId": "the id of the message to execute the tool on",
"name": "the tool name/key that was run"
},
"content": "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null.",
"dataRequest": {
"name": "the name (tool key for the route) of the tool or function used. ex: SearchIngredientMatches",
"respondParameters": "json-encoded string of the data to respond with",
"runId": "the run id of the execution (required for responding)",
"toolCallId": "the id of this tool in the execution (required for responding)"
},
"messageId": "text",
"threadId": "text",
"usage": {
"model": "the base model used under passios tools (for billing clarity)",
"tokensUsed": 1
}
}Type - vision
Tools of type vision are actions that can be run on images. The image must be sent with the vision tool request, either by URL or a base64-encoded string in the requests image field.
Execute a 'vision' type tool on an image. Inputs and outputs are not stored in conversation history to avoid excessive token usage
Thread ID
Name of the tool to execute (as seen in name field from GET tools)
url or base64 encoded image. https://example.com/image.jpg || data:image/jpeg;base64,<Base64ImageData>POST /v2/products/nutrition-advisor/threads/:threadId/messages/tools/vision/:toolName HTTP/1.1
Host: api.passiolife.com
Content-Type: application/json
Accept: */*
Content-Length: 129
{
"image": "url or base64 encoded image. https://example.com/image.jpg || data:image/jpeg;base64,<Base64ImageData>",
"message": null
}{
"actionResponse": {
"data": "json-encoded string of the data result of the tool execution. see tool definitions for details https://passio.gitbook.io/nutrition-advisor-api/interaction-guide/advisor-tools",
"messageId": "the id of the message to execute the tool on",
"name": "the tool name/key that was run"
},
"content": "json-encoded string of response content. Will be empty if dataReqeust or actionResponse is not null.",
"dataRequest": {
"name": "the name (tool key for the route) of the tool or function used. ex: SearchIngredientMatches",
"respondParameters": "json-encoded string of the data to respond with",
"runId": "the run id of the execution (required for responding)",
"toolCallId": "the id of this tool in the execution (required for responding)"
},
"messageId": "text",
"threadId": "text",
"usage": {
"model": "the base model used under passios tools (for billing clarity)",
"tokensUsed": 1
}
}Last updated