Overview
An overview of interacting with the Advisor
Last updated
An overview of interacting with the Advisor
Last updated
After following authorization steps to receive your token, you can begin to interact with the advisor.
Once you have your API client authorized, you will need to begin a conversation.
By default, responses will come with markdown style formatting. You can pass the query param
?plainText=true to this route in order to receive plain text, with no markdown formatting.
Project for this uses ReactMarkdown package to display the messages with markdown.
Now you can begin the conversation. The content of message can be any serializable type.
The advisor has 2 types of input sensing intelligence. These are designed to help you reduce token usage, by only suggesting actions or requesting data added to the conversation when it is deemed necessary.
The AdvisorResponse model contains a field called contentToolHints.
Whenever you receive a message (ie, non-tool related) AdvisorResponse the content field will contain the message. the contentToolHints will provide an array of tool names that could be useful to run on this message, at your discretion.
For Example
User sends: generate me a recipe using salmon and asparagus for dinner
The advisor will respond with a recipe for the user. Logically, this recipe is going to contain references to some fruits, veggies, meats, herbs, etc. Because there is reference to food items the response will contain: contentToolHints: ['SearchIngredientMatches'] This informs you that this particular tool may be worth running on the message.
check my last 3 days of eating and tell me what I can improve
In this case you would receive a response containing a DataRequest with some info you should provide. In this case it would tell you it needs 3 days of the users meal log data.
This design prevents having to include all the users meal logs at all times in the conversation history, to save you token usage costs (as every token in a given conversation contributes to the cost of all future messages in that conversation).
The AdvisorResponse object represents the response received from the Nutrition Advisor API. It contains information about the conversation thread, the advisor's message, and any additional data or actions related to the response.
threadId (string): The unique identifier of the conversation thread.
messageId (string): The unique identifier of the advisor's response message.
content (string): The main content of the advisor's response. Empty if message has a dataRequest or actionResponse. Should be JSON Decoded from string when received.
To respond to a DataRequest
RespondParameters will return a JSON encoded string of a key/value object containing information on the data being requested. For instance, when detecting meal logs you might receive:
{"daysBack": 3}
To give you some idea of how many logs to supply.
The data field will contain a JSON encoded string of the objects returned by the tool.
The AdvisorResponseUsage object provides information about the token usage for the advisor's response.
Note: The InputTokens and OutputTokens fields are not included in the JSON representation of the AdvisorResponseUsage object.
The different tool types have different methods of execution described below.
Tools of type target are run on specific messages in the thread history.
These are often used in a Context Menu sense, for instance after asking the Advisor to generate a recipe, once you are happy with the recipe you get, you can set up a right-click action that runs the SearchIngredientMatches tool on the messageId to link all of the ingredients specified to passios food database.
Tools of type vision are actions that can be run on images. The image must be sent with the vision tool request, either by URL or a base64-encoded string in the requests image field.
In this case, it lets you know there are food items in the message that could be linked to Passio's Nutrition Database. In theIntegration, this hint might prompt the UI displaying an action to "Log this meal", which would then run the extraction, link ingredients, and create a meal log for the user.
You can pass the name of of type inputsense (currently limit of 1 per message) to run certain actions if they are detected as necessary. For instance, if DetectMealLogsRequired is given as an inputSensors and the users message was something like:
contentToolHints ([]string): An array of tool names (or empty) that the advisor has sensed could be useful to run on this response. See
dataRequest (, optional): If the advisor requires additional information, this field contains the details of the data request.
actionResponse (, optional): If the advisor performs an action based on the message, this field contains the details of the action response.
usage (, optional): Information about the token usage for the response.
The AdvisorDataRequest object represents a data request from the advisor when additional information is needed to provide a complete response. This can be triggered on if related parameters are enabled.
The AdvisorActionResponse object represents an action performed by the advisor based on the message. This will be received after submitting a .
This route will list the tools available for you to run, and differentiate which tools can be enabled for messages going to the advisor (resulting in possible ) or if they can be executed on messages from the advisor ().
Using the tool name and type from the route, you can execute a tool on a message.
Tools of type inputsense can be used with outgoing user messages. Pass the tool name in the inputSenseors field when sending a user message to enabled this tool. See
Show the tool actions available to use. See tool documentation at: https://passio.gitbook.io/nutrition-advisor-api/interaction-guide/overview
Add a message to the Nutrition Advisor thread
The ID of the thread thread
["tool names to enable (only tools of type 'inputsense' are valid)"]Fulfill a data request
Thread ID
Message ID
the run id of the execution (required for responding)the id of this tool in the execution (required for responding)Execute an advisor interraction tool on a message. See available tools and documentation at: https://passio.gitbook.io/nutrition-advisor-api/interaction-guide/advisor-tools
Thread ID
Tool name to execute (only tools of type 'target' are valid)
the id of the message to execute the tool onExecute a 'vision' type tool on an image. Inputs and outputs are not stored in conversation history to avoid excessive token usage
Thread ID
Name of the tool to execute (as seen in name field from GET tools)
url or base64 encoded image. https://example.com/image.jpg || data:image/jpeg;base64,<Base64ImageData>