Advisor Tools

the currently available tools that can be executed on messages

Our Tools allow integrated actions between our AI Nutritionist and our database behind our Nutrition Mobile App SDK and our REST Nutrition API services.

Targeted Tools

Target tools can execute some action on an existing message in the conversation history, and return some data related to its content.

See overview of Executing Target Tools

Extract Ingredients (SearchIngredientMatches)

When executed on a message, this tool will find the names of all individual ingredients mentioned within the message, and infer amounts in grams for each.

Each of these food items are then linked to an item from our Nutrition Database and returned in same format as results from a text search with our REST NutritionAPI, with an additional few fields added by a wrapper to link back to your message content.

Return Type

After execution, the return value can be fetched via the actionResponse object, in the data field. The value will be a JSON encoded string.

[
  {
    "ingredientName": "banana", // this will be the string from the message
    "portionSize": "1 medium",
    "weightGrams": 300,
    
    "brandName": "the brand name of the food item if available. brand name can sometimes appear in food name",
    "displayName": "display name of the food item",
    "iconId": "the id to use for icon lookup",
    "labelId": "the labelid from the result, if avaialbe",
    "longName": "the food name",
    "nutritionPreview": {
      "calories": 0,
      "carbs": 0,
      "fat": 0,
      "portion": {
        "name": "string",
        "quantity": 0,
        "suggestedQuantity": [
          0
        ],
        "weight": {
          "unit": "string",
          "value": 0
        }
      },
      "protein": 0
    },
    "recipeId": "the recipeid from the result, if avaialbe",
    "referenceId": "the referenceid from the result",
    "resultId": "the resultid from search",
    "score": 0,
    "scoredName": "the name search used to score the item",
    "shortName": "the synonym or recipe matched; otherwise, the food name",
    "synonymId": "the synonymid from the result, if avaialbe",
    "type": "type of match"
  }
]

You can use the ingredientName to relate to your original message, as this will be the string from the message that was used to locate the ingredient information.

More detailed information on the food item can be retrieved using Nutrition API

InputSense Tools

InputSense tools can be enabled when sending user messages. These help reduce cost by preventing you from needing to provide all data at the start of a conversation.

These tools automatically sense if they will need any additional data based on the content of the users message. If they are necessary, your response will contain a dataRequest object. You then Respond To The DataRequest, supplying the data requested, and the following response will answer the users question, with the added context of the data supplied.

See Sending User Messages to see how to enable one in a request

Detect Meal Logs Required (DetectMealLogsRequired)

If enabled, when a user sends a message, Nutrition Advisor will read their message, and determine if the user has sad anything like review my last 2 days of eating or what can i improve in my diet.

These questions imply that the Advisor should have some history on the users eating habits. When detected, rather than receiving an answer immediately, the advisor will respond with a DataRequest

Note if the tool is not enabled, or it didn't think more data was needed, it will respond as usual, and leave dataRequest null.

DataRequest Respond Parameters

If necessary, the dataRequest.respondParameters field will be a JSON encoded string containing

{
    "daysBack": 0.0
}

The provided daysBack float value gives you an idea of how much data the user was asking to be reviewed, so you can send the appropriate amount.

Responding

You can respond with data of any format.

You could send back a json dump, a csv, some plain text saying "i ate an apple", and Advisor will sort out the rest!

The POST body should contain a JSON encoded string of your data (even if the original content is just a string itself).

Enabling This Tool

This tool can be enabled via a query parameter on the route when sending a user message by passing its tool nmae DetectMealLogsRequired in the inputSensors of your message request.

Vision Tools

Vision tools require an image to be sent with the tool request; either by URL or base64 encoded string.

Depending on the tool run, the Advisor will view your image, infer the requested information, and provide an actionResponse object containing the resulting data.

See Vision Tools Overview for more information

Visual Food Extraction(VisualFoodExtraction)

Nutrition Advisor will look at your image, find all food items/ingredients/drinks etc it can locate, and link them to our nutritional database, returning summary info for each food-item in the image.

Advisor will also supply its best-guess of amount (weight in grams) of each item.

Further information, nutrients, etc can be collected for this item via Nutrition API

The advisorResponse.data field will contain a JSON encoded string of an array like the following:

[
  {
    "ingredientName": "banana", 
    "portionSize": "1 medium",
    "weightGrams": 300,
    
    "brandName": "the brand name of the food item if available. brand name can sometimes appear in food name",
    "displayName": "display name of the food item",
    "iconId": "the id to use for icon lookup",
    "labelId": "the labelid from the result, if avaialbe",
    "longName": "the food name",
    "nutritionPreview": {
      "calories": 0,
      "carbs": 0,
      "fat": 0,
      "portion": {
        "name": "string",
        "quantity": 0,
        "suggestedQuantity": [
          0
        ],
        "weight": {
          "unit": "string",
          "value": 0
        }
      },
      "protein": 0
    },
    "recipeId": "the recipeid from the result, if avaialbe",
    "referenceId": "the referenceid from the result",
    "resultId": "the resultid from search",
    "score": 0,
    "scoredName": "the name search used to score the item",
    "shortName": "the synonym or recipe matched; otherwise, the food name",
    "synonymId": "the synonymid from the result, if avaialbe",
    "type": "type of match"
  }
]