Get Log Probs

[davide221]

Is there some way to get all the log probs of multiple possible next tokens with the library? So far I managed to get only the probabilities for the chosen token with this example

curl -X POST https://api.together.xyz/v1/chat/completions \
     -H 'Content-Type: application/json' \
     -H "Authorization: Bearer $TOGETHER_API_KEY"\
     -d '{
     "model": "codellama/CodeLlama-70b-Instruct-hf",
     "stream": false,
     "max_tokens": 10,
     "messages": [
       {"role": "user", "content": "write an async function in python"}
     ],
     "logprobs": 1
 }'

{
  "id": "85d3d3ff1a0d8c57-EWR",
  "object": "chat.completion",
  "created": 1709240335,
  "model": "codellama/CodeLlama-70b-Instruct-hf",
  "prompt": [],
  "choices": [
    {
      "finish_reason": "length",
      "logprobs": {
        "tokens": [
          "1",
          ".",
          " Define",
          " the",
          " function",
          " with",
          " the",
          " async",
          " keyword",
          "."
        ],
        "token_logprobs": [
          -1.0029297,
          -0.07861328,
          -2.1367188,
          -0.921875,
          -0.6479492,
          -1.1318359,
          -0.35668945,
          -0.6743164,
          -0.023162842,
          -1.1484375
        ]
      },
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1. Define the function with the async keyword."
      }
    }
  ],
  "usage": {
    "prompt_tokens": 33,
    "completion_tokens": 10,
    "total_tokens": 43
  }
}