Setup
- Install a current LiteLLM release and start with a direct Python SDK call, not a second proxy layer.
- Use openai/ followed by the exact TokenReply model ID when LiteLLM needs a provider prefix.
- Set api_base to the TokenReply /v1 URL and api_key to a TokenReply key.
- Add retries or a LiteLLM proxy only after the direct request succeeds.
from litellm import completion
response = completion(
model="openai/YOUR_MODEL_ID",
api_key="sk-your-token-here",
api_base="https://api.tokenreply.com/v1",
messages=[{"role": "user", "content": "Say hello."}],
)
print(response.choices[0].message.content)
How to verify
- A plain completion returns an OpenAI-shaped response with non-empty message content.
- Authentication failures are handled as authentication errors rather than retried indefinitely.
- Streaming and tool calls are verified separately before production use.
Limitations
- LiteLLM provider prefixes and parameter handling can change between releases; pin and test the version you deploy.
- Running LiteLLM Proxy in front of TokenReply adds another authentication, retry, and logging layer.
- Do not assume every catalog model accepts every OpenAI parameter.
Troubleshooting
| Symptom | What it usually means | What to do |
|---|
| LiteLLM reports an unknown provider | The model name does not include the provider prefix LiteLLM uses for custom OpenAI-compatible calls. | Try openai/YOUR_MODEL_ID while leaving the TokenReply catalog ID unchanged after the slash. |
|---|
| A proxy deployment loops or double retries | Both LiteLLM Proxy and the application retry the same upstream failure. | Choose one retry owner and cap attempts with exponential backoff. |
|---|
