Skip to content

feat: add LiteLLM as unified LLM provider#3182

Open
RheagalFire wants to merge 2 commits into
ModelEngine-Group:mainfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add LiteLLM as unified LLM provider#3182
RheagalFire wants to merge 2 commits into
ModelEngine-Group:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire

@RheagalFire RheagalFire commented Jun 2, 2026

Copy link
Copy Markdown

Summary

  • Adds LiteLLM as a model provider, giving nexent users access to 100+ LLM providers (OpenAI, Anthropic, Google Gemini, Azure, Bedrock, Ollama, etc.) through litellm.completion() as an SDK dependency.
  • Two integration points: backend provider for model discovery + SDK model for inference.

Changes

  • backend/consts/provider.py: added LITELLM = "litellm" to ProviderEnum
  • backend/services/providers/litellm_provider.py: new LiteLLMModelProvider for model discovery via /v1/models
  • backend/services/model_provider_service.py: wired LiteLLM into get_provider_models() dispatch
  • sdk/nexent/core/models/litellm_llm.py: new LiteLLMModel with streaming, token tracking, tool calling, drop_params=True
  • sdk/nexent/core/models/__init__.py: registered LiteLLMModel
  • test/sdk/test_litellm_model.py: 17 tests (init, streaming, credentials, edge cases, live E2E)

Tests

Unit tests (17/17 pass):

TestInit::test_basic PASSED
TestInit::test_with_credentials PASSED
TestCall::test_streaming_output PASSED
TestCall::test_api_key_forwarded PASSED
TestCall::test_api_key_omitted_when_none PASSED
TestCall::test_api_base_forwarded PASSED
TestCall::test_drop_params_set PASSED
TestCall::test_response_format PASSED
TestCall::test_stop_sequences PASSED
TestCall::test_token_tracking PASSED
TestEdgeCases::test_empty_stream PASSED
TestEdgeCases::test_chunk_without_choices PASSED
TestEdgeCases::test_context_length_exceeded PASSED
TestEdgeCases::test_auth_error_propagates PASSED
TestEdgeCases::test_import_error PASSED
TestEdgeCases::test_stop_event_interrupts PASSED
TestLiveE2E::test_live_streaming PASSED
============================== 17 passed in 3.64s ==============================

Live E2E (Anthropic claude-sonnet-4-6 via Azure Foundry, streaming):

Live E2E response: 'OK'

Risk / Compatibility

  • Additive only. Existing providers and OpenAIModel untouched.
  • litellm is lazy-imported so the base install is unaffected.
  • drop_params=True ensures cross-provider compatibility.

Example usage

from nexent.core.models import LiteLLMModel

# Any LiteLLM-supported provider
model = LiteLLMModel(
    model_id="anthropic/claude-sonnet-4-20250514",
    api_key="sk-ant-...",
)

# Or Google Gemini
model = LiteLLMModel(model_id="gemini/gemini-2.5-flash")
# export GEMINI_API_KEY=...

response = model([{"role": "user", "content": "Hello!"}])
print(response.content)

@RheagalFire

Copy link
Copy Markdown
Author

cc @Dallas98 @WMC001

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant