Features
- Multi-provider support: Ollama, OpenAI and an extensible adapter architecture
- Streaming: real-time streaming responses from models
- OpenAI-compatible API: a drop-in replacement for OpenAI’s API
- Multiple interfaces: HTTP server, Discord bot, CLI and library usage
- Embeddable: integrates into existing Axum applications
- Type-safe: strong typing throughout, built on Tokio
Architecture
- Core layer: router, adapters and types (pure inference logic)
- Service layer: provider management and model resolution
- Interface layer: HTTP APIs, Discord bot, CLI
- Application layer: full server or embeddable components
Quickstart
Add Omniference to your project.
Source code
Omniference on GitHub (MIT license).