Provides a self-hosted web interface and API for interacting with large language models via llama.cpp.
Serge is a robust, self-hosted web interface designed for seamless interaction with large language models (LLMs) such as Alpaca, leveraging the efficient llama.cpp library. It offers a fully dockerized environment, ensuring easy deployment and management, alongside a user-friendly API for programmatic access. The solution features a SvelteKit frontend for an intuitive chat experience, uses Redis for persistent chat history and parameter storage, and integrates FastAPI with LangChain for a powerful backend that wraps llama.cpp calls, making LLM access straightforward and private.
Key Features
- Web interface built with SvelteKit
- Persists chat history and parameters using Redis
- Self-hosted chat interface for LLMs
- Robust API powered by FastAPI and LangChain
- Fully Dockerized for easy deployment
- 5,724 GitHub stars
Use Cases
- Building custom applications that interact with local LLMs via an API
- Setting up a private, self-contained AI chat environment
- Running local LLMs without cloud API keys