Harbor
Run a complete local LLM stack with one command
About Harbor
Harbor is a containerized CLI toolkit that brings a complete pre-wired LLM stack with hundreds of services to your local machine. It handles Docker Compose orchestration, configuration, and cross-service connectivity so you can run LLM backends like Ollama, llama.cpp, or vLLM alongside frontends like Open WebUI and supporting services like SearXNG, ComfyUI, and LangFuse — all with minimal configuration.
Pros & Cons
Pros
- Single command spins up an entire local AI development environment with dozens of pre-wired services
- Supports all major LLM inference engines and frontends with automatic cross-service connectivity
- Shared cache system prevents redundant downloads of large model files across services
- Completely free and open source under MIT license with active development
- Harbor Boost modules add middleware capabilities like caching, routing, and request modification
- Eliminates complex Docker Compose configuration for multi-service AI stacks
Cons
- Not designed for production deployment — strictly a local development and experimentation tool
- Requires Docker knowledge and comfortable with containerized environments
- Still in v0.x pre-release stage, so breaking changes may occur between updates
- Resource-intensive as running multiple AI services simultaneously demands significant hardware
- Users experienced with Docker and Linux administration may not need the abstraction layer
Key Features
One-Command Setup
Spin up a complete local LLM stack with a single harbor up command — backends, frontends, and services all pre-wired together
100+ Supported Services
Supports major inference engines (Ollama, llama.cpp, vLLM, TGI, LiteLLM, TabbyAPI, Aphrodite, SGLang) plus dozens of frontends and utilities
Pre-Configured Integrations
Services are automatically connected — running SearXNG enables Web RAG in Open WebUI, Speaches provides OpenAI-compatible STT and TTS out of the box
Harbor App UI
A companion desktop app providing a clean interface for managing services, configurations, and workflows
Harbor Boost
Modular enhancement system that adds capabilities like caching, routing, and middleware to your LLM stack
Harbor Bench
Built-in benchmarking tool for evaluating and comparing LLM performance across different backends and models
Shared Cache System
Host cache is shared and reused across services — Hugging Face models, Ollama weights, and other artifacts are downloaded once
Service CLI Access
Access service CLIs (hf, ollama, etc.) directly via Docker without local installation of each tool
Cross-Platform Support
Built on Docker Compose, works across Linux, macOS, and Windows with consistent behavior
Pricing
- All services included
- Unlimited local usage
- Community support
- MIT license
- Harbor App UI
Best For
Local LLM Development
Set up a complete local AI development environment with inference engines, frontends, and supporting services for model experimentation
AI Service Evaluation
Quickly spin up and compare different LLM backends, frontends, and tooling to find the best combination for your workflow
Offline AI Workflows
Run AI models and services entirely locally for privacy-sensitive work without sending data to external APIs
AI Tool Prototyping
Rapidly prototype applications using local LLMs with pre-connected RAG, image generation, voice chat, and code assistance services
Similar Tools
Visual Studio Code
Free, open-source code editor from Microsoft
Chat2DB
AI-powered SQL client that turns natural language into database queries
Chroma
The open-source AI-native vector database for search and retrieval
Zed
The fastest AI code editor — built in Rust for speed and collaboration
Activepieces
Open-source, AI-first business automation
Flux
Open-source AI image generator with photorealistic output and clean text rendering