Effortlessly run LLM backends, APIs, frontends, and services with one command.
!GitHub repo size
!GitHub repo file or directory count
!GitHub language count
!Harbor Ko-fi
Setup your local LLM stack effortlessly.
``bashStarts fully configured Open WebUI and Ollama
harbor up
Harbor is a CLI and companion app that lets you spin up a complete local LLM stack—backends like Ollama, llama.cpp, or vLLM, frontends like Open WebUI, plus supporting services like SearXNG for web search, Speaches for voice chat, and ComfyUI for image generation—all pre-wired to work together with a single
command. No manual setup: just pick the services you want and Harbor handles the Docker Compose orchestration, configuration, and cross-service connectivity so you can focus on actually using your models.!Screenshot of Harbor CLI and App together
Documentation
- Installing Harbor
Guides to install Harbor CLI and App
- Harbor User Guide
High-level overview of working with Harbor
- Harbor App
Overview and manual for the Harbor companion application
- Harbor Services
Catalog of services available in Harbor
- Harbor CLI Reference
Read more about Harbor CLI commands and options.
Read about supported services and the ways to configure them.
- Join our Discord
Get help, share your experience, and contribute to the project.
What can Harbor do?
!Diagram outlining Harbor's service structure
#### ✦ Local LLMs
Run LLMs and related services locally, with no or minimal configuration, typically in a single command or click.
`bash
All backends are pre-connected to Open WebUI
harbor up ollama
harbor up llamacpp
harbor up vllmSet and remember args for llama.cpp
harbor llamacpp args -ngl 32
`#### Cutting Edge Inference
Harbor supports most of the major inference engines as well as a few of the lesser-known ones.
bash
We sincerely hope you'll never try to run all of them at once
harbor up vllm llamacpp tgi litellm tabbyapi aphrodite sglang ktransformers mistralrs airllm
`#### Tool Use
Enjoy the benefits of MCP ecosystem, extend it to your use-cases.
bash
Manage MCPs with a convenient Web UI
harbor up metamcpConnect MCPs to Open WebUI
harbor up metamcp mcpo
`#### Generate Images
Harbor includes ComfyUI + Flux + Open WebUI integration.
bash
Use FLUX in Open WebUI in one command
harbor up comfyui
`#### Local Web RAG / Deep Research
Harbor includes SearXNG that is pre-connected to a lot of services out of the box: Perplexica, ChatUI, Morphic, Local Deep Research and more.
bash
SearXNG is pre-connected to Open WebUI
harbor up searxngAnd to many other services
harbor up searxng chatui
harbor up searxng morphic
harbor up searxng perplexica
harbor up searxng ldr
`#### LLM Workflows
Harbor includes multiple services for build LLM-based data and chat workflows: Dify, LitLytics, n8n, Open WebUI Pipelines, FloWise, LangFlow
bash
Use Dify in Open WebUI
harbor up dify
`#### Talk to your LLM
Setup voice chats with your LLM in a single command. Open WebUI + Speaches
bash
Speaches includes OpenAI-compatible SST and TTS
and connected to Open WebUI out of the box
harbor up speaches
`#### Chat from the phone
You can access Harbor services from your phone with a QR code. Easily get links for local, LAN or Docker access.
bash
Print a QR code to open the service on your phone
harbor qr
Print a link to open the service on your phone
harbor url webui
`#### Chat from anywhere
Harbor includes a built-in tunneling service to expose your Harbor to the internet.
> [!WARN]
> Be careful exposing your computer to the Internet, it's not safe.
bash
Expose default UI to the internet
harbor tunnelExpose a specific service to the internet
⚠️ Ensure to configure authentication for the service
harbor tunnel vllmHarbor comes with traefik built-in and pre-configured
for all included services
harbor up traefik
`#### LLM Scripting
Harbor Boost allows you to easily script workflows and interactions with downstream LLMs.
bash
Use Harbor Boost to script LLM workflows
harbor up boost
`#### Config Profiles
Save and manage configuration profiles for different scenarios. For example - save llama.cpp args for different models and contexts and switch between them easily.
bash
Save and use config profiles
harbor profile save llama4
harbor profile use defaultImport profiles from a URL
harbor profile use https://example.com/path/to/harbor-profile.env
`#### Command History
Harbor keeps a local-only history of recent commands. Look up and re-run easily, standalone from the system shell history.
bash
Lookup recently used harbor commands
harbor history
`#### Eject
Ready to move to your own setup? Harbor will give you a docker-compose file replicating your setup.
bash
Eject from Harbor into a standalone Docker Compose setup
Will export related services and variables into a standalone file.
harbor eject searxng llamacpp > docker-compose.harbor.yml
`---
Services
##### UIs
Open WebUI ⦁︎
ComfyUI ⦁︎
LibreChat ⦁︎
HuggingFace ChatUI ⦁︎
Lobe Chat ⦁︎
Hollama ⦁︎
parllama ⦁︎
BionicGPT ⦁︎
AnythingLLM ⦁︎
Chat Nio ⦁︎
mikupad ⦁︎
oterm ⦁︎
omnichain ⦁︎
ol1
##### Backends
Ollama ⦁︎
llama.cpp ⦁︎
vLLM ⦁︎
TabbyAPI ⦁︎
Aphrodite Engine ⦁︎
mistral.rs ⦁︎
openedai-speech ⦁︎
Speaches ⦁︎
Parler ⦁︎
text-generation-inference ⦁︎
LMDeploy ⦁︎
AirLLM ⦁︎
SGLang ⦁︎
KTransformers ⦁︎
Nexa SDK ⦁︎
KoboldCpp ⦁︎
Modular MAX
##### Satellites
Harbor Bench ⦁︎
Harbor Boost ⦁︎
SearXNG ⦁︎
Perplexica ⦁︎
Dify ⦁︎
Plandex ⦁︎
LiteLLM ⦁︎
LangFuse ⦁︎
Open Interpreter ⦁
︎cloudflared ⦁︎
cmdh ⦁︎
fabric ⦁︎
txtai RAG ⦁︎
TextGrad ⦁︎
Aider ⦁︎
aichat ⦁︎
autogpt ⦁︎
lm-evaluation-harness ⦁︎
JupyterLab ⦁︎
ol1 ⦁︎
OpenHands ⦁︎
LitLytics ⦁︎
Repopack ⦁︎
n8n ⦁︎
Bolt.new ⦁︎
Open WebUI Pipelines ⦁︎
Qdrant ⦁︎
K6 ⦁︎
Promptfoo ⦁︎
Webtop ⦁︎
OmniParser ⦁︎
Flowise ⦁︎
Langflow ⦁︎
OptiLLM ⦁︎
Morphic ⦁︎
SQL Chat ⦁︎
gptme ⦁︎
traefik ⦁︎
Latent Scope ⦁︎
RAGLite ⦁︎
llama-swap ⦁︎
LibreTranslate ⦁︎
MetaMCP ⦁︎
mcpo ⦁︎
SuperGateway ⦁︎
Local Deep Research ⦁︎
LocalAI ⦁︎
AgentZero ⦁︎
Airweave ⦁︎
Docling ⦁︎
Browser Use ⦁︎
Unsloth ⦁︎
Windmill
See services documentation for a brief overview of each.
CLI Tour
bash
Run Harbor with default services:
Open WebUI and Ollama
harbor upRun Harbor with additional services
Running SearXNG automatically enables Web RAG in Open WebUI
harbor up searxngSpeaches includes OpenAI-compatible SST and TTS
and connected to Open WebUI out of the box
harbor up speachesRun additional/alternative LLM Inference backends
Open Webui is automatically connected to them.
harbor up llamacpp tgi litellm vllm tabbyapi aphrodite sglang ktransformersRun different Frontends
harbor up librechat chatui bionicgpt hollamaGet a free quality boost with
built-in optimizing proxy
harbor up boostUse FLUX in Open WebUI in one command
harbor up comfyuiUse custom models for supported backends
harbor llamacpp model https://huggingface.co/user/repo/model.ggufAccess service CLIs without installing them
Caches are shared between services where possible
harbor hf scan-cache
harbor hf download google/gemma-2-2b-it
harbor ollama listShortcut to HF Hub to find the models
harbor hf find gguf gemma-2
Use HFDownloader and official HF CLI to download models
harbor hf dl -m google/gemma-2-2b-it -c 10 -s ./hf
harbor hf download google/gemma-2-2b-itWhere possible, cache is shared between the services
harbor tgi model google/gemma-2-2b-it
harbor vllm model google/gemma-2-2b-it
harbor aphrodite model google/gemma-2-2b-it
harbor tabbyapi model google/gemma-2-2b-it-exl2
harbor mistralrs model google/gemma-2-2b-it
harbor opint model google/gemma-2-2b-it
harbor sglang model google/gemma-2-2b-itConvenience tools for docker setup
harbor logs llamacpp
harbor exec llamacpp ./scripts/llama-bench --help
harbor shell vllmTell your shell exactly what you think about it
harbor opint
harbor aider
harbor aichat
harbor cmdhUse fabric to LLM-ify your linux pipes
cat ./file.md | harbor fabric --pattern extract_extraordinary_claims | grep "LK99"Open services from the CLI
harbor open webui
harbor open llamacpp
Print yourself a QR to quickly open the
service on your phone
harbor qr
Feeling adventurous? Expose your Harbor
to the internet
harbor tunnelConfig management
harbor config list
harbor config set webui.host.port 8080Create and manage config profiles
harbor profile save l370b
harbor profile use default
Import profile from a URL
harbor profile use https://example.com/path/to/harbor-profile.envLookup recently used harbor commands
harbor historyEject from Harbor into a standalone Docker Compose setup
Will export related services and variables into a standalone file.
harbor eject searxng llamacpp > docker-compose.harbor.ymlRun a built-in LLM benchmark with
your own tasks
harbor bench runGimmick/Fun Area
Argument scrambling, below commands are all the same as above
Harbor doesn't care if it's "vllm model" or "model vllm", it'll
figure it out.
harbor model vllm
harbor vllm modelharbor config get webui.name
harbor get config webui_name
harbor tabbyapi shell
harbor shell tabbyapi
50% gimmick, 50% useful
Ask harbor about itself
harbor how to ping ollama container from the webui?
`Harbor App Demo
https://github.com/user-attachments/assets/a5cd2ef1-3208-400a-8866-7abd85808503
In the demo, Harbor App is used to launch a default stack with Ollama and Open WebUI services. Later, SearXNG is also started, and WebUI can connect to it for the Web RAG right out of the box. After that, Harbor Boost is also started and connected to the WebUI automatically to induce more creative outputs. As a final step, Harbor config is adjusted in the App for the
- If you're comfortable with Docker and Linux administration - you likely don't need Harbor to manage your local LLM environment. However, while growing it - you're also likely to eventually arrive to a similar solution. I know this for a fact, since that's exactly how Harbor came to be.
- Harbor is not designed as a deployment solution, but rather as a helper for the local LLM development environment. It's a good starting point for experimenting with LLMs and related services.
- Workflow/setup centralisation - you can be sure where to find a specific config or service, logs, data and configuration files.
- Convenience factor - single CLI with a lot of services and features, accessible from anywhere on your host.