Home Lab
The lab runs on a single-GPU workstation. Models are served by vLLM and current interacted with via Open WebUI. Everything is containerised.
Current State
| component | current |
|---|---|
| GPU | NVIDIA RTX Pro 5000 (48 GB) |
| Models | Qwen3.8-27B (default), Recursive's Flint |
| Inference | vLLM |
| Front end | Open WebUI |
| Routing | Caddy, reachable over Tailscale only |
Models
The GPU has room for one model at a time. Qwen is the default. Flint swaps in through a compose override file. Flint's fast kernels target B200/GB200, so on this card vLLM falls back to its standard kernels (Marlin for the NVFP4 MoE, FlashInfer for attention).
| setting | Qwen3.8-27B | Flint |
|---|---|---|
| architecture | 27B dense | 40B MoE, 2.5B active |
| weights | FP8 (~31 GB) | NVFP4 GGUF (23 GB) |
| context length | 64K | 262K (full) |
| KV cache | FP8 | BF16 |
| max concurrent seqs | 8 | 8 |
| speculative decoding | MTP, 3 tokens | – |
| reasoning | qwen3 parser, low effort by default |
glm47 parser |
| tool calls | qwen3_xml parser |
glm47 parser |
| throughput | – | ~240 tokens/s, single stream |
Infra
Networking in managed via Tailscale so access is restricted to approved devices. Caddy proxies inbound requests to Open WebUI.
Containers
| container | role |
|---|---|
| vllm | Serves the model on the GPU. Weights load from local disk with the Hugging Face Hub offline. |
| open-webui | Chat UI. Chats, users and settings persist to a local data directory. |
| caddy | Reverse proxy on the host network, bound to the Tailscale IP. |
| monitor | Checks vLLM and Open WebUI every minute and sends a Telegram alert on failure. |