# vLLM / LocalAI

Model serving you run. vLLM is Apache-2.0; LocalAI is MIT — Hugging Face Inference Endpoints and Replicate without the hosted inference meter.

**Replaces:** Hugging Face Inference Endpoints / Replicate

**Tier:** Medium

## Problem

Hugging Face Inference Endpoints rent a GPU endpoint. Replicate meters model runs. You wanted an OpenAI-compatible server on hardware you control. vLLM is Apache-2.0 (vllm-project/vllm). LocalAI is MIT (already in the catalog). Ollama is the MIT daemon on /best/local-llm. Hugging Face text-generation-inference is archived — not this path. This is not Hugging Face's every hardware SKU, Replicate's hosted model catalog, or free tokens. You still bring weights and usually a GPU. A chat UI stays on /replace/chatgpt.

## Stack

- [vLLM](https://vllm.ai) — Apache-2.0 high-throughput OpenAI-compatible serving engine
- [LocalAI](https://localai.io) — MIT OpenAI-compatible engine you host
- [Ollama](https://ollama.com) — MIT local daemon and API if the job is one box

## Estimated SaaS cost saved

| You stop paying | Typical / year | Self-host instead |
|---|---:|---|
| Hugging Face Endpoints / Replicate (small team) | $1,200 | vLLM or LocalAI + weights you run |

**About $1,200 / year** on a small team, before you count lock-in.

## Deploy on Railway

A GPU host is the honest path for vLLM. LocalAI and Ollama can run smaller models on CPU. No one-click inference cluster in this repo — Start on Railway. Chat UIs stay on /stacks/open-webui-litellm.

[Start on Railway](https://c17.io/go/railway/vllm-inference?from=%2Fstacks%2Fvllm-inference.md)

[Open WebUI if this is the chat window, not the serving engine](https://c17.io/stacks/open-webui-litellm)

## AGENT_PROMPT

```
Replace Hugging Face Inference Endpoints or Replicate with vLLM (throughput) or LocalAI (smaller box).

- Official docs. Persist weights. Point one client at the OpenAI-compatible URL. Do not claim the hosted model catalog.
- Hugging Face TGI is archived — do not deploy it as the path. Chat UIs stay on /stacks/open-webui-litellm.
- Write AGENT_PROMPT.md.
```

## FAQ

### vLLM or LocalAI?

vLLM if you want throughput on a GPU you already run. LocalAI if you want a MIT OpenAI-compatible box. Ollama if the job is one daemon — /compare/vllm-vs-localai and /compare/ollama-vs-vllm.

### Hugging Face or Replicate?

Same published stack. You run the server. The hosted catalog stays behind.

### Do I still pay a GPU?

Usually. The software is the server. Weights and machines are yours.

Source: https://c17.io/stacks/vllm-inference