adam/foundry
1
0
Fork 0
Code Issues 7 Pull requests 6 Projects 1 Releases Packages Wiki Activity Actions

feat: LiteLLM proxy stack #1

Merged
adam merged 7 commits from litellm-proxy into main 2026-06-26 17:40:32 -05:00
Conversation 0 Commits 7 Files changed 6 +568
adam commented 2026-06-26 15:23:33 -05:00
Owner
Copy link

Summary

  • Adds stacks/proxy/ with docker-compose.yml, .env, and config.yaml for LiteLLM unified gateway
  • config.yaml is git-tracked and bind-mounted from the stack directory (no manual placement needed)
  • Model picks reviewed against OpenRouter pricing as Ollama cloud cost proxy
  • Fallback chains drop tiers on failure rather than peer-to-peer (peer fallbacks do not help when Ollama cloud is down)

Models

Role Model OR out $/M
Primary (default) glm-5.2:cloud $3.00
Primary (coding) kimi-k2.7-code:cloud $3.50
Primary (heavy reasoning) deepseek-v4-pro:cloud $0.87
Primary (fast) + aux-fast deepseek-v4-flash:cloud $0.18
Primary (large coding) qwen3-coder:480b-cloud $1.80
Primary (large general) qwen3.5:122b-cloud $2.45
aux-small gpt-oss:20b-cloud $0.14
aux-medium nemotron-3-super:cloud (120B MoE) $0.40
On-demand gpt-oss:120b, nemotron-3-nano:30b —
Embeddings nomic-embed-text:latest (local) —

Before merging

  • Add secrets to 1Password foundry/litellm item
  • Create Postgres litellm DB + user
  • Set secrets in Dockhand UI for proxy stack

Test plan

  • curl http://192.168.4.200:4000/health → 200 OK
  • Primary model call succeeds (glm-5.2)
  • Cache hit on repeated identical request
  • Fallback triggers when primary model fails
  • Hermes config updated to point at LiteLLM
## Summary - Adds `stacks/proxy/` with docker-compose.yml, .env, and config.yaml for LiteLLM unified gateway - `config.yaml` is git-tracked and bind-mounted from the stack directory (no manual placement needed) - Model picks reviewed against OpenRouter pricing as Ollama cloud cost proxy - Fallback chains drop tiers on failure rather than peer-to-peer (peer fallbacks do not help when Ollama cloud is down) ## Models | Role | Model | OR out $/M | |---|---|---| | Primary (default) | glm-5.2:cloud | $3.00 | | Primary (coding) | kimi-k2.7-code:cloud | $3.50 | | Primary (heavy reasoning) | deepseek-v4-pro:cloud | $0.87 | | Primary (fast) + aux-fast | deepseek-v4-flash:cloud | $0.18 | | Primary (large coding) | qwen3-coder:480b-cloud | $1.80 | | Primary (large general) | qwen3.5:122b-cloud | $2.45 | | aux-small | gpt-oss:20b-cloud | $0.14 | | aux-medium | nemotron-3-super:cloud (120B MoE) | $0.40 | | On-demand | gpt-oss:120b, nemotron-3-nano:30b | — | | Embeddings | nomic-embed-text:latest (local) | — | ## Before merging - [ ] Add secrets to 1Password `foundry/litellm` item - [ ] Create Postgres `litellm` DB + user - [ ] Set secrets in Dockhand UI for proxy stack ## Test plan - [ ] `curl http://192.168.4.200:4000/health` → 200 OK - [ ] Primary model call succeeds (glm-5.2) - [ ] Cache hit on repeated identical request - [ ] Fallback triggers when primary model fails - [ ] Hermes config updated to point at LiteLLM
- Architecture: Hermes/Hindsight -> LiteLLM (:4000) -> Ollama
- All models via Ollama cloud (no external API keys needed)
- Official ghcr.io image first, Forgejo mirror build as follow-up
- Redis caching, fallback chains, Postgres for virtual keys
- config.yaml at /mnt/user/appdata/litellm/ (matches existing stack patterns)
- Swap aux-medium from gemma4:31b to nemotron-3-super (120B MoE, better value)
- Drop gemini-3-flash, minimax-m3, qwen3.5:397b (unavailable or no clear role)
- Fix qwen3.5 tag: 397b → 122b (largest available on Ollama cloud)
- Simplify fallback chains: drop-tier on failure instead of peer-to-peer
- Add OR pricing as cost proxy in model inventory table
- Mark open questions 1 and 2 resolved

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
docker-compose.yml, .env, and config.yaml for the LiteLLM proxy stack.
config.yaml is git-tracked and bind-mounted from the stack directory.
Also adds stacks/proxy to CLAUDE.md stack table.

Model picks:
- Primaries: glm-5.2, kimi-k2.7-code, deepseek-v4-pro/flash, qwen3-coder, qwen3.5:122b
- Aux: aux-small=gpt-oss:20b, aux-medium=nemotron-3-super, aux-fast=deepseek-v4-flash
- On-demand: nemotron-3-nano, gpt-oss:120b
- Fallbacks drop tiers rather than peer-to-peer

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
claude user + token already provisioned; new session just needs to
write the token file and credential helper on hermes (192.168.4.210).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Use hermes user (not claude); token generated and tied to hermes account.
hermes added as write collaborator on adam/foundry.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
adam merged commit 8533e8c08d into main 2026-06-26 17:40:32 -05:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
lab-honing
Lab-access honing backlog (from 2026-07-11 session review)

No labels
lab-honing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
adam/foundry!1
No description provided.