{"id":13765,"date":"2026-05-30T09:45:58","date_gmt":"2026-05-30T08:45:58","guid":{"rendered":"https:\/\/innovatenews.site\/index.php\/2026\/05\/30\/i-replaced-cloud-llms-with-local-models-running-off-a-proxmox-lxc-and-the\/"},"modified":"2026-05-30T09:46:01","modified_gmt":"2026-05-30T08:46:01","slug":"i-replaced-cloud-llms-with-local-models-running-off-a-proxmox-lxc-and-the","status":"publish","type":"post","link":"https:\/\/innovatenews.site\/index.php\/2026\/05\/30\/i-replaced-cloud-llms-with-local-models-running-off-a-proxmox-lxc-and-the\/","title":{"rendered":"I replaced cloud LLMs with local models running off a Proxmox LXC, and the\u2026"},"content":{"rendered":"
\n

The topic I replaced cloud LLMs with local models running off a Proxmox LXC, and the\u2026<\/strong> is currently the subject of lively discussion \u2014 readers and analysts are keeping a close eye on developments.<\/p>\n

This is taking place in a dynamic environment: companies\u2019 decisions and competitors\u2019 reactions can quickly change the picture.<\/p>\n

Whether it\u2019s Perplexity\u2019s reliable and transparent nature or Claude Code\u2019s programming capabilities, there\u2019s no denying that cloud-based large language models can be a godsend for productivity. Most cloud LLMs ship with beginner-friendly UIs, and the fact that you don\u2019t have to put in extra work just to get them up and running makes them pretty convenient for the average user.<\/p>\n

But I\u2019ve spent the last couple of months moving away from cloud LLMs for my everyday tasks, partly since I don\u2019t want external servers gaining access to my data, and also because I\u2019d rather avoid the extra charges incurred by paid API usage. After migrating through a bunch of setups, I\u2019ve honed in on a local LLM server running on my old Proxmox workstation, and it works surprisingly well for everything from simple prompting to OCR analysis, voice assistant inference backend, and automation pipelines.<\/p>\n

\"\"<\/figure>\n

Like most LLM-hosting enthusiasts, I started my journey by hosting local models on Ollama, and it served me well for the first couple of weeks. After all, pulling LLMs and deploying them is a piece of cake on Ollama, with a bunch of self-hosted apps supporting this inference engine natively. However, its extra performance overhead and lack of advanced tools became pretty apparent once I started looking into ways to maximize the efficiency on my local models. Once I started wanting to run bulky models (and I\u2019ll go over them in a bit), it became clear that Ollama won\u2019t work well for my needs, so I switched to llama.cpp instead.<\/p>\n

Rather, I began using the llama-server functionality to create an LLM server that remains operational 24\/7 and hooks up to the rest of my FOSS arsenal thanks to its OpenAI-compatible API. I also went with a Proxmox LXC, as I can still share my old graphics card with Immich, Frigate, and other apps that need its computational prowess when my LLMs are inactive. Thanks to GPU passthrough, my llama-server LXC gets native-level performance, and I\u2019ve upped its RAM resources all the way to 24GB (out of 32GB) to ensure it can fit MoE models (and I\u2019ll go over them in a bit). On my aged system, I simply ran the ls -l \/dev\/nvidia* command to get the device IDs (195, 235, and 237 for my GPU), pasted the following syntax into the LXC\u2019s config file, and installed the graphics card drivers inside the LXC to configure GPU passthrough, before compiling llama.cpp\u2019s Vulkan variant.<\/p>\n

During my Ollama days, I was starting to get frustrated by the accuracy (or rather, the lack thereof) of local models. Sure, 4B, 7B, and even 9B models could handle simple inference requests, but anything requiring detailed troubleshooting or complex reasoning would be too much for them to handle \u2013 and in some cases, they\u2019d end up spouting complete nonsense. That\u2019s when I started looking into bulkier models \u2013 LLMs that could crunch 20B+ parameters. But considering that my broke self only has a Pascal card (specifically, a GTX 1080), I couldn\u2019t run conventional models without using the –ngl flag to offload entire layers from my GPU and causing the performance to plummet.<\/p>\n

\"\"<\/figure>\n

However, Mixture of Experts models let me offload the less frequently accessed resources onto my CPU and RAM, with the attention weights and other demanding units still remaining on my GPU. As such, I can host models like GPT-OSS-20B and Gemma4-26B-A4B on my VRAM-starved card at respectable token rates, with the latter even managing 15+ t\/s with a fairly large context window.<\/p>\n

As for their reasoning capabilities, I\u2019d say they’re solid competitors to cloud models. While I still prefer the Qwen3.6-35B-A3B for hardcore coding tasks, Gemma4 is pretty effective at rewriting code, providing autosuggestions, and aiding my troubleshooting needs. Likewise, it has yet to hallucinate or provide irrelevant information when I use it for RAG analysis in Paperless AI, Open Notebook, and Blinko. While we\u2019re on this subject\u2026<\/p>\n

Besides its terrific performance, llama-server also deploys an interface for accessing LLMs via a web browser \u2013 and it\u2019s fairly useful for simple prompts and queries. It even supports MCP servers, and as long as I set the context window fairly high (and run the –webui-mcp-proxy flag), I have no issues controlling Obsidian, Home Assistant, TrueNAS, and a bunch of other apps via MCP tools on llama-server\u2019s web interface.<\/p>\n

However, I prefer Open WebUI for the majority of my tasks, and its ChatGPT-like interface makes it fairly accessible. But the real draw of Open WebUI is the sheer number of customization options and integrations that I can pair it (and by extension, my llama-server LLMs) with. There\u2019s the open terminal facility, which lets me execute Python code on the browser, and connecting it with SearXNG lets my Gemma4 instance access websites on the Internet instead of relying solely on its trained knowledge base. It even supports ComfyUI, and I often use Open WebUI to trigger the upscaling workflows I\u2019ve configured on the app.<\/p>\n

I\u2019ve been building my LLM pipelines for a couple of months, and it\u2019s really mind-boggling how much you can accomplish with them. Once you venture past the 20B mark, the reasoning capabilities of self-hosted models skyrocket to the point where they\u2019re good enough to replace their cloud counterparts for coding workloads. And with MoE models becoming more popular, it\u2019s possible to run competent clankers without dealing with slow token generation rates on an old GPU or throwing thousands of bucks on a new system.<\/p>\n