{"id":5658,"date":"2026-04-22T21:59:09","date_gmt":"2026-04-22T20:59:09","guid":{"rendered":"https:\/\/innovatenews.site\/index.php\/2026\/04\/22\/googles-gemma-4-shines-on-local-systems-both-big-and-small\/"},"modified":"2026-04-22T22:01:28","modified_gmt":"2026-04-22T21:01:28","slug":"googles-gemma-4-shines-on-local-systems-both-big-and-small","status":"publish","type":"post","link":"https:\/\/innovatenews.site\/index.php\/2026\/04\/22\/googles-gemma-4-shines-on-local-systems-both-big-and-small\/","title":{"rendered":"Google\u2019s Gemma 4 shines on local systems \u2013 both big and small"},"content":{"rendered":"
\n

The topic Google\u2019s Gemma 4 shines on local systems \u2013 both big and small<\/strong> is currently the subject of lively discussion \u2014 readers and analysts are keeping a close eye on developments.<\/p>\n

This is taking place in a dynamic environment: companies\u2019 decisions and competitors\u2019 reactions can quickly change the picture.<\/p>\n

Google\u2019s Gemma 4 comes touted as the latest evolution of Google\u2019s multi-modal model offerings. Gemma 4 not only offers reasoning and tool use, but vision and audio functionality, and it\u2019s available in a range of model sizes that target servers and local devices.<\/p>\n

What\u2019s striking about Gemma 4 is that even at the higher end of its size range, it\u2019s still decently performant on personal hardware. Google claims this is due to innovations in the architecture of the model, but the proof is in the trying. Gemma 4 is quite responsive.<\/p>\n

To that end, I took Gemma 4 for a spin on my own hardware to see how it fared for its advertised tasks.<\/p>\n

\"\"<\/figure>\n

Each of these model sizes is available in a slew of community-created editions, thanks to Gemma 4\u2019s Apache 2 licensing. For instance, the 26B A4B model comes in a community edition with more compact quantizations (4-bit, 6-bit, etc.), which I used as one of the model mixes for this article.<\/p>\n

I ran each model using my now-standard test bed: LM Studio 0.4.10 on an AMD Ryzen 5 3600 6-core CPU (32GB RAM) and an Nvidia GeForce RTX 5060 (8GB VRAM).<\/p>\n

The 26B model was at the upper end of what I could run comfortably on my test hardware. I wasn\u2019t able to fit the entire model into GPU memory, but I set the first 12 layers to run on the GPU (7.51GB VRAM), and I set the context length to 16384 tokens (total: 18.76GB RAM).<\/p>\n

Getting good performance out of models that don\u2019t fit in VRAM is always a challenge. However, Gemma 4 has, courtesy of its \u201cmixture of experts\u201d design, a feature to boost performance. LM Studio exposes this feature through a setting currently tagged as experimental. You can choose how many layers of the model to \u201cforce MoE [Mixture of Experts] weights onto the CPU,\u201d which conserves VRAM and can speed up inference.<\/p>\n

The MoE (mixture of experts) experimental setting in LM Studio. For models that use an MoE design, this setting forces the weights for that aspect of the model to be run on the CPU instead of the GPU. With Gemma 4, this resulted in a major speed boost for models too big to fit in memory.<\/p>\n

Without the MoE forcing, the overall inference time and token generation speed cratered; the model could barely manage an average of 1.5 tokens per second even for simple queries. With MoE forcing turned on (with the maximum number of layers supported, 30), token generation speed jumped to anywhere from 5 to 13 tokens per second, depending on the rest of the system\u2019s load. That\u2019s still a far cry from the speed of the smaller models, but a lot more workable.<\/p>\n

\"\"<\/figure>\n

For faster time-to-first-token results, you can disable thinking, at the possible cost of less robust output. For the code-generation query, Gemma 4 spent 6 minutes 26 seconds thinking, and over 8 minutes generating the response (5,013 tokens, 9.55 tokens per second). The resulting code and explanation was not significantly more advanced or detailed than the non-thinking version.<\/p>\n

Response from Gemma 4\u2019s 26B parameter model to a query to generate code. This larger version of the model runs less quickly when it can\u2019t fit entirely in memory, but its mixture-of-experts design helped offset that limitation.<\/p>\n

When I switched to the LM Studio Community edition of the E4B model, I put all 42 layers on the GPU and kept the context at 16,384, all of which fit comfortably in VRAM with room to spare. The results were a major jump in speed: 72 tokens per second. The smaller model was less specific for certain queries \u2014 the code-generation query in particular didn\u2019t generate a comprehensive code example, only a conceptual framework for one \u2014 but still did a decent job of analyzing the problem and suggesting constructive approaches. The \u201cunsloth\u201d edition of the E4B model, despite being slightly smaller, was about as performant and useful.<\/p>\n

Examples of Gemma 4\u2019s 26B parameter version generating image captions. The smaller versions of the model tended not to editorialize. The larger version sometimes needed specific guidance to be less verbose or florid.<\/p>\n

For the \u201cmake this program more modular\u201d prompt, I got roughly equivalent results across all incarnations of the model in terms of the advice given. The only major difference was that the smaller models ran far faster \u2014 73.85 and 71.73 tokens per second vs. 9.3 for the big model.<\/p>\n

The biggest takeaway from running Gemma 4 locally is how the mix-of-experts design in one of the larger incarnations of the model make it useful even on systems where the model doesn\u2019t fit entirely into VRAM. The smaller incarnations of the model, even at lower quantizations, still work well, too. They also deliver results many times faster, and free up much more memory for larger context windows. Thus, the smaller models are well worth experimenting with as the first model of choice before moving up to their bigger brothers.<\/p>\n