The topic I stopped uploading my notes to ChatGPT and NotebookLM after setting up my local LLMs is currently the subject of lively discussion — readers and analysts are keeping a close eye on developments.
This is taking place in a dynamic environment: companies’ decisions and competitors’ reactions can quickly change the picture.
Pairing everyday productivity applications with AI models is one of the best ways to bolster their capabilities. Note takers, for example, can utilize large language models to manage all your ideas, and even generate new ones from your existing knowledge base. In most cases, you’ll find AI-based note-taking workflows relying on cloud-based LLMs.
But despite their superior reasoning capabilities and easy access, I steer clear of cloud models when it comes to managing something as private as my note collection. Instead, I use locally-deployed LLMs to handle all inference tasks for my note-taking needs. With the right collection of FOSS applications, embedding models, and good ol’ LLMs, I haven’t had any issues moving my personal knowledge management system off the cloud.
I built a local AI setup out of two old GPUs that sell for cheap, and it beats a single new card
Let’s start with what really makes AI models a solid upgrade for note-takers. Conventional LLMs are good at summarizing documents, sure, but you’ll need a specialized type of model to skim through pages upon pages of notes and retrieving the right information. Specifically, I’m talking about embedding models, which convert typical notes into a vector map, where text (or even images/audio, for that matter) with similar contexts are grouped together, while unrelated data gets scattered far apart.
Embedding models, in turn, make way for Retrieval-Augmented Generation workflows on AI-supported note-taking apps. Every time I ask my RAG-powered apps about a specific note, it funnels the query through the embedding model, which generates its own vector and compares it with the one for my note collection to find all relevant ideas. These text snippets are tossed at the LLM, and the model uses them to create a context-aware answer.
The best part? Typical embedding models don’t need top-of-the-line GPUs to run. Nomic Embed Text meshes well with most of my RAG-powered tools, and the V1.5 variant of this model is light enough to run on a mere Raspberry Pi. Of course, there’s Qwen3-Embedding (8B) for folks who require maximum precision, but low-parameter embedding models work without any issues with typical note takers.
As for the general-purpose LLM, I rely entirely on mixture-of-experts models these days. Since these LLMs can run the attention parts and router mechanism on the VRAM while offloading the lesser-used expert aspects to the CPU and system memory, they can fit even on VRAM-constrained GPUs without causing massive drops in performance. For my note-takers, I use a Gemma-4-26B-A4B that runs on my GTX at 15 tokens/second, though I sometimes swap to GPT-OSS-20B (or even to the RTX 3080 Ti-powered Qwen3.6-35B-A3B powering my coding tasks) when I want faster inference speeds.
Llama.cpp is an open-source framework that runs large language models locally on your computer.
Just like my inference engines, I also rely on locally-deployed note-takers to avoid the same privacy risks as their cloud counterparts. Obsidian with the right plugins is a solid option, but I prefer Blinko to manage my ideas, and it’s mostly due to the way the latter handles notes. Rather than forcing me to plan out my directories in advance before I can even jot down my thoughts, this FOSS app lets me use a custom note format called “blinko” where I can hastily add ideas and go over them later.
I typically use the blinko format for most of my ideas, be it bullet points summarizing meetings, quick lecture notes, or just random fleeting thoughts. Once I’m done ruminating on them, I switch them to the notes format and use my LLMs to auto-tag them. But since Blinko supports RAG functionality, it serves as a personal knowledge management vault where my LLMs can retrieve information from.
That way, I get precise, personalized answers sourced from my note library instead of just the data my LLMs were trained on. Since my MoE models generate precise responses to my queries, I often turn some of the AI-generated answers into their own notes, thereby adding even more information to my knowledge base.
Although I rely on Blinko for most of my notes, I can’t end this article without mentioning my Open Notebook workflow. Since it’s essentially a self-hosted variant of NotebookLM, I hoard all the PDFs, blog posts, and custom notes I’d need for academics and research within Open Notebook.
Similar to Blinko, Open Notebook also supports RAG search provisions, and it’s even more useful here because my AI models can sift through dozens of sources in addition to my ideas before generating relevant answers. Since Open Notebook can turn my documentation into summarized podcasts, I’ve armed it with text-to-speech and speech-to-text models via Speaches, and they work pretty darn well for a completely local pipeline.