This article explains a set-up I have found useful for LLM applications using a locally-hosted small-language model. My wife runs a popular events calendar for Baroque music in Bavaria, and we use AI to find and post events. A local LLM is fine for this purpose, since we define the workflow in terms of narrow tasks.

I run small LLMs on a separate Linux server equipped with 64 GB RAM, a 20-core Intel i7, and an Nvidia RTX 4060 Ti with 16 GB VRAM and 4352 CUDA cores. The set-up described below allows me to use the models hosted there from a laptop or separate development workstation, and makes for a flexible arrangement.