This article explains a set-up I have found useful for LLM applications using a locally-hosted small-language model. My wife runs a popular events calendar for Baroque music in Bavaria, and we use AI to find and post events. A local LLM is fine for this purpose, since we define the workflow in terms of narrow tasks.

I run small LLMs on a separate Linux server equipped with 64 GB RAM, a 20-core Intel i7, and an Nvidia RTX 4060 Ti with 16 GB VRAM and 4352 CUDA cores. The set-up described below allows me to use the models hosted there from a laptop or separate development workstation, and makes for a flexible arrangement.

A couple of the challenges running AI applications at scale are managing cost, and ensuring content is compliant. LLM calls and token costs can rapidly spin out of control, and prompts are sent to the LLMs without filtering, a potential risk with external providers. Go’s standard libraries provide sophisticated networking capability, including an elegant reverse proxy implementation. This seemed like a good fit for these problems.

In this post, we will use reverse proxy to create an endpoint for LLM calls, that intercepts and inspects incoming prompts, before forwarding these largely unchanged to the LLM provider if they pass inspection, or block them if not. For those that pass through, the LLM reply is again intercepted by the same reverse proxy, and token counts are extracted from the LLM completions. These counts can be aggregated, and high-usage accounts can be quickly flagged and even blocked.

In this post, we combine the fast MCP server from the last post with the near-instantaneous queries on Parquet files from the post before that. This approach is not only fast, but good for data sovereignty and scale. With only 250 lines of code, we get a lightning-fast MCP server that puts Apache DataFusion behind a tool call, so an LLM can do analytics on enterprise-scale Parquet data without a notebook, a warehouse bill, or a single row of raw data entering the context window.

MCP (Model Context Protocol) servers, which make data and analytics discoverable and accessible by large language models, are built using the JSON RPC protocol, and can thus be written in Rust (or most other languages) with the standard HTTP libraries. But rmcp has become the standard way to do this, and I wanted to explore how this works, and how it can expose the fast analytics made possible by Rust. This entry shows a basic MCP server that makes it possible to get a file system directory listing, and to get some attributes of one of those files.

I first got into AI in the late 1980s while at Air Canada. At that time, AI research and applications were dominated by the languages LISP and Prolog. I experimented with both, and was fascinated with Prolog’s ability to solve problems that you defined as facts, and rules. During a recent visit to Bletchley Park north of London, I reflected on how this ability could have helped the tedious code-breaking efforts there, had it been available, and picked up a book of logic puzzles in the shop, with the intent of solving them using AI and Prolog.