A couple of the challenges running AI applications at scale are managing cost, and ensuring content is compliant. LLM calls and token costs can rapidly spin out of control, and prompts are sent to the LLMs without filtering, a potential risk with external providers. Go’s standard libraries provide sophisticated networking capability, including an elegant reverse proxy implementation. This seemed like a good fit for these problems.

In this post, we will use reverse proxy to create an endpoint for LLM calls, that intercepts and inspects incoming prompts, before forwarding these largely unchanged to the LLM provider if they pass inspection, or block them if not. For those that pass through, the LLM reply is again intercepted by the same reverse proxy, and token counts are extracted from the LLM completions. These counts can be aggregated, and high-usage accounts can be quickly flagged and even blocked.