# Rotating keys on an internal LLM endpoint: common mistakes and fixes

# The business side of “Rotating keys on an internal LLM endpoint”

Short-lived tokens; no shared ‘team’ key in chat apps. Private deployment of large language models is attractive for companies that cannot send documents to public APIs. The technology is accessible; the harder part is governance: who may use the system, for what data, and who is accountable when it is wrong.

# Start with data classes

List the kinds of data the system will see: public material, internal documents, customer records, financial data. Decide which classes are allowed, where they are stored, and how long logs are kept. Write this on one page and get it signed by whoever owns compliance. Local inference does not remove the need for a policy; it makes the policy enforceable.

# Access and keys

• Give each person or service its own key; avoid a shared team key pasted in chat.
• Rotate keys on a calendar and revoke them on offboarding the same day.
• Keep SSH access to inference hosts key-only, with a documented break-glass procedure.
• Log the caller, model version and timestamp for every request.

# Guardrails and testing

Put simple input and output filters in front of the main model, version their rules, and test them. Run a light red-team exercise before each launch: prompt injection through documents, attempts to extract other users’ data, and requests the assistant should refuse. Tools that can write or send things should start disabled and be added only after a read-only phase.

# Cost and delivery

Explain costs to finance with measured numbers: hardware, power from a plug meter, admin time and evaluation effort, compared with a realistic cloud scenario. A delivery package for a private deployment should include the hardware bill of materials, a runbook, evaluation results and an access matrix.

# Questions leadership will ask

• What data does the system see, and where is it stored?
• Who can use it, and how is access removed when someone leaves?
• What happens when the answer is wrong, and who is responsible?
• What does it cost per month, including power and staff time?
• How do we switch it off quickly if something goes wrong?

Prepare short, factual answers before the first meeting. A one-page summary with an owner, a data policy and a kill switch usually moves the conversation from “is it safe” to “where do we start”.

More on this topic: MicroAI One software. See also private deployment services.

# Bottom line

Local AI rewards patience and record keeping more than raw spending. Decide what the system must do, measure it on your own hardware and documents, write down every change with a date and an owner, and review the setup after each significant upgrade. Small, documented steps are easier to debug, easier to hand over and easier to defend when someone asks why a particular choice was made. When in doubt, test on a smaller scale first and expand only when the evidence supports it.

# Quick reminders

• Cap concurrent sessions before lowering quantisation.
• Read the model licence before commercial use.
• Test a restore, not just a backup.
• Review the setup again after every major model upgrade.

Published by MicroAI One (本地AI 1号网), a Shenzhen team working on local LLM hardware, software and deployment.