# SSH hardening for inference boxes: lessons from real setups

# Making “SSH hardening for inference boxes” work inside a company

Keys only, fail2ban or equivalent, no password root. Internal assistants built on local models succeed when they are treated like any other production service: owned, monitored and documented. They fail when they remain a demo that slowly turns into a dependency.

# Ownership

Name one owner for the service and one backup. The owner approves model changes, maintains the runbook and answers questions from security and finance. Without a name, issues bounce between IT and the team that started the pilot.

# Boundaries

Write down what the assistant does and does not do. Summaries, drafting and search over approved documents are typical. Posting journal entries, approving payments, interpreting contracts or giving medical and legal advice should stay with people. When a request crosses a boundary, the assistant should say so plainly and offer the manual path.

# Security basics

• Separate system instructions from user input and from retrieved text.
• Treat tool results and uploaded files as untrusted.
• Keep audit logs in a place users cannot edit.
• Review findings from internal testing with named owners and deadlines.

# Monitoring

Watch the things that cause outages: GPU temperature, disk space, process crashes and queue length. Alert on those, not on every individual error. Publish a simple status page so colleagues know when the service is degraded.

# Reviews

# A worked example

A professional services firm set up a local assistant for drafting proposals. Before launch they classified data into three levels and allowed only the first two. Each consultant received a personal key, logs recorded the caller and model version, and an output filter blocked client names from appearing in drafts meant for other clients. A short internal test found that pasted documents could carry instructions that the model followed, so they separated retrieved text from instructions and added a review step. The launch was slower than a demo, but the security team signed off and the service has an owner who answers questions.

More on this topic: custom local-AI builds. See also private deployment services.

# Bottom line

Local AI rewards patience and record keeping more than raw spending. Decide what the system must do, measure it on your own hardware and documents, write down every change with a date and an owner, and review the setup after each significant upgrade. Small, documented steps are easier to debug, easier to hand over and easier to defend when someone asks why a particular choice was made. When in doubt, test on a smaller scale first and expand only when the evidence supports it.

# Quick reminders

• Measure on your own prompts before quoting any speed.
• Photograph cable routing before closing the case.
• Log the model name and version with every request.
• Write the date and owner on every change record.

Published by MicroAI One (本地AI 1号网), a Shenzhen team working on local LLM hardware, software and deployment.