# Chinese tokenization impact on retrieval: notes from a Shenzhen bench
Contents
# Framing “Chinese tokenization impact on retrieval”
Test your splitter on Chinese PDFs; do not assume English defaults. Local knowledge bases fail in predictable ways: outdated documents outrank current ones, tables get flattened into nonsense, and exact codes never match because embeddings blur them. Each failure has a simple, boring fix.
# Fix the documents
Assign an owner for each collection. Store a document date and version as required fields. Convert tables to structured text or keep them as separate records instead of letting a paragraph splitter cut them in half. Remove duplicates so the same paragraph does not dominate results.
# Fix retrieval
Use hybrid retrieval: keyword search for identifiers and rare terms, vectors for meaning. Filter by metadata before ranking, for example by department, product line or validity date. Return a few passages with their sources so users can check the answer themselves.
# Fix expectations
The assistant should summarise and point to sources, not invent policy. Publish a short list of things it will not do: legal interpretation, medical advice, payroll decisions. Show the source document name and date with every answer, and make it easy to report a wrong one.
# A two-week pilot
- Pick one team and one document collection.
- Write thirty real questions with expected sources.
- Run the pilot, collect feedback, and fix ingestion issues first.
- Only then expand to more collections or users.
# Questions for your pilot users
- Did the answer cite the document you expected, and was it the current version?
- Was anything important missing that you know exists in the documents?
- Did the assistant refuse when it should have, and answer when it should have?
- How long did it take to verify the answer using the cited source?
Collect these answers in a simple form during the pilot. Patterns in the feedback usually point to ingestion or retrieval problems, which are cheaper to fix than switching models.
Further reading from MicroAI One: local AI tutorials, and MicroAI One software.
# Bottom line
Local AI rewards patience and record keeping more than raw spending. Decide what the system must do, measure it on your own hardware and documents, write down every change with a date and an owner, and review the setup after each significant upgrade. Small, documented steps are easier to debug, easier to hand over and easier to defend when someone asks why a particular choice was made. When in doubt, test on a smaller scale first and expand only when the evidence supports it.
# Quick reminders
- Keep a printed fallback for offline demos.
- Publish what the assistant will refuse to do.
- Leave memory headroom for the OS and the embedding model.
- Use one models directory shared by every runtime.
Published by MicroAI One (本地AI 1号网), a Shenzhen team working on local LLM hardware, software and deployment.