Collect everyday work
Evaluating AI requires a set of real work tasks: routine requests, difficult cases and questions the source data cannot answer. An acceptable result is defined in advance for each case. Model versions can then be compared under the same conditions.
Define a correct result first
Imagine an assistant handling shop enquiries. Correctness includes identifying the problem, finding the appropriate returns policy, avoiding excessive promises and choosing an authorised next step. Combining everything into a judgement that an answer is good makes fixes harder. A retrieval error will not be solved by better writing. Criteria should reflect actions that actually follow the answer.
Collect ordinary requests, borderline cases and situations with insufficient data. Record expected behaviour, not necessarily one ideal sentence. An assistant may correctly request an order number or refer a case to a specialist. Such answers should not automatically rank below confident resolutions, or the evaluation itself will encourage invention and excessive autonomy.
Do not confuse tuning with an exam
Some examples can improve instructions and retrieval. Others should remain independent tests after changes. If a team continually tunes for one familiar set, the result demonstrates familiarity with that set rather than readiness for real work. Add new pilot errors and periodically refresh the evaluation sample while retaining a change history.
Repeat important cases and record consistency. One successful answer does not establish behaviour under slightly different wording. Compare the same document and tool versions, or the cause of changed results becomes unclear. A record needs the task, configuration version, response, assessment and reason for failure, without unnecessary personal data.
Evaluate quality alongside operation
A system may become more accurate while making employees wait and read excessive text. Alongside quality, assess time to a usable answer, manual corrections and processing cost. Separate errors by consequence: awkward wording and an incorrect external operation require different responses. Averages must not conceal rare but serious failures.
