Specification · 10
Data preparation for AI
Organising data and documents so they can be used for a defined AI task.
When this is useful
An AI application may need information spread across documents, business systems and shared folders. Some sources overlap or disagree; others lack the context needed to interpret them.
My contribution
I identify the relevant material, define how it should be organised, indexed and checked, and carry out agreed preparation work where appropriate.
Keyword and vector index candidates are joined by passage ID; three of six remain eligible after version, expiry and access checks, with the current policy passage selected.
What it covers
- Sources, intended uses and access requirements
- Duplicates, conflicting versions and authoritative records
- Document structure, metadata and links between records
- Indexes and retrieval rules for finding relevant records and passages
- Preparation steps and checks against the intended task
- Update responsibilities and treatment of outdated material
The output
A preparation specification covering source structure, indexes and retrieval checks, with organised source material where included in the assignment.
Preparing information for an internal assistant
An illustrative preparation plan for answering questions about orders, products and procedures. Each source needs its own treatment.
| Source | Preparation | Check |
|---|---|---|
| Order records | Keep the order ID, current status and status timestamp together. | An answer refers to the correct order and states when its status was updated. |
| Product documents | Link each document to its product and version; separate superseded material. | The answer uses the version that applies to the requested product. |
| Internal procedures | Record the owner, effective date and staff access group. | A user receives only material their access permits, with its source identified. |
| Search indexes | Keep passage IDs, source versions and access fields aligned across keyword and vector indexes. | Retrieved passages are current, permitted and traceable to their source. |
Illustrative example. These are preparation requirements and proposed checks; they do not represent a deployed assistant or verified access controls.
For an enquiry, describe the question and the systems involved.
contact@fiftystork.com ↗