AI deserves a fair hearing and a sceptical one. It can support useful work in digital product development. It can also generate convincing errors, obscure responsibility and consume budget in search of a problem.
The most important early distinction is between using AI to help a product team and putting AI into the product itself.
An internal tool may help summarise research, draft test cases or suggest code under review. An in-product system may recommend an action, generate client-facing content or make a prediction using user data. The second creates a new service behaviour and a different level of risk, evidence and governance.
Do not let a general claim about “AI in product development” blur those two decisions.
Where AI can earn a place in the team
Research synthesis and discovery support
AI tools can cluster comments, propose themes, summarise long material and help a researcher explore a large evidence set. They should not replace direct access to participants or the researcher’s judgement about context, contradiction and power.
Test synthesis against a human-coded sample. Preserve links from a theme to the source material so the team can inspect how it was formed. Remove or protect personal, confidential and commercially sensitive information according to the applicable rules and supplier arrangements.
A fluent summary can erase minority views or make an early hypothesis look settled. Use it to widen questions and reduce clerical effort, not to manufacture certainty.
Prototyping and design exploration
Generative tools can create variations, draft interface copy and help teams make an idea tangible. Speed is valuable when the purpose is learning.
Label generated material and review it for accessibility, brand, legal, cultural and factual issues before it reaches users. Avoid testing a prototype that appears more complete than the underlying service. A convincing interface can cause stakeholders to approve assumptions that no system or operating process can fulfil.
Engineering assistance
Code assistants can suggest implementations, explanations, tests and refactoring. Their output remains code the organisation is responsible for.
Apply the same or stronger review, testing, security scanning and dependency controls used for human-written code. Establish which repositories and data may be sent to a supplier, how output provenance and licensing are handled, and when generated code is prohibited. Productivity should be measured across delivery and maintenance, including review and correction, rather than by lines produced.
Quality and test support
AI can propose test cases, identify unusual patterns in logs or help prioritise defects. It does not automatically reduce human error. A model may miss the same edge case repeatedly, generate invalid tests or flood the team with low-value findings.
Evaluate precision, recall or another task-appropriate measure against known examples. Keep deterministic tests for critical requirements. Use people to investigate material findings and assess user consequence.
Where AI changes the product
Search, recommendation and assistance
An AI-supported search or recommendation can help users navigate complex information. Define what it is allowed to retrieve, how sources are shown, how freshness is controlled and what happens when confidence is low.
Recommendation quality is not only click-through. Measure whether the result helps the intended task, whether important options are systematically hidden and whether the system creates harmful or unfair differences between users.
Generation and transformation
A product may draft text, summarise documents, classify requests or extract structured information. Design for verification. Show source material where possible, preserve uncertainty and prevent an output from entering a consequential workflow without the required review.
The source article’s strongest warning remains useful: poor data does not necessarily make an AI system cautious. It can produce a confident, plausible and wrong answer. The same applies to incomplete context and ambiguous instructions.
Adaptive experiences
Personalisation can alter content, sequence or assistance based on behaviour and attributes. Begin with a user need that cannot be met more simply. Collecting additional data or inferring sensitive interests to rearrange a page may create more risk than value.
Give users appropriate control and explanation. Test performance across relevant groups and contexts. An experience that adapts successfully for frequent users may confuse a new or infrequent user.
The challenges that decide whether value survives
Data fitness
Assess relevance, provenance, quality, representativeness, permission, freshness and security. More data is not automatically better. The minimum suitable data can reduce exposure and make evaluation easier.
Document which data the model learned from where that is known, which data enters the live system and which output is stored. Supplier claims should be checked against contracts and technical configuration.
Bias and uneven harm
Historical data can reproduce existing inequity. A model can also create new differences through labels, thresholds, product design or feedback loops.
Identify affected groups and plausible harms before selecting metrics. Involve people with domain, legal, social and user knowledge. Aggregate performance can conceal severe failure for a smaller group.
Explanation and contestability
The necessary explanation depends on the decision. A product team debugging a low-risk recommendation needs different evidence from a professional relying on an output that affects a client.
Define who must understand what, at which moment, and what they can do when the system is wrong. “A human reviewed it” is weak protection if the reviewer lacks source evidence, time or authority to challenge the output.
Security, privacy and intellectual property
AI changes data flows, attack surfaces and supplier dependencies. Threat modelling should cover prompts, retrieved content, model output, user abuse, data leakage and changes to the underlying service. Legal and privacy review should address the actual use, jurisdiction, content and relationships involved.
Do not paste confidential client material or proprietary code into a tool merely because it is available through a personal account.
Integration and operation
A successful demonstration may rely on curated cases, close developer support and a fixed model version. Production introduces volume, latency, cost, monitoring, supplier change, edge cases and incidents.
Plan fallback behaviour. If the AI service is unavailable or degrades, can the product continue safely? Define ownership for model and prompt changes, evaluation, complaints and retirement.
Measure the problem before the model
State the current baseline and the expected change. For an internal assistant, measure task quality, completion time, rework, reviewer effort and incidents. For an in-product capability, add user outcomes, unequal performance, reliance, complaints, service cost and possible harm.
An experiment is a legitimate outcome when uncertainty is high. Call it an experiment, define what it can expose and avoid promising production value. A project can also end with evidence that AI adds little beyond a simpler rule, search improvement or process change.
Use a decision gate:
- Is the problem material and well understood?
- Is AI necessary or meaningfully better than a simpler approach?
- Can the team obtain and govern suitable data?
- Can performance and harm be evaluated in realistic conditions?
- Is there a safe operating and fallback model?
NIST’s AI Risk Management Framework resources organise voluntary risk work around governance, mapping, measurement and management. At the time of writing, NIST states that AI RMF 1.0 is being revised, so teams should check the current material and adapt it to their use and obligations.
Start with a bounded decision
Choose a real task and define the evidence required to continue. Test on safely prepared material, with representative cases and qualified review. Include failure cases deliberately. Decide in advance what result leads to expansion, revision or stop.
AI can shorten a cycle, improve access to information or enable a valuable product behaviour. It can also move error faster. The product team’s job is to establish which is happening in this context.
If you want to examine a use case with Distinction, book a short discovery call. We have a commercial interest in AI product work. A useful first conversation should still be capable of concluding that the problem needs better research, data or process rather than an AI feature.
