Most mature content management systems can satisfy a long list of familiar requirements. Personalisation, approvals, experimentation, integrations and multi-channel delivery appear across vendor pages and comparison tables. Score enough platforms against enough features and several will finish within a few points of each other.
The resulting paralysis is a problem with the evaluation method. Feature breadth tells you what a platform can do under some conditions. It says much less about whether your organisation can operate it, govern it and afford it over the life of the investment.
The more useful question is fit: fit with the people who publish, the developers and partners who will support them, the systems around the CMS, the organisation’s risk obligations and the likely path out of the platform one day.
This article sets out an eight-part matrix and a four-stage evaluation. It complements our separate guidance on the stakeholder and governance side of platform selection. One deals with what to evaluate; the other deals with who decides and how.
Treat the demonstration as a prepared sample
A vendor demonstration shows the platform in favourable conditions: organised content, known journeys, expert configuration and a presenter who knows every step. That is appropriate for an introduction. It is weak evidence of ordinary use.
The show-home analogy from the source article still holds. Everything looks coherent while the rooms are staged. Move in with your own furniture, constraints and daily mess, and the experience changes.
In one platform selection we encountered after contract signature, an impressive personalisation demonstration had helped shape the decision. The client’s marketing team later discovered that maintaining comparable rules required technical support they had not planned for. The capability existed. The operating model needed to use it did not.
That distinction should govern the evaluation. For every attractive feature, ask who configures it, how often, using which skills, with what support and at what ongoing cost. A tick in a grid becomes useful only when it is connected to a real task and an owner.
Eight criteria for organisational fit
These criteria deliberately mix technical, operational and commercial concerns. Set any absolute requirements first. A platform that fails a mandatory security, accessibility, regulatory or architectural condition should not survive because it performs well elsewhere.
1. Total cost over the intended life
Model more than the licence. Include discovery, implementation, content work, migration, integrations, hosting, support, training, upgrades, specialist skills and a reasonable allowance for change. Include the eventual cost of extracting content and moving again.
The period should match the organisation’s decision horizon rather than an arbitrary five-year rule. Use ranges where estimates are uncertain and show which assumptions drive the result. A precise total built on invented certainty is less useful than a range whose dependencies are visible.
2. Editorial usability for the people doing the work
Ask actual editors to complete representative tasks: create a complex page, reuse an approved component, schedule a change, obtain approval, correct an error and update content across more than one location.
Observe where they need technical help. The goal is not a frictionless interface under every condition. It is an operating experience that fits the volume, skills and governance of the publishing team. If ordinary changes require a support ticket, price that dependence and its effect on speed.
3. Developer flexibility within your delivery model
“Flexible” has meaning only in relation to the people available. An architecture may offer extensive control while demanding skills the organisation neither employs nor plans to retain. Another may constrain deep customisation while making common work safer and faster.
Evaluate the development workflow, testing, deployment, observability, documentation and availability of skills. Include the organisation’s appetite for owning custom code. Technical possibility is different from maintainable flexibility.
4. Integration with specific systems and processes
“Has an API” is the start of a question. Examine the integrations that matter: what data moves, in which direction, how identities and permissions are handled, how failures are detected and who supports the connection after launch.
Test the highest-risk integration with realistic data and volumes. Check versioning, rate limits, authentication, documentation and responsibility across suppliers. A named connector may reduce effort, although its presence does not guarantee that it supports your required workflow.
5. Product and supplier direction
Assess the supplier’s product investment, roadmap, ownership, support model and treatment of previous changes. The purpose is risk judgement, not a prediction that any vendor will remain unchanged.
Ask how customers are notified of deprecations, what support periods apply, how pricing has evolved and how much of the platform depends on acquired or third-party services. Record the evidence and uncertainty. Our Replatform Reckoning guide covers end-of-life risk in more detail.
6. Hosting and governance fit
Map the hosting model against the organisation’s security, resilience, privacy, records, procurement and regulatory needs. Establish responsibility for patching, monitoring, incidents, recovery, subcontractors and data location.
Regulated organisations should have the relevant legal, compliance, risk and security specialists determine the applicable requirements. A platform evaluation can organise their evidence. It should not convert a general vendor assurance into a compliance conclusion.
7. Exit and migration conditions
Every platform decision creates a future migration problem. Examine how content, assets, metadata, relationships, redirects and audit information can be exported. Identify proprietary structures and presentation logic that may be difficult to reproduce elsewhere.
The aim is not to make exit effortless. It is to understand lock-in, preserve choices where they matter and avoid discovering at the next migration that the organisation cannot recover essential content in a usable form.
8. The implementation and support ecosystem
Evaluate the partners as carefully as the product. Relevant questions include experience with organisations of comparable complexity, continuity of the proposed team, capability across content and change as well as engineering, and the quality of support after release.
Also consider concentration risk. A small partner ecosystem may be acceptable when the organisation has strong internal capability. It carries a different risk when almost all knowledge will sit with one supplier.
Weight before vendors shape the conversation
Weighting turns a generic list into an organisational decision. A smaller firm with limited technical capacity may place greater emphasis on editorial usability, support and predictable cost. A larger organisation with a dedicated platform team may give more weight to integration, developer workflow and multi-site governance.
Agree the weighting before demonstrations. Record the reasoning and distinguish three types of criterion:
- mandatory conditions, where failure removes the option;
- weighted criteria, where relative performance matters; and
- differentiators, which matter only after the fundamentals are satisfied.
Weights should be revisable when new evidence changes the underlying need. They should not move merely to make a preferred product win. Keep the original decision record and explain every later change.
The downloadable eight-criteria CMS decision matrix includes weighted scoring, longlist and shortlist views, and a section for proof-of-concept success criteria. It is vendor-neutral and can be completed without specialist platform knowledge, although technical, security and commercial specialists should contribute where their judgement is required.
Move through four evidence stages
Stage 1: Longlist
Use published information and existing organisational knowledge to identify plausible options and remove those that fail clear mandatory conditions. Four to six candidates is often manageable, though the number should reflect the market and complexity of the brief.
Record unknowns instead of filling them with assumptions. The longlist is a screening step rather than an early attempt to pick a winner.
Stage 2: Shortlist
Score the remaining platforms using documentation, contracts, product information, independent analysis and relevant case material. Ask vendors the same core questions so the evidence is comparable.
Identify where scores depend on claims that still need testing. Two platforms finishing close together is an acceptable result. It shows which uncertainties the next stage must resolve.
Stage 3: Proof of concept
Run a bounded test with your content, roles, integrations and delivery team. Define the questions, evidence and stop date before work begins. A proof of concept that expands into a miniature implementation increases cost without necessarily improving the decision.
Test the conditions most likely to reveal poor fit. That may be a difficult content model, a permission workflow, an integration failure or an editor completing an urgent change without a developer. The glossy path from the demonstration has already been shown.
Stage 4: References and due diligence
Speak with organisations whose complexity and operating model resemble yours. Vendor references are useful, and independently identified customers can broaden the picture. Ask what required more effort than expected, how the supplier responded to problems, what skills remained necessary after launch and what the customer would change in its selection process.
Complete commercial, security, legal and financial due diligence at the level appropriate to the investment. Reference calls add experience; they do not replace those controls.
Make the rationale durable
Three choices create disproportionate regret: allowing presentation quality to dominate evidence, selecting a platform before evaluating its delivery ecosystem, and comparing first-year prices without the surrounding costs.
A defensible decision does not guarantee a painless implementation. It shows that the organisation agreed what mattered before vendor influence, tested the riskiest assumptions in realistic conditions and understood the material trade-offs at the time.
That record also helps the next phase. The implementation team inherits the constraints, success measures and unresolved risks rather than starting again from a winning sales proposal.
If you want support applying the matrix, you can book a platform evaluation workshop. Distinction facilitates these evaluations, so we have a commercial interest in that option. The matrix is equally capable of supporting a well-governed internal process. The aim is a platform your organisation can live with after the demonstration team has left.
