Relevance & Urgency Test
Evaluates whether AI is genuinely the best solution for a real, pressing problem - not just following hype.
Deploying AI at scale is expensive and difficult. If the problem wouldn't justify significant investment without AI, adding AI won't change that. Many AI pilots fail because they're solutions looking for problems, or because simpler alternatives (better workflows, other tools) would work better and cost less. The strongest projects tackle problems that are already urgent and costly, with clear evidence that AI's specific capabilities - throughput, pattern recognition, or personalization - make it the optimal choice.
What Good Looks Like
✓ Concrete, quantified problem statement with baseline data (e.g., "median wait time is 41 days")
✓ Clear identification of who is most affected, especially marginalized groups
✓ Documented needs assessment conducted with affected communities
✓ Analysis of simpler alternatives with evidence for why they won't work
✓ Clear reasoning for why AI's specific capabilities (not just "innovation") are necessary
✓ Scan of existing tools/solutions with explanation of differentiation
✓ Evidence the problem is urgent and costly enough to justify AI investment
✓ Baseline comparison showing AI would be faster, more accurate, or more accessible than status quo
What to Watch Out For
✗ Can't explain why AI is better than simpler alternatives (improved workflows, better staffing)
✗ No quantifiable problem statement (vague claims like "wait times are too long")
✗ No evidence that affected communities actually want or need this solution
✗ Building something that already exists elsewhere without clear differentiation
✗ Timeline suggests rushing to deploy without proper needs validation
✗ Problem defined by outsiders without community input
✗ Using AI because it's trendy, not because it's the optimal solution
Tests To Apply
□ Is there a concrete, quantified problem statement with baseline data?
□ Have they identified who is most affected and validated the problem with those communities?
□ Have they documented why simpler alternatives won't work (with evidence, not assumptions)?
□ Did they conduct a needs assessment with representative users from affected communities?
□ Have they identified nearby tools/solutions and explained why theirs is needed?
□ Is there a human baseline comparison (e.g., "10% faster than human support")?
□ Are there pre-defined metrics that would prove the problem was misdiagnosed?
□ Is the problem urgent and costly enough to justify AI's expense and complexity?
Key Questions to Ask
-
What's the quantified gap you're addressing and who's most affected?
-
What alternatives did you consider and why did you choose AI over simpler solutions?
-
How did you validate this need with the community you're serving?
-
What would success look like in 90 days, and what would cause you to stop?
-
If this weren't an AI solution, would you still invest in solving this problem?
Apply the Cross-Cutting Lenses
After evaluating the core criteria above, apply these two additional lenses to assess equity outcomes and evidence quality.
Equity & Safety Check
When evaluating Relevance & Urgency through the equity and safety lens, assess whether the problem being solved actually matters to the communities being served and whether rushing to deploy could cause harm.
Gate Assessment:
🟢 CONTINUE: Problem validated by community, clear evidence AI is optimal approach, safety timeline prioritized
🟡 ADJUST: Some community validation, but gaps in understanding most affected groups, needs refinement
🔴 STOP: Problem defined without community input, or speed prioritized over safety for vulnerable users
Check for:
□ Was the problem definition co-created with affected communities, not assumed by outsiders?
□ Are urgency claims based on needs of the most marginalized users, not just the easiest to serve?
□ Could rushing deployment to capture "first mover advantage" bypass necessary safety testing?
□ Are alternative solutions (that might be simpler/safer) being dismissed because they're not "innovative"?
□ Is there evidence the AI approach reduces harm compared to current state (not just maintains status quo)?
□ Are rollback triggers defined if the urgent problem turns out to be misdiagnosed?
Evidence & Uncertainty Check
When evaluating Relevance & Urgency through the evidence and uncertainty lens, assess whether the "need" is backed by data and whether simpler solutions have been properly ruled out.
Quality Grade:
🅰️ A (Strong): Quantified problem with baseline data, documented alternatives analysis, community-validated need
🅱️ B (Moderate): Problem articulated with some data, alternatives considered but not rigorously tested
🅲 C (Weak): Vague problem statement, no baseline data, "we should use AI because it's cool" - do not fund
Check for:
□ Is there quantified baseline data showing the problem's scale and who's most affected?
□ Have they documented why simpler alternatives won't work (with evidence, not assumptions)?
□ Is there a needs assessment conducted with representative users, not just convenient samples?
□ Do they acknowledge uncertainty about whether AI is actually necessary vs. better workflows/more staff?
□ Have they identified nearby tools/solutions and explained why theirs is needed (not duplicative)?
□ Are there pre-defined metrics that would prove the problem was misdiagnosed?