Page Title

Why This Matters Personal data is both the backbone of one of AI’s biggest unlocks - providing personalized solutions at scale - one of its biggest issues - creating security risks for the data given to it. and also an emerging security risk for communities that are often already over-surveiled or under-served. Technology has generally improved the delivery of social impact. AI is unique in that it’s level of force-multiplication is unprecedented, but the risks are greater than ever. For communities already over-surveilled or under-served, the stakes are higher: a leak or misuse can compromise safety and privacy. The social sector should follow the lead of federal regulators who are already tightening rules accordingly: AI’s progression is seemingly defined by its shift from technology being reactive to proactive. This is evidenced by the emerging “agentic” era, where AI can make decisions and take actions across tools. This further creates immense potential and risks ([1]). For an AI solution to actually expand equity — quicker service delivery, improved accessibility, or contextually-aware customizations — privacy and security must be embedded foundationally, not bolted on later. Federal regulators are already tightening rules accordingly. The FTC updated COPPA to protect children by preventing their data from being shared without a separate opt-in and stronger retention/deletion safeguards. ([2]). The HHS requires concrete safeguards (access controls, encryption, audits) when AI systems touch protected health information. ([3]). The wheel doesn’t need to be reinvented. In fact, the public sector has resources that can be adapted such as NIST frameworks and federal AI governance memos that all point to the same idea: optimize useful data use and minimize harms through consent, minimization, and continuous risk management. ([4])

Why This Matters Deploying AI at scale is expensive and difficult. Any good AI solution should be able to clearly demonstrate that AI is the optimal way to solve a real, timely problem - rather than simply following the “shiny new toy”. Those building human-centered solutions, have a responsibility to the communities they serve to maximize every dollar and resource. While AI can be a great enabler in doing so, chasing hype by forcing AI to solve a problem could lead to overlooking simpler, higher quality solutions. A needs assessment demonstrating clear value for AI over alternative solutions is table stakes for a human-centered AI solution’s sustainability.For a human-centered AI solution to be sustainable, a needs assessment clearly demonstrating the value of AI over alternative solutions is an essential prerequisite. The market reality backs this up: many AI pilots don’t reach production because of unclear value population, and some analysts warn that over 40% of “agentic AI” projects may be scrapped by 2027 due to unclear value and rising costs. ([7]) The strongest AI projects are those tackling problems that are already urgent and costly. If the challenge wouldn’t justify effort or investment on its own, layering AI is unlikely to change that. Especially when serving people who are facing current barriers, close a real gap now and have a credible path to value measure soon.

Why This Matters Believing that AI is the right tool for a problem is one thing; proving it is another. Another significant technological shift with AI is that identical inputs can lead to vastly different outputs. That’s an inherent feature of the tech that is further complicated by it’s potential to deliver biased results, hallucinate false information, and, especially with solutions built on foundation models, experience sudden changes to capabilities (also know as model drift). General outcome metrics like ‘number of users served’ aren’t enough, because they don’t show what the AI itself is adding. Good AI-specific metrics isolate whether the technology is responsible for the change. Measures like time saved, error rates, and subgroup equity outcomes are more reliable signals in early pilots as they point directly to the AI’s contribution rather than just overall activity. Without a clear AI evaluation plan, teams risk confusing correlation with causation. Positive outcomes may be attributed to AI when they actually stem from other factors (e.g. staff, funding or even luck), thereby overlooking limitations of their AI integration and tanking their scalability or sustainability. This can also be a huge issue for equity. Accenture (2024) and Stanford CRFM (2023) both found that some GenAI pilots that looked “successful” at the top line actually created uneven outcomes when you broke down results by subgroup, meaning average numbers hid widening disparities. For AI to help marginalized communities, there needs to be proof that it actually narrows gaps, not just moves numbers around. Before deployment, HCAI solutions should define intended benefits with metrics and have mechanisms in place to monitor for and respond to degradation.

Why This Matters Just because AI is efficient, does not mean that it is cheap. It’s true that prototyping is quicker, cheaper and more accessible that ever, but real-world usage includes various costs. These include API calls to foundation models, high data storage fees, and the need for experts to scale, monitor and fix issues. AI usage often grows faster than expected as it’s cost structure represents yet another departure from traditional technology. AI solutions incur costs every time they send a prompt to a foundation model. Even as unit prices keep falling (the cost to get GPT-3.5-level answers fell ~280× between Nov 2022 and Oct 2024), yet total bills still rise as organizations scale usage and adopt more powerful models. ([9]) Finance and engineering teams are already flagging this. The percentage of organizations actively managing AI spend and building cost visibility has more than doubled in the last year ([10]). Organizations that rent AI cloud compute see an average of ~40% year-over-year growth in usage as they experiment and scale.[11] While cost scaling with tech deployment isn’t new, the slope for AI can much be steeper than typical software because AI bills rise with every prompt, longer contexts, and more powerful models—even as unit prices fall. Bottom line: if a solution doesn’t show its total cost of ownership (TCO)—build and run— there’s a high chance that a promising HCAI demo can’t be sustained.

Why This Matters For HCAI, clear timelines protect people from broken promises and endless pilots that never reach them. AI demos often look “almost ready,” but much of the hardest work is the process of converting demos to production. This includes things like policy reviews, data access and security checks, staff training, workflow integrations, and monitoring setup. Without a calendar-backed plan for those steps, pilots drift—burning time, budgets, and trust while communities wait. Some analysts even warn of a potential pilot purgatory and predict that over 40% of “agentic AI” projects may be canceled by 2027 because timelines don't allow value to be shown or cost to be controlled fast enough.[8][8] Though there’s room for nuance, HCAI solutions should prioritize quality over speed when they’re in tension. Timelines that account for the most common blockers (workforce capacity, policy compliance, and up-to-date use policies [15]) force teams to do the unglamorous work up front which may delay a product launch, in exchange for assurance that the tool can be used safely and consistently. That work is also critical to making a solution scalable, as tackling those problems upfront are much easier than retroactively adjusting a large, deployed system. GAO’s recent reviews echo this: workforce capacity, policy compliance, and up-to-date use policies are common blockers that timelines must account for explicitly. Timelines aren’t just important to launch - AI should also be treated as a living system that must be monitored continuously and de-commissioned when risk exceeds tolerance, preventing potential harm created by open-ended experiments.[16][16] Timelines are also accountability for funders. Clear start date, mid-point checks, and decision date (scale, adjust, or stop) force teams to show evidence (e.g., reduced wait times and no widening equity gaps) before expanding impact claims.

Why This Matters Building has never been more accessible due to AI’s ability to democratize expertise and work efficiently. But it’s much easier to sketch a great AI idea than to run one. Most HCAI don’t fail due to the AI’s functionality; but rather because of the difficulty of launching and sustaining the tool with the necessary people, data, and systems. [17]. People: There are justified concerns over job displacement stemming from AI. It’s true that AI is already replacing a lot of processed work (CITE), but effective HCAI solutions depend on humans more than ever. Deploying automated solutions in high-risk contexts requires human oversight and lived expertise. Data: AI is only as good as the data given to it. The best model infrastructure can’t help AI systems if its data, which is essentially its source of truth, isn’t treated as high priority. Feasibility hinges on data that is high correct, complete, and representative of all users Systems: HCAI are never “just” AI - they require necessary infrastructure to ensure that the tool is secure, monitorable, and trust worthy. Just as a beautiful home requires plumbing, AI requires supportive infrastructure. Implementability begets equity. If an MVP doesn’t fit the languages, accessibility needs or other context of intended beneficiaries, the benefits go to the easiest-to-reach users while everyone else waits, often exacerbating divides. Feasibility for HCAI solutions is all about fit whether that’s with real users, real data or real workflows.

Why This Matters As mentioned throughout this framework, AI isn’t static. Model drift is the real phenomenon that AI that worked yesterday can slip tomorrow, often by vendor-made decisions. When building on top of foundational models, as most do, users feel those shifts even if code hasn’t been touched. That’s risky in human-impact contexts: small changes can redefine functionality and raise error rates or widen equity gaps unless. It’s critical that HCAI solutions have mechanisms to be on top of potential changes and are able to correct course. Public guidance even assumes that change is the default/ Explainability is critical to resilience. Being able to see sources, reasoning, or uncertainty empowers people to understand why the AI responded as it did. For users, this furthers trust by facilitating their ability to notice shifts, flag issues, and choose safe alternatives. Internally, it allows teams to move quickly by pinpointing where issues are stemming from. Change resilience can be complex, but following compliance guidelines can kill two birds with one stone. Not only will it protect an HCAI solution from legal (and often ethical) risks, but it also promotes trustworthy design. Early compliance is a feature - it reduces approval friction, signals reliability to partners, and becomes a real differentiator as the AI landscape keeps shifting.

Why This Matters AI is primarily known for changing how work gets done, but it also changes the nature of control. AI can proactively make decisions, prescribe advice and trigger follow up steps. What will never move to the machine is responsibility. If an AI creates a negative outcome, the humans who decided to deploy it are responsible. When no one is able to answer for what the system does or no one is watching, the tool is viewed as unaccountable and dangerous. HCAI solutions are especially complex. A generic B2B AI product can underwrite a small percentage of errors as a business cost. An HCAI solution could be immediately viewed as untrustworthy or dangerous even from one unideal outcome. While the primary incentive for safety is to avoid harm, setting up the infrastructure for accountability and oversight helps an HCAI solution avoid eroding trust, weakening consent, and inviting reputational and ethical risks. Accountability and oversight are the primary protectors for users and legitimacy. They clarify who owns decisions, who explains them, and who answers when things go wrong. They prevent users from feeling locked out into an opaque result. There’s internal incentives too. Accountability as a feature reduces friction with partners and speeds up review processes. Oversight is a vehicle for learning as incidences become signals on what to improve to better guide product decisions. The benefits of clear ownership and transparency are distributed. They tell communities, funders and regulators that the organization intends to be answerable over time. It orients AI to keep it in service of people, builds trust when uncertainty is unavoidable and sustains a secure foundation needed to scale.

Why This Matters The most important part of an HCAI solution is not the AI; it’s people who will use it. A technically strong project will fail if people do not trust it. Legitimacy doesn’t come from scoring high on evaluations; adoption hinges on users feeling seen, respected and empowered. Public sentiment around AI generally ranges from cautious to fearful. Half of US adults are more concerned than excited about AI, primarily citing societal risks. [21] . Trust patterns vary by age or geography [22], so trust can even vary just by who you’re trying to serve. Lived expertise is not an add-on; it’s core to quality. Technical wins often miss real workflows, constraints or other cultural contexts. Early, genuine involvement of those potentially affected by a system improves contextual fit and acceptance. As shown throughout the framework, transparency and explainability are core attributes of making AI trustworthy as they empower people to feel in control of the tools they’re using.