A Means to Assess Social Investment Risk – and a Plea on Behalf of the People who Need Nonprofit Organizations to Deliver the Value they Promise

Please login/logout below:

Written by David E. K. Hunter

May 2010

Table of Contents

Summary

This article focuses on the hundred thousand registered nonprofit human service agencies that, annually, have combined budgets totaling $141,215 million. Is society getting the social value such enormous resources should yield? In human terms, are those people who depend on nonprofits to improve their lives and prospects getting the help they need and the results they deserve?

I repeat the observation I made in a previous article in this journal that while nonprofits work incredibly hard, with passion and dedication, and often in extremely difficult circumstances to solve society’s most intractable problems, there is virtually no credible evidence that most nonprofit organizations actually produce any social value. And I address the counter-arguments adduced by those who have criticized this assertion. On the question of my commitment to the nonprofit sector and its organizations, I state unequivocally that ultimately my first loyalty is to the people whom nonprofits serve: the individuals and families who are poor, hungry, sick, disabled, structurally marginalized, and desperate to improve their lives and prospects but who face enormous obstacles in doing so.

Because it seems likely that for the most part nonprofits will not reform themselves without a major push from funders, I argue that the sector needs enlightened social investing, by which I mean the channeling of resources to nonprofits with the measurable objective of supporting the betterment of intolerable social conditions, the melioration of suffering, and the improved well-being of those whose lives and prospects promise little comfort, health, security, safety, or adequate resources. How can a social investor know which nonprofit agencies are likely to be delivering high social value? This article closes with a discussion of a social investment risk assessment tool that my colleagues and I have developed to help nonprofit funders make well-informed choices in selecting nonprofits to support.

Historical Context

In America¹, especially after the Revolution, functions that had once been mostly under the auspices of the church gradually moved to civic organizations and local government. Almshouses were built to house the poor and disabled, formerly private or church-based institutions of higher learning were put under the authority of government, and better-off families were paid to help the less fortunate (in some New England villages, needy families were “auctioned off” to the lowest bidders – that is, to families who would charge the least for harboring them – thus laying the foundation for the current foster care system).

At the same time, government began to rely on organized charities to perform essential things that were beyond its own capacities. For example, the U.S. Sanitary Commission among other things provided health services on battlefields.

In a watershed moment after World War II, America turned with ideological finality away from becoming a European-style welfare state. While the government grew steadily on a rich diet of universal taxation, public policies encouraged the emergence of nonprofit organizations to provide direct services (paid for by public funds) to targeted populations that were deemed both in need and worthy of such services.

^This section synthesizes and summarizes materials in Robbins (2006) and Hall (2006).

The Contemporary Nonprofit Sector in America

Since World War II the number of registered nonprofit organizations (to this day still called “charities”) grew exponentially: from 17,450 in 1943 to 573,265 in 1996 and over 1.9 million today. The so-called “nonprofit sector” was formally created by and codified in the U.S. tax laws wherein nonprofits and their funders are granted tax-exempt status. Presently, the U.S. tax code defines nonprofits as organizations that (a) are incorporated for purposes that meet public needs, (b) are self-governed, and (c) do not distribute profits to officers or shareholders. This article focuses on the first of these functions, holding it as essentially empty and meaningless in the absence of evidence that public needs actually are being met by the nonprofits that claim to do so.

It is usual to divide the nonprofit sector into three categories:

Public charities: While including the nation’s churches and congregations, this category mostly consists of organizations active in the arts, education, health care and human services. It also encompasses grantmaking public charities, also known as public foundations. Public foundations² get their funds from many sources, including private foundations, individuals and families, corporations, and government. Like private foundations, these organizations tend to fund public charities.
Private foundations: Private foundations are endowed and supported by a founding individual, a family or a corporation. Typically, they fund public charities.
Other nonprofits: This catchall category includes social and recreational clubs, trade associations, labor unions, veterans’ organizations and advocacy groups.

The growth of the nonprofit sector – especially in the number of foundations, but certainly also of public charities – is stunning, as shown in Table 1.

Table 1: Size of the U.S. Nonprofit Sector, 1995–2006

Nonprofits	1995(Millions)	2000(Millions)	2006(Millions)	Growth1995–2006
Public charities	0.57	0.69	1.00	75%
Private foundations	0.04	0.06	0.12	300%
Other nonprofits	0.49	0.55	0.45	-8%
All nonprofits	1.10	1.30	1.57^a	43%

Source: National Center for Charitable Statistics a Note that a comparison with Table 2 shows an increase of nonprofit budgets by about $50 million from 2005 to 2006.
^a Note that a comparison with Table 2 shows an increase of nonprofit budgets by about $50 million from 2005 to 2006.

The societal significance of the nonprofit sector becomes even more apparent when we consider the amount of money that it uses. In 2006, nonprofits accounted for 8.11% of all wages and salaries paid in the United States (National Center for Charitable Statistics 2009). And as Table 2 shows, the sector has revenues of over a trillion dollars. Of this amount, the more than 100,000 nonprofit social or human service agencies, which make up 32.3% of the sector, have aggregated annual expenditures that exceed $141 billion.

Table 2: Number and Financial Scope of Reporting Public Charities by Subsector, 2005

Reporting public charities	Number	%	Expenses($ Millions)	%
Arts, culture, humanities	35,840	11.5	23,927	2.3
Education^a	57,991	18.7	158,679	15.1
Environment, animals	13,399	4.3	9,807	0.9
Health^b	41,243	13.3	637,323	60.5
Human services	100,436	32.3	141,215	13.4
International/foreign affairs	5,075	1.6	20,535	1.9
Public and societal benefit^c	37,677	12.1	53,052	5.0
Religion-related	18,600	6.0	8,867	0.8
Unknown/unclassified	422	0.1	83	0.0
All reporting public charities	310,683	100	1,053,487	100

Source: National Center for Charitable Statistics a Including institutions of higher learning, which tend to have large budgets and hence a relatively high share of the sector’s revenues. b Including hospitals, which of necessity have very large budgets and a disproportionate share (considerably more than half) of the sector’s revenues. c Including foundations.
^a Including institutions of higher learning, which tend to have large budgets and hence a relatively high share of the sector’s revenues.
^b Including hospitals, which of necessity have very large budgets and a disproportionate share (considerably more than half) of the sector’s revenues.
^c Including foundations.

So it hardly seems unreasonable to ask: is society getting the “bang” it deserves to get for so many “bucks”?

This article focuses on the human services domain of the nonprofit sector, and builds on the arguments I put forward in “The End of Charity: How to Fix the Nonprofit Sector through Effective Social Investing” (Hunter 2009) published in the first issue of this journal. That article presented “three impolitic, unpleasant truths of which I have become persuaded over the course of my career working with and on behalf of nonprofit organizations” and went on to list them:

Unpleasant truth number 1: While nonprofits work incredibly hard, with passion and dedication, and often in incredibly difficult circumstances to solve society’s most intractable problems, there is virtually no credible evidence that most nonprofit organizations actually produce any social value.

Unpleasant truth number 2: Because so few nonprofits are willing to face this fact and ask themselves whether they are doing any good at all, or even as much good as they may be doing harm, we cannot rely on direct service nonprofits to fix themselves without a serious push.

Unpleasant truth number 3: In general, nonprofits do what their funders tell them to do.

The original article stimulated considerable response, which tended to polarize at both ends of the love–hate continuum. Those who liked the article – including leaders of some of the nation’s highest-performing nonprofits, as well as nationally recognized program evaluators – thanked me for having the courage to say what many of them knew to be true but what there is little public will to face. The responses of those who were angry about the article (especially about my assertion regarding the paucity of evidence that most nonprofits achieve what they promise, i.e., deliver social value) tended to cluster into the following groups:

Dismissal. I was written off, mostly by people who don’t know me, as an armchair pundit who should come down out of his ivory tower and get to know real people and real organizations in the nonprofit sector. I won’t address this other than to invite readers to look over my website at www.dekhconsulting.com and decide for themselves whether I have the bona fides to claim credible knowledge of the sector.
Epistemological outrage. Some critics lectured me to the effect that I am in no position to assert what is true; at best, they thundered, I could assert what I believe is true. Well, I guess it’s my task to draw attention to English usage and point out that “truths of which I have become persuaded” in fact means the same thing as “We hold these truths to be self-evident” – namely, that I (like the authors of our Declaration of Independence) believe these things to be true (and do so with deep conviction).
Demands for proof. While scientifically literate people know that it is inherently impossible to prove a negative (Popper 1957, 1959), nonetheless I was challenged to “prove” my assertion that there is virtually no credible evidence that most nonprofit organizations actually produce any social value. Standing on centuries of scientific practice, I’d say that the burden of proof falls directly on the shoulders of these critics: it is their responsibility to prove such evidence exists. It’s worth remembering that there are over 100,000 nonprofit human service providers in this country. What percentage has rigorous evaluation data proving their programs yield the impacts they promise? How many keep count simply of the unduplicated individuals and families they serve? How many collect outcome data? Anybody who has spent even a small time among human service nonprofits will know the answer: very, very few.³
Anecdotal triumphalism. Several critics, some of whom are leaders of nonprofit intermediaries, took it upon themselves to recount the great contributions that nonprofits have made to our society – indeed, to the world at large. True enough, but what does this prove? Certainly not that my assertion is wrong. It simply shows that there are instances of wonderful things nonprofits have accomplished. Evaluators often say: “No stories without numbers” – that is, we can’t know the significance of a story, of an instance, without knowing how representative it is of the state of things in the world.⁴ Even a thousand such stories wouldn’t invalidate my point, as I happen to know at least some of these critics understand fully (personal communications).
Wild accusations. Finally, some folks worked themselves into such a lather that they accused my assertion of being “destructive” to the nonprofit sector. They argued in a variety of ways that the sector needs tender, loving care; nurturing, not criticism; a gentle hand up, not a kick in the butt. Some of them, while making this argument, did so in spite of accepting what I had to say about the sector. So here, as far as I can tell, we have a disagreement about means rather than about reality. These critics are incrementalists who want to nudge the sector and its organizations toward higher levels of performance tenderly and slowly, while keeping the reality of the sector’s dysfunctions shrouded from the hostile world (including funders!). They are concerned about the fragility of nonprofit organizations, and deeply worried on their behalf. In essence they are saying: The world will stop supporting nonprofits if the truth gets out!

Well, maybe so. But I doubt it. Rather, I think that what would happen is that nonprofits would be supported…differently.

To be clear: While I share these critics’ concerns, and in fact spend all my professional time working as hard and creatively as I know how in order to help (mostly) human service nonprofits become better and more effective at what they do, become organizationally stronger, and become more sustainable, ultimately my first loyalty is to the people whom nonprofits serve: the individuals and families who are poor, hungry, sick, disabled, structurally marginalized, desperate to improve their lives and prospects but who face enormous obstacles in doing so. These people deserve nonprofits that can and do deliver the goods, that reliably provide effective supports and opportunities as promised, which measurably help to improve their skills, knowledge, achievements and attainments. Too many generations of such people have, in my view, been offered false hope and empty promises by nonprofits that work with good intentions but to no effect.

So I find the incrementalist argument fundamentally elitist and socially unacceptable. As Geoffrey Canada, leader of the Harlem Children’s Zone, has been quoted as saying about public education, while American public schools need more money rather than less, it is his “fundamental belief that the folk who care about public education the most, who really want to see it work, are destroying it” (Tough 2008, 131). I take this to mean that Canada is mad as hell and not willing to take it any more, that he is rejecting incrementalism, and that he is calling for creative ways to destroy the education system’s status quo in order to build a system that can deliver on education’s promise to our children. Today. Not tomorrow, next week or month, or next year. Not in a decade. Now. That is exactly what I want for the nonprofit sector in general, for nonprofit social services most particularly, and for the intended beneficiaries of these services most urgently.

^According to the website of the Foundation Center (2010), “A foundation is a non-governmental entity that is established as a nonprofit corporation or a charitable trust, with a principal purpose of making grants to unrelated organizations, institutions, or individuals for scientific, educational, cultural, religious, or other charitable purposes. This broad definition encompasses two foundation types: private foundations and grantmaking public charities.”
^Fortunately, this is changing. But not nearly fast enough, in my view.
^I like to add: “And no numbers without stories.” This is because numbers alone will never elucidate the human condition. We need stories – qualitative data – to understand what the numbers mean.

Why I Advocate for Clearheaded Social Investing over Sentimental Support for Nonprofits

The nonprofit sector was created to meet public needs – not to perpetuate any particular nonprofit organization. To the extent that nonprofits do so meaningfully, they deserve our support. To the extent they don’t, there is a breach of public trust.

In my previous paper in this journal (Hunter 2009) I argued that nonprofits more often than not do what their funders require of them. I am not alone in this view, which is presented persuasively in “The Nonprofit Starvation Cycle” (Gregory and Howard 2009). It is with this conviction – that for the most part nonprofits will not reform themselves without a major push from funders – that I argue for enlightened social investing, by which I mean the channeling of resources to nonprofits with the objective of supporting the betterment of intolerable social conditions, the melioration of suffering, and the improved well-being of those whose lives and prospects promise little comfort, health, security, safety or adequate resources.⁵

While in my previous paper I tried to make the case for social investing as a constructive alternative to other ways of funding nonprofits, here I address the question of how. Specifically, how can one know whether one is making a high-risk social investment (where good social value is likely to be a distant prospect and hence a lot of nonfinancial support is required, in addition to the financial investment, to realize the social return on the investment), whether the risk is moderate, or whether it is low (essentially, amounting to the purchase of immediate social value).

^The choice to focus on human service organizations and private foundations is shaped by our desire to be where the unmet need is greatest, while avoiding higher education and major health care institutions that are well served by funders, consultants and advocates.

Assessing the Risk of a Social Investment

To support rational and pragmatic social investing, my colleagues⁶ and I have developed an instrument that uses clear, easily applied and meaningful metrics to calculate the potential risk and value of an investment in a nonprofit organization. A social investor needs to understand the implications related to making a specific investment. For instance, when making a “blue chip” investment the investor basically “purchases” social value while taking little risk of failure. The investor can make the deal without much subsequent work to ensure that it will bear fruit. However, a “social venture” investment is quite different. Like investing in a commercial start-up, a social venture investment is more risky. Social value is not a sure thing, and perhaps not even likely in the short run. A social venture investment requires the investor to work with, and monitor the development of, the organization to help cultivate it into a high-performer capable of generating social value.

How can one differentiate up front between “blue chip” investments that purchase social value with minimal risk and “social venture” investments that bet on long-term potential? And how can one know when the risk for any kind of investment is simply too high, that the poor prospects of creating social value through it are not worth the money and effort? What makes for a good bet, and what makes for a poor one?

While of course any such decision must rest on the intangibles of good sense and judgment, it seems sensible to find ways to support the selection of nonprofits in which to invest by helping to bring appropriate focus and pertinent information into the process. That is the purpose of the social investment risk assessment tool discussed in the remaining parts of this article.

Basic Assumptions

We have made some key assumptions to create our social investment risk assessment tool (SIRA):⁷

Whereas investing in the commercial sector ultimately must lead to investors’ financial gain and greater shareholder profits, we assume that in the nonprofit sector the fundamental investment objective is the creation of social value – a measurable change for the better (or at the very least preventing a change for the worst) in some domain of the world where neither the commercial sector nor government is doing enough to accomplish what needs to be done.
We also assume that the social investor will support whole organizations (rather than single programs within organizations) and, as the investment dictates, dedicate energy and nonfinancial resources (technical assistance, consultation, etc.) to strengthening them and helping them to deliver their programs and services reliably, sustainably, and at high levels of quality.
Finally, we assume that social investors will be clear on what kinds and levels of social value they are seeking to produce through their investments. For example, an early childhood intervention program results in quite a different set of outcomes than a juvenile justice diversionary program, and investors ought to be clear on such differences and fully conscious of the logic they follow when selecting for investment nonprofit organizations and the programs they run. Our investment instrument will not suggest which outcomes are worth investing in – knowing what the research says about such matters is up to the investor.⁸ Our instrument will inform the next step – selecting among nonprofit organizations that are working for similar outcomes – to make either “blue chip” or “social venture” investments.⁹

^The first draft of the tool was created by me along with Steve Butz (then president of Social Solutions). Numerous experts from various domains of the nonprofit sector reviewed and made substantive contributions to subsequent drafts – for which my colleagues and I are very grateful. I have been the primary author of these subsequent drafts, with contributions from Ingvild Bjornvold (of Social Solutions) and Marie Louise Refsgaard (of Ramboll Management). The latter two participated with me in beta testing the tool, which we now regard as operational. The costs for alpha and beta testing were borne by Hunter Consulting LLC, Social Solutions Inc., and Ramboll Management SA.
^Because this point has been the subject of some confusion, I want to emphasize that the SIRA is designed to assess investment risk – not measure or evaluate the kind of social value produced by an organization. Another way of saying this is that the tool looks at how likely it is that a given organization can deliver the social value it promises – not at the inherent worth of that social value.
^This is parallel with the commercial sector, where an investor must know what kinds of products are likely to have market value before selecting, for investment, an organization that makes such products.
^We recognize that some nonprofit organizations – like trade organizations or learning communities – exist to address or promote the specific needs of members and do not necessarily seek to deliver social value beyond that framework. This paper, and our investment tool, focuses on the vast majority of nonprofit organizations that exist to advance some kind of wider social good.

Introducing the SIRA

But is there a measurable way to assess an organization’s ability to create social value that is meaningful and has utility across all the domains where nonprofit organizations labor? While it remains widely believed that the answer is still “no,” we are convinced that in fact the answer is “yes.” By this we mean that although some of the contents will of necessity shift from one domain to the next, the basic assumptions that inform this tool will remain quite consistent across domains. The remainder of this article presents version 3.0 (23rd revision) of the first of our SIRA tools, designed to be used to assess investment risk in the domain of human services.¹⁰

The SIRA rates three performance domains – tactical data use, strategic data use, and program value – that, in the view of the SIRA’s creators, together characterize the social value of a human service organization’s work. Two indicators define each domain:

1. Tactical Data Use Domain

a. Data Integrity Indicator – the requirement that performance data be accurate, timely and complete.
b. Outcomes Focus Indicator – the requirement that performance data include the collection of client outcome data and that performance management take client outcomes into account.¹¹

2. Strategic Data Use Domain

a. Making Essential Adjustments Indicator – the requirement that performance data be consequential, i.e., that they be used to improve on what is being done, and how work is undertaken, when results are not satisfactory.
b. Relating Staff Efforts to Outcomes Indicator – the requirement that performance management focus on the relationship between what the organization actually does (as opposed to what it claims to do) and what results are achieved.

3. Program Value Domain

a. Capacity to Deliver Program/Services with Fidelity Indicator – the requirement that the organization have well-institutionalized policies, procedures, systems and practices to ensure that the services it delivers conform to its design specifications (implementation standards) and deliver results as expected (performance standards).
b. Evidence of Program Impact Indicator – the requirement that an organization have reasonable evidence that its programs and services produce results (outcomes, impacts) as intended.

To assess these indicators we have developed a short but comprehensive set of simple and straightforward questions that, when answered and scored, produce valid and useful ratings along a 10-point scale of organizations in each of the three performance domains.

Organizations that rate high (7–10 points) in one or more domains are the riskiest investments. That does not mean I am arguing that they should be avoided. But it does mean that a social investor should proceed with open eyes and the understanding that simply giving such organizations money most likely will not generate much in the way of social value. Rather, they will need considerable time and significant nonfinancial support – and a high degree of involvement from investors – to help them improve to the point where they are reliable sources of public good. Some obvious examples would include many beloved grassroots, community-based, neighborhood, and faith-based organizations.¹²

I believe that the intended results of such a high-risk investment should be the development of the organization itself, not (in the foreseeable future) any outcomes it seeks for the people it serves. Hence I term them social venture investments, and would highlight that such investments have the potential for creating enormous social value over time.

Investments in organizations with domain ratings in the medium range (4–6) are somewhat less risky, but most will require no less by way of vigilance, nonfinancial support and investor involvement if they are to mature into high social value institutions offering low-risk investment opportunities. (Again, the opportunity costs are significant and should not be underestimated.) But, then too, such organizations offer investors the chance to roll up their sleeves and help those nonprofits that they find attractive do what is necessary to become high-performing organizations – a real contribution to society when done successfully.

By now it will come as no surprise that I see only those organizations in the low risk range (1–3 points) in all three domains as “blue chip” social investment opportunities, where investors’ dollars are essentially purchasing proven social value and there is little need for further investor efforts. However, even with such organizations there is a need for periodic reassessment. By using the SIRA initially, and at least annually thereafter, investors can usefully track the investment types (“blue chip” to “social venture”) and risk levels across their social investment portfolios, plan the kind and degree of their involvement with the nonprofits in which they invest, and calculate how best to make future investments align with and enhance the social value that they care about.

But to be clear: A nonprofit’s SIRA rating is only a snapshot of the organization at a single point in time. Organizations that are rated as “risky” investments on a given date may, in fact, be on an upward trajectory over a two- or three-year period – a trend that a social venture investor might reasonably expect to see realized. Such organizations are perhaps the best long-run “bets” of the social venture investor, provided they continue to improve and when appropriate undertake rigorous program evaluations.¹³ Of course, in the same line of thinking, a “risky” organization trending the wrong way through periodic use of the assessment tool should be avoided altogether – unless the investment is made very intentionally to arrest or reverse the organization’s decline (in which case there should be a time limit after which no further investments are made if no improvement is evident).

^Specifically this version of SIRA is intended to be used for assessing investment risk with regard to human service organizations that are working to make changes in people’s lives – i.e., produce outcomes or impacts. We are working on a more simple tool to assess investment risk for human services (such as soup kitchens and homeless shelters) that are concerned principally with providing aid to people in need without the expectation that this will change their lives much beyond the moment – that is, the focus is on outputs (not outcomes), and the risk factor to be assessed is one of service quality (not resulting changes).
^The two indicators of the Tactical Data Use Domain are not entirely independent. In our view, performance management data that do not include client outcome data are inherently incomplete. But we believe the second indicator is necessary as a way to rate the degree to which an organization uses outcome data as a way to think about, learn from, and manage performance
^Social investors must face the ethical conundrum that any such investment has an inherently high opportunity cost for program participants – research and development projects (of which such investments are an example) of necessity divert clients who need help away from organizations that could help them immediately and with greater likelihood of success.
^The issue of when it makes sense to undertake both formative (implementation) and summative (impact) program evaluations is of great importance to the nonprofit sector, and deserves to be considered deeply in its own right. I have addressed this matter elsewhere (Hunter 2006).

The Intellectual Foundation of the SIRA

There are many people who believe that it is impossible to use metrics when assessing inherently qualitative phenomena. And surely “social investment risk” is fundamentally a qualitative matter, one that involves judgments grounded in assumptions about the world and how it works, and practical experiences gained from working in the world. Yet there is a field of research that insists on the utility of using quantitative methods to solve qualitative problems. It is called clinical biostatistics (Feinstein 1977) and it has become a bedrock of today’s medicine, where we know that practice entails the applied conjoining of art and science. Clinical biostatistics insists that both the practitioner and the statistician must contribute to the understanding of qualitative phenomena such as the relative contributions of ball players to their teams’ success, how first responders should proceed in disasters, what constitutes high quality in youth programming or – in the matter under discussion here – how risky it is to invest in a given nonprofit if one cares about the social value that likely will result. It rejects the protests of practitioners who claim that their organizations’ work is too nuanced, precious and ephemeral to submit to systematic (often declaimed as reductionistic) assessment. (And it equally rejects the arrogance of quantifiers who think all important things can be reduced to numbers alone.)

The SIRA tool is not a means to rate the investment risk of a nonprofit organization in a mechanistic manner. The ratings it produces are not intended ever to be dispositive mechanistically, in their own right. But just as physicians can diagnose certain conditions more accurately when assisted by computers – if the computer rating systems have been constructed with meaningful contributions by clinical experts, as the SIRA was – so too, I believe, social investors will be able to make much better decisions if they pay serious attention to the ratings that SIRA will produce.

SIRA Indicators and Some of the Questions Used to Assess Them

The entire SIRA tool is available both on my personal website and that of the Alliance for Effective Social Investing – http://alleffedtive.org – and I hope interested readers will review it in detail. In fact, my collaborators and I invite nonprofits to use the tool at no charge as a way to analyze their organizations and diagnose how to become less of a risk for social investors interested in getting a measurable social return on their investments.¹⁴

Table 3 presents a sampling of some of the thirty items that are used to rate each of the six indicators.¹⁵ For the purpose of producing ratings that can be used reliably to inform social investing, the ratings should be made by trained analysts based on a series of on-site interviews and a review of key documents.

Table 3: Sample SIRA Items and the Indicators They Rate (marked with x)^a

Rating Items	1. Tactical Data Use		2. Strategic Data Use		3. Program Value
	Indicator a. Data Integrity	Indicator b. Outcomes Focus	Indicator a. Making Essential Adjustments	Indicator b. Relating Staff Efforts to Outcomes	Indicator a. Capacity to Deliver Program/Servi-ces with Fidelity	Indicator b. Evidence of Program Impact
The organization’s core programming consists of one or more programs or an array of services intended to achieve measurable outcomes or impacts for intended beneficiaries.		x			x	x
The organization monitors those outcomes for clients in its core programming that it believes are produced by staff activities/services (including negative outcomes when these become apparent).		x		x	x
For its core programming, the organization monitors how much time (hours and minutes) each client spends participating in programming or using services intended to produce outcomes or impacts over the course of a week.	x			x	x
In the past year, the organization has made a significant change (e.g., restructuring, changes in personnel, adding/eliminating program[s], program redesign, change[s] to service frequency/intensity) due to knowledge it gained from positive and/or negative performance data it collected and analyzed.		x	x	x	x
In its core programming, the organization has implemented systems and processes for assuring operational adherence to original program design.				x	x	x
The organization’s front-line staff members are held accountable for how well their efforts succeed in helping clients achieve targeted outcomes.		x	x	x	x
The organization’s front-line staff and their supervisors review aggregated client outcome data in documented meetings to manage performance.	x	x		x	x
The percent of core (outcome-producing) program participants that have received a full dosage of programming/services necessary for them to achieve key outcomes – as called for by the theory of change – is ___.					x
The organization averages _______ percent front-line staff turnover annually.					x
The organization’s theory of change has which of the following characteristics. It is: *Plausible* (i.e., it has face validity, it “makes sense” to the informed reviewer) *Doable* (i.e., it can be executed fully within available resources) *Measurable* (i.e., its various key elements can be monitored using operational data – qualitative and quantitative)*Testable* (i.e., its implementation standards, performance standards, and success criteria support program management and external evaluation)					x	x
The evidentiary basis for the organization’s practicesb is best described in the following way:— The organization’s core program model(s) as a whole has/have been proven effective by one or more impact evaluations (done by one or more other organizations using this/these model(s), or by the present organization) in one or more scientifically valid experimental or quasi-experimental evaluations. —The program model(s) incorporate(s) elements taken from other, similar programs that have been proven valid by research done in one or more scientifically valid impact evaluations(s) using experimental					x	x

^a Note that most questions contribute to the ratings of more than one indicator, because performance data can be used in a variety of ways. So too can an awareness on the part of management and front-line staff of the evidence of a program’s effectiveness in generating participant outcomes (e.g., motivating a more rigorous approach to working with each service recipient).
^b This may refer to the practices of one program for a single-service agency, or those in multiple programs for a multi-service agency
^c Implementation standards are specifications regarding how a program or service must be delivered: where, by whom, how often, for how long, and using what activities.
^d Performance standards are specifications of the organization’s expectations in relation to the rate(s) of achieving success – e.g., the number and percent of program participants or service recipients who achieve each targeted outcome.

At a minimum, the ratings should be based on interviews with the following people:

executive director and chief operating officer
person directly in charge of all programming or services
person in charge of reporting to funders (typically director of quality assurance/performance management or development director)
each site director (for all sites – unless there are so many sites that a significant sampling is a reasonable alternative)
each program director (at all sites or a sample of sites)
one direct service staff member from each program (at all sites or a sample of sites)

The analyst should also review and take into account information derived from each of the following documents:

the most recent strategy that has been adopted by the board of directors
written, operational versions of the organization’s mission, goals and objectives
the most recent operational theory of change
any existing program logic models that are operational
all external evaluation reports
a sample of current, internal performance reports used in the management of each site and each program – including but not limited to unduplicated counts of participants served, participant demographics, service utilization by participants, activities/efforts delivered by staff to participants, quality indicator reports, etc.
relevant personnel policies and procedures that pertain to:
- job descriptions
- performance evaluations and compensation
- service quality management
- professional accountability
- data entry
- data use in supervision and program management
- data integrity standards and processes for assuring data integrity
a staff performance review form

^We have tested both individual questions and the tool as a whole with a large number of nonprofit organizations, the leaders of which have agreed that the resulting assessments fairly capture the state of their organizations and the level of investment risk they constitute – that is, they affirmed the validity of the tool. We hope that others will use it as a springboard for refining their thoughts about how best to assess the ability of social service agencies to deliver solid social value reliably, measurably, and sustainably.
^The scoring system and rating instructions are omitted here, but are part of the tool itself. This version of the SIRA is designed to be used by trained analysts who are informed about qualitative assessment and its methodological requirements.

The End of Charity

I will end this article by repeating what I said at the end of my previous article (Hunter 2009) because, in spite of all the controversy it created, I still hold to its basic premises and do so with a deep conviction that social investing done well holds more promise than any other means to promote the improvement of the nonprofit sector and therefore the lives and prospects of the people its agencies serve.

The transition from charitable giving to intentional social investing has major implications, some of which admittedly are a cause for discomfort and concern to many people and organizations in the nonprofit sector.

Perhaps easiest for them to accept is the hope (and for some investors even the expectation, yet to be proven true) that social investing will promote the cultivation and growth of high-performing nonprofit social service organizations. Less comfortable, doubtless, are the intended corollaries. Social investing, if widely adopted, will help channel funding streams that are directed by measurable performance rather than feel-good stories, habits of giving and rank sentimentality. And social investing has the potential (yet to be realized) to advance a selection process that either forces poor performers to evolve and improve, or weeds them out.

The bottom line is that social investing, if it succeeds, offers the potential to reduce the enormous cost to society of funding and sustaining organizations that are not high-performers and cannot justify their claims that they produce the social value promised in their mission statements. It means that scarce resources will be better spent, and people using social services will more likely benefit and achieve what they have been promised (emphasis added).

The nonprofit sector is essential to our nation’s future. It doesn’t need apologists. It needs critical friends. I count myself among the latter, and hope that this article and the social investment risk assessment tool it discusses will contribute constructively to efforts to bring clearheaded and sober thought, predictability of funding streams, and accountability for performance to a sector where these attributes long have been kept at bay by sentimentality and good intentions.

References

Feinstein, A. R. (1977). Clinical Biostatistics. St. Louis: Mosby.

Foundation Center. (2010). Frequently asked questions: What is a foundation? Available at http://foundationcenter.org/getstarted/faqs/html/foundfun.html (accessed April 22, 2010).

Gregory, A. G. G., and D. Howard. (2009). The Nonprofit Starvation Cycle. Stanford Social Innovation Review, Fall.

Hall, P.D. (2006). A Historical Overview of Philanthropy, Voluntary Associations, and Nonprofit Organizations in the United States, 1600–2000. In W. W. Powell and R. Steinberg (Eds.), The Non-Profit Sector: A Research Handbook (2nd ed.). New Haven and London: Yale University Press, 32–65.

Hunter, D. E. K. (2006). Daniel and the Rhinoceros. Evaluation and Program Planning 29(2), 180–185.

Hunter, D. E. K. (2009). The End of Charity: How to Fix the Nonprofit Sector through Effective Social Investing. Philadelphia Social Innovations Journal, October.

National Center for Charitable Statistics. (2009). Statistics: Quick facts about nonprofits. Available at http://nccs.urban.org/statistics/quickfacts.cfm (accessed April 22, 2010).

Popper, K. (1957). The Poverty of Historicism. London: Routledge.

Popper, K. (1959). The Logic of Scientific Discovery. London: Routledge.

Robbins, K. C. (2006). The Nonprofit Sector in Historical Perspective: Traditions of Philanthropy in the West. In W. W. Powell and R. Steinberg (Eds.), The Non-Profit Sector: A Research Handbook (2nd ed.). New Haven and London: Yale University Press, 13–31.

Tough, P. (2008). Whatever It Takes: Geoffrey Canada’s Quest to Change Harlem and America. New York: Houghton Mifflin Harcourt.

Comments (0) Add Comment