The International Journal
of Not-for-Profit Law
Volume 8, Issue 3, May 2006
By Douglas Rutzen and Michelle Coulton1
The Voluntary Sector Initiative (“VSI”) is a five-year, $94.6 million initiative designed to strengthen the voluntary sector’s capacity to engage in policy dialogue and to fortify the relationship between the voluntary sector and the federal government. The VSI included a number of components, including amendments to the legal and regulatory framework under which the voluntary sector operates. Social Development Canada has invited the International Center for Not-for-Profit Law (ICNL)2 to prepare this paper, which discusses possible methodologies to assess the impact of these legal and regulatory reforms.
This paper is written from a practitioner’s perspective. ICNL has worked in more than 90 countries to improve the legal environment in which the voluntary sector operates. As part of this work, ICNL regularly engages in projects to assess the impact of legal and regulatory reform. For example, ICNL has assisted with the development and implementation of the World Bank’s ARVIN Framework,3 the implementation of USAID’s NGO Sustainability Index in Europe and Eurasia,4 and the refinement of the legal/regulatory component of CIVICUS’s Civil Society Index.5 ICNL is also working with partners in countries from Turkey to Mexico to assess the impact of prospective and enacted legislative reform.
Part 1 of this paper provides a brief introduction to various Civil Society Indices. These initiatives provide a useful conceptual framework to assess the relationship between the enabling environment and the development of the voluntary sector. Indices are also a useful tool to determine whether the VSI’s legal and regulatory reforms are generally enabling or disabling. At the same time, we recognize that they are not specifically designed to provide data on the practical impact of certain reforms that have received considerable attention within the voluntary sector in Canada (e.g., increased due diligence requirements for large donations, intermediate sanctions, etc.). To inform discussions relating to these provisions, we build on existing models and suggest an evaluation methodology to assess the micro, meso, and macro-level impact of specific VSI reforms.
A Brief Introduction to Indices
This section explores Johns Hopkins University’s Global Civil Society Index,6 CIVICUS’ Civil Society Index,7 and USAID’s NGO Sustainability Index.8 Volumes have been published on these Indices, and it would be impossible to describe these Indices in a comprehensive fashion in this short paper. We therefore provide an introduction to these Indices below and refer readers to other publications for additional detail.
The Johns Hopkins University Global Civil Society Index
The Johns Hopkins Comparative Nonprofit Project published its latest “Global Civil Society Index” (“GCSI”) in 2004. The GCSI assesses civil society along three dimensions:
- capacity, or the level of effort or activity mobilized by the sector9;
- sustainability, or the ability to secure resources to operate over time10; and
- impact, or the contribution of civil society to social, economic and political life.11
The legal environment is encompassed under the concept of “sustainability” and assessed using a specially designed Civil Society Legal Environment Scale.12 This Scale applies the concept of transaction costs to analyze legislative provisions.13 Specifically, the Scale identifies and examines provisions that affect the “demand” for civil society organizations and the trust that individuals have in organizations (e.g., rules that prohibit the distribution of profit, establish internal governance requirements, impose reporting and transparency requirements, etc.).14 The Scale then examines provisions that affect the ease with which organizations can form and operate – thus affecting the “supply” of civil society organizations. Examples include provisions affecting registration/incorporation requirements, minimum capitalization requirements, and tax/fiscal benefits for organizations and their donors.15
The GCSI utilizes a coding system and detailed scoring worksheets. Scores are integers between 0 and 2, and are assigned based on memoranda developed by legal experts. Scores are then validated by local experts and summed to form the “demand” and “supply” scores for each country.16 To normalize the different indicators for composite scoring, Johns Hopkins uses a scale that rates each country on its percentage of the maximum value achieved by any country for that indicator.
The GCSI recognizes that the law as written may diverge from the law as applied.17 To address this issue, the GCSI draws on governance indicators developed by the World Bank. Specifically, the GCSI uses an average of two indicators developed by the Bank:
- the “government effectiveness” index, which measures the capacity of a government to enforce legislation; and
- the “rule of law” index, which measures compliance and enforcement of governing legislation.18
The GSCI then “weights” de jure provisions by this composite measure of de facto operation of the law to derive a more nuanced measure of the legal and regulatory framework for civil society in a particular country.
In summary, the GCSI provides a useful conceptual framework for considering the legal and regulatory factors that affect civil society. It also identifies twenty-four key features of the legal and regulatory framework, which adds further rigor to the analysis.
The CIVICUS Civil Society Index
The CIVICUS Civil Society Index (CSI) assesses the state of civil society along four dimensions:
- The structure of civil society;
- The external environment in which civil society exists and functions;
- The values practiced and promoted in the civil society arena; and
- The impact of activities pursued by civil society actors.19
These dimensions comprise twenty-five sub-dimensions. Most relevant to this study, the “external environment” contains the following factors:
- Political context;
- Basic freedoms and rights;
- Socio-economic context;
- Legal environment;
- State-civil society relations; and
- Private sector-civil society relations.20
These sub-dimensions are then measured according to a number of indicators. For the “legal environment,” the CSI provides four main indicators:
- CSO Registration: How supportive is the CSO registration process? Is the process simple, quick, inexpensive, following legal provisions, and consistently applied?
- Allowable Advocacy Activities: To what extent are CSOs free to engage in advocacy and criticize the government?
- Tax Laws Favorable to CSOs: How favorable is the tax system to CSOs? How narrow/broad is the range of CSOs that are eligible for tax exemptions, if any? How significant are these exemptions?
- Tax Benefits for Philanthropy: How broadly available are tax deductions or credits, or other tax benefits, to encourage individual and corporate giving?21
Other sub-dimensions are also relevant. For example, components under state-civil society relations include:
- Autonomy: To what extent are CSOs free to operate without excessive government interference? Is government oversight reasonably designed and limited to protect legitimate public interests?
- Dialogue: To what extent does the state dialogue with civil society? How inclusive and institutionalized are the terms and rules of engagement, if they exist?
- Cooperation/Support: How narrow/broad is the range of CSOs that receive state grants?
The CSI uses a variety of data collection methods depending on the indicator. Methods include: regional stakeholder consultations, individual questionnaires, group discussions, community surveys, a review of appropriate media, and fact-finding. Central to this process is the National Index Team (NIT), which selects a diverse group of stakeholders for the National Advisory Group (NAG). The NIT compiles secondary data for review by the NAG, decides on the definition of “civil society” for use in the country, adapts the proposed methodology to include indicators appropriate to this definition and the country, and contextualizes civil society within the broader society by considering other society actors and power relationships. All findings are submitted to the civil society expert who then prepares a draft country report. Using this report and the scoring guidelines, the NAG then meets to assign scores to the indicators. These are aggregated into sub-dimension and dimension scores, which, along with the draft country report, are reviewed at a national workshop. The actors and stakeholders at this workshop analyze the findings and propose plans of action for strengthening problem areas. All of this information is published as part of the final country report.22
CIVICUS uses a 0-3 scale. Similar to the GCSI, the CSI also provides guidelines on the key attributes of various indicator scores. Ratings for each indicator are then averaged for each sub-dimension. Indicators are scored on an integer scale, but other numerical values, for example the sub-dimension scores, are averages and may therefore be expressed as decimals. Sub-dimensions are averaged and placed into four dimensions. The dimension scores are then plotted, producing the “Civil Society Diamond.”23
USAID’s NGO Sustainability Index
The United States Agency for International Development (USAID) initiated the NGO Sustainability Index in 1997. It has become an annual publication (now implemented by Management Systems International and ICNL under a contract with USAID), providing data on the “strength and continued viability” of the nonprofit sector in Central and Eastern Europe and the countries of the former Soviet Union.24
The Sustainability Index assesses sustainability along seven dimensions: legal environment, organizational capacity, financial viability, advocacy, service provision, infrastructure, and public image. Each dimension is then subdivided into sub-dimensions with corresponding indicators. For example, the legal environment comprises seven components categorized under the following headings, with illustrative questions presented below:
- Registration: Is a favorable registration law in place, and are NGOs easily able to register in practice?
- Operation: Are there appropriate rules governing internal management, permissible activities, financial reporting, and government oversight?
- Administrative Impediments and State Harassment: Are NGOs free from harassment by the central government, local government, and tax police?
- Local Legal Capacity: Is competent legal advice available to NGOs, particularly outside of the capital?
- Taxation: Are there organizational tax exemptions and tax incentives for philanthropy?
- Earned Income: Does the law allow NGOs to generate income from the sale of goods and services?
USAID, either directly or through a contracted party depending on the country, convenes a panel of local stakeholders to score various components. USAID encourages a cross-sectoral panel, including government officials, NGO representatives and others. Moreover, an effort is made to include representatives from outside of the capital city, and some panels calculate scores for different parts of the country.
Scores are calculated on a seven-point scale to facilitate comparisons to Indices developed by Freedom House. This is the most expansive scoring scale of the three Indices; Johns Hopkins provides three scoring options (0, 1, and 2), by contrast, and CIVICUS provides four (0, 1, 2, 3). Also, the NGO Sustainability Index not only allows but also encourages decimal scoring. Scores are then averaged to determine a dimension score.
The NGO Sustainability Index provides scoring guidelines identifying the key attributes of each integer along the seven point scale. These are general guidelines, however, and do not define the key attributes of any particular indicator. For example, the instructions provide the following guidelines for a score of “7”:
NGO sector’s sustainability significantly impeded by practices/policies in this area, generally as a result of an authoritarian government that aggressively opposes the development of independent NGOs.
In contrast, a score of “1” is appropriate when:
NGO sector’s sustainability enhanced significantly by practices/policies in this area. While the needed reforms may not be complete, the local NGO community recognizes which reforms or developments are still needed, and has a plan and the ability to pursue them itself.
The NGO Sustainability Index also identifies key attributes of countries that are in the “consolidation” phase (for scores between 1-3); “mid-transition” phase (for scores between 3-5), and the “early transition” phase (for scores between 5-7).
A narrative report is also produced to support and explain scoring. The narrative reports and scores are then sent to USAID/Washington, which convenes an Editorial Board of experts that reviews reports and scores to help promote comparability across the region. If differences arise, USAID/Washington typically discusses these issues with the local USAID mission and then determines final scores.
Strengths and Limitations of Indices
Indices provide an important conceptual lens through which to view the legal and regulatory environment for civil society. They are a marked improvement over the disparate hunches and estimations about the relationship between law and civil society, which marked prior dialogue on the subject.
It is also interesting that, though significant conceptual differences remain, there is increasing consensus about the key attributes of an enabling environment for civil society. To a greater or lesser degree, all three Indices include indicators concerning the following:
- The legal existence of civic organizations (i.e., registration/incorporation);
- Internal governance;
- Advocacy/lobbying activities;
- Reporting and governmental oversight;
- Incentives for philanthropy; and
- Organizational tax benefits.
Two of the Indices also refer explicitly to the right of civil society to engage in fee-for-service activities and to engage in self-regulatory initiatives.
Of course, there are also some stand-out provisions. Only one Index contains an indicator explicitly relating to protections against arbitrary dissolution by the state. That said, this issue is the downstream component of the right to establish an organization, so this indicator may be implicit in other Indices. In addition, the following provisions are referenced in only one Index: the ability to form unincorporated associations, and the existence of institutionalized forms of engagement between the government and civil society (such as compacts). In a separate forum, it would be interesting to analyze these divergences to see if further consensus can be reached concerning appropriate indicators.
Indices also provide a rich source of data and have provided a quantum leap in understanding the relationship between the law and civil society. Moreover, resource constraints often limit the ability to assess the impact of multifaceted reform packages, and Indices help capture the “essence” of the NGO legal framework. They can help answer the question of whether a reform package was, on the whole, enabling or disabling for civil society. In addition, the use of standardized templates enables comparability across countries and time.
However, we must also recognize the limitations of Indices. Weighting, for example, is a key issue, and has at least two interrelated components. First, existing Indices seem to give equal weight to all sub-dimensions. For example, the USAID NGO Sustainability Index gives equal weighting to state-sanctioned harassment of the sector and the availability of legal services. Although the two issues are obviously related, one could argue that – at least in certain restrictive contexts – state harassment may better reflect the state of civil society than the availability of legal advice.
It is also important to recognize weighting issues implicit in the design of any Index. In other words, the relative weight of a provision may depend on how it is incorporated into an Index. Taking a simple example, the NGO Sustainability Index asks a stand-alone question about “earned income” issues. Accordingly, this issue accounts for one-sixth of the total legal environment score for a country. In the GSCI, however, a subcomponent of this issue is addressed – unrelated business activity – and is one of nine indicators under “Financing,” which in turn is only one of three components of the “supply side” score. As such, earned income issues have a far smaller impact on the total legal environment score under the GSCI than under the NGO Sustainability Index.
Third, while the Indices provide an important conceptual framework and tend to capture the essence of a country’s legal framework, they are less well-equipped to assess the specific impact of incremental change in sophisticated environments, which is a feature of a number of VSI reforms. For example, we understand that operational charities are now subjected to a 3.5 disbursement quota. Perhaps it would be possible to shoehorn this issue under the CSI indicator relating to “capitalization,” but this seems like a stretch.
Even if this conceptual leap can be made, problems arise because the ratings are rather coarse (integer ratings on a scale of 0, 1, 2, or 3), and it is not clear what value should be assigned to intermediate sanctions. Should this account for no change, or for 1/8 of a point, 1/4 of a point, or 1 point? Moreover, is the net change positive or negative? Reasonable people could spend considerable time arguing over such matters.
Our proposition is that for these sorts of provisions, it is not all that useful to engage in lengthy theoretical debates about the numerical value of the reform. Instead, for select reforms, we think it would more productive to shortcut the debate over numerical values (which after all are merely proxies for practical impact) and instead focus on assessing the actual effect of reform. For example, it would be interesting to evaluate this provision’s practical impact – if any – on the activities, funding, and capitalization of operational charities.
Armed with this more detailed information, the government and other stakeholders can make more informed judgments about whether the reform achieved intended objectives, whether unintended consequences were incurred, and whether further reform is necessary. This information would seem more interesting than a bald numerical proxy and is similar to the approach used in the ARVIN assessment tool developed by the World Bank and ICNL.
Of course, designing a system to assess the practical impact of select provisions is easier said than done. The following section seeks to advance the discussion of how such a methodology might be constructed.
Toward an Evaluation of Practical Impact
Before looking at the VSI regulatory reform, program theory relating to voluntary sector regulation merits consideration.25 If Canada’s voluntary sector regulatory regime is understood as a program in itself, of which the VSI reforms are a refining or sub-program, then the reasons or objectives behind Canada’s regulatory framework – together with the linkages between the framework and the attainment of these objectives – may help inform the program theory relating to reform. A full discussion of nonprofit sector regulatory program theory is too great a task to tackle here and for this reason we leave aside the issue of the link between program and outcomes entirely, and touch only briefly on regulation objectives.
Regulatory theory has received surprisingly little recent attention; it merits greater research, particularly in relation to the nonprofit sector.26 Nevertheless, theories can be understood as implicit within, and thus extrapolated from, literature dealing with the sector’s regulation, if not attributable to a particular author. Revisiting the Johns Hopkins Civil Society Legal Environment Scale, splitting the effects of the legal environment on sector sustainability into demand and supply-side impacts suggests a regulatory objective of market correction or balance.27 Applying Le Grand’s theory might suggest that voluntary sector regulation is necessary to ensure that nonprofit actors exhibit knightly rather than a default, knavish, behavior.28 Again, there is an idea of optimizing or balancing: that the sector is not proscribed altogether in this model suggests dual premises: a) the sector is desirable, but b) leaving it unchecked would lead to under-performance or abuse. Different models may offer alternative views about what is being held in balance; likewise, different ideologies would identify differing optimal points along the regulatory continuum. For example, rights-based theories, where regulation seeks to balance people’s competing freedoms and rights, may more naturally give rise to an enabling, light-touch model.29 By contrast, where regulation seeks to maintain the political status quo, the optimal may be more restrictive.
For this reason it is important that Canada satisfies itself as to its regulatory program theory, viewing it as something to be defined, agreed upon, and owned internally. Some theories can be found in the VSI literature but would merit further exploration, such as to “enhance public trust and confidence”;30 to balance “the need to ensure public confidence in voluntary organizations, and the need to ensure a supportive and enabling environment for them”;31 increasing effective performance and credibility;32 and contributing to the integrity of the charitable sector and the social well-being of Canadians.33 For other recent thinking, Canada may wish to consider regulatory program theory from other countries. For example, the UK’s dedicated “Better Regulation Task Force” is stimulating debate around regulatory theory, giving rise to ideas about the purpose of regulation such as to “improve standards & protect rights”34 and to “improve our economic performance and quality of life.”35
Although the design of regulatory reform may be informed by an understanding of the objectives motivating Canada’s voluntary sector regulatory framework and the means by which these are attained, such an understanding may be of limited relevance to the evaluation of the reform. It cannot be assumed that the goals and means for the reforms correlate with those of the original program, because reform may require fine-turning/adjusting a balance. For example, regulation may emphasize ensuring compliance, whereas reform/refinement may emphasize enabling by cutting red-tape (or vice-versa) – an adjustment in the opposite direction along the regulation scale. Therefore, reforms cannot necessarily be evaluated in terms of the overarching regulatory objectives, but must be assessed in terms of their own (albeit related) objectives.
One possible approach for an impact evaluation of the regulatory reforms is outlined as follows:
- Identification of reforms to assess;
- Development of indicators;
- Selection of metrics;
- Data collection, aggregation and validation;
- Contextualization (in terms of outstanding problems – those not addressed; new problems created); and
- Analysis and learning
The following sections unpack each stage sequentially after providing some contextual comments.
Although concerned with the evaluation of democracy assistance programs, Gordon Crawford’s critique of recent evaluation attempts is helpful in informing the question of whether an existing methodology might suitably be used to evaluate particular VSI regulatory reforms.36 Distinguishing two main approaches, he identifies the main weaknesses of logical framework (logframe) or results-based evaluations as too narrow and too focused on quantitative data to take account of wider impacts (e.g., effects beyond a program’s pre-defined objectives and more qualitative changes). Additionally, these evaluations typically fail to take sufficient account of pluralism (different stakeholders may understand problems, objectives, and effects differently, so it matters who does the scoring).37 Moreover, country impact studies often deal inadequately with methodological difficulties (most notably attribution, by making leaps from micro changes to macro conclusions).38 He recommends that to overcome these difficulties, an appropriate methodology would take account of the dynamic context in which an intervention was made,39 link micro changes to macro impacts through a meso stage,40 and be genuinely participatory.41
Identification of reforms to assess
Canada’s reforms can be understood at various levels:
- Micro – individual initiatives (e.g., a shortened tax return; educating the public on how to make a complaint; the introduction of intermediate sanctions)42;
- Meso1 – clusters or themes of reforms, such as the groupings within the JRT final report43 or within Budget 200444;
- Meso2 – voluntary sector regulatory reform package;
- Macro – VSI as a whole, or, alternatively, overall alterations to Canadian legal framework, etc.
Although policy-makers may need to draw broad conclusions at more macro levels (e.g., an appraisal of the VSI’s overall impact may inform a judgment on whether the it was worth the investment), it is perhaps more useful for them to distinguish and understand the different effects (by degree or by design) of different reforms, which contribute to the overall impact.
A comprehensive evaluation of each of the 60 JRT recommendations pursued would be neither feasible nor sensible. We therefore propose that critical or contentious reforms for evaluation be selected through dialogue with key stakeholders. This may help overcome the pluralism problem of different stakeholders with different interests and differing perceptions of reforms’ relative importance.
In addition, as we discuss below under “Contextualization,” we strongly recommend that the evaluation not only identify reforms that were undertaken but also reforms that were rejected. For example, we understand that the VSI did not address issues relating to the definition of charity, issues relating to political activities by charities, and certain challenges relating to cross-border philanthropy.
Development of indicators
A second step is to articulate the problems that the reform was designed to address. It is by reference to problems or intended goals that “improvement” or policy effectiveness may be meaningfully assessed.45 Thus, indicators may be developed to show to what extent a problem was remedied or a goal attained.
A review of literature around the VSI reveals a number of articulated problems and goals.46 One common theme is the goal to improve the Canadian quality of life, although other ultimate objectives are possible. For each reform, it might be possible to describe how it is linked with an ultimate goal by tracing a pathway through a series of sub-goals (i.e., a theory of change).47 For example, shortened tax returns may affect filing rates, which may in turn affect public information levels, thus public trust, thus donations, thus the sector’s financial resources, thus more cost-effective service delivery, thus Canadian quality of life. By taking measurements at the various stages, it is possible to test the theory of change. Sub-goals along the pathway essentially form indicators to show whether the reforms are having the intended effects.
This basic model is complicated by a number of factors:
- Each reform may have multiple effects – designed or unintended, positive or negative (e.g., intermediate sanctions may affect compliance but also increase supply-side transactions costs; in addition, they may increase public trust while straining governmental/sector relations). Theoretically, to understand the full impact of a given reform, it would be necessary to look at various plausible pathways. The conceptual frameworks set forth by various Indices may be helpful in this regard.
- Conversely, several reforms may affect a given (sub)goal (e.g., intermediate sanctions, compliance education, and shortened tax returns might all affect filing rates, which may then affect both “supply-side” and “demand-side” indicators). It would therefore be inaccurate to ascribe a change to just one reform where there is a plausible pathway from several reforms. Clustering reforms according to goals may be helpful in this situation. Several groupings are described in the VSI literature.48 We suggest that some of these clusters may be appropriate, although some reforms may meaningfully appear in more than one cluster.
- More complex still is the existence of external variables, from the wider VSI or beyond (e.g., the economic climate may affect giving, and a particular scandal may affect trust in NPOs). Progressing to higher-order goals increases the number of variables potentially contributing to a discernible change. Effectively, a complex pyramidal network of plausible pathways feed into the ultimate goal, only some of which are traced from the VSI regulatory reforms. Consequently, at higher levels, attribution confidence levels diminish.
Beginning at micro levels and gradually building up permits greater confidence than by making cosmic leaps from the micro to macro levels in the chain of attribution.49 At the same time, it is also useful to test theories of change by measuring macro levels to ensure that higher-level objectives are actually being achieved.
Broad participation should be integral to the process of selecting which reforms to assess, articulating the problems that a given reform sought to address, describing plausible theories of change pathways, and identifying the critical pathways to test. A process of dialogue and negotiation between different groups lends legitimacy to the evaluation, by taking different perceptions of the original problems and critical reforms into account, capturing unintended effects and plausible pathways more comprehensively (including those beyond the sector), and providing wider context.
Selection of metrics
Once indicators have been identified (i.e., points along the plausible pathways), appropriate metrics and data collection methods can be selected. Possible data collection methods could include surveys, key informant interviews, literature reviews, cases studies, and other methodologies, depending on the indicator. Additionally, we offer the following suggestions for selecting metrics.
- Using several metrics per indicator will permit verification through triangulation. The use of both qualitative and quantitative data enables hard evidence to be supplied where possible, while capturing those changes that cannot be easily quantified.
- Despite the difficulties, there is value in collecting data further up the pathways. Although attribution becomes increasingly tenuous, it is by observing changes through to a macro level that the wider impact of the reforms can be understood (i.e., micro changes are aggregated and in some sense contextualized).
- In keeping with the participatory principle, effects on different groups/from different perspectives should be measured, including those beyond the voluntary sector. Again, this method helps to provide a fuller picture of a reform’s likely impact and the context within which it operates. Dialogue may also be appropriate to identify metrics where none immediately presents itself.
- The design of metrics should include metadata to permit disaggregation to trace effects on different groups and sub-sectors (e.g.: by Province, organization size, activity, cultural group, etc). For example, it might be interesting to compare the impact of reform in major cities versus rural areas, and on particular ethnic groups.
- In the case of preventative reforms (such as the due diligence requirement on larger donations), consideration should be given to measuring what did not happen. How to evaluate an absence or negative is beyond the scope of this paper, but we recognize that it may present even greater challenges than assessing those changes that did occur.
Data Collection and Collation
Once the evaluation has been designed, the data can be collected and collated, by feeding them into the hypothetical pathways and observing the magnitudes of change at each stage/indicator. Certain results may be verified by using aspects of the existing Indices.
Comparisons of magnitudes at consecutive stages on a path may suggest stronger or weaker linkages. For example, the average time burden of completing tax returns may be shown to have decreased from a day to an hour, but filing rates may not have increased as much as expected. This evidence may suggest that shortening tax returns was not an effective way to encourage filing returns (i.e., weak link). Although this analysis is oversimplified, by looking at the data within the web of pathways, it may be possible to identify the more likely “cause-effect” type patterns.
“Cross-purposes” of pathways may also be detectable, whereby a reform may contribute to one goal but undermine another (e.g., intermediate sanctions may increase filing but also give rise to tension between the sector and the government). Though complicated (how do you decide whether such a reform’s overall effect is “good” or “bad”?), tensions between policy objectives inevitably exist. These data may therefore help inform choices between competing policy priorities by addressing the questions of “did it work and was it worth the trade-off?”
Data may also be disaggregated at this stage (according to metadata) for different key groups. This analysis will be important in later determining, for example, if new organizations receive most fines, or if smaller organizations continue to file late returns, or if certain reforms had a disparate impact on certain groups.
Evaluation would be incomplete without consideration of context. Among other issues, it is important to consider unaddressed issues and rejected recommendations. Just as each reform can be understood as intended to address particular problems, so there may be problems that the regulatory reform sought not to address or proposed solutions that were not taken forward (e.g., defining “charity” in legislation, lack of clarity around political activities, and issues relating to cross-border philanthropy).50
This stage in the evaluation process is critical, lending legitimacy and context to any conclusions. However effective the reforms are shown to be in achieving their aims, their significance – and ultimately success – can only be fully understood by reference to the context in which they were introduced. This wider consideration helps one understand attribution and impact, and reveals whether the problems the reforms sought to address were appropriate (most pressing, most prevalent, most costly) as well as what work remains to be done.
Analysis and Learning
At this stage, conclusions can be drawn on the effectiveness of the regulatory reforms. The evaluation process proposed lends itself to the drawing of full, nuanced conclusions, whereby the impacts of the reforms may be understood from multiple perspectives, in terms of their efficacy or ability to solve a range of problems, the likely mechanism(s) by which they took effect, and their inter-relationships, contextualized by the wider sectoral and social context. Sophisticated conclusions can be drawn and should be encouraged.
Evaluation using a trial solution/theory of change model enables learning to be implemented and shared. Rather than evaluating simply in terms of “better or worse,” this approach seeks to address the questions of how effectively the reforms resolved the problems (e.g., how good was the theory of change?) and why; what else happened; and what remains to be done; what might have been done differently. Conclusions in this form lend themselves to such practical applications as the refinement of policies and reforms; the introduction of new or elimination of undesirable initiatives; identification of new problems to be tackled; and policy transfer between departments and states. Additionally, connections, trade-offs, and compromises between competing policy priorities may be better understood and reviewed.
Key Features & Limitations
Evaluation itself has an impact. In Canada’s situation, evaluation through a carefully facilitated participatory process has the potential to significantly and positively affect the VSI objectives (e.g., civil society vibrancy, sector-government relations).
The process we have described has attempted to address the limitations of existing methodologies as they apply to evaluating the impact of VSI regulatory reform. Its key features are that it is participatory; “aggregatable” through plausible, sequential pathways to improve attribution confidence; “disaggregatable” to reveal differential impacts according to key factors; and designed to capture outcomes or impacts against defined problems; as well as focusing resources on critical reforms/clusters. In effect, the process attempts to hybridize logframe-style indicators with participatory dialogue to decide which reforms to assess, what improvements might look like, how to measure them, and what data to collect. This helps to account for pluralism and provide a wider context (both beyond the sector itself, beyond intended effects, and beyond abstract or prescribed “improvements”) while ensuring some accepted and meaningful framework in which to assess impact.
Recognizing that selecting one evaluation methodology over another inevitably involves trade-offs and compromise, we have identified some of the issues the suggested approach raises.
- Participatory evaluation overcomes some problems (pluralism, context) but raises others.51 These include barriers to participation (e.g., capacity, culture, distrust), self-selection yielding a distorted picture, logistics (full participation would be unfeasible), and the inevitability that someone will take a lead in the process for conclusions to be drawn (e.g., deciding which views to canvass, and their weightings, or merely to adopt a participatory model).
- Evaluation can only be participatory to the extent that the sponsor is prepared to relinquish control over the process, which is difficult for the sponsor to do in some contexts.
- Acknowledging resource limitations, we proposed a selective rather than complete approach. This approach inevitably involves a trade-off, risking the omission of the wrong things, and forcing the optimal balance between cost and completeness to require careful negotiation.
- The problem of attribution in terms of cause/effect is difficult, but we have suggested that attribution confidence levels can be increased through triangulation between different metrics and narrowing the gap between micro reforms and macro goals by tracing meso stages of aggregation through the pathways. Indices are also useful tools for triangulation of results.
- A model in which an evaluation is developed for and within a specific context lends itself well to understanding the impact of technical reforms within their context, but less well to comparison between situations (across time or country), for which existing Indices are better suited.
- We have assumed the VSI’s delimitation, but in practice its parameters may be less clear. Although the VSI was the five-year initiative that has now concluded,52 some of the recommendations have yet to be implemented fully, and for others, a more extended time period may be necessary before the impact of change is fully known.
There are a number of questions from the original conference brief which this paper has not addressed. We have neither commented on whether the VSI has improved the regulatory framework – a task for the evaluation itself – nor defined what constitutes improvement, suggesting instead that improvements should be defined by reference to their context and the problems the regulatory reforms were intended to solve, through the voicing of and negotiation between different perspectives. Neither have we come up with a magic formula to prove attribution.
Supplementing the contribution of Indices, we have suggested a practical framework within which an evaluation method might be developed to expand the practical impact of specific VSI reforms. We have not offered a “ready-to-use” method: we believe its development should be collaborative and context-specific. In our suggested approach, we have attempted to recognize both the inherent constraints and necessary trade-offs in evaluation (pluralism, attribution, limited resources, etc.) and Canada’s need for a practical solution.
The dearth of suitable models for evaluating the impact of specific regulatory change creates both a challenge and opportunity for Canada. We recognize that much work remains to be done, and we at ICNL are prepared to provide further assistance to the Government of Canada as it embarks on this ground-breaking initiative.
1 Douglas Rutzen is the President of the International Center for Not-for-Profit Law (“ICNL”). Michelle Coulton is reading MA Voluntary Sector Studies at University of East London and is on a career break from the Charity Commission for England and Wales. This paper was presented at the Social Development Canada (Audit & Evaluation Directorate) Conference on Methodology, Ottawa, December 5-6, 2005.
The authors wish to thank Jennifer Green for her research assistance, and Erin Means for her assistance in finalizing this paper. The authors also express their particular appreciation to Lester Salamon for his help in thinking through key aspects of this paper. This article reflects the views of the authors only, and not those of Social Development Canada, the Charity Commission for England and Wales, or any other entity.
2 The International Center for Not-For-Profit Law is an international not-for-profit organization that promotes an enabling legal environment for civil society, freedom of association, and public participation around the world. It works with local partners through participative programs to provide assistance in the design and implementation of civil society laws and initiatives and to develop skills in policy formulation and legislative activity. For more information visit <https://www.icnl.org>.
3 See Catherine Shea, Assessment of Legal and Regulatory Environment in Albania (presentation given at the Enabling Environment for Civic Engagement Advisory Board Meeting, 6-7 June 2004), World Bank Website <https://lnweb18.worldbank.org/ESSD/sdvext.nsf/66ByDocName/NewsandEventsLearningSeries-CivicEnagagementEvent1> [accessed 2 November 2005].
4 See USAID Website <https://www.usaid.gov/locations/europe_eurasia/dem_gov/ngoindex> [accessed 2 November 2005].
5 See CIVICUS Website <https://www.civicus.org/new/CSI_overview.asp?c=FD8912> [accessed 2 November 2005].
6 See Salamon, Lester M., S. Wojciech Sokolowski, and Associates, Global Civil Society: Dimensions of the Nonprofit Sector, Volume Two (Bloomfield, CT: Kumarian Press, 2004).
7 See Heinrich, Volkhart Finn, Assessing and Strengthening Civil Society Worldwide: A Project Description of the CIVICUS Civil Society Index: a participatory needs assessment & action-planning tool for civil society, Index Paper Series, Vol. 2, Issue 1, 2004.
9 See Salamon, Lester M., S. Wojciech Sokolowski, and Associates, Global Civil Society: Dimensions of the Nonprofit Sector, Volume Two (Bloomfield, CT: Kumarian Press, 2004) , 67.
10 Ibid. 70.
11 Ibid. 73.
12 Ibid. 72.
13 Salamon, Lester M, and Stefan Toepler, The Influence of the Legal Environment on the Development of the Nonprofit Sector, Center for Civil Society Studies: Working Paper Series No. 17 (Baltimore, MD: Johns Hopkins University, 2000), 4.
14 See Salamon, Lester M., S. Wojciech Sokolowski, and Associates, Global Civil Society: Dimensions of the Nonprofit Sector, Volume Two (Bloomfield, CT: Kumarian Press, 2004) , 72
19 See Heinrich, Volkhart Finn, Assessing and Strengthening Civil Society Worldwide: A Project Description of the CIVICUS Civil Society Index: a participatory needs assessment & action-planning tool for civil society, Index Paper Series, Vol. 2, Issue 1, 2004, 7.
20 Ibid. 35-48.
21 Ibid. 42.
22 See Salamon, Lester M., S. Wojciech Sokolowski, and Associates, Global Civil Society: Dimensions of the Nonprofit Sector, Volume Two (Bloomfield, CT: Kumarian Press, 2004) , 9.
23 Ibid. 28-29.
25 Nelson and Bickel define program theory as “a summary of outcomes a program is intended to achieve and […] the strategies and interventions it uses to get there. In other words, program theory is a statement of how a program is supposed to work.” Catherine Awsumb Nelson and Bill Bickel, InfoLink Program Theory ( University of Pittsburgh Learning Research and Development Center, 7 August 2000) <https://itclass.heinz.cmu.edu/infolink2003/InfoLink03/docs/ProgramTheory.pdf> [accessed 22 November 2005].
26 A notable exception is David Campbell and Sol Picciotto (eds.), New Directions in Regulatory Theory (Oxford: Blackwell, 2002).
27 See Salamon, Lester M., S. Wojciech Sokolowski, and Associates, Global Civil Society: Dimensions of the Nonprofit Sector, Volume Two (Bloomfield, CT: Kumarian Press, 2004) , 71-2.
28 See Julian Le Grand, Motivation, Agency, and Public Policy: Of Knights and Knaves, Pawns and Queens (Oxford: Oxford University Press, 2003).
29 See, for example, Leon E. Irish, Robert Kushen, Karla W. Simon, Guidelines for Laws Affecting Civic Organizations (New York: Open Society Institute, 2004).
30 See Joint Regulatory Table, Final Report of the Joint Regulatory Table: Strengthening Canada’s Charitable Sector: Regulatory Reform, (Canada: Voluntary Sector Initiative, May 2003), 20, Voluntary Sector Initiative Website <https://www.vsi-isbc.ca/eng/regulations/reports.cfm> [accessed 2 November 2005].
31 Working Together: A Government of Canada/Voluntary Sector Joint Initiative: Report of the Joint Tables (Aug. 1999), 45.
32 (Broadbent) Panel on Accountability and Governance in the Voluntary Sector, Building on Strength: Improving Governance & Accountability in Canada’s Voluntary Sector ( Ottawa: 1999), 7.
33 CRA Charities Directorate mission statement, <https://www.cra-arc.gc.ca/tax/charities/mission_vision-e.html> [accessed 22 November 2005].
34 Better Regulation Task Force , Formal Government Response. Less is more: reducing burdens, improving outcomes. A better regulation taskforce report (London: Better Regulation Executive, July 2005), 3.
35 Better Regulation Task Force, Regulation – Less is More. Reducing Burdens, Improving Outcomes. A BRTF Report to the Prime Minister, (London: Better Regulation Executive, March 2005), 11; see also Imaginative Thinking for Better Regulation (September 2003) and Better Regulation for Civil Society (November 2005) <https://www.brtf.gov.uk> [28 November 2005].
36 See Crawford, Gordon, “Promoting Democracy From Without – Learning From Within (Part 1),” Democratization, Vol. 10, No. 1, Spring 2003, 77-98. See also Part 2 in Democratization, Vol. 10, No. 2, Summer 2003, 1-20.
37 Ibid. Part 1, 79-86.
38 Ibid Part 1, 86-95.
39 Ibid, Part 2, 2-5.
40 Ibid, Part 2, 5-6.
41 Ibid, Part 2, 6-7.
42 See Department of Finance Canada, New Agenda for Achievement: The Budget Plan 2004 (Ottawa: Public Works and Government Services Canada, 2004), especially Chapter 4 and Annex 9, Department of Finance Canada Website <https://www.fin.gc.ca/budget04/bp/bptoce.htm> [accessed 2 November 2005].
43 See Joint Regulatory Table, Final Report of the Joint Regulatory Table: Strengthening Canada’s Charitable Sector: Regulatory Reform, (Canada: Voluntary Sector Initiative, May 2003), Voluntary Sector Initiative Website <https://www.vsi-isbc.ca/eng/regulations/reports.cfm> [accessed 2 November 2005].
44 Department of Finance Canada, The Budget Plan (2004).
45 See the model of policy analysis developed by the Centre for Institutional Studies, University of East London, as set out in John Pratt, Michael Locke and Tyrrell Burgess, Popper and problems, problems with Popper , Readings in Institutional Studies, 1 (1994).
46 See for example (Broadbent) Panel on Accountability and Governance in the Voluntary Sector, Building on Strength: Improving Governance & Accountability in Canada’s voluntary sector (Ottawa: 1999); Arthur Drache and Frances Boyle, Charities, Public Benefit and the Canadian Income Tax System: A Proposal for Reform (Ottawa: Drache, Burke-Robertson, & Buchmayer, 1998); and Ontario Law Reform Commission, Report on the Law of Charities (Ontario: Government of Ontario, 1996).
47 See J. Connell, A. Kurbisch, L. Schorr and C. Weiss, New Approaches to Evaluating Community Initiatives: Concepts, Methods and Contexts (Washington, DC: The Aspen Institute, 1995); see also Theory of Change Website <https://www.theoryofchange.org/html/basics.html> [accessed 2 November 2005].
48 See notes 24 and 25 above, for example.
49 Crawford, Gordon, “Promoting Democracy From Without – Learning From Within (Part 1),” Democratization, Vol. 10, No. 1, Spring 2003, Part 1, 87-88.
50 Both Arthur Drache, “Unintended Consequences Taint Reform of Federal Charity Law,” Not-for-Profit News (October 2005), and Kathy L. Brock, “Judging the VSI: Reflections on the Relationship Between the Federal Government and the Voluntary Sector,” The Philanthropist, 19 (2005), 168-81, point to issues that the VSI did not address.
51 See also C. C. Rebien, Evaluating Development Assistance in Theory and Practice (Aldershot: Averbury, 1996).