News and Events

Impact Research NZ

High quality research and evaluation within New Zealand’s social services sector

  • Home
    Home This is where you can find all the blog posts throughout the site.
  • Categories
    Categories Displays a list of categories from this blog.
  • Tags
    Tags Displays a list of tags that have been used in the blog.
  • Bloggers
    Bloggers Search for your favorite blogger from this site.
  • Team Blogs
    Team Blogs Find your favorite team blogs here.
  • Login
    Login Login form
Super User

Super User

Super User has not set their biography yet

“Dancing with data”: Investing in capacity building for non-government organisations (NGOs)

Author(s): 

Annie Weir and Christa Fouche

§  Developing evaluation capacity with non-government organisations (NGOs) in New Zealand is in vogue, with funders increasingly keen to demonstrate that their investments in social-service programmes are outcomes-focused and providers keen to demonstrate the difference they are making. This article presents a case study of how a large philanthropic trust, focused on family social health and wellbeing, engaged with their grant recipients to improve both outcome-focused evaluation practices and their own evaluation of grants. Partnering with a community funding broker and a research company, the Trust enabled an evaluation capacity-building programme, Dancing with Data. This programme was conducted as three distinct workshops several months apart, with 34 grant recipients ranging from small to large NGOs. Through this initiative, these agencies have developed an evaluation framework focused on the value and cumulative impact of funded projects. An evaluation of this programme with participants and key stakeholders highlighted the challenges and benefits of such a collaborative approach to developing the skills and knowledge needed to commission or undertake effective outcome-focused evaluations.

Journal issue: 

Evaluation Matters—He Take Tō Te Aromatawai 2: 2016

Start page: 

131

 Free full text: 

 PDF iconEM2016_1_131.pdf

DOI: 

http://dx.doi.org/10.18296/em.0014

Hits: 324
0

Evaluation Matters — He Take Tō Te Aromatawai 2: 2016

Eval Matters Issue 2Hot of the press – the latest issue of ANZEA’s journal, Evaluation Matters — He Take Tō Te Aromatawai has been released and is free to access online at NZCER.

Hard copies of the journal are available for purchase from NZCER’s website.

Articles and Author(s)

 Evaluating in traumatic contexts: Considering the contextual, ethical, emotional, and political aspects

 Author(s): 

Carol Mutch and Annie Weir

§  This article is drawn from the personal experience of two evaluators working in a post-disaster context. While the evaluators, also the authors of this article, were both experienced in a range of evaluation settings, they found that little in the evaluation literature prepared for them for working in such a physically exhausting and emotionally draining context. The article is set out in three parts. First, the authors synthesise the literature on evaluating in disaster and other traumatic contexts. Secondly, they outline the evaluation they undertook for The Salvation Army in the wake of the 2010/2011 Canterbury earthquakes. Finally, the authors reflect on their experiences by discussing the evaluation context, design, ethics, implementation and outcomes in order to add to our growing understanding of evaluating in traumatic contexts.

Journal issue: 

 Evaluation Matters—He Take Tō Te Aromatawai 2: 2016

Start page: 

23

 Free full text: 

PDF iconEM2016_1_023.pdf

 DOI: 

  http://dx.doi.org/10.18296/em.0010

Hits: 314
0

Follow the link below to recently published journal articles:
http://www.nzcer.org.nz/nzcerpress/evaluation-matters

Mutch, C., & Weir, A.  (2016). Evaluating in traumatic contexts:  Considering the contextual, ethical, emotional, and political aspects.  Evaluation Matters – He Take Tō Te Aromatawai 2.  doi:10.18296/em.0010

Weir, A., & Fouche, C. (2016). ‘Dancing with Data’: Investing in capacity building for Non-Government Organisations, Evaluation Matters - He Take Tō Te Aromatawai, 2:2016, 1-12. 

Hits: 322
0

Panel: “Outcomes and Accountability: Holy Grail, Wholly Possible or Black Hole”:

On both sides of the Tasman, funders (and many other third sector organisations) have been in search of the elusive 'outcomes'.Guided by the Government Procurement Branch of the Ministry of Business, Innovation and Employment, the New Zealand government is moving to implement a "Streamlined Contracting Framework" across all government funders, including the standard use of an outcome measurement tool, like Results Based Accountability (RBA) or equivalent. Its an issue at the top of the agenda for many policy makers, researcher, funders and third sector organisations.

The Keynote panel:
Dr Annie Weir, Director of Impact Research, Auckland has worked extensively with RBA and other outcomes measures
Dr Michael Houlbrook, Senior Lecturer, University of Sydney, whose research looked at the impact of imposed RBA on small community-based organisations.

Kate Frykberg, Executive director of the Todd Foundation, Chair of Philanthropy NZ, Executive Director of the Todd Foundation and a prominent NZ blogger on philanthropy & community. See conference site for details.

Hits: 485
0

Abstract:
The paper presented a case study of how a large philanthropic trust focused on family social health and wellbeing engaged with their grant recipients to improve outcome-focused evaluation practices. The Trust also aimed to improve their evaluation of grants and to create an evaluation framework to establish the value and cumulative impact of projects funded. The challenges and benefits of a collaborative approach to developing the skills and knowledge needed to undertake or commission effective outcome-focused evaluation are explored. 
   
Partnering with a community funding broker and a research company, the Trust provided an evaluation capacity building programme, conducted over several months for 33 grant recipients ranging from small to large NGOs. The ‘Dancing with Data Evaluation Programme’ was based on the premise that evaluation provides a platform for collaborative learning, accountability and focused strategy.

The study aimed to determine how successful the programme was in contributing to building an organisation’s evaluation capacity. Qualitative methods were used including a document review, face-to-face interviews of key stakeholders and a survey of participants.

The learning objectives of the programme were successfully met as evidenced by the majority of participants completing an organisation-wide evaluation plan and utilizing a wider range of evaluation tools to evidence outcomes to inform their development and funding applications. Participants valued sharing evaluation experiences and some formed new collaborations. The majority of evaluation plans met or exceeded expectations with a few needing further assistance to complete. Both participants and the Trust saw value in integrating evaluation into their operational and strategic plans. The Trust utilized the findings from the evaluation of the programme to inform their own evaluation practices.

The implications for evaluation practice are that a genuine partnership between funders and recipients to build evaluation capacity requires dedicated funding, time, commitment to mutual learning and improving current practice.

Hits: 665
0

The Authors:
Dr Annie Weir, Director, Impact Research NZ and Honorary Research Fellow, School of Critical Studies in Education, Faculty of Education, University of Auckland.
Dr Christa Fouche, Head of School, Social Work and Human Services, University of Auckland, Auckland and Research Associate Impact Research NZ.

About the Article:
The article reports on a study undertaken in Auckland and Hamilton on the effectiveness of community-based dementia day programmes. The small-scale pilot study was aimed at investigating the elements that make up an effective client-focused dementia day programme and the methods employed by organisations to measure the outcomes of these programmes. A mixed methods approach was employed with multiple stakeholders. The research revealed that effective day programmes comprised five core elements, and that surveys, reporting and auditing processes are routinely used to measure the quality of outcomes of day programmes. Although these findings are reflective of a specific context, it raises concerns about the nature and availability of evidence informing decisions regarding the design and implementation of day programmes internationally.

Hits: 496
0

Author: Dr Annie Weir, Director, Impact Research NZ, Honorary Research Fellow, School of Critical Studies in Education, Faculty of Education, University of Auckland.

Abstract:
The Department of Corrections Innovation Fund supports their aim to reduce recidivism. The Whakamanahia Wahine community-based programme received funding for two pilots in 2013 and 2014 which were attended by between ten and 14 low-risk female offenders from the Bay of Plenty. The programme goals were to improve individual well-being and personal development and to reduce recidivism. The programme was based on a holistic approach to building sustainable skills for personal wellbeing including self-efficacy around educational attainment, building personal wellbeing skills, fostering positive relationships and creating personal pathway plans. A three-month follow-up evaluation from the first pilot showed zero recidivism amongst the contacted participants. The programme for the second pilot for ten participants kept mostly the format of the first pilot, however based on participant and stakeholder feedback  the programme length was extended (from six weeks to eight weeks),  a co-facilitator and mentors were included as well as selecting educational goals based on the participants’ individual needs. These programme elements are supported by the international literature on good practice. The evaluations and results of these two pilots support the possibility for future mainstreaming of funds to offer this programme more widely.

Hits: 603
0

The Social Workers in Schools (SWiS) service, funded by the Ministry of Social Development (MSD) through Child Youth and Family (CYF), was established in 1999 in response to concerns raised by schools about the impact that social and family problems have on students and their ability to learn. Access to SWiS in decile 1 to 3 primary and intermediate schools has never been better, but how is SWiS working? The Ministry of Social Development (MSD) requires service providers (non-government organisations (NGOs) to report using a Results Based Accountability Framework (RBA).


This seminar presents the findings of an outcomes- focused evaluation, using RBA, of the delivery of SWiS by a large NGO to over 100 schools in the upper North Island. Both qualitative and quantitative data were collected through surveys of child clients, their parents/caregivers and school staff. Results show that parents and caregivers report that their children are more settled, happier at school, more engaged in their learning and have developed new strategies for dealing with challenging situations. School staff value the SWiS service because it allows them to concentrate on their primary responsibilities. It is also evident from the evaluation that SWiS helps to strengthen relationships between home and school.

 

 

Hits: 1558
0

Dr Annie Weir presented at the New Zealand Council of Christian Social Services Conference, “The Next Stretch”, which focused on services for older people.  Annie’s presentation What the key ingredients to effective Day Programmes. Presented findings from recent research undertaken by Impact Research NZ that explored day programmes for older people in New Zealand from the perspectives of a wide range of stakeholders. View Dr Weir's presentation

Hits: 952
0

Dr Annie Weir presented at the National Family Start Hui on the results of the Waitakere Family Start service Results Based Accountability evaluation pilot.The service is a collaboration between the West Fono Health Trust and Family Works Northern. Dr Weir’s presentation can be viewed here.

Hits: 1191
0

This practical evaluation programme was offered in partnership with Auckland Communities Foundation and funded by SKYCITY Auckland Community Trust Board. Three cohorts of participants from social service NGOs completed the programme, which comprised three workshops of two hours duration, with an expectation that development activities would be completed between the workshops; an overall commitment of approximately 12 hours over the duration of programme. Read more

Hits: 1378
0

Results-Based Accountability

Evaluating program outcomes in a social services organisation in New Zealand

In times of financial constraint, social services organisations reliant on external funding from government, community and philanthropic sources are under increasing pressure to demonstrate to their stakeholders the difference they are making to their client's lives. This can present a challenge for organisations, encouraging them to move beyond traditional methods of financial accountability. From collecting data on the level and extent of their services, they need to explore new accountability tools that best demonstrate client outcomes to stakeholders. Read more  

Hits: 1149
0

Dr Annie Weir has submitted a paper for consideration to present at the Australasian Evaluation Society International Conference to be held in Brisbane in September.  The paper entitled  “Safer Parties Together: One Voice” discusses the findings of an evaluation of safer dance parties and large events in Auckland, a collaborative project between Impact Research NZ and the Auckland Council Safety Committee in 2102,  which identified a set of best practice guidelines.

Hits: 1395
0

Dr Annie Weir and Dr Richard Griffiths are to co-present at the 2013 Aotearoa New Zealand Evaluation Society Conference in July on “Effective client-focused dementia day programmes in the community: improving the quality of life for people living with dementia and their families”. The session will present and discuss the findings of a research project undertaken by Impact Research early in 2013.

Hits: 1287
0

Presentation of Results Based Accountability
Dr Annie Weir (Director of Impact Research) and Dr Rod Watts (CEO, Presbyterian Support Northern) recently joined forces to present a workshop on Results Based Accountability at the New Council of Christian Social Services (NZCCSS) Conference April 18-19 2012 held at the University of Auckland. The workshop covered:

  • The context of the development of Results Based Accountability in New Zealand
  • An opportunity to reflect on current practice in monitoring, evaluation and reporting
  • A brief introduction to Results Based Accountability (also known as Outcomes Based Accountability (e.g. UK, Australia)
  • New Zealand Case studies on Results Based Accountability


Follow this link to have a look at the presentation: AW-RW-RBA-NZCCSS-April-2012-1.pdf

Hits: 1243
0
Hits: 3517
0

Dr Annie Weir has been appointed Director Impact Research. Dr Weir holds a PhD in Education from Victoria University of Wellington. Dr Weir has experience working with Maori and Pasifika, the public sector, industry and business, health and education sectors. She has extensive experience in working in senior management in education in New Zealand. Press-Release-New-Appointment-Dr-Annie-Weir.pdf

Hits: 1592
0

Dr Annie Weir has submitted a paper for consideration to present at the Australasian Evaluation Society International Conference to be held in Brisbane in September.  The paper entitled  “Safer Parties Together: One Voice” discusses the findings of an evaluation of safer dance parties and large events in Auckland, a collaborative project between Impact Research NZ and the Auckland Council Safety Committee in 2102,  which identified a set of best practice guidelines.

Hits: 1146
0

Our People
At Impact Research NZ we believe that our people are our most valuable assets. With researchers and Research Associates who are leaders in their fields, our combined experience and expertise encompasses a broad range of social sector interests.
Read more


Our Research and Evaluation
Research and evaluation are now seen as essential elements in the provision of the social services, and provide the means of measuring outcomes for clients, and informing service improvement.

Read more


Contact Us
Impact Research NZ
PO Box 9591 Newmarket
Auckland 1149
Phone: 09 5208696
Email: This email address is being protected from spambots. You need JavaScript enabled to view it.
Read more