Australian Evaluation Society – aes19 – International Evaluation Conference – Day 3 Reflection

By Jess Dart with Lee-Anne Molony and Carina Calzoni

A slide from Jane Davidson's presentation: "Unboxing the core like our lives depend on it – because they do" - Jane Davidson (Real Evaluation LLC)

We all came in a little worse for wear on the final day of #aes19SYD after a really great gala dinner at Luna Park – AES treating us to an amazing venue once again. The gay and lesbian choir were so awesome, I was really moved by their amazing music.

My highlight of the day was definitely the morning key note with Jane Davidson — she was wonderfully clear, refreshing and as practical as ever. If you’ve never come across her work, it’s a must read. She started by warning us about the bias of validated tools and said sometimes, maybe it’s better to simply ask people. She moved onto stressing the centrality of values and making evaluative judgements – that is what evaluation is all about: making evaluative judgements. Without evaluative judgements it’s just applied social research. She gave us an interesting example of using rubrics to evaluate gender justice and social inclusion. In this case, the rubric was used to actually describe the context, rather than the program, with a “Harmful” level being the baseline and the target was to move ‘up the rubric’ into higher levels of gender justice and social inclusion after a number of years.

We talked about who should develop the rubric, and she reminded us that rubrics are not about what we as evaluators think, they should instead reflect what stakeholders expect to see, as well as drawing on any existing standards, etc. There was an audience question about what happens when stakeholders don’t agree on the values. She said when there is no consensus, root for the views of those most marginalised by the system. Her take home message here was that if you want to understand the health of a system, check how its working for those who are traditionally the most disadvantaged: if it’s working for them, it’s working for all.

I also had a good run of short papers and some fabulous 5 minute “ignite” sessions too – I love this format. In particular, I enjoyed hearing about Julian King’s ideas around value for money. Julian has just finished his PhD on this topic and has landed his thoughts in a good place. His approach combines the more holistic approach to value for money favoured by Department of International Development (DFID) (the 4 Es: Economy, Efficiency, Effectiveness and Equity) with some work on values and rubrics. Love the approach. I also enjoyed attending the session by Adrian Field (another Kiwi from the Kinnect Group) who was talking about socio-technical systems.

A highlight for Carina was a session about the development of a new ethical framework for evaluation in Aboriginal and Torres Strait Islander settings. The project was funded by Department of Premier and Cabinet and Indigenous Affairs. It involved engaging with researchers, evaluators and practitioners to develop a set of ethical principles and apply these to identify examples of evaluation practice that are rigorous, cultural appropriate and endorsed by the community.  It was brilliant project and so much has been produced that is accessible to everyone. Hopefully it will be picked up by all evaluators and really change the practice of Indigenous evaluation in Australia. Check out the framework and practice examples.

Lee-Anne loved the theme of values – and got more clarity on the steps of surfacing values to then determine criteria to then develop awesome (as Jane would say) rubrics. She started off at the rubrics workshop with Jane, Kate and Nan on Sunday to see what had changed since she attended one of Jane’s first ever workshops on rubrics about 10 years ago. This was complemented by Mathea Roorda’s paper on Bringing values into evaluation, which described a tool for more explicitly considering values, based on value theoryWhile values clarity is the starting point for good criteria and rubric development, Jane’s plenary yesterday on ‘Unboxing the core like our lives depend on it – because they do’ reminded us that values are actually the main game. With Andrew Hawkins and Brad Astbury’s point about how even the key Randomised Controlled Trials (RCT) advocates among the ‘big seven’ theorists came to soften on this over time still ringing in her ears as she listened to Jane encourage us to seek Radical Values Transparency (RVT), Lee-Anne decided that this year’s conference was a lot about RVT instead of RCT! 

We were totally exhausted by the end of the three days, and got loads out of the conference! 

Well done AES!!

Leave a Reply

avatar
  Subscribe  
Notify of

CLEAR HORIZON ACADEMY

  • (+61) 3 9425 7777
  • Email us
  • 129 Chestnut Street, Cremorne, 3121, VIC

Sign up to our Newsletter

The Clear Horizon Academy is a learning initiative of Clear Horizon.

We acknowledge the Traditional Custodians of the lands on which Clear Horizon is located and where we conduct our business. We pay our respects to Elders, past, present and emerging. Clear Horizon is deeply committed to honouring Indigenous peoples’ unique cultural and spiritual relationships to the land, waters and seas; and to transforming evaluation practice to uphold and reflect Indigenous values.
Register your interest We will inform you when the product arrives in stock. Just leave your valid email address below.
Email We won't share your address with anybody else.
Dr Jess Dart 3

Dr Jess Dart

CEO & Founder

Inventor of practical methodologies and highly demanded facilitator, Jess navigates complexity with comfort and helps her clients to become clear about their desired outcomes and how to get there.

Recipient of the 2018 AES Award for outstanding contribution to evaluation, Jess has over 25 years experience in evaluating and designing social change initiatives and strategies in Australia and overseas.

In October 2005, Jess founded Clear Horizon Consulting and is now CEO. She is also a board member of the Australasian Evaluation society.

Jess is passionate about developing and designing real world evaluation and strategy for social justice and sustainability. She particularly works with systems change interventions, large scale strategy and social innovation. After completing her PhD she co-authored the Most Significant Change (MSC) guide alongside Dr Rick Davies, which is now translated into 12 different languages.

MSC is well suited to evaluation in the complex emergent context. The latest innovation by Jess, Collaborative Outcomes Reporting (COR), is a collaborative form of impact evaluation.

Jess is also an active mum and has two teenage boys. In a quiet moment, she loves reading far-future science fiction and enjoys long distance running.