By Jess Dart with Lee-Anne Molony and Carina Calzoni
We all came in a little worse for wear on the final day of #aes19SYD after a really great gala dinner at Luna Park – AES treating us to an amazing venue once again. The gay and lesbian choir were so awesome, I was really moved by their amazing music.
My highlight of the day was definitely the morning key note with Jane Davidson — she was wonderfully clear, refreshing and as practical as ever. If you’ve never come across her work, it’s a must read. She started by warning us about the bias of validated tools and said sometimes, maybe it’s better to simply ask people. She moved onto stressing the centrality of values and making evaluative judgements – that is what evaluation is all about: making evaluative judgements. Without evaluative judgements it’s just applied social research. She gave us an interesting example of using rubrics to evaluate gender justice and social inclusion. In this case, the rubric was used to actually describe the context, rather than the program, with a “Harmful” level being the baseline and the target was to move ‘up the rubric’ into higher levels of gender justice and social inclusion after a number of years.
We talked about who should develop the rubric, and she reminded us that rubrics are not about what we as evaluators think, they should instead reflect what stakeholders expect to see, as well as drawing on any existing standards, etc. There was an audience question about what happens when stakeholders don’t agree on the values. She said when there is no consensus, root for the views of those most marginalised by the system. Her take home message here was that if you want to understand the health of a system, check how its working for those who are traditionally the most disadvantaged: if it’s working for them, it’s working for all.
I also had a good run of short papers and some fabulous 5 minute “ignite” sessions too – I love this format. In particular, I enjoyed hearing about Julian King’s ideas around value for money. Julian has just finished his PhD on this topic and has landed his thoughts in a good place. His approach combines the more holistic approach to value for money favoured by Department of International Development (DFID) (the 4 Es: Economy, Efficiency, Effectiveness and Equity) with some work on values and rubrics. Love the approach. I also enjoyed attending the session by Adrian Field (another Kiwi from the Kinnect Group) who was talking about socio-technical systems.
A highlight for Carina was a session about the development of a new ethical framework for evaluation in Aboriginal and Torres Strait Islander settings. The project was funded by Department of Premier and Cabinet and Indigenous Affairs. It involved engaging with researchers, evaluators and practitioners to develop a set of ethical principles and apply these to identify examples of evaluation practice that are rigorous, cultural appropriate and endorsed by the community. It was brilliant project and so much has been produced that is accessible to everyone. Hopefully it will be picked up by all evaluators and really change the practice of Indigenous evaluation in Australia. Check out the framework and practice examples.
Lee-Anne loved the theme of values – and got more clarity on the steps of surfacing values to then determine criteria to then develop awesome (as Jane would say) rubrics. She started off at the rubrics workshop with Jane, Kate and Nan on Sunday to see what had changed since she attended one of Jane’s first ever workshops on rubrics about 10 years ago. This was complemented by Mathea Roorda’s paper on Bringing values into evaluation, which described a tool for more explicitly considering values, based on value theory. While values clarity is the starting point for good criteria and rubric development, Jane’s plenary yesterday on ‘Unboxing the core like our lives depend on it – because they do’ reminded us that values are actually the main game. With Andrew Hawkins and Brad Astbury’s point about how even the key Randomised Controlled Trials (RCT) advocates among the ‘big seven’ theorists came to soften on this over time still ringing in her ears as she listened to Jane encourage us to seek Radical Values Transparency (RVT), Lee-Anne decided that this year’s conference was a lot about RVT instead of RCT!
We were totally exhausted by the end of the three days, and got loads out of the conference!
Well done AES!!