Evaluation Frameworks & Methodologies

Building the right frameworks with the correct methodologies helps to pave out success within any given project

We’ve all been there. At some point, during a project, panic starts to take hold as people realise they need to measure success but didn’t really plan for it.

This is not a good place to find yourself for all sorts of reasons. But let’s come at this from the other direction.

Why are evaluation frameworks important and useful?

Having an agreed framework that details what success looks like means, first and foremost, that consideration has really been given to it. I know, it seems silly to suggest that people actually begin projects without knowing how to judge their success, but they really do — all the time. If you can’t build an evaluation framework relatively easily, it probably means the objectives aren’t clear — do that first.

If you know what success looks like and how it will be measured at the beginning of a project, you also know what data to collect as you go along so you can populate the metrics. If there are data items which need to be collected over and above what already exists, you can put in place actions to make that happen. Where innovation is concerned, there is almost always something new that needs to be collected.

We have had instances where a customer realised early on that the data doesn’t exist and can’t be collected for a certain group of beneficiaries, and therefore we would never be able to tell whether the project was successful or not. The beginning of the project is the best time to realise this. Realising they had chosen a group of beneficiaries whose outcomes were not possible to measure precipitated a change in the project direction which prevented months of wasted effort. This leads on to another important consideration.

Who are the beneficiaries and what are the benefits?

It may or may not be obvious who the beneficiaries are. When we deploy into an enterprise environment, we usually have four distinct groups of beneficiaries for whom we collect data. Each group stands to benefit in different ways, so not only does that mean four distinct levels of metrics, it can also imply different types of data are required.

Getting the data right

Everyone wants to know the numbers. Financial return on investment #ROI is certainly important, and often forms the hub of the business case for further investment, but it’s seldom the whole story. The mid-1990s concept of the triple bottom line is not talked about so much now, partly because it’s been subsumed into what many organisations would refer to as CSR (Corporate Social Responsibility). Its legacy, however, is that organisational success has come to be judged on more than just profitability. Along with financial returns, comes social and ecological benefits as well as, in some cases, benefits in other areas.

In our case there are personal benefits to our subscribers that take the form of improved health, wellbeing or social outcomes. These are related to, but distinct from, the organisational economic benefits to, say, a health trust of having fewer admissions to hospital of patients who are less ill when they do arrive.

Furthermore, alongside the health, wellbeing and social benefits to our subscribers, are also benefits to their loved ones which are less easily measured such as peace of mind, and a reduction in anxiety around certain events or times of day.

Don’t forget qualitative data

“Not everything that can be counted counts, and not everything that counts can be counted” — Einstein

Did I mention that the numbers are important? They go a long way to telling the story; but they rarely tell the whole story, and the pieces they miss out are often those that make the story relatable at a human level. I’ll give you an example. If I told you that the overwhelming majority of family members have improved peace of mind and quality of life by checking in with our wellbeing insights, it doesn’t convey the central message nearly as well as the actual words those family members said:

“I always check the app at night and sleep better knowing Dad is in bed and settled.”

“Even though Dad gets up at night, I know he is safe because he quickly goes back to bed.”

“I feel now I have more time for my family.”

It is not for us to determine what weight is given to all these benefits by decision makers, simply to construct the evaluation framework to make sure the full range of evidence is available when a decision is taken.

Building the right evaluation framework measuring the right things in the right way is a crucial early step in any project. It validates aims and objectives, crystallises the benefits and the beneficiaries and defines success. It can save huge amounts of time and embarrassment, and forms the spine of the business case for future investment.

Written by James Brown, Partnerships Director at MySense

Originally published at https://www.mysense.ai on August 18, 2021.