How to Measure Automation KPIs

When I first started creating automated processes, I didn’t even measure their performance. I just created them, ran them, and hoped for the best. Which, to be fair, was initially fine. If you are creating automation solutions just for yourself, then you could argue that monitoring performance is somewhat redundant. However, if you’re in a team and promoting your automated processes, or trying to show the improvements they have brought, you will need to set some benchmarks.

This all came to me when I was trying to convince my colleagues that automation was great, and they should get on board with using it. It can be surprisingly difficult to convince people who aren’t “into automation” that it is a great tool. But, your purpose for automation key performance indicators (KPIs) may not be the same as mine originally was. Some general usages of KPIs might be:

• Monitor improved efficiency/gauging progress
• Setting targets/goals to achieve
• Making a case for automation implementation
• Decision making

How to Monitor Automation KPIs

There are a few options you have here, but in my view, the best option is creating a ‘KPI Dashboard’. Again, a variety of options for a dashboard. I tend to go with simplicity, so I just created an excel dashboard. The core of this is obtaining important data points. My initial KPI dashboard was very simplistic and just focussed on some really basic metrics.

How did I get the data? At the time I was using Python to run all of my automated tasks. So, I simply created an excel file for the Dashboard, and my first tab was just a log of all of my automated processes. Each Python script I made would include measurement of time: When the process started, and when it ended. Therefore calculating the time it took to run each process. Each time a script ran, it would add the following datapoints to the log’s last line:

1. Name of Process
2. Date of Process
3. Time of Process
4. Average Manual Process Time
5. Actual Automated Time
6. Time Saved

Manual Time vs Automated Time

Let me touch on the manual process time for a moment. In order to obtain this, whenever I created an automated process, I would time myself three times completing the manual task. Nothing fancy, I would just start the stopwatch on my iPhone and time it. I then took an average of the three attempts, and then the script would hard-code that time into the log each time. Based on this, it would calculate the time saved.

For example. If a manual process took 20 minutes (1,200 seconds), but the automated process took 36 seconds. The script would log that the process saved 1,164 seconds in that specific instance.

Fundamentally, the success of a KPI dashboard is in the actual underlying data. Get this right, and you can do whatever you want with the dashboard within reason. The trick here is to just sit down and think exactly what data metrics you may need for your specific scenario.

Remember, there is a lot you can do with simple data. You can create averages, extrapolations, visuals, estimates, etc. One useful metric I used was a hypothetical cost saving. I took a rough salary of someone in the team handling the manual tasks @ $30/hour and then I calculated the monthly monetary saving from the time which was saved running the automated processes.

How Often to Look at Automation KPIs

Trust me, it can be very easy to check on these every single day. Don’t. You’ll drive yourself mad.

I used to report the metrics to the team on a quarterly basis but would check how they were looking every two weeks roughly. This was primarily a check to make sure nothing was going wrong with the logging of the data more than anything else.

Common Automation KPI Metrics

1. Total Automated Proccesses Complete
2. Average Time of Automated Process
3. How Often Automated Processes Ran (EG – Average Daily Number)
4. Error Tracking
5. Estimated Value
6. Process Break Tracking