OKRs For Your Tests

1

OKRs for Your Tests: Elevate Your Approach to Test Design

Abstract The OKR (Objective and Key Results) system is a performance management tool meant to be used for optimizing productivity and time allocation. Its aim: to transform the way people set goals for themselves and build a purpose driven environment, ensuring seemingly difficult objectives are fulfilled. However, what if this could be applicable to tests themselves as well? Although goal-setting is inherently a human-driven activity, what if we hand the same or at least similar type of responsibilities and challenges to our own tests? Give suites an objective that should be aimed for and measurable results that will define whether it’s been fulfilled or not, transforming them into active and goal-oriented components of the development lifecycle.

The purpose of this paper is to broaden the perspective from which tests can be designed and managed, and make sure they contribute directly to the overall business goals and toward meaningful outcomes. Too many times have test suites been designed, executed and even automated with no clear purpose in mind, resulting in them existing just to fill a quota or reach certain coverage. By using this approach you can orchestrate hundreds of tests ready to support you in achieving a shared goal.

1

Introduction to the OKR system

The OKR (Objective and Key Results) system is a tool to help mold behaviour for better time allotment and focus. The development of this framework traces its origins to Andrew Grove, who introduced it to Intel in the 1970s. It is a simple yet proven to be efficient approach for defining measurable goals and tracking progress. The idea was also introduced to Google by John Doerr (also the author of Measure What Matters, which inspired the idea of this paper), and it later became central to Google’s culture and way of working. It consists of objectives and key results, which are the two main concepts upon which the entire framework is built upon.

The objectives themselves are concise and well defined goals that should be striven for. Specific objectives produce a higher level of output than vaguely worded ones, and that is the reason they are made. The key results serve as criteria for tracking the completion of those objectives. These criteria are measurable and clear, while the objectives are action oriented and inspirational. Combining these two together and using them coherently as a system by individuals, teams and entire organizations has shown to be a

2

Jana Markovikj

simple yet powerful method for achieving results. OKRs help drive people forward, not constrain them to additional rules, representing a blueprint that can be shaped around the goals and intentions of those creating it. This system also gives a lot of visibility on what exactly is going on at each point in time within the team, project or entire organization.

It’s really a practice that shapes the work, and can be an amazing way to approach the testing process itself. If done well, it can surface primary goals and channel coordination.

1.1

Objectives

The objectives are the directions, they drive the work towards where it should be going.

These are set by individuals, teams or on the level of the entire organization. When all people involved choose a course of action, they are more likely to see it through. Not only clear and concrete, but objectives should also be challenging. When it comes to testing, objectives can help align the process with business goals, and that testing efforts contribute directly to the product. These are not necessarily measurable, however they should be defined well enough so that they can be asserted as achieved or not.

Defining a clear objective can initially be challenging, but with experience and adjustments the process becomes more straightforward. This is why it’s also a valuable mental exercise that encourages people to rethink their approach to challenges and broadens their perspectives on possible solutions. An example of a good team objective would be: Enhance product reliability to improve customer satisfaction. This defines a clear goal with a reason, which can be determined as accomplished or not by the ones who set it in the first place. It is well-defined and sufficiently challenging to motivate everyone involved to aim for its achievement.

1.2

Key Results

Key results have to be measurable, they are the means of paving the path towards the objective itself. They can be estimated percentage wise on a 0-100% scale or with any numerical value which can later be used to determine whether it has been completed. By the end of the OKR time frame (which is established in advance, but can also be adjusted later on), each key result should have a definitive outcome. Was it done or was it not done?

As a best practice, teams, individuals, or organizations can aim for a target success rate on their key results (for example, Doerr recommends 70% as the target rate [01]).

This success rate can encourage those working on the objective to stay engaged and competitive in their decisions. When defining key results, it’s important to choose readily measurable indicators which can quickly be assessed during execution so that corrections can be made when necessary.

It’s also important to avoid having too many key results for a single objective, a smaller but concrete set is more effective than a larger, extensive one. Focus on results, however not so much that OKRs are made to be just more tasks and metrics to follow.

OKRs for Your Tests: Elevate Your Approach to Test Design

They’re meant to guide, not micromanage. The goal is to reflect the most important areas to track progress and redefine where the focus should be. And example of a good key result regarding testing would be: Achieve 90% automated test coverage on all critical user flows by the end of the quarter. This is specific, measured by percentage, and has a clear deadline. The time frame for achieving a key result does not need to be stated explicitly, but could also be determined implicitly by the overall OKR lifespan.

1.3

Determining completion

Given the constituents of an OKR, to truly determine whether it has been achieved, a score on a scale from 0.0 to 1.0 is given. A target score is set, and this score is influenced by the completion of key results. It’s important to take the time to review OKRs regularly, so that individuals and teams can assess their progress, make necessary adjustments and ensure priorities are aligned. Think of this process as a way to refocus and stay committed to the intentions that have been made when the OKRs were first created.

The timeline for one can vary, depending on how challenging it is and how extreme the key results may be. It might span a sprint, quarter or several years. When defining OKRs and their duration, it’s important to consider the realistic capacity and availability of the people working on them. They are meant to be guidelines, not deadlines. When the need arises, they can be changed, dropped or renewed. OKRs are meant to be flexible, and their key results can be modified or even discarded if it’s so determined. Ultimately, this system is a tool not a weapon, meant to support progress, not constrain it.

Completion of an objective isn’t decided by the passing of its lifespan, but by the satisfaction of its creators.

2

OKRs for tests

When it comes to testing, this system can be used to help manage both automated and manual test cases. The OKRs themselves can be established by individual testers, teams or even on the level of the entire organization. However, creating these from the perspective of the tests themselves can make the goal feel more alive and purposeful. It’s a different way of approaching the design and grouping of cases that can prove very efficient and helpful. For testers themselves it can be a good mental exercise that encour ages rethinking test purposes.

When writing tests, quality engineers tend to focus on flow coverage and high-risk areas. However, increasing the number of cases can reduce the clarity of the objective and increase the complexity in execution and documentation. The tests themselves are treated as isolated instances that should be executed and determined to have passed or failed-not taking into account a broader purpose for their execution beyond documentation and metrics.

However, if testers were to approach them with a mindset around OKRs, they can be redesigned into achieving a higher, shared goal and speed up progress and quality man-

4

Jana Markovikj

agement. The OKR framework can be used to refocus the aim of entire manual and automation suites, making them more than numbers to be documented. The objectives can vary depending on testing priorities within the team or project, and each of them can challenge the design and grouping of tests. They too need a process of “discipline” to keep them from trying to accomplish everything. Since when it comes to writing them, we tend to try and achieve it all with less directed input.

2.1

Grouping tests for an objective

Grouping test cases and suites to reach an objective can enhance their effectiveness throughout the development life cycle. This would also ensure that they are not only aligned with project goals but they also serve their intended purpose, since it would be them reaching the designated objective. Prioritization of critical areas based on OKRs can make sure testing efforts are directed where they have the greatest impact, and help move the project forward. Giving the tests themselves a goal to reach might seem unusual, however approaching and grouping them with this mentality can be incredibly powerful when applied with intention.

Rather than organizing tests by flows, priority or automation, they can be grouped by the objective they aim to achieve. An example for this would be grouping both manual and automation tests based on their impact on product quality. Say you have a number of manual and automation tests already written and used throughout the project, and you want to reach an objective of Increasing application security by checking for vulnerabilities. By aligning them to this objective, you can focus on their execution, update steps where needed, and ultimately ensure they contribute to that goal. Instead of just categorizing them, assign them.

2.2

Designing test cases for an objective

Grouping suites to reach an objective can be useful, however we can push this further by designing and writing new ones as the project evolves. An example for ensuring successful stress testing would be designing tests to push the limits of the application, and giving them that purpose. If our objective is to fully test the limits and thresholds, then the tests themselves would have the task to achieve it. The key results could be measured by the status and values of metrics after the execution of these tests. This will also show significant promise to clients, as well as create better deliverables.

How OKRs would help in this scenario is that they would shift the tester’s mindset towards designing cases with a goal in mind. When writing, a quality engineer can focus on delivering valuable results with the tests being the catalysts, instead of simply for coverage or compliance. If the objective is Increasing application security (as mentioned in the previous passage), each test case written will execute with a means to an end-the end being creating a more secure product. Cases will be working together to reach this goal, and testers will be able to demonstrate a more directed and strategic approach to their design.

OKRs for Your Tests: Elevate Your Approach to Test Design

2.3

Using outcomes to determine key results

After completing test runs, the next step usually involves reporting, documentation and visualization of results. These could also be used to determine whether key results have been achieved or not, and how tests have contributed to the overall objective. By using them in this manner to track OKRs, efficiency can be improved and the process of execution can be more proactive. Don’t just complete the task, learn from it as well. If the objective is to Improve application security by testing critical areas, and tests have been designed and grouped to support that, key results could be measured by the number of simulated attacks, targeted areas covered, vulnerabilities caught or pass rates.

The results will not only determine the status of the tests, but the status of the objective as well. In addition to this, testers can use the OKRs to showcase the impact of their work by tying results directly to business goals and communicating the value of testing efforts more meaningfully to clients.

3

Timelines, values and collaboration

When creating OKRs, a common challenge is determining the appropriate number of objectives and key results. These can vary, but it’s generally recommended to have no more than five key results per objective. This is because they monitor how we get to the goal, and are specific and time-bound enough to present a significant workload. When it comes to writing them for tests themselves, it’s important to consider the full scope of work involved: grouping and designing, but also execution and reporting. Once these tasks are finished, the key results can be determined complete or not, which in turn would also give the score of the objective.

Another common challenge is setting the timeline, which can range from a week, a sprint or even quarters. These timelines are usually tied to the key results, since they represent the measurable parts of the framework. For example, if a tester only needs to group and execute tests for an objective, it would take much less time than writing them from scratch as well. This will naturally become easier as OKRs are used more and adjustments are made depending on the work. Writing them should not create pressure or fear, but rather motivation to improve planning and time management.

3.1

Being concrete but not rigid

Since key results are the way we get to the end goal, they should be specific in how they are to be measured, with no uncertainties whether they are done or not. However, a tester should stay flexible and if the climate has changed and an objective no longer seems relevant as written, key results can be modified mid-cycle. The same would be for the objective, if it’s too impractical to even have it at a certain point and everyone agrees to it, it can be modified or discarded.

A few extremely well-chosen objectives give a clear message about what testers should say ‘yes’ and what to say ‘no’ to. Grove gives a limit of three to five OKRs per

6

Jana Markovikj

cycle, so as to drive focus [01]. And as a general suggestion, each objective is said to be tied to five or fewer key results. When assigning your own test suites and cases objectives, make sure they are not given too many, so as to not overload them. The key results themselves should be specific, measurable and time-bound. For example, a key result written for automation tests might be: Maintain a 98%+ pass rate across three consecutive CI runs. This type of statement removes uncertainty by clearly outlining what should be measured and over what period.

3.2

Setting the ideal time frames

When determining timelines for OKRs, the objective should be looked at as a sum of its parts-those parts being the key results. When writing them and deciding on their allocated time for achievement, quality engineers should focus on giving enough time to avoid burnout, but enough pressure to stay focused. In the way this framework is proposed to be used, the tests are the ones accomplishing the work, and testers are the ones designing them to do so.

Grouping would require a lot less time than when writing them as well. If an objective involved repeated test executions, like stress and load tests, then this should also be taken into account when allotting the time. If at a certain point testers determine they need more to properly design and execute their tests in order to fulfill the objective, key results can and should be modified. Clear-cut time frames can intensify commitment, however they should not present rigid constraints.

3.3

Creating a network

OKRs are also a cooperative contract to establish priorities and define how progress will be measured. They are not islands. To the contrary, they create networks – to connect everyone’s most vital work [01]. Even when creating them specifically for testing purposes, the entire team or engineers across teams should have a say in how they could be accomplished. Suppose you’re aiming to improve your automation tests by creating OKRs for them, you can include different QA experts across your organization to gather more insight. This will not only improve your own work, but also motivate the others and encourage collaboration, connecting theirs as well.

The key is transparency, and OKRs are meant to be visible to anyone interested in seeing them. If the ones you have written for your tests are available for everyone else to see, they can drive motivation and also hold accountability. They can also be used (if applicable) by other testers to improve their own work, instigating a collaborative process of focus across teams, countries and time zones.

3.4

Examples

While explanations are insightful, nothing helps more than a concrete example. Below are samples of OKRs written for tests, which can be created by individuals or by multi ple QA engineers collaboratively.

OKRs for Your Tests: Elevate Your Approach to Test Design

These are intended to be used as a reference, not as exact templates. Every project, testing framework and process are different, so OKRs should be tailored to their context.

Once each key result has been determined as completed or not, it contributes to the final score of the objective, which is from 0.0 to 1.0. For instance, if 3 out of 4 key results are achieved, then the objective would be scored as 0.75.

Tab. 1:

Example of an OKR meant for a test suite

Objective: Become a lean and reliable safety net for every code change KEY RESULTS 1. Run in under 10 minutes across the full suite 2. Catch 90% of critical bugs before code reaches production 3. Maintain a flake rate of less than 5% across all environments 4. Cover 100% of business-critical workflows with automated scenarios 5. Be triggered automatically on every code commit with zero manual intervention

This OKR serves as an example of one that a test suite might be assigned to improve se curity during code changes. The objective itself is clear, well-defined and challenging.

The key results are specific and measurable. The tests within the suite take on the “challenge” of meeting these results, since they are, in fact, the ones that should be running in under 10 minutes and triggered automatically. It’s the tester’s job to make sure they get there.

Tab. 2:

Example of an OKR meant for test execution

Objective: Improve effectiveness and efficiency of execution KEY RESULTS 1. Execute 100% within the planned time frame for the next sprint 2. Achieve bug detection rate of at least 90% for critical features 3. Reduce time spent on testing regression by 10%

8

Jana Markovikj

This is a more generalized objective, however still defined enough to direct focus where it needs to be directed. It’s targeting specific areas that could use improvement, and assigning them to the entire testing framework.

By using OKRs this way, aspiring generalizations can be translated into actionable, coordinated plans, and decisions can be properly examined and if needed corrected once results begin to roll in.

4

Outcomes

The purpose of using this framework directly for tests is to be able to determine concrete outcomes that directly affect the quality and efficiency of the product. Documentation does not have to be pass and fail rate only, but also objective and key result fulfillment.

The tests are meant to support our work, not hinder it. So creating and executing them with a means to an end can prove to be a much more efficient way of testing any type of product.

Their results can be directly juxtaposed to the key results and their objective, making sure we have done what we have set out to do. Presenting them to clients and teams and how they directly impact on goals will further show the quality of our testing process and work. By shifting our mindset, we can achieve greater focus on what’s important when working, and designing more robust and purpose-driven testing frameworks.

5

Conclusion

It’s vital not to get stuck in repetitive testing cycles that bring about little real progress. This is why a good way to make sure it doesn’t come to that is to rethink the testing entirely. The OKR system has proven to have helped multiple companies boost their productivity and achieve their most aspirational goals, and it can do the same for testing professionals as well. It offers a way to really broaden the perspective from which we design, group, execute and document tests. When it comes to presenting the work, sometimes it’s important to stop only selling to testers and start selling to stakeholders and clients as well.

There is no better way to demonstrate what was done by presenting the goals achieved and the way efforts directly impacted the product. By adopting this system testers can not only increase the effectiveness of their test frameworks, but also truly show the difference they are making.

References [01] J. Doerr: Measure What Matters. 2017