Scorecards are one of the most powerful ways in Cortex to promote accountability among teams and enforce standardization across services and resources. 

Scorecards gather and organize existing data within Cortex to provide greater insight into the performance of your service architecture. You can use Scorecards to establish best practices, track migration, or define maturity standards. For more information about how to use Scorecards, check out the Real World Examples article in our Help Desk.

Before using Scorecards, make sure you have entities in your catalogs — Scorecards evaluate services and resources, so if none exist, you won’t get much out of a Scorecard. You should also make sure essential integrations are set up, like Git tools.

You can find Scorecards in the navigation bar.

As with other features in Cortex, you’ll see all of your organization's Scorecards populate under All, while Scorecards related to services and resources you own will appear under Mine.

Those with the admin or manager permission levels will see the option to Create Scorecard.

Starting from scratch

When you create a new Scorecard, you have the option to use a Scorecard template or Start from scratch. To Start from scratch, select that option from Getting started, and click Continue.

The first step in creating a Scorecard is entering some key details about it, just like with entities. Add a name and identifier for the Scorecard, as well as a description. 

Keep in mind that the Scorecard's name and description will be visible across Cortex, so make sure it’s obvious what services or resources are being scored on. For example, you may name your Scorecard “Security” — it’s pretty clear what that Scorecard is designed to measure. Beyond that, you may link to a confluence wiki page in the description, as well as a brief summary of how you intend to use the Scorecard. The description features full markdown support, so you can enter anything that is relevant in that field.

By default, Scorecards will be refreshed every 4 hours, but you can adjust the timeframe under Evaluation window. You may choose a longer evaluation window for Scorecards that are designed to measure long-term performance, while shorter evaluation windows may be more appropriate for Scorecards set up for shorter-term projects.

You can also toggle Notifications on for your Scorecard. When notifications are enabled, users who own services being evaluated by the Scorecard will receive alerts when there are relevant updates, like score changes.

Apply to specific entities

In the Apply to specific entities section, you'll decide which entities the Scorecard will apply to. 

First, decide whether the Scorecard is evaluating Services, Resources, Domains, or Teams. Scorecards can only evaluate one of these primary entity types at a time. 

By default, the Scorecard will apply to all entities of the selected type, but you can narrow down the selection to more specific subsets. With Basic filtering, you can choose to Include groups of entities or Exclude groups. For example, if you're creating a services-based Scorecard, you might choose to include those that belong in a Tier 1 services group.

For a resource-based Scorecard, you also have the option to include or exclude by resource type, such as s3 or RDS. 

With Advanced filtering, you can write CQL expressions that dictate what is included or excluded from the Scorecard. You can write your custom query within the provided field or use the CQL Builder to create your filter.

Note that advanced filtering only works with data that exists within Cortex, like service groups, owners, and other custom data, but does not extend to third party integrations. 

Define evaluation rules

Next, you’ll Define evaluation rules — these are the criteria your entities will be scored against. Rules can reference entities' metadata within Cortex, and can pull data from third-party integrations.

There are two ways to set up your Scorecard's evaluation: Level progression, which allows you to establish progressive levels as you’re adding rules, and Point-based rules, where each rule has a weighted value.

Writing rules

Regardless of which option you choose, you'll need to write some rules — otherwise, your Scorecard won't measure anything.

There are two ways you can build a rule: through the Cortex UI or with the CQL Editor. The Cortex UI is a great way to get started quickly, since you can choose from pre-configured rules for each integration. 

To use the Form editor, first select an integration from the dropdown menu — your options here will depend on the integrations that have been set up already. Next, select a rule from the dropdown menu.

Finally, enter information in the required fields to finish configuring the rule. These fields will vary depending on the integration/rule you've selected.

At this stage, you can add more information to make the rule more user friendly, like adding a Title. If you do not add a title, your rule will display as a CQL expression — for this reason, we strongly recommend adding a title.

In the Failure message field, you can enter a message that will display when an entity is failing a rule. This is a good opportunity to give team members some direction on how they can pass the rule.

You can Restrict rule to specific groups to apply a more stringent filter to the entities in your Scorecard. This would make sense if a group of entities should be evaluated by the Scorecard as a whole, but a specific rule in the Scorecard does not apply to that group.

In the Points section, you can adjust the number of points a certain rule is worth. By default, the point value is 1. You can use points to signify each rule's importance — rules worth more points are more critical than those with fewer points.

Once you're familiar with Cortex and CQL, you can use the CQL Editor to write your own expressions for each rule. This option will make more sense when writing complex rules that rely on custom data.

Most of the time, it’ll make sense to work with an integration through the Form editor, especially in the beginning. This is one of the reasons we recommend setting up integrations before creating a Scorecard — it's easy and intuitive to build rules based on integrations.

Once you've defined your rule through the CQL Editor, you can enter a readable title and description, failure message, filters, and points.

Level progression

Within a single Scorecard, you might define 10 or 20 different evaluation rules. Although you can use points to assign weighted values to each rule, level progression makes it immediately obvious which ones are most important. With levels, you can set up a gamified system that encourages developers to make progress toward goals and improve entities' performance.

To start building levels for your Scorecard, select Level progression in the Define evaluation rules section, and then Add level.

This will open a modal where you can enter a Name and Description for your level, and choose a custom color. Levels are fully customizable, so you can go with something classic — like Bronze, Silver, and Gold — or something unique that represents your organization. 

Levels are designed to inspire progress on important goals, so think about level names that will motivate team members.

Once you've created a level, you can begin adding rules to it. Select Add rule to [level name] to add rules to a given level.

This will open the Form editor for adding Scorecard rules. Keep in mind that your first level should include the most essential rules, so developers know to prioritize these.

Once you've finished adding rules for the level, you can add another level and repeat the process. You can't duplicate a rule across levels, since developers will have already completed that task earlier. You can, however, write similar rules with different thresholds if you want developers to progress in stages.

Once you have more than one level, you can rearrange the order by using the up and down arrows next to each level.

Level progression is also cumulative, so an entity must maintain its passing status with rules in order to maintain its level. When you add a level, it includes All rules from [the prior level]. An entity could pass all the rules in the second level, but if it's failing a rule from the first, it won't be marked as passing the second level.

Let's take a look at the DORA Metrics Scorecard as an example:

Let's say an entity's last commit was within 24 hours, there were zero rollbacks in the last 7 days, and the ratios of incidents and rollbacks to deploys in the last 7 days are both zero, but it hasn't averaged at least one deploy per day in the last week. 

In this case, the entity would have No Level. Even though it's passing all of the rules in the Bronze level, it's failing a rule in the Steel level; because of this, it has not achieved the Steel level, the first one that an entity can pass. Once the entity averages at least one deploy a day in the last week, it would automatically achieve the Bronze level, since it's passing the other four rules required for that level.

This kind of gamification motivates developers to not only progress through the levels, but to maintain the quality of their services and resources over time.

You can add as many levels as you want to a Scorecard. You can also add as many rules to each level as makes sense for the Scorecard, but keep in mind that an entity must pass all rules in a given level in order to progress to the next one. You want to make sure each level is achievable, even if challenging, so developers stay motivated.

Starting with a Scorecard template

Scorecard templates are a great way to get started quickly as you get more familiar with Cortex, since they come with pre-built rules and levels. To work with a Scorecard template, select any of the Scorecard templates built into Cortex, and click Continue.

From there, you’ll be taken to an overview page where you can view the Integrations used by the Scorecard, as well as the template’s Levels and rules.

If you don't have the necessary integrations established for the Scorecard template, you'll need to configure them in Settings first.

On the next page, you have the opportunity to edit the Scorecard's Name and Description. At this stage, you'll also filter entities for the Scorecard, and determine whether it will apply to services, resources, domains, or teams. Before saving your Scorecard, you can also make adjustments to the template's predefined levels.

The draft toggle allows you to keep your Scorecard private until it’s ready to be viewed more widely. While in draft status, your Scorecard will not appear in reports or send notifications, and it’ll only be visible to admins and managers, so it won’t be noisy for developers. 

If you’re testing a Scorecard, or aren’t ready for others to see it, keep the draft toggle engaged.

Viewing and editing a Scorecard

Once you select Save Scorecard, your Scorecard will automatically evaluate all the entities that apply based on the filters you set up in Apply to specific entities, and you'll be taken to the Scorecard's page.

At the top of the page, you'll see high-level information about the Scorecard: the Median level entities have achieved, the % at the highest level, the % without level, and the number of Services. You'll also be able to find the Scorecard's description and its identifier from this page, as well as which entities the Scorecard applies to.

In the Scores tab, you can see at a glance which entities have achieved which levels, and how many rules remain before that entity can progress to the next level. 

Click on any entity name to get more information about its progress. In the sidebar, you can see the specific tasks the entity must pass to advance, the rules it's currently failing and passing, and its progress over time.

This makes it easy for developers to understand exactly what they should be doing, and lays out the priorities in a clear way. The visibility enabled by Scorecards then allows developers to target specific entities for improvement.

In the Rules tab, you can find all of the Scorecard's rules laid out for easily readability. 

Click the carat next to any rule to see the expression behind the rule. Note that if you haven't given a rule a human-readable name, the expression will appear instead of a rule name.

From the Reports tab, you can automatically generate a Bird's Eye report, Progress report, or Report Card based on that Scorecard.

From the Rule Exemptions tab, you can see exemptions that have requested for specific entities. With rule exemptions, entities are not marked as passing or failing — the rule simply does not apply to those entities.

If you're the owner of the Scorecard, then you can also Deny or Approve requested exemptions, as well as Revoke Exemptions that have been granted.

Editing a Scorecard

To edit a Scorecard, click Edit on that Scorecard's home page, above the description.

The Edit Scorecard page is identical to the Create Scorecard page. You can modify your Scorecard’s name and description, edit levels, add or delete rules, and toggle the draft status. You can also adjust the filter criteria and change the services or resources being evaluated by the Scorecard to fine-tune your insights over time.

Any time you edit and save your Scorecard, Cortex will automatically reevaluate the relevant entities to reflect your changes.