Acting on Impact Results

Once VEKTIS calculates an impact score, the question shifts from “is the change real?” to “what do I do about it?” This guide walks through what each impact zone means for your decisions, plus how to share results in a way that lands.

If you haven’t seen impact scores yet, start with Understanding Your Results — this page assumes you know how the score is computed.

The decision frame

Every post-release measurement gives you two pieces of information:

• Strength — the impact score (0–100), grouped into three zones: Significant, Confirmed, Signal Detected.
• Direction — whether the change moved toward or away from your target.

Combine those two and you get four practical situations. Each calls for a different response.

Situation 1: Significant Impact, target direction (green, 70–100)

What it means: Strong evidence your feature moved the metric in the direction you wanted. The change is well outside normal fluctuation — it’s almost certainly real.

What to do:

• Replicate the pattern. Look at what you did and ask whether the same approach applies to adjacent surfaces in your product. The decisions, mechanics, and tradeoffs that worked here often transfer.
• Lock in the win. If this was an experiment or A/B test, ship it to 100% if you haven’t already. If you measured a feature at full rollout, document the design choices that drove the result so the next person who touches that surface doesn’t undo them.
• Tell the people who fund this work. A green Significant score with a clear delta is the strongest impact signal VEKTIS can produce — it’s the moment to share with your manager, your stakeholders, and your customers if appropriate. See Sharing impact below.
• Keep measuring for a few weeks. A Significant score on day 3 can soften by week 6 as novelty effects wear off. Plan to add 2–3 more post-release measurements over the following month so you know the impact is sustained, not just a launch spike.

Tip

Significant impact in the right direction is the rarest outcome in measurement work. Most features land in Confirmed or Signal Detected. When you see a green 70+, it’s worth taking time to understand WHY — that learning is more valuable than the metric itself.

Situation 2: Significant Impact, wrong direction (red, 70–100 negative)

What it means: Strong evidence the metric moved opposite your target. Something significant happened, and it’s likely your feature.

What to do:

• Treat it as urgent but not panic-worthy. A single measurement on day 3 isn’t a final verdict, but it’s enough signal to warrant immediate investigation.
• Form a hypothesis fast. Common explanations: the feature is harder to use than you expected; it broke something adjacent; user behavior shifted in a way you didn’t anticipate; a measurement issue (different data source, methodology change). Pick the most likely one and investigate in code or product before adding more measurements.
• Consider whether to roll back. If the metric you’re tracking ties directly to revenue or retention, the cost of waiting often exceeds the cost of reverting. If the metric is softer (engagement, satisfaction), it’s reasonable to add a few more measurements to confirm the trend before rolling back.
• Add more post-release measurements quickly. A single red Significant value is strong but not conclusive. Add 2–3 more measurements within the next few days. If they all stay red, your hypothesis is correct.
• Document what you learned. Even a feature that moved the metric the wrong way is information. Capture what you tried, what you predicted, and what actually happened. The hypothesis log is more valuable than any single feature.

Situation 3: Confirmed Impact (40–69)

What it means: The change is outside the normal range, but not by a wide margin. Likely a real change — but a couple more data points will tell you whether it holds.

What to do — green (target direction):

• Keep measuring before you celebrate. A Confirmed result often firms up to Significant with more data, but it can also soften back to Signal Detected. Plan to add measurements over the next 2–4 weeks.
• Don’t make irreversible decisions yet. Confirmed is enough to keep the feature shipped, but not enough to redesign your whole approach around the result.
• Look for confirming evidence in other places. Does qualitative feedback (support tickets, user interviews, sales calls) align with what the metric is showing? Triangulating across signals firms up the picture faster than waiting for the score to harden.

What to do — red (wrong direction):

• Watch closely, but don’t roll back yet. Confirmed in the wrong direction warrants paying attention but not acting on a single data point. The signal might soften.
• Investigate what could be causing the regression. Even if the score is “only” Confirmed, the cost of an unmitigated negative trend grows fast. Look at the same hypothesis space as Situation 2 — usability, breakage, behavior shift, measurement issue.
• Set a tripwire. “If the next measurement is also red and at or above this score, I’ll [roll back / redesign / pause].” Pre-committing the action prevents the slow drift of “let’s just measure one more.”

Situation 4: Signal Detected (0–39)

What it means: The change is within or near the normal range. Could be regular fluctuation, could be the early signal of real impact that hasn’t accumulated yet.

Don’t conclude “the feature didn’t work.” Signal Detected ≠ no impact. It means you don’t have enough evidence yet — either because the change is genuinely small, your baseline strength is too weak to detect a small change, or you haven’t given it enough time.

What to do:

• Add more post-release measurements over time. Patterns emerge with more data. A Signal Detected on day 3 can become a Confirmed by week 4 as the underlying change accumulates.
• Check baseline strength. If your baseline is Weak (only 2 data points), the “normal range” is estimated from very little data — small real changes can hide in the noise. Add more baselines if you can; otherwise, more post-release measurements will help.
• Consider whether the metric you picked is the right one. A feature can have real impact on a different metric than the one you targeted. If qualitative feedback says the feature is working but the numbers don’t move, you might be measuring the wrong surface.
• Set a “no impact” threshold for yourself. “If after 4 weeks and 6 post-release measurements the score is still under 40, I’ll conclude this feature didn’t move the needle.” Pre-committing saves you from indefinite measuring.

Note

Signal Detected is the most common outcome in product work. Most features don’t move the needle in a way that’s statistically clear. That’s not a failure — it’s a calibration. The features that DO produce Significant scores are the ones worth pattern-matching on.

When the data is mixed

Real measurement is messy. You’ll often see:

• One Significant green, then three Signal Detecteds. The first measurement caught a launch spike; the subsequent ones reflect steady state. Steady state is what matters for long-term decisions.
• Confirmed green on one metric, Confirmed red on another. Pick the metric that ties most directly to the business outcome you’re trying to drive. If your feature improves engagement but reduces conversion, the conversion answer probably matters more.
• Significant impact with a Weak baseline. The score might be inflated. Add baseline measurements retroactively if you can (using historical data from before release), or treat the score as suggestive rather than conclusive.

Sharing impact

The moment metrics become useful is the moment they leave your screen and reach a decision-maker. Three patterns that work:

The single-sentence summary

For a Significant green result, lead with the simplest possible framing:

Checkout completion rate moved from 12.3% to 15.1% after we shipped the simplified flow. VEKTIS rates this as Significant Impact.

The number, the direction, the feature, and the score — in one sentence. Anyone in your organization can read this without needing to know what a baseline is or how impact scores work.

The before-and-after view

For more context, expand to a tiny table:

Metric	Before	After	Impact
Checkout completion rate	12.3% (avg of 6 baselines)	15.1% (avg of 4 post-release)	Significant (green)

Three columns is enough — before, after, verdict. Resist adding more.

The honest middle ground

For a Confirmed or Signal Detected result, don’t overclaim:

Initial post-release data on the new checkout flow looks promising — Confirmed Impact in the right direction. Two more weeks of measurements will tell us whether the pattern holds.

Confidence-shaping language (“looks promising”, “initial data”, “will tell us”) is more credible than overstating a soft signal. Decision-makers learn whether you’re a reliable narrator faster than they learn the underlying numbers.

What NOT to do

• Don’t share a screenshot of one Signal Detected result and frame it as “the feature didn’t work.” One data point isn’t enough evidence. If you’ve decided the feature didn’t deliver, say so based on the pattern across multiple measurements, not a single early one.
• Don’t cherry-pick measurements. If you have 6 post-release values and one of them shows Significant green while the other 5 are Signal Detected, the honest summary is the latter, not the former.
• Don’t make the score the headline. The score is a tool for understanding what the data is telling you. The headline should be the decision or insight that follows from it: “We’re rolling this out to 100%” or “We’re going to investigate why this regressed” or “We need more data before deciding.”
• Don’t compare scores across Dev Items as if they’re commensurable. A Significant score on one feature isn’t “twice as good” as a Confirmed score on another. The score reflects strength of evidence about a specific change, not the magnitude of business value.

Common questions

Should I act on the first post-release measurement? For Significant red (wrong direction) results, yes — investigate immediately. For everything else, no — wait for 2–3 measurements before making decisions you can’t easily reverse.

How long should I keep measuring? Until either (a) the result has stabilized (3 consecutive measurements in the same zone with the same direction), or (b) you’ve collected at least 4–6 post-release measurements, whichever comes later. After that, the marginal information from another data point is small.

What if I shipped multiple features in the same release? Impact scores can’t isolate which feature drove the change — they only show that a change happened. Either ship features in separate releases when measurement matters, or accept that the impact you measure is the joint effect.

Related

• Understanding Your Results — how the score is computed
• Impact Scores — reference for the three zones
• Measuring After You Ship — adding post-release data points
• Troubleshooting — what to check when impact didn’t calculate