Canary Rollout and Validation Gates

Connecting the earlier stages to rollout

Not re-teaching Production Automation

select_canary(), run_stage(), evaluate_gate(), and rollback_stage() use the exact mechanics Production Automation already taught and verified -- nr.filter(), napalm_configure()/napalm_get(), a threshold gate, napalm_rollback. Nothing about how those individual pieces work is re-explained here. This lesson's job is connecting them to what Lessons 1-3 actually produced.

Lesson 1's non-compliant devices + Lesson 3's approved config
select_canary()one host from the detected subset, not a static role tag
run_stage(canary, approved_config)apply, then validate against the Git-evidenced line
evaluate_gate()
PASS -> expand to the rest of the detected subsetsame approved_config
FAIL -> abort -> rollback_stage(canary)

Quick checkConcept check

How does canary selection here differ from Production Automation's own capstone?

It doesn't differ at all -- identical codeProduction Automation selected a canary via a static data.role field set in inventory ahead of time; here, select_canary() operates on the list of devices detect_noncompliance() found at runtime -- the canary is whichever affected device sorts first, not a pre-assigned tagThis lesson doesn't select a canaryThe canary is chosen randomly

The code

capstone_lib.py (Lesson 4 portion)

from nornir_napalm.plugins.tasks import napalm_rollback


def select_canary(nr, affected_hostnames):
  affected_sorted = sorted(affected_hostnames)
  canary_host = affected_sorted[0]
  rest_hosts = affected_sorted[1:]
  canary_nr = nr.filter(filter_func=lambda h: h.name == canary_host)
  rest_nr = nr.filter(filter_func=lambda h: h.name in rest_hosts)
  return canary_nr, rest_nr


def _apply_and_validate(task, configuration):
  task.run(task=napalm_configure, configuration=configuration)
  task.run(task=napalm_get, getters=["get_config"])


def run_stage(nr_subset, configuration):
  agg = nr_subset.run(task=_apply_and_validate, configuration=configuration)
  stage_result = {}
  for hostname, multi_result in agg.items():
      if multi_result.failed:
          stage_result[hostname] = {"applied": False, "validated": False}
          continue
      applied = not multi_result[1].failed
      running_config = multi_result[2].result["get_config"]["running"]
      validated = configuration.strip() in running_config
      stage_result[hostname] = {"applied": applied, "validated": validated}
  return stage_result


def evaluate_gate(stage_result, threshold=1.0):
  total = len(stage_result)
  passed = sum(1 for r in stage_result.values() if r["applied"] and r["validated"])
  pass_rate = passed / total if total else 0.0
  return "PASS" if pass_rate >= threshold else "FAIL"


def rollback_stage(nr_subset):
  return nr_subset.run(task=napalm_rollback)

Real, verified code. run_stage() validates against the "configuration" parameter it was actually given -- Lesson 3's Git-evidenced artifact, passed in directly, not a separately hardcoded expectation.

Quick checkIdentify the correct code

In run_stage(), why does validated check `configuration.strip() in running_config` instead of checking against a separate constant like REQUIRED_NTP_LINE?

There's no difference -- both would always produce the same resultChecking against the configuration parameter that was actually passed in (Lesson 3's read-back artifact) proves the stage validates against what was genuinely approved and applied -- not a second, independently-defined expectation that could silently drift from itREQUIRED_NTP_LINE doesn't exist in this moduleThis is a performance optimization

Scenario A — real, verified: canary passes, rollout expands

terminal output — real, verified run

canary: {'RTR1': {'applied': True, 'validated': True}}
decision: PASS
expanded: {
'RTR2': {'applied': True, 'validated': True},
'SW-EOS1': {'applied': True, 'validated': True}
}
rolled_back: False

Real, verified. RTR1 (canary) both applied and validated correctly; the gate decided PASS; the same approved configuration expanded to RTR2 and SW-EOS1 -- both applying and validating successfully.

Quick checkPredict output

In Scenario A, why does "expanded" contain RTR2 and SW-EOS1 but not RTR1?

This is a bug -- RTR1 should appear twiceselect_canary() already separated RTR1 into the canary group and RTR2/SW-EOS1 into the rest group -- run_stage() only runs against whichever subset it's called with, so the canary and the expansion never overlapRTR1 was excluded because it's a router, not a switchThe expansion re-applies to every device including the canary

Scenario B — real, verified: canary fails, aborts, rolls back

Not just the happy path

A separate fixture scenario represents the canary's commit succeeding but its post-change validation failing. No fixtures for the rollout stage exist for RTR2/SW-EOS1 in this scenario (they still have detection-stage fixtures, since detection touches the whole fleet) -- so if the code mistakenly tried to expand after a failed gate, it would fail loudly with a real error rather than silently succeeding.

terminal output — real, verified run

canary: {'RTR1': {'applied': True, 'validated': False}}
decision: FAIL
expanded: {}
rolled_back: True

Real, verified. RTR1's configuration was applied (committed) but validation found the required line still missing -- a genuine post-change verification failure, not a connection error. The gate correctly decided FAIL, expansion never ran, and rollback genuinely succeeded.

Quick checkScenario

In Scenario B, what specifically proves RTR2 and SW-EOS1 were never touched during rollout, rather than the report simply failing to mention them?

Nothing proves it -- it's just claimed in the reportTheir fixture directories in this scenario contain only detection-stage data (get_config.1), with no load_merge_candidate/compare_config/commit_config fixtures -- if run_stage() had mistakenly run against them, NAPALM's mock driver would have raised a real NotImplementedError instead of silently succeedingRTR2 and SW-EOS1 don't exist in this scenario's inventoryA separate lock file prevents them from being touched

Key takeaways

select_canary() operates on Lesson 1's runtime-detected affected devices, not a static inventory role -- the one genuinely new wiring in this stage.
run_stage() validates against the exact configuration parameter it receives -- Lesson 3's Git-evidenced artifact, not a separately maintained constant.
Scenario A, real and verified: canary applies and validates, gate decides PASS, the same approved configuration expands to the rest of the affected devices.
Scenario B, real and verified: canary applies but fails validation, gate decides FAIL, expansion never runs, rollback genuinely succeeds -- with fixture absence proving the rest of the fleet was truly untouched.
None of napalm_configure(), napalm_rollback(), or nr.filter()'s own mechanics are re-explained here -- all assumed from Production Automation.

What you'll learn

Connecting the earlier stages to rollout

The code

Scenario A — real, verified: canary passes, rollout expands

Scenario B — real, verified: canary fails, aborts, rolls back