Project 3: Reviewed Change to Staged Rollout

The business problem

Scenario

The team already reviews changes in Git and already stages rollouts through a canary gate -- separately. Production Automation's own module notes state this plainly: its rollout code assumes a change is "already reviewed and approved" but never actually reads that approval from anywhere real. In practice, this leaves a gap: what actually rolls out to the canary is whatever the operator typed into the rollout script that day, not provably the same thing that was reviewed and merged. This project closes exactly that gap.

Requirements: the configuration applied during rollout must come directly from a real, committed Git change -- not be retyped or reconstructed; the canary/gate/expand-or-abort mechanics must work exactly as Production Automation already proved they do; both a successful and a failed rollout must be demonstrated for real; the final report must show the change's Git evidence and its rollout outcome together, as one artifact.

Quick checkConcept check

What specific gap does this project close that neither Git / NetDevOps nor Production Automation closed on its own?

It adds a completely new rollout mechanism not seen beforeIt connects the two: the configuration a staged rollout actually applies is read directly from a real Git commit, instead of Production Automation's rollout code assuming -- without checking -- that a review already happenedIt replaces Git with a different review systemIt removes the need for a canary stage entirely

Skills reused — not re-taught

Prerequisites, referenced only

This project assumes you can already do the following without a refresher: commit a file with GitPython (Git / NetDevOps), filter a Nornir inventory with nr.filter(), run napalm_configure()/napalm_get() as a composed apply-then-validate stage, evaluate a pass/fail threshold, and trigger napalm_rollback() (all Production Automation). None of those mechanics are re-explained below.

Approved changea real Git commit, not a hardcoded string
Canary selectionnr.filter(role="canary") -- Production Automation
Apply stagenapalm_configure() -- read from the commit itself
Validatenapalm_get() -- does the applied config match?
Gate decision
PASS -> expand to the rest of the fleetsame configuration, same commit
FAIL -> abort -> rollbacknapalm_rollback() on the canary only
One audit report: Git evidence + rollout outcome, together

Quick checkScenario

Why does reading the configuration directly from the Git commit object (rather than a hardcoded Python string, as in Production Automation's own capstone) matter here?

It doesn't matter -- either approach produces identical behaviorIt's the only way to prove that what gets rolled out is genuinely the same thing that went through review and got merged -- a hardcoded string could silently drift from what was actually approvedGit commits execute faster than string literalsNAPALM requires configuration to come from a Git object

The real script

rollout_lib.py

import os
import git
from nornir import InitNornir
from nornir_napalm.plugins.tasks import napalm_get, napalm_configure, napalm_rollback

CHANGE_FILE = "sw-fleet.cfg"


def init_change_repo(repo_path):
  repo = git.Repo.init(repo_path)
  with repo.config_writer() as cw:
      cw.set_value("user", "name", os.environ.get("GIT_AUTHOR_NAME", "Network Automation Curriculum"))
      cw.set_value("user", "email", os.environ.get("GIT_AUTHOR_EMAIL", "automation@example.local"))
  return repo


def record_approved_change(repo, config_line, message):
  # Stands in for a change that already went through Git / NetDevOps's
  # real branch-diff-merge review and landed on trunk. This project
  # doesn't re-run that workflow -- it picks up at its result: one
  # real, committed change, ready to roll out.
  path = os.path.join(repo.working_tree_dir, CHANGE_FILE)
  with open(path, "a") as f:
      f.write(config_line)
  repo.index.add([CHANGE_FILE])
  return repo.index.commit(message)


def read_approved_change(repo):
  # Reads the configuration directly from the commit object itself, not
  # the working-tree file -- proof that what rolls out is exactly what
  # was reviewed and merged, not a retyped copy of it.
  commit = next(repo.iter_commits(max_count=1))
  blob = commit.tree / CHANGE_FILE
  configuration = blob.data_stream.read().decode("utf-8")
  return commit, configuration


def load_inventory(inventory_dir):
  return InitNornir(
      inventory={
          "plugin": "SimpleInventory",
          "options": {
              "host_file": f"{inventory_dir}/hosts.yaml",
              "group_file": f"{inventory_dir}/groups.yaml",
              "defaults_file": f"{inventory_dir}/defaults.yaml",
          },
      },
  )


def _apply_and_validate(task, configuration, expected_line):
  task.run(task=napalm_configure, configuration=configuration)
  task.run(task=napalm_get, getters=["get_config"])


def stage_change(nr_subset, configuration, expected_line):
  agg = nr_subset.run(task=_apply_and_validate, configuration=configuration, expected_line=expected_line)
  stage_result = {}
  for hostname, multi_result in agg.items():
      if multi_result.failed:
          stage_result[hostname] = {"applied": False, "validated": False}
          continue
      applied = not multi_result[1].failed
      running_config = multi_result[2].result["get_config"]["running"]
      validated = expected_line.strip() in running_config
      stage_result[hostname] = {"applied": applied, "validated": validated}
  return stage_result


def evaluate_gate(stage_result, threshold=1.0):
  total = len(stage_result)
  passed = sum(1 for r in stage_result.values() if r["applied"] and r["validated"])
  pass_rate = passed / total if total else 0.0
  return "PASS" if pass_rate >= threshold else "FAIL"


def rollout(nr, repo, expected_line, threshold=1.0):
  commit, configuration = read_approved_change(repo)

  canary = nr.filter(role="canary")
  rest = nr.filter(role="rest")

  canary_result = stage_change(canary, configuration, expected_line)
  decision = evaluate_gate(canary_result, threshold=threshold)

  report = {
      "change_commit": commit.hexsha[:8],
      "change_message": commit.message.strip(),
      "canary": canary_result,
      "decision": decision,
      "expanded": {},
      "rolled_back": False,
  }

  if decision == "PASS":
      report["expanded"] = stage_change(rest, configuration, expected_line)
  else:
      rollback_agg = canary.run(task=napalm_rollback)
      report["rolled_back"] = not rollback_agg.failed

  return report

The complete, real, verified library. stage_change()/evaluate_gate() intentionally mirror Production Automation's own run_stage()/evaluate_gate() shape -- the new code here is record_approved_change(), read_approved_change(), and rollout()'s use of a real commit as its configuration source.

Quick checkIdentify the correct code

What does `commit.tree / CHANGE_FILE` actually do in read_approved_change()?

A syntax error -- Python doesn't support dividing objectsGitPython overloads the / operator on a commit's tree object to look up a file by path inside that specific commit -- retrieving its committed content directly from Git's object database, not the working directoryIt calculates the file's size in the commitIt deletes the file from the commit

Scenario A — real, verified: canary passes, rollout expands

terminal output — real, verified run

--- Approved change (Git evidence) ---
7df41b5b  Add guest VLAN uplink description to SW fleet (reviewed and merged)

--- Rollout report (Scenario A: canary passes) ---
{
"change_commit": "7df41b5b",
"change_message": "Add guest VLAN uplink description to SW fleet (reviewed and merged)",
"canary": {
  "SW1": {
    "applied": true,
    "validated": true
  }
},
"decision": "PASS",
"expanded": {
  "SW2": {
    "applied": true,
    "validated": true
  },
  "SW3": {
    "applied": true,
    "validated": true
  }
},
"rolled_back": false
}

Real, verified. SW1's canary stage both applied and validated correctly, the gate decided PASS, and the same reviewed configuration expanded to SW2 and SW3 -- with the exact commit hash and message carried through into the report.

Quick checkPredict output

In Scenario A's report, why does "change_commit" show the same value "7df41b5b" that also appeared when the change was first committed, rather than a new hash generated during rollout?

This is a coincidencerollout() calls read_approved_change(), which reads the existing commit rather than creating a new one -- the rollout consumes an already-committed change, it doesn't create oneThe rollout process re-commits the change, producing an identical hash by chanceGit hashes are always the same across different commits

Scenario B — real, verified: canary fails, aborts, rolls back

Not just the happy path

A separate fixture scenario represents the canary's commit succeeding but its post-change validation failing -- the same real distinction Production Automation's own capstone drew between "applied" and "validated." No fixtures exist for SW2/SW3 in this scenario, so if the code incorrectly tried to expand after a failed gate, it would fail loudly with a real error rather than silently succeeding.

terminal output — real, verified run

--- Approved change (Git evidence) ---
21ab168d  Add guest VLAN uplink description to SW fleet (reviewed and merged)

--- Rollout report (Scenario B: canary gate fails) ---
{
"change_commit": "21ab168d",
"change_message": "Add guest VLAN uplink description to SW fleet (reviewed and merged)",
"canary": {
  "SW1": {
    "applied": true,
    "validated": false
  }
},
"decision": "FAIL",
"expanded": {},
"rolled_back": true
}

Real, verified. SW1's change was applied (committed) but validation failed -- the deployed config didn't actually reflect the expected line. The gate correctly decided FAIL, expansion never happened (an empty "expanded"), and rollback genuinely succeeded.

Quick checkScenario

In Scenario B, "expanded" is an empty object. What would you need to see to be confident this means "the rest of the fleet was genuinely never touched" rather than "the code just forgot to record it"?

Nothing further is needed -- an empty object is always sufficient proofThat no fixture files exist for SW2/SW3 in this scenario, so any code path that mistakenly tried to reach them would raise a real, visible error instead of silently returning an empty resultA screenshot of the empty objectRe-running the script a second time

What this does and doesn't prove

Never claimed here

This ran against NAPALM's real mock driver and a real local Git repository -- never a real device, real production fleet, or real GitHub/GitLab remote. The integration itself (a rollout reading its configuration from a real Git commit) is genuinely proven; nothing here claims this was tested against a live network.

Deliberately narrower than the Capstone

This project touches two modules (Git / NetDevOps and Production Automation), one change type, and a three-host fleet -- not the full curriculum. The Capstone, next, is reserved for the first genuinely cumulative project.

How to present this in your portfolio

Portfolio guidance

Problem statement: "Our staged rollout process couldn't prove that what it deployed to the canary was the same thing that got reviewed and approved." Architecture: a real Git commit as the single source of truth for the change, read directly from the commit object, driving Production Automation's already-proven canary/gate/rollback mechanics. Evidence: both scenarios above, in full -- a reviewer will trust "both outcomes shown" far more than only the success path. Limitations to state honestly: mock driver, three-host fleet, one change type; no real GitHub/GitLab remote or CI/CD trigger. Extension ideas: trigger the rollout automatically on a merge to trunk, or extend the audit report to include the original review diff alongside the rollout outcome.

Key takeaways

The gap this project closes: Git / NetDevOps never touched rollout; Production Automation assumed review happened without reading it from anywhere real.
read_approved_change() reads configuration directly from a Git commit object (commit.tree / filename) -- proof the rollout uses exactly what was reviewed, not a retyped copy.
Scenario A, real and verified: canary applies and validates, gate decides PASS, the same reviewed change expands to the rest of the fleet.
Scenario B, real and verified: canary applies but fails validation, gate decides FAIL, expansion never runs, rollback genuinely succeeds.
Deliberately narrower than the Capstone -- two modules, one change type, a three-host fleet, not the full curriculum.

What you'll learn