Multi-Device Partial Failure and Structured Results

Three buckets isn't enough anymore

The Netmiko capstone's result model — compliant, non-compliant, unreachable — was honest and useful at the time. It's too coarse now: this module's tool can retry, skip an unnecessary change, preview a change, or make one. A device that succeeded on the third attempt and one that succeeded on the first look identical in a 3-bucket model. This lesson builds a result that actually reflects what happened.

Combining everything into one run

Lesson 2's call_with_retry() took the connection function as an argument — flexible, but this project only ever retries one specific kind of call: getting a device's connection. Here's that same retry pattern, specialized to exactly that job:

connect_with_retry.py

def connect_with_retry(device, max_attempts=3, delay=2):
  """The same retry policy from Lesson 2, applied specifically to get_connection()."""
  for attempt in range(1, max_attempts + 1):
      try:
          return get_connection(device), attempt
      except TimeoutError:
          if attempt == max_attempts:
              raise
          time.sleep(delay)
      except PermissionError:
          raise

Same bounded, selective retry logic as call_with_retry() — it just calls a fixed get_connection() internally instead of taking a function as an argument, and returns the attempt count alongside the connection so run_device() below can record it.

run_device.py

def run_device(device, dry_run=True):
  hostname = device["hostname"]
  try:
      connection, attempts = connect_with_retry(device)
  except TimeoutError:
      logging.warning(f"{hostname}: unreachable after 3 attempts")
      return {"hostname": hostname, "status": "unreachable", "attempts": 3, "retried": True}
  except PermissionError:
      logging.warning(f"{hostname}: authentication failed")
      return {"hostname": hostname, "status": "auth-failed", "attempts": 1, "retried": False}

  outcome = apply_ntp_config(connection, REQUIRED_NTP_SERVER, dry_run=dry_run)
  retried = attempts > 1

  if outcome["dry_run"] and not outcome["changed"] and outcome["verified"] is None:
      status = "dry-run-preview"
  elif outcome["changed"]:
      status = "changed" if outcome["verified"] else "change-not-verified"
  else:
      status = "already-compliant"

  logging.info(f"{hostname}: {status} (attempts={attempts})")
  return {"hostname": hostname, "status": status, "attempts": attempts, "retried": retried}

connect_with_retry() is the function shown just above; apply_ntp_config() is Lesson 3 and 4's pattern, dry-run included. This function's only new job is turning their combined results into one status per device — five possible outcomes now, not three.

Quick checkConcept check

Why does run_device() need attempts and retried in its result, when the old 3-bucket model never tracked that?

It doesn't actually matter — extra fields are just clutterBecause a device that needed retries succeeded differently than one that connected cleanly on the first try — that distinction is exactly the kind of detail an audit-friendly report should preservePython requires every dictionary to have the same fieldsattempts and retried are required by the logging module

Logging: beyond print()

setup_logging.py

import logging

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")

logging.info("SW1: already-compliant (attempts=1)")
logging.warning("SW4: unreachable after 3 attempts")

basicConfig() sets this up once, at the start of a script. logging.info() and logging.warning() both timestamp every line automatically and label its severity — a small step up from print() that matters the moment someone needs to reconstruct what happened, and when.

Why this matters for automation specifically

A report that only says "SW4: unreachable" is missing the one piece of information someone reviewing it later actually needs first: when. logging.basicConfig()'s timestamp is exactly that — audit-friendly means someone can reconstruct the sequence of events afterward, not just the final outcome.

Quick checkIdentify the correct code

Which line correctly logs a warning-level message with an automatic timestamp, using Python's logging module?

print(f"WARNING: {hostname} unreachable")logging.warning(f"{hostname}: unreachable after 3 attempts")logging.print(f"{hostname}: unreachable")warning.log(hostname)

Tracing a mixed run

Five devices, five different real outcomes — this is what a realistic run actually looks like, not an edge case:

mixed_run_output

2026-08-25 22:22:59 INFO SW1: already-compliant (attempts=1)
2026-08-25 22:22:59 INFO SW2: dry-run-preview (attempts=1)
2026-08-25 22:22:59 INFO SW3: dry-run-preview (attempts=3)
2026-08-25 22:22:59 WARNING SW4: unreachable after 3 attempts
2026-08-25 22:22:59 WARNING SW5: authentication failed

This is real, actually-run output: SW1 was already correct, SW2 would need a change (previewed, not applied), SW3 needed two retries before it even got that far, SW4 never responded across 3 attempts, and SW5's credentials were rejected — five devices, five distinct, honestly-recorded outcomes.

Quick checkScenario

In the run above, what happens when device SW4 is unreachable after 3 attempts?

The entire script stops, and SW5 is never checkedSW4 is recorded as 'unreachable' and the run continues — SW5 still gets checked next, exactly as the non-stopping loop from earlier modules already establishedSW4 is silently skipped with no record at allSW4 is retried indefinitely until it responds

Quick checkMyth check

Is a run where 2 out of 5 devices need attention (unreachable or auth-failed) a sign that the automation itself is broken?

Yes — a reliable script should have zero failures every timeNo — partial failure across a real inventory is the normal, expected outcome; the goal is handling and reporting it honestly, not eliminating itYes, but only if more than half the devices failNo — failures should always be hidden from the final report

Key takeaways

A 3-bucket result model (compliant/non-compliant/unreachable) can't represent retries, dry-run previews, or verified changes — this lesson's richer outcome model can.
run_device() combines Lesson 2's retry wrapper and Lessons 3–4's pre-check/change/post-check/dry-run pattern into one per-device result.
Python's logging module adds automatic timestamps and severity levels — a real step toward audit-friendly output, not just print() with extra words.
A realistic run produces a genuine mix of outcomes across devices — that's the normal case this module is built to handle, not a rare edge case.
Every piece built in this lesson is what Lesson 6 assembles into the complete, reliable capstone.

What you'll learn

Three buckets isn't enough anymore

Combining everything into one run

Logging: beyond print()

Tracing a mixed run