Advanced Project

Multi-Device Compliance Report

Advanced65 min

Objective

Build a data-driven compliance engine that evaluates every device in a fleet against a shared set of policy rules, safely isolating a rule failure caused by missing device data from a device record too malformed to evaluate at all.

Builds on Multi-vendor Automation and Reliable Automation and Real-world Projects.

Prerequisites

Comfortable with dictionaries, lambda functions, and iterating a list of rule definitions
Familiar with the configuration-vs-state distinction (Configuration vs. State)
Familiar with structured, per-device results that isolate one device's problem from the rest (Multi-Device Partial Failure and Structured Results)

Deliverables

A reusable rule-evaluation function that checks one device against one policy rule safely, even when the field a rule needs is missing
A fleet-wide evaluation that separates devices too malformed to evaluate from devices that were evaluated and found non-compliant
Per-rule failure counts across the fleet, not just a per-device pass/fail verdict
One structured fleet compliance report a stakeholder could read without re-deriving anything by hand

Local/offline only. The fleet and compliance rules below are fixed sample data -- no NetBox, no live device scan, no real policy system.

The problem

Not the same task as the existing interface or source-of-truth labs

"Interface Status Audit" checks three fixed, hardcoded conditions against interface records. "Investigating a Source-of-Truth Compliance Gap" reconciles two separate device lists. This project is neither: it's a small policy engine -- the rules themselves are data, not separate hand-written functions, so the same evaluation logic applies uniformly no matter how many rules exist or what they check.

A fleet needs to be checked against a shared compliance policy: NTP configured, logging configured, a login banner present, and a minimum OS version. Some devices pass everything. Some fail one rule. One device is missing a field a rule needs entirely. And one record in the fleet data is so malformed it can't be evaluated at all -- it never even names which device it is. Your job: build an evaluator that treats the rules as a list to iterate, not a list of special cases to hand-code, and that tells the difference between "failed a rule" and "couldn't be evaluated" instead of crashing on the second one.

Starter data

FLEET -- safe sample data

FLEET = [
  {"name": "RTR1", "ntp_servers": ["10.50.10.2"], "logging_hosts": ["10.50.10.50"], "banner_configured": True, "os_version": "15.4"},
  {"name": "RTR2", "ntp_servers": [], "logging_hosts": ["10.50.10.50"], "banner_configured": True, "os_version": "15.4"},
  {"name": "SW1", "ntp_servers": ["10.50.10.2"], "logging_hosts": ["10.50.10.50"], "banner_configured": False, "os_version": "12.2"},
  {"name": "SW2", "ntp_servers": ["10.50.10.2"], "logging_hosts": ["10.50.10.50"], "banner_configured": True, "os_version": None},
  {"site": "branch2", "banner_configured": True},
]

Fixed sample data. RTR1 passes every rule; RTR2 has no NTP server configured; SW1 has no banner and an old OS version; SW2 is missing its os_version entirely; the last record has no 'name' field at all -- it isn't even a usable device record.

COMPLIANCE_RULES -- the policy, as data

COMPLIANCE_RULES = [
  {"rule_id": "ntp-configured", "description": "At least one NTP server must be configured",
   "check": lambda d: bool(d.get("ntp_servers"))},
  {"rule_id": "logging-configured", "description": "At least one logging host must be configured",
   "check": lambda d: bool(d.get("logging_hosts"))},
  {"rule_id": "banner-configured", "description": "A login banner must be configured",
   "check": lambda d: d.get("banner_configured") is True},
  {"rule_id": "min-os-version", "description": "OS version must be at least 15.2",
   "check": lambda d: version_at_least(d.get("os_version"), "15.2")},
]

Each rule is a dict with an id, a description, and a check function -- your evaluator iterates this list generically, without knowing what any individual rule actually checks.

Milestones

Load the fleet and the compliance rule setrules as data, not hardcoded functions
Normalize the version comparisonsafely handle a missing os_version
Evaluate one device against every rulecollect every failed rule, not just the first
Separate malformed records from evaluable onesa missing 'name' field is a different problem
Evaluate the whole fleetaggregate per-rule failure counts across every device
Produce the final fleet compliance reportcompliant / non-compliant / skipped, plus rule detail

Task

COMPLIANCE_RULES is given -- write three functions:

version_at_least(version, minimum) -- return False if version is None. Otherwise, parse both version and minimum as dotted version strings (e.g. "15.4" → (15, 4)) and return whether version's tuple is greater than or equal to minimum's.
evaluate_device(device, rules) -- for every rule in rules, call rule["check"](device). Collect the rules that returned False into a failed_rules list (each entry: {"rule_id": ..., "description": ...}). Return {"device": device["name"], "compliant": ..., "failed_rules": ...}.
evaluate_fleet(fleet, rules) -- for every record in fleet: if it has no "name" field, add it to a skipped list with a reason instead of evaluating it. Otherwise, evaluate it with evaluate_device. Combine the results into {"summary": {...}, "devices": [...], "skipped": [...]}, where summary includes total_devices, evaluated, skipped, compliant, non_compliant, and a rule_failure_counts dict (one entry per rule ID, counting how many evaluated devices failed it).

starter.py

FLEET = [
  {"name": "RTR1", "ntp_servers": ["10.50.10.2"], "logging_hosts": ["10.50.10.50"], "banner_configured": True, "os_version": "15.4"},
  {"name": "RTR2", "ntp_servers": [], "logging_hosts": ["10.50.10.50"], "banner_configured": True, "os_version": "15.4"},
  {"name": "SW1", "ntp_servers": ["10.50.10.2"], "logging_hosts": ["10.50.10.50"], "banner_configured": False, "os_version": "12.2"},
  {"name": "SW2", "ntp_servers": ["10.50.10.2"], "logging_hosts": ["10.50.10.50"], "banner_configured": True, "os_version": None},
  {"site": "branch2", "banner_configured": True},
]


def version_at_least(version, minimum):
  # Your code here. Handle version being None safely.
  ...


COMPLIANCE_RULES = [
  {"rule_id": "ntp-configured", "description": "At least one NTP server must be configured",
   "check": lambda d: bool(d.get("ntp_servers"))},
  {"rule_id": "logging-configured", "description": "At least one logging host must be configured",
   "check": lambda d: bool(d.get("logging_hosts"))},
  {"rule_id": "banner-configured", "description": "A login banner must be configured",
   "check": lambda d: d.get("banner_configured") is True},
  {"rule_id": "min-os-version", "description": "OS version must be at least 15.2",
   "check": lambda d: version_at_least(d.get("os_version"), "15.2")},
]


def evaluate_device(device, rules):
  # Your code here.
  ...


def evaluate_fleet(fleet, rules):
  # Your code here.
  ...


if __name__ == "__main__":
  report = evaluate_fleet(FLEET, COMPLIANCE_RULES)
  print(report["summary"])

Copy into a local .py file and run with plain python3 -- no install needed. version_at_least(), evaluate_device(), and evaluate_fleet() are intentionally incomplete.

Hints

Expected output

terminal output -- real, verified run

--- Fleet Compliance Report ---
Total devices:   5
Evaluated:       4
Skipped:         1
Compliant:       1
Non-compliant:   3

RTR1: compliant
RTR2: non-compliant (ntp-configured)
SW1: non-compliant (banner-configured, min-os-version)
SW2: non-compliant (min-os-version)

Rule failure counts: {'ntp-configured': 1, 'logging-configured': 0, 'banner-configured': 1, 'min-os-version': 2}

Skipped: missing required 'name' field -- {'site': 'branch2', 'banner_configured': True}

Real, verified output from a working solution against the exact starter data above. Your own report's formatting can differ, but the counts and per-device classifications must match.

Validation checklist

Loading the sample fleet yields 5 raw device entries before any evaluation
Exactly 1 device is skipped for missing its 'name' field entirely, and is not counted as compliant or non-compliant
Exactly 4 devices are evaluated
RTR1 is the only device classified as fully compliant
RTR2 is non-compliant for exactly one reason: ntp-configured
SW1 is non-compliant for exactly two reasons: banner-configured and min-os-version
SW2 is non-compliant for exactly one reason: min-os-version, caused by a missing os_version field, not a crash
The rule_failure_counts for min-os-version equals 2 (SW1 and SW2), matching the two non-compliant devices that failed it

Self-check only -- nothing here is submitted or graded. Compare against your own output manually.

Solution

Try the task yourself first -- the full solution and explanation stay hidden until you click above.

Key takeaways

Representing policy rules as data -- a list of {rule_id, description, check} dicts -- lets one generic evaluator apply any number of rules without hand-coding a branch per rule.
'Failed a rule because a field is missing' (SW2) and 'can't be evaluated because the record itself is malformed' (the last fleet entry) are genuinely different findings, and this project's skipped list keeps the second one from silently corrupting the first.
A per-rule failure count answers a different, equally real question than a per-device pass/fail verdict does: which policy is actually causing the most fleet-wide non-compliance.
This is fixed, offline sample data -- no NetBox, live scan, or real policy system was involved.

Related lessons

Why Multi-vendor Automation Is Hard, and How nornir_napalm Closes the Gap

The normalization problem this project's rule engine solves for policy checks instead of driver selection.

Read lesson

What Makes Automation Reliable

The mindset this project applies: don't just collect device data, decide what in it actually matters.

Read lesson

Multi-Device Partial Failure and Structured Results

The exact per-device isolation this project's 'one bad record doesn't stop the fleet' handling reuses.

Read lesson

Project 2: Multi-Vendor Configuration Compliance Report

A related but different compliance project -- that one checks NAPALM-sourced device state against a single baseline; this one evaluates a whole fleet against multiple independent policy rules at once.

Read lesson

Done with this project?