# Autonomous Fault Detection & Dynamic Reallocation > Telemetry monitoring for silent heartbeat timeouts, battery exhaustion, and kinematic stalls, coupled with dynamic task revocation, avoidance pheromone deposition over disabled units, and immediate peer reallocation. - Repository: ivaziz/swarm_car_search - GitHub: https://github.com/ivaziz/swarm_car_search - Human wiki: https://grok-wiki.com/public/wiki/ivaziz-swarm-car-search-66a3d00cb601 - Complete Markdown: https://grok-wiki.com/public/wiki/ivaziz-swarm-car-search-66a3d00cb601/llms-full.txt ## Source Files - `swarm/failure_handling/failure_detector.py` - `swarm/failure_handling/recovery_manager.py` - `swarm/core/robot_agent.py` - `swarm/task_allocation/task_manager.py` - `tests/test_failure_handling.py` ---
Relevant source files The following files were used as context for generating this wiki page: - [swarm/failure_handling/failure_detector.py](swarm/failure_handling/failure_detector.py) - [swarm/failure_handling/recovery_manager.py](swarm/failure_handling/recovery_manager.py) - [swarm/core/robot_agent.py](swarm/core/robot_agent.py) - [swarm/task_allocation/task_manager.py](swarm/task_allocation/task_manager.py) - [swarm/core/swarm_coordinator.py](swarm/core/swarm_coordinator.py) - [swarm/core/swarm_state.py](swarm/core/swarm_state.py) - [swarm/pheromone/pheromone_updater.py](swarm/pheromone/pheromone_updater.py) - [tests/test_failure_handling.py](tests/test_failure_handling.py)
# Autonomous Fault Detection & Dynamic Reallocation In distributed multi-robot search missions, individual agents navigate dense, unpredictable urban terrain where physical obstacles, radio frequency (RF) shadowing, mechanical drivetrain failures, and power depletion are persistent operational risks. Without autonomous fault handling, a single immobilized or disconnected robot can create persistent blind spots, hold uncompleted frontier tasks indefinitely, and induce catastrophic bottlenecks across the fleet. The fault tolerance subsystem in `swarm_car_search` resolves these challenges through a closed-loop resilience architecture. The system pairs continuous multi-modal telemetry monitoring in [`FailureDetector`](swarm/failure_handling/failure_detector.py) with rapid remediation in [`RecoveryManager`](swarm/failure_handling/recovery_manager.py). When an agent ceases transmission, exhausts its battery reserve, gets entrapped, or throws an unrecoverable internal exception, the framework revokes task commitments, marks the disabled unit as a physical obstacle using high-potency stigmergic avoidance pheromones, and immediately dispatches orphaned objectives to the most suitable active peers. --- ## System Architecture & Resilience Pipeline The fault detection and recovery pipeline operates across both centralized coordination loops and decentralized peer-to-peer state tracking. It monitors high-frequency telemetry, identifies failure patterns against configurable thresholds, and executes sequential remediation without human intervention. ```mermaid flowchart TD subgraph Fleet ["Robot Fleet Layer (swarm/core/robot_agent.py)"] R1["Robot Agent 1\n(Active / Stalled)"] R2["Robot Agent 2\n(Idle Candidate)"] R3["Robot Agent 3\n(Navigating Candidate)"] HB["Heartbeat & Pose Telemetry\n(create_heartbeat)"] R1 --> HB end subgraph Monitoring ["Telemetry Ingestion & Anomaly Detection (swarm/failure_handling/failure_detector.py)"] FD["FailureDetector\n(record_telemetry / check_failures)"] HB --> FD FD --> T1{"Heartbeat Silent?\n(> 6.0s)"} FD --> T2{"Battery Critical?\n(< 10%)"} FD --> T3{"Kinematic Stall?\n(Delta pos < 0.3m for 8.0s)"} FD --> T4{"State == FAILED?\n(Explicit Fault)"} end subgraph Recovery ["Recovery & Dynamic Reallocation (swarm/failure_handling/recovery_manager.py)"] RM["RecoveryManager\n(handle_failure)"] T1 -- Yes --> FE["RobotFailureEvent"] T2 -- Yes --> FE T3 -- Yes --> FE T4 -- Yes --> FE FE --> RM A1["1. Deactivate Peer\n(SwarmState.mark_peer_inactive)"] A2["2. Revoke Orphaned Task\n(TaskManager.release_task -> PENDING)"] A3["3. Deposit Avoidance Pheromone\n(tau_a kernel at wreck coords)"] A4["4. Dynamic Suitability Reallocation\n(Dist * Battery * Workload)"] RM --> A1 RM --> A2 RM --> A3 RM --> A4 end subgraph StateAndStigmergy ["State & Environment Updates"] TM["TaskManager\n(Pending Task Queue)"] PM["PheromoneMap\n(Layer: AVOIDANCE)"] SS["Decentralized SwarmState\n(Peer Registry)"] A1 --> SS A2 --> TM A3 --> PM A4 --> R2 end ``` The system operates continuously within the primary coordination loop in [`SwarmCoordinator.step_coordination()`](swarm/core/swarm_coordinator.py#L226-L281), while individual agents maintain local peer watchdogs via [`SwarmState.check_peer_timeouts()`](swarm/core/swarm_state.py#L96-L109). Sources: [swarm/core/swarm_coordinator.py:226-281](), [swarm/failure_handling/recovery_manager.py:45-72]() --- ## Failure Detection Engine: Multi-Mode Anomaly Identification The [`FailureDetector`](swarm/failure_handling/failure_detector.py#L58-L69) ingests state telemetry (`Pose2D`, `battery_level`, `RobotState`, `assigned_task_id`, and `timestamp`) on each tick and checks against four distinct failure modalities. ``` +----------------------------------------------------------------------------------------------------+ | FAILURE DETECTION MODES | +----------------------+-----------------------------+-----------------------------------------------+ | Mode (FailureType) | Default Threshold Trigger | Root Cause & System Response | +----------------------+-----------------------------+-----------------------------------------------+ | HEARTBEAT_TIMEOUT | silence > 6.0s | RF blackout, transceiver destruction, crash. | | | | Declares robot unreachable; frees its task. | +----------------------+-----------------------------+-----------------------------------------------+ | BATTERY_CRITICAL | battery_level < 0.10 (10%) | Cell exhaustion below operational reserve. | | | | Preemptively halts agent before complete death.| +----------------------+-----------------------------+-----------------------------------------------+ | STALLED | displacement < 0.3m | Physical wheel trap, urban obstacle lockup. | | | over duration > 8.0s | Triggered only in NAVIGATING or SEARCHING. | +----------------------+-----------------------------+-----------------------------------------------+ | EXPLICIT_FAULT | state == RobotState.FAILED | Internal diagnostic failure, motor burnout, | | | | or unrecoverable sensor/SLAM fault. | +----------------------+-----------------------------+-----------------------------------------------+ ``` ### Telemetry Tracking & Kinematic Stall Anchoring To prevent false-positive stall alerts when a robot is legitimately waiting in `IDLE`, pausing during `VERIFYING`, or broadcasting in `REPORTING`, stall evaluation is strictly restricted to active transit states: `RobotState.NAVIGATING` and `RobotState.SEARCHING`. The detector establishes a spatial anchor (`stall_anchor_pose`) and tracks the duration an agent remains within `stall_distance_threshold`: $$\Delta d = \sqrt{(x_{\text{current}} - x_{\text{anchor}})^2 + (y_{\text{current}} - y_{\text{anchor}})^2}$$ If $\Delta d \ge 0.3\,\text{m}$, the anchor updates to the current pose and the timer resets. If $\Delta d < 0.3\,\text{m}$ persists beyond `stall_timeout_sec` ($8.0\,\text{s}$), a `FailureType.STALLED` event is triggered. ```python # swarm/failure_handling/failure_detector.py:101-118 # Evaluate movement for stall tracking if state in (RobotState.NAVIGATING, RobotState.SEARCHING): if rec.stall_anchor_pose is None: rec.stall_anchor_pose = pose rec.stall_start_time = timestamp else: dist = math.hypot(pose.x - rec.stall_anchor_pose.x, pose.y - rec.stall_anchor_pose.y) if dist >= self.config.stall_distance_threshold: # Made sufficient displacement; reset stall anchor rec.stall_anchor_pose = pose rec.stall_start_time = timestamp else: # Idle, returning, or reporting: not considered stalled rec.stall_anchor_pose = None rec.stall_start_time = None ``` ### Failure Event Lifecycle & FSM State Representation ```mermaid stateDiagram-v2 [*] --> Monitored: record_telemetry() state Monitored { Healthy: Active Heartbeats & Valid Kinematics } Healthy --> HeartbeatTimeout: silence > heartbeat_timeout_sec Healthy --> BatteryCritical: battery < battery_critical_threshold Healthy --> KinematicStall: dist < 0.3m for > stall_timeout_sec Healthy --> ExplicitFault: state == RobotState.FAILED state DetectedFailure { HeartbeatTimeout BatteryCritical KinematicStall ExplicitFault } DetectedFailure --> FailureEventGenerated: RobotFailureEvent Instantiated FailureEventGenerated --> RecoveryExecuted: RecoveryManager.handle_failure() RecoveryExecuted --> FailedRegistry: Registered in _failed_robots FailedRegistry --> Monitored: reset_robot() (Reboot / Reset) ``` Sources: [swarm/failure_handling/failure_detector.py:16-56](), [swarm/failure_handling/failure_detector.py:119-198]() --- ## The Recovery Pipeline: Revocation, Stigmergic Marking, and Peer Dispatch When [`FailureDetector.check_failures()`](swarm/failure_handling/failure_detector.py#L119-L198) emits a [`RobotFailureEvent`](swarm/failure_handling/failure_detector.py#L25-L34), the [`RecoveryManager`](swarm/failure_handling/recovery_manager.py#L45-L72) executes a 4-step remediation protocol via `handle_failure()`. ```mermaid sequenceDiagram autonumber participant SC as SwarmCoordinator participant FD as FailureDetector participant RM as RecoveryManager participant TM as TaskManager participant PU as PheromoneUpdater / Map participant SS as SwarmStates (Peers) participant Peer as Best Surviving Peer SC->>FD: check_failures(timestamp) FD-->>SC: [RobotFailureEvent(r_fail, last_pose, task_id)] SC->>RM: handle_failure(event, task_manager, swarm_states, ...) Note over RM,SS: Step 1: Decentralized Peer Deactivation RM->>SS: mark_peer_inactive(r_fail) Note over RM,TM: Step 2: Task Revocation & Release RM->>TM: release_task(task_id, timestamp) TM-->>TM: Status -> PENDING, assigned_robot = None RM->>SS: active_tasks[task_id].mark_released(timestamp) Note over RM,PU: Step 3: Stigmergic Wreck Avoidance Deposit RM->>PU: deposit_avoidance(x, y, amount=12.0, radius=2.5) PU-->>PU: Inject radial Gaussian tau_a obstacle kernel Note over RM,Peer: Step 4: Multi-Criteria Dynamic Reassignment RM->>RM: Score surviving peers (Suitability = dist * battery * workload) RM->>Peer: assign_task(chosen_task, timestamp) RM->>TM: assign_task(task_id, peer_id, timestamp) RM-->>SC: RecoveryResult(actions=[DEACTIVATED, RELEASED, AVOIDANCE, REASSIGNED]) ``` ### 1. Peer Inactivation Across Swarm States The failed agent is marked inactive in every surviving robot's local [`SwarmState`](swarm/core/swarm_state.py#L86-L95). This purges the agent from peer availability lists, halts stale coordinate updates, and prevents peer-to-peer consensus deadlocks. ### 2. Orphaned Task Revocation If the disabled unit was navigating toward or executing an active [`SearchTask`](swarm/task_allocation/frontier_task.py), ownership is immediately revoked. The task's status is reset from `ASSIGNED` or `IN_PROGRESS` back to `PENDING`, clearing its assigned robot field: ```python # swarm/failure_handling/recovery_manager.py:90-99 if orphaned_task is not None and orphaned_task.status in (TaskStatus.ASSIGNED, TaskStatus.IN_PROGRESS): result.orphaned_task_id = orphaned_task.task_id task_manager.release_task(orphaned_task.task_id, event.timestamp) result.actions_taken.append(RecoveryAction.TASK_RELEASED) # Synchronize task release in decentralized swarm states for sstate in swarm_states.values(): if orphaned_task.task_id in sstate.active_tasks: sstate.active_tasks[orphaned_task.task_id].mark_released(event.timestamp) ``` ### 3. Wreck Avoidance Pheromone Deposition An immobilized robot in an urban corridor represents a physical obstacle. To prevent healthy robots from colliding with or queuing behind the disabled unit, the [`RecoveryManager`](swarm/failure_handling/recovery_manager.py#L101-L108) instructs the [`PheromoneUpdater`](swarm/pheromone/pheromone_updater.py#L78-L80) to deposit high-intensity avoidance pheromone ($\tau_a$) directly onto the obstacle grid: $$\tau_a(\mathbf{x}) \leftarrow \tau_a(\mathbf{x}) + A_{\text{wreck}} \cdot \exp\left(-\frac{\|\mathbf{x} - \mathbf{x}_{\text{wreck}}\|^2}{2\sigma^2}\right)$$ Default configuration parameters: - `wreck_avoidance_amount` ($A_{\text{wreck}}$): `12.0` - `wreck_avoidance_radius` ($r_{\text{avoid}}$): `2.5` meters ($\sigma = r_{\text{avoid}} / 2 = 1.25\,\text{m}$) Because the ACO path planner and task scoring engine penalize paths crossing high $\tau_a$ concentrations, surviving robots naturally navigate around the disabled vehicle. ### 4. Immediate Dynamic Reallocation with Multi-Criteria Scoring Rather than waiting for normal periodic task allocation ticks, [`RecoveryManager`](swarm/failure_handling/recovery_manager.py#L110-L165) calculates suitability scores for all active peers: $$S(r_i) = F_{\text{dist}}(r_i) \times F_{\text{batt}}(r_i) \times F_{\text{workload}}(r_i)$$ Where: - **Distance Factor**: $F_{\text{dist}}(r_i) = \frac{1.0}{1.0 + 0.05 \cdot \|\mathbf{p}_i - \mathbf{c}_{\text{task}}\|}$ - **Battery Factor**: $F_{\text{batt}}(r_i) = \max(0.1, E_{\text{batt}, i})$ - **Workload Factor**: - $1.0$ if the candidate is in `RobotState.IDLE` or `RobotState.REASSIGNING`. - $0.65$ if the candidate is in `RobotState.NAVIGATING` on background exploration (preemptible). Robots handling urgent `TaskType.TARGET_CONFIRMATION` are non-preemptible. The candidate with the highest suitability score is chosen, and the task is immediately assigned and committed in [`TaskManager`](swarm/task_allocation/task_manager.py#L120-L125). Sources: [swarm/failure_handling/recovery_manager.py:73-165](), [swarm/pheromone/pheromone_updater.py:49-69]() --- ## Decentralized State Synchronization & Local Peer Watchdogs While the [`SwarmCoordinator`](swarm/core/swarm_coordinator.py) executes centralized safety monitoring during simulation runs, the architecture supports decentralized swarm resilience. Each robot instance maintains an independent [`SwarmState`](swarm/core/swarm_state.py#L50-L63) that tracks peer heartbeats: ```python # swarm/core/swarm_state.py:96-109 def check_peer_timeouts(self, current_time: float, timeout_sec: float) -> List[str]: """ Evaluates peer heartbeats against timeout threshold. Marks silent peers as inactive, releases any assigned tasks, and returns list of newly timed-out robot IDs. """ timed_out: List[str] = [] for peer_id, peer in self.peers.items(): if peer.is_active: if (current_time - peer.last_heartbeat) > timeout_sec: self.mark_peer_inactive(peer_id) timed_out.append(peer_id) return timed_out ``` ``` +----------------------------------------------------------------------------------------------------+ | SUBSYSTEM ROLES & FAILURE RESPONSES | +--------------------------+-------------------------------------------------------------------------+ | Component | Role in Failure Handling | +--------------------------+-------------------------------------------------------------------------+ | FailureDetector | Ingests telemetry, tracks stall anchors, validates thresholds, | | | and emits RobotFailureEvent. | +--------------------------+-------------------------------------------------------------------------+ | RecoveryManager | Coordinates peer deactivation, task revocation, avoidance deposition, | | | and immediate peer suitability scoring. | +--------------------------+-------------------------------------------------------------------------+ | RobotAgent | Executes onboard FSM, emits periodic heartbeats, switches to | | | COMMUNICATION_LOST fallback if isolated, and handles task preemption. | +--------------------------+-------------------------------------------------------------------------+ | SwarmState | Maintains peer active/inactive maps and releases local task locks | | | upon detecting peer heartbeat timeouts. | +--------------------------+-------------------------------------------------------------------------+ | TaskManager | Manages global task lifecycle (PENDING, ASSIGNED, IN_PROGRESS), | | | resetting status and clearing robot bindings upon task release. | +--------------------------+-------------------------------------------------------------------------+ | PheromoneUpdater | Injects radial Gaussian avoidance kernels (tau_a) at failure coordinates| | | to divert peer traffic. | +--------------------------+-------------------------------------------------------------------------+ ``` Sources: [swarm/core/swarm_state.py:86-109](), [swarm/core/robot_agent.py:275-288]() --- ## Verification & Fault Injection Scenarios The test suite in [`tests/test_failure_handling.py`](tests/test_failure_handling.py) verifies failure detection and recovery mechanics across unit, integration, and decentralized boundary conditions. ```mermaid flowchart LR subgraph TestSuites ["Test Suites in tests/test_failure_handling.py"] T1["TestFailureDetector\n- Heartbeat timeout\n- Low battery (<10%)\n- Kinematic stall\n- Explicit FSM fault\n- Robot reset"] T2["TestRecoveryManager\n- Task release to PENDING\n- Avoidance pheromone > 10.0\n- RecoveryAction audit trail"] T3["TestSwarmCoordinatorResilience\n- Dynamic reassignment\n- RF communication loss\n- Telemetry metrics tracking"] T4["TestDecentralizedSwarmStateTimeout\n- Peer timeout evaluation\n- Local task status release"] end ``` ### Verified Test Cases 1. **Heartbeat Silence Detection (`test_heartbeat_timeout`)**: - Asserts an agent transmitting at $t = 0.0\,\text{s}$ remains healthy at $t = 3.0\,\text{s}$ under a $5.0\,\text{s}$ threshold, and transitions to `FailureType.HEARTBEAT_TIMEOUT` at $t = 6.0\,\text{s}$. 2. **Critical Battery Cutoff (`test_battery_critical`)**: - Confirms that a telemetry record with $8\%$ battery against a $10\%$ threshold immediately generates a `FailureType.BATTERY_CRITICAL` event. 3. **Kinematic Wheel Stall (`test_stalled_robot`)**: - Confirms that an agent moving only $0.05\,\text{m}$ over $7.0\,\text{s}$ while in `RobotState.NAVIGATING` exceeds the $6.0\,\text{s}$ stall timeout and triggers `FailureType.STALLED`. 4. **Wreck Marking & Task Reclamation (`test_task_release_and_avoidance_deposit`)**: - Asserts that upon failure of `robot_fail`, its assigned task reverts to `TaskStatus.PENDING`, and the avoidance layer in `PheromoneMap` at the crash coordinates records $\tau_a > 10.0$. 5. **Dynamic Swarm Peer Takeover (`test_dynamic_reassignment_on_simulated_failure`)**: - Injects a simulated motor driver failure (`MOTOR_DRIVER_BURNOUT`) on `robot_1` via `SwarmCoordinator.simulate_robot_failure()`. - Verifies that `robot_2` assumes ownership of the orphaned task within a single coordination tick ($0.1\,\text{s}$ interval) and transitions to `RobotState.NAVIGATING`. Sources: [tests/test_failure_handling.py:35-139](), [tests/test_failure_handling.py:140-195](), [tests/test_failure_handling.py:196-297](), [tests/test_failure_handling.py:298-338]() --- ## Summary The autonomous fault detection and dynamic reallocation architecture provides continuous operational resilience for multi-robot search missions. By decoupling telemetry monitoring in [`FailureDetector`](swarm/failure_handling/failure_detector.py) from remedial action in [`RecoveryManager`](swarm/failure_handling/recovery_manager.py), the system isolates broken units, prevents urban bottlenecks via stigmergic avoidance pheromone deposition, and redistributes orphaned tasks based on proximity, battery reserves, and current workload. This design guarantees mission continuity without requiring centralized operator intervention.