feature/oob multi target

README.md

@@ -28,6 +28,9 @@ system debug. For details on what data is collected and analyzed, see the [plugi
       - [Global args](#global-args)
       - [Plugin config: **'--plugin-configs' command**](#plugin-config---plugin-configs-command)
       - [Post-action plugins](#post-action-plugins)
+        - [Config structure](#config-structure)
+        - [Condition fields](#condition-fields)
+        - [Example: run OsPlugin if DmesgPlugin finds error-level events](#example-run-osplugin-if-dmesgplugin-finds-error-level-events)
       - [Reference config: **'gen-reference-config' command**](#reference-config-gen-reference-config-command)
 
 ## Installation
@@ -192,6 +195,8 @@ In-band (SSH) connection:
 }
 ```
 
+A sample is in `config/connection-config_inband.example.json`. Use `password` or `key_filename`.
+
 Redfish (BMC) connection for Redfish-only plugins:
 
 ```json
@@ -210,6 +215,51 @@ Redfish (BMC) connection for Redfish-only plugins:
 
 - `api_root` (optional): Redfish API path (e.g. `redfish/v1`). If omitted, the default `redfish/v1` is used. Override this when your BMC uses a different API version path.
 
+OOB SSH plugins use this same single-host `RedfishConnectionManager` block and open SSH to that BMC. A sample is in `config/connection-config_oob.example.json`.
+
+#### Redfish multi-target
+
+Redfish plugins can collect from multiple BMCs concurrently. In-band plugins and OOB SSH plugins stay single-host. If this config has no top-level `host`, OOB SSH plugins are skipped. Add a top-level `host` when those plugins should still run against one BMC.
+
+```json
+{
+    "RedfishConnectionManager": {
+        "targets": [
+            {
+                "target_key": "node-a",
+                "host": "bmc-node-a.example.com",
+                "username": "admin",
+                "password": "secret",
+                "use_https": true,
+                "verify_ssl": false,
+                "timeout_seconds": 30
+            },
+            {
+                "target_key": "node-b",
+                "host": "bmc-node-b.example.com",
+                "username": "admin",
+                "password": "secret",
+                "use_https": true,
+                "verify_ssl": false,
+                "timeout_seconds": 30
+            }
+        ],
+        "max_workers": 32
+    }
+}
+```
+
+Multi-target mode applies to Redfish plugins (`RedfishEndpointPlugin`, `RedfishOemDiagPlugin`, and other plugins based on `OOBandDataPlugin`). Targets are collected concurrently. Wall-clock time follows the slowest target.
+
+A target that fails to connect or collect does not fail the run when another target succeeds. That plugin result is a warning, and analysis still runs for the targets that returned data. The run fails when every target fails, or when analysis of collected data reports an error.
+
+- `targets`: list of per-target connection parameters. Each entry accepts the same fields as the single-target config plus an optional `target_key` (used as the result key; defaults to the host string).
+- `max_workers` (optional): maximum concurrent collection threads. Defaults to `min(len(targets), 32)` and is capped at 32.
+
+Per-target results are written to `<plugin>/<collector>/<target_key>/`. Single-target results stay in `<plugin>/<collector>/`. The same layout is used for the analyzer directory and for the AMC SSH-proxy plugin.
+
+A ready-to-edit sample is in `config/connection-config_redfish_multi_target.example.json`. Replace the example hosts and password, then pass it with `--connection-config`.
+
 **Notes:**
 - If using SSH keys, specify `key_filename` instead of `password`.
 - The remote user must have permissions to run the requested plugins and access required files. If needed, use the `--skip-sudo` argument to skip plugins requiring sudo.
@@ -472,9 +522,11 @@ Use a plugin config that points at your LogService and lists the types to collec
 
 The RedfishEndpointPlugin collects Redfish URIs (GET responses) and optionally runs checks on the returned JSON. It requires a Redfish connection config (same as RedfishOemDiagPlugin).
 
+**Multi-target support:** `RedfishEndpointPlugin` collects from each BMC in the `targets` list at the same time. Use the [Redfish multi-target](#redfish-multi-target) connection config. The same `uris` and `checks` apply to every target. Per-target results are written to `redfish_endpoint_plugin/redfish_endpoint_collector/<target_key>/` under the run log directory. A BMC that cannot be reached is reported as a warning when another target succeeds.
+
 **How to run**
 
-1. Create a connection config (e.g. `connection-config.json`) with `RedfishConnectionManager` and your BMC host, credentials, and API root.
+1. Create a connection config (e.g. `connection-config.json`) with `RedfishConnectionManager` and your BMC host (single target) or `targets` list (multi-target).
 2. Create a plugin config with `uris` to collect and optional `checks` for analysis (see example below). For example save as `plugin_config_redfish_endpoint.json`.
 3. Run:
    ```sh

config/connection-config_inband.example.json

@@ -0,0 +1,8 @@
+{
+  "InBandConnectionManager": {
+    "hostname": "host.example.com",
+    "port": 22,
+    "username": "admin",
+    "password": "placeholder"
+  }
+}

config/connection-config_oob.example.json

@@ -0,0 +1,12 @@
+{
+  "RedfishConnectionManager": {
+    "host": "bmc.example.com",
+    "port": 443,
+    "username": "admin",
+    "password": "placeholder",
+    "use_https": true,
+    "verify_ssl": false,
+    "timeout_seconds": 30,
+    "api_root": "redfish/v1"
+  }
+}

config/connection-config_redfish_multi_target.example.json

@@ -0,0 +1,25 @@
+{
+  "RedfishConnectionManager": {
+    "max_workers": 2,
+    "targets": [
+      {
+        "target_key": "node-a",
+        "host": "bmc-node-a.example.com",
+        "username": "admin",
+        "password": "placeholder",
+        "use_https": true,
+        "verify_ssl": false,
+        "timeout_seconds": 30
+      },
+      {
+        "target_key": "node-b",
+        "host": "bmc-node-b.example.com",
+        "username": "admin",
+        "password": "placeholder",
+        "use_https": true,
+        "verify_ssl": false,
+        "timeout_seconds": 30
+      }
+    ]
+  }
+}

nodescraper/base/oobanddataplugin.py

@@ -23,14 +23,21 @@
 # SOFTWARE.
 #
 ###############################################################################
-from typing import Generic
+from pathlib import Path
+from typing import Any, Generic, Optional, Union
 
+from nodescraper.base.redfishcollectortask import _target_dir_name
 from nodescraper.connection.redfish import (
     RedfishConnectionManager,
     RedfishConnectionParams,
+    collected_multi_target_data,
 )
+from nodescraper.enums import EventPriority, ExecutionStatus, SystemInteractionLevel
 from nodescraper.generictypes import TAnalyzeArg, TCollectArg, TDataModel
 from nodescraper.interfaces import DataPlugin
+from nodescraper.models import TaskResult
+from nodescraper.taskresulthooks.filesystemloghook import hooks_for_fixed_directory
+from nodescraper.utils import resolve_log_dir_name
 
 
 class OOBandDataPlugin(
@@ -43,6 +50,120 @@ class OOBandDataPlugin(
     ],
     Generic[TDataModel, TCollectArg, TAnalyzeArg],
 ):
-    """Base class for out-of-band (OOB) plugins that use Redfish connection."""
+    """Base class for OOB plugins (Redfish). Adds multi-target support.
+
+    Multi-target collection is handled transparently by RedfishDataCollector's
+    __init_subclass__ wrapper; this class only needs to override analyze() to run
+    the analyzer once per target after collection completes.
+    """
 
     CONNECTION_TYPE = RedfishConnectionManager
+
+    def run(  # type: ignore[override]
+        self,
+        collection: bool = True,
+        analysis: bool = True,
+        max_event_priority_level: Union[EventPriority, str] = EventPriority.CRITICAL,
+        system_interaction_level: Union[
+            SystemInteractionLevel, str
+        ] = SystemInteractionLevel.INTERACTIVE,
+        preserve_connection: bool = False,
+        data: Optional[Any] = None,
+        collection_args: Optional[Any] = None,
+        analysis_args: Optional[Any] = None,
+    ):
+        """Run plugin. For multi-target OK runs, the summary includes per-target collection detail."""
+        result = super().run(
+            collection=collection,
+            analysis=analysis,
+            max_event_priority_level=max_event_priority_level,
+            system_interaction_level=system_interaction_level,
+            preserve_connection=preserve_connection,
+            data=data,
+            collection_args=collection_args,
+            analysis_args=analysis_args,
+        )
+        cm = self.connection_manager
+        if collected_multi_target_data(cm):
+            # DataPlugin.run() replaces the message with "Plugin tasks completed successfully"
+            # for OK status, discarding per-target detail. Restore it for multi-target runs.
+            if result.status == ExecutionStatus.OK and getattr(
+                self.collection_result, "message", None
+            ):
+                result.message = self.collection_result.message
+
+        return result
+
+    def analyze(
+        self,
+        max_event_priority_level: Optional[Union[EventPriority, str]] = EventPriority.CRITICAL,
+        analysis_args: Optional[Union[TAnalyzeArg, dict]] = None,
+        data: Optional[Any] = None,
+    ) -> TaskResult:
+        """Analyze collected data for one BMC or once per Redfish target.
+
+        Args:
+            max_event_priority_level: Priority limit for events.
+            analysis_args: Analyzer arguments.
+            data: Pre-collected data for a single-target run.
+
+        Returns:
+            TaskResult: Analysis result for the targets that returned data.
+        """
+        cm = self.connection_manager
+        multi_target_data: dict = collected_multi_target_data(cm)
+
+        if not multi_target_data:
+            return super().analyze(
+                max_event_priority_level=max_event_priority_level,
+                analysis_args=analysis_args,
+                data=data,
+            )
+
+        if self.ANALYZER is None:
+            self.analysis_result = TaskResult(
+                status=ExecutionStatus.NOT_RAN,
+                parent=self.__class__.__name__,
+                message=f"Data analysis not supported for {self.__class__.__name__}",
+            )
+            return self.analysis_result
+
+        if (
+            analysis_args is not None
+            and isinstance(analysis_args, dict)
+            and hasattr(self, "ANALYZER_ARGS")
+            and self.ANALYZER_ARGS is not None
+        ):
+            analysis_args = self.ANALYZER_ARGS.model_validate(analysis_args)  # type: ignore[assignment]
+
+        plugin_log_dir = (
+            Path(self.log_path) / resolve_log_dir_name(self.__class__.__name__)
+            if self.log_path
+            else None
+        )
+        analyzer_name = resolve_log_dir_name(self.ANALYZER.__name__)
+
+        analysis_results: list[TaskResult] = []
+        for target_key, target_data in multi_target_data.items():
+            safe_key = _target_dir_name(target_key)
+            target_hooks = self.task_result_hooks
+            if plugin_log_dir is not None:
+                target_hooks = hooks_for_fixed_directory(
+                    self.task_result_hooks,
+                    str(plugin_log_dir / analyzer_name / safe_key),
+                )
+            analyzer_task = self.ANALYZER(
+                system_info=self.system_info.model_copy(),
+                logger=self.logger,
+                max_event_priority_level=max_event_priority_level or EventPriority.CRITICAL,
+                parent=safe_key,
+                task_result_hooks=target_hooks,
+                event_reporter=self.event_reporter,
+                session_id=self.session_id,
+            )
+            analysis_results.append(analyzer_task.analyze_data(target_data, analysis_args))
+
+        self.analysis_result = self._aggregate_collection_results(
+            self.__class__.__name__, analysis_results
+        )
+        return self.analysis_result