Skip to content

Commit 030485c

Browse files
squirrelscLiliDeng
authored andcommitted
Add enabled switch for environments and nodes
This change introduces an `enabled` boolean field at both the environment and node levels, allowing selective loading of configurations through runbook variables. Example: environment: - name: my_env enabled: $(use_first_env) # Variable-controlled nodes: - type: local name: node1 enabled: true - type: local name: node2 enabled: false # Skip this node
1 parent 855b70d commit 030485c

File tree

5 files changed

+229
-15
lines changed

5 files changed

+229
-15
lines changed

docs/run_test/runbook.rst

Lines changed: 118 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ Runbook Reference
99
- `Use variable and secrets <#use-variable-and-secrets>`__
1010
- `Use partial runbook <#use-partial-runbook>`__
1111
- `Use extensions <#use-extensions>`__
12+
- `Conditionally enable/disable environments or nodes <#conditionally-enable-disable-environments-or-nodes>`__
1213

1314
- `Reference <#reference>`__
1415

@@ -89,6 +90,7 @@ Runbook Reference
8990
- `environments <#environments>`__
9091

9192
- `name <#name-4>`__
93+
- `enabled <#enabled>`__
9294
- `topology <#topology>`__
9395
- `nodes <#nodes>`__
9496
- `nodes_requirement <#nodes-requirement>`__
@@ -153,8 +155,8 @@ name ``hello``.
153155
154156
Below section demonstrates how to configure test cases with retry, repetition,
155157
and timeout settings. The first test case will automatically retry up to 2 times
156-
if it fails, redeploying the environment for each retry attempt. The second test
157-
case demonstrates stress testing by running 3 times unconditionally (regardless
158+
if it fails, redeploying the environment for each retry attempt. The second test
159+
case demonstrates stress testing by running 3 times unconditionally (regardless
158160
of pass/fail) with a custom timeout of 1 hour.
159161

160162
.. code:: yaml
@@ -257,6 +259,59 @@ modules for test cases or extended features.
257259
path: ../../extensions
258260
- ../../lisa/microsoft/testsuites/core
259261
262+
Conditionally enable/disable environments or nodes
263+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
264+
265+
You can use the ``enabled`` field to conditionally enable or disable entire
266+
environments or individual nodes within an environment. This is particularly
267+
useful when combined with variables for dynamic configuration.
268+
269+
Below example shows how to enable/disable environments based on a variable:
270+
271+
.. code:: yaml
272+
273+
variable:
274+
- name: use_prod
275+
value: true
276+
- name: use_dev
277+
value: false
278+
279+
environment:
280+
environments:
281+
- name: production_env
282+
enabled: $(use_prod) # Controlled by variable
283+
nodes:
284+
- type: local
285+
- name: dev_env
286+
enabled: $(use_dev) # This environment will be skipped
287+
nodes:
288+
- type: local
289+
290+
Below example shows how to selectively disable specific nodes within an environment:
291+
292+
.. code:: yaml
293+
294+
environment:
295+
environments:
296+
- name: multi_node_env
297+
nodes:
298+
- name: primary_node
299+
type: local
300+
enabled: true # Always enabled
301+
- name: secondary_node
302+
type: local
303+
enabled: false # Temporarily disabled
304+
- name: optional_node
305+
type: remote
306+
address: 192.168.1.100
307+
enabled: $(include_remote_node) # Variable-controlled
308+
309+
This allows you to:
310+
311+
- Temporarily disable environments or nodes without deleting their configuration
312+
- Use variables to control which environments/nodes are active
313+
- Maintain multiple environment configurations and switch between them dynamically
314+
260315
Use transformers
261316
~~~~~~~~~~~~~~~~
262317

@@ -794,7 +849,7 @@ test execution logs and code context from the LISA framework.
794849
The log_agent notifier uses a multi-agent AI system that combines:
795850

796851
- **LogSearchAgent**: Specialized in searching and analyzing log files for error patterns
797-
- **CodeSearchAgent**: Examines source code files and analyzes implementations related to errors
852+
- **CodeSearchAgent**: Examines source code files and analyzes implementations related to errors
798853
- **Magentic Orchestration**: Coordinates the agents to provide comprehensive analysis
799854

800855
The analysis results are attached to test result messages and made available to
@@ -809,7 +864,7 @@ downstream notifiers and reporting systems.
809864

810865
2. **Required Python packages** (automatically included with LISA):
811866
- python-dotenv
812-
- semantic-kernel
867+
- semantic-kernel
813868
- azure-ai-inference
814869
- retry
815870

@@ -827,7 +882,7 @@ azure_openai_api_key
827882

828883
type: str, optional, default: ""
829884

830-
Azure OpenAI API key for authentication. If not set, the notifier will use
885+
Azure OpenAI API key for authentication. If not set, the notifier will use
831886
default authentication methods available in the environment.
832887

833888
Note: This value is automatically marked as secret and will be masked in logs.
@@ -897,7 +952,7 @@ Example of log_agent notifier:
897952
5. **Evidence Gathering**: Searches for supporting evidence in logs
898953
6. **Root Cause Analysis**: Provides comprehensive analysis with actionable insights
899954

900-
The AI analysis results are stored in the test result message's ``analysis["AI"]``
955+
The AI analysis results are stored in the test result message's ``analysis["AI"]``
901956
field and can be consumed by other notifiers like HTML or custom reporting systems.
902957

903958
environment
@@ -925,6 +980,30 @@ type: str, optional, default is empty
925980

926981
The name of the environment.
927982

983+
enabled
984+
'''''''
985+
986+
type: bool, optional, default is true
987+
988+
Controls whether the environment is loaded and used during test execution. When
989+
set to ``false``, the environment will be skipped during initialization. This is
990+
useful for definining multiple similar environments in the same runbook.
991+
992+
Example:
993+
994+
.. code:: yaml
995+
996+
environment:
997+
environments:
998+
- name: prod_env
999+
enabled: true # This environment will be loaded
1000+
nodes:
1001+
- type: local
1002+
- name: dev_env
1003+
enabled: $(use_dev_env) # Variable-controlled
1004+
nodes:
1005+
- type: local
1006+
9281007
topology
9291008
''''''''
9301009

@@ -939,6 +1018,32 @@ List of node, it can be a virtual machine on Azure or Hyper-V, bare metal or
9391018
others. For more information, refer to :ref:`write_test/concepts:node and
9401019
environment`.
9411020

1021+
Each node supports an ``enabled`` field:
1022+
1023+
**enabled** (bool, optional, default is true): Controls whether the node is
1024+
loaded during environment initialization. When set to ``false``, the node will
1025+
be skipped. This is useful for selecting specific nodes from the same
1026+
environment configuration.
1027+
1028+
Example:
1029+
1030+
.. code:: yaml
1031+
1032+
environment:
1033+
environments:
1034+
- name: test_env
1035+
nodes:
1036+
- name: node1
1037+
type: local
1038+
enabled: true # This node will be loaded
1039+
- name: node2
1040+
type: local
1041+
enabled: false # This node will be skipped
1042+
- name: node3
1043+
type: remote
1044+
address: 192.168.1.100
1045+
enabled: $(enable_node3) # Variable-controlled
1046+
9421047
nodes_requirement
9431048
'''''''''''''''''
9441049

@@ -1038,15 +1143,15 @@ timeout
10381143

10391144
type: int, optional, default is 0
10401145

1041-
Timeout in seconds for each test case. When a test case runs, LISA uses the
1042-
maximum value between the timeout specified in the runbook and the test case's
1043-
own metadata timeout. If this field is set to 0 (default) or not specified, only
1044-
the test case's metadata timeout is used (which defaults to 3600 seconds / 1 hour
1045-
if not explicitly set in the test case). This allows you to extend timeouts for
1146+
Timeout in seconds for each test case. When a test case runs, LISA uses the
1147+
maximum value between the timeout specified in the runbook and the test case's
1148+
own metadata timeout. If this field is set to 0 (default) or not specified, only
1149+
the test case's metadata timeout is used (which defaults to 3600 seconds / 1 hour
1150+
if not explicitly set in the test case). This allows you to extend timeouts for
10461151
specific test runs without modifying the test case code.
10471152

1048-
Note that this timeout applies to the overall test case execution. Any additional
1049-
command-level timeouts set within the test case code itself will not be affected
1153+
Note that this timeout applies to the overall test case execution. Any additional
1154+
command-level timeouts set within the test case code itself will not be affected
10501155
by this setting.
10511156

10521157
.. code:: yaml

lisa/environment.py

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -438,6 +438,12 @@ def _reset(self) -> None:
438438

439439
has_default_node = False
440440
for node_runbook in self.runbook.nodes:
441+
# Skip disabled nodes
442+
if not node_runbook.enabled:
443+
node_name = node_runbook.name or "unnamed"
444+
self.log.info(f"skipping to load disabled node: {node_name}")
445+
continue
446+
441447
self.create_node_from_exists(
442448
node_runbook=node_runbook,
443449
)
@@ -545,9 +551,17 @@ def load_environments(
545551
environments_runbook = root_runbook.environments
546552
for environment_runbook in environments_runbook:
547553
id_ = _get_environment_id()
554+
name = environment_runbook.name or f"customized_{id_}"
555+
556+
# Skip disabled environments
557+
if not environment_runbook.enabled:
558+
log = _get_init_logger()
559+
log.info(f"skipping to load disabled environment: {name}")
560+
continue
561+
548562
env = environments.from_runbook(
549563
runbook=environment_runbook,
550-
name=environment_runbook.name or f"customized_{id_}",
564+
name=name,
551565
is_predefined_runbook=True,
552566
id_=id_,
553567
)

lisa/schema.py

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1246,6 +1246,10 @@ class Node(TypedSchema, ExtendableSchemaMixin):
12461246
name: str = ""
12471247
is_default: bool = field(default=False)
12481248

1249+
# A node is disabled if it's False. It helps to disable node by
1250+
# variables.
1251+
enabled: bool = True
1252+
12491253

12501254
@dataclass_json()
12511255
@dataclass
@@ -1372,6 +1376,10 @@ class Environment:
13721376
)
13731377
nodes_requirement: Optional[List[NodeSpace]] = None
13741378

1379+
# An environment is disabled if it's False. It helps to disable environment
1380+
# by variables.
1381+
enabled: bool = True
1382+
13751383
_original_nodes_requirement: Optional[List[NodeSpace]] = None
13761384

13771385
def __post_init__(self, *args: Any, **kwargs: Any) -> None:

selftests/test_environment.py

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -292,3 +292,90 @@ def test_create_from_custom_local_remote(self) -> None:
292292
self.assertEqual(r_n.custom_remote_field, CUSTOM_REMOTE)
293293
done += 1
294294
self.assertEqual(2, done)
295+
296+
def test_disabled_environment_not_loaded(self) -> None:
297+
# Create runbook with 3 environments: 2 enabled, 1 disabled
298+
data = {
299+
constants.ENVIRONMENTS: [
300+
{
301+
"name": "enabled_env_1",
302+
constants.NODES: [
303+
{
304+
constants.TYPE: constants.ENVIRONMENTS_NODES_LOCAL,
305+
}
306+
],
307+
},
308+
{
309+
"name": "disabled_env",
310+
"enabled": False,
311+
constants.NODES: [
312+
{
313+
constants.TYPE: constants.ENVIRONMENTS_NODES_REMOTE,
314+
constants.ENVIRONMENTS_NODES_REMOTE_ADDRESS: "1.2.3.4",
315+
constants.ENVIRONMENTS_NODES_REMOTE_PORT: 22,
316+
constants.ENVIRONMENTS_NODES_REMOTE_USERNAME: "user",
317+
constants.ENVIRONMENTS_NODES_REMOTE_PASSWORD: "pass",
318+
}
319+
],
320+
},
321+
{
322+
"name": "enabled_env_2",
323+
constants.NODES: [
324+
{
325+
constants.TYPE: constants.ENVIRONMENTS_NODES_LOCAL,
326+
}
327+
],
328+
},
329+
]
330+
}
331+
runbook = schema.load_by_type(schema.EnvironmentRoot, data)
332+
envs = load_environments(runbook)
333+
334+
# Only 2 environments should be loaded (disabled one skipped)
335+
self.assertEqual(2, len(envs))
336+
self.assertIn("enabled_env_1", envs)
337+
self.assertIn("enabled_env_2", envs)
338+
self.assertNotIn("disabled_env", envs)
339+
340+
def test_disabled_node_not_loaded(self) -> None:
341+
# Create environment with 3 nodes: 2 enabled, 1 disabled
342+
data = {
343+
constants.ENVIRONMENTS: [
344+
{
345+
"name": "test_env",
346+
constants.NODES: [
347+
{
348+
"name": "node1",
349+
constants.TYPE: constants.ENVIRONMENTS_NODES_LOCAL,
350+
},
351+
{
352+
"name": "node2",
353+
"enabled": False,
354+
constants.TYPE: constants.ENVIRONMENTS_NODES_REMOTE,
355+
constants.ENVIRONMENTS_NODES_REMOTE_ADDRESS: "1.2.3.4",
356+
constants.ENVIRONMENTS_NODES_REMOTE_PORT: 22,
357+
constants.ENVIRONMENTS_NODES_REMOTE_USERNAME: "user",
358+
constants.ENVIRONMENTS_NODES_REMOTE_PASSWORD: "pass",
359+
},
360+
{
361+
"name": "node3",
362+
constants.TYPE: constants.ENVIRONMENTS_NODES_LOCAL,
363+
},
364+
],
365+
},
366+
]
367+
}
368+
runbook = schema.load_by_type(schema.EnvironmentRoot, data)
369+
envs = load_environments(runbook)
370+
371+
# Environment should be loaded
372+
self.assertEqual(1, len(envs))
373+
env = envs.get("test_env")
374+
assert env
375+
376+
# Only 2 nodes should be loaded (disabled one skipped)
377+
self.assertEqual(2, len(env.nodes))
378+
node_names = [node.name for node in env.nodes.list()]
379+
self.assertIn("node1", node_names)
380+
self.assertIn("node3", node_names)
381+
self.assertNotIn("node2", node_names)

selftests/test_platform.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -156,7 +156,7 @@ def test_prepared_env_not_success_with_exception(self) -> None:
156156
"no capability found for environment: Environment("
157157
"name='customized_0', topology='subnet', nodes_raw=[{'type': 'local', "
158158
"'capability': {'core_count': {'min': 4}}}], nodes_requirement=None, "
159-
"_original_nodes_requirement=None)",
159+
"enabled=True, _original_nodes_requirement=None)",
160160
str(cm.exception),
161161
)
162162

0 commit comments

Comments
 (0)