[FR] Detection Rule PR Guidelines and Issue Forms (#3850)

2024-07-10 17:18:45 -05:00
parent 59a10be7c8
commit c62321f810
28 changed files with 861 additions and 267 deletions
@@ -0,0 +1,34 @@
+## Bug - Guidelines
+
+These guidelines serve as a reminder set of considerations when addressing a bug in the code.
+
+### Documentation and Context
+
+- [ ] Provide detailed documentation (description, screenshots, reproducing the bug, etc.) of the bug if not already documented in an issue.
+- [ ] Include additional context or details about the problem.
+- [ ] Ensure the fix includes necessary updates to the release documentation and versioning.
+
+### Code Standards and Practices
+
+- [ ] Code follows established design patterns within the repo and avoids duplication.
+- [ ] Code changes do not introduce new warnings or errors.
+- [ ] Variables and functions are well-named and descriptive.
+- [ ] Any unnecessary / commented-out code is removed.
+- [ ] Ensure that the code is modular and reusable where applicable.
+- [ ] Check for proper exception handling and messaging.
+
+### Testing
+
+- [ ] New unit tests have been added to cover the bug fix or edge cases.
+- [ ] Existing unit tests have been updated to reflect the changes.
+- [ ] Provide evidence of testing and detecting the bug fix (e.g., test logs, screenshots).
+- [ ] Validate that any rules affected by the bug are correctly updated.
+- [ ] Ensure that performance is not negatively impacted by the changes.
+- [ ] Verify that any release artifacts are properly generated and tested.
+
+### Additional Checks
+
+- [ ] Ensure that the bug fix does not break existing functionality.
+- [ ] Review the bug fix with a peer or team member for additional insights.
+- [ ] Verify that the bug fix works across all relevant environments (e.g., different OS versions).
+- [ ] Confirm that all dependencies are up-to-date and compatible with the changes.
@@ -0,0 +1,34 @@
+## Enhancement - Guidelines
+
+These guidelines serve as a reminder set of considerations when addressing adding a feature to the code.
+
+### Documentation and Context
+
+- [ ] Describe the feature enhancement in detail (alternative solutions, description of the solution, etc.) if not already documented in an issue.
+- [ ] Include additional context or screenshots.
+- [ ] Ensure the enhancement includes necessary updates to the documentation and versioning.
+
+### Code Standards and Practices
+
+- [ ] Code follows established design patterns within the repo and avoids duplication.
+- [ ] Code changes do not introduce new warnings or errors.
+- [ ] Variables and functions are well-named and descriptive.
+- [ ] Any unnecessary / commented-out code is removed.
+- [ ] Ensure that the code is modular and reusable where applicable.
+- [ ] Check for proper exception handling and messaging.
+
+### Testing
+
+- [ ] New unit tests have been added to cover the enhancement.
+- [ ] Existing unit tests have been updated to reflect the changes.
+- [ ] Provide evidence of testing and validating the enhancement (e.g., test logs, screenshots).
+- [ ] Validate that any rules affected by the enhancement are correctly updated.
+- [ ] Ensure that performance is not negatively impacted by the changes.
+- [ ] Verify that any release artifacts are properly generated and tested.
+
+### Additional Checks
+
+- [ ] Ensure that the enhancement does not break existing functionality.
+- [ ] Review the enhancement with a peer or team member for additional insights.
+- [ ] Verify that the enhancement works across all relevant environments (e.g., different OS versions).
+- [ ] Confirm that all dependencies are up-to-date and compatible with the changes.
@@ -0,0 +1,31 @@
+## Hunt: New - Guidelines
+
+Welcome to the `hunting` folder within the `detection-rules` repository! This directory houses a curated collection of threat hunting queries designed to enhance security monitoring and threat detection capabilities using the Elastic Stack.
+
+### Documentation and Context
+
+- [ ] Detailed description of the Hunt.
+- [ ] List any new fields required in ECS/data sources.
+- [ ] Link related issues or PRs.
+- [ ] Include references.
+- [ ] Field Usage: Ensure standardized fields for compatibility across different data environments and sources.
+
+### Hunt Metadata Checks
+
+- [ ] `author`: The name of the individual or organization authoring the rule.
+- [ ] `creation_date` matches the date of creation PR initially merged.
+- [ ] `min_stack_version` supports the widest stack versions.
+- [ ] `name` and `description` are descriptive and typo-free.
+- [ ] `language`: The query language(s) used in the rule, such as `KQL`, `EQL`, `ES|QL`, `OsQuery`, or `YARA`.
+- [ ] `query` is inclusive, not overly exclusive, considering performance for diverse environments.
+- [ ] `integration` aligns with the `index`. Ensure updates if the integration is newly introduced.
+- [ ] `setup` includes necessary steps to configure the integration.
+- [ ] `note` includes additional information (e.g., Triage and analysis investigation guides, timeline templates).
+- [ ] `tags` are relevant to the threat and align with `EXPECTED_HUNT_TAGS` in `definitions.py`.
+- [ ] `threat`, `techniques`, and `subtechniques` map to ATT&CK whenever possible.
+
+### Testing and Validation
+
+- [ ] Evidence of testing and detecting the expected threat.
+- [ ] Check for the existence of coverage to prevent duplication.
+- [ ] Generate Markdown: Run `python generate_markdown.py` to update the documentation.
@@ -0,0 +1,39 @@
+## Hunt: Tuning - Guidelines
+
+These guidelines serve as a reminder set of considerations when tuning an existing Hunt.
+
+### Documentation and Context
+
+- [ ] Detailed description of the suggested changes.
+- [ ] Provide example JSON data or screenshots.
+- [ ] Evidence of reducing benign events mistakenly identified as threats (False Positives).
+- [ ] Evidence of enhancing detection of true threats that were previously missed (False Negatives).
+- [ ] Evidence of optimizing resource consumption and execution time of detection rules (Performance).
+- [ ] Evidence of specific environment factors influencing customized hunt tuning (Contextual Tuning).
+- [ ] Evidence of improvements by modifying sensitivity (Threshold Adjustments).
+- [ ] Evidence of refining hunts to better detect deviations from typical behavior (Behavioral Tuning).
+- [ ] Evidence of improvements based on time-based patterns (Temporal Tuning).
+- [ ] Reasoning for adjusting priority or severity levels of alerts (Severity Tuning).
+- [ ] Evidence of improving the quality integrity of data used by hunts (Data Quality).
+- [ ] Ensure necessary updates to release documentation and versioning.
+- [ ] Field Usage: Ensure standardized fields for compatibility across different data environments and sources.
+
+### Hunt Metadata Checks
+
+- [ ] `author`: The name of the individual or organization authoring the rule.
+- [ ] `updated_date` matches the date of tuning PR merged.
+- [ ] `min_stack_version` supports the widest stack versions.
+- [ ] `name` and `description` are descriptive and typo-free.
+- [ ] `language`: The query language(s) used in the rule, such as `KQL`, `EQL`, `ES|QL`, `OsQuery`, or `YARA`.
+- [ ] `query` is inclusive, not overly exclusive. Review to ensure the original intent of the hunt is maintained.
+- [ ] `integration` aligns with the `index`. Ensure updates if the integration is newly introduced.
+- [ ] `setup` includes necessary steps to configure the integration.
+- [ ] `note` includes additional information (e.g., Triage and analysis investigation guides, timeline templates).
+- [ ] `tags` are relevant to the threat and align with `EXPECTED_HUNT_TAGS` in `definitions.py`.
+- [ ] `threat`, `techniques`, and `subtechniques` map to ATT&CK whenever possible.
+
+### Testing and Validation
+
+- [ ] Generate Markdown: Run `python generate_markdown.py` to update the documentation.
+- [ ] Validate the tuned hunt's performance and ensure it does not negatively impact the stack.
+- [ ] Ensure the tuned hunt has a low false positive rate.
@@ -0,0 +1,20 @@
+## Rule: Deprecation - Guidelines
+
+These guidelines serve as a reminder set of considerations when recommending the deprecation of a rule.
+
+### Documentation and Context
+
+- [ ] Description of the reason for deprecation.
+- [ ] Include any context or historical data supporting the deprecation decision.
+
+### Rule Metadata Checks
+
+- [ ] `deprecated = true` added to the rule metadata.
+- [ ] `updated_date` should be the date of the PR.
+
+### Testing and Validation
+
+- [ ] A prior rule tuning occurred for the rule where `Deprecated - ` is prepended to the rule name, and the rule has already been released.
+- [ ] Rule has be moved to the `_deprecated` directory.
+- [ ] Double check gaps potentially or inadvertently introduced.
+- [ ] Provide evidence that the rule is no longer needed or has been replaced (e.g., alternative rules, updated detection methods).
@@ -0,0 +1,33 @@
+## Rule: New - Guidelines
+
+These guidelines serve as a reminder set of considerations when proposing a new rule.
+
+### Documentation and Context
+
+- [ ] Detailed description of the rule.
+- [ ] List any new fields required in ECS/data sources.
+- [ ] Link related issues or PRs.
+- [ ] Include references.
+
+### Rule Metadata Checks
+
+- [ ] `creation_date` matches the date of creation PR initially merged.
+- [ ] `min_stack_version` should support the widest stack versions.
+- [ ] `name` and `description` should be descriptive and not include typos.
+- [ ] `query` should be inclusive, not overly exclusive, considering performance for diverse environments. Non ecs fields should be added to `non-ecs-schema.json` if not available in an integration.
+- [ ] `min_stack_comments` and `min_stack_version` should be included if the rule is only compatible starting from a specific stack version.
+- [ ] `index` pattern should be neither too specific nor too vague, ensuring it accurately matches the relevant data stream (e.g., use logs-endpoint.process-* for process data).
+- [ ] `integration` should align with the `index`. If the integration is newly introduced, ensure the manifest, schemas, and `new_rule.yaml` template are updated.
+- [ ] `setup` should include the necessary steps to configure the integration.
+- [ ] `note` should include any additional information (e.g. Triage and analysis investigation guides, timeline templates).
+- [ ] `tags` should be relevant to the threat and align/added to the `EXPECTED_RULE_TAGS` in the definitions.py file.
+- [ ] `threat`, `techniques`, and `subtechniques` should map to ATT&CK always if possible.
+
+#### New BBR Rules
+- [ ] `building_block_type` should be included if the rule is a building block and the rule should be located in the `rules_building_block` folder.
+- [ ] `bypass_bbr_timing` should be included if adding custom lookback timing to the rule.
+
+### Testing and Validation
+
+- [ ] Provide evidence of testing and detecting the expected threat.
+- [ ] Check for existence of coverage to prevent duplication.
@@ -0,0 +1,30 @@
+## Rule: Tuning - Guidelines
+
+These guidelines serve as a reminder set of considerations when tuning an existing rule.
+
+### Documentation and Context
+
+- [ ] Detailed description of the suggested changes.
+- [ ] Provide example JSON data or screenshots.
+- [ ] Provide evidence of reducing benign events mistakenly identified as threats (False Positives).
+- [ ] Provide evidence of enhancing detection of true threats that were previously missed (False Negatives).
+- [ ] Provide evidence of optimizing resource consumption and execution time of detection rules (Performance).
+- [ ] Provide evidence of specific environment factors influencing customized rule tuning (Contextual Tuning).
+- [ ] Provide evidence of improvements made by modifying sensitivity by changing alert triggering thresholds (Threshold Adjustments).
+- [ ] Provide evidence of refining rules to better detect deviations from typical behavior (Behavioral Tuning).
+- [ ] Provide evidence of improvements of adjusting rules based on time-based patterns (Temporal Tuning).
+- [ ] Provide reasoning of adjusting priority or severity levels of alerts (Severity Tuning).
+- [ ] Provide evidence of improving quality integrity of our data used by detection rules (Data Quality).
+- [ ] Ensure the tuning includes necessary updates to the release documentation and versioning.
+
+### Rule Metadata Checks
+
+- [ ] `updated_date` matches the date of tuning PR merged.
+- [ ] `min_stack_version` should support the widest stack versions.
+- [ ] `name` and `description` should be descriptive and not include typos.
+- [ ] `query` should be inclusive, not overly exclusive. Review to ensure the original intent of the rule is maintained.
+
+### Testing and Validation
+
+- [ ] Validate that the tuned rule's performance is satisfactory and does not negatively impact the stack.
+- [ ] Ensure that the tuned rule has a low false positive rate.
@@ -0,0 +1,46 @@
+## Enhancement - Guidelines
+
+These guidelines serve as a reminder set of considerations when addressing adding a new schema feature to the code.
+
+### Documentation and Context
+
+- [ ] Describe the feature enhancement in detail (alternative solutions, description of the solution, etc.) if not already documented in an issue.
+- [ ] Include additional context or screenshots.
+- [ ] Ensure the enhancement includes necessary updates to the documentation and versioning.
+
+### Code Standards and Practices
+
+- [ ] Code follows established design patterns within the repo and avoids duplication.
+- [ ] Code changes do not introduce new warnings or errors.
+- [ ] Variables and functions are well-named and descriptive.
+- [ ] Any unnecessary / commented-out code is removed.
+- [ ] Ensure that the code is modular and reusable where applicable.
+- [ ] Check for proper exception handling and messaging.
+
+### Testing
+
+- [ ] New unit tests have been added to cover the enhancement.
+- [ ] Existing unit tests have been updated to reflect the changes.
+- [ ] Provide evidence of testing and validating the enhancement (e.g., test logs, screenshots).
+- [ ] Validate that any rules affected by the enhancement are correctly updated.
+- [ ] Ensure that performance is not negatively impacted by the changes.
+- [ ] Verify that any release artifacts are properly generated and tested.
+
+### Additional Schema Related Checks
+
+- [ ] Ensure that the enhancement does not break existing functionality. (e.g., run `make test-cli`)
+- [ ] Review the enhancement with a peer or team member for additional insights.
+- [ ] Verify that the enhancement works across all relevant environments (e.g., different OS versions).
+- [ ] Confirm that all dependencies are up-to-date and compatible with the changes.
+- [ ] Link to the relevant Kibana PR or issue provided
+- [ ] Exported detection rule(s) from Kibana to showcase the feature(s)
+- [ ] Converted the exported ndjson file(s) to toml in the detection-rules repo
+- [ ] Re-exported the toml rule(s) to ndjson and re-imported into Kibana
+- [ ] Updated necessary unit tests to accommodate the feature
+- [ ] Applied min_compat restrictions to limit the feature to a specified minimum stack version
+- [ ] Executed all unit tests locally with a test toml rule to confirm passing
+- [ ] Included Kibana PR implementer as an optional reviewer for insights on the feature
+- [ ] Implemented requisite downgrade functionality
+- [ ] Cross-referenced the feature with product documentation for consistency
+- [ ] Incorporated a comprehensive test rule in unit tests for full schema coverage
+- [ ] Conducted system testing, including fleet, import, and create APIs (e.g., run `make test-remote-cli`)