Move etc under detection_rules (#1885)

* Move etc directory under detection_rules
* Prepend original `etc` path with `detection_rules`
* Update docstrings in util and CODEOWNERS
* Add resiliency to tags to account for the old directory structure
* Bug fix: remove unused param caused by commit 6ed1a39efe

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

.github/CODEOWNERS

@@ -1,13 +1,13 @@
 # detection-rules code owners
 # POC: Elastic Security Intelligence and Analytics Team
 
-tests/**/*.py     @brokensound77 @rw-access
-detection_rules/  @brokensound77 @rw-access
-tests/            @brokensound77 @rw-access
+tests/**/*.py     @brokensound77 @mikaayenson @terrancedejesus
+detection_rules/  @brokensound77 @mikaayenson @terrancedejesus
+tests/            @brokensound77 @mikaayenson @terrancedejesus
 
 # skip rta-mapping to avoid the spam
-etc/packages.yml  @brokensound77 @rw-access
-etc/*.json        @brokensound77 @rw-access
-etc/*.json        @brokensound77 @rw-access
-etc/*/*           @brokensound77 @rw-access
+detection_rules/etc/packages.yml  @brokensound77 @mikaayenson @terrancedejesus
+detection_rules/etc/*.json        @brokensound77 @mikaayenson @terrancedejesus
+detection_rules/etc/*.json        @brokensound77 @mikaayenson @terrancedejesus
+detection_rules/etc/*/*           @brokensound77 @mikaayenson @terrancedejesus
 

.github/paths-labeller.yml

@@ -3,10 +3,10 @@
   - "./**/*.md"
 - "schema":
   - "detection_rules/beats.py"
-  - "etc/beats_schemas/**/*"
+  - "detection_rules/etc/beats_schemas/**/*"
   - "detection_rules/ecs.py"
-  - "etc/ecs_schemas/**/*"
-  - "etc/api_schemas/**/*"
+  - "detection_rules/etc/ecs_schemas/**/*"
+  - "detection_rules/etc/api_schemas/**/*"
   - "detection_rules/schemas/**/*"
 - "python":
   - "detection_rules/**/*.py"

.github/workflows/lock-versions.yml

@@ -49,8 +49,8 @@ jobs:
         env:
           BRANCHES: "${{github.event.inputs.branches}}"
         run: |
-          ./etc/lock-multiple.sh $BRANCHES
-          git add etc/version.lock.json
+          ./detection_rules/etc/lock-multiple.sh $BRANCHES
+          git add detection_rules/etc/version.lock.json
 
       - name: Create Pull Request
         uses: peter-evans/create-pull-request@v3

CONTRIBUTING.md

@@ -8,24 +8,26 @@ These guidelines will also help you post meaningful issues that will be more eas
 
 ## Table of Contents
 
-- [Effective issue creation in Detection Rules](#effective-issue-creation-in-detection-rules)
+- [Contributing to Detection Rules](#contributing-to-detection-rules)
+  - [Table of Contents](#table-of-contents)
+  - [Effective issue creation in Detection Rules](#effective-issue-creation-in-detection-rules)
     - [Why we create issues before contributing code or new rules](#why-we-create-issues-before-contributing-code-or-new-rules)
     - [What a good issue looks like](#what-a-good-issue-looks-like)
-  - ["My issue isn’t getting enough attention"](#my-issue-isnt-getting-enough-attention)
+    - ["My issue isn't getting enough attention"](#my-issue-isnt-getting-enough-attention)
     - ["I want to help!"](#i-want-to-help)
-- [How we use Git and GitHub](#how-we-use-git-and-github)
+  - [How we use Git and GitHub](#how-we-use-git-and-github)
     - [Forking](#forking)
     - [Branching](#branching)
     - [Commit messages](#commit-messages)
     - [What goes into a Pull Request](#what-goes-into-a-pull-request)
-- [Our approach to detection engineering](#our-approach-to-detection-engineering)
+  - [Our approach to detection engineering](#our-approach-to-detection-engineering)
     - [Rule metadata](#rule-metadata)
     - [Using Elastic Common Schema (ECS)](#using-elastic-common-schema-ecs)
     - [Creating a rule with the CLI](#creating-a-rule-with-the-cli)
     - [Testing a rule with the CLI](#testing-a-rule-with-the-cli)
-- [Writing style](#writing-style)
-- [Signing the contributor license agreement](#signing-the-contributor-license-agreement)
-- [Submitting a Pull Request](#submitting-a-pull-request)
+  - [Writing style](#writing-style)
+  - [Signing the contributor license agreement](#signing-the-contributor-license-agreement)
+  - [Submitting a Pull Request](#submitting-a-pull-request)
     - [What to expect from a code review](#what-to-expect-from-a-code-review)
     - [How we handle merges](#how-we-handle-merges)
 
@@ -167,7 +169,7 @@ Our rules should be written generically when possible. We use [Elastic Common Sc
 
 If the relevant [categorization values](https://www.elastic.co/guide/en/ecs/current/ecs-category-field-values-reference.html) are already defined for ECS, we use these to narrow down the event type before adding the query. Typically, the query starts with the broadest grouping possible and gets narrower for each clause. For example, we might write `event.category:process and event.type:start and process.name:net.exe and process.args:group`. First, we match process events with `event.category`, then narrow to creation events with `event.type`. Of the process creation events, we're looking for the process `net.exe` with `process.name` and finally we check the arguments `group` by looking at `process.args`. This flow has little effect on the generated Elasticsearch query, but is the most intuitive to read for rule developers.
 
-Sometimes, it might not make sense for ECS to standardize a field, value, or category. Occasionally, we may encounter fields that specific to a single use-case or vendor. When that happens, we add an exception in [etc/non-ecs-schema.json](etc/non-ecs-schema.json). We automatically detect beats by looking at the index patterns used in a rule. If we see `winlogbeat-*`, for example, then we can validate the rule against ECS + Winlogbeat. When using a particular beat, please use `event.module` and `event.dataset` to make the rule more precise and to better nudge the validation logic. Similar to our logic flow for ECS categorization, we recommend searches progress from `event.module` → `event.dataset` → `event.action` → `<additional criteria>`.
+Sometimes, it might not make sense for ECS to standardize a field, value, or category. Occasionally, we may encounter fields that specific to a single use-case or vendor. When that happens, we add an exception in [detection_rules/etc/non-ecs-schema.json](detection_rules/etc/non-ecs-schema.json). We automatically detect beats by looking at the index patterns used in a rule. If we see `winlogbeat-*`, for example, then we can validate the rule against ECS + Winlogbeat. When using a particular beat, please use `event.module` and `event.dataset` to make the rule more precise and to better nudge the validation logic. Similar to our logic flow for ECS categorization, we recommend searches progress from `event.module` → `event.dataset` → `event.action` → `<additional criteria>`.
 
 When a Pull Request is missing a necessary ECS change, please add an issue to [elastic/ecs](https://github.com/elastic/ecs) and link it from the pull request. We don't want to leave PRs blocked for too long, so if the ECS issue isn't progressing, then we can add a note and use the vendor- or beat-specific fields. We'll create another issue, reminding us to update the rule logic to switch to the ECS field when it becomes available. To maximize compatibility, we may add an `or` clause for a release or two to handle the different permutatations. After a few releases, we'll remove this and strictly require the ECS fields.
 

README.md

@@ -11,12 +11,13 @@ This repository was first announced on Elastic's blog post, [Elastic Security op
 
 
 ## Table of Contents
-- [Overview of this repository](#overview-of-this-repository)
-- [Getting started](#getting-started)
-- [Red Team Automation](rta)
-- [How to contribute](#how-to-contribute)
-- [Licensing](#licensing)
-- [Questions? Problems? Suggestions?](#questions-problems-suggestions)
+- [Detection Rules](#detection-rules)
+  - [Table of Contents](#table-of-contents)
+  - [Overview of this repository](#overview-of-this-repository)
+  - [Getting started](#getting-started)
+  - [How to contribute](#how-to-contribute)
+  - [Licensing](#licensing)
+  - [Questions? Problems? Suggestions?](#questions-problems-suggestions)
 
 
 ## Overview of this repository
@@ -26,7 +27,7 @@ Detection Rules contains more than just static rule files. This repository also
 | folder                                |  description                                                                        |
 |-------------------------------------- |------------------------------------------------------------------------------------ |
 | [`detection_rules/`](detection_rules) | Python module for rule parsing, validating and packaging                            |
-| [`etc/`](etc)                         | Miscellaneous files, such as ECS and Beats schemas                                  |
+| [`detection_rules/etc/`](etc)         | Miscellaneous files, such as ECS and Beats schemas                                  |
 | [`kibana/`](kibana)                   | Python library for handling the API calls to Kibana and the Detection Engine        |
 | [`kql/`](kql)                         | Python library for parsing and validating Kibana Query Language                     |
 | [`rta/`](rta)                         | Red Team Automation code used to emulate attacker techniques, used for rule testing |

detection_rules/beats.py

@@ -56,7 +56,7 @@ def _decompress_and_save_schema(url, release_name):
 
     # remove all non-beat directories
     fs = {k: v for k, v in fs.get("folders", {}).items() if k.endswith("beat")}
-    print(f"Saving etc/beats_schema/{release_name}.json")
+    print(f"Saving detection_rules/etc/beats_schema/{release_name}.json")
 
     compressed = gzip_compress(json.dumps(fs, sort_keys=True, cls=DateTimeEncoder))
     path = get_etc_path("beats_schemas", release_name + ".json.gz")

detection_rules/devtools.py

@@ -189,7 +189,7 @@ class GitChangeEntry:
 def prune_staging_area(target_stack_version: str, dry_run: bool):
     """Prune the git staging area to remove changes to incompatible rules."""
     exceptions = {
-        "etc/packages.yml",
+        "detection_rules/etc/packages.yml",
     }
 
     target_stack_version = Version(target_stack_version)[:2]