Commit Graph

92 Commits

Author SHA1 Message Date
Eric Forte 90a2043bc4 [FR] 8.12 Release Preparation update Main Branch to 8.13 (#3313)
* 8.12 Release Prep update Main Branch to 8.13

* Fix typo in integrations

* Updated Schemas
2023-12-11 14:58:06 -05:00
Mika Ayenson 7514c0a206 [FR] Add Support for ES|QL Rule Type and Remote Validation (#3281)
* add suuport for esql type
* add unit tests
* set clients in RemoteConnector from auth methods
* thread remote rules; add engine test
* Add versions to remote validation results

---------

Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
2023-12-08 12:46:28 -07:00
Jonhnathan aeb1f91320 [Security Content] Introduce Investigate Plugin in Investigation Guides (#3080)
* [Security Content] Introduce Investigate Plugin in Investigation Guides
* Add compatibility note
* Update Transform format
* update transform unit tests for investigate
* updated docs with transform

---------

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
2023-12-08 11:54:40 -07:00
Mika Ayenson bc39c20eaf FR] Add Core Support for ES|QL Rule Type (#3292) 2023-11-28 13:03:09 -06:00
Mika Ayenson 93ad4b0959 Add UEBA Tag (#3277) 2023-11-20 13:51:13 -06:00
Terrance DeJesus cdeb398ab3 [FR] Adjust Prebuilt Rules Packaging to Use Elastic Package v3 (#3252)
* Adding support for elastic package version 3

* replaced OS with Pathlib where applicable

* added sub-dataclasses for V3

* fixed flake errors

* adjusted registry dataclasses to inherit base
2023-11-01 12:47:40 -04:00
Mika Ayenson d0b0216362 [FR] Support missing events (#3153) 2023-10-31 16:20:52 -05:00
Mika Ayenson a808130390 Cleanup saved_query references (#3205) 2023-10-26 18:07:33 -05:00
Terrance DeJesus 3ab57fb8a7 [FR] Adding Support for missing_field_strategy Field in Alert Suppression (#3201)
* adding missing field strategy option to alert suppression

* fixed linting errors

* added validate methods for alertsuppression dataclass

* fixed linting errors

* replaced old variable with new variable

* removing test rule

* adding post_load to queryruledata

* changed post_load to validates_schema

* updated unit testing for alert suppression

* fixed linting errors

* changed validates method name to validates_exceptions

* removed min compat for fields
2023-10-19 18:16:54 -04:00
Terrance DeJesus b4f8fc3290 [FR] 8.11 Release Preparation and Update Main Branch to 8.12 (#3182)
* prepping for 8.12 branch

* added ananlytic manifests and schemas

* fix linting issues

* updated analytic package manifests and schemas
2023-10-13 13:37:21 -04:00
Justin Ibarra 7f8a9849c4 [New Rule] File Compressed or Archived into Common Format (#3173)
* [New Rule] File Compressed or Archived into Common Format
* new build-threat-map-entry-command

---------

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>
2023-10-11 11:34:34 -07:00
Apoorva Joshi 747ee7d593 [New Rule] Adding Lateral Movement Rules from Advanced Analytic LMD Package (#3119)
* Adding Lateral Movement Detection rules

* added tags; adjusted tests; updated manifests and schemas

* added default value to build_integrations_schema

* combined analytic and non-dataset packages for related integrations

* adjusted machine learning definitions

* adjusted machine learning definitions

* removed splat for machine learning list due to 3.8 constraints

---------

Co-authored-by: terrancedejesus <terrance.dejesus@elastic.co>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
2023-09-27 14:53:38 -04:00
Mika Ayenson 20de1d8d1d [FR] Add support for samples in eql 0.9.18 (#3000) 2023-09-07 09:01:28 -05:00
Terrance DeJesus 08b646aa94 [FR] 8.10 Release Preparation and Update Main Branch to 8.11 (#3012)
* prepping for 8.11 branch

* fixed lint errors

* added 8.11 to stack schema map

* trimmed version lock file; adjusted new terms validation

* reverting changes to version lock, stack schema and workflow
2023-08-16 14:23:44 -04:00
Mika Ayenson 3f9e7aced1 [Bug] Strip Non-Public Fields Prior to Uploading Rules (#2986) 2023-08-02 12:38:48 -05:00
Mika Ayenson 2ff4584456 load unsupported rule type from schema (#2893) 2023-06-29 15:32:32 -04:00
Terrance DeJesus 35d373b2bd [FR] 8.9 Release Preparation and Update Main Branch to 8.10 (#2891)
* adding new branch and refreshed schema

* fixed flake errors
2023-06-29 11:39:11 -04:00
Mika Ayenson cec41b4072 [FR Build a limited compatible rule ndjson for older stacks (#2885) 2023-06-29 10:18:24 -04:00
Jonhnathan a7e605a0e5 [Rule Tuning] [BUG] Revert PowerShell Query modifications from #2823 (#2889)
* Revert query mods done in https://github.com/elastic/detection-rules/pull/2823

* Add exception to unit test

* fixed linting

* proper linting fix

* updated to add to definitions.py

* fix linting

---------

Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
2023-06-28 15:55:43 -03:00
Terrance DeJesus 48cf95c8eb [Rule Tuning] Change Network Rules to Use Network Packet Capture Integration (#2665)
* updated indexes and updated dates

* added network_traffic integration tag to rules

* reverting changes to resolve conflicts

* metadata changes; indexes changed; schemas and manifest updated

* updated default telnet port connection rule

* updating integration manifests

* adjusted rules; updated integrations; deduplicate packages
2023-06-26 17:35:49 -04:00
Terrance DeJesus d829b145ef [Bug] Fix Tag Navigator Generation (#2875)
* bug fix for tag navigator generation

* addressing flake errors

* added unit test to ensure prefix exists

* updated unit test case sensitivity

* moved expected tags to definitions.py

* removed expected prefixes

* revert downloadable updates JSON file
2023-06-23 10:44:55 -04:00
eric-forte-elastic 6449cecd08 [FR] Add support for building block rules (BBR) (#2822)
* added test bbr

* initial implementation

* Added Unit test and exempted bbr from integrations

* fixed linting

* Add schema validation to building block rules

* add separate error messages

* fixed linting

* Add testing bbr validation

* fixed linting

* Add default values

* fixed linting

* added defaults

* fixed linting

* cleaned up test rule

* removed .gitkeep

* read .gitkeep

* Switch to using validates_schema

* addressing some linting

* fixed linting

* Update detection_rules/schemas/definitions.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* add env variable check

* fix skip function

* updated name

* Update detection_rules/schemas/definitions.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Add bbr validation unit test

* Clean up comments

* fix linting

* Move convert time to utils

* Moved to rules_building_block

* Add check for only bbr in bbr dir

* fix linting

* additional linting fix

* Changed to bbr rule loader

* fixed bbr default

* Updated error messages and README

* fixed more linting

* Updating root level README

* Fixed convert_time_span calls

* fixed typo in unit test logic and updated txt

* fixed error message

* updated comment for clarity

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Updated validation methods for clarity

* fix doctring location

* Fixed typo

* updated error messages.

* removed excess whitespace

* Add per rule bypass

* Add single rule bypass

* Split unit tests

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

---------

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2023-06-20 09:00:30 -04:00
Terrance DeJesus 792da36fb9 [Bug] Add Cloud Defend to definitions.NON_DATASET_PACKAGES (#2764)
* updating code to include cloud defend package

* updated integration manifests and schemas
2023-04-28 11:23:48 -04:00
Terrance DeJesus fadb5c2343 [FR] 8.8 Release Preparation and Update Main Branch to 8.9 (#2734)
* [FR] 8.8 Release Preparation and Update Main Branch to 8.9

* fixed flake errors
2023-04-24 10:13:07 -04:00
Mika Ayenson e9ebb1f2d8 [Bug] Rename 8.7 schemas from *.master and strip build time fields (#2707) 2023-04-11 10:56:20 -04:00
Terrance DeJesus 7e28b8fc50 [FR] Support Rule Alert Suppression in Rule Schema (#2660)
* adding initial solution for alert suppression support in rule schema

* reverting rule changes

* fixing flake errors

* reverting rule changes

* adding unit tests

* addressing flake errors

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update detection_rules/schemas/definitions.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* adjusting rule.py after commits

* adjusted test_group_field_in_schemas to check integrations

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* nested AlertSuppressDuration class under mapping class

* adjusted dataclass naming

* added unit test to ensure rule is KQL

* fixing flake errors

* added docstrings

---------

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
2023-03-27 15:37:35 -04:00
Terrance DeJesus 8a7ad13611 [FR] 8.7 Release Preparation and Update Main Branch to 8.8 (#2533)
* adding preparations for 8.8 release

* addressed flake single new line error

* froze and updated API schemas

* updated get_intregration_manifests

* adjusted boolean in find_latest_integration_version
2023-02-08 17:27:21 -05:00
Terrance DeJesus fb2b4529c5 [FR] Adapt PyPi semver Library and Remove Custom (#2503)
* removed custom semver and replaced with pypi

* updated beats.py version references

* updated bump-versions CLI command to use semver and change logic

* updated schemas __init__, test_version_lock and unstage incompatible rules CLI

* updated test_stack_schema_map in TestVersions unittest

* updated test_all_rules unit testing Version() references

* updated stack_compat.py for get_restricted_field references)

* updated version_lock.py Version() references

* updated docs.py Version() reference for parse_registry

* updated devtools.py Version() reference for trim-version-lock

* updated mixins.py Version() reference in validate_field_compatibility

* adjusted schemas.__init__ Version() reference in get_stack_schemas

* adjusted ecs.py Version() references

* adjusted integrations.py Version() references

* adjusted rule.py Version() references

* sorted imports

* replaced custom semver with pypi semver in unit test files

* addressed unit test and flake errors

* changed semver strings casted to version_lock.py

* fixed sorting in integrations.py

* updated bump-pkgs-versions CLI command

* adjusted semantic version in unstage-incompatible-rules command

* adjusted semver import to VersionInfo

* added semver 3 and adjusted import names

* added option_minor_and_patch parameter where version is major.minor

* updated bump-pkg-versions to always save to packages.yml

* removed leftover split call & updated find latest compatible version command

* updated integrations.py, version_lock.py and schemas.__init__.py

* changed fstring reference in downgrade function

* reverted formatting changes for detection_rules __init__.py

* added newline to detection_rules __init__.py

* adjusted finding latest_release for attack package logic

* adjusted unstage-incompatible-rules command logic comparing versions

* removing changes from misc.py related to auto-formatting

* adding newline to misc.py

* fixed bug in downgrade function calling decorators

* added semantic version validation on migrate decorator function

* added expected type returned from find_latest_integration_version in integrations.py

* add comment about stripped versions for version lock file

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

---------

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2023-02-07 14:26:29 -05:00
Mika Ayenson 1784429aa7 [FR] Add Integration Schema Query Validation (#2470) 2023-02-02 16:22:44 -05:00
Jonhnathan 0acbe1d832 [New Rule] Multiple Alerts Involving a User (#2401)
* [New Rule] Multiple Alerts Involving a User

* Update definitions.py

* update query

* Update multiple_alerts_involving_user.toml

Co-authored-by: Colson Wilhoit <48036388+DefSecSentinel@users.noreply.github.com>
2023-01-03 12:25:40 -03:00
Terrance DeJesus 7e459dd585 [FR] Add support for New Terms Fields and Window Start History (#2360)
* adding support new_terms_fields and window_start_history

* adjusted rule.py to address flake errors

* added assertion error if history_window_start does not exist

* removed sample rule

* removed self.rule_id from DataValidator

* added new_terms to RuleType

* changed new terms to its own class in rule.py

* removed nonexisting function call in DataValidator class

* adjusted new_terms field value in dataclass

* changed literal type for history_window_start; view-rule working

* removing test TOML rule

* addressed flake errors for missing newlines

* added validation option and adjusted object referencing

* adjusted validation method call in post_validation

* addressed flake errors for multiple spaces

* added transform method to NewTermsRuleData class

* added validation for min stack version and new terms array length restraints

* added validation for unique new terms array

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* removed historywindowstart definition and adjusted subclass

* removed test rule from commit

* adjusted if/else for data transform method check

* adjusted stack-schema-map; validation method name

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* added assertion for history_window_start field value

* added variables for feature min stack and extended field min stack

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* addressed flake errors for continuation line with same indent

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2022-12-05 14:07:33 -05:00
Terrance DeJesus 57b8f630de initial commit with changes for 8.7 branch creation (#2406) 2022-11-21 12:55:01 -05:00
Jonhnathan a7caa4baf3 [New Rule] Multiple Alerts in Different ATT&CK Tactics on a Single Host (#2399)
* [New Rule] Multiple Alerts in Different ATT&CK Tactics on a Single Host

* Update definitions.py

* Update rules/cross-platform/multiple_alerts_different_tactics_host.toml

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
2022-11-18 17:38:27 -03:00
Terrance DeJesus facaef1389 Prep for 8.6 Branch Creation (#2308)
* adding prep for 8.6

* updated ecs and beats packages

* addressed flake errors
2022-09-21 17:01:02 -04:00
Terrance DeJesus 2a3b584433 Prep for 8.5 branch (#2220)
* adding first commit

* renamed branch

* adjusted packages, stack schema and updated schemas

* updated integrations manifest

* adjusted comments to be a little more organized

* adjusted stack-schema-map

* refreshed ecs and beats schema, adjusted stack schema map accordingly
2022-08-09 17:14:42 -04:00
Justin Ibarra cc01d3fb1a Add support for restricted fields (#2053)
* Add support for restricted fields (fields valid only in min/max stack versions)
* add test to ensure rule backports wont exceed min compat
2022-06-27 10:02:15 -05:00
Justin Ibarra 744f56d98e [Bug] resolves bug in Rule version methods (#2021)
* [Bug] resolves bug in Rule version methods

* comment out unused code with notes
2022-06-07 15:40:46 -08:00
Justin Ibarra e850f39526 [Bug] Fix test_matrix_to_lock_version_defaults test (#2014) 2022-06-02 16:34:54 -08:00
Terrance DeJesus 35b1a69ff5 Prep for Creation of 8.4 Branch (#2001)
* prepping for 8.4 branch

* adjusted schemas init file

* adjusted target matrix to only backport to 7.16, updated api schemas

* adjusted the lock-versions workflow to account for 7.16 and up support only

* Add test for version lock to schema map correlation

* decouple from static 7.13 references

* keep patch version for lock

* Update detection_rules/etc/packages.yml

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

Co-authored-by: Jonhnathan <jonhnathancesar@gmail.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
2022-06-02 14:59:18 -04:00
Justin Ibarra 0428e161a8 Refresh ECS/beats schemas up to 8.2 (#1995) 2022-05-25 11:51:43 -08:00
Justin Ibarra 22679e16d2 Add delta command to determine changes to endpoint rules between tags (#1943)
* update git tag loader to be compatible with lock validation
* add diff command
* default to query for missing rules
2022-05-03 12:30:11 -08:00
Justin Ibarra c803160e4f Validate version lock and deprecation files on load and save (#1884)
* Validate version lock and deprecation files on load and save
* add missing types for previous lock entries
* bump marshmallow_dataclass
2022-04-26 22:17:20 -08:00
Terrance DeJesus 648daf1237 Prep for Creation of 8.3 Branch (#1906)
* updating with changes for 8.3 prep
* adding updates
* adjusted version in packages.yml
2022-04-01 13:33:18 -08:00
Terrance DeJesus e72031a71a added comprehensive timeline template definitions (#1905) 2022-04-01 08:51:54 -08:00
Mika Ayenson 84b7ce6582 update beats master branch ref to main (#1853)
* update beats master branch ref to main

* update filename of master beat schema to main

* delete old main beats schema

* rebuilt main beats archive
2022-03-18 10:06:34 -04:00
Colson Wilhoit e0dda91f26 Prep for creation of 8.2 branch (#1762) 2022-02-08 18:43:55 -09:00
Justin Ibarra 2e78da5c9a Prepare for creation of 8.1 branch (#1700) 2022-01-25 18:11:59 -09:00
Justin Ibarra d753ecb8d8 Add pattern for "name" in rule schema (#1669) 2022-01-25 12:03:27 -09:00
Justin Ibarra 95d7e9b6f5 Prepare for creation of 7.16 release branch (#1611) 2021-11-15 09:39:34 -09:00
Justin Ibarra ab17dfcc28 [Bug] Tighten definitions validation patterns (#1396)
* [Bug] Anchor validation patterns
* Deprecate rule with invalid rule_id and duplicate as new one

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2021-10-26 10:26:20 -05:00