Commit Graph

102 Commits

Author SHA1 Message Date
Eric Forte 39782b4295 [FR] Update utility path computation to use pathlib (#3699)
* update

* Updated to pathlib

* Linting

* Add string cast where needed

* Add additional string conversion as needed

* Str conversions to support eql lib

* Attack typo

* Typo in test script

* Updated for more pathlib

* Linting

* Update to convert string to path object

* Fix typo

(cherry picked from commit f43fbfba0d)
2024-05-23 21:39:55 +00:00
shashank-elastic f27479ee12 Package Manifest changes to add capabilities (#3706)
Removed changes from:
- detection_rules/etc/packages.yaml

(selectively cherry picked from commit f73022b900)
2024-05-23 20:49:50 +00:00
Jonhnathan 0ab70f13a4 [Rule Tuning] Add Initial SentinelOne Compatibility to Windows DRs (#3627)
* [Rule Tuning] Add Initial SentinelOne Compatibility

* updated definitions.py; updated tags; fixed unit tests

* added prerelease versions for s1 integration; updated build CLI commands to allow prerelease; bumped min-stacks

* updating manifests and integrations

* fixing flake errors

* min_stack

---------

Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: terrancedejesus <terrance.dejesus@elastic.co>
Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>

(cherry picked from commit d023ad66b1)
2024-05-20 12:59:37 +00:00
shashank-elastic 891da3623d Prepare For Next Elastic Stack 8.15 (#3670)
Removed changes from:
- detection_rules/etc/packages.yml

(selectively cherry picked from commit 50a8b52cd5)
2024-05-14 19:10:09 +00:00
Mika Ayenson b75a9f902b [New Rule] Potential Abuse of Resources by High Token Count and Large Response Sizes (#3644)
(cherry picked from commit 2ffb0e7fe2)
2024-05-03 23:08:58 +00:00
Eric Forte 645fa593a1 [Bug] New Terms Rule Import Failing (#3569)
* initial patch

* Update definitions to allow for brackets in name

* Update to prompt for required fields.

* Update detection_rules/cli_utils.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

---------

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

(cherry picked from commit fa75876322)
2024-04-04 21:45:02 +00:00
Jonhnathan 7838042839 [Rule Tuning] Replace KQL exceptions for Query DSL Exceptions (#3505)
* [Rule Tuning] Replace KQL exceptions for Query DSL Exceptions

* update min_stack

* build out schema in more detail for Filters

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* Remove enum for definition

* remove unused import

* remove $state store

* transform state

* add call to super

* add return type hint

* use dataclass metadata

* use Literal type

---------

Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mika.ayenson@elastic.co>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

Removed changes from:
- rules/windows/collection_mailbox_export_winlog.toml
- rules/windows/collection_posh_clipboard_capture.toml
- rules/windows/defense_evasion_posh_assembly_load.toml
- rules/windows/defense_evasion_posh_compressed.toml
- rules/windows/discovery_posh_suspicious_api_functions.toml
- rules/windows/discovery_privileged_localgroup_membership.toml
- rules/windows/execution_posh_hacktool_functions.toml
- rules/windows/execution_posh_psreflect.toml
- rules_building_block/collection_posh_compression.toml
- rules_building_block/defense_evasion_powershell_clear_logs_script.toml
- rules_building_block/discovery_posh_generic.toml
- rules_building_block/lateral_movement_posh_winrm_activity.toml

(selectively cherry picked from commit 67ca13c1ce)
2024-04-01 20:53:09 +00:00
shashank-elastic 7043173371 Prepare For Next Elastic Stack Minor Release (#3490)
Removed changes from:
- detection_rules/etc/packages.yml

(selectively cherry picked from commit a4094df732)
2024-03-06 16:03:19 +00:00
Ruben Groenewoud 984f2a6fbf [FR] NON_DATASET_PACKAGE list & Data Source tag for Auditd_manager (#3430)
* [FR] Add Auditd_Manager to NON_DATASET_PACKAGE

* Changed alphabetical order

---------

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>

(cherry picked from commit a637bcec38)
2024-02-19 08:42:19 +00:00
Mika Ayenson bde05d63c6 [FR] Add support for Threshold Alert Suppression (#3433)
(cherry picked from commit c3ca01ebcc)
2024-02-12 16:01:10 +00:00
Eric Forte 908168725a [FR] 8.12 Release Preparation update Main Branch to 8.13 (#3313)
* 8.12 Release Prep update Main Branch to 8.13

* Fix typo in integrations

* Updated Schemas

Removed changes from:
- detection_rules/etc/packages.yml

(selectively cherry picked from commit 90a2043bc4)
2023-12-11 20:03:26 +00:00
Mika Ayenson 53f924d52e [FR] Add Support for ES|QL Rule Type and Remote Validation (#3281)
* add suuport for esql type
* add unit tests
* set clients in RemoteConnector from auth methods
* thread remote rules; add engine test
* Add versions to remote validation results

---------

Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

(cherry picked from commit 7514c0a206)
2023-12-08 19:52:16 +00:00
Jonhnathan 094f3ead92 [Security Content] Introduce Investigate Plugin in Investigation Guides (#3080)
* [Security Content] Introduce Investigate Plugin in Investigation Guides
* Add compatibility note
* Update Transform format
* update transform unit tests for investigate
* updated docs with transform

---------

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

(cherry picked from commit aeb1f91320)
2023-12-08 18:59:53 +00:00
Mika Ayenson 53c4ff1fdc FR] Add Core Support for ES|QL Rule Type (#3292)
(cherry picked from commit bc39c20eaf)
2023-11-28 19:08:40 +00:00
Mika Ayenson 61bbcfec52 Add UEBA Tag (#3277)
(cherry picked from commit 93ad4b0959)
2023-11-20 19:56:28 +00:00
Terrance DeJesus 4c6fc3496c [FR] Adjust Prebuilt Rules Packaging to Use Elastic Package v3 (#3252)
* Adding support for elastic package version 3

* replaced OS with Pathlib where applicable

* added sub-dataclasses for V3

* fixed flake errors

* adjusted registry dataclasses to inherit base

Removed changes from:
- detection_rules/etc/packages.yml

(selectively cherry picked from commit cdeb398ab3)
2023-11-01 16:54:01 +00:00
Mika Ayenson 8415bedf0f [FR] Support missing events (#3153)
(cherry picked from commit d0b0216362)
2023-10-31 21:27:08 +00:00
Mika Ayenson 55661b1239 Cleanup saved_query references (#3205)
(cherry picked from commit a808130390)
2023-10-26 23:13:57 +00:00
Terrance DeJesus 9c38931287 [FR] Adding Support for missing_field_strategy Field in Alert Suppression (#3201)
* adding missing field strategy option to alert suppression

* fixed linting errors

* added validate methods for alertsuppression dataclass

* fixed linting errors

* replaced old variable with new variable

* removing test rule

* adding post_load to queryruledata

* changed post_load to validates_schema

* updated unit testing for alert suppression

* fixed linting errors

* changed validates method name to validates_exceptions

* removed min compat for fields

(cherry picked from commit 3ab57fb8a7)
2023-10-19 22:23:31 +00:00
Terrance DeJesus 96bc049852 [FR] 8.11 Release Preparation and Update Main Branch to 8.12 (#3182)
* prepping for 8.12 branch

* added ananlytic manifests and schemas

* fix linting issues

* updated analytic package manifests and schemas

Removed changes from:
- detection_rules/etc/packages.yml

(selectively cherry picked from commit b4f8fc3290)
2023-10-13 17:43:55 +00:00
Justin Ibarra 7f8a9849c4 [New Rule] File Compressed or Archived into Common Format (#3173)
* [New Rule] File Compressed or Archived into Common Format
* new build-threat-map-entry-command

---------

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>
2023-10-11 11:34:34 -07:00
Apoorva Joshi 747ee7d593 [New Rule] Adding Lateral Movement Rules from Advanced Analytic LMD Package (#3119)
* Adding Lateral Movement Detection rules

* added tags; adjusted tests; updated manifests and schemas

* added default value to build_integrations_schema

* combined analytic and non-dataset packages for related integrations

* adjusted machine learning definitions

* adjusted machine learning definitions

* removed splat for machine learning list due to 3.8 constraints

---------

Co-authored-by: terrancedejesus <terrance.dejesus@elastic.co>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
2023-09-27 14:53:38 -04:00
Mika Ayenson 20de1d8d1d [FR] Add support for samples in eql 0.9.18 (#3000) 2023-09-07 09:01:28 -05:00
Terrance DeJesus 08b646aa94 [FR] 8.10 Release Preparation and Update Main Branch to 8.11 (#3012)
* prepping for 8.11 branch

* fixed lint errors

* added 8.11 to stack schema map

* trimmed version lock file; adjusted new terms validation

* reverting changes to version lock, stack schema and workflow
2023-08-16 14:23:44 -04:00
Mika Ayenson 3f9e7aced1 [Bug] Strip Non-Public Fields Prior to Uploading Rules (#2986) 2023-08-02 12:38:48 -05:00
Mika Ayenson 2ff4584456 load unsupported rule type from schema (#2893) 2023-06-29 15:32:32 -04:00
Terrance DeJesus 35d373b2bd [FR] 8.9 Release Preparation and Update Main Branch to 8.10 (#2891)
* adding new branch and refreshed schema

* fixed flake errors
2023-06-29 11:39:11 -04:00
Mika Ayenson cec41b4072 [FR Build a limited compatible rule ndjson for older stacks (#2885) 2023-06-29 10:18:24 -04:00
Jonhnathan a7e605a0e5 [Rule Tuning] [BUG] Revert PowerShell Query modifications from #2823 (#2889)
* Revert query mods done in https://github.com/elastic/detection-rules/pull/2823

* Add exception to unit test

* fixed linting

* proper linting fix

* updated to add to definitions.py

* fix linting

---------

Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
2023-06-28 15:55:43 -03:00
Terrance DeJesus 48cf95c8eb [Rule Tuning] Change Network Rules to Use Network Packet Capture Integration (#2665)
* updated indexes and updated dates

* added network_traffic integration tag to rules

* reverting changes to resolve conflicts

* metadata changes; indexes changed; schemas and manifest updated

* updated default telnet port connection rule

* updating integration manifests

* adjusted rules; updated integrations; deduplicate packages
2023-06-26 17:35:49 -04:00
Terrance DeJesus d829b145ef [Bug] Fix Tag Navigator Generation (#2875)
* bug fix for tag navigator generation

* addressing flake errors

* added unit test to ensure prefix exists

* updated unit test case sensitivity

* moved expected tags to definitions.py

* removed expected prefixes

* revert downloadable updates JSON file
2023-06-23 10:44:55 -04:00
eric-forte-elastic 6449cecd08 [FR] Add support for building block rules (BBR) (#2822)
* added test bbr

* initial implementation

* Added Unit test and exempted bbr from integrations

* fixed linting

* Add schema validation to building block rules

* add separate error messages

* fixed linting

* Add testing bbr validation

* fixed linting

* Add default values

* fixed linting

* added defaults

* fixed linting

* cleaned up test rule

* removed .gitkeep

* read .gitkeep

* Switch to using validates_schema

* addressing some linting

* fixed linting

* Update detection_rules/schemas/definitions.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* add env variable check

* fix skip function

* updated name

* Update detection_rules/schemas/definitions.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Add bbr validation unit test

* Clean up comments

* fix linting

* Move convert time to utils

* Moved to rules_building_block

* Add check for only bbr in bbr dir

* fix linting

* additional linting fix

* Changed to bbr rule loader

* fixed bbr default

* Updated error messages and README

* fixed more linting

* Updating root level README

* Fixed convert_time_span calls

* fixed typo in unit test logic and updated txt

* fixed error message

* updated comment for clarity

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Updated validation methods for clarity

* fix doctring location

* Fixed typo

* updated error messages.

* removed excess whitespace

* Add per rule bypass

* Add single rule bypass

* Split unit tests

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

---------

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2023-06-20 09:00:30 -04:00
Terrance DeJesus 792da36fb9 [Bug] Add Cloud Defend to definitions.NON_DATASET_PACKAGES (#2764)
* updating code to include cloud defend package

* updated integration manifests and schemas
2023-04-28 11:23:48 -04:00
Terrance DeJesus fadb5c2343 [FR] 8.8 Release Preparation and Update Main Branch to 8.9 (#2734)
* [FR] 8.8 Release Preparation and Update Main Branch to 8.9

* fixed flake errors
2023-04-24 10:13:07 -04:00
Mika Ayenson e9ebb1f2d8 [Bug] Rename 8.7 schemas from *.master and strip build time fields (#2707) 2023-04-11 10:56:20 -04:00
Terrance DeJesus 7e28b8fc50 [FR] Support Rule Alert Suppression in Rule Schema (#2660)
* adding initial solution for alert suppression support in rule schema

* reverting rule changes

* fixing flake errors

* reverting rule changes

* adding unit tests

* addressing flake errors

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update detection_rules/schemas/definitions.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* adjusting rule.py after commits

* adjusted test_group_field_in_schemas to check integrations

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* nested AlertSuppressDuration class under mapping class

* adjusted dataclass naming

* added unit test to ensure rule is KQL

* fixing flake errors

* added docstrings

---------

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
2023-03-27 15:37:35 -04:00
Terrance DeJesus 8a7ad13611 [FR] 8.7 Release Preparation and Update Main Branch to 8.8 (#2533)
* adding preparations for 8.8 release

* addressed flake single new line error

* froze and updated API schemas

* updated get_intregration_manifests

* adjusted boolean in find_latest_integration_version
2023-02-08 17:27:21 -05:00
Terrance DeJesus fb2b4529c5 [FR] Adapt PyPi semver Library and Remove Custom (#2503)
* removed custom semver and replaced with pypi

* updated beats.py version references

* updated bump-versions CLI command to use semver and change logic

* updated schemas __init__, test_version_lock and unstage incompatible rules CLI

* updated test_stack_schema_map in TestVersions unittest

* updated test_all_rules unit testing Version() references

* updated stack_compat.py for get_restricted_field references)

* updated version_lock.py Version() references

* updated docs.py Version() reference for parse_registry

* updated devtools.py Version() reference for trim-version-lock

* updated mixins.py Version() reference in validate_field_compatibility

* adjusted schemas.__init__ Version() reference in get_stack_schemas

* adjusted ecs.py Version() references

* adjusted integrations.py Version() references

* adjusted rule.py Version() references

* sorted imports

* replaced custom semver with pypi semver in unit test files

* addressed unit test and flake errors

* changed semver strings casted to version_lock.py

* fixed sorting in integrations.py

* updated bump-pkgs-versions CLI command

* adjusted semantic version in unstage-incompatible-rules command

* adjusted semver import to VersionInfo

* added semver 3 and adjusted import names

* added option_minor_and_patch parameter where version is major.minor

* updated bump-pkg-versions to always save to packages.yml

* removed leftover split call & updated find latest compatible version command

* updated integrations.py, version_lock.py and schemas.__init__.py

* changed fstring reference in downgrade function

* reverted formatting changes for detection_rules __init__.py

* added newline to detection_rules __init__.py

* adjusted finding latest_release for attack package logic

* adjusted unstage-incompatible-rules command logic comparing versions

* removing changes from misc.py related to auto-formatting

* adding newline to misc.py

* fixed bug in downgrade function calling decorators

* added semantic version validation on migrate decorator function

* added expected type returned from find_latest_integration_version in integrations.py

* add comment about stripped versions for version lock file

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

---------

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2023-02-07 14:26:29 -05:00
Mika Ayenson 1784429aa7 [FR] Add Integration Schema Query Validation (#2470) 2023-02-02 16:22:44 -05:00
Jonhnathan 0acbe1d832 [New Rule] Multiple Alerts Involving a User (#2401)
* [New Rule] Multiple Alerts Involving a User

* Update definitions.py

* update query

* Update multiple_alerts_involving_user.toml

Co-authored-by: Colson Wilhoit <48036388+DefSecSentinel@users.noreply.github.com>
2023-01-03 12:25:40 -03:00
Terrance DeJesus 7e459dd585 [FR] Add support for New Terms Fields and Window Start History (#2360)
* adding support new_terms_fields and window_start_history

* adjusted rule.py to address flake errors

* added assertion error if history_window_start does not exist

* removed sample rule

* removed self.rule_id from DataValidator

* added new_terms to RuleType

* changed new terms to its own class in rule.py

* removed nonexisting function call in DataValidator class

* adjusted new_terms field value in dataclass

* changed literal type for history_window_start; view-rule working

* removing test TOML rule

* addressed flake errors for missing newlines

* added validation option and adjusted object referencing

* adjusted validation method call in post_validation

* addressed flake errors for multiple spaces

* added transform method to NewTermsRuleData class

* added validation for min stack version and new terms array length restraints

* added validation for unique new terms array

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* removed historywindowstart definition and adjusted subclass

* removed test rule from commit

* adjusted if/else for data transform method check

* adjusted stack-schema-map; validation method name

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* added assertion for history_window_start field value

* added variables for feature min stack and extended field min stack

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* addressed flake errors for continuation line with same indent

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2022-12-05 14:07:33 -05:00
Terrance DeJesus 57b8f630de initial commit with changes for 8.7 branch creation (#2406) 2022-11-21 12:55:01 -05:00
Jonhnathan a7caa4baf3 [New Rule] Multiple Alerts in Different ATT&CK Tactics on a Single Host (#2399)
* [New Rule] Multiple Alerts in Different ATT&CK Tactics on a Single Host

* Update definitions.py

* Update rules/cross-platform/multiple_alerts_different_tactics_host.toml

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
2022-11-18 17:38:27 -03:00
Terrance DeJesus facaef1389 Prep for 8.6 Branch Creation (#2308)
* adding prep for 8.6

* updated ecs and beats packages

* addressed flake errors
2022-09-21 17:01:02 -04:00
Terrance DeJesus 2a3b584433 Prep for 8.5 branch (#2220)
* adding first commit

* renamed branch

* adjusted packages, stack schema and updated schemas

* updated integrations manifest

* adjusted comments to be a little more organized

* adjusted stack-schema-map

* refreshed ecs and beats schema, adjusted stack schema map accordingly
2022-08-09 17:14:42 -04:00
Justin Ibarra cc01d3fb1a Add support for restricted fields (#2053)
* Add support for restricted fields (fields valid only in min/max stack versions)
* add test to ensure rule backports wont exceed min compat
2022-06-27 10:02:15 -05:00
Justin Ibarra 744f56d98e [Bug] resolves bug in Rule version methods (#2021)
* [Bug] resolves bug in Rule version methods

* comment out unused code with notes
2022-06-07 15:40:46 -08:00
Justin Ibarra e850f39526 [Bug] Fix test_matrix_to_lock_version_defaults test (#2014) 2022-06-02 16:34:54 -08:00
Terrance DeJesus 35b1a69ff5 Prep for Creation of 8.4 Branch (#2001)
* prepping for 8.4 branch

* adjusted schemas init file

* adjusted target matrix to only backport to 7.16, updated api schemas

* adjusted the lock-versions workflow to account for 7.16 and up support only

* Add test for version lock to schema map correlation

* decouple from static 7.13 references

* keep patch version for lock

* Update detection_rules/etc/packages.yml

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

Co-authored-by: Jonhnathan <jonhnathancesar@gmail.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
2022-06-02 14:59:18 -04:00
Justin Ibarra 0428e161a8 Refresh ECS/beats schemas up to 8.2 (#1995) 2022-05-25 11:51:43 -08:00