Commit Graph

40 Commits

Author SHA1 Message Date
Mika Ayenson 7514c0a206 [FR] Add Support for ES|QL Rule Type and Remote Validation (#3281)
* add suuport for esql type
* add unit tests
* set clients in RemoteConnector from auth methods
* thread remote rules; add engine test
* Add versions to remote validation results

---------

Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
2023-12-08 12:46:28 -07:00
Jonhnathan aeb1f91320 [Security Content] Introduce Investigate Plugin in Investigation Guides (#3080)
* [Security Content] Introduce Investigate Plugin in Investigation Guides
* Add compatibility note
* Update Transform format
* update transform unit tests for investigate
* updated docs with transform

---------

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
2023-12-08 11:54:40 -07:00
Mika Ayenson bc39c20eaf FR] Add Core Support for ES|QL Rule Type (#3292) 2023-11-28 13:03:09 -06:00
Mika Ayenson 93ad4b0959 Add UEBA Tag (#3277) 2023-11-20 13:51:13 -06:00
Mika Ayenson d0b0216362 [FR] Support missing events (#3153) 2023-10-31 16:20:52 -05:00
Mika Ayenson a808130390 Cleanup saved_query references (#3205) 2023-10-26 18:07:33 -05:00
Terrance DeJesus 3ab57fb8a7 [FR] Adding Support for missing_field_strategy Field in Alert Suppression (#3201)
* adding missing field strategy option to alert suppression

* fixed linting errors

* added validate methods for alertsuppression dataclass

* fixed linting errors

* replaced old variable with new variable

* removing test rule

* adding post_load to queryruledata

* changed post_load to validates_schema

* updated unit testing for alert suppression

* fixed linting errors

* changed validates method name to validates_exceptions

* removed min compat for fields
2023-10-19 18:16:54 -04:00
Justin Ibarra 7f8a9849c4 [New Rule] File Compressed or Archived into Common Format (#3173)
* [New Rule] File Compressed or Archived into Common Format
* new build-threat-map-entry-command

---------

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>
2023-10-11 11:34:34 -07:00
Apoorva Joshi 747ee7d593 [New Rule] Adding Lateral Movement Rules from Advanced Analytic LMD Package (#3119)
* Adding Lateral Movement Detection rules

* added tags; adjusted tests; updated manifests and schemas

* added default value to build_integrations_schema

* combined analytic and non-dataset packages for related integrations

* adjusted machine learning definitions

* adjusted machine learning definitions

* removed splat for machine learning list due to 3.8 constraints

---------

Co-authored-by: terrancedejesus <terrance.dejesus@elastic.co>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
2023-09-27 14:53:38 -04:00
Mika Ayenson 20de1d8d1d [FR] Add support for samples in eql 0.9.18 (#3000) 2023-09-07 09:01:28 -05:00
Mika Ayenson 3f9e7aced1 [Bug] Strip Non-Public Fields Prior to Uploading Rules (#2986) 2023-08-02 12:38:48 -05:00
Mika Ayenson 2ff4584456 load unsupported rule type from schema (#2893) 2023-06-29 15:32:32 -04:00
Mika Ayenson cec41b4072 [FR Build a limited compatible rule ndjson for older stacks (#2885) 2023-06-29 10:18:24 -04:00
Jonhnathan a7e605a0e5 [Rule Tuning] [BUG] Revert PowerShell Query modifications from #2823 (#2889)
* Revert query mods done in https://github.com/elastic/detection-rules/pull/2823

* Add exception to unit test

* fixed linting

* proper linting fix

* updated to add to definitions.py

* fix linting

---------

Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
2023-06-28 15:55:43 -03:00
Terrance DeJesus 48cf95c8eb [Rule Tuning] Change Network Rules to Use Network Packet Capture Integration (#2665)
* updated indexes and updated dates

* added network_traffic integration tag to rules

* reverting changes to resolve conflicts

* metadata changes; indexes changed; schemas and manifest updated

* updated default telnet port connection rule

* updating integration manifests

* adjusted rules; updated integrations; deduplicate packages
2023-06-26 17:35:49 -04:00
Terrance DeJesus d829b145ef [Bug] Fix Tag Navigator Generation (#2875)
* bug fix for tag navigator generation

* addressing flake errors

* added unit test to ensure prefix exists

* updated unit test case sensitivity

* moved expected tags to definitions.py

* removed expected prefixes

* revert downloadable updates JSON file
2023-06-23 10:44:55 -04:00
eric-forte-elastic 6449cecd08 [FR] Add support for building block rules (BBR) (#2822)
* added test bbr

* initial implementation

* Added Unit test and exempted bbr from integrations

* fixed linting

* Add schema validation to building block rules

* add separate error messages

* fixed linting

* Add testing bbr validation

* fixed linting

* Add default values

* fixed linting

* added defaults

* fixed linting

* cleaned up test rule

* removed .gitkeep

* read .gitkeep

* Switch to using validates_schema

* addressing some linting

* fixed linting

* Update detection_rules/schemas/definitions.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* add env variable check

* fix skip function

* updated name

* Update detection_rules/schemas/definitions.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Add bbr validation unit test

* Clean up comments

* fix linting

* Move convert time to utils

* Moved to rules_building_block

* Add check for only bbr in bbr dir

* fix linting

* additional linting fix

* Changed to bbr rule loader

* fixed bbr default

* Updated error messages and README

* fixed more linting

* Updating root level README

* Fixed convert_time_span calls

* fixed typo in unit test logic and updated txt

* fixed error message

* updated comment for clarity

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Updated validation methods for clarity

* fix doctring location

* Fixed typo

* updated error messages.

* removed excess whitespace

* Add per rule bypass

* Add single rule bypass

* Split unit tests

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

---------

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2023-06-20 09:00:30 -04:00
Terrance DeJesus 792da36fb9 [Bug] Add Cloud Defend to definitions.NON_DATASET_PACKAGES (#2764)
* updating code to include cloud defend package

* updated integration manifests and schemas
2023-04-28 11:23:48 -04:00
Terrance DeJesus 7e28b8fc50 [FR] Support Rule Alert Suppression in Rule Schema (#2660)
* adding initial solution for alert suppression support in rule schema

* reverting rule changes

* fixing flake errors

* reverting rule changes

* adding unit tests

* addressing flake errors

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update detection_rules/schemas/definitions.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* adjusting rule.py after commits

* adjusted test_group_field_in_schemas to check integrations

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* nested AlertSuppressDuration class under mapping class

* adjusted dataclass naming

* added unit test to ensure rule is KQL

* fixing flake errors

* added docstrings

---------

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
2023-03-27 15:37:35 -04:00
Mika Ayenson 1784429aa7 [FR] Add Integration Schema Query Validation (#2470) 2023-02-02 16:22:44 -05:00
Jonhnathan 0acbe1d832 [New Rule] Multiple Alerts Involving a User (#2401)
* [New Rule] Multiple Alerts Involving a User

* Update definitions.py

* update query

* Update multiple_alerts_involving_user.toml

Co-authored-by: Colson Wilhoit <48036388+DefSecSentinel@users.noreply.github.com>
2023-01-03 12:25:40 -03:00
Terrance DeJesus 7e459dd585 [FR] Add support for New Terms Fields and Window Start History (#2360)
* adding support new_terms_fields and window_start_history

* adjusted rule.py to address flake errors

* added assertion error if history_window_start does not exist

* removed sample rule

* removed self.rule_id from DataValidator

* added new_terms to RuleType

* changed new terms to its own class in rule.py

* removed nonexisting function call in DataValidator class

* adjusted new_terms field value in dataclass

* changed literal type for history_window_start; view-rule working

* removing test TOML rule

* addressed flake errors for missing newlines

* added validation option and adjusted object referencing

* adjusted validation method call in post_validation

* addressed flake errors for multiple spaces

* added transform method to NewTermsRuleData class

* added validation for min stack version and new terms array length restraints

* added validation for unique new terms array

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* removed historywindowstart definition and adjusted subclass

* removed test rule from commit

* adjusted if/else for data transform method check

* adjusted stack-schema-map; validation method name

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* added assertion for history_window_start field value

* added variables for feature min stack and extended field min stack

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* addressed flake errors for continuation line with same indent

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2022-12-05 14:07:33 -05:00
Jonhnathan a7caa4baf3 [New Rule] Multiple Alerts in Different ATT&CK Tactics on a Single Host (#2399)
* [New Rule] Multiple Alerts in Different ATT&CK Tactics on a Single Host

* Update definitions.py

* Update rules/cross-platform/multiple_alerts_different_tactics_host.toml

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
2022-11-18 17:38:27 -03:00
Justin Ibarra 744f56d98e [Bug] resolves bug in Rule version methods (#2021)
* [Bug] resolves bug in Rule version methods

* comment out unused code with notes
2022-06-07 15:40:46 -08:00
Justin Ibarra 22679e16d2 Add delta command to determine changes to endpoint rules between tags (#1943)
* update git tag loader to be compatible with lock validation
* add diff command
* default to query for missing rules
2022-05-03 12:30:11 -08:00
Justin Ibarra c803160e4f Validate version lock and deprecation files on load and save (#1884)
* Validate version lock and deprecation files on load and save
* add missing types for previous lock entries
* bump marshmallow_dataclass
2022-04-26 22:17:20 -08:00
Terrance DeJesus e72031a71a added comprehensive timeline template definitions (#1905) 2022-04-01 08:51:54 -08:00
Justin Ibarra d753ecb8d8 Add pattern for "name" in rule schema (#1669) 2022-01-25 12:03:27 -09:00
Justin Ibarra ab17dfcc28 [Bug] Tighten definitions validation patterns (#1396)
* [Bug] Anchor validation patterns
* Deprecate rule with invalid rule_id and duplicate as new one

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2021-10-26 10:26:20 -05:00
Justin Ibarra 163d9e3864 Update cardinality field in schema for threshold rules (#1349)
* Make cardinality array in schema for threshold rules
* update master, 7.12, 7.13, and 7.14 schemas with cardinality fix
* fix 7.12 downgrade to handle cardinality as an array

* Add two new rules to detect agent spoofing


Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2021-07-21 08:32:54 -08:00
Justin Ibarra 0ec8d67e78 Refactor experimental ML CLI and code (#1218)
* move github and ml to their own files
* refactor release and ml commands
* update ML readmes
* add unzip_to_dict function
* prompt for model ID in remove-model
* update experimental rule upload process
* update remove-scripts-pipelines to take multiple options

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
Co-authored-by: Apoorva <appujo@gmail.com>
2021-06-02 20:37:12 -08:00
Ross Wolf eb40c52c7c Port historical schemas to jsonschema (#1084)
* Port historical schemas to jsonschema
* Add marshmallow-json dependency
* Mark etc/api_schemas as binary
* Remove gitattributes attempt
* Lint fix
* Apply PR feedback
* Additional PR feedback
* Extract stack version from packages.yml
* Fix the backport schemas
* Cache the schema reads
* Add migration for #1167
* Make a separate 'migration not found' error
2021-05-13 14:27:32 -06:00
Andrew Pease 92eaa5b18a [New Rule] Threat intel indicator match rule (#1133) 2021-04-26 07:07:04 -05:00
Justin Ibarra cabe9239c0 Add threat_match rule type (#1138) 2021-04-22 09:03:57 -08:00
Ross Wolf 8789dd7c90 Separate out query validation from the class hierarchy (#1136)
* Separate out query validation from the class hierarchy
* Rename to *RuleData for consistency
* Apply suggestions from code review
* Fix lint error
2021-04-21 14:55:26 -06:00
Justin Ibarra e656a984b3 Update threshold rule schema to disallow empty field string (#1099) 2021-04-15 16:22:45 -06:00
Ross Wolf 07be6b701d Change the asset .type field (#1075) 2021-04-05 10:50:58 -06:00
Ross Wolf 1e6e49a2cb Change the JSON schema for the security_rule Kibana asset (#1066)
* Change the JSON schema for the security_rule Kibana asset
* Use the asset type for the folder name
2021-03-30 13:31:02 -06:00
Ross Wolf c0af222e7e Move Rule into a dataclass (#1029)
* WIP: Convert Rule to a dataclass
* Fix make release
* Lint fixes
* Remove dead code
* Fix lint and tests
* Use Python 3.8 in GitHub actions
* Update README to 3.8+
* Add Python 3.8 assertion
* Fix is_dirty property
* Remove incorrect pop from contents
* Add mixin with from_dict() and to_dict() methods
* Bypass validation for deprecated rules
* Fix rule_prompt
* Fix dict_hash usage
* Fix rule_event_search
* Switch to definitions.Date
* Fix toml-lint command, ignoring 'unneeded defaults'
* Moved severity Literal to definitions.Severity
* Remove BaseMarshmallowDataclass
* Fix lint and tests
* Add maturity to metadata for rule prompt loop
* Fix typo in devtools
* Use rule loader to load single rule in toml-lint
* Add Schema hint to __schema method
* Add MITREAttackURL definition
* Fix is_dirty to compare sha<-->sha
* Normalize the autoformatted rule output for API and toml-lint
* Make the package hash match
* Make the rule object mutable but not rule contents
* Restore the rules
2021-03-24 10:24:32 -06:00
Justin Ibarra fc9dfde2c4 Generate an integrations package from a release (#983)
* Generate an integrations package files during a release build
2021-03-09 13:30:12 -09:00