Commit Graph

124 Commits

Author SHA1 Message Date
Terrance DeJesus 06319b7a13 [Rule Tuning] Add KEEP Command to all ES|QL Rules (#4146)
* updating ES|QL rules to include KEEP command

* fixed some ES|QL rules with typos; added validation for KEEP command

* fixed ES|QL errors from missing fields

* fixed flake errors

* updated date

* added best practices to hunt docs
2024-10-09 21:08:38 -04:00
Terrance DeJesus 281926052c [Rule Tuning] Add METADATA checks for non-aggregate ES|QL queries and fix existing (#4126)
* fixed existing rules;added query checks

* fixed flake errors

* added re.DOTALL to regex pattern, adjusted pattern slightly; reverted some rules

* removed valueError and replaced ValidationError

* adjusted validation error output based on feedback

* Update rules/integrations/aws/persistence_iam_user_created_access_keys_for_another_user.toml

Co-authored-by: Isai <59296946+imays11@users.noreply.github.com>

* Update rules/integrations/aws/persistence_iam_user_created_access_keys_for_another_user.toml

Co-authored-by: Isai <59296946+imays11@users.noreply.github.com>

* added space for failure

* updated to use re.compile

---------

Co-authored-by: Isai <59296946+imays11@users.noreply.github.com>
2024-10-09 15:25:36 -04:00
Mika Ayenson 10ba6ad5a6 [FR] Add Alert Suppression for Addtional Rule Types (#3986) 2024-08-15 15:03:45 -05:00
Eric Forte 47d7a3acaa [DaC] Beta Release (#3889)
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
Co-authored-by: Mika Ayenson <mika.ayenson@elastic.co>
2024-08-06 18:07:12 -04:00
Mika Ayenson 2110ad53f0 [FR] Support new_terms schema import/export w/custom format (#3890)
* [FR] Support new_terms schema import/export w/custom format

* fix formatter for filters

* handle both rule formats when parsing data view
2024-07-12 17:17:09 -05:00
Eric Forte ec6038b9d9 Added Schema Check for Data View ID and Index (#3830) 2024-07-09 15:05:12 -04:00
Mika Ayenson 259efaf716 [FR] Loosen Filters Schema Validation (#3753) 2024-06-18 15:57:14 -05:00
Justin Ibarra 54ff270c62 [New Rule] AWS S3 Bucket Enumeration or Brute Force (#3635)
* [New Rule] AWS S3 Bucket Enumeration or Brute Force
Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com>
---------

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com>
2024-05-01 15:00:33 -06:00
Justin Ibarra c567d3731a Refresh Kibana module with API updates (#3466)
* Refresh Kibana module with API updates
* add import/export commands
* rename repo commands
* add RawRuleCollection and DictRule objects
* save exported rules to files; rule.from_rule_resource
* strip unknown fields in schema
* add remote cli test
* update docs
* bump kibana lib version

---------

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
2024-04-26 11:12:50 -06:00
Eric Forte fbb6df506e Update default (#3574) 2024-04-04 20:27:14 -04:00
Eric Forte 1566c29bae [Bug] KQL fails validation on uppercase keywords (#3568)
* add todo

* Add a normalize_kql_keywords function to utils

* update rule loader to normalize and warn

* optimized loading

* fix linting

* Moved conversion to kql module.

* Updated unit test

* Refactor KQL parser to normalize keywords via flag

* Fix logic typo

* Update detection_rules/utils.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update lib/kql/kql/__init__.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Updated to fix unit tests and remove warnings

* linting typo

* Added comments

* remove unused imports

* Update kql.parse default

---------

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2024-04-04 18:03:30 -04:00
Jonhnathan 67ca13c1ce [Rule Tuning] Replace KQL exceptions for Query DSL Exceptions (#3505)
* [Rule Tuning] Replace KQL exceptions for Query DSL Exceptions

* update min_stack

* build out schema in more detail for Filters

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* Remove enum for definition

* remove unused import

* remove $state store

* transform state

* add call to super

* add return type hint

* use dataclass metadata

* use Literal type

---------

Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mika.ayenson@elastic.co>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2024-04-01 17:44:50 -03:00
Mika Ayenson bb907a4d76 [FR] Add support for investigation_fields (#3550) 2024-04-01 11:52:46 -05:00
Mika Ayenson 8724077a0e [FR] Add support for dataviews in the rule schema (#3510) 2024-03-14 17:43:27 -05:00
Mika Ayenson 542053719b [FR] Skip eql optimizations on parsing query for unique fields (#3443) 2024-02-20 20:25:51 -06:00
Mika Ayenson c3ca01ebcc [FR] Add support for Threshold Alert Suppression (#3433) 2024-02-12 09:55:46 -06:00
Terrance DeJesus d7b62395e7 [FR] Add --include-metadata argument to export-rules command (#3365)
* added --include-metadata argument to export-rules command

* added type hinting in method definitions

* changed add_metadata to include_metadata

* adjusted argument name to include_metadata in command

* Update detection_rules/main.py

Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>

* fixed flake error

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

---------

Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2024-01-04 16:02:48 -05:00
Mika Ayenson face95058f [Bug] Use integration schemas for required_field types (#3303) 2023-12-11 11:32:38 -06:00
Mika Ayenson 7514c0a206 [FR] Add Support for ES|QL Rule Type and Remote Validation (#3281)
* add suuport for esql type
* add unit tests
* set clients in RemoteConnector from auth methods
* thread remote rules; add engine test
* Add versions to remote validation results

---------

Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
2023-12-08 12:46:28 -07:00
Jonhnathan aeb1f91320 [Security Content] Introduce Investigate Plugin in Investigation Guides (#3080)
* [Security Content] Introduce Investigate Plugin in Investigation Guides
* Add compatibility note
* Update Transform format
* update transform unit tests for investigate
* updated docs with transform

---------

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
2023-12-08 11:54:40 -07:00
Terrance DeJesus 5358361754 Adjust ESQLRuleData to Inherit QueryRuleData Dataclass (#3297)
* adjusting inheritance of ESQL rule data

* update tests to handle missing index from QueryRuleData

* removed test es|ql rule

---------

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
2023-11-30 09:06:34 -05:00
Mika Ayenson f7b9a1f8df Update QueryRuleData (#3294) 2023-11-29 09:43:04 -06:00
Mika Ayenson bc39c20eaf FR] Add Core Support for ES|QL Rule Type (#3292) 2023-11-28 13:03:09 -06:00
Eric Forte 66c1d7f3b4 [Bug] Fix typo in downgrade_contents_from_rule (#3272)
* Fix missing to_dict()

* Update pyproject.toml
2023-11-14 23:06:04 -05:00
Terrance DeJesus 829f5ea885 [Bug] Add Integration Schema Validation to NewTermsRuleData.validate Method (#3227)
* adjusted validation method to include integration schema checks

* fixed linting errors

* re-factored NewTermsRuleData and added unit testing
2023-11-02 16:52:18 -04:00
shashank-elastic 7254c582c5 Move Setup information into setup filed (#3206) 2023-10-23 19:28:18 +05:30
Terrance DeJesus 3ab57fb8a7 [FR] Adding Support for missing_field_strategy Field in Alert Suppression (#3201)
* adding missing field strategy option to alert suppression

* fixed linting errors

* added validate methods for alertsuppression dataclass

* fixed linting errors

* replaced old variable with new variable

* removing test rule

* adding post_load to queryruledata

* changed post_load to validates_schema

* updated unit testing for alert suppression

* fixed linting errors

* changed validates method name to validates_exceptions

* removed min compat for fields
2023-10-19 18:16:54 -04:00
Apoorva Joshi a5a606e804 [New Rule] Adding DGA Rules from Advanced Analytic DGA Package (#3102)
* Adding DGA rules

* Adding references

* updated rule tags and queries

* Updating min stack version

* added logic to handle ml jobs

* added code comments for clarity

* removing subbed security docs folder

* added event dataset to queries for endpoint; updated note

* removed event dataset

---------

Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: terrancedejesus <terrance.dejesus@elastic.co>
2023-10-16 15:48:54 -04:00
Apoorva Joshi 747ee7d593 [New Rule] Adding Lateral Movement Rules from Advanced Analytic LMD Package (#3119)
* Adding Lateral Movement Detection rules

* added tags; adjusted tests; updated manifests and schemas

* added default value to build_integrations_schema

* combined analytic and non-dataset packages for related integrations

* adjusted machine learning definitions

* adjusted machine learning definitions

* removed splat for machine learning list due to 3.8 constraints

---------

Co-authored-by: terrancedejesus <terrance.dejesus@elastic.co>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
2023-09-27 14:53:38 -04:00
Mika Ayenson 20de1d8d1d [FR] Add support for samples in eql 0.9.18 (#3000) 2023-09-07 09:01:28 -05:00
Apoorva Joshi 9482bda414 Adding related integrations to ML rules (#2972)
* Adding related integrations to ML rules

* added adjustments to determine related integrations for ML rules

* fixed lint errors

* Empty commit

* Empty commit

* Empty commit

---------

Co-authored-by: Apoorva Joshi <apoorvajoshi@Apoorvas-MBP.lan>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: terrancedejesus <terrance.dejesus@elastic.co>
Co-authored-by: Apoorva Joshi <apoorvajoshi@Apoorvas-MBP.fritz.box>
2023-08-22 14:39:18 -04:00
Terrance DeJesus 08b646aa94 [FR] 8.10 Release Preparation and Update Main Branch to 8.11 (#3012)
* prepping for 8.11 branch

* fixed lint errors

* added 8.11 to stack schema map

* trimmed version lock file; adjusted new terms validation

* reverting changes to version lock, stack schema and workflow
2023-08-16 14:23:44 -04:00
Mika Ayenson 3f9e7aced1 [Bug] Strip Non-Public Fields Prior to Uploading Rules (#2986) 2023-08-02 12:38:48 -05:00
Terrance DeJesus 9f29129585 [FR] Add EQL Rule Type Configuration Fields (#2918)
* adding initial EQL fields to EQLRuleData

* added validation

* adjusted validation

* fixed flake errors

* adjusted type linting; variable names

* added a min_compat to EQL Rule fields

* Update detection_rules/rule_validators.py

* Update detection_rules/rule_validators.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

---------

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2023-07-13 11:20:14 -04:00
Terrance DeJesus 48cf95c8eb [Rule Tuning] Change Network Rules to Use Network Packet Capture Integration (#2665)
* updated indexes and updated dates

* added network_traffic integration tag to rules

* reverting changes to resolve conflicts

* metadata changes; indexes changed; schemas and manifest updated

* updated default telnet port connection rule

* updating integration manifests

* adjusted rules; updated integrations; deduplicate packages
2023-06-26 17:35:49 -04:00
eric-forte-elastic 6449cecd08 [FR] Add support for building block rules (BBR) (#2822)
* added test bbr

* initial implementation

* Added Unit test and exempted bbr from integrations

* fixed linting

* Add schema validation to building block rules

* add separate error messages

* fixed linting

* Add testing bbr validation

* fixed linting

* Add default values

* fixed linting

* added defaults

* fixed linting

* cleaned up test rule

* removed .gitkeep

* read .gitkeep

* Switch to using validates_schema

* addressing some linting

* fixed linting

* Update detection_rules/schemas/definitions.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* add env variable check

* fix skip function

* updated name

* Update detection_rules/schemas/definitions.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Add bbr validation unit test

* Clean up comments

* fix linting

* Move convert time to utils

* Moved to rules_building_block

* Add check for only bbr in bbr dir

* fix linting

* additional linting fix

* Changed to bbr rule loader

* fixed bbr default

* Updated error messages and README

* fixed more linting

* Updating root level README

* Fixed convert_time_span calls

* fixed typo in unit test logic and updated txt

* fixed error message

* updated comment for clarity

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Updated validation methods for clarity

* fix doctring location

* Fixed typo

* updated error messages.

* removed excess whitespace

* Add per rule bypass

* Add single rule bypass

* Split unit tests

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

---------

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2023-06-20 09:00:30 -04:00
Justin Ibarra 2c76527922 Make call to TOMLRuleContents.to_dict from TOMLRuleContents.to_api_format (#2742)
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2023-04-25 12:33:43 -04:00
Terrance DeJesus 71d12bdda4 [Bug] Unit Tests Passing for Rules with Integrations Not Reflected in Manifests (#2682)
* add promotion to rulemeta schema class and updated promotion rules

* add promotion to rulemeta schema class and updated promotion rules

* adjusted test_integration_tag and okta rule missing dataset

* fixed flake errors

* updated manifests and schemas to include cloud defend
2023-04-03 09:42:40 -04:00
Justin Ibarra 411ec36ff0 Validate markdown plugin fields (#2602) 2023-03-28 09:17:50 -04:00
Terrance DeJesus 7e28b8fc50 [FR] Support Rule Alert Suppression in Rule Schema (#2660)
* adding initial solution for alert suppression support in rule schema

* reverting rule changes

* fixing flake errors

* reverting rule changes

* adding unit tests

* addressing flake errors

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update detection_rules/schemas/definitions.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* adjusting rule.py after commits

* adjusted test_group_field_in_schemas to check integrations

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* nested AlertSuppressDuration class under mapping class

* adjusted dataclass naming

* added unit test to ensure rule is KQL

* fixing flake errors

* added docstrings

---------

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
2023-03-27 15:37:35 -04:00
Terrance DeJesus f8d26f4ce0 [Bug] Removed Strip Calls in Favor of F-Strings with Major and Minor Versions (#2541)
* removed strip calls in favor of f-strings with major and minor versions

* changed variable reference in minor_release of bump-pkg-versions
2023-02-10 13:18:53 -05:00
Terrance DeJesus 58ba72d5bf patch fix for 2503 update addressing separate bugs (#2528) 2023-02-07 16:09:17 -05:00
Terrance DeJesus 4054eb43d1 patch fix for 2503 (#2527) 2023-02-07 15:40:51 -05:00
Terrance DeJesus fb2b4529c5 [FR] Adapt PyPi semver Library and Remove Custom (#2503)
* removed custom semver and replaced with pypi

* updated beats.py version references

* updated bump-versions CLI command to use semver and change logic

* updated schemas __init__, test_version_lock and unstage incompatible rules CLI

* updated test_stack_schema_map in TestVersions unittest

* updated test_all_rules unit testing Version() references

* updated stack_compat.py for get_restricted_field references)

* updated version_lock.py Version() references

* updated docs.py Version() reference for parse_registry

* updated devtools.py Version() reference for trim-version-lock

* updated mixins.py Version() reference in validate_field_compatibility

* adjusted schemas.__init__ Version() reference in get_stack_schemas

* adjusted ecs.py Version() references

* adjusted integrations.py Version() references

* adjusted rule.py Version() references

* sorted imports

* replaced custom semver with pypi semver in unit test files

* addressed unit test and flake errors

* changed semver strings casted to version_lock.py

* fixed sorting in integrations.py

* updated bump-pkgs-versions CLI command

* adjusted semantic version in unstage-incompatible-rules command

* adjusted semver import to VersionInfo

* added semver 3 and adjusted import names

* added option_minor_and_patch parameter where version is major.minor

* updated bump-pkg-versions to always save to packages.yml

* removed leftover split call & updated find latest compatible version command

* updated integrations.py, version_lock.py and schemas.__init__.py

* changed fstring reference in downgrade function

* reverted formatting changes for detection_rules __init__.py

* added newline to detection_rules __init__.py

* adjusted finding latest_release for attack package logic

* adjusted unstage-incompatible-rules command logic comparing versions

* removing changes from misc.py related to auto-formatting

* adding newline to misc.py

* fixed bug in downgrade function calling decorators

* added semantic version validation on migrate decorator function

* added expected type returned from find_latest_integration_version in integrations.py

* add comment about stripped versions for version lock file

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

---------

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2023-02-07 14:26:29 -05:00
Mika Ayenson 1784429aa7 [FR] Add Integration Schema Query Validation (#2470) 2023-02-02 16:22:44 -05:00
Terrance DeJesus 4312d8c958 [FR] Add Endpoint, APM and Windows Integration Tags to Rules and Supportability (#2429)
* initial commit

* addressing flake errors

* added apm to _get_packagted_integrations logic

* addressed flake errors

* adjusted integration schema and updated rules to be a list

* updated several rules and removed a unit test

* updated rules with logs-* only index patterns

* Update tests/test_all_rules.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* addressed flake errors

* integration is none is windows, endpoint or apm

* adding rules with accepted incoming changes from main

* fixed tag and tactic alignment errors from unit testing

* adjusted unit testing logic for integration tags; added more exclusion rules

* adjusted test_integration logic to be rule resistent and skip if -8.3

* adjusted comments for unit test skip

* fixed merge conflicts from main

* changing test_integration_tag to remove logic for rule version comparisons

* added integration tag to new rule

* adjusted rules updated_date value

* ignore guided onboarding rule in unit tests

* added integration tag to new rule

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2023-01-04 09:30:07 -05:00
Terrance DeJesus e9169b4cfa [Bug] Add Non-ECS Checks to New Terms Rule Validation (#2435)
* initial commit with changes to new terms validation

* adjusted validation to call KQLValidator for flattened ECS variable

* changed call to KQLValidator instead of super; validate from same variable

* removed testing rules

* removed commented line

* Version() called on all string versions prior to comparison logic

* adjusted assert error punctuation
2022-12-19 12:44:42 -05:00
Terrance DeJesus 7e459dd585 [FR] Add support for New Terms Fields and Window Start History (#2360)
* adding support new_terms_fields and window_start_history

* adjusted rule.py to address flake errors

* added assertion error if history_window_start does not exist

* removed sample rule

* removed self.rule_id from DataValidator

* added new_terms to RuleType

* changed new terms to its own class in rule.py

* removed nonexisting function call in DataValidator class

* adjusted new_terms field value in dataclass

* changed literal type for history_window_start; view-rule working

* removing test TOML rule

* addressed flake errors for missing newlines

* added validation option and adjusted object referencing

* adjusted validation method call in post_validation

* addressed flake errors for multiple spaces

* added transform method to NewTermsRuleData class

* added validation for min stack version and new terms array length restraints

* added validation for unique new terms array

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* removed historywindowstart definition and adjusted subclass

* removed test rule from commit

* adjusted if/else for data transform method check

* adjusted stack-schema-map; validation method name

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* added assertion for history_window_start field value

* added variables for feature min stack and extended field min stack

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* addressed flake errors for continuation line with same indent

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2022-12-05 14:07:33 -05:00
Mika Ayenson aa8239652d [FR] Add endgame schema validation to detection-rule query (#2257) 2022-10-19 09:54:47 -04:00
Terrance DeJesus b31a1b761c [FR] Re-factor Build Integrations Manifest (#2274)
* adjusted how integrations list is created

* removed unused import and addressed linting errors

* adjusted integration_manifest dictionary to only load latest major

* adjusted manifests sourcing from GH to EPR CDN

* addressed flake errors

* added some additional comments and formatting

* updaing integration-manifests file

* adjusted test_integration testing

* addressed flake errors

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* added folder unit tests

* updated unit test to remove network calls

* Update tests/test_all_rules.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2022-09-28 09:33:49 -04:00