f0f7d217c0
[FR] Refactor Schema Validation & Support Multi-Dataset Sequence Validation ( #5059 )
2025-09-10 13:11:04 -05:00
35b000b7ab
[FR] Add negate DOES NOT MATCH capability to IM rule type (>=9.2) ( #5041 )
2025-09-09 10:58:53 -05:00
3c1de72f6b
[FR] Add support for 5 group_by fields in threshold rules (>=9.2) ( #5040 )
2025-09-04 09:24:36 -05:00
ff46a7ab4a
fix: Allow different order of the metadata fields in ESQL queries ( #4956 )
...
* Initial commit
* Python project version bump
2025-08-02 02:26:39 +02:00
a9ad66935c
[FR] [DAC] Add Arbitrary File location Support for Local Creation Date ( #4915 )
...
* Add support for local file contents
* Update Rule Params
* Update CLI docs
* Update to Pathlib
* Format updating
* Delete duplicate
* Update logic to handle just local_contents path
* Update to Glob Based Approach
* Updated to use RawRuleCollection
* Fix Logging Typo
* New utils functions no longer needed
* Update naming for convention
2025-07-31 14:35:00 -04:00
1dc3926203
[New Rules] External Promotion Alerts ( #4903 )
2025-07-31 11:00:50 -05:00
1fb60d6475
fix: type hinting fixes and additional code checks ( #4790 )
...
* first pass
* Adding a dedicated code checking workflow
* Type fixes
* linting config and python version bump
* Type hints
* Drop incorrect config option
* More fixes
* Style fixes
* CI adjustments
* Pyproject fixes
* CI & pyproject fixes
* Proper version bump
* Tests formatting
* Resolve cirtular dependency
* Test fixes
* Make sure the tests are formatted correctly
* Check tweaks
* Bumping python version in CI images
* Pin marshmallow do 3.x because 4.x is not supported
* License fix
* Convert path to str
* Making myself a codeowner
* Missing kwargs param
* Adding a missing kwargs to `set_score`
* Update .github/CODEOWNERS
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com >
* Dropping unnecessary raise
* Dropping skipped test
* Drop unnecessary var
* Drop unused commented-out func
* Disable typehinting for the whole func
* Update linting command
* Invalid type hist on the input param
* Incorrect field type
* Incorrect value used fix
* Stricter values check
* Simpler function call
* Type condition fix
* TOML formatter fix
* Simpligy output conditions
* Formatting
* Use proper types instead of aliases
* MITRE attack fixes
* Using pathlib.Path for an argument
* Use proper method to update a set from a dict
* First round of `ruff` fixes
* More fixes
* More fixes
* Hack against cyclic dependency
* Ignore `PLC0415`
* Remove unused markers
* Cleanup
* Fixing the incorrect condition
* Update .github/CODEOWNERS
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com >
* Set explicit default values for optional fields
* Update the guidelines
* Adding None Defaults
---------
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com >
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co >
2025-07-01 08:20:55 -05:00
d72cb92d59
Bringing back "fix: Cleaning up the hashable content for the rule" ( #4621 ) ( #4668 )
2025-04-28 21:59:55 +05:30
b7a324b2e8
Revert "fix: Cleaning up the hashable content for the rule ( #4621 )" ( #4654 )
...
This reverts commit 80c4f7eacc
2025-04-24 19:05:17 +02:00
80c4f7eacc
fix: Cleaning up the hashable content for the rule ( #4621 )
2025-04-24 14:33:26 +05:30
66996ac597
Fix typo in error message ( #4489 )
2025-02-24 20:16:43 +05:30
06319b7a13
[Rule Tuning] Add KEEP Command to all ES|QL Rules ( #4146 )
...
* updating ES|QL rules to include KEEP command
* fixed some ES|QL rules with typos; added validation for KEEP command
* fixed ES|QL errors from missing fields
* fixed flake errors
* updated date
* added best practices to hunt docs
2024-10-09 21:08:38 -04:00
281926052c
[Rule Tuning] Add METADATA checks for non-aggregate ES|QL queries and fix existing ( #4126 )
...
* fixed existing rules;added query checks
* fixed flake errors
* added re.DOTALL to regex pattern, adjusted pattern slightly; reverted some rules
* removed valueError and replaced ValidationError
* adjusted validation error output based on feedback
* Update rules/integrations/aws/persistence_iam_user_created_access_keys_for_another_user.toml
Co-authored-by: Isai <59296946+imays11@users.noreply.github.com >
* Update rules/integrations/aws/persistence_iam_user_created_access_keys_for_another_user.toml
Co-authored-by: Isai <59296946+imays11@users.noreply.github.com >
* added space for failure
* updated to use re.compile
---------
Co-authored-by: Isai <59296946+imays11@users.noreply.github.com >
2024-10-09 15:25:36 -04:00
10ba6ad5a6
[FR] Add Alert Suppression for Addtional Rule Types ( #3986 )
2024-08-15 15:03:45 -05:00
47d7a3acaa
[DaC] Beta Release ( #3889 )
...
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com >
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
Co-authored-by: Mika Ayenson <mika.ayenson@elastic.co >
2024-08-06 18:07:12 -04:00
2110ad53f0
[FR] Support new_terms schema import/export w/custom format ( #3890 )
...
* [FR] Support new_terms schema import/export w/custom format
* fix formatter for filters
* handle both rule formats when parsing data view
2024-07-12 17:17:09 -05:00
ec6038b9d9
Added Schema Check for Data View ID and Index ( #3830 )
2024-07-09 15:05:12 -04:00
259efaf716
[FR] Loosen Filters Schema Validation ( #3753 )
2024-06-18 15:57:14 -05:00
54ff270c62
[New Rule] AWS S3 Bucket Enumeration or Brute Force ( #3635 )
...
* [New Rule] AWS S3 Bucket Enumeration or Brute Force
Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com >
---------
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com >
Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com >
2024-05-01 15:00:33 -06:00
c567d3731a
Refresh Kibana module with API updates ( #3466 )
...
* Refresh Kibana module with API updates
* add import/export commands
* rename repo commands
* add RawRuleCollection and DictRule objects
* save exported rules to files; rule.from_rule_resource
* strip unknown fields in schema
* add remote cli test
* update docs
* bump kibana lib version
---------
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com >
2024-04-26 11:12:50 -06:00
fbb6df506e
Update default ( #3574 )
2024-04-04 20:27:14 -04:00
1566c29bae
[Bug] KQL fails validation on uppercase keywords ( #3568 )
...
* add todo
* Add a normalize_kql_keywords function to utils
* update rule loader to normalize and warn
* optimized loading
* fix linting
* Moved conversion to kql module.
* Updated unit test
* Refactor KQL parser to normalize keywords via flag
* Fix logic typo
* Update detection_rules/utils.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Update lib/kql/kql/__init__.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Updated to fix unit tests and remove warnings
* linting typo
* Added comments
* remove unused imports
* Update kql.parse default
---------
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
2024-04-04 18:03:30 -04:00
67ca13c1ce
[Rule Tuning] Replace KQL exceptions for Query DSL Exceptions ( #3505 )
...
* [Rule Tuning] Replace KQL exceptions for Query DSL Exceptions
* update min_stack
* build out schema in more detail for Filters
* Update detection_rules/rule.py
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
* Remove enum for definition
* remove unused import
* remove $state store
* transform state
* add call to super
* add return type hint
* use dataclass metadata
* use Literal type
---------
Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com >
Co-authored-by: Mika Ayenson <Mika.ayenson@elastic.co >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
2024-04-01 17:44:50 -03:00
bb907a4d76
[FR] Add support for investigation_fields ( #3550 )
2024-04-01 11:52:46 -05:00
8724077a0e
[FR] Add support for dataviews in the rule schema ( #3510 )
2024-03-14 17:43:27 -05:00
542053719b
[FR] Skip eql optimizations on parsing query for unique fields ( #3443 )
2024-02-20 20:25:51 -06:00
c3ca01ebcc
[FR] Add support for Threshold Alert Suppression ( #3433 )
2024-02-12 09:55:46 -06:00
d7b62395e7
[FR] Add --include-metadata argument to export-rules command ( #3365 )
...
* added --include-metadata argument to export-rules command
* added type hinting in method definitions
* changed add_metadata to include_metadata
* adjusted argument name to include_metadata in command
* Update detection_rules/main.py
Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com >
* fixed flake error
* Update detection_rules/rule.py
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
---------
Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com >
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
2024-01-04 16:02:48 -05:00
face95058f
[Bug] Use integration schemas for required_field types ( #3303 )
2023-12-11 11:32:38 -06:00
7514c0a206
[FR] Add Support for ES|QL Rule Type and Remote Validation ( #3281 )
...
* add suuport for esql type
* add unit tests
* set clients in RemoteConnector from auth methods
* thread remote rules; add engine test
* Add versions to remote validation results
---------
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com >
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
2023-12-08 12:46:28 -07:00
aeb1f91320
[Security Content] Introduce Investigate Plugin in Investigation Guides ( #3080 )
...
* [Security Content] Introduce Investigate Plugin in Investigation Guides
* Add compatibility note
* Update Transform format
* update transform unit tests for investigate
* updated docs with transform
---------
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com >
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
2023-12-08 11:54:40 -07:00
5358361754
Adjust ESQLRuleData to Inherit QueryRuleData Dataclass ( #3297 )
...
* adjusting inheritance of ESQL rule data
* update tests to handle missing index from QueryRuleData
* removed test es|ql rule
---------
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com >
2023-11-30 09:06:34 -05:00
f7b9a1f8df
Update QueryRuleData ( #3294 )
2023-11-29 09:43:04 -06:00
bc39c20eaf
FR] Add Core Support for ES|QL Rule Type ( #3292 )
2023-11-28 13:03:09 -06:00
66c1d7f3b4
[Bug] Fix typo in downgrade_contents_from_rule ( #3272 )
...
* Fix missing to_dict()
* Update pyproject.toml
2023-11-14 23:06:04 -05:00
829f5ea885
[Bug] Add Integration Schema Validation to NewTermsRuleData.validate Method ( #3227 )
...
* adjusted validation method to include integration schema checks
* fixed linting errors
* re-factored NewTermsRuleData and added unit testing
2023-11-02 16:52:18 -04:00
7254c582c5
Move Setup information into setup filed ( #3206 )
2023-10-23 19:28:18 +05:30
3ab57fb8a7
[FR] Adding Support for missing_field_strategy Field in Alert Suppression ( #3201 )
...
* adding missing field strategy option to alert suppression
* fixed linting errors
* added validate methods for alertsuppression dataclass
* fixed linting errors
* replaced old variable with new variable
* removing test rule
* adding post_load to queryruledata
* changed post_load to validates_schema
* updated unit testing for alert suppression
* fixed linting errors
* changed validates method name to validates_exceptions
* removed min compat for fields
2023-10-19 18:16:54 -04:00
a5a606e804
[New Rule] Adding DGA Rules from Advanced Analytic DGA Package ( #3102 )
...
* Adding DGA rules
* Adding references
* updated rule tags and queries
* Updating min stack version
* added logic to handle ml jobs
* added code comments for clarity
* removing subbed security docs folder
* added event dataset to queries for endpoint; updated note
* removed event dataset
---------
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
Co-authored-by: terrancedejesus <terrance.dejesus@elastic.co >
2023-10-16 15:48:54 -04:00
747ee7d593
[New Rule] Adding Lateral Movement Rules from Advanced Analytic LMD Package ( #3119 )
...
* Adding Lateral Movement Detection rules
* added tags; adjusted tests; updated manifests and schemas
* added default value to build_integrations_schema
* combined analytic and non-dataset packages for related integrations
* adjusted machine learning definitions
* adjusted machine learning definitions
* removed splat for machine learning list due to 3.8 constraints
---------
Co-authored-by: terrancedejesus <terrance.dejesus@elastic.co >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
2023-09-27 14:53:38 -04:00
20de1d8d1d
[FR] Add support for samples in eql 0.9.18 ( #3000 )
2023-09-07 09:01:28 -05:00
9482bda414
Adding related integrations to ML rules ( #2972 )
...
* Adding related integrations to ML rules
* added adjustments to determine related integrations for ML rules
* fixed lint errors
* Empty commit
* Empty commit
* Empty commit
---------
Co-authored-by: Apoorva Joshi <apoorvajoshi@Apoorvas-MBP.lan >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
Co-authored-by: terrancedejesus <terrance.dejesus@elastic.co >
Co-authored-by: Apoorva Joshi <apoorvajoshi@Apoorvas-MBP.fritz.box >
2023-08-22 14:39:18 -04:00
08b646aa94
[FR] 8.10 Release Preparation and Update Main Branch to 8.11 ( #3012 )
...
* prepping for 8.11 branch
* fixed lint errors
* added 8.11 to stack schema map
* trimmed version lock file; adjusted new terms validation
* reverting changes to version lock, stack schema and workflow
2023-08-16 14:23:44 -04:00
3f9e7aced1
[Bug] Strip Non-Public Fields Prior to Uploading Rules ( #2986 )
2023-08-02 12:38:48 -05:00
9f29129585
[FR] Add EQL Rule Type Configuration Fields ( #2918 )
...
* adding initial EQL fields to EQLRuleData
* added validation
* adjusted validation
* fixed flake errors
* adjusted type linting; variable names
* added a min_compat to EQL Rule fields
* Update detection_rules/rule_validators.py
* Update detection_rules/rule_validators.py
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
---------
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
2023-07-13 11:20:14 -04:00
48cf95c8eb
[Rule Tuning] Change Network Rules to Use Network Packet Capture Integration ( #2665 )
...
* updated indexes and updated dates
* added network_traffic integration tag to rules
* reverting changes to resolve conflicts
* metadata changes; indexes changed; schemas and manifest updated
* updated default telnet port connection rule
* updating integration manifests
* adjusted rules; updated integrations; deduplicate packages
2023-06-26 17:35:49 -04:00
6449cecd08
[FR] Add support for building block rules (BBR) ( #2822 )
...
* added test bbr
* initial implementation
* Added Unit test and exempted bbr from integrations
* fixed linting
* Add schema validation to building block rules
* add separate error messages
* fixed linting
* Add testing bbr validation
* fixed linting
* Add default values
* fixed linting
* added defaults
* fixed linting
* cleaned up test rule
* removed .gitkeep
* read .gitkeep
* Switch to using validates_schema
* addressing some linting
* fixed linting
* Update detection_rules/schemas/definitions.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* add env variable check
* fix skip function
* updated name
* Update detection_rules/schemas/definitions.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Add bbr validation unit test
* Clean up comments
* fix linting
* Move convert time to utils
* Moved to rules_building_block
* Add check for only bbr in bbr dir
* fix linting
* additional linting fix
* Changed to bbr rule loader
* fixed bbr default
* Updated error messages and README
* fixed more linting
* Updating root level README
* Fixed convert_time_span calls
* fixed typo in unit test logic and updated txt
* fixed error message
* updated comment for clarity
* Update detection_rules/rule.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Update detection_rules/rule.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Updated validation methods for clarity
* fix doctring location
* Fixed typo
* updated error messages.
* removed excess whitespace
* Add per rule bypass
* Add single rule bypass
* Split unit tests
* Update detection_rules/rule.py
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
* Update detection_rules/rule.py
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
---------
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
2023-06-20 09:00:30 -04:00
2c76527922
Make call to TOMLRuleContents.to_dict from TOMLRuleContents.to_api_format ( #2742 )
...
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
2023-04-25 12:33:43 -04:00
71d12bdda4
[Bug] Unit Tests Passing for Rules with Integrations Not Reflected in Manifests ( #2682 )
...
* add promotion to rulemeta schema class and updated promotion rules
* add promotion to rulemeta schema class and updated promotion rules
* adjusted test_integration_tag and okta rule missing dataset
* fixed flake errors
* updated manifests and schemas to include cloud defend
2023-04-03 09:42:40 -04:00
411ec36ff0
Validate markdown plugin fields ( #2602 )
2023-03-28 09:17:50 -04:00
Mika Ayenson, PhD