security-tools/sigma-rules
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
3,633 Commits 1 Branch 0 Tags
main
Commit Graph

148 Commits

Author SHA1 Message Date
Mika Ayenson, PhD cc66323d1d [Bug] Omit ES|QL engine columns from required_fields (#6027) 
* Omit Esql.* columns from ES|QL rule required_fields

Kibana treats required_fields as index mappings. ES|QL stats and
similar commands expose Esql.* and Esql_priv.* result columns that
are not mapped on source indices, which produced noisy validation
warnings for shipped rules.

Filter those names when building required_fields. Add a check in
test_esql_endpoint_alerts_index when remote ES|QL validation runs.

Fixes #6026.

* Move required_fields check to its own remote test

* Iterate production rules in required_fields test

* Use direct get_required_fields call in remote test

Skip to_api_format() and call data.get_required_fields(index) directly,
gated on ESQLRuleData. Mirrors the ESQLValidator scope of the fix and
avoids the unrelated packaging steps that to_api_format runs per rule.

* Bump version to 1.6.30

* Centralize ES|QL dynamic field prefix tuple

Define ESQL_DYNAMIC_FIELD_PREFIXES = ("Esql.", "Esql_priv.") in
schemas/definitions.py and reuse it in QueryValidator.get_required_fields,
ESQLValidator.validate_columns_index_mapping, and the remote test.
Single source of truth and consistent ordering across the codebase.
 2026-05-01 17:37:31 -05:00
Eric Forte 9736407ef3 [FR] [DAC] Initial Yaml Support (#5821) 
* Initial Yaml Support
 2026-04-10 11:29:15 -04:00
Eric Forte 75ffa5ec4e [FR] [DaC] Add fine-grained bypass env var for ES|QL keep and metadata validation (#5869) 
* Add fine grain 'keep' req bypass

* Add metadata bypass
 2026-03-24 14:36:45 -04:00
Eric Forte 26d37dd62e [Bug] Ignore Other Keep Wildcards (#5792) 
* Ignore other Keep Wildcards

* Added a unit test for multiple keeps

* Add keep star unit tests
 2026-03-09 19:33:27 -04:00
Eric Forte f306404fe5 [Bug] CLI adds frequency field to system actions (.cases), causing import failure (#5690) 
* No frequency field to cases
 2026-02-11 15:18:20 -05:00
Eric Forte f74c04d11a [Bug] ESQL validation keep Clause Reported Missing Metadata Fields (#5717) 
* Update Keep Field to Handle Comments

* Update for handling inline comments

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>
 2026-02-11 15:02:23 -05:00
Eric Forte d252cae4ee Ignore Keep * for ES|QL hash calc (#5638) 
* Ignore Keep * for ES|QL hash calc

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>


---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
 2026-01-27 23:01:27 -05:00
Eric Forte 891aa8b6d5 [FR] Add keep metadata check to esql schema test (#5441) 
* Add keep metadata check to esql schema test

* Update unit tests

* Allow for keep *

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>

---------

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
 2026-01-14 16:03:24 -05:00
shashank-elastic 1ce072a4e5 Prep for Release 9.3 (#5548) 2026-01-12 21:07:07 +05:30
Mika Ayenson, PhD f40a383b7e [New Rules] Add MITRE ATLAS framework support and GenAI threat detection rules (#5352) 
Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>
Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com>
 2025-12-05 12:26:56 -06:00
Eric Forte 7604c20d9e [FR] Add ESQL rules to dataset exception (#5249) 
* Add ESQL rules to dataset exception

* Add unit test
 2025-10-27 11:03:48 -04:00
Sergey Polzunov c7246313f7 feat: ESQL query validation against Elastic cluster (#4955) 
* Add remote ESQL validation
---------

Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
Co-authored-by: Mika Ayenson <mika.ayenson@elastic.co>
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
 2025-10-15 15:17:07 -04:00
Eric Forte a5c100a65b [Bug] Add unit tests and fix Alert Suppression schema validation for ThresholdQueryRuleData (#5196) 
* Add schema validation for AlertSuppressionMapping

* Add support for indicator match alert suppression

* Add unit tests

* Update order and remove validates_schema method

* Add comments

* Add test for query rule duration only
 2025-10-09 16:21:21 -04:00
Mika Ayenson, PhD f0f7d217c0 [FR] Refactor Schema Validation & Support Multi-Dataset Sequence Validation (#5059) 2025-09-10 13:11:04 -05:00
Mika Ayenson, PhD 35b000b7ab [FR] Add negate DOES NOT MATCH capability to IM rule type (>=9.2) (#5041) 2025-09-09 10:58:53 -05:00
Mika Ayenson, PhD 3c1de72f6b [FR] Add support for 5 group_by fields in threshold rules (>=9.2) (#5040) 2025-09-04 09:24:36 -05:00
Sergey Polzunov ff46a7ab4a fix: Allow different order of the metadata fields in ESQL queries (#4956) 
* Initial commit

* Python project version bump
 2025-08-02 02:26:39 +02:00
Eric Forte a9ad66935c [FR] [DAC] Add Arbitrary File location Support for Local Creation Date (#4915) 
* Add support for local file contents

* Update Rule Params

* Update CLI docs

* Update to Pathlib

* Format updating

* Delete duplicate

* Update logic to handle just local_contents path

* Update to Glob Based Approach

* Updated to use RawRuleCollection

* Fix Logging Typo

* New utils functions no longer needed

* Update naming for convention
 2025-07-31 14:35:00 -04:00
Mika Ayenson, PhD 1dc3926203 [New Rules] External Promotion Alerts (#4903) 2025-07-31 11:00:50 -05:00
Sergey Polzunov 1fb60d6475 fix: type hinting fixes and additional code checks (#4790) 
* first pass

* Adding a dedicated code checking workflow

* Type fixes

* linting config and python version bump

* Type hints

* Drop incorrect config option

* More fixes

* Style fixes

* CI adjustments

* Pyproject fixes

* CI & pyproject fixes

* Proper version bump

* Tests formatting

* Resolve cirtular dependency

* Test fixes

* Make sure the tests are formatted correctly

* Check tweaks

* Bumping python version in CI images

* Pin marshmallow do 3.x because 4.x is not supported

* License fix

* Convert path to str

* Making myself a codeowner

* Missing kwargs param

* Adding a missing kwargs to `set_score`

* Update .github/CODEOWNERS

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* Dropping unnecessary raise

* Dropping skipped test

* Drop unnecessary var

* Drop unused commented-out func

* Disable typehinting for the whole func

* Update linting command

* Invalid type hist on the input param

* Incorrect field type

* Incorrect value used fix

* Stricter values check

* Simpler function call

* Type condition fix

* TOML formatter fix

* Simpligy output conditions

* Formatting

* Use proper types instead of aliases

* MITRE attack fixes

* Using pathlib.Path for an argument

* Use proper method to update a set from a dict

* First round of `ruff` fixes

* More fixes

* More fixes

* Hack against cyclic dependency

* Ignore `PLC0415`

* Remove unused markers

* Cleanup

* Fixing the incorrect condition

* Update .github/CODEOWNERS

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* Set explicit default values for optional fields

* Update the guidelines

* Adding None Defaults

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
 2025-07-01 08:20:55 -05:00
Sergey Polzunov d72cb92d59 Bringing back "fix: Cleaning up the hashable content for the rule" (#4621) (#4668) 2025-04-28 21:59:55 +05:30
Sergey Polzunov b7a324b2e8 Revert "fix: Cleaning up the hashable content for the rule (#4621)" (#4654) 
This reverts commit 80c4f7eacc.
 2025-04-24 19:05:17 +02:00
Sergey Polzunov 80c4f7eacc fix: Cleaning up the hashable content for the rule (#4621) 2025-04-24 14:33:26 +05:30
shashank-elastic 66996ac597 Fix typo in error message (#4489) 2025-02-24 20:16:43 +05:30
Terrance DeJesus 06319b7a13 [Rule Tuning] Add KEEP Command to all ES|QL Rules (#4146) 
* updating ES|QL rules to include KEEP command

* fixed some ES|QL rules with typos; added validation for KEEP command

* fixed ES|QL errors from missing fields

* fixed flake errors

* updated date

* added best practices to hunt docs
 2024-10-09 21:08:38 -04:00
Terrance DeJesus 281926052c [Rule Tuning] Add METADATA checks for non-aggregate ES|QL queries and fix existing (#4126) 
* fixed existing rules;added query checks

* fixed flake errors

* added re.DOTALL to regex pattern, adjusted pattern slightly; reverted some rules

* removed valueError and replaced ValidationError

* adjusted validation error output based on feedback

* Update rules/integrations/aws/persistence_iam_user_created_access_keys_for_another_user.toml

Co-authored-by: Isai <59296946+imays11@users.noreply.github.com>

* Update rules/integrations/aws/persistence_iam_user_created_access_keys_for_another_user.toml

Co-authored-by: Isai <59296946+imays11@users.noreply.github.com>

* added space for failure

* updated to use re.compile

---------

Co-authored-by: Isai <59296946+imays11@users.noreply.github.com>
 2024-10-09 15:25:36 -04:00
Mika Ayenson 10ba6ad5a6 [FR] Add Alert Suppression for Addtional Rule Types (#3986) 2024-08-15 15:03:45 -05:00
Eric Forte 47d7a3acaa [DaC] Beta Release (#3889) 
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
Co-authored-by: Mika Ayenson <mika.ayenson@elastic.co>
 2024-08-06 18:07:12 -04:00
Mika Ayenson 2110ad53f0 [FR] Support new_terms schema import/export w/custom format (#3890) 
* [FR] Support new_terms schema import/export w/custom format

* fix formatter for filters

* handle both rule formats when parsing data view
 2024-07-12 17:17:09 -05:00
Eric Forte ec6038b9d9 Added Schema Check for Data View ID and Index (#3830) 2024-07-09 15:05:12 -04:00
Mika Ayenson 259efaf716 [FR] Loosen Filters Schema Validation (#3753) 2024-06-18 15:57:14 -05:00
Justin Ibarra 54ff270c62 [New Rule] AWS S3 Bucket Enumeration or Brute Force (#3635) 
* [New Rule] AWS S3 Bucket Enumeration or Brute Force
Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com>
---------

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com>
 2024-05-01 15:00:33 -06:00
Justin Ibarra c567d3731a Refresh Kibana module with API updates (#3466) 
* Refresh Kibana module with API updates
* add import/export commands
* rename repo commands
* add RawRuleCollection and DictRule objects
* save exported rules to files; rule.from_rule_resource
* strip unknown fields in schema
* add remote cli test
* update docs
* bump kibana lib version

---------

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
 2024-04-26 11:12:50 -06:00
Eric Forte fbb6df506e Update default (#3574) 2024-04-04 20:27:14 -04:00
Eric Forte 1566c29bae [Bug] KQL fails validation on uppercase keywords (#3568) 
* add todo

* Add a normalize_kql_keywords function to utils

* update rule loader to normalize and warn

* optimized loading

* fix linting

* Moved conversion to kql module.

* Updated unit test

* Refactor KQL parser to normalize keywords via flag

* Fix logic typo

* Update detection_rules/utils.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update lib/kql/kql/__init__.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Updated to fix unit tests and remove warnings

* linting typo

* Added comments

* remove unused imports

* Update kql.parse default

---------

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
 2024-04-04 18:03:30 -04:00
Jonhnathan 67ca13c1ce [Rule Tuning] Replace KQL exceptions for Query DSL Exceptions (#3505) 
* [Rule Tuning] Replace KQL exceptions for Query DSL Exceptions

* update min_stack

* build out schema in more detail for Filters

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* Remove enum for definition

* remove unused import

* remove $state store

* transform state

* add call to super

* add return type hint

* use dataclass metadata

* use Literal type

---------

Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mika.ayenson@elastic.co>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
 2024-04-01 17:44:50 -03:00
Mika Ayenson bb907a4d76 [FR] Add support for investigation_fields (#3550) 2024-04-01 11:52:46 -05:00
Mika Ayenson 8724077a0e [FR] Add support for dataviews in the rule schema (#3510) 2024-03-14 17:43:27 -05:00
Mika Ayenson 542053719b [FR] Skip eql optimizations on parsing query for unique fields (#3443) 2024-02-20 20:25:51 -06:00
Mika Ayenson c3ca01ebcc [FR] Add support for Threshold Alert Suppression (#3433) 2024-02-12 09:55:46 -06:00
Terrance DeJesus d7b62395e7 [FR] Add --include-metadata argument to export-rules command (#3365) 
* added --include-metadata argument to export-rules command

* added type hinting in method definitions

* changed add_metadata to include_metadata

* adjusted argument name to include_metadata in command

* Update detection_rules/main.py

Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>

* fixed flake error

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

---------

Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
 2024-01-04 16:02:48 -05:00
Mika Ayenson face95058f [Bug] Use integration schemas for required_field types (#3303) 2023-12-11 11:32:38 -06:00
Mika Ayenson 7514c0a206 [FR] Add Support for ES|QL Rule Type and Remote Validation (#3281) 
* add suuport for esql type
* add unit tests
* set clients in RemoteConnector from auth methods
* thread remote rules; add engine test
* Add versions to remote validation results

---------

Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
 2023-12-08 12:46:28 -07:00
Jonhnathan aeb1f91320 [Security Content] Introduce Investigate Plugin in Investigation Guides (#3080) 
* [Security Content] Introduce Investigate Plugin in Investigation Guides
* Add compatibility note
* Update Transform format
* update transform unit tests for investigate
* updated docs with transform

---------

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
 2023-12-08 11:54:40 -07:00
Terrance DeJesus 5358361754 Adjust ESQLRuleData to Inherit QueryRuleData Dataclass (#3297) 
* adjusting inheritance of ESQL rule data

* update tests to handle missing index from QueryRuleData

* removed test es|ql rule

---------

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
 2023-11-30 09:06:34 -05:00
Mika Ayenson f7b9a1f8df Update QueryRuleData (#3294) 2023-11-29 09:43:04 -06:00
Mika Ayenson bc39c20eaf FR] Add Core Support for ES|QL Rule Type (#3292) 2023-11-28 13:03:09 -06:00
Eric Forte 66c1d7f3b4 [Bug] Fix typo in downgrade_contents_from_rule (#3272) 
* Fix missing to_dict()

* Update pyproject.toml
 2023-11-14 23:06:04 -05:00
Terrance DeJesus 829f5ea885 [Bug] Add Integration Schema Validation to NewTermsRuleData.validate Method (#3227) 
* adjusted validation method to include integration schema checks

* fixed linting errors

* re-factored NewTermsRuleData and added unit testing
 2023-11-02 16:52:18 -04:00
shashank-elastic 7254c582c5 Move Setup information into setup filed (#3206) 2023-10-23 19:28:18 +05:30