Commit Graph

31 Commits

Author SHA1 Message Date
Eric Forte 5adc118f92 [Bug] ES|QL Validation Add Reverse Lookup Check Against Kibana Value (#5747)
* Add reverse lookup check against Kibana value

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
2026-02-20 15:29:51 -05:00
Eric Forte 29d4aeb37a [Bug] [DAC] Auto Gen Schema Fails on Certain Subqueries (#5256)
* Add alignment checking for sub-queries

* Allow field to be over written with original field

* Update rule prompt to allow for int 0 values

* Support custom schema index overwrite

---------

Co-authored-by: shashank-elastic <91139415+shashank-elastic@users.noreply.github.com>
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
2025-11-12 11:21:53 -05:00
Sergey Polzunov c7246313f7 feat: ESQL query validation against Elastic cluster (#4955)
* Add remote ESQL validation
---------

Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
Co-authored-by: Mika Ayenson <mika.ayenson@elastic.co>
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
2025-10-15 15:17:07 -04:00
Mika Ayenson, PhD f0f7d217c0 [FR] Refactor Schema Validation & Support Multi-Dataset Sequence Validation (#5059) 2025-09-10 13:11:04 -05:00
Mika Ayenson, PhD 1dc3926203 [New Rules] External Promotion Alerts (#4903) 2025-07-31 11:00:50 -05:00
Sergey Polzunov 1fb60d6475 fix: type hinting fixes and additional code checks (#4790)
* first pass

* Adding a dedicated code checking workflow

* Type fixes

* linting config and python version bump

* Type hints

* Drop incorrect config option

* More fixes

* Style fixes

* CI adjustments

* Pyproject fixes

* CI & pyproject fixes

* Proper version bump

* Tests formatting

* Resolve cirtular dependency

* Test fixes

* Make sure the tests are formatted correctly

* Check tweaks

* Bumping python version in CI images

* Pin marshmallow do 3.x because 4.x is not supported

* License fix

* Convert path to str

* Making myself a codeowner

* Missing kwargs param

* Adding a missing kwargs to `set_score`

* Update .github/CODEOWNERS

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* Dropping unnecessary raise

* Dropping skipped test

* Drop unnecessary var

* Drop unused commented-out func

* Disable typehinting for the whole func

* Update linting command

* Invalid type hist on the input param

* Incorrect field type

* Incorrect value used fix

* Stricter values check

* Simpler function call

* Type condition fix

* TOML formatter fix

* Simpligy output conditions

* Formatting

* Use proper types instead of aliases

* MITRE attack fixes

* Using pathlib.Path for an argument

* Use proper method to update a set from a dict

* First round of `ruff` fixes

* More fixes

* More fixes

* Hack against cyclic dependency

* Ignore `PLC0415`

* Remove unused markers

* Cleanup

* Fixing the incorrect condition

* Update .github/CODEOWNERS

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* Set explicit default values for optional fields

* Update the guidelines

* Adding None Defaults

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
2025-07-01 08:20:55 -05:00
shashank-elastic 318ab3ffa0 Enhance Readability of KQL validation check failures (#4329) 2025-01-06 22:18:05 +05:30
shashank-elastic 2ff2965cb9 Enhance Readability of validation check failures (#4299) 2024-12-13 19:03:47 +05:30
Eric Forte 0c38662cf3 [FR] [DAC] Add Support for Known Types to Auto-generated Schemas (#3985)
* Add support for autogen known type

* Add support for ML packages

* rename known_type to field_type
2024-08-28 10:48:00 -04:00
Eric Forte 47d7a3acaa [DaC] Beta Release (#3889)
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
Co-authored-by: Mika Ayenson <mika.ayenson@elastic.co>
2024-08-06 18:07:12 -04:00
Eric Forte a4a0bc6a7e [Bug] Query validation failing to capture InSet edge case with ip field types (#3572)
* Move test case to separate file

---------

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
Co-authored-by: shashank-elastic <91139415+shashank-elastic@users.noreply.github.com>
2024-05-06 07:58:42 -04:00
Eric Forte fbb6df506e Update default (#3574) 2024-04-04 20:27:14 -04:00
Eric Forte 1566c29bae [Bug] KQL fails validation on uppercase keywords (#3568)
* add todo

* Add a normalize_kql_keywords function to utils

* update rule loader to normalize and warn

* optimized loading

* fix linting

* Moved conversion to kql module.

* Updated unit test

* Refactor KQL parser to normalize keywords via flag

* Fix logic typo

* Update detection_rules/utils.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update lib/kql/kql/__init__.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Updated to fix unit tests and remove warnings

* linting typo

* Added comments

* remove unused imports

* Update kql.parse default

---------

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2024-04-04 18:03:30 -04:00
Mika Ayenson 8724077a0e [FR] Add support for dataviews in the rule schema (#3510) 2024-03-14 17:43:27 -05:00
Mika Ayenson 542053719b [FR] Skip eql optimizations on parsing query for unique fields (#3443) 2024-02-20 20:25:51 -06:00
Mika Ayenson a873abbb5b [FR] Update Validate Integrations to Check Fields Across All Schema Variations (#3372) 2024-01-18 15:42:22 -06:00
Mika Ayenson 7514c0a206 [FR] Add Support for ES|QL Rule Type and Remote Validation (#3281)
* add suuport for esql type
* add unit tests
* set clients in RemoteConnector from auth methods
* thread remote rules; add engine test
* Add versions to remote validation results

---------

Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
2023-12-08 12:46:28 -07:00
Terrance DeJesus 5358361754 Adjust ESQLRuleData to Inherit QueryRuleData Dataclass (#3297)
* adjusting inheritance of ESQL rule data

* update tests to handle missing index from QueryRuleData

* removed test es|ql rule

---------

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
2023-11-30 09:06:34 -05:00
Mika Ayenson bc39c20eaf FR] Add Core Support for ES|QL Rule Type (#3292) 2023-11-28 13:03:09 -06:00
Mika Ayenson 20de1d8d1d [FR] Add support for samples in eql 0.9.18 (#3000) 2023-09-07 09:01:28 -05:00
Terrance DeJesus 9f29129585 [FR] Add EQL Rule Type Configuration Fields (#2918)
* adding initial EQL fields to EQLRuleData

* added validation

* adjusted validation

* fixed flake errors

* adjusted type linting; variable names

* added a min_compat to EQL Rule fields

* Update detection_rules/rule_validators.py

* Update detection_rules/rule_validators.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

---------

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2023-07-13 11:20:14 -04:00
Terrance DeJesus 73970eb2f2 [FR] Add Support for Multi-Fields and Validation in Rules (#2882) 2023-06-28 20:35:33 -04:00
eric-forte-elastic 8ef2f6557b Patch to allow integration validation if ECS/beats fails (#2701)
* Updated for AND logic

* Added case for no package_intregrations

* Fixed linting

* Added unit test for new functionality

* Fixed linting

* Added valid query tests

* Add unit test for event.dataset

* Switched type calls to isinstance calls

* Removed  unused stack validation call

* Added additional error type

* Fixed linting

* Cleaned up error handling

* fixed linting

* Added proper type hints

* Fixed typo in Unions

* Updated unit test with additional test cases

* Updated  test_invalid_queries unit test

* Fixed linting

* Added kql to unit tests

* Updated tests

* Fixed error handling

* Fixed style issues

* updating integration manifests and schemas

---------

Co-authored-by: terrancedejesus <terrance.dejesus@elastic.co>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
2023-04-18 15:43:35 -04:00
Mika Ayenson 60115443a4 Validate against beats and integrations schemas (#2524) 2023-02-08 12:01:31 -05:00
Mika Ayenson 1784429aa7 [FR] Add Integration Schema Query Validation (#2470) 2023-02-02 16:22:44 -05:00
Mika Ayenson aa8239652d [FR] Add endgame schema validation to detection-rule query (#2257) 2022-10-19 09:54:47 -04:00
Mika Ayenson c76a397969 Add new required_fields as a build-time restricted field (#2059)
* Add new `require_field` restricted field
* validate new fields against BaseRuleData schema and global constant

Co-authored-by: Terrance DeJesus <terrance.dejesus@elastic.co>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
2022-07-06 11:49:44 -04:00
Mika Ayenson 1f015ebe85 1554 update eql schemas to fail validation on text fields (#1866)
* Ensure kql2eql conversion doesnt support `text` fields

* Add unit test cases for`text` not supported in eql

* test `field not recognized` in the rule_validator and output a verbose message.

* use elasticsearch_type_family to lookup text mappings

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
2022-03-23 16:22:26 -04:00
Justin Ibarra 2e78da5c9a Prepare for creation of 8.1 branch (#1700) 2022-01-25 18:11:59 -09:00
Justin Ibarra 781953a0a0 Add min_stack_version to rule metadata (#1173)
* Add min_stack_version to metadata of rule structure
* validate all "stack versions" between defined and current package
* Use master schemas if min_stack_version > current_package

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2021-06-30 13:26:27 -08:00
Ross Wolf 8789dd7c90 Separate out query validation from the class hierarchy (#1136)
* Separate out query validation from the class hierarchy
* Rename to *RuleData for consistency
* Apply suggestions from code review
* Fix lint error
2021-04-21 14:55:26 -06:00