10 Commits

Author SHA1 Message Date
Mika Ayenson, PhD cc66323d1d [Bug] Omit ES|QL engine columns from required_fields (#6027)
* Omit Esql.* columns from ES|QL rule required_fields

Kibana treats required_fields as index mappings. ES|QL stats and
similar commands expose Esql.* and Esql_priv.* result columns that
are not mapped on source indices, which produced noisy validation
warnings for shipped rules.

Filter those names when building required_fields. Add a check in
test_esql_endpoint_alerts_index when remote ES|QL validation runs.

Fixes #6026.

* Move required_fields check to its own remote test

* Iterate production rules in required_fields test

* Use direct get_required_fields call in remote test

Skip to_api_format() and call data.get_required_fields(index) directly,
gated on ESQLRuleData. Mirrors the ESQLValidator scope of the fix and
avoids the unrelated packaging steps that to_api_format runs per rule.

* Bump version to 1.6.30

* Centralize ES|QL dynamic field prefix tuple

Define ESQL_DYNAMIC_FIELD_PREFIXES = ("Esql.", "Esql_priv.") in
schemas/definitions.py and reuse it in QueryValidator.get_required_fields,
ESQLValidator.validate_columns_index_mapping, and the remote test.
Single source of truth and consistent ordering across the codebase.
2026-05-01 17:37:31 -05:00
Eric Forte 26d37dd62e [Bug] Ignore Other Keep Wildcards (#5792)
* Ignore other Keep Wildcards

* Added a unit test for multiple keeps

* Add keep star unit tests
2026-03-09 19:33:27 -04:00
Eric Forte 070b457659 Test remote_cli update test indices 2026-01-27 20:08:19 +05:30
Eric Forte 891aa8b6d5 [FR] Add keep metadata check to esql schema test (#5441)
* Add keep metadata check to esql schema test

* Update unit tests

* Allow for keep *

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>

---------

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
2026-01-14 16:03:24 -05:00
Eric Forte 634de61d6d [FR] ES|QL remote validation support newline split indices (#5356)
* Updated regex pattern for multiline

* Add line split unit test
2025-12-03 11:50:51 -05:00
Eric Forte 033145adf4 [Bug] Add synthetic properties check to remote ESQL validation (#5308)
* Add synthetic properties check

* Add additional unit test for schema conflicts
2025-11-13 15:25:42 -05:00
Eric Forte 7604c20d9e [FR] Add ESQL rules to dataset exception (#5249)
* Add ESQL rules to dataset exception

* Add unit test
2025-10-27 11:03:48 -04:00
Sergey Polzunov c7246313f7 feat: ESQL query validation against Elastic cluster (#4955)
* Add remote ESQL validation
---------

Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
Co-authored-by: Mika Ayenson <mika.ayenson@elastic.co>
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
2025-10-15 15:17:07 -04:00
Sergey Polzunov 1fb60d6475 fix: type hinting fixes and additional code checks (#4790)
* first pass

* Adding a dedicated code checking workflow

* Type fixes

* linting config and python version bump

* Type hints

* Drop incorrect config option

* More fixes

* Style fixes

* CI adjustments

* Pyproject fixes

* CI & pyproject fixes

* Proper version bump

* Tests formatting

* Resolve cirtular dependency

* Test fixes

* Make sure the tests are formatted correctly

* Check tweaks

* Bumping python version in CI images

* Pin marshmallow do 3.x because 4.x is not supported

* License fix

* Convert path to str

* Making myself a codeowner

* Missing kwargs param

* Adding a missing kwargs to `set_score`

* Update .github/CODEOWNERS

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* Dropping unnecessary raise

* Dropping skipped test

* Drop unnecessary var

* Drop unused commented-out func

* Disable typehinting for the whole func

* Update linting command

* Invalid type hist on the input param

* Incorrect field type

* Incorrect value used fix

* Stricter values check

* Simpler function call

* Type condition fix

* TOML formatter fix

* Simpligy output conditions

* Formatting

* Use proper types instead of aliases

* MITRE attack fixes

* Using pathlib.Path for an argument

* Use proper method to update a set from a dict

* First round of `ruff` fixes

* More fixes

* More fixes

* Hack against cyclic dependency

* Ignore `PLC0415`

* Remove unused markers

* Cleanup

* Fixing the incorrect condition

* Update .github/CODEOWNERS

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* Set explicit default values for optional fields

* Update the guidelines

* Adding None Defaults

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
2025-07-01 08:20:55 -05:00
Mika Ayenson 7514c0a206 [FR] Add Support for ES|QL Rule Type and Remote Validation (#3281)
* add suuport for esql type
* add unit tests
* set clients in RemoteConnector from auth methods
* thread remote rules; add engine test
* Add versions to remote validation results

---------

Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
2023-12-08 12:46:28 -07:00