* Omit Esql.* columns from ES|QL rule required_fields
Kibana treats required_fields as index mappings. ES|QL stats and
similar commands expose Esql.* and Esql_priv.* result columns that
are not mapped on source indices, which produced noisy validation
warnings for shipped rules.
Filter those names when building required_fields. Add a check in
test_esql_endpoint_alerts_index when remote ES|QL validation runs.
Fixes#6026.
* Move required_fields check to its own remote test
* Iterate production rules in required_fields test
* Use direct get_required_fields call in remote test
Skip to_api_format() and call data.get_required_fields(index) directly,
gated on ESQLRuleData. Mirrors the ESQLValidator scope of the fix and
avoids the unrelated packaging steps that to_api_format runs per rule.
* Bump version to 1.6.30
* Centralize ES|QL dynamic field prefix tuple
Define ESQL_DYNAMIC_FIELD_PREFIXES = ("Esql.", "Esql_priv.") in
schemas/definitions.py and reuse it in QueryValidator.get_required_fields,
ESQLValidator.validate_columns_index_mapping, and the remote test.
Single source of truth and consistent ordering across the codebase.
* [Bug] KQL Validation Add Wildcard w/ Space token value
## Summary
Fixes KQL parser to support wildcard values containing spaces (e.g., `*S3 Browser*`), which work in Kibana but were rejected by our unit tests.
**Issue:** #5750
## Changes
### Grammar (`lib/kql/kql/kql.g`)
- Added `WILDCARD_LITERAL` token with priority 3 to match wildcard patterns containing spaces
- Uses negative lookahead to stop before `or`/`and`/`not` keywords
- Added to `value` rule (not `literal`) so field names remain unaffected
### Parser (`lib/kql/kql/parser.py`)
- Handle new `WILDCARD_LITERAL` token type as wildcards
- Quoted strings (`"*text*"`) now treated as literals, matching Kibana behavior
## Behavior
| Query | Before | After |
|-------|--------|-------|
| `field: *S3 Browser*` | ❌ Parse error | ✅ Wildcard |
| `field: *test*` | ✅ Wildcard | ✅ Wildcard |
| `common.*: value` | ✅ Works | ✅ Works |
| `field: "*text*"` | Wildcard | ✅ Literal (matches Kibana) |
## Test plan
- [x] All 63 existing KQL unit tests pass
- [x] New wildcard-with-spaces patterns parse correctly
- [x] Wildcard field names (`common.*`) still work
- [x] Keywords (`or`, `and`, `not`) correctly recognized as separators
- [x] Tested against rule file from PR #5694
* update pyproject version
* update kibana and kql pyproject.toml versions
update kibana and kql pyproject.toml versions
* update wildcard_literal pattern to account for false matches with leading keywords
Add Negative lookahead at start of Pattern 2 - uses (?!(?:or|and|not)\b) at the start to prevent matching values that begin with keywords like 'not /path*'
* adding NOT keyword token and support for wildcard in the middle of spaced phrase
# KQL Parser Changes - Wildcard Spaces and NOT Prefix Fix
## Overview
This update fixes two issues in the KQL parser:
1. **Wildcard values with spaces** - Values like `*S3 Browser*` now parse correctly
2. **NOT prefix false match** - Values like `not /tmp/go-build*` are no longer incorrectly consumed as a single wildcard literal
## Files Modified
### `lib/kql/kql/kql.g` (Grammar)
**Added `optional_not` rule** to handle `NOT` as an explicit grammar element:
```
?list_of_values: "(" or_list_of_values ")"
| optional_not value
?optional_not: NOT optional_not
|
```
**Expanded `WILDCARD_LITERAL`** with 4 patterns to support all wildcard-with-space cases:
| Pattern | Description | Example |
|---------|-------------|---------|
| 1 | Starts with `*` | `*S3 Browser`, `*S3 Browser*` |
| 2 | Ends with `*` (doesn't start with `*`) | `S3 Browser*` |
| 3a | `*` appears after a space | `S3 B*owser` |
| 3b | `*` appears before a space | `S3* Browser` |
### `lib/kql/kql/parser.py`
Added methods to handle the new grammar rules:
- `list_of_values()` - handles `optional_not value` structure
- `optional_not()` - counts NOT occurrences and wraps values with `NotValue`
### `lib/kql/kql/kql2eql.py`
Added corresponding methods for EQL conversion:
- `list_of_values()` - handles `optional_not value` structure
- `optional_not()` - counts NOT occurrences and wraps with `eql.ast.Not`
## Test Results
All 63 kuery tests pass. Verified wildcard cases:
| Input | Result |
|-------|--------|
| `field: *S3 Browser*` | `field:*S3\ Browser*` |
| `field: S3 Browser*` | `field:S3\ Browser*` |
| `field: *S3 Browser` | `field:*S3\ Browser` |
| `field: S3 B*owser` | `field:S3\ B*owser` |
| `field: S3* Browser` | `field:S3*\ Browser` |
| `field: foo* bar* baz` | `field:foo*\ bar*\ baz` |
| `process.executable: not /tmp/go-build*` | `not process.executable:/tmp/go-build*` |
| `field < value` | `field < value` (range expression, not wildcard) |
## Technical Notes
### Pattern 3a Fix
Pattern 3a requires at least one character AFTER the `*` (uses `[...]+` instead of `[...]*`). This prevents Pattern 2 from incorrectly matching shorter strings like `S3 B*` when the full value is `S3 B*owser`.
### NOT Keyword Handling
The `optional_not` grammar approach explicitly parses `NOT` as a keyword before the value, preventing it from being consumed as part of a wildcard literal. This is safer than regex-only approaches because:
- `NOT` token only matches the exact word "not" (case-insensitive)
- Values like `notafile*` are still parsed as `UNQUOTED_LITERAL`
- Edge case: literal value "not" must be quoted: `field: "not"`
* Changes to Addresses Review Comments
### Changes to Addresses Review Comments @Mikaayenson
1. **Fixed regex patterns to prevent trailing whitespace capture** (`kql.g`)
- Added `(?=\s|$|[()":{}])` lookahead to all WILDCARD_LITERAL patterns
- This ensures patterns stop at boundaries without capturing trailing whitespace
2. **Removed `.rstrip()` workaround** (`parser.py`)
- No longer needed since regex now handles boundaries correctly
3. **Added explicit WILDCARD_LITERAL handling** (`kql2eql.py`)
- Now checks `token.type == "WILDCARD_LITERAL"` explicitly
- Mirrors the approach used in `parser.py`
4. **Added unit tests** (`tests/kuery/test_parser.py`)
- `test_wildcard_with_spaces` - all 4 WILDCARD_LITERAL patterns
- `test_wildcard_with_spaces_and_keywords` - wildcards with `and`/`or` boundaries
- `test_not_prefix_with_wildcard` - NOT keyword not consumed as wildcard
- `test_quoted_wildcard_as_literal` - quoted wildcards are literal strings
- `test_triple_not_optimization` - `not not not foo` → `not foo`
* changed test directory from tmp
* changed format of new tests
* Update pyproject.toml
Update pyproject.toml
---------
Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>
* Add schema validation for AlertSuppressionMapping
* Add support for indicator match alert suppression
* Add unit tests
* Update order and remove validates_schema method
* Add comments
* Add test for query rule duration only
* Add whitespace checking for KQL parse
* Add unit test for blank space check
* Bump patch version
* Add test cases for newline blank space
* Add additional unit tests
* Update to only walk tree once
---------
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
* Bump Version
* updated
* Bump patch version
* Optimization should only occur on single values
* Wildcard semantically equivalent to query_string*
* Add unit test for optimization
* Move code-checks to yml
* Add tests path to code-checks
* Add lib path for code-checks
* Install deps from local
* Update DSL optimization unit test
---------
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
* first pass
* Adding a dedicated code checking workflow
* Type fixes
* linting config and python version bump
* Type hints
* Drop incorrect config option
* More fixes
* Style fixes
* CI adjustments
* Pyproject fixes
* CI & pyproject fixes
* Proper version bump
* Tests formatting
* Resolve cirtular dependency
* Test fixes
* Make sure the tests are formatted correctly
* Check tweaks
* Bumping python version in CI images
* Pin marshmallow do 3.x because 4.x is not supported
* License fix
* Convert path to str
* Making myself a codeowner
* Missing kwargs param
* Adding a missing kwargs to `set_score`
* Update .github/CODEOWNERS
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
* Dropping unnecessary raise
* Dropping skipped test
* Drop unnecessary var
* Drop unused commented-out func
* Disable typehinting for the whole func
* Update linting command
* Invalid type hist on the input param
* Incorrect field type
* Incorrect value used fix
* Stricter values check
* Simpler function call
* Type condition fix
* TOML formatter fix
* Simpligy output conditions
* Formatting
* Use proper types instead of aliases
* MITRE attack fixes
* Using pathlib.Path for an argument
* Use proper method to update a set from a dict
* First round of `ruff` fixes
* More fixes
* More fixes
* Hack against cyclic dependency
* Ignore `PLC0415`
* Remove unused markers
* Cleanup
* Fixing the incorrect condition
* Update .github/CODEOWNERS
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
* Set explicit default values for optional fields
* Update the guidelines
* Adding None Defaults
---------
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
* Delete RTAs
* Delete RTA-related orchestration code
* Drop RTAs from tests
* Remove RTAs from README
* Further cleanup
* Readme update
* Version bump and no more RTAs
* Styling fixes
* Drop RTAs from config files
* Drop `rule-mapping.yaml`
* Bring back event collector / normalizer
* Drop rta mention
* Cleanup rta leftovers
* Style fix
---------
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
* [Rule Tuning] Tighten Up Windows EventLog Indexes, Improve tags
* Format & order
* Update pyproject.toml
* Update credential_access_cookies_chromium_browsers_debugging.toml
Mika Ayenson, PhD