Commit Graph

184 Commits

Author SHA1 Message Date
Eric Forte 7410ec7db9 [Rule Tuning] Updated ESQL Rules Based on Validation Results (#5151)
* Updated ESQL rules based on validation results

* Patch bump

* Updated regex patterns

* added missing azure fields to non-ecs-schema.json; adjusted okta query logic to use LIKE instead of RLIKE

* fixed incorrect field in non-ecs-schema.json; changed logs-azure.signinlogs* sightings to logs-azure.signinlogs-*

* Add and

* Additional non-ecs fields

* Add EOF

* Add kibana.alert.rule.name

* removed azure.platforlogs.identity.claim.objectid; updated query for 'c07f7898-5dc3-11f0-9f27-f661ea17fbcd'

* Field removed from query removing from keep

* Patch Bump

---------

Co-authored-by: terrancedejesus <terrance.dejesus@elastic.co>
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
2025-09-30 00:36:29 -04:00
Eric Forte 42be8bc8ba [Bug] Add Required to the Annotation (#5159)
* Add Required to the Annotation

* Additional required fields

* remove nonempty sting validation

* Required Types via Annotated and Dataclass

* remove space

* Remove inline comment

* Switch to getting a list

* Fix typo and sort

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
2025-09-29 18:30:50 -04:00
shashank-elastic e147188939 Add SIEM package category (#5128) 2025-09-18 19:15:53 +05:30
Eric Forte 80c01cf665 [Bug] Annotated Fields Ignored (#5125)
* Add Note for stop gap
2025-09-17 17:34:42 -04:00
github-actions[bot] 8f79d58f3f Lock versions for releases: 8.18,8.19,9.0,9.1 (#5123) 2025-09-16 19:56:59 +05:30
Eric Forte 99ebad576b Added handling for unauth error (#5115) 2025-09-16 18:25:10 +05:30
Eric Forte b2b9d677c7 [Bug] Github Gist API Now Requires Auth (#5119)
* Add headers to public call
2025-09-16 08:18:48 -04:00
elastic-renovate-prod[bot] 39b6f19eb9 Pin dependencies (#5086)
Co-authored-by: Shashank K S <Shashank.Suryanarayana@elastic.co>
2025-09-12 22:46:24 +05:30
Mika Ayenson, PhD f0f7d217c0 [FR] Refactor Schema Validation & Support Multi-Dataset Sequence Validation (#5059) 2025-09-10 13:11:04 -05:00
shashank-elastic 6adee51410 Fix Ruff failures (#5083) 2025-09-10 22:24:07 +05:30
shashank-elastic a6dfd2c0e1 Add test_min_stack_version_supported testcase (#5077) 2025-09-10 20:12:36 +05:30
Mika Ayenson, PhD 35b000b7ab [FR] Add negate DOES NOT MATCH capability to IM rule type (>=9.2) (#5041) 2025-09-09 10:58:53 -05:00
Eric Forte cbb892b4bc [Bug] Incorrect Integrations Schema Parsing for Nested Fields (#5058)
* Add proper handling for nested fields

* Updated schemas

* bump patch

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
2025-09-04 14:12:33 -04:00
Mika Ayenson, PhD 3c1de72f6b [FR] Add support for 5 group_by fields in threshold rules (>=9.2) (#5040) 2025-09-04 09:24:36 -05:00
github-actions[bot] f2291e0261 Lock versions for releases: 8.18,8.19,9.0,9.1 (#5049) 2025-09-01 23:19:12 +05:30
shashank-elastic 93ac471574 Monthly Schema Updates (#5046) 2025-09-01 20:42:42 +05:30
shashank-elastic ee70674e2c Add all rule types DaC testing (#4969) 2025-08-20 19:04:57 +05:30
Eric Forte dde448ee6b [Bug] Rule Toml Write Formatting Wrongly Formats \\\\x (#4978)
* Fix rule and mitigate py toml

* Bump patch version

* Add reference to issue

* Add unit test for path issues

* Update comment

* Certain strings were not properly escaped

* Updated to use json instead of repr

* replace _old_dump_str with json.dumps

* Bump Version
2025-08-18 17:03:51 -04:00
github-actions[bot] fb76ec1b2d Lock versions for releases: 8.18,8.19,9.0,9.1 (#4991) 2025-08-18 22:36:37 +05:30
github-actions[bot] 154283f457 Lock versions for releases: 8.18,8.19,9.0,9.1 (#4963) 2025-08-06 08:58:16 +05:30
Eric Forte a726da5e83 [Bug] [DAC] Custom Rules Filter Discrepancy on Stacks Upgraded to 8.18 (#4945)
* Update Custom Rules KQL

* Bump Patch Version

* Update detection_rules/kbwrap.py

Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com>

* Use or instead of and

* Bump patch version

* Fix results len typo

---------

Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com>
2025-08-05 09:42:25 -04:00
github-actions[bot] c210a88b1f Lock versions for releases: 8.18,8.19,9.0,9.1 (#4960) 2025-08-04 22:37:59 +05:30
shashank-elastic 2c2b15368c Update latest integration manifests and schema and investigation guides (#4957) 2025-08-04 19:30:01 +05:30
Sergey Polzunov ff46a7ab4a fix: Allow different order of the metadata fields in ESQL queries (#4956)
* Initial commit

* Python project version bump
2025-08-02 02:26:39 +02:00
Eric Forte a9ad66935c [FR] [DAC] Add Arbitrary File location Support for Local Creation Date (#4915)
* Add support for local file contents

* Update Rule Params

* Update CLI docs

* Update to Pathlib

* Format updating

* Delete duplicate

* Update logic to handle just local_contents path

* Update to Glob Based Approach

* Updated to use RawRuleCollection

* Fix Logging Typo

* New utils functions no longer needed

* Update naming for convention
2025-07-31 14:35:00 -04:00
Eric Forte bf3071d3d1 [FR] Add white space checking for KQL parse (#3789)
* Add whitespace checking for KQL parse

* Add unit test for blank space check

* Bump patch version

* Add test cases for newline blank space

* Add additional unit tests

* Update to only walk tree once

---------

Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
2025-07-31 14:23:53 -04:00
Mika Ayenson, PhD 1dc3926203 [New Rules] External Promotion Alerts (#4903) 2025-07-31 11:00:50 -05:00
Mika Ayenson, PhD f2fac1bc48 [FR] [DAC] Add existing mitre threat information on import (#4948) 2025-07-31 09:44:09 -05:00
github-actions[bot] f348e92f06 Lock versions for releases: 8.18,8.19,9.0,9.1 (#4926) 2025-07-22 21:19:44 +05:30
Eric Forte 0cb1e596b3 [Bug] [DAC] Kibana Export Rules Rule Name Filter Exports All Rules (#4917)
* Add check for not rule_id
2025-07-22 11:32:17 -04:00
github-actions[bot] 3bec392e66 Lock versions for releases: 8.18,8.19,9.0,9.1 (#4924) 2025-07-22 18:10:32 +05:30
github-actions[bot] b3c681e475 Lock versions for releases: 8.18,8.19,9.0,9.1 (#4922) 2025-07-22 12:50:27 +05:30
shashank-elastic bbdde20f7b Fix variable usage impacting schema build performance (#4910) 2025-07-15 21:20:30 +05:30
Sergey Polzunov c0631d2df2 fix: Better aligning prompt behaviour with jsonschema types (#4894)
* Check for `["array"]` in addition to `"array"`

* version bump

* Exclude non-ecs-schema.json from CI check
2025-07-11 07:10:47 -05:00
Marc-Antoine Leclercq 1b12ecff87 Clarify authentication settings to Kibana related to #4495 (#4819)
* Update CLI.md

Removing mentions of kibana_user and kibana_password since #4495 removed them entirely.

* Bump patch version

* Bump patch version

---------

Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
2025-07-10 15:21:01 -04:00
Eric Forte 03f977246f [FR] Updates to KQL Lib Parsing and Install (#3605)
* Bump Version

* updated

* Bump patch version

* Optimization should only occur on single values

* Wildcard semantically equivalent to query_string*

* Add unit test for optimization

* Move code-checks to yml

* Add tests path to code-checks

* Add lib path for code-checks

* Install deps from local

* Update DSL optimization unit test

---------

Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
2025-07-10 15:03:08 -04:00
dependabot[bot] 932163e9cd Bump setuptools from 75.2.0 to 78.1.1 and lock marshmallow-dataclass[union] to 8.6.1 (#4730)
* Bump setuptools from 75.2.0 to 78.1.1

Bumps [setuptools](https://github.com/pypa/setuptools) from 75.2.0 to 78.1.1.
- [Release notes](https://github.com/pypa/setuptools/releases)
- [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst)
- [Commits](https://github.com/pypa/setuptools/compare/v75.2.0...v78.1.1)

---
updated-dependencies:
- dependency-name: setuptools
  dependency-version: 78.1.1
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump Package Version

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>
Co-authored-by: shashank-elastic <91139415+shashank-elastic@users.noreply.github.com>
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
2025-07-09 18:08:31 -04:00
Eric Forte 898be50e95 [Bug] Fix Filter Support for Import Rules (#4852)
* Fix Filter Support for Import Rules

* Patch Bump

* Update Remove CLI Test Script

* Ruff formatting
2025-07-09 10:07:42 -04:00
github-actions[bot] 52a3652965 Lock versions for releases: 8.18,8.19,9.0,9.1 (#4887) 2025-07-08 15:05:39 +05:30
shashank-elastic 9b292b97ea Prep 8.19/9.1 (#4869)
* Prep 8.19/9.1 Release

* Download Beats Schema

* Download API Schema

* Download 8.18.3 Beats Schema

* Download Latest Integrations manifest and schema

* Comment old schemas

* Update Patch version
2025-07-07 11:27:48 -04:00
Terrance DeJesus 016cdf2cbb [New Rule] Microsoft Entra ID Suspicious Cloud Device Registration (#4802)
* new rule Microsoft Entra ID Suspicious Cloud Device Registration

* adjusted backticks in non-ecs and rule

* linted

* adjusted uuid; bumped patch version
2025-07-02 10:03:08 -04:00
Sergey Polzunov cdb346cb77 fix: Skip invalid YAML files in Beats dist (#4865)
* Skip invalid YAML files but keep them in the branch

* Typo fix

* Patch version bump

* Adding a schema generation command to `test_cli.bash` flow
2025-07-02 13:39:35 +02:00
Sergey Polzunov 1fb60d6475 fix: type hinting fixes and additional code checks (#4790)
* first pass

* Adding a dedicated code checking workflow

* Type fixes

* linting config and python version bump

* Type hints

* Drop incorrect config option

* More fixes

* Style fixes

* CI adjustments

* Pyproject fixes

* CI & pyproject fixes

* Proper version bump

* Tests formatting

* Resolve cirtular dependency

* Test fixes

* Make sure the tests are formatted correctly

* Check tweaks

* Bumping python version in CI images

* Pin marshmallow do 3.x because 4.x is not supported

* License fix

* Convert path to str

* Making myself a codeowner

* Missing kwargs param

* Adding a missing kwargs to `set_score`

* Update .github/CODEOWNERS

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* Dropping unnecessary raise

* Dropping skipped test

* Drop unnecessary var

* Drop unused commented-out func

* Disable typehinting for the whole func

* Update linting command

* Invalid type hist on the input param

* Incorrect field type

* Incorrect value used fix

* Stricter values check

* Simpler function call

* Type condition fix

* TOML formatter fix

* Simpligy output conditions

* Formatting

* Use proper types instead of aliases

* MITRE attack fixes

* Using pathlib.Path for an argument

* Use proper method to update a set from a dict

* First round of `ruff` fixes

* More fixes

* More fixes

* Hack against cyclic dependency

* Ignore `PLC0415`

* Remove unused markers

* Cleanup

* Fixing the incorrect condition

* Update .github/CODEOWNERS

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* Set explicit default values for optional fields

* Update the guidelines

* Adding None Defaults

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
2025-07-01 08:20:55 -05:00
Isai bf1dc2547f [Rule Tunings] AWS SSM Command Document Created by Rare User (#4848)
* [Rule Tunings] AWS SSM Command Document Created by Rare User

## AWS SSM Command Document Created by Rare User
Rule executes as expected and has very few alerts in telemetry. However, it is one of the rules timing out occasionally.
- reduced execution window
- reduced new terms history window
- replaced wildcards with the flattened field in the query, which should improve performance
- replaced `aws.cloudtrail.user_identity.arn` with combination of `cloud.account.id` and `user.name` to account for Assumed Roles. This will only evaluate the role instead of each individual role session, which will improve performance.
- added investigation fields
- corrected tags
- added mitre technique

## AWS SSM `SendCommand` Execution by Rare User"
- added investigation fields
- added tag

* update pyproject.toml

update pyproject.toml version
2025-06-27 13:24:27 -04:00
Terrance DeJesus 3bd9ab8d1d [New Rule] Excessive Microsoft 365 Mailbox Items Accessed (#4825)
* new rule Excessive Microsoft 365 Mailbox Items Accessed

* bumping patch version
2025-06-26 12:51:11 -04:00
Terrance DeJesus c986138333 [Rule Tuning] Suspicious Microsoft 365 Mail Access by Unusual ClientAppId (#4806)
* tuning rule Suspicious Microsoft 365 Mail Access by Unusual ClientAppId

* adjusted tactic tag

* updating patch version

* updating patch version

* bumping patch version
2025-06-24 14:20:33 -04:00
Samirbous 4b20d69c03 [Tuning] Elevation via SCM rules (#4837)
* Update privilege_escalation_krbrelayup_service_creation.toml

* Update privilege_escalation_windows_service_via_unusual_client.toml

* Update non-ecs-schema.json

* Update non-ecs-schema.json

* Update pyproject.toml
2025-06-20 09:52:59 +01:00
Terrance DeJesus 0aefedd6f1 [New Rule] Suspicious ADRS Token Request by Microsoft Auth Broker (#4801)
* new rule Suspicious ADRS Token Request by Microsoft Auth Broker

* bumping patch version

* updating patch version
2025-06-18 14:41:04 -04:00
Terrance DeJesus 0c68fcb7d9 [New Rule] Entra ID User Signed In from Unusual Device (#4804)
* new rule Entra ID User Signed In from Unusual Device

* adjusted patch version

* adjusted patch version

* updating patch version
2025-06-18 14:13:42 -04:00
github-actions[bot] fcad19fa18 Lock versions for releases: 8.14,8.15,8.16,8.17,8.18,9.0 (#4820) 2025-06-18 18:11:09 +05:30