Commit Graph

568 Commits

Author SHA1 Message Date
shashank-elastic bbdde20f7b Fix variable usage impacting schema build performance (#4910) 2025-07-15 21:20:30 +05:30
Sergey Polzunov c0631d2df2 fix: Better aligning prompt behaviour with jsonschema types (#4894)
* Check for `["array"]` in addition to `"array"`

* version bump

* Exclude non-ecs-schema.json from CI check
2025-07-11 07:10:47 -05:00
Eric Forte 898be50e95 [Bug] Fix Filter Support for Import Rules (#4852)
* Fix Filter Support for Import Rules

* Patch Bump

* Update Remove CLI Test Script

* Ruff formatting
2025-07-09 10:07:42 -04:00
github-actions[bot] 52a3652965 Lock versions for releases: 8.18,8.19,9.0,9.1 (#4887) 2025-07-08 15:05:39 +05:30
shashank-elastic 9b292b97ea Prep 8.19/9.1 (#4869)
* Prep 8.19/9.1 Release

* Download Beats Schema

* Download API Schema

* Download 8.18.3 Beats Schema

* Download Latest Integrations manifest and schema

* Comment old schemas

* Update Patch version
2025-07-07 11:27:48 -04:00
Terrance DeJesus 016cdf2cbb [New Rule] Microsoft Entra ID Suspicious Cloud Device Registration (#4802)
* new rule Microsoft Entra ID Suspicious Cloud Device Registration

* adjusted backticks in non-ecs and rule

* linted

* adjusted uuid; bumped patch version
2025-07-02 10:03:08 -04:00
Sergey Polzunov cdb346cb77 fix: Skip invalid YAML files in Beats dist (#4865)
* Skip invalid YAML files but keep them in the branch

* Typo fix

* Patch version bump

* Adding a schema generation command to `test_cli.bash` flow
2025-07-02 13:39:35 +02:00
Sergey Polzunov 1fb60d6475 fix: type hinting fixes and additional code checks (#4790)
* first pass

* Adding a dedicated code checking workflow

* Type fixes

* linting config and python version bump

* Type hints

* Drop incorrect config option

* More fixes

* Style fixes

* CI adjustments

* Pyproject fixes

* CI & pyproject fixes

* Proper version bump

* Tests formatting

* Resolve cirtular dependency

* Test fixes

* Make sure the tests are formatted correctly

* Check tweaks

* Bumping python version in CI images

* Pin marshmallow do 3.x because 4.x is not supported

* License fix

* Convert path to str

* Making myself a codeowner

* Missing kwargs param

* Adding a missing kwargs to `set_score`

* Update .github/CODEOWNERS

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* Dropping unnecessary raise

* Dropping skipped test

* Drop unnecessary var

* Drop unused commented-out func

* Disable typehinting for the whole func

* Update linting command

* Invalid type hist on the input param

* Incorrect field type

* Incorrect value used fix

* Stricter values check

* Simpler function call

* Type condition fix

* TOML formatter fix

* Simpligy output conditions

* Formatting

* Use proper types instead of aliases

* MITRE attack fixes

* Using pathlib.Path for an argument

* Use proper method to update a set from a dict

* First round of `ruff` fixes

* More fixes

* More fixes

* Hack against cyclic dependency

* Ignore `PLC0415`

* Remove unused markers

* Cleanup

* Fixing the incorrect condition

* Update .github/CODEOWNERS

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* Set explicit default values for optional fields

* Update the guidelines

* Adding None Defaults

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
2025-07-01 08:20:55 -05:00
Isai bf1dc2547f [Rule Tunings] AWS SSM Command Document Created by Rare User (#4848)
* [Rule Tunings] AWS SSM Command Document Created by Rare User

## AWS SSM Command Document Created by Rare User
Rule executes as expected and has very few alerts in telemetry. However, it is one of the rules timing out occasionally.
- reduced execution window
- reduced new terms history window
- replaced wildcards with the flattened field in the query, which should improve performance
- replaced `aws.cloudtrail.user_identity.arn` with combination of `cloud.account.id` and `user.name` to account for Assumed Roles. This will only evaluate the role instead of each individual role session, which will improve performance.
- added investigation fields
- corrected tags
- added mitre technique

## AWS SSM `SendCommand` Execution by Rare User"
- added investigation fields
- added tag

* update pyproject.toml

update pyproject.toml version
2025-06-27 13:24:27 -04:00
Terrance DeJesus 3bd9ab8d1d [New Rule] Excessive Microsoft 365 Mailbox Items Accessed (#4825)
* new rule Excessive Microsoft 365 Mailbox Items Accessed

* bumping patch version
2025-06-26 12:51:11 -04:00
Terrance DeJesus c986138333 [Rule Tuning] Suspicious Microsoft 365 Mail Access by Unusual ClientAppId (#4806)
* tuning rule Suspicious Microsoft 365 Mail Access by Unusual ClientAppId

* adjusted tactic tag

* updating patch version

* updating patch version

* bumping patch version
2025-06-24 14:20:33 -04:00
Ruben Groenewoud d59addb710 [Rule Tuning] Sharpening Kubernetes Rules Indices (#4822)
* [Rule Tuning] Sharpening Kubernetes Rules Indices

* ++
2025-06-24 14:11:31 +02:00
Samirbous 4b20d69c03 [Tuning] Elevation via SCM rules (#4837)
* Update privilege_escalation_krbrelayup_service_creation.toml

* Update privilege_escalation_windows_service_via_unusual_client.toml

* Update non-ecs-schema.json

* Update non-ecs-schema.json

* Update pyproject.toml
2025-06-20 09:52:59 +01:00
Terrance DeJesus 0aefedd6f1 [New Rule] Suspicious ADRS Token Request by Microsoft Auth Broker (#4801)
* new rule Suspicious ADRS Token Request by Microsoft Auth Broker

* bumping patch version

* updating patch version
2025-06-18 14:41:04 -04:00
Terrance DeJesus 0c68fcb7d9 [New Rule] Entra ID User Signed In from Unusual Device (#4804)
* new rule Entra ID User Signed In from Unusual Device

* adjusted patch version

* adjusted patch version

* updating patch version
2025-06-18 14:13:42 -04:00
github-actions[bot] fcad19fa18 Lock versions for releases: 8.14,8.15,8.16,8.17,8.18,9.0 (#4820) 2025-06-18 18:11:09 +05:30
Jonhnathan 1f71191c85 [New Rules] SPN Spoofing / Coercion Rules (#4815)
* [New Rules] SPN Spoofing / Coercion Rules

* Apply suggestions from code review

Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>

* Update rules/windows/credential_access_kerberos_coerce.toml

* Update rules/windows/credential_access_kerberos_coerce_dns.toml

Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>

* Update rules/windows/credential_access_kerberos_coerce.toml

Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>

* .

* Update rules/windows/credential_access_kerberos_coerce_dns.toml

* Update rules/windows/credential_access_kerberos_coerce_dns.toml

* Update pyproject.toml

* missing tag

---------

Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>
2025-06-17 18:50:28 -03:00
Isai 74d1715f6e [Rule Tuning] AWS EC2 User Data Retrieval for EC2 Instance (#4808)
* [Rule Tuning] AWS EC2 User Data Retrieval for EC2 Instance

- changed execution window
- explicitly added flattened fields to query, to reduce wildcard usage
- added investigation fields
- changed new terms field to evaluate `user.name` over `aws.cloudtrail.user_identity.arn` so that only the role name for Assumed Role identitites is being evaluated instead of each individual session. This should greatly impact performance as most instances of this rule in telemetry is triggered by Assumed Roles.

* Apply suggestions from code review

* remove instanceId parameter

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
2025-06-17 14:51:18 -04:00
Isai cd003fc9a7 [New Rule] AWS CloudTrail Log Evasion (#4788)
* [New Rule] AWS CloudTrail Log Evasion

Identifies the evasion of cloudtrail logging for IAM actions involving policy creation, modification or attachment. When making certain policy-related API calls, an adversary may pad the associated policy document with whitespaces to trigger CloudTrail’s logging size constraints, resulting in incomplete logging where critical details about the policy are omitted. By exploiting this gap, threat actors can bypass monitoring performed through CloudTrail and can effectively obscure unauthorized changes. This rule looks for IAM API calls with the requestParameters property containing reason:”requestParameters too large” and omitted:true.

This is a known gap in AWS with no immediate remediation steps. While the size constraint issue affects additional services, IAM policy-related API calls are the only that pose a security risk which is why this rule is scoped specifically to `event.provider: iam.amazonaws.com`.  For additional background on the evasion technique refer to Permisso's [research](https://permiso.io/blog/cloudtrail-logging-evasion-where-policy-size-matters).

* aligning IG and rule name

* added investigation fields

added investigation fields

* change severity

* updating pyproject version
2025-06-17 13:58:26 -04:00
Isai e84892a115 [Rule Tuning] AWS EC2 Deprecated AMI Discovery (#4784)
* [Rule Tuning] AWS EC2 Deprecated AMI Discovery

Rule triggers as expected
Telemetry shows only known FP risks from tools that are intentionally including deprecated AMIs in their searches (these should be excluded by customers)
- changed the query to reduce use of multiple wildcards
- changed the execution window
- removed unnecessary parts of IG
- added to the highlighted fields

* update non-ecs-schema.json

update non-ecs-schema.json with field "aws.cloudtrail.flattened.request_parameters.ownersSet.items.owner"

* update version in pyproject.toml

update version in pyproject.toml

* Update pyproject.toml
2025-06-17 13:19:22 -04:00
Eric Forte 9f2d4d9247 [Bug] Makefile test-remote-cli Defined Twice (#4751)
* Minor Bug Fixes for Paths
2025-06-13 11:45:54 -04:00
Eric Forte 5b3dac0a14 [FR] Add Ability to Filter Rule Exports from Kibana (#4783)
* Add ability to filter on custom rules and filter exports
2025-06-09 12:21:15 -04:00
github-actions[bot] 4cf3d28367 Lock versions for releases: 8.14,8.15,8.16,8.17,8.18,9.0 (#4758) 2025-06-02 21:53:59 +05:30
shashank-elastic 89fe4c977c Refresh Integration Manifest & Schema (#4755) 2025-06-02 20:14:43 +05:30
github-actions[bot] 72ec8199ae Lock versions for releases: 8.14,8.15,8.16,8.17,8.18,9.0 (#4732) 2025-05-20 08:26:21 +05:30
shashank-elastic 43cdc7ff51 Refresh MITRE version (#4729) 2025-05-19 22:49:33 +05:30
Emmanuel Ferdman 2ad2d68c4a Resolve datetime.utcfromtimestamp deprecation (#4719) 2025-05-19 21:35:07 +05:30
Terrance DeJesus 8f27c24528 [New Rule] Suspicious Email Access by First-Party Application via Microsoft Graph (#4704)
* new rule 'Suspicious Email Access by First-Party Application via Microsoft Graph'

* updated patch version

---------

Co-authored-by: Colson Wilhoit <48036388+DefSecSentinel@users.noreply.github.com>
2025-05-09 20:49:08 -04:00
github-actions[bot] acab8b4c6e Lock versions for releases: 8.14,8.15,8.16,8.17,8.18,9.0 (#4703) 2025-05-07 07:34:20 +05:30
Eric Forte 639d748ec2 [FR] Add check-version-lock dev command (#4650)
* Add check-version-lock dev command

* Bump the version

* Add Check Double Bumps to lock-versions workflow

* Replace return with ctx aware exit

* Bump Version

* Update Double Bump Modulo calculation

* Update if formatting

* Undo formatting typo

* Add logic to process the local file

* Update for descriptiveness

* Allow double bump branch for testing

* Pass github token

* Re-restrict to main

* Patch version bump

* Add comment if no double bumps found

* Bump Version
2025-05-06 13:26:23 -04:00
shashank-elastic e4856d3c2c Refresh ecs, beats, integration manifests & schemas (#4699) 2025-05-05 23:06:40 +05:30
shashank-elastic b3adc6d3ea Deprecate Experimental ML command (#4669) 2025-05-02 21:01:46 +05:30
Samirbous dddc2a7bb9 [New] Microsoft 365 OAuth Redirect to Device Registration for User (#4694)
* [New] Microsoft 365 OAuth Redirect to Device Registration for User Principal

https://github.com/elastic/ia-trade-team/issues/590

* Update non-ecs-schema.json

* Update pyproject.toml

* Update credential_access_antra_id_device_reg_via_oauth_redirection.toml

* Update credential_access_antra_id_device_reg_via_oauth_redirection.toml

* Update credential_access_antra_id_device_reg_via_oauth_redirection.toml

* fixed investigation guide formatting; fixed unit test failure

* updated patch version

---------

Co-authored-by: terrancedejesus <terrance.dejesus@elastic.co>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
2025-05-02 08:36:10 +01:00
Terrance DeJesus bae7835f6a [New Rule] MSFT Tenant OAuth Phishing via First-Party VSCode Client (#4642)
* new rules for MSFT Oauth phishing in Azure, Entra and Microsoft 365

* changed m365 file name

* fixed duplicate tactics

* updaing non-ecs for graph activity logs

* updating rules; investigation guides; formatting, linting errors
2025-05-01 22:38:41 -04:00
Sergey Polzunov ba959f2ceb fix: Fixing leftover references to sha256 method (#4690)
* Fixing missed old method name usage

* Patch version bump
2025-04-30 20:34:15 +02:00
github-actions[bot] fc1e6145cc Lock versions for releases: 8.14,8.15,8.16,8.17,8.18,9.0 (#4679) 2025-04-30 18:11:35 +05:30
Sergey Polzunov d72cb92d59 Bringing back "fix: Cleaning up the hashable content for the rule" (#4621) (#4668) 2025-04-28 21:59:55 +05:30
shashank-elastic 97e6d8b706 Lock versions for releases: 8.14,8.15,8.16,8.17,8.18,9.0 (#4665) 2025-04-25 20:35:09 +05:30
Sergey Polzunov b7a324b2e8 Revert "fix: Cleaning up the hashable content for the rule (#4621)" (#4654)
This reverts commit 80c4f7eacc.
2025-04-24 19:05:17 +02:00
Sergey Polzunov 80c4f7eacc fix: Cleaning up the hashable content for the rule (#4621) 2025-04-24 14:33:26 +05:30
shashank-elastic 54fadc8e2e Add 8.18 and 9.0 beats schemas (#4641) 2025-04-24 05:36:45 +05:30
Samirbous ea31143b83 [New] Suspicious Azure Sign-in via Visual Studio Code (#4639)
* Create initial_access_entra_login_visual_code_phish.toml

* Update non-ecs-schema.json

* Update initial_access_entra_susp_visual_code_signin.toml

* Update pyproject.toml

* Update initial_access_entra_susp_visual_code_signin.toml

* Update non-ecs-schema.json
2025-04-23 14:06:05 +01:00
Jonhnathan 364d9dd3bc [New Rule] Threat Intel Email Indicator Match (#4598)
* [New Rule] Threat Intel Email Indicator Match

* Update threat_intel_indicator_match_email.toml

* Update pyproject.toml

* Adds IG

* Update rules/threat_intel/threat_intel_indicator_match_email.toml

* Update rules/threat_intel/threat_intel_indicator_match_email.toml

Co-authored-by: Isai <59296946+imays11@users.noreply.github.com>

* Update rules/threat_intel/threat_intel_indicator_match_email.toml

Co-authored-by: Isai <59296946+imays11@users.noreply.github.com>

---------

Co-authored-by: Isai <59296946+imays11@users.noreply.github.com>
2025-04-22 12:15:36 -03:00
Eric Forte 62feac3348 [Bug] Update Schema Prompt to include new_terms_fields (#4567)
* Update Schema Prompt to include new_terms_fields

* Version Bump

* Ensure list of strings

* Update utils to support comma deliminated strings

* Also remove excess quotes

* Bump patch version

* Remove Union

* bump version
2025-04-17 10:45:51 -04:00
Frederik Berg 6cb238bedb [Enhancement] Add flag to export rules via KQL search on name (#4594)
* Add flag to export rules via KQL search on name

* Add KQL to help text

Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>

* version patch bump

* flake8 trimming

* pyproject bump

* Bump version

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
2025-04-16 18:40:46 -04:00
Frederik Berg 9b682b752c Feature exclude tactic name (#4593)
* Added new cli flag to exclude tactic name in rule file name

* added a shortcut for the flag and adjusted CLI readme

* Add no tactic flag also to import to prevent warnings

* Added info about unit test

* version bump

* Added no_tactic_filename as config option + fixed linting

* pyproject version bump

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>
2025-04-16 16:02:14 -04:00
Eric Forte 033c82858c [FR] Add Support for Local Dates Flag (#4582)
* Add support for local dates flag

* Use two variables

* Add support for import-rules-to-repo

* Revert arg formatting

* Update comment

* Pass Rule Path as Path Object

* Update to rule loader function

* Streamline metadata function

* Also support dictionaries

* Bump patch version

* Reduce complexity

* Add if path exists check

* Fix version bump
2025-04-16 15:41:09 -04:00
Terrance DeJesus ba16e27edb [Rule Tuning] Tuning Azure Service Principal Credentials Added (#4570)
* tuning 'Azure Service Principal Credentials Added'

* updated patch version

* added investigation guide

* updating patch version

* updating patch version
2025-04-16 13:58:17 -04:00
Terrance DeJesus 1a6669e5a6 [Rule Tuning] Adjusting Microsoft Entra ID Rare Authentication Requirement for Principal User (#4562)
* tuning 'Microsoft Entra ID Rare Authentication Requirement for Principal User'

* updated MITRE ATT&CK mappings

* updated index target

* updated patch version

* updating patch version

* bumping patch version

* updating patch version
2025-04-16 12:21:41 -04:00
Eric Forte ea7de8230c [FR] Add Kibana Action Connector Error to Exception List Workaround (#4583)
* Add error catch for workaround

* Switch to set for efficiency

* Patch version bump

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
2025-04-15 09:18:50 -04:00