Commit Graph

189 Commits

Author SHA1 Message Date
Sergey Polzunov 59e394f36b [doc fix] Adjust wording in the docs for Kibana import/export commands (#5600)
* Wording fix

* Version bump

* Style fixes

* Style fix for tests
2026-02-04 11:17:58 +01:00
Eric Forte 070b457659 Test remote_cli update test indices 2026-01-27 20:08:19 +05:30
Eric Forte 891aa8b6d5 [FR] Add keep metadata check to esql schema test (#5441)
* Add keep metadata check to esql schema test

* Update unit tests

* Allow for keep *

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>

---------

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
2026-01-14 16:03:24 -05:00
Mika Ayenson, PhD f40a383b7e [New Rules] Add MITRE ATLAS framework support and GenAI threat detection rules (#5352)
Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>
Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com>
2025-12-05 12:26:56 -06:00
Eric Forte a8dbf2cf16 [FR] Expand CUSTOM_RULES_DIR to support user relative paths (#5390)
* Add user relative path support
2025-12-03 12:19:29 -05:00
Eric Forte 634de61d6d [FR] ES|QL remote validation support newline split indices (#5356)
* Updated regex pattern for multiline

* Add line split unit test
2025-12-03 11:50:51 -05:00
shashank-elastic 5db396f084 Skip unit test for protected prebuilt-rules on DAC env (#5323) 2025-11-17 21:41:46 +05:30
Jonhnathan 8b74ba7136 [Rule Tuning] Remove host.os.type Unit Test Exception (#5317) 2025-11-14 08:46:24 -08:00
Eric Forte 033145adf4 [Bug] Add synthetic properties check to remote ESQL validation (#5308)
* Add synthetic properties check

* Add additional unit test for schema conflicts
2025-11-13 15:25:42 -05:00
Eric Forte 7604c20d9e [FR] Add ESQL rules to dataset exception (#5249)
* Add ESQL rules to dataset exception

* Add unit test
2025-10-27 11:03:48 -04:00
shashank-elastic 9345e0ec27 Add unit test for protected prebuilt-rules (#5242) 2025-10-24 19:15:52 +05:30
Sergey Polzunov c7246313f7 feat: ESQL query validation against Elastic cluster (#4955)
* Add remote ESQL validation
---------

Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
Co-authored-by: Mika Ayenson <mika.ayenson@elastic.co>
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
2025-10-15 15:17:07 -04:00
Eric Forte a5c100a65b [Bug] Add unit tests and fix Alert Suppression schema validation for ThresholdQueryRuleData (#5196)
* Add schema validation for AlertSuppressionMapping

* Add support for indicator match alert suppression

* Add unit tests

* Update order and remove validates_schema method

* Add comments

* Add test for query rule duration only
2025-10-09 16:21:21 -04:00
Mika Ayenson, PhD f0f7d217c0 [FR] Refactor Schema Validation & Support Multi-Dataset Sequence Validation (#5059) 2025-09-10 13:11:04 -05:00
shashank-elastic 6adee51410 Fix Ruff failures (#5083) 2025-09-10 22:24:07 +05:30
shashank-elastic a6dfd2c0e1 Add test_min_stack_version_supported testcase (#5077) 2025-09-10 20:12:36 +05:30
Sergey Polzunov ff46a7ab4a fix: Allow different order of the metadata fields in ESQL queries (#4956)
* Initial commit

* Python project version bump
2025-08-02 02:26:39 +02:00
Eric Forte bf3071d3d1 [FR] Add white space checking for KQL parse (#3789)
* Add whitespace checking for KQL parse

* Add unit test for blank space check

* Bump patch version

* Add test cases for newline blank space

* Add additional unit tests

* Update to only walk tree once

---------

Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
2025-07-31 14:23:53 -04:00
Mika Ayenson, PhD 1dc3926203 [New Rules] External Promotion Alerts (#4903) 2025-07-31 11:00:50 -05:00
Eric Forte 03f977246f [FR] Updates to KQL Lib Parsing and Install (#3605)
* Bump Version

* updated

* Bump patch version

* Optimization should only occur on single values

* Wildcard semantically equivalent to query_string*

* Add unit test for optimization

* Move code-checks to yml

* Add tests path to code-checks

* Add lib path for code-checks

* Install deps from local

* Update DSL optimization unit test

---------

Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
2025-07-10 15:03:08 -04:00
Sergey Polzunov 1fb60d6475 fix: type hinting fixes and additional code checks (#4790)
* first pass

* Adding a dedicated code checking workflow

* Type fixes

* linting config and python version bump

* Type hints

* Drop incorrect config option

* More fixes

* Style fixes

* CI adjustments

* Pyproject fixes

* CI & pyproject fixes

* Proper version bump

* Tests formatting

* Resolve cirtular dependency

* Test fixes

* Make sure the tests are formatted correctly

* Check tweaks

* Bumping python version in CI images

* Pin marshmallow do 3.x because 4.x is not supported

* License fix

* Convert path to str

* Making myself a codeowner

* Missing kwargs param

* Adding a missing kwargs to `set_score`

* Update .github/CODEOWNERS

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* Dropping unnecessary raise

* Dropping skipped test

* Drop unnecessary var

* Drop unused commented-out func

* Disable typehinting for the whole func

* Update linting command

* Invalid type hist on the input param

* Incorrect field type

* Incorrect value used fix

* Stricter values check

* Simpler function call

* Type condition fix

* TOML formatter fix

* Simpligy output conditions

* Formatting

* Use proper types instead of aliases

* MITRE attack fixes

* Using pathlib.Path for an argument

* Use proper method to update a set from a dict

* First round of `ruff` fixes

* More fixes

* More fixes

* Hack against cyclic dependency

* Ignore `PLC0415`

* Remove unused markers

* Cleanup

* Fixing the incorrect condition

* Update .github/CODEOWNERS

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* Set explicit default values for optional fields

* Update the guidelines

* Adding None Defaults

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
2025-07-01 08:20:55 -05:00
Sergey Polzunov d72cb92d59 Bringing back "fix: Cleaning up the hashable content for the rule" (#4621) (#4668) 2025-04-28 21:59:55 +05:30
Sergey Polzunov b7a324b2e8 Revert "fix: Cleaning up the hashable content for the rule (#4621)" (#4654)
This reverts commit 80c4f7eacc.
2025-04-24 19:05:17 +02:00
Sergey Polzunov 80c4f7eacc fix: Cleaning up the hashable content for the rule (#4621) 2025-04-24 14:33:26 +05:30
shashank-elastic 2b3095a13c Update Max signals value to supported limits (#4556) 2025-03-27 09:02:25 +05:30
Sergey Polzunov 5f54eb8006 chore: Removing RTAs (#4437)
* Delete RTAs

* Delete RTA-related orchestration code

* Drop RTAs from tests

* Remove RTAs from README

* Further cleanup

* Readme update

* Version bump and no more RTAs

* Styling fixes

* Drop RTAs from config files

* Drop `rule-mapping.yaml`

* Bring back event collector / normalizer

* Drop rta mention

* Cleanup rta leftovers

* Style fix

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
2025-03-05 12:35:57 +01:00
Jonhnathan c0f12ddecf [Rule Tuning] Tighten Up Windows EventLog Indexes, Improve tags (#4464)
* [Rule Tuning] Tighten Up Windows EventLog Indexes, Improve tags

* Format & order

* Update pyproject.toml

* Update credential_access_cookies_chromium_browsers_debugging.toml
2025-02-19 12:54:31 -03:00
shashank-elastic aded9deb79 Modify Unit Test to Support Alert Suppression for EQL Sequences (#4457) 2025-02-14 00:14:28 +05:30
shashank-elastic 818467f132 Replace master doc URLs with current (#4439) 2025-02-03 21:27:50 +05:30
Mika Ayenson fe8c81d762 [FR] Generate investigation guides (#4358) 2025-01-22 11:17:38 -06:00
shashank-elastic 2ff2965cb9 Enhance Readability of validation check failures (#4299) 2024-12-13 19:03:47 +05:30
Jonhnathan 81292aee8a [Rule Tuning] 3rd Party EDR - Add Crowdstrike FDR support - 1 (#4220)
* [Rule Tuning] 3rd Party EDR - Add Crowdstrike FDR support - 1

* Update Integrations unit tests

* Update test_all_rules.py
2024-11-04 11:32:22 -03:00
shashank-elastic 275c7288a3 Add testcase to check for related_integrations based on index (#4096) 2024-10-22 00:17:30 +05:30
Terrance DeJesus 50e23ba242 [Hunting] Re-factor Hunting Library Code (#4085)
* updating python code for hunting library

* fixed okta queries; added MITRE search capability

* fixed hunting unit test imports

* fixed duplicate UUID; fixed duplicate index entry bug

* fixed technique finding sub-technique in search

* added more unit tests

* linted

* flake errors addressed; fixed unit test import; fixed markdown generate bug

* added description for generate-markdown command

* updated README

* adjusted YAML index, adjusted code for index changes

* adjusted relative imports; updated CODEOWNERS

* adding updates; moving to different branch for main dependencies

* finished run-query command; made some code adjustments

* removed some comments

* revised makefile; fixed unit tests; adjusted detection rules pyproject

* updated README

* updated README

* adjusted unit tests; adjusted hunt guidelines; updated makefile; adjusted several commands

* adjusted package to be more object-oriented

* removed unused variable

* Add simple breakdown stats

* addressed feedback; added keyword option for search

* Update hunting/README.md

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* Update detection_rules/etc/test_hunting_cli.bash

Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>

* addressing feedback

* addressed feedback

* added message for unknown index; fixed function call

* fixed search command

* fixed flake error

---------

Co-authored-by: Mika Ayenson <Mika.ayenson@elastic.co>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>
2024-10-03 12:47:40 -04:00
shashank-elastic a53f7d55a3 Testcase to check if Rule Type: BBR tag is present for all BBR rules (#4048) 2024-09-02 21:29:31 +05:30
Terrance DeJesus 2559b7bb41 [Rule Tuning] Tuning AWS Rules for SAML Provider Updates and Assumed Roles via STS (#3898)
* tuning AWS rules for SAML provider updates and assumed roles via STS

* fixed mitre mapping

* adjusted new terms and added user ID to query

* reverting new terms value change

* adding non-ecs to new term checks

* fixing mitre mapping

* Update rules/integrations/aws/privilege_escalation_sts_temp_creds_via_assume_role.toml

* reverting file removal to add diff changes

* changeing rule contents

* reverting rule changes

* added rule contents

* changed file name

* linted

* reverting lint
2024-08-20 11:53:46 -04:00
Mika Ayenson 10ba6ad5a6 [FR] Add Alert Suppression for Addtional Rule Types (#3986) 2024-08-15 15:03:45 -05:00
shashank-elastic e607d521b8 Add Unit Test test_index_or_data_view_id_present (#3967) 2024-08-12 17:48:05 +05:30
Eric Forte 47d7a3acaa [DaC] Beta Release (#3889)
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
Co-authored-by: Mika Ayenson <mika.ayenson@elastic.co>
2024-08-06 18:07:12 -04:00
shashank-elastic 59a10be7c8 Unit Test to validate from field in toml file (#3866) 2024-07-10 22:41:53 +05:30
shashank-elastic 50f0fb3518 Test case to check updated_date (#3818) 2024-07-03 19:17:27 +05:30
Jonhnathan 54d5b442cf [Rule Tuning] Add Initial Microsoft Defender for Endpoint Compatibility to Windows DRs (#3825)
* [Rule Tuning] Add Initial Microsoft Defender for Endpoint Compatibility to Windows DRs

* .

* Update integration-schemas.json.gz

* Fix integration manifests
2024-06-26 11:06:27 -03:00
Terrance DeJesus 632e169f7a [Hunt Tuning] Add Descriptions, Collapse Queries and Re-Generate Docs (#3791)
* add description to hunting schema; change queries to be a list

* update createremotethreat by process hunt

* update dll hijack and masquerading as MSFT library

* remove sysmon specific dDLL hijack via masquerading MSFT library

* updated Masquerading Attempts as Native Windows Binaries

* updates Rare DLL Side-Loading by Occurrence

* updates Rare LSASS Process Access Attempts

* update DNS Queries via LOLBins with Low Occurence Frequency

* updated Low Occurrence of Drivers Loaded on Unique Hosts

* updates Excessive RDP Network Activity by Host and User

* updates Excessive SMB Network Activity by Process ID

* updated Executable File Creation by an Unusual Microsoft Binary

* Frequency of Process Execution and Network Logon by Source Address

* updates Frequency of Process Execution and Network Logon by Source Address

* updated Execution via Remote Services by Client Address

* updated Startup Execution with Low Occurrence Frequency by Unique Host

* updated Low Frequency of Process Execution via WMI by Unique Agent

* updated Low Frequency of Process Execution via Windows Scheduled Task by Unique Agent

* updated Low Occurence of Process Execution via Windows Services with Unique Agent

* Updated High Count of Network Connection Over Extended Period by Process

* update Libraries Loaded by svchost with Low Occurrence Frequency

* updated Microsoft Office Child Processes with Low Occurrence Frequency by Unique Agent

* updated Network Discovery via Sensitive Ports by Unusual Process

* updated PE File Transfer via SMB_Admin Shares by Agent or User

* updated Persistence via Run Key with Low Occurrence Frequency

* updates Persistence via Startup with Low Occurrence Frequency by Unique Host

* updates "Persistence via Run Key with Low Occurrence Frequency"; adjusted file names to remove data source

* updates "Low Occurrence of Suspicious Launch Agent or Launch Daemon"

* updates "Egress Network Connections with Total Bytes Greater than Threshold"

* updates "Rundll32 Execution Aggregated by Command Line"

* updates "Scheduled tasks Creation by Action via Registry"

* updates "Scheduled Tasks Creation for Unique Hosts by Task Command"

* updates "Suspicious Base64 Encoded Powershell Command"

* updates "Suspicious DNS TXT Record Lookups by Process"

* updates "Unique Windows Services Creation by Service File Name"

* Updates "Unique Windows Services Creation by Service File Name"

* updates "Windows Command and Scripting Interpreter from Unusual Parent Process"

* updates "Windows Logon Activity by Source IP"

* updates "Suspicious Network Connections by Unsigned Mach-O"

* updates LLM hunting queries

* re-generated markdown files; updated generate markdown py file

* updated test_hunt_data

* Update hunting/macos/queries/suspicious_network_connections_by_unsigned_macho.toml

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* Update hunting/windows/queries/drivers_load_with_low_occurrence_frequency.toml

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* Update hunting/windows/queries/domain_names_queried_via_lolbins_and_with_low_occurence_frequency.toml

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* Update hunting/windows/queries/excessive_rdp_network_activity_by_source_host_and_user.toml

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* Update hunting/windows/queries/excessive_rdp_network_activity_by_source_host_and_user.toml

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* updated missing integrations

* updated MD docs according to recent hunting changes

* Update hunting/windows/queries/executable_file_creation_by_an_unusual_microsoft_binary.toml

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>

* Update hunting/windows/queries/detect_rare_dll_sideload_by_occurrence.toml

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>

* Update hunting/windows/queries/detect_masquerading_attempts_as_native_windows_binaries.toml

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>

* Update hunting/windows/queries/detect_dll_hijack_via_masquerading_as_microsoft_native_libraries.toml

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>

* Update hunting/llm/queries/aws_bedrock_dos_resource_exhaustion_detection.toml

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>

* added enrichment policy link to rule

* Update hunting/windows/docs/execution_via_windows_management_instrumentation_by_occurrence_frequency_by_unique_agent.md

Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>

* Update hunting/windows/docs/windows_command_and_scripting_interpreter_from_unusual_parent.md

Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>

* Update hunting/windows/docs/windows_command_and_scripting_interpreter_from_unusual_parent.md

Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>

* Update hunting/windows/docs/rundll32_execution_aggregated_by_cmdline.md

Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>

* Update hunting/windows/docs/microsoft_office_child_processes_with_low_occurrence_frequency.md

Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>

* Update hunting/windows/docs/microsoft_office_child_processes_with_low_occurrence_frequency.md

Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>

* Update hunting/windows/queries/execution_via_windows_management_instrumentation_by_occurrence_frequency_by_unique_agent.toml

Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>

* Update hunting/windows/queries/execution_via_windows_management_instrumentation_by_occurrence_frequency_by_unique_agent.toml

Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>

* Update hunting/index.md

Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>

* Update hunting/windows/docs/execution_via_network_logon_by_occurrence_frequency_by_top_source_ip.md

Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>

* Update hunting/windows/queries/execution_via_network_logon_by_occurrence_frequency_by_top_source_ip.toml

Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>

---------

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>
Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>
2024-06-25 09:35:36 -04:00
Mika Ayenson a131e021ed Revert "Test case to check updated_date (#3764)"
This reverts commit 7621a54fcc.
2024-06-21 11:05:57 -05:00
shashank-elastic 7621a54fcc Test case to check updated_date (#3764) 2024-06-21 18:43:32 +05:30
shashank-elastic 675cad2ed4 Incorrect Integration Index Check (#3794) 2024-06-21 18:29:39 +05:30
Terrance DeJesus 020ca4be24 [New Rule] Rapid7 Threat Command CVEs Correlation (#3718)
* new rule 'Rapid7 Threat Command CVEs Correlation'

* Update rules/threat_intel/threat_intel_rapid7_threat_command.toml

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* updated threat index and tags

* changed 'indicator match' to 'threat match' for tags

* removed timeline

* updating integrations to match main

* re-adding rapid7 threat command integration manifest and schema

* reverting changes; removing timeline

* changed max signals to 10000

---------

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
2024-06-12 18:01:44 -04:00
Justin Ibarra 48e85439e0 [New hunts] 50 ES|QL Windows Hunt Queries (#3642)
* [New Hunt] Initial add of Windows hunt queries

* Add markdown files

* Added license to schema and md generation

* add hunt index; minor tweaks to script

* minor tweaks from feedback

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>
Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>

* Update hunting/macos/queries/suspicious_network_connections_by_unsigned_macho.toml

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>

* convert integrations to list

* Update script to generate integration links

* validate generated integrations links

* Update hunting/windows/docs/execution_via_remote_services_by_client_address.md

* Update hunting/windows/queries/execution_via_network_logon_by_occurrence_frequency_by_top_source_ip.toml

* Update hunting/windows/queries/execution_via_remote_services_by_client_address.toml

* Update hunting/windows/docs/execution_via_network_logon_by_occurrence_frequency_by_top_source_ip.md

* Update hunting/windows/queries/execution_via_network_logon_by_occurrence_frequency.toml

* Update hunting/windows/docs/execution_via_network_logon_by_occurrence_frequency.md

* update docs with naming information

* Create suspicious_base64_encoded_powershell_commands.toml

* Create scheduled_task_creation_by_action_via_registry.toml

* Create suspicious_base64_encoded_powershell_commands.md

* Create scheduled_task_creation_by_action_via_registry.md

* Update index.md

---------

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>
Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>
2024-06-12 09:09:09 -07:00
Ruben Groenewoud ec223a4a05 [New Rule] Suspicious File Modification (#3746)
* [New Rule] Suspicious File Modification

* Update persistence_suspicious_file_modifications.toml

* Update rules/linux/persistence_suspicious_file_modifications.toml

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>

* Update rules/linux/persistence_suspicious_file_modifications.toml

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>

* Updates

* Update rules/integrations/fim/persistence_suspicious_file_modifications.toml

---------

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
2024-06-11 13:03:20 +02:00
shashank-elastic f9b3534cdd Test deprecated rule modification (#3727) 2024-06-07 19:24:36 +05:30