Commit Graph

125 Commits

Author SHA1 Message Date
Mika Ayenson, PhD f0f7d217c0 [FR] Refactor Schema Validation & Support Multi-Dataset Sequence Validation (#5059) 2025-09-10 13:11:04 -05:00
shashank-elastic 6adee51410 Fix Ruff failures (#5083) 2025-09-10 22:24:07 +05:30
shashank-elastic a6dfd2c0e1 Add test_min_stack_version_supported testcase (#5077) 2025-09-10 20:12:36 +05:30
Mika Ayenson, PhD 1dc3926203 [New Rules] External Promotion Alerts (#4903) 2025-07-31 11:00:50 -05:00
Sergey Polzunov 1fb60d6475 fix: type hinting fixes and additional code checks (#4790)
* first pass

* Adding a dedicated code checking workflow

* Type fixes

* linting config and python version bump

* Type hints

* Drop incorrect config option

* More fixes

* Style fixes

* CI adjustments

* Pyproject fixes

* CI & pyproject fixes

* Proper version bump

* Tests formatting

* Resolve cirtular dependency

* Test fixes

* Make sure the tests are formatted correctly

* Check tweaks

* Bumping python version in CI images

* Pin marshmallow do 3.x because 4.x is not supported

* License fix

* Convert path to str

* Making myself a codeowner

* Missing kwargs param

* Adding a missing kwargs to `set_score`

* Update .github/CODEOWNERS

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* Dropping unnecessary raise

* Dropping skipped test

* Drop unnecessary var

* Drop unused commented-out func

* Disable typehinting for the whole func

* Update linting command

* Invalid type hist on the input param

* Incorrect field type

* Incorrect value used fix

* Stricter values check

* Simpler function call

* Type condition fix

* TOML formatter fix

* Simpligy output conditions

* Formatting

* Use proper types instead of aliases

* MITRE attack fixes

* Using pathlib.Path for an argument

* Use proper method to update a set from a dict

* First round of `ruff` fixes

* More fixes

* More fixes

* Hack against cyclic dependency

* Ignore `PLC0415`

* Remove unused markers

* Cleanup

* Fixing the incorrect condition

* Update .github/CODEOWNERS

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* Set explicit default values for optional fields

* Update the guidelines

* Adding None Defaults

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
2025-07-01 08:20:55 -05:00
shashank-elastic 2b3095a13c Update Max signals value to supported limits (#4556) 2025-03-27 09:02:25 +05:30
Sergey Polzunov 5f54eb8006 chore: Removing RTAs (#4437)
* Delete RTAs

* Delete RTA-related orchestration code

* Drop RTAs from tests

* Remove RTAs from README

* Further cleanup

* Readme update

* Version bump and no more RTAs

* Styling fixes

* Drop RTAs from config files

* Drop `rule-mapping.yaml`

* Bring back event collector / normalizer

* Drop rta mention

* Cleanup rta leftovers

* Style fix

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
2025-03-05 12:35:57 +01:00
Jonhnathan c0f12ddecf [Rule Tuning] Tighten Up Windows EventLog Indexes, Improve tags (#4464)
* [Rule Tuning] Tighten Up Windows EventLog Indexes, Improve tags

* Format & order

* Update pyproject.toml

* Update credential_access_cookies_chromium_browsers_debugging.toml
2025-02-19 12:54:31 -03:00
shashank-elastic aded9deb79 Modify Unit Test to Support Alert Suppression for EQL Sequences (#4457) 2025-02-14 00:14:28 +05:30
shashank-elastic 818467f132 Replace master doc URLs with current (#4439) 2025-02-03 21:27:50 +05:30
Mika Ayenson fe8c81d762 [FR] Generate investigation guides (#4358) 2025-01-22 11:17:38 -06:00
Jonhnathan 81292aee8a [Rule Tuning] 3rd Party EDR - Add Crowdstrike FDR support - 1 (#4220)
* [Rule Tuning] 3rd Party EDR - Add Crowdstrike FDR support - 1

* Update Integrations unit tests

* Update test_all_rules.py
2024-11-04 11:32:22 -03:00
shashank-elastic 275c7288a3 Add testcase to check for related_integrations based on index (#4096) 2024-10-22 00:17:30 +05:30
shashank-elastic a53f7d55a3 Testcase to check if Rule Type: BBR tag is present for all BBR rules (#4048) 2024-09-02 21:29:31 +05:30
Mika Ayenson 10ba6ad5a6 [FR] Add Alert Suppression for Addtional Rule Types (#3986) 2024-08-15 15:03:45 -05:00
shashank-elastic e607d521b8 Add Unit Test test_index_or_data_view_id_present (#3967) 2024-08-12 17:48:05 +05:30
Eric Forte 47d7a3acaa [DaC] Beta Release (#3889)
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
Co-authored-by: Mika Ayenson <mika.ayenson@elastic.co>
2024-08-06 18:07:12 -04:00
shashank-elastic 59a10be7c8 Unit Test to validate from field in toml file (#3866) 2024-07-10 22:41:53 +05:30
shashank-elastic 50f0fb3518 Test case to check updated_date (#3818) 2024-07-03 19:17:27 +05:30
Jonhnathan 54d5b442cf [Rule Tuning] Add Initial Microsoft Defender for Endpoint Compatibility to Windows DRs (#3825)
* [Rule Tuning] Add Initial Microsoft Defender for Endpoint Compatibility to Windows DRs

* .

* Update integration-schemas.json.gz

* Fix integration manifests
2024-06-26 11:06:27 -03:00
Mika Ayenson a131e021ed Revert "Test case to check updated_date (#3764)"
This reverts commit 7621a54fcc.
2024-06-21 11:05:57 -05:00
shashank-elastic 7621a54fcc Test case to check updated_date (#3764) 2024-06-21 18:43:32 +05:30
shashank-elastic 675cad2ed4 Incorrect Integration Index Check (#3794) 2024-06-21 18:29:39 +05:30
Terrance DeJesus 020ca4be24 [New Rule] Rapid7 Threat Command CVEs Correlation (#3718)
* new rule 'Rapid7 Threat Command CVEs Correlation'

* Update rules/threat_intel/threat_intel_rapid7_threat_command.toml

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* updated threat index and tags

* changed 'indicator match' to 'threat match' for tags

* removed timeline

* updating integrations to match main

* re-adding rapid7 threat command integration manifest and schema

* reverting changes; removing timeline

* changed max signals to 10000

---------

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
2024-06-12 18:01:44 -04:00
Ruben Groenewoud ec223a4a05 [New Rule] Suspicious File Modification (#3746)
* [New Rule] Suspicious File Modification

* Update persistence_suspicious_file_modifications.toml

* Update rules/linux/persistence_suspicious_file_modifications.toml

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>

* Update rules/linux/persistence_suspicious_file_modifications.toml

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>

* Updates

* Update rules/integrations/fim/persistence_suspicious_file_modifications.toml

---------

Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com>
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
2024-06-11 13:03:20 +02:00
shashank-elastic f9b3534cdd Test deprecated rule modification (#3727) 2024-06-07 19:24:36 +05:30
shashank-elastic 63e91c2f12 Back-porting Version Trimming (#3704) 2024-05-23 00:45:10 +05:30
Mika Ayenson 2c3dbfc039 Revert "Back-porting Version Trimming (#3681)"
This reverts commit 71d2c59b5c.
2024-05-22 13:51:46 -05:00
shashank-elastic 71d2c59b5c Back-porting Version Trimming (#3681) 2024-05-23 00:11:50 +05:30
Jonhnathan d023ad66b1 [Rule Tuning] Add Initial SentinelOne Compatibility to Windows DRs (#3627)
* [Rule Tuning] Add Initial SentinelOne Compatibility

* updated definitions.py; updated tags; fixed unit tests

* added prerelease versions for s1 integration; updated build CLI commands to allow prerelease; bumped min-stacks

* updating manifests and integrations

* fixing flake errors

* min_stack

---------

Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: terrancedejesus <terrance.dejesus@elastic.co>
Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com>
2024-05-20 09:50:57 -03:00
Mika Ayenson 79f575b33c [FR] Normalize yml ext to yaml (#3675) 2024-05-15 15:18:39 -05:00
Mika Ayenson f07a9e6fbc [FR] Add max_signal note, unit test, and rule tuning (#3669) 2024-05-14 11:15:12 -05:00
Eric Forte fbb6df506e Update default (#3574) 2024-04-04 20:27:14 -04:00
Eric Forte 1566c29bae [Bug] KQL fails validation on uppercase keywords (#3568)
* add todo

* Add a normalize_kql_keywords function to utils

* update rule loader to normalize and warn

* optimized loading

* fix linting

* Moved conversion to kql module.

* Updated unit test

* Refactor KQL parser to normalize keywords via flag

* Fix logic typo

* Update detection_rules/utils.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update lib/kql/kql/__init__.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Updated to fix unit tests and remove warnings

* linting typo

* Added comments

* remove unused imports

* Update kql.parse default

---------

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2024-04-04 18:03:30 -04:00
Eric Forte a9cc323d09 [Bug] Threshold Rule Importing Failures (#3560)
* remove threshold specific req

* fix test event override

---------

Co-authored-by: shashank-elastic <91139415+shashank-elastic@users.noreply.github.com>
2024-04-03 14:15:09 -04:00
Jonhnathan f5254f3b5e [Rule Tuning] Improve Compatibility in WIndows Detection Rules - Part 1 (#3501)
* Initial commit

* Date bump
2024-03-13 10:27:44 -03:00
Mika Ayenson c3ca01ebcc [FR] Add support for Threshold Alert Suppression (#3433) 2024-02-12 09:55:46 -06:00
Terrance DeJesus 164b7d4028 removed query var; using is_sequence method; removed integration var (#3395) 2024-01-22 15:23:07 -05:00
Terrance DeJesus 1c10c37468 [Rule Tuning] Update timestamp_override Unit Tests and Fix Rules Missing Field (#3368)
* updated timestamp override unit test; fixed rules missing this field

* fixed flake error

* simplified and consolidated logic

* Update tests/test_all_rules.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* Update tests/test_all_rules.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* added comments

* updated logic; added comments; removed unused variables

* removed custom python script

* updated dates

* removed deprecated rule change

* updated dates

---------

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2024-01-17 14:14:38 -05:00
Terrance DeJesus eafec1d857 [Bug] Fix BBR Folder Location Requirements for Specific Integrations (#3348)
* fixing bug in BBR rule folder location

* fixed export rules missing BBR rules

* adjusted directory loading

* Update tests/test_all_rules.py

Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>

---------

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>
2023-12-19 15:36:45 -05:00
Apoorva Joshi 9a9f5437f2 Update Advanced Analytics config guides (#3302)
* Updating config guides for Advanced Analytics rules

* More updates

* Update setup instructions for LMD

* Adding more guides

* update TestRuleTiming unit test to ignore advanced analytic rules

* fixed flake error

* Moving config guides under setup instead of note

* Removing leading and trailing whitespace

* Updates as requested by PM

* Updating related integrations, minor updates to setup guides

* fixing unit tests to ignore analytic packages with multiple integration tags

* Update tests/test_all_rules.py

* fixing linting errors

---------

Co-authored-by: Kirti Kirti <kirti.kirti@elastic.co>
Co-authored-by: terrancedejesus <terrance.dejesus@elastic.co>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
2023-12-13 07:53:41 -08:00
Mika Ayenson 7514c0a206 [FR] Add Support for ES|QL Rule Type and Remote Validation (#3281)
* add suuport for esql type
* add unit tests
* set clients in RemoteConnector from auth methods
* thread remote rules; add engine test
* Add versions to remote validation results

---------

Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
2023-12-08 12:46:28 -07:00
Jonhnathan aeb1f91320 [Security Content] Introduce Investigate Plugin in Investigation Guides (#3080)
* [Security Content] Introduce Investigate Plugin in Investigation Guides
* Add compatibility note
* Update Transform format
* update transform unit tests for investigate
* updated docs with transform

---------

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
2023-12-08 11:54:40 -07:00
Terrance DeJesus 5358361754 Adjust ESQLRuleData to Inherit QueryRuleData Dataclass (#3297)
* adjusting inheritance of ESQL rule data

* update tests to handle missing index from QueryRuleData

* removed test es|ql rule

---------

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
2023-11-30 09:06:34 -05:00
Terrance DeJesus 829f5ea885 [Bug] Add Integration Schema Validation to NewTermsRuleData.validate Method (#3227)
* adjusted validation method to include integration schema checks

* fixed linting errors

* re-factored NewTermsRuleData and added unit testing
2023-11-02 16:52:18 -04:00
Terrance DeJesus e6ead7b5f7 [FR] Adjust Risk Severity Unit Test to Match UI (#3218)
* adjusted risk score to include ranges allowed upstream

* Update tests/test_all_rules.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

---------

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2023-10-25 10:25:46 -04:00
Terrance DeJesus 3ab57fb8a7 [FR] Adding Support for missing_field_strategy Field in Alert Suppression (#3201)
* adding missing field strategy option to alert suppression

* fixed linting errors

* added validate methods for alertsuppression dataclass

* fixed linting errors

* replaced old variable with new variable

* removing test rule

* adding post_load to queryruledata

* changed post_load to validates_schema

* updated unit testing for alert suppression

* fixed linting errors

* changed validates method name to validates_exceptions

* removed min compat for fields
2023-10-19 18:16:54 -04:00
Terrance DeJesus 3e212e2b74 [FR] Add ML Jobs to Schemas and Unit Test for Validation (#3161)
* adding machine learning job id validation

* Update rules/ml/credential_access_ml_auth_spike_in_logon_events_from_a_source_ip.toml

* Update tests/test_all_rules.py

* adding integration manifests and schemas from main

* rebuilt manifests and schemas with lmd

* fixed unit test linting

* adding manifests and schemas for other analytic packages

* updated manifests and schemas; adjusted unit test for verbosity

* sorted imports
2023-10-12 10:51:12 -04:00
eric-forte-elastic 4828ae07df [FR] Added asset tag to expected tags (#3115)
* Added asset tag to expected tags

* removed *

* Add regex wildcard tag support

* Updated tag format test location

* Updated to use env variable

* fixed typo
2023-09-28 14:09:05 -04:00
Apoorva Joshi 747ee7d593 [New Rule] Adding Lateral Movement Rules from Advanced Analytic LMD Package (#3119)
* Adding Lateral Movement Detection rules

* added tags; adjusted tests; updated manifests and schemas

* added default value to build_integrations_schema

* combined analytic and non-dataset packages for related integrations

* adjusted machine learning definitions

* adjusted machine learning definitions

* removed splat for machine learning list due to 3.8 constraints

---------

Co-authored-by: terrancedejesus <terrance.dejesus@elastic.co>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
2023-09-27 14:53:38 -04:00