f66da9d350
[FR] Update Python Dependency Versions ( #3515 )
...
(cherry picked from commit 5c3523954e
2024-03-19 19:14:23 +00:00
3354460843
[FR] Independently package kql / kibana and bump to py3.12 ( #3514 )
...
(cherry picked from commit d26981f712
2024-03-15 01:25:26 +00:00
f1542e6ef5
[FR] Add support for dataviews in the rule schema ( #3510 )
...
(cherry picked from commit 8724077a0e
2024-03-14 22:48:17 +00:00
98cfc81c4a
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12,8.13 ( #3491 )
...
(cherry picked from commit bf3932f384
2024-03-06 17:45:13 +00:00
8cd057dc58
Prepare For Next Elastic Stack Minor Release ( #3490 )
...
Removed changes from:
- detection_rules/etc/packages.yml
(selectively cherry picked from commit a4094df732
2024-03-06 16:02:44 +00:00
111b881503
[Bug] Fix URL links in autogenerated security docs ( #3474 )
...
* added content() class method for guide and setup
* removed non-existent variable
* removed unnecessary newlines
* adjusted levels for titles
* reverting changes
* added method to convert markdown links to asciidoc
* adjusted regex to include trailing periods
* fixing linting errors
* adjusted regex pattern
* added content() class method for guide and setup
* stripped # out of investigation guide, setup or note
* adjusted formatting outcome
* changed function call
* fixed linting errors
* fixing auto-formatting for rule asciidoc
* fixing URL link removal
* fixing URL link removal
* removed strip() from string for setup
* fixed linting errors
* fixed linting errors
* adjusting code formatting for convert_markdown_to_asciidoc
(cherry picked from commit 8e0ca421ca
2024-02-23 21:54:58 +00:00
8f31731cc3
[FR] Skip eql optimizations on parsing query for unique fields ( #3443 )
...
(cherry picked from commit 542053719b
2024-02-21 02:30:32 +00:00
d6c57956c0
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12 ( #3459 )
...
(cherry picked from commit 7815d23110
2024-02-20 17:31:57 +00:00
cfc0b41e20
[FR] NON_DATASET_PACKAGE list & Data Source tag for Auditd_manager ( #3430 )
...
* [FR] Add Auditd_Manager to NON_DATASET_PACKAGE
* Changed alphabetical order
---------
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
(cherry picked from commit a637bcec38
2024-02-19 08:41:46 +00:00
c36803d464
[FR] Add support for Threshold Alert Suppression ( #3433 )
...
(cherry picked from commit c3ca01ebcc
2024-02-12 16:00:34 +00:00
90d069bb08
[Bug] Adjust build-release CLI and fix links when generating security docs ( #3434 )
...
* removed historical argument; added setup string; fixed links
* fixing flake errors
* added types for command arguments
* adjusted get_release_diff to append strings for release tags
* set fetch-depth to 0 for integrations checkout in workflow
* changed the name of the workflow
* removed TODOs
* adjusted release docs workflow to remove prefix for release tags
* adjusted URL replacement only if pointed to docs site
* added elastic website to regex pattern
* add docstrings; adjusted regex; add note for stopgap
* added a note about the regex pattern for elastic URLs
(cherry picked from commit 06b97ec79b
2024-02-12 15:13:10 +00:00
98b7a409fc
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12 ( #3431 )
...
* Locked versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12
* Update detection_rules/etc/version.lock.json
* updated downloadable updates file to reconcile changes
* Removed spacing from downloadable updates file
---------
Co-authored-by: terrancedejesus <terrancedejesus@users.noreply.github.com >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
(cherry picked from commit 827dfa7327
2024-02-06 19:53:46 +00:00
5c0880e53a
[Bug] Update Prebuilt Detection Rules Release Process ( #3403 )
...
* release fleet workflow updates; build package integration reference changes
* updated commit hash extraction to output to env
* adjusted bump-pkg-versions to only include release if necessary
* fixed flake errors
* add historical argument for build-release set to yes by default
* Update detection_rules/devtools.py
* fixed fleet workflow; updated registry data references
* updated job names
* removed extract commit hash job and consolidated into fleet pr job
* added echo statement for current branch before checkout
* removed id from extract commit hash
(cherry picked from commit 7df7ab5101
2024-02-06 14:04:04 +00:00
f31a1917dc
changed the kibana version of packages.yml ref issue 3374
2024-01-25 17:35:09 -05:00
80be303533
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12 ( #3402 )
...
* Locked versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12
* Update detection_rules/etc/version.lock.json
---------
Co-authored-by: terrancedejesus <terrancedejesus@users.noreply.github.com >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
(cherry picked from commit d093336125
2024-01-23 21:41:46 +00:00
f0028e1457
[New Rules] UEBA GItHub BBRs and Rules ( #3174 )
...
* [New Rules] UEBA GItHub BBRs and Rules
A new set of BBRs and rules that will be used to trigger new UEBA GitHub threshold Rules.
* Update rules/integrations/github/impact_github_member_removed_from_organization.toml
* Apply suggestions from code review
Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com >
* edited BBR rules
-removed newly added member rule
* updated integration manifests and schemas
* Updated min_stack for some rules based on newest GitHub integration schema manifest
* testing min_stack bump to 8.8 for new fields
* removing offending rule to troubleshoot seperately
* added UEBA tags and created UEBA threshold rule
* updated non-ecs-schema to add signal.rule.tags
* updated non-ecs-schema with kibana.alert.workflow_status
* updated rule.threat.tactic
* added user.name to non-ecs-schema
* added quotes to kibana.alert.workflow_status value
* removed trailing space from rule name
* update tags and optimize query for UEBA threshold rule
* removed integration field from Higher-Order rule
* Apply suggestions from code review
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
* adjusted new_terms order and rule types based on review feedback
* Apply suggestions from code review
Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com >
* remove user.name from detection_rules/etc/non-ecs-schema.json
* fix json formatting
---------
Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com >
Co-authored-by: Colson Wilhoit <48036388+DefSecSentinel@users.noreply.github.com >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
(cherry picked from commit 442435830f
2024-01-22 17:53:12 +00:00
e7fd90f2b1
[FR] Update Validate Integrations to Check Fields Across All Schema Variations ( #3372 )
...
(cherry picked from commit a873abbb5b
2024-01-18 21:47:18 +00:00
0afe7715f0
[FR] Update _event_sort to use datetime instead of time ( #3375 )
...
* Update _event_sort to use datetime
* remove unused time
* added type hints
(cherry picked from commit 6170db6231
2024-01-09 16:03:47 +00:00
bb7bf106f7
[FR] Add --include-metadata argument to export-rules command ( #3365 )
...
* added --include-metadata argument to export-rules command
* added type hinting in method definitions
* changed add_metadata to include_metadata
* adjusted argument name to include_metadata in command
* Update detection_rules/main.py
Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com >
* fixed flake error
* Update detection_rules/rule.py
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
---------
Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com >
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
(cherry picked from commit d7b62395e7
2024-01-04 21:07:25 +00:00
f882c20919
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12 ( #3358 )
...
* Locked versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12
* Update detection_rules/etc/deprecated_rules.json
---------
Co-authored-by: terrancedejesus <terrancedejesus@users.noreply.github.com >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
(cherry picked from commit f37d13f29b
2024-01-02 17:30:17 +00:00
49d2a748d0
[Bug] Fix BBR Folder Location Requirements for Specific Integrations ( #3348 )
...
* fixing bug in BBR rule folder location
* fixed export rules missing BBR rules
* adjusted directory loading
* Update tests/test_all_rules.py
Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com >
---------
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com >
(cherry picked from commit eafec1d857
2023-12-19 20:41:40 +00:00
5b8e686583
[Tuning] Remote Scheduled Task Creation ( #3337 )
...
* Update non-ecs-schema.json
* add timestamp override
---------
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
(cherry picked from commit 07b952b7bc
2023-12-14 23:44:37 +00:00
760735c90b
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12 ( #3319 )
...
* Locked versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12
* Update detection_rules/etc/version.lock.json
---------
Co-authored-by: terrancedejesus <terrancedejesus@users.noreply.github.com >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
(cherry picked from commit a39a52360a
2023-12-12 18:27:49 +00:00
b70bbe0841
[New Rule] Adding Detection for Stolen Credentials Used to Login to Okta Account After MFA Reset ( #3265 )
...
* adding new rule 'Stolen Credentials Used to Login to Okta Account After MFA Reset'
* updated non-ecs; linted rule; updated description
* adjusted interval and maxspan
* Update rules/integrations/okta/persistence_stolen_credentials_used_to_login_to_okta_account_after_mfa_reset.toml
Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com >
* Update rules/integrations/okta/persistence_stolen_credentials_used_to_login_to_okta_account_after_mfa_reset.toml
Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com >
* Update rules/integrations/okta/persistence_stolen_credentials_used_to_login_to_okta_account_after_mfa_reset.toml
Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com >
* Update rules/integrations/okta/persistence_stolen_credentials_used_to_login_to_okta_account_after_mfa_reset.toml
---------
Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com >
(cherry picked from commit 93d71acb91
2023-12-12 15:36:56 +00:00
1fc8e591d7
[FR] 8.12 Release Preparation update Main Branch to 8.13 ( #3313 )
...
* 8.12 Release Prep update Main Branch to 8.13
* Fix typo in integrations
* Updated Schemas
Removed changes from:
- detection_rules/etc/packages.yml
(selectively cherry picked from commit 90a2043bc4
2023-12-11 20:02:56 +00:00
1f776c8ebe
[Bug] Use integration schemas for required_field types ( #3303 )
...
(cherry picked from commit face95058f
2023-12-11 17:37:17 +00:00
111ce46b75
[FR] Add Support for ES|QL Rule Type and Remote Validation ( #3281 )
...
* add suuport for esql type
* add unit tests
* set clients in RemoteConnector from auth methods
* thread remote rules; add engine test
* Add versions to remote validation results
---------
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com >
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
(cherry picked from commit 7514c0a206
2023-12-08 19:51:44 +00:00
87f8498b68
[Security Content] Introduce Investigate Plugin in Investigation Guides ( #3080 )
...
* [Security Content] Introduce Investigate Plugin in Investigation Guides
* Add compatibility note
* Update Transform format
* update transform unit tests for investigate
* updated docs with transform
---------
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com >
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
(cherry picked from commit aeb1f91320
2023-12-08 18:59:26 +00:00
2168afc8f8
Adjust ESQLRuleData to Inherit QueryRuleData Dataclass ( #3297 )
...
* adjusting inheritance of ESQL rule data
* update tests to handle missing index from QueryRuleData
* removed test es|ql rule
---------
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com >
(cherry picked from commit 5358361754
2023-11-30 14:11:53 +00:00
7c22714ce0
Update QueryRuleData ( #3294 )
...
(cherry picked from commit f7b9a1f8df
2023-11-29 15:47:40 +00:00
2c5e0fa2f7
FR] Add Core Support for ES|QL Rule Type ( #3292 )
...
(cherry picked from commit bc39c20eaf
2023-11-28 19:08:07 +00:00
9d34fc21ae
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11 ( #3291 )
...
* Locked versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11
* Update detection_rules/etc/version.lock.json
---------
Co-authored-by: terrancedejesus <terrancedejesus@users.noreply.github.com >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
(cherry picked from commit ba7b2722c2
2023-11-28 17:35:29 +00:00
98b331e2ce
Add UEBA Tag ( #3277 )
...
(cherry picked from commit 93ad4b0959
2023-11-20 19:55:56 +00:00
eb835a5079
[Bug] Fix typo in downgrade_contents_from_rule ( #3272 )
...
* Fix missing to_dict()
* Update pyproject.toml
(cherry picked from commit 66c1d7f3b4
2023-11-15 04:10:54 +00:00
f1da3e86bc
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11 ( #3270 )
...
* Locked versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11
* Update detection_rules/etc/version.lock.json
---------
Co-authored-by: terrancedejesus <terrancedejesus@users.noreply.github.com >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
(cherry picked from commit 9195eedb9c
2023-11-13 19:50:48 +00:00
39346f4b5b
[Bug] Add Integration Schema Validation to NewTermsRuleData.validate Method ( #3227 )
...
* adjusted validation method to include integration schema checks
* fixed linting errors
* re-factored NewTermsRuleData and added unit testing
(cherry picked from commit 829f5ea885
2023-11-02 20:57:37 +00:00
029c826043
[FR] Adjust Prebuilt Rules Packaging to Use Elastic Package v3 ( #3252 )
...
* Adding support for elastic package version 3
* replaced OS with Pathlib where applicable
* added sub-dataclasses for V3
* fixed flake errors
* adjusted registry dataclasses to inherit base
Removed changes from:
- detection_rules/etc/packages.yml
(selectively cherry picked from commit cdeb398ab3
2023-11-01 16:53:22 +00:00
98e8748f9f
[FR] Support missing events ( #3153 )
...
(cherry picked from commit d0b0216362
2023-10-31 21:26:25 +00:00
f32a83791a
[New Rule] Adding Beaconing Rules from Advanced Analytic Beaconing Package ( #3128 )
...
* Adding beaconing rules
* Update rules/integrations/beaconing/command_and_control_beaconing_high_confidence.toml
Co-authored-by: Kirti Sodhi <109447885+sodhikirti07@users.noreply.github.com >
* Update rules/integrations/beaconing/command_and_control_beaconing.toml
Co-authored-by: Kirti Sodhi <109447885+sodhikirti07@users.noreply.github.com >
* Updating min stack version
* added beaconing to manifests and schemas; updated rules
---------
Co-authored-by: Kirti Sodhi <109447885+sodhikirti07@users.noreply.github.com >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
Co-authored-by: terrancedejesus <terrance.dejesus@elastic.co >
Co-authored-by: Colson Wilhoit <48036388+DefSecSentinel@users.noreply.github.com >
(cherry picked from commit a4f9cf4616
2023-10-30 14:11:53 +00:00
86bca86b6c
Cleanup saved_query references ( #3205 )
...
(cherry picked from commit a808130390
2023-10-26 23:13:17 +00:00
46c430d4d4
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11 ( #3223 )
...
* Locked versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11
* Update detection_rules/etc/deprecated_rules.json
---------
Co-authored-by: terrancedejesus <terrancedejesus@users.noreply.github.com >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
(cherry picked from commit ab6f28a380
2023-10-24 18:06:52 +00:00
60475f6aa0
Move Setup information into setup filed ( #3206 )
...
(cherry picked from commit 7254c582c5
2023-10-23 14:04:26 +00:00
21bff0b9d2
[FR] Adding Support for missing_field_strategy Field in Alert Suppression ( #3201 )
...
* adding missing field strategy option to alert suppression
* fixed linting errors
* added validate methods for alertsuppression dataclass
* fixed linting errors
* replaced old variable with new variable
* removing test rule
* adding post_load to queryruledata
* changed post_load to validates_schema
* updated unit testing for alert suppression
* fixed linting errors
* changed validates method name to validates_exceptions
* removed min compat for fields
(cherry picked from commit 3ab57fb8a7
2023-10-19 22:22:50 +00:00
cad094abbd
[New Rule] Adding DGA Rules from Advanced Analytic DGA Package ( #3102 )
...
* Adding DGA rules
* Adding references
* updated rule tags and queries
* Updating min stack version
* added logic to handle ml jobs
* added code comments for clarity
* removing subbed security docs folder
* added event dataset to queries for endpoint; updated note
* removed event dataset
---------
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
Co-authored-by: terrancedejesus <terrance.dejesus@elastic.co >
(cherry picked from commit a5a606e804
2023-10-16 19:54:30 +00:00
045de05e46
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11 ( #3183 )
...
* Locked versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11
* Update detection_rules/etc/version.lock.json
---------
Co-authored-by: terrancedejesus <terrancedejesus@users.noreply.github.com >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
(cherry picked from commit 2b0735024e
2023-10-13 19:16:31 +00:00
685cc8f628
[FR] 8.11 Release Preparation and Update Main Branch to 8.12 ( #3182 )
...
* prepping for 8.12 branch
* added ananlytic manifests and schemas
* fix linting issues
* updated analytic package manifests and schemas
Removed changes from:
- detection_rules/etc/packages.yml
(selectively cherry picked from commit b4f8fc3290
2023-10-13 17:43:21 +00:00
d72996c401
[New Rule] Migrate Lateral Movement Detection Rules ( #3175 )
...
* adding LMD rules
* added setup note; updated references
* adds 2.0.0 lmd manifest and schema
* adjusted min-stack for non-ML rules
(cherry picked from commit 1e514afa57
2023-10-12 19:07:54 +00:00
0308e32ea0
[FR] Add ML Jobs to Schemas and Unit Test for Validation ( #3161 )
...
* adding machine learning job id validation
* Update rules/ml/credential_access_ml_auth_spike_in_logon_events_from_a_source_ip.toml
* Update tests/test_all_rules.py
* adding integration manifests and schemas from main
* rebuilt manifests and schemas with lmd
* fixed unit test linting
* adding manifests and schemas for other analytic packages
* updated manifests and schemas; adjusted unit test for verbosity
* sorted imports
(cherry picked from commit 3e212e2b74
2023-10-12 14:57:00 +00:00
7c563fb834
[New Rule] File Compressed or Archived into Common Format ( #3173 )
...
* [New Rule] File Compressed or Archived into Common Format
* new build-threat-map-entry-command
---------
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com >
Co-authored-by: Jonhnathan <26856693+w0rk3r@users.noreply.github.com >
(cherry picked from commit 7f8a9849c4
2023-10-11 18:40:16 +00:00
f67291561e
[FR] Only supporting known compatible rule file types ( #3167 )
...
* Only supporting known compatible file types
* Add --ignore-invalid-files flag
* Added support to ignore invalid rule files
* Update detection_rules/utils.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Update detection_rules/utils.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Update detection_rules/utils.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Update detection_rules/utils.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Update detection_rules/main.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* reverting main
* add punctuation
---------
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
(cherry picked from commit 9f61ce4923
2023-10-11 15:49:41 +00:00
Mika Ayenson