39782b4295
[FR] Update utility path computation to use pathlib ( #3699 )
...
* update
* Updated to pathlib
* Linting
* Add string cast where needed
* Add additional string conversion as needed
* Str conversions to support eql lib
* Attack typo
* Typo in test script
* Updated for more pathlib
* Linting
* Update to convert string to path object
* Fix typo
(cherry picked from commit f43fbfba0d
2024-05-23 21:39:55 +00:00
f27479ee12
Package Manifest changes to add capabilities ( #3706 )
...
Removed changes from:
- detection_rules/etc/packages.yaml
(selectively cherry picked from commit f73022b900
2024-05-23 20:49:50 +00:00
18fcd83683
Back-porting Version Trimming ( #3704 )
...
(cherry picked from commit 63e91c2f12
2024-05-22 19:18:10 +00:00
0ab70f13a4
[Rule Tuning] Add Initial SentinelOne Compatibility to Windows DRs ( #3627 )
...
* [Rule Tuning] Add Initial SentinelOne Compatibility
* updated definitions.py; updated tags; fixed unit tests
* added prerelease versions for s1 integration; updated build CLI commands to allow prerelease; bumped min-stacks
* updating manifests and integrations
* fixing flake errors
* min_stack
---------
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
Co-authored-by: terrancedejesus <terrance.dejesus@elastic.co >
Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com >
(cherry picked from commit d023ad66b1
2024-05-20 12:59:37 +00:00
6e25eabf71
[FR] Add --force flag to update-lock-versions ( #3693 )
...
* Add --force flag to update-lock-versions
* Add type hinting
(cherry picked from commit 707ca32ab1
2024-05-18 00:33:11 +00:00
06ef471c39
[FR] Normalize yml ext to yaml ( #3675 )
2024-05-15 17:08:01 -05:00
2d96f10725
[FR] Normalize yml ext to yaml ( #3675 )
...
Removed changes from:
- detection_rules/etc/packages.yml
(selectively cherry picked from commit 79f575b33c
2024-05-15 20:27:01 +00:00
ed48d9fd57
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12,8.13,8.14 ( #3676 )
...
(cherry picked from commit f3585da503
2024-05-15 11:41:56 +00:00
891da3623d
Prepare For Next Elastic Stack 8.15 ( #3670 )
...
Removed changes from:
- detection_rules/etc/packages.yml
(selectively cherry picked from commit 50a8b52cd5
2024-05-14 19:10:09 +00:00
33e44b29fc
[FR] Bundle KQL & Kibana libs into base dependencies ( #3662 )
...
(cherry picked from commit 78837549e8
2024-05-13 19:36:55 +00:00
e45c7db95e
[Bug] Update Rule Formatter ( #3668 )
...
* Update Rule Formatter
* Only apply fix to Note
(cherry picked from commit 094ef22604
2024-05-13 19:07:19 +00:00
947e8fd965
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12,8.13 ( #3650 )
...
* Locked versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12,8.13
* Bumping status checks
* undo bump
---------
Co-authored-by: eric-forte-elastic <eric-forte-elastic@users.noreply.github.com >
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co >
(cherry picked from commit 84437bac03
2024-05-06 16:52:30 +00:00
2bd230ff60
[Bug] Query validation failing to capture InSet edge case with ip field types ( #3572 )
...
* Move test case to separate file
---------
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
Co-authored-by: shashank-elastic <91139415+shashank-elastic@users.noreply.github.com >
(cherry picked from commit a4a0bc6a7e
2024-05-06 12:07:00 +00:00
b75a9f902b
[New Rule] Potential Abuse of Resources by High Token Count and Large Response Sizes ( #3644 )
...
(cherry picked from commit 2ffb0e7fe2
2024-05-03 23:08:58 +00:00
c97395d606
[Bug] Fix missing indexes on navigator build ( #3636 )
...
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com >
(cherry picked from commit 2668f5f762
2024-05-01 21:58:13 +00:00
b83887e73d
[New Rule] AWS S3 Bucket Enumeration or Brute Force ( #3635 )
...
* [New Rule] AWS S3 Bucket Enumeration or Brute Force
Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com >
---------
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com >
Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com >
(cherry picked from commit 54ff270c62
2024-05-01 21:08:19 +00:00
809279b62b
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12,8.13 ( #3630 )
...
(cherry picked from commit ca78f550fd
2024-04-30 12:43:58 +00:00
09a7e2e81b
Refresh Kibana module with API updates ( #3466 )
...
* Refresh Kibana module with API updates
* add import/export commands
* rename repo commands
* add RawRuleCollection and DictRule objects
* save exported rules to files; rule.from_rule_resource
* strip unknown fields in schema
* add remote cli test
* update docs
* bump kibana lib version
---------
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com >
(cherry picked from commit c567d3731a
2024-04-26 17:20:37 +00:00
dfd261590b
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12,8.13 ( #3615 )
...
(cherry picked from commit 374f21fbc4
2024-04-23 12:36:46 +00:00
608a0ff0c2
[Rule Tuning] Windows BBR Rule Tuning - 1 ( #3579 )
...
* [Rule Tuning] Windows BBR Rule Tuning - 1
* Update non-ecs-schema.json
* Update rules_building_block/command_and_control_certutil_network_connection.toml
Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com >
* Update rules_building_block/collection_common_compressed_archived_file.toml
Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com >
* Update defense_evasion_dll_hijack.toml
---------
Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com >
Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com >
(cherry picked from commit d0dfa479bb
2024-04-08 13:46:29 +00:00
a2cb089d12
updated to v14.0 mitre ATT&CK ( #3289 )
...
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
Co-authored-by: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com >
(cherry picked from commit 0cb42983c1
2024-04-05 18:38:20 +00:00
dee8c947de
Update default ( #3574 )
...
(cherry picked from commit fbb6df506e
2024-04-05 00:35:15 +00:00
72ba0b16a9
[Bug] KQL fails validation on uppercase keywords ( #3568 )
...
* add todo
* Add a normalize_kql_keywords function to utils
* update rule loader to normalize and warn
* optimized loading
* fix linting
* Moved conversion to kql module.
* Updated unit test
* Refactor KQL parser to normalize keywords via flag
* Fix logic typo
* Update detection_rules/utils.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Update lib/kql/kql/__init__.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Updated to fix unit tests and remove warnings
* linting typo
* Added comments
* remove unused imports
* Update kql.parse default
---------
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
(cherry picked from commit 1566c29bae
2024-04-04 22:10:57 +00:00
645fa593a1
[Bug] New Terms Rule Import Failing ( #3569 )
...
* initial patch
* Update definitions to allow for brackets in name
* Update to prompt for required fields.
* Update detection_rules/cli_utils.py
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
---------
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
(cherry picked from commit fa75876322
2024-04-04 21:45:02 +00:00
5a28e1ecac
[Bug] Add explicit format preserver ( #3566 )
...
(cherry picked from commit c35652c8c8
2024-04-04 20:58:27 +00:00
ec275e8d99
[Bug] Threshold Rule Importing Failures ( #3560 )
...
* remove threshold specific req
* fix test event override
---------
Co-authored-by: shashank-elastic <91139415+shashank-elastic@users.noreply.github.com >
(cherry picked from commit a9cc323d09
2024-04-03 18:23:39 +00:00
fe9217892f
Deprecate Releasing to a patch kibana version workflow ( #3552 )
...
(cherry picked from commit 3fbffa24ed
2024-04-03 03:12:07 +00:00
112ae41cd3
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12,8.13 ( #3567 )
...
* Locked versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12,8.13
* Update detection_rules/etc/deprecated_rules.json
---------
Co-authored-by: shashank-elastic <shashank-elastic@users.noreply.github.com >
Co-authored-by: shashank-elastic <91139415+shashank-elastic@users.noreply.github.com >
(cherry picked from commit 8d5bd3b0f6
2024-04-02 18:37:42 +00:00
7838042839
[Rule Tuning] Replace KQL exceptions for Query DSL Exceptions ( #3505 )
...
* [Rule Tuning] Replace KQL exceptions for Query DSL Exceptions
* update min_stack
* build out schema in more detail for Filters
* Update detection_rules/rule.py
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
* Remove enum for definition
* remove unused import
* remove $state store
* transform state
* add call to super
* add return type hint
* use dataclass metadata
* use Literal type
---------
Co-authored-by: Samirbous <64742097+Samirbous@users.noreply.github.com >
Co-authored-by: Mika Ayenson <Mika.ayenson@elastic.co >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
Removed changes from:
- rules/windows/collection_mailbox_export_winlog.toml
- rules/windows/collection_posh_clipboard_capture.toml
- rules/windows/defense_evasion_posh_assembly_load.toml
- rules/windows/defense_evasion_posh_compressed.toml
- rules/windows/discovery_posh_suspicious_api_functions.toml
- rules/windows/discovery_privileged_localgroup_membership.toml
- rules/windows/execution_posh_hacktool_functions.toml
- rules/windows/execution_posh_psreflect.toml
- rules_building_block/collection_posh_compression.toml
- rules_building_block/defense_evasion_powershell_clear_logs_script.toml
- rules_building_block/discovery_posh_generic.toml
- rules_building_block/lateral_movement_posh_winrm_activity.toml
(selectively cherry picked from commit 67ca13c1ce
2024-04-01 20:53:09 +00:00
e74f7a4d6b
[FR] Add support for investigation_fields ( #3550 )
...
(cherry picked from commit bb907a4d76
2024-04-01 16:59:59 +00:00
69d2f4b607
Fix create PR in release workflow ( #3528 )
...
(cherry picked from commit 8b215eac41
2024-04-01 15:54:59 +00:00
e7416a6a68
[FR] Add required-fields option to import-rules ( #3546 )
...
(cherry picked from commit b6a7e7ebda
2024-03-28 23:37:15 +00:00
6bf3a82f51
Update sort parameter ( #3531 )
...
(cherry picked from commit 3503786154
2024-03-25 15:54:13 +00:00
dda6a33f70
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12,8.13 ( #3526 )
...
* Locked versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12,8.13
* Update detection_rules/etc/deprecated_rules.json
---------
Co-authored-by: shashank-elastic <shashank-elastic@users.noreply.github.com >
Co-authored-by: shashank-elastic <91139415+shashank-elastic@users.noreply.github.com >
(cherry picked from commit eaf4658620
2024-03-21 15:09:40 +00:00
edf52a578c
[FR] Update Python Dependency Versions ( #3515 )
...
(cherry picked from commit 5c3523954e
2024-03-19 19:15:12 +00:00
434b3ffcc0
[FR] Independently package kql / kibana and bump to py3.12 ( #3514 )
...
(cherry picked from commit d26981f712
2024-03-15 01:26:12 +00:00
2af0c64945
[FR] Add support for dataviews in the rule schema ( #3510 )
...
(cherry picked from commit 8724077a0e
2024-03-14 22:48:44 +00:00
59812dac4e
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12,8.13 ( #3491 )
...
(cherry picked from commit bf3932f384
2024-03-06 17:45:52 +00:00
7043173371
Prepare For Next Elastic Stack Minor Release ( #3490 )
...
Removed changes from:
- detection_rules/etc/packages.yml
(selectively cherry picked from commit a4094df732
2024-03-06 16:03:19 +00:00
f8a7fe9cec
[Bug] Fix URL links in autogenerated security docs ( #3474 )
...
* added content() class method for guide and setup
* removed non-existent variable
* removed unnecessary newlines
* adjusted levels for titles
* reverting changes
* added method to convert markdown links to asciidoc
* adjusted regex to include trailing periods
* fixing linting errors
* adjusted regex pattern
* added content() class method for guide and setup
* stripped # out of investigation guide, setup or note
* adjusted formatting outcome
* changed function call
* fixed linting errors
* fixing auto-formatting for rule asciidoc
* fixing URL link removal
* fixing URL link removal
* removed strip() from string for setup
* fixed linting errors
* fixed linting errors
* adjusting code formatting for convert_markdown_to_asciidoc
(cherry picked from commit 8e0ca421ca
2024-02-23 21:55:30 +00:00
2312455d7a
[FR] Skip eql optimizations on parsing query for unique fields ( #3443 )
...
(cherry picked from commit 542053719b
2024-02-21 02:31:01 +00:00
c772b2a842
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12 ( #3459 )
...
(cherry picked from commit 7815d23110
2024-02-20 17:32:25 +00:00
984f2a6fbf
[FR] NON_DATASET_PACKAGE list & Data Source tag for Auditd_manager ( #3430 )
...
* [FR] Add Auditd_Manager to NON_DATASET_PACKAGE
* Changed alphabetical order
---------
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
(cherry picked from commit a637bcec38
2024-02-19 08:42:19 +00:00
bde05d63c6
[FR] Add support for Threshold Alert Suppression ( #3433 )
...
(cherry picked from commit c3ca01ebcc
2024-02-12 16:01:10 +00:00
00fe4c8283
[Bug] Adjust build-release CLI and fix links when generating security docs ( #3434 )
...
* removed historical argument; added setup string; fixed links
* fixing flake errors
* added types for command arguments
* adjusted get_release_diff to append strings for release tags
* set fetch-depth to 0 for integrations checkout in workflow
* changed the name of the workflow
* removed TODOs
* adjusted release docs workflow to remove prefix for release tags
* adjusted URL replacement only if pointed to docs site
* added elastic website to regex pattern
* add docstrings; adjusted regex; add note for stopgap
* added a note about the regex pattern for elastic URLs
(cherry picked from commit 06b97ec79b
2024-02-12 15:13:42 +00:00
10d36f6872
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12 ( #3431 )
...
* Locked versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12
* Update detection_rules/etc/version.lock.json
* updated downloadable updates file to reconcile changes
* Removed spacing from downloadable updates file
---------
Co-authored-by: terrancedejesus <terrancedejesus@users.noreply.github.com >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
(cherry picked from commit 827dfa7327
2024-02-06 19:54:15 +00:00
7201490af1
[Bug] Update Prebuilt Detection Rules Release Process ( #3403 )
...
* release fleet workflow updates; build package integration reference changes
* updated commit hash extraction to output to env
* adjusted bump-pkg-versions to only include release if necessary
* fixed flake errors
* add historical argument for build-release set to yes by default
* Update detection_rules/devtools.py
* fixed fleet workflow; updated registry data references
* updated job names
* removed extract commit hash job and consolidated into fleet pr job
* added echo statement for current branch before checkout
* removed id from extract commit hash
(cherry picked from commit 7df7ab5101
2024-02-06 14:04:40 +00:00
df82c11b4a
Lock versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12 ( #3402 )
...
* Locked versions for releases: 8.3,8.4,8.5,8.6,8.7,8.8,8.9,8.10,8.11,8.12
* Update detection_rules/etc/version.lock.json
---------
Co-authored-by: terrancedejesus <terrancedejesus@users.noreply.github.com >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
(cherry picked from commit d093336125
2024-01-23 21:42:17 +00:00
cfb4f1a013
[New Rules] UEBA GItHub BBRs and Rules ( #3174 )
...
* [New Rules] UEBA GItHub BBRs and Rules
A new set of BBRs and rules that will be used to trigger new UEBA GitHub threshold Rules.
* Update rules/integrations/github/impact_github_member_removed_from_organization.toml
* Apply suggestions from code review
Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com >
* edited BBR rules
-removed newly added member rule
* updated integration manifests and schemas
* Updated min_stack for some rules based on newest GitHub integration schema manifest
* testing min_stack bump to 8.8 for new fields
* removing offending rule to troubleshoot seperately
* added UEBA tags and created UEBA threshold rule
* updated non-ecs-schema to add signal.rule.tags
* updated non-ecs-schema with kibana.alert.workflow_status
* updated rule.threat.tactic
* added user.name to non-ecs-schema
* added quotes to kibana.alert.workflow_status value
* removed trailing space from rule name
* update tags and optimize query for UEBA threshold rule
* removed integration field from Higher-Order rule
* Apply suggestions from code review
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
* adjusted new_terms order and rule types based on review feedback
* Apply suggestions from code review
Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com >
* remove user.name from detection_rules/etc/non-ecs-schema.json
* fix json formatting
---------
Co-authored-by: Ruben Groenewoud <78494512+Aegrah@users.noreply.github.com >
Co-authored-by: Colson Wilhoit <48036388+DefSecSentinel@users.noreply.github.com >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
(cherry picked from commit 442435830f
2024-01-22 17:53:42 +00:00
8a80d74136
[FR] Update Validate Integrations to Check Fields Across All Schema Variations ( #3372 )
...
(cherry picked from commit a873abbb5b
2024-01-18 21:47:51 +00:00
Eric Forte