Commit Graph

191 Commits

Author SHA1 Message Date
Mika Ayenson 0358ec9d9a Release ER Production RTAs to DR (#2270) 2022-09-08 12:50:39 -04:00
Justin Ibarra 332ea40100 Cleanup rule survey code (#1923)
* Cleanup rule survey code

* default to only unique-ing on process name for lucene rules

* fix bug in kibana url parsing by removing redundant port from domain

* update search-alerts columns and nest fields

* fix rule.contents.data.index

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2022-09-06 15:53:47 -06:00
Justin Ibarra d37eac8d9d Add test that newly introduced build-time fields for a min_stack for … (#2262)
* add test that newly introduced build-time fields for a min_stack for applicable rules.

* account for rules without min_stack_version

* limit test to >= stack ver
2022-08-25 21:56:16 -06:00
Terrance DeJesus 5a04aaf671 [Bug] Integrations-Pr Command (Elastic-Package Linting and Version Adjustments) (#2054)
* started solution for integrations-pr bug

* Update devtools.py

* Update detection_rules/devtools.py

* Update detection_rules/devtools.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2022-08-24 14:01:30 -04:00
github-actions[bot] 6ff7d2284d Lock versions for releases: 7.16,8.0,8.1,8.2,8.3,8.4 (#2261)
* Locked versions for releases: 7.16,8.0,8.1,8.2,8.3,8.4

* adjusting version lock file to increase current version by 100

Co-authored-by: terrancedejesus <terrancedejesus@users.noreply.github.com>
Co-authored-by: Terrance DeJesus <terrance.dejesus@elastic.co>
2022-08-24 13:26:35 -04:00
github-actions[bot] cb2ca45d56 Locked versions for releases: 7.16,8.0,8.1,8.2,8.3,8.4 (#2236)
Co-authored-by: terrancedejesus <terrancedejesus@users.noreply.github.com>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
2022-08-10 09:18:59 -04:00
Terrance DeJesus 2a3b584433 Prep for 8.5 branch (#2220)
* adding first commit

* renamed branch

* adjusted packages, stack schema and updated schemas

* updated integrations manifest

* adjusted comments to be a little more organized

* adjusted stack-schema-map

* refreshed ecs and beats schema, adjusted stack schema map accordingly
2022-08-09 17:14:42 -04:00
Mika Ayenson 89cdae87c5 only add related_integration if on the correct stack (#2234) 2022-08-08 18:41:56 -04:00
Mika Ayenson 7d973a3b07 add new field related_integrations to the post build (#2060)
* add new field `related_integrations` to the post build

* add exception for endpoint `integration`

* Skip rules without related integrations

* lint

* refactor related_integrations to TOMLRuleContents class

* update to reflect required_fields updates

* add todo

* add new line for linting

* related_integrations updates, get_packaged_integrations returns list of dictionaries, started work on integrations py

* build_integrations_manifest command completed

* initial test completed for post-building related_integrations

* removed get_integration_manifest method from rule, removed global integrations path

* moved integration related methods to integrations.py and fixed flake issues

* adjustments for PipedQuery from eql sequence rules and packages with no integration

* adjusted github client import for integrations.py

* Update detection_rules/devtools.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/devtools.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* added integration manifest schema, made adjustments

* Update detection_rules/integrations.py

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* removed get_integrations_package to consolidate code

* removed type list return

* adjusted import flake errors

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* adjusted indentation error

* adjusted rule.get_packaged_integrations to account for kql.ast.OrExpr if event.dataset is not set

* Update detection_rules/devtools.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/devtools.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* adjusted find_least_compatible_version in integrations.py

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* fixed flake issues

* adjusted get_packaged_integrations

* iterate the ast for literal event.dataset values

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* made small adjustments to address errors during build manifests command

* addressing integrations.find_least_compatible method to return None instead of raise error only

* Update detection_rules/integrations.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
Co-authored-by: Terrance DeJesus <terrance.dejesus@elastic.co>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
2022-08-08 13:44:36 -04:00
Samirbous b15f0de9a4 [Rules Tuning] Diverse Windows Rules - FPs reduction (#2213)
* [Rules Tuning] 7 diverse Windows rules

Excluding FP patterns while avoiding breaking compat with winlogbeat and 4688 events lack of codesign metadata.

* Update initial_access_suspicious_ms_exchange_process.toml

* Update privilege_escalation_persistence_phantom_dll.toml

* Update execution_psexec_lateral_movement_command.toml

* Update persistence_remote_password_reset.toml

* Update non-ecs-schema.json

* Update persistence_remote_password_reset.toml

* Update non-ecs-schema.json

* Update discovery_privileged_localgroup_membership.toml
2022-08-02 18:37:07 +02:00
Isai c1486407aa [New Rule] Kubernetes Pod Created with Sensitive hostPath Volume (#2094)
* [New Rule] Kubernetes Pod Created with Sensitive hostPath Volume

created new rule toml and updated non-ecs-schema with k8s fields

* Update rules/integrations/kubernetes/privilege_escalation_pod_created_with_sensitive_hospath_volume.toml

Co-authored-by: Colson Wilhoit <48036388+DefSecSentinel@users.noreply.github.com>
Co-authored-by: Jonhnathan <jonhnathancesar@gmail.com>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
2022-07-28 13:09:26 -04:00
Terrance DeJesus e8c39d19a7 [Rule Tuning] Missing MITRE ATT&CK Mappings (#2073)
* initial commit with eggshell mitre mapping added

* adding updated rules

* [Rule Tuning] MITRE for GCP rules

I've added Mitre references for the 4 GCP rules missing. Changed 3 of the rules from "Impact" to "Defense Evasion" based on the technique used and it's matched tactic.

* [Rule Tuning] Endgame Rule name updates for Mitre

Updated Endgame rule names for those with Mitre tactics to match the tactics.

* Update rules/integrations/aws/persistence_redshift_instance_creation.toml

Co-authored-by: Jonhnathan <jonhnathancesar@gmail.com>

* Update rules/integrations/aws/exfiltration_rds_snapshot_restored.toml

Co-authored-by: Jonhnathan <jonhnathancesar@gmail.com>

* adding 10 updated rules for google_workspace, ml and o365

* adding 22 rule updates for mitre att&ck mappings

* adding 24 rule updates related mainly to ML rules

* adding 3 rules related to detection via ML

* adding adjustments

* adding adjustments with solutions to recent pytest errors

* removed tabs from tags

* adjusted mappings and added techniques

* adjusted endgame rule mappings per review

* adjusted names to match different tactics

* added execution and defense evasion tag

* adjustments to address errors from merging with main

* added newlines to rules missing them at the end of the file

Co-authored-by: imays11 <59296946+imays11@users.noreply.github.com>
Co-authored-by: Jonhnathan <jonhnathancesar@gmail.com>
2022-07-22 14:30:34 -04:00
Samirbous d312f49117 [New Rule] Suspicious HTML File Creation (#2068)
* [New Rule] Suspicious HTML File Creation

* Update initial_access_evasion_suspicious_htm_file_creation.toml

* Update non-ecs-schema.json

* Update initial_access_evasion_suspicious_htm_file_creation.toml

* Update rules/windows/initial_access_evasion_suspicious_htm_file_creation.toml

Co-authored-by: Jonhnathan <jonhnathancesar@gmail.com>
2022-07-22 16:21:53 +02:00
Mika Ayenson a52751494e 2058 add setup field to metadata (#2061)
* Convert config header to setup in note field
* Parse note field into separate setup and note field with marko gfm
* only validate and parse note on elastic authored rules and add CLI description for new DR_BYPASS_NOTE_VALIDATION_AND_PARSE environment variable

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
2022-07-18 15:41:32 -04:00
Mika Ayenson c76a397969 Add new required_fields as a build-time restricted field (#2059)
* Add new `require_field` restricted field
* validate new fields against BaseRuleData schema and global constant

Co-authored-by: Terrance DeJesus <terrance.dejesus@elastic.co>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
2022-07-06 11:49:44 -04:00
Justin Ibarra cc01d3fb1a Add support for restricted fields (#2053)
* Add support for restricted fields (fields valid only in min/max stack versions)
* add test to ensure rule backports wont exceed min compat
2022-06-27 10:02:15 -05:00
Mika Ayenson 4ef1a1a627 Update cli documentation for search-alerts (#2051)
* Add cli documentation for search-alerts and table fields
2022-06-24 09:58:58 -04:00
Mika Ayenson 4fdd978183 test automatically prevent future merges when a backport fails (#1909)
automatically prevent future merges when a backport fails
2022-06-23 14:59:25 -04:00
github-actions[bot] fd9c9f8abf Locked versions for releases: 7.16,8.0,8.1,8.2,8.3 (#2041)
Co-authored-by: terrancedejesus <terrancedejesus@users.noreply.github.com>
2022-06-17 11:44:07 -04:00
Isai 63fda01fdd [New Rule] Kubernetes execution_user_exec_to_pod (#1979)
* Create execution_user_exec_to_pod.toml

* Update execution_user_exec_to_pod.toml

* Update rules/integrations/kubernetes/execution_user_exec_to_pod.toml

* Update non-ecs-schema.json

* Update execution_user_exec_to_pod.toml

* Update rules/integrations/kubernetes/execution_user_exec_to_pod.toml

Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>

* Update execution_user_exec_to_pod.toml

* Update execution_user_exec_to_pod.toml

* Update execution_user_exec_to_pod.toml

* toml-linted file and add to false positive

toml-linted the file and added to the false positive description

* Create notepad.sct

Added this back into the repo, deleted by mistake.

* added min_stack_version based on integration

min stack version determined by integration support of necessary fields

Co-authored-by: Jonhnathan <jonhnathancesar@gmail.com>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
Co-authored-by: Colson Wilhoit <48036388+DefSecSentinel@users.noreply.github.com>
2022-06-09 17:52:45 -04:00
Justin Ibarra 744f56d98e [Bug] resolves bug in Rule version methods (#2021)
* [Bug] resolves bug in Rule version methods

* comment out unused code with notes
2022-06-07 15:40:46 -08:00
Justin Ibarra e850f39526 [Bug] Fix test_matrix_to_lock_version_defaults test (#2014) 2022-06-02 16:34:54 -08:00
Justin Ibarra f57950a3c9 Collapse unsupported previous version entries (#2013)
* Collapse unsupported previous version entries
* drop the last entry in the matrix test
2022-06-02 15:18:12 -08:00
Terrance DeJesus 35b1a69ff5 Prep for Creation of 8.4 Branch (#2001)
* prepping for 8.4 branch

* adjusted schemas init file

* adjusted target matrix to only backport to 7.16, updated api schemas

* adjusted the lock-versions workflow to account for 7.16 and up support only

* Add test for version lock to schema map correlation

* decouple from static 7.13 references

* keep patch version for lock

* Update detection_rules/etc/packages.yml

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

Co-authored-by: Jonhnathan <jonhnathancesar@gmail.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
2022-06-02 14:59:18 -04:00
Justin Ibarra 0428e161a8 Refresh ECS/beats schemas up to 8.2 (#1995) 2022-05-25 11:51:43 -08:00
Mika Ayenson e1266a6fd3 Skip previous validation on pre/post load/dump (#1942)
* Build out the dataclasses for a base entry and version lock explicitly
* Ensure previous field does not have a nested previous
* Test validation on version lock for previous fields.
2022-05-25 13:34:03 -04:00
Samirbous 19ff825a91 [New rule] Remote Computer Account DnsHostName Update (#1962)
* [New rule] Remote Computer Account DnsHostName Update

Identifies remote update to a computer account DnsHostName attribute, if the new value is set a valid domain controller DNS hostname and the subject computer name is not a domain controller then it's high likely a preparation step to exploit CVE-2022-26923 in an attempt to elevate privileges from a standard domain user to domain admin privileges :

* added MS ref url

* Update rules/windows/privilege_escalation_suspicious_dnshostname_update.toml

Co-authored-by: Jonhnathan <jonhnathancesar@gmail.com>

* Update rules/windows/privilege_escalation_suspicious_dnshostname_update.toml

Co-authored-by: Jonhnathan <jonhnathancesar@gmail.com>

Co-authored-by: Jonhnathan <jonhnathancesar@gmail.com>
2022-05-11 19:40:34 +02:00
Justin Ibarra 8168551c59 Manually reconciled versions from forked rule package generation bug (#1950) 2022-05-04 10:04:10 -08:00
Justin Ibarra 22679e16d2 Add delta command to determine changes to endpoint rules between tags (#1943)
* update git tag loader to be compatible with lock validation
* add diff command
* default to query for missing rules
2022-05-03 12:30:11 -08:00
Mika Ayenson 6219fc06b9 Move etc under detection_rules (#1885)
* Move etc directory under detection_rules
* Prepend original `etc` path with `detection_rules`
* Update docstrings in util and CODEOWNERS
* Add resiliency to tags to account for the old directory structure
* Bug fix: remove unused param caused by commit 6ed1a39efe

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
2022-05-02 10:11:21 -04:00
Justin Ibarra c803160e4f Validate version lock and deprecation files on load and save (#1884)
* Validate version lock and deprecation files on load and save
* add missing types for previous lock entries
* bump marshmallow_dataclass
2022-04-26 22:17:20 -08:00
Justin Ibarra b3e789a202 Remove deprecated elasticsearch parameter (#1913) 2022-04-12 12:06:11 -08:00
Justin Ibarra ad99c6b489 Update elasticsearch dependency to 8.1 (#1911) 2022-04-06 11:52:22 -08:00
Justin Ibarra 6bdfddac8e Expand timestamp override tests (#1907)
* Expand timestamp_override tests
* removed timestamp_override from eql sequence rules
* add config entry for eql rules with beats index and t_o
* add timestamp_override to missing fields
2022-04-01 15:27:08 -08:00
Terrance DeJesus 648daf1237 Prep for Creation of 8.3 Branch (#1906)
* updating with changes for 8.3 prep
* adding updates
* adjusted version in packages.yml
2022-04-01 13:33:18 -08:00
Terrance DeJesus e72031a71a added comprehensive timeline template definitions (#1905) 2022-04-01 08:51:54 -08:00
Justin Ibarra 5214209f8d reset evasion rules (#1902) 2022-03-29 15:47:48 -08:00
Justin Ibarra 17ef6c558c [Bug] Fix bug in version_lock.py (#1880) 2022-03-24 15:41:16 -08:00
Justin Ibarra 11ec9c230e Prevent changes to rule type for locked rules (#1855)
* add rule type to the rule lock_info
* add check in VersionLock; add type to version.lock
* print changes only on save
2022-03-24 11:56:27 -08:00
Justin Ibarra f4c94af994 [Bug] Version bump with previous (#1870)
* save changes to top level for route C; verbose prints
* update top level on forked rule without overriding min_stack_version
* add check to ensure previous version !> current
2022-03-24 11:12:06 -08:00
Mika Ayenson 1f015ebe85 1554 update eql schemas to fail validation on text fields (#1866)
* Ensure kql2eql conversion doesnt support `text` fields

* Add unit test cases for`text` not supported in eql

* test `field not recognized` in the rule_validator and output a verbose message.

* use elasticsearch_type_family to lookup text mappings

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
2022-03-23 16:22:26 -04:00
Mika Ayenson 84b7ce6582 update beats master branch ref to main (#1853)
* update beats master branch ref to main

* update filename of master beat schema to main

* delete old main beats schema

* rebuilt main beats archive
2022-03-18 10:06:34 -04:00
Justin Ibarra 6653acb21c [Github Workflows] Only generate navigator files on push to main (#1814)
* [Github Workflows] Only generate navigator files on push to main

* fix workflow logic syntax
2022-03-04 09:55:11 -09:00
Justin Ibarra bb105a3c43 Replace * in navigator filenames (#1813) 2022-03-04 08:45:55 -09:00
Justin Ibarra 254b4eb23f Generate ATT&CK navigator layer files and links (#1787)
* Generate attack layer files and build with package
* add update-navigator-gists command
* add workflow to update navigator gists on pushes to main
* Add coverage readme
* fix keys for links
* update navigator layer names
* purge gist files prior to update; add badge
* Update how the navigator links are displayed
* moved navigator code to dedicated and refactored to dataclasses
* convert gist links to permalink versions
* alphabetize; catch 404 for gist update
2022-03-04 08:20:44 -09:00
Justin Ibarra a5eb02ac28 Refresh ATT&CK to v10.1 (#1791) 2022-02-24 16:37:23 -09:00
Justin Ibarra d373db7659 Ensure github module is installed before running PR commands (#1777)
* Ensure github module is installed before running PR commands

* move go and elastic-package assertions to top of command

* update error msg for missing pkg

* remove redundant github assertion

Co-authored-by: Colson Wilhoit <48036388+DefSecSentinel@users.noreply.github.com>
2022-02-24 14:49:01 -09:00
Mika Ayenson 0aeb7399d4 [Bug] Fix toml-lint ordering of Mitre metadata #1249 (#1774)
* Order the MITRE metadata by recursively sorting the rule object before writing.

* Refactor order_rule into the rule_formatter module.

* sort test_toml.json according to rule_formatter spec

* rename var to obj since this will traverse all data in the rule

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
2022-02-22 13:57:49 -05:00
Colson Wilhoit e0dda91f26 Prep for creation of 8.2 branch (#1762) 2022-02-08 18:43:55 -09:00
Justin Ibarra 2828633919 [Bug] Fix AttributeError in RuleCollection dupe check (#1747) 2022-01-31 15:57:46 -09:00