Commit Graph

78 Commits

Author SHA1 Message Date
Ross Wolf 8ee1b2ffd4 Fix the version lock update code (#1064)
* Fix the version lock update code
* Add Rule.lock_info() method
2021-03-25 14:48:31 -06:00
Ross Wolf c0af222e7e Move Rule into a dataclass (#1029)
* WIP: Convert Rule to a dataclass
* Fix make release
* Lint fixes
* Remove dead code
* Fix lint and tests
* Use Python 3.8 in GitHub actions
* Update README to 3.8+
* Add Python 3.8 assertion
* Fix is_dirty property
* Remove incorrect pop from contents
* Add mixin with from_dict() and to_dict() methods
* Bypass validation for deprecated rules
* Fix rule_prompt
* Fix dict_hash usage
* Fix rule_event_search
* Switch to definitions.Date
* Fix toml-lint command, ignoring 'unneeded defaults'
* Moved severity Literal to definitions.Severity
* Remove BaseMarshmallowDataclass
* Fix lint and tests
* Add maturity to metadata for rule prompt loop
* Fix typo in devtools
* Use rule loader to load single rule in toml-lint
* Add Schema hint to __schema method
* Add MITREAttackURL definition
* Fix is_dirty to compare sha<-->sha
* Normalize the autoformatted rule output for API and toml-lint
* Make the package hash match
* Make the rule object mutable but not rule contents
* Restore the rules
2021-03-24 10:24:32 -06:00
Ross Wolf 6963c5a445 Change asset type to security_rule (#1054)
* Change asset type to security_rule
* Add notice.txt
2021-03-19 08:55:02 -06:00
Justin Ibarra d4cc4432ce Add tests to ensure rules are properly deprecated (#1050)
* Add tests to ensure rules are properly deprecated
* add deprecate-rule command
2021-03-16 21:31:33 -08:00
Ross Wolf 93f8f2dd94 Change asset type for integration to security-rule (#1048) 2021-03-16 16:05:30 -06:00
Ross Wolf 5c2da0b5c4 Move Rule.build to cli_utils.rule_prompt (#1024)
* Move Rule.build to cli_utils.rule_prompt
* Fix build_threat_map_entry lint
* Fix license and add docstring
2021-03-09 16:37:53 -07:00
Justin Ibarra fc9dfde2c4 Generate an integrations package from a release (#983)
* Generate an integrations package files during a release build
2021-03-09 13:30:12 -09:00
Justin Ibarra 0e0b2ea1a4 Update schema for threshold rule type for 7.12 (#976)
* Update schema for threshold rule type for 7.12
* add downgrade function to drop new fields
* update existing threshold rules
2021-03-05 14:35:50 -09:00
Justin Ibarra 0ef7d87b34 [Rule Tuning] Fix inconsistent rule indexes (#974)
* [Rule Tuning] Fix inconsistent rule indexes
* cleaned up tests that load rules to leverage setUpClass
2021-03-05 11:16:02 -09:00
Justin Ibarra 3fc34b86f2 Update License to Elastic v2 (#944) 2021-03-03 22:12:11 -09:00
Justin Ibarra b04218ec21 [CLI] Add repo option to kibana-diff command (#952) 2021-02-17 23:49:40 -09:00
Justin Ibarra d57394816f [FR] Index rules from repo to elasticsearch (#932)
* Add commands to generate index rules files or index them directly to elasticsearch
* files generated at package creation
* add readme explaining these index files
2021-02-10 10:37:26 -09:00
Justin Ibarra b8116a5b77 Add GitHub PR rule loader (#670)
* add load_gh_pr_rules function
* add dev package-stats command
* add dev search-rule-prs command, which extends the same functionality in rule-search to rules in PR
2021-02-08 21:35:44 -09:00
Justin Ibarra 56dc4745b5 Add export-rules command (#639)
* Add export-rule command to CLI
* add `export` method to packaging class
2021-02-08 20:43:16 -09:00
brokensound77 bf32dec5a4 Merge remote-tracking branch 'upstream/main' into mergeback/7.11-to-main
# Conflicts:
#	rules/linux/defense_evasion_deletion_of_bash_command_line_history.toml
2021-01-28 10:41:39 -09:00
Ross Wolf 1708ea3252 Loosen query DSL filter schema validation (#895) 2021-01-20 12:21:46 -07:00
Justin Ibarra 6177458bd8 Add empty technique array to rules (#828)
* [Rule Tuning] Add empty arrays in place of tactic only threat mappings
* dynamically insert empty technique array in payload
* use replace_id as function parameter
2021-01-11 08:58:18 -09:00
Justin Ibarra 67413cee47 Update ML-DGA docs (#750) 2020-12-21 16:25:24 -09:00
Justin Ibarra 992eabd6dc update incomplete bug fix from 736 for 7.11 -> 7.10 downgrade logic 2020-12-18 22:04:19 -09:00
Justin Ibarra 5561738f28 update incomplete bug fix from 736 for 7.11 -> 7.10 downgrade logic 2020-12-18 22:01:06 -09:00
Justin Ibarra 425e0ddf64 Add flattened subtechniques to rule-search (#739) 2020-12-18 14:21:37 -09:00
Justin Ibarra c1a0438f45 [Rule Tuning] Update ATT&CK threat mappings to reflect changes (#706)
* replaced/removed all revoked/deprecated techniques
* tests will fail on revoked (changed) techniques
* tests will fail on deprecated techniques
* tests will fail when techniques are mapped to an invalid tactic
2020-12-18 12:46:16 -09:00
Ross Wolf 7dcb666d81 Fix 7.11 -> 7.10 ATT&CK downgrade logic for optional techiques (#736) 2020-12-18 09:28:05 -07:00
Ross Wolf 331d321648 Make threat.technique optional (#727) 2020-12-17 20:22:59 -09:00
Justin Ibarra b6aa6c6548 Auth to Kibana connector using an existing cookie (#711) 2020-12-15 13:20:46 -07:00
Justin Ibarra 7c2abc68d7 [Docs] Update ML_DGA.md (#707) 2020-12-09 13:06:35 -09:00
Justin Ibarra e272800a5d Add ATT&CK sub-technique support to CLI (#614)
* Add Mitre sub-technique support to CLI
* Add subtechnique enum to schema
* Add test to prevent duplicative tactics in mapping
2020-12-08 21:56:55 -09:00
Justin Ibarra 0ed1e1df71 Add support to validate against dev ECS and beats schemas (#691) 2020-12-08 13:29:56 -09:00
Justin Ibarra 200fbe939e [Bug] Allow duplicative queries across different rule types (#704) 2020-12-08 13:16:59 -09:00
Ross Wolf 8c92ae7348 Add ATT&CK subtechniques to the schema (#337)
* Add ATT&CK subtechniques to the schema
* Switch subtechniques to the 7.11 schema
* Make technique still required
* Lint fixes
* Cleanup EQL constant
* Trim more cruft
* Restore EQL for 710

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
2020-12-08 14:57:30 -07:00
Brent Murphy 6a296c64c5 [New Rule] Microsoft 365 Exchange DKIM Signing Configuration Disabled (#578)
* [New Rule] O365 Exchange DKIM Signing Configuration Disabled

* rebrand to m365

* still req non ecs schema

* Remove the ECS override

* Update _flatten_schema logic

* Allow fields with * in the path

* Allow explicit fields to overwrite implicit * fields

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2020-12-08 16:38:00 -05:00
Justin Ibarra 366e5002e1 [FR] Add experimental ML DGA CLI support (#361)
* Add DGA model commands
* Add upload/delete ML job command
* Add DGA release management commands
* Add Manifest handling
* Add GithubClient object
2020-12-01 22:25:33 -09:00
Justin Ibarra 97ee8cc9ac Refresh beats and ecs schemas and default to use latest to validate (#570)
* Refresh beats and ecs schemas and default to use latest to validate
* remove incorrect ecs_version from zoom rule
* remove stale ecs_version from rules
2020-12-01 13:24:20 -09:00
Justin Ibarra ad4a2ef0eb Add test commands to search and survey rule hits (#485) 2020-11-17 13:08:00 -09:00
brokensound77 75d37e9271 Merge remote-tracking branch 'upstream/main' into mergeback/7.10-to-main 2020-11-12 00:59:31 -09:00
Ross Wolf 8ca32f1423 Fix ClientError (NoneType) suffix 2020-11-09 11:08:36 -07:00
Justin Ibarra 3b597bdb72 fix auth args in get_es_client 2020-10-30 09:19:50 -08:00
Justin Ibarra 3827d01a65 fix bugs in es client retrieval 2020-10-29 21:20:49 -08:00
Justin Ibarra a575cf9ff3 [Rule Tuning] Use cidrMatch for eql rules checking multiple IPs (#431) 2020-10-29 11:06:24 -08:00
Ross Wolf 7da343e89f Fix kibana upload command (#425) 2020-10-28 10:16:36 -06:00
Ross Wolf a0a8d63baf Merge branch '7.10' into main 2020-10-28 09:40:15 -06:00
Brent Murphy 2e422f7159 [Rule Tuning] Minor Rule Tweaks for 7.10 (#400)
* Tweak Rules for 7.10

* Add endpoint index for packetbeat rules

* update unit test to account for Network tag as well

* update modified date, add endpoint tag

* use Host instead of Endpoint

* Update packaging.py

* add v back to changelog url

* Add "tag" comment to get_markdown_rule_info

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2020-10-22 09:07:04 -04:00
Justin Ibarra 0a992d716a [Rule Tuning] Update EQL rules for 7.10 (#399)
* update syntax to reflect eql changes
* use more case-insensitivity
* comment out missing fields for winlogbeat compatibility
2020-10-21 12:35:18 -08:00
Stijn Holzhauer 60b3d47efd Add kibana-upload --space option (#251)
Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
2020-10-08 12:21:54 -06:00
Justin Ibarra bd680a2bd4 Re-organize commands under more specific click groups (#356)
* Restructure commands under more specific click groups
* standardize CLI error handling
* add global debug options
* move es and kibana clients into their click groups
* move commands and groups to dedicated files 
* distinguish variable names for better env/config parsing
2020-10-07 12:15:33 -08:00
Justin Ibarra bf202b6b6c [New Rule] Initial converted EQL rules (#304)
* 18 converted eql rules (not all prod)
2020-09-30 21:40:55 -08:00
Justin Ibarra 7c1e9c1ed5 Update package summary extras produced during package generation (#341)
* update summary.txt
* add summary.xlsx
* add changelog entry autogeneration
2020-09-30 14:43:45 -08:00
Justin Ibarra a212008f8c [Rule Tuning] Remove event.module from rules for compatibility with agent integrations (#342) 2020-09-30 09:41:33 -08:00
shravaka fa12340ff0 [Bug fix] Add missing parenthesis for -kibana-url 2020-09-30 09:32:43 -06:00
Justin Ibarra 065bcd8018 Refresh ATT&CK data to v7.2 and expand threat validation (#330)
* refresh to latest ATT&CK 7.2
* add new unit test to further validate threat mappings
* updated threat mappings in rules to reflect changes
* new func to download and refresh mitre data based on version
2020-09-23 22:03:29 -08:00