Commit Graph

44 Commits

Author SHA1 Message Date
Ross Wolf f6839e98d1 Simplify version locking code and fix 7.13.0 lock (#1295)
* Update version lock overwrite command
* Fix tooling and restore old version lock
* Lint fix
* Fix tests
* Remove dead code
* Filter to prod+deprecated rules
* Cast set -> list
* Store deprecation info
* Add correct version.lock.json (finally)
* Fix "stack_version" typo
* Remove stack_version
* Back out main.py changes

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
2021-06-16 18:02:47 -06:00
Justin Ibarra e46f5e96d3 Fix create-rule bug (#1246) 2021-06-01 08:31:36 -08:00
Ross Wolf eb40c52c7c Port historical schemas to jsonschema (#1084)
* Port historical schemas to jsonschema
* Add marshmallow-json dependency
* Mark etc/api_schemas as binary
* Remove gitattributes attempt
* Lint fix
* Apply PR feedback
* Additional PR feedback
* Extract stack version from packages.yml
* Fix the backport schemas
* Cache the schema reads
* Add migration for #1167
* Make a separate 'migration not found' error
2021-05-13 14:27:32 -06:00
Justin Ibarra 1fb0b6726e Fix rule filenames during packaging (#1158) 2021-05-05 11:27:04 -08:00
Justin Ibarra 3d7f5d73a4 Allow ML rules to accept a single or array of job IDs (#1167) 2021-05-05 11:12:12 -08:00
Andrew Pease 92eaa5b18a [New Rule] Threat intel indicator match rule (#1133) 2021-04-26 07:07:04 -05:00
Justin Ibarra cabe9239c0 Add threat_match rule type (#1138) 2021-04-22 09:03:57 -08:00
Ross Wolf 8789dd7c90 Separate out query validation from the class hierarchy (#1136)
* Separate out query validation from the class hierarchy
* Rename to *RuleData for consistency
* Apply suggestions from code review
* Fix lint error
2021-04-21 14:55:26 -06:00
Justin Ibarra e656a984b3 Update threshold rule schema to disallow empty field string (#1099) 2021-04-15 16:22:45 -06:00
Ross Wolf 6ed1a39efe Add a RuleCollection object instead of a "loader" module (#1063)
* Add a RuleCollection object instead of a "loader" module
* Remove legacy loader code
* Remove more legacy loader
* Freeze the default collection
* Change RULE_LOADER default
* Rename to _toml_load_cache
* Use rglob magic
* Typo should've been a string
* Remove no longer needed glob import
* Fix pycharm import bad ordering
* Restore the detection_rules/schemas imports
* Put more imports back for a smaller diff
* Check cache in _deserialize_toml
* Add multi collection and single collection decorators
* Reorder RuleCollection methods
* Move filter method up
2021-04-05 14:23:37 -06:00
Ross Wolf 07be6b701d Change the asset .type field (#1075) 2021-04-05 10:50:58 -06:00
Ross Wolf 1e6e49a2cb Change the JSON schema for the security_rule Kibana asset (#1066)
* Change the JSON schema for the security_rule Kibana asset
* Use the asset type for the folder name
2021-03-30 13:31:02 -06:00
Ross Wolf 8ee1b2ffd4 Fix the version lock update code (#1064)
* Fix the version lock update code
* Add Rule.lock_info() method
2021-03-25 14:48:31 -06:00
Ross Wolf c0af222e7e Move Rule into a dataclass (#1029)
* WIP: Convert Rule to a dataclass
* Fix make release
* Lint fixes
* Remove dead code
* Fix lint and tests
* Use Python 3.8 in GitHub actions
* Update README to 3.8+
* Add Python 3.8 assertion
* Fix is_dirty property
* Remove incorrect pop from contents
* Add mixin with from_dict() and to_dict() methods
* Bypass validation for deprecated rules
* Fix rule_prompt
* Fix dict_hash usage
* Fix rule_event_search
* Switch to definitions.Date
* Fix toml-lint command, ignoring 'unneeded defaults'
* Moved severity Literal to definitions.Severity
* Remove BaseMarshmallowDataclass
* Fix lint and tests
* Add maturity to metadata for rule prompt loop
* Fix typo in devtools
* Use rule loader to load single rule in toml-lint
* Add Schema hint to __schema method
* Add MITREAttackURL definition
* Fix is_dirty to compare sha<-->sha
* Normalize the autoformatted rule output for API and toml-lint
* Make the package hash match
* Make the rule object mutable but not rule contents
* Restore the rules
2021-03-24 10:24:32 -06:00
Ross Wolf 5c2da0b5c4 Move Rule.build to cli_utils.rule_prompt (#1024)
* Move Rule.build to cli_utils.rule_prompt
* Fix build_threat_map_entry lint
* Fix license and add docstring
2021-03-09 16:37:53 -07:00
Justin Ibarra fc9dfde2c4 Generate an integrations package from a release (#983)
* Generate an integrations package files during a release build
2021-03-09 13:30:12 -09:00
Justin Ibarra 3fc34b86f2 Update License to Elastic v2 (#944) 2021-03-03 22:12:11 -09:00
Justin Ibarra b04218ec21 [CLI] Add repo option to kibana-diff command (#952) 2021-02-17 23:49:40 -09:00
Justin Ibarra d57394816f [FR] Index rules from repo to elasticsearch (#932)
* Add commands to generate index rules files or index them directly to elasticsearch
* files generated at package creation
* add readme explaining these index files
2021-02-10 10:37:26 -09:00
Justin Ibarra 56dc4745b5 Add export-rules command (#639)
* Add export-rule command to CLI
* add `export` method to packaging class
2021-02-08 20:43:16 -09:00
Justin Ibarra 6177458bd8 Add empty technique array to rules (#828)
* [Rule Tuning] Add empty arrays in place of tactic only threat mappings
* dynamically insert empty technique array in payload
* use replace_id as function parameter
2021-01-11 08:58:18 -09:00
Justin Ibarra c1a0438f45 [Rule Tuning] Update ATT&CK threat mappings to reflect changes (#706)
* replaced/removed all revoked/deprecated techniques
* tests will fail on revoked (changed) techniques
* tests will fail on deprecated techniques
* tests will fail when techniques are mapped to an invalid tactic
2020-12-18 12:46:16 -09:00
Ross Wolf 331d321648 Make threat.technique optional (#727) 2020-12-17 20:22:59 -09:00
Justin Ibarra e272800a5d Add ATT&CK sub-technique support to CLI (#614)
* Add Mitre sub-technique support to CLI
* Add subtechnique enum to schema
* Add test to prevent duplicative tactics in mapping
2020-12-08 21:56:55 -09:00
Justin Ibarra 366e5002e1 [FR] Add experimental ML DGA CLI support (#361)
* Add DGA model commands
* Add upload/delete ML job command
* Add DGA release management commands
* Add Manifest handling
* Add GithubClient object
2020-12-01 22:25:33 -09:00
Justin Ibarra 97ee8cc9ac Refresh beats and ecs schemas and default to use latest to validate (#570)
* Refresh beats and ecs schemas and default to use latest to validate
* remove incorrect ecs_version from zoom rule
* remove stale ecs_version from rules
2020-12-01 13:24:20 -09:00
Justin Ibarra a575cf9ff3 [Rule Tuning] Use cidrMatch for eql rules checking multiple IPs (#431) 2020-10-29 11:06:24 -08:00
Justin Ibarra 0a992d716a [Rule Tuning] Update EQL rules for 7.10 (#399)
* update syntax to reflect eql changes
* use more case-insensitivity
* comment out missing fields for winlogbeat compatibility
2020-10-21 12:35:18 -08:00
Justin Ibarra bf202b6b6c [New Rule] Initial converted EQL rules (#304)
* 18 converted eql rules (not all prod)
2020-09-30 21:40:55 -08:00
Justin Ibarra 7c1e9c1ed5 Update package summary extras produced during package generation (#341)
* update summary.txt
* add summary.xlsx
* add changelog entry autogeneration
2020-09-30 14:43:45 -08:00
Justin Ibarra a212008f8c [Rule Tuning] Remove event.module from rules for compatibility with agent integrations (#342) 2020-09-30 09:41:33 -08:00
Justin Ibarra 065bcd8018 Refresh ATT&CK data to v7.2 and expand threat validation (#330)
* refresh to latest ATT&CK 7.2
* add new unit test to further validate threat mappings
* updated threat mappings in rules to reflect changes
* new func to download and refresh mitre data based on version
2020-09-23 22:03:29 -08:00
Justin Ibarra 6ad3344af3 Collect unique query fields per rule (#296) 2020-09-23 14:36:34 -08:00
Ross Wolf 9d22970e21 Add EQL rules and schema validation (#297)
* Add EQL rules and schema validation
* Lint nitpick
* Rename get_schema_from_eql
* Add EQL default language
* Rename parsed_kql to parsed_query
* Fix parsed_kql method call in loader
* Autopopulate dependent values
2020-09-16 08:36:48 -06:00
Justin Ibarra 6b7ea7e66c Fix kibana-diff command (#198) 2020-09-02 12:19:17 -05:00
Justin Ibarra 8f5ddbb121 Add better CLI support for handling Kibana exported rules (#83) 2020-07-27 23:31:19 -05:00
Ross Wolf d15da0ada1 Add versioned schemas with a downgrade path (#84)
* Add versioned schemas with a downgrade path
* Remove and move unused variables
* Add missing license
* Skip NotField for output_index
* Add strip_additional_properties for kibana import
* Remove stray comment
* Apply suggestions from code review

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
2020-07-23 11:39:35 -06:00
Ross Wolf 16fb306254 Add command to upload to kibana (#58)
* Add upload command to kibana
* Restore skipped fields
* Change prefix to DR_
* Add note to manage_versions call
* Reorder requirements.txt to trigger build
2020-07-20 15:58:28 -06:00
Justin Ibarra 1cfb8f92bb Windows DNS server vulnerability (CVE-2020-1350) rules (#69) 2020-07-17 14:32:52 -05:00
Justin Ibarra 7647699e2b Add support for threshold rules (#65) 2020-07-16 19:06:34 -05:00
Justin Ibarra 916917a619 Update rule.py 2020-07-15 09:40:07 -05:00
Ross Wolf db4f50d4b8 Improve the validation and testing time (#61)
* Improve the validation and testing time
* Lint fix
* Cache schema validation
2020-07-15 08:05:55 -06:00
Ross Wolf e2d97b0a74 Remove unreachable and legacy code
Co-Authored-By: Justin Ibarra <brokensound77@users.noreply.github.com>
2020-06-30 10:12:23 -06:00
Ross Wolf 3b305d3003 Add rule loader and dependencies
Co-Authored-By: Justin Ibarra <brokensound77@users.noreply.github.com>
2020-06-29 23:17:42 -06:00