Commit Graph

58 Commits

Author SHA1 Message Date
Terrance DeJesus 2a3b584433 Prep for 8.5 branch (#2220)
* adding first commit

* renamed branch

* adjusted packages, stack schema and updated schemas

* updated integrations manifest

* adjusted comments to be a little more organized

* adjusted stack-schema-map

* refreshed ecs and beats schema, adjusted stack schema map accordingly
2022-08-09 17:14:42 -04:00
Justin Ibarra cc01d3fb1a Add support for restricted fields (#2053)
* Add support for restricted fields (fields valid only in min/max stack versions)
* add test to ensure rule backports wont exceed min compat
2022-06-27 10:02:15 -05:00
Justin Ibarra 744f56d98e [Bug] resolves bug in Rule version methods (#2021)
* [Bug] resolves bug in Rule version methods

* comment out unused code with notes
2022-06-07 15:40:46 -08:00
Justin Ibarra e850f39526 [Bug] Fix test_matrix_to_lock_version_defaults test (#2014) 2022-06-02 16:34:54 -08:00
Terrance DeJesus 35b1a69ff5 Prep for Creation of 8.4 Branch (#2001)
* prepping for 8.4 branch

* adjusted schemas init file

* adjusted target matrix to only backport to 7.16, updated api schemas

* adjusted the lock-versions workflow to account for 7.16 and up support only

* Add test for version lock to schema map correlation

* decouple from static 7.13 references

* keep patch version for lock

* Update detection_rules/etc/packages.yml

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

Co-authored-by: Jonhnathan <jonhnathancesar@gmail.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
2022-06-02 14:59:18 -04:00
Justin Ibarra 0428e161a8 Refresh ECS/beats schemas up to 8.2 (#1995) 2022-05-25 11:51:43 -08:00
Justin Ibarra 22679e16d2 Add delta command to determine changes to endpoint rules between tags (#1943)
* update git tag loader to be compatible with lock validation
* add diff command
* default to query for missing rules
2022-05-03 12:30:11 -08:00
Justin Ibarra c803160e4f Validate version lock and deprecation files on load and save (#1884)
* Validate version lock and deprecation files on load and save
* add missing types for previous lock entries
* bump marshmallow_dataclass
2022-04-26 22:17:20 -08:00
Terrance DeJesus 648daf1237 Prep for Creation of 8.3 Branch (#1906)
* updating with changes for 8.3 prep
* adding updates
* adjusted version in packages.yml
2022-04-01 13:33:18 -08:00
Terrance DeJesus e72031a71a added comprehensive timeline template definitions (#1905) 2022-04-01 08:51:54 -08:00
Mika Ayenson 84b7ce6582 update beats master branch ref to main (#1853)
* update beats master branch ref to main

* update filename of master beat schema to main

* delete old main beats schema

* rebuilt main beats archive
2022-03-18 10:06:34 -04:00
Colson Wilhoit e0dda91f26 Prep for creation of 8.2 branch (#1762) 2022-02-08 18:43:55 -09:00
Justin Ibarra 2e78da5c9a Prepare for creation of 8.1 branch (#1700) 2022-01-25 18:11:59 -09:00
Justin Ibarra d753ecb8d8 Add pattern for "name" in rule schema (#1669) 2022-01-25 12:03:27 -09:00
Justin Ibarra 95d7e9b6f5 Prepare for creation of 7.16 release branch (#1611) 2021-11-15 09:39:34 -09:00
Justin Ibarra ab17dfcc28 [Bug] Tighten definitions validation patterns (#1396)
* [Bug] Anchor validation patterns
* Deprecate rule with invalid rule_id and duplicate as new one

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2021-10-26 10:26:20 -05:00
Ross Wolf 7b8b18cb20 Update main to point to 7.16 (#1457)
* Update main to point to 7.16
* Add 7.16 -> 7.15 migration
* Update stack-schema-map
* Update conditions.kibana.version
2021-08-26 14:23:55 -06:00
Justin Ibarra 2d517432e3 Bump package versions (#1418)
* Bump package versions
* Add 7.14 migration; use master schema map if one does not exist
* add test to ensure an entry exists in the stack-schema-map for the current package version

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2021-08-18 21:25:53 -08:00
Justin Ibarra 163d9e3864 Update cardinality field in schema for threshold rules (#1349)
* Make cardinality array in schema for threshold rules
* update master, 7.12, 7.13, and 7.14 schemas with cardinality fix
* fix 7.12 downgrade to handle cardinality as an array

* Add two new rules to detect agent spoofing


Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2021-07-21 08:32:54 -08:00
Justin Ibarra 781953a0a0 Add min_stack_version to rule metadata (#1173)
* Add min_stack_version to metadata of rule structure
* validate all "stack versions" between defined and current package
* Use master schemas if min_stack_version > current_package

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2021-06-30 13:26:27 -08:00
Ross Wolf e897a67604 Fix fleet package generation (#1296)
* Fix fleet package generation
* Add .lstrip()
* Lint fix
* Add newline
2021-06-17 06:16:09 -06:00
Ross Wolf 61e5b44c44 [Fleet] Update template and packaging code for fleet packages (#1280)
* Update template and packaging code for fleet packages
* Fix linting
2021-06-15 07:54:50 -06:00
Justin Ibarra 0ec8d67e78 Refactor experimental ML CLI and code (#1218)
* move github and ml to their own files
* refactor release and ml commands
* update ML readmes
* add unzip_to_dict function
* prompt for model ID in remove-model
* update experimental rule upload process
* update remove-scripts-pipelines to take multiple options

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
Co-authored-by: Apoorva <appujo@gmail.com>
2021-06-02 20:37:12 -08:00
Ross Wolf eb40c52c7c Port historical schemas to jsonschema (#1084)
* Port historical schemas to jsonschema
* Add marshmallow-json dependency
* Mark etc/api_schemas as binary
* Remove gitattributes attempt
* Lint fix
* Apply PR feedback
* Additional PR feedback
* Extract stack version from packages.yml
* Fix the backport schemas
* Cache the schema reads
* Add migration for #1167
* Make a separate 'migration not found' error
2021-05-13 14:27:32 -06:00
Andrew Pease 92eaa5b18a [New Rule] Threat intel indicator match rule (#1133) 2021-04-26 07:07:04 -05:00
Justin Ibarra cabe9239c0 Add threat_match rule type (#1138) 2021-04-22 09:03:57 -08:00
Ross Wolf 8789dd7c90 Separate out query validation from the class hierarchy (#1136)
* Separate out query validation from the class hierarchy
* Rename to *RuleData for consistency
* Apply suggestions from code review
* Fix lint error
2021-04-21 14:55:26 -06:00
Justin Ibarra e656a984b3 Update threshold rule schema to disallow empty field string (#1099) 2021-04-15 16:22:45 -06:00
Ross Wolf 791c911b9e Merge branch '7.12' into main 2021-04-15 16:17:59 -06:00
Ross Wolf 5669988e0b Remove unnecessary required=False check 2021-04-15 16:16:42 -06:00
Justin Ibarra 462fab3ff8 Update threshold rule schema to disallow empty field string (#1098)
* Update threshold rule schema to disallow empty field string
* lock versions for rule changes
2021-04-14 04:56:38 -08:00
Ross Wolf 07be6b701d Change the asset .type field (#1075) 2021-04-05 10:50:58 -06:00
Ross Wolf 1e6e49a2cb Change the JSON schema for the security_rule Kibana asset (#1066)
* Change the JSON schema for the security_rule Kibana asset
* Use the asset type for the folder name
2021-03-30 13:31:02 -06:00
Ross Wolf c0af222e7e Move Rule into a dataclass (#1029)
* WIP: Convert Rule to a dataclass
* Fix make release
* Lint fixes
* Remove dead code
* Fix lint and tests
* Use Python 3.8 in GitHub actions
* Update README to 3.8+
* Add Python 3.8 assertion
* Fix is_dirty property
* Remove incorrect pop from contents
* Add mixin with from_dict() and to_dict() methods
* Bypass validation for deprecated rules
* Fix rule_prompt
* Fix dict_hash usage
* Fix rule_event_search
* Switch to definitions.Date
* Fix toml-lint command, ignoring 'unneeded defaults'
* Moved severity Literal to definitions.Severity
* Remove BaseMarshmallowDataclass
* Fix lint and tests
* Add maturity to metadata for rule prompt loop
* Fix typo in devtools
* Use rule loader to load single rule in toml-lint
* Add Schema hint to __schema method
* Add MITREAttackURL definition
* Fix is_dirty to compare sha<-->sha
* Normalize the autoformatted rule output for API and toml-lint
* Make the package hash match
* Make the rule object mutable but not rule contents
* Restore the rules
2021-03-24 10:24:32 -06:00
Justin Ibarra d4cc4432ce Add tests to ensure rules are properly deprecated (#1050)
* Add tests to ensure rules are properly deprecated
* add deprecate-rule command
2021-03-16 21:31:33 -08:00
Justin Ibarra fc9dfde2c4 Generate an integrations package from a release (#983)
* Generate an integrations package files during a release build
2021-03-09 13:30:12 -09:00
Justin Ibarra 0e0b2ea1a4 Update schema for threshold rule type for 7.12 (#976)
* Update schema for threshold rule type for 7.12
* add downgrade function to drop new fields
* update existing threshold rules
2021-03-05 14:35:50 -09:00
Justin Ibarra 3fc34b86f2 Update License to Elastic v2 (#944) 2021-03-03 22:12:11 -09:00
Justin Ibarra b8116a5b77 Add GitHub PR rule loader (#670)
* add load_gh_pr_rules function
* add dev package-stats command
* add dev search-rule-prs command, which extends the same functionality in rule-search to rules in PR
2021-02-08 21:35:44 -09:00
Justin Ibarra 56dc4745b5 Add export-rules command (#639)
* Add export-rule command to CLI
* add `export` method to packaging class
2021-02-08 20:43:16 -09:00
brokensound77 bf32dec5a4 Merge remote-tracking branch 'upstream/main' into mergeback/7.11-to-main
# Conflicts:
#	rules/linux/defense_evasion_deletion_of_bash_command_line_history.toml
2021-01-28 10:41:39 -09:00
Ross Wolf 1708ea3252 Loosen query DSL filter schema validation (#895) 2021-01-20 12:21:46 -07:00
Justin Ibarra 6177458bd8 Add empty technique array to rules (#828)
* [Rule Tuning] Add empty arrays in place of tactic only threat mappings
* dynamically insert empty technique array in payload
* use replace_id as function parameter
2021-01-11 08:58:18 -09:00
Justin Ibarra 992eabd6dc update incomplete bug fix from 736 for 7.11 -> 7.10 downgrade logic 2020-12-18 22:04:19 -09:00
Justin Ibarra 5561738f28 update incomplete bug fix from 736 for 7.11 -> 7.10 downgrade logic 2020-12-18 22:01:06 -09:00
Ross Wolf 7dcb666d81 Fix 7.11 -> 7.10 ATT&CK downgrade logic for optional techiques (#736) 2020-12-18 09:28:05 -07:00
Ross Wolf 331d321648 Make threat.technique optional (#727) 2020-12-17 20:22:59 -09:00
Justin Ibarra e272800a5d Add ATT&CK sub-technique support to CLI (#614)
* Add Mitre sub-technique support to CLI
* Add subtechnique enum to schema
* Add test to prevent duplicative tactics in mapping
2020-12-08 21:56:55 -09:00
Justin Ibarra 0ed1e1df71 Add support to validate against dev ECS and beats schemas (#691) 2020-12-08 13:29:56 -09:00
Ross Wolf 8c92ae7348 Add ATT&CK subtechniques to the schema (#337)
* Add ATT&CK subtechniques to the schema
* Switch subtechniques to the 7.11 schema
* Make technique still required
* Lint fixes
* Cleanup EQL constant
* Trim more cruft
* Restore EQL for 710

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
2020-12-08 14:57:30 -07:00