* add test that newly introduced build-time fields for a min_stack for applicable rules.
* account for rules without min_stack_version
* limit test to >= stack ver
* Convert config header to setup in note field
* Parse note field into separate setup and note field with marko gfm
* only validate and parse note on elastic authored rules and add CLI description for new DR_BYPASS_NOTE_VALIDATION_AND_PARSE environment variable
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
* prepping for 8.4 branch
* adjusted schemas init file
* adjusted target matrix to only backport to 7.16, updated api schemas
* adjusted the lock-versions workflow to account for 7.16 and up support only
* Add test for version lock to schema map correlation
* decouple from static 7.13 references
* keep patch version for lock
* Update detection_rules/etc/packages.yml
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
Co-authored-by: Jonhnathan <jonhnathancesar@gmail.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
* Build out the dataclasses for a base entry and version lock explicitly
* Ensure previous field does not have a nested previous
* Test validation on version lock for previous fields.
* Move etc directory under detection_rules
* Prepend original `etc` path with `detection_rules`
* Update docstrings in util and CODEOWNERS
* Add resiliency to tags to account for the old directory structure
* Bug fix: remove unused param caused by commit 6ed1a39efe
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
* Expand timestamp_override tests
* removed timestamp_override from eql sequence rules
* add config entry for eql rules with beats index and t_o
* add timestamp_override to missing fields
* Ensure kql2eql conversion doesnt support `text` fields
* Add unit test cases for`text` not supported in eql
* test `field not recognized` in the rule_validator and output a verbose message.
* use elasticsearch_type_family to lookup text mappings
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
* Add DeprecatedCollection to RuleCollection to bypass validation
* use DeprecatedRule properties in RuleCollection
* use RuleCollection filter for max/min filtering in Package
* Save the stack versions in the lock file
* Support tracking of multiple stacks in the lock
* Update the version locking logic
* Fix bugs and test lock file
* Restore version lock
* Fix lint errors
* Call both click.echo and verbose echo separately
* Change when the change_rules message is output
* Bump package versions
* Add 7.14 migration; use master schema map if one does not exist
* add test to ensure an entry exists in the stack-schema-map for the current package version
Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
* Add the following properties to EQLRuleData:
- max_span
- look_back
- interval_ratio
* Add the following tests:
- test_eql_lookback
- test_eql_interval_to_maxspan
Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
* Make cardinality array in schema for threshold rules
* update master, 7.12, 7.13, and 7.14 schemas with cardinality fix
* fix 7.12 downgrade to handle cardinality as an array
* Add two new rules to detect agent spoofing
Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
* Add min_stack_version to metadata of rule structure
* validate all "stack versions" between defined and current package
* Use master schemas if min_stack_version > current_package
Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
* Update version lock overwrite command
* Fix tooling and restore old version lock
* Lint fix
* Fix tests
* Remove dead code
* Filter to prod+deprecated rules
* Cast set -> list
* Store deprecation info
* Add correct version.lock.json (finally)
* Fix "stack_version" typo
* Remove stack_version
* Back out main.py changes
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
* Port historical schemas to jsonschema
* Add marshmallow-json dependency
* Mark etc/api_schemas as binary
* Remove gitattributes attempt
* Lint fix
* Apply PR feedback
* Additional PR feedback
* Extract stack version from packages.yml
* Fix the backport schemas
* Cache the schema reads
* Add migration for #1167
* Make a separate 'migration not found' error
* Add a RuleCollection object instead of a "loader" module
* Remove legacy loader code
* Remove more legacy loader
* Freeze the default collection
* Change RULE_LOADER default
* Rename to _toml_load_cache
* Use rglob magic
* Typo should've been a string
* Remove no longer needed glob import
* Fix pycharm import bad ordering
* Restore the detection_rules/schemas imports
* Put more imports back for a smaller diff
* Check cache in _deserialize_toml
* Add multi collection and single collection decorators
* Reorder RuleCollection methods
* Move filter method up
* WIP: Convert Rule to a dataclass
* Fix make release
* Lint fixes
* Remove dead code
* Fix lint and tests
* Use Python 3.8 in GitHub actions
* Update README to 3.8+
* Add Python 3.8 assertion
* Fix is_dirty property
* Remove incorrect pop from contents
* Add mixin with from_dict() and to_dict() methods
* Bypass validation for deprecated rules
* Fix rule_prompt
* Fix dict_hash usage
* Fix rule_event_search
* Switch to definitions.Date
* Fix toml-lint command, ignoring 'unneeded defaults'
* Moved severity Literal to definitions.Severity
* Remove BaseMarshmallowDataclass
* Fix lint and tests
* Add maturity to metadata for rule prompt loop
* Fix typo in devtools
* Use rule loader to load single rule in toml-lint
* Add Schema hint to __schema method
* Add MITREAttackURL definition
* Fix is_dirty to compare sha<-->sha
* Normalize the autoformatted rule output for API and toml-lint
* Make the package hash match
* Make the rule object mutable but not rule contents
* Restore the rules
Mika Ayenson