Commit Graph

114 Commits

Author SHA1 Message Date
Ross Wolf 92937a1ad1 [CI] Fix kibana PR command again (#1386) 2021-07-27 16:29:50 -06:00
Ross Wolf 64977b01bd Fix kibana_pr for click.Context (#1385) 2021-07-27 16:03:28 -06:00
Justin Ibarra 7759fa2500 Ensure EQL rules with maxspan have a long enough lookback window (#1361)
* Add the following properties to EQLRuleData:
   - max_span
   - look_back
   - interval_ratio

* Add the following tests:
   - test_eql_lookback
   - test_eql_interval_to_maxspan

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2021-07-22 13:53:13 -08:00
Ross Wolf 5ba1c26cf1 Fix metadata.extended (#1377) 2021-07-22 10:29:30 -06:00
Ross Wolf 1882f4456c [Fleet] Track integrations in folder and metadata (#1372)
* Track integrations in folder and metadata
* Remove duplicate entry
* Update note and tests
2021-07-21 15:24:56 -06:00
Justin Ibarra 163d9e3864 Update cardinality field in schema for threshold rules (#1349)
* Make cardinality array in schema for threshold rules
* update master, 7.12, 7.13, and 7.14 schemas with cardinality fix
* fix 7.12 downgrade to handle cardinality as an array

* Add two new rules to detect agent spoofing


Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2021-07-21 08:32:54 -08:00
Ross Wolf 816e31cd38 Add optional integration field to the schema (#1359) 2021-07-19 12:52:44 -06:00
Ross Wolf 77c23da1db [CI] Publish to integrations from on-demand job (#1340)
* Add command to publish integrations PR
* Add workflow_dispatch job to publish package
* Get working directory dynamically
* Fix the repo settings
* Get the absolute path for local-repo
* Filter out 'main' branch
* Update the description for target_branch
* Fix workflow definition
* Move 'if' into job
* Update ref format
* Remove unnecessary E501 suppression
* Add a link to the full commit hash
* s/partial_args/prefix_args
2021-07-14 16:19:41 -06:00
Ross Wolf 5b0f72ffc3 [CI/CD] Create on-demand job to release from Kibana (#1334)
* Add on-demand job to release to Kibana
* Update the inputs structure
* Archive the artifacts
2021-07-12 14:34:54 -06:00
Ross Wolf cf736046f1 Add command to unstage incompatible rules from git (#1317)
* Add devtools unstage-incompatible-rules command
* Create ephemeral GitChangeEntry for R->D+A
* Undo changes to Github job
* Fix typo in comment
* s/previous_path/original_path
2021-07-08 13:44:04 -06:00
Justin Ibarra 781953a0a0 Add min_stack_version to rule metadata (#1173)
* Add min_stack_version to metadata of rule structure
* validate all "stack versions" between defined and current package
* Use master schemas if min_stack_version > current_package

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2021-06-30 13:26:27 -08:00
Ross Wolf f1476b1637 Extend metadata with [metadata.extended] section (#1306)
* Extend metadata with `[metadata.extended]` section
* Remove whitespace
* Comment that it's a dict
2021-06-25 17:02:11 -06:00
Ross Wolf e897a67604 Fix fleet package generation (#1296)
* Fix fleet package generation
* Add .lstrip()
* Lint fix
* Add newline
2021-06-17 06:16:09 -06:00
Ross Wolf f6839e98d1 Simplify version locking code and fix 7.13.0 lock (#1295)
* Update version lock overwrite command
* Fix tooling and restore old version lock
* Lint fix
* Fix tests
* Remove dead code
* Filter to prod+deprecated rules
* Cast set -> list
* Store deprecation info
* Add correct version.lock.json (finally)
* Fix "stack_version" typo
* Remove stack_version
* Back out main.py changes

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
2021-06-16 18:02:47 -06:00
Ross Wolf 61e5b44c44 [Fleet] Update template and packaging code for fleet packages (#1280)
* Update template and packaging code for fleet packages
* Fix linting
2021-06-15 07:54:50 -06:00
Justin Ibarra 0ec8d67e78 Refactor experimental ML CLI and code (#1218)
* move github and ml to their own files
* refactor release and ml commands
* update ML readmes
* add unzip_to_dict function
* prompt for model ID in remove-model
* update experimental rule upload process
* update remove-scripts-pipelines to take multiple options

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
Co-authored-by: Apoorva <appujo@gmail.com>
2021-06-02 20:37:12 -08:00
Justin Ibarra e46f5e96d3 Fix create-rule bug (#1246) 2021-06-01 08:31:36 -08:00
Ross Wolf b0270d059f Add a command to create a Kibana PR (#1208)
* Add a command to create a Kibana PR
* Reformat code
* Fix docstring whitespace
* Make a hidden token prompt
* Fix E501
2021-05-17 14:57:21 -06:00
Ross Wolf eb40c52c7c Port historical schemas to jsonschema (#1084)
* Port historical schemas to jsonschema
* Add marshmallow-json dependency
* Mark etc/api_schemas as binary
* Remove gitattributes attempt
* Lint fix
* Apply PR feedback
* Additional PR feedback
* Extract stack version from packages.yml
* Fix the backport schemas
* Cache the schema reads
* Add migration for #1167
* Make a separate 'migration not found' error
2021-05-13 14:27:32 -06:00
Brent Murphy e40276c12b [Bug] Update main.py to fix toml-lint (#1202) 2021-05-13 09:43:13 -06:00
Justin Ibarra 1fb0b6726e Fix rule filenames during packaging (#1158) 2021-05-05 11:27:04 -08:00
Justin Ibarra 3d7f5d73a4 Allow ML rules to accept a single or array of job IDs (#1167) 2021-05-05 11:12:12 -08:00
Andrew Pease 92eaa5b18a [New Rule] Threat intel indicator match rule (#1133) 2021-04-26 07:07:04 -05:00
Justin Ibarra a0a3143a52 Refresh beats and ecs schemas (#1140)
* download new beats and ecs schemas
* add beats download func by version and download v7.11.2
2021-04-22 09:49:06 -08:00
Justin Ibarra cabe9239c0 Add threat_match rule type (#1138) 2021-04-22 09:03:57 -08:00
Ross Wolf 8789dd7c90 Separate out query validation from the class hierarchy (#1136)
* Separate out query validation from the class hierarchy
* Rename to *RuleData for consistency
* Apply suggestions from code review
* Fix lint error
2021-04-21 14:55:26 -06:00
Justin Ibarra e656a984b3 Update threshold rule schema to disallow empty field string (#1099) 2021-04-15 16:22:45 -06:00
Ross Wolf 791c911b9e Merge branch '7.12' into main 2021-04-15 16:17:59 -06:00
Ross Wolf 5669988e0b Remove unnecessary required=False check 2021-04-15 16:16:42 -06:00
Justin Ibarra b0f449339d add branch_name option to kibana-commit command 2021-04-14 21:16:09 -08:00
Justin Ibarra 462fab3ff8 Update threshold rule schema to disallow empty field string (#1098)
* Update threshold rule schema to disallow empty field string
* lock versions for rule changes
2021-04-14 04:56:38 -08:00
Justin Ibarra 92313b479a Lock 7.12 rule versions (#1083)
* lock versions for 7.12 rules
* Update promotion rules to match kibana with timestamp_override field
2021-04-06 10:48:17 -08:00
Ross Wolf b12437c88c Remove dead code in the rule loader 2021-04-05 14:30:26 -06:00
Ross Wolf 6ed1a39efe Add a RuleCollection object instead of a "loader" module (#1063)
* Add a RuleCollection object instead of a "loader" module
* Remove legacy loader code
* Remove more legacy loader
* Freeze the default collection
* Change RULE_LOADER default
* Rename to _toml_load_cache
* Use rglob magic
* Typo should've been a string
* Remove no longer needed glob import
* Fix pycharm import bad ordering
* Restore the detection_rules/schemas imports
* Put more imports back for a smaller diff
* Check cache in _deserialize_toml
* Add multi collection and single collection decorators
* Reorder RuleCollection methods
* Move filter method up
2021-04-05 14:23:37 -06:00
Ross Wolf 07be6b701d Change the asset .type field (#1075) 2021-04-05 10:50:58 -06:00
Ross Wolf 1e6e49a2cb Change the JSON schema for the security_rule Kibana asset (#1066)
* Change the JSON schema for the security_rule Kibana asset
* Use the asset type for the folder name
2021-03-30 13:31:02 -06:00
Ross Wolf 8ee1b2ffd4 Fix the version lock update code (#1064)
* Fix the version lock update code
* Add Rule.lock_info() method
2021-03-25 14:48:31 -06:00
Ross Wolf c0af222e7e Move Rule into a dataclass (#1029)
* WIP: Convert Rule to a dataclass
* Fix make release
* Lint fixes
* Remove dead code
* Fix lint and tests
* Use Python 3.8 in GitHub actions
* Update README to 3.8+
* Add Python 3.8 assertion
* Fix is_dirty property
* Remove incorrect pop from contents
* Add mixin with from_dict() and to_dict() methods
* Bypass validation for deprecated rules
* Fix rule_prompt
* Fix dict_hash usage
* Fix rule_event_search
* Switch to definitions.Date
* Fix toml-lint command, ignoring 'unneeded defaults'
* Moved severity Literal to definitions.Severity
* Remove BaseMarshmallowDataclass
* Fix lint and tests
* Add maturity to metadata for rule prompt loop
* Fix typo in devtools
* Use rule loader to load single rule in toml-lint
* Add Schema hint to __schema method
* Add MITREAttackURL definition
* Fix is_dirty to compare sha<-->sha
* Normalize the autoformatted rule output for API and toml-lint
* Make the package hash match
* Make the rule object mutable but not rule contents
* Restore the rules
2021-03-24 10:24:32 -06:00
Ross Wolf 6963c5a445 Change asset type to security_rule (#1054)
* Change asset type to security_rule
* Add notice.txt
2021-03-19 08:55:02 -06:00
Justin Ibarra d4cc4432ce Add tests to ensure rules are properly deprecated (#1050)
* Add tests to ensure rules are properly deprecated
* add deprecate-rule command
2021-03-16 21:31:33 -08:00
Ross Wolf 93f8f2dd94 Change asset type for integration to security-rule (#1048) 2021-03-16 16:05:30 -06:00
Ross Wolf 5c2da0b5c4 Move Rule.build to cli_utils.rule_prompt (#1024)
* Move Rule.build to cli_utils.rule_prompt
* Fix build_threat_map_entry lint
* Fix license and add docstring
2021-03-09 16:37:53 -07:00
Justin Ibarra fc9dfde2c4 Generate an integrations package from a release (#983)
* Generate an integrations package files during a release build
2021-03-09 13:30:12 -09:00
Justin Ibarra 0e0b2ea1a4 Update schema for threshold rule type for 7.12 (#976)
* Update schema for threshold rule type for 7.12
* add downgrade function to drop new fields
* update existing threshold rules
2021-03-05 14:35:50 -09:00
Justin Ibarra 0ef7d87b34 [Rule Tuning] Fix inconsistent rule indexes (#974)
* [Rule Tuning] Fix inconsistent rule indexes
* cleaned up tests that load rules to leverage setUpClass
2021-03-05 11:16:02 -09:00
Justin Ibarra 3fc34b86f2 Update License to Elastic v2 (#944) 2021-03-03 22:12:11 -09:00
Justin Ibarra b04218ec21 [CLI] Add repo option to kibana-diff command (#952) 2021-02-17 23:49:40 -09:00
Justin Ibarra d57394816f [FR] Index rules from repo to elasticsearch (#932)
* Add commands to generate index rules files or index them directly to elasticsearch
* files generated at package creation
* add readme explaining these index files
2021-02-10 10:37:26 -09:00
Justin Ibarra b8116a5b77 Add GitHub PR rule loader (#670)
* add load_gh_pr_rules function
* add dev package-stats command
* add dev search-rule-prs command, which extends the same functionality in rule-search to rules in PR
2021-02-08 21:35:44 -09:00
Justin Ibarra 56dc4745b5 Add export-rules command (#639)
* Add export-rule command to CLI
* add `export` method to packaging class
2021-02-08 20:43:16 -09:00