Commit Graph

18 Commits

Author SHA1 Message Date
Justin Ibarra 0ec8d67e78 Refactor experimental ML CLI and code (#1218)
* move github and ml to their own files
* refactor release and ml commands
* update ML readmes
* add unzip_to_dict function
* prompt for model ID in remove-model
* update experimental rule upload process
* update remove-scripts-pipelines to take multiple options

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
Co-authored-by: Apoorva <appujo@gmail.com>
2021-06-02 20:37:12 -08:00
Ross Wolf 6ed1a39efe Add a RuleCollection object instead of a "loader" module (#1063)
* Add a RuleCollection object instead of a "loader" module
* Remove legacy loader code
* Remove more legacy loader
* Freeze the default collection
* Change RULE_LOADER default
* Rename to _toml_load_cache
* Use rglob magic
* Typo should've been a string
* Remove no longer needed glob import
* Fix pycharm import bad ordering
* Restore the detection_rules/schemas imports
* Put more imports back for a smaller diff
* Check cache in _deserialize_toml
* Add multi collection and single collection decorators
* Reorder RuleCollection methods
* Move filter method up
2021-04-05 14:23:37 -06:00
Ross Wolf c0af222e7e Move Rule into a dataclass (#1029)
* WIP: Convert Rule to a dataclass
* Fix make release
* Lint fixes
* Remove dead code
* Fix lint and tests
* Use Python 3.8 in GitHub actions
* Update README to 3.8+
* Add Python 3.8 assertion
* Fix is_dirty property
* Remove incorrect pop from contents
* Add mixin with from_dict() and to_dict() methods
* Bypass validation for deprecated rules
* Fix rule_prompt
* Fix dict_hash usage
* Fix rule_event_search
* Switch to definitions.Date
* Fix toml-lint command, ignoring 'unneeded defaults'
* Moved severity Literal to definitions.Severity
* Remove BaseMarshmallowDataclass
* Fix lint and tests
* Add maturity to metadata for rule prompt loop
* Fix typo in devtools
* Use rule loader to load single rule in toml-lint
* Add Schema hint to __schema method
* Add MITREAttackURL definition
* Fix is_dirty to compare sha<-->sha
* Normalize the autoformatted rule output for API and toml-lint
* Make the package hash match
* Make the rule object mutable but not rule contents
* Restore the rules
2021-03-24 10:24:32 -06:00
Justin Ibarra 3fc34b86f2 Update License to Elastic v2 (#944) 2021-03-03 22:12:11 -09:00
Justin Ibarra d57394816f [FR] Index rules from repo to elasticsearch (#932)
* Add commands to generate index rules files or index them directly to elasticsearch
* files generated at package creation
* add readme explaining these index files
2021-02-10 10:37:26 -09:00
Justin Ibarra 7c2abc68d7 [Docs] Update ML_DGA.md (#707) 2020-12-09 13:06:35 -09:00
Justin Ibarra 366e5002e1 [FR] Add experimental ML DGA CLI support (#361)
* Add DGA model commands
* Add upload/delete ML job command
* Add DGA release management commands
* Add Manifest handling
* Add GithubClient object
2020-12-01 22:25:33 -09:00
Justin Ibarra ad4a2ef0eb Add test commands to search and survey rule hits (#485) 2020-11-17 13:08:00 -09:00
Justin Ibarra 3b597bdb72 fix auth args in get_es_client 2020-10-30 09:19:50 -08:00
Justin Ibarra 3827d01a65 fix bugs in es client retrieval 2020-10-29 21:20:49 -08:00
Justin Ibarra bd680a2bd4 Re-organize commands under more specific click groups (#356)
* Restructure commands under more specific click groups
* standardize CLI error handling
* add global debug options
* move es and kibana clients into their click groups
* move commands and groups to dedicated files 
* distinguish variable names for better env/config parsing
2020-10-07 12:15:33 -08:00
shravaka fa12340ff0 [Bug fix] Add missing parenthesis for -kibana-url 2020-09-30 09:32:43 -06:00
Ross Wolf 453553f685 Change the way we get environment variables (#280)
* Change the way we get environment variables
* Change environ to getenv
* Read from envvar, then config file
* Switch to get_path
* Lint: Remove unused import
* Add --cloud-id/--elasticsearch-url
* Fix comment copy-pasta
2020-09-16 10:23:22 -06:00
Ross Wolf 464d5e645a Fix kibana-upload and remove cumbersome dataclasses (#216)
* Fix kibana-upload and remove cumbersom dataclasses

* Linting fixes
2020-09-01 05:47:27 -06:00
Justin Ibarra 28c869fb5f Expand documentation on CLI and workflows (#130) 2020-08-18 14:27:51 -05:00
Ross Wolf 0455307577 Downgrade rule version before uploading to Kibana (#97)
* Downgrade version before uploading to Kibana
* Update downgrade exception format
* Update s/siem/detection

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
2020-07-28 11:03:47 -06:00
Ross Wolf 16fb306254 Add command to upload to kibana (#58)
* Add upload command to kibana
* Restore skipped fields
* Change prefix to DR_
* Add note to manage_versions call
* Reorder requirements.txt to trigger build
2020-07-20 15:58:28 -06:00
Ross Wolf 3b305d3003 Add rule loader and dependencies
Co-Authored-By: Justin Ibarra <brokensound77@users.noreply.github.com>
2020-06-29 23:17:42 -06:00