f43fbfba0d
[FR] Update utility path computation to use pathlib ( #3699 )
...
* update
* Updated to pathlib
* Linting
* Add string cast where needed
* Add additional string conversion as needed
* Str conversions to support eql lib
* Attack typo
* Typo in test script
* Updated for more pathlib
* Linting
* Update to convert string to path object
* Fix typo
2024-05-23 17:36:51 -04:00
371e24b2ed
Revert "[FR] Update Utility Path Computation to use Pathlib ( #3659 )"
...
This reverts commit 23567c1d0c
2024-05-21 16:14:45 -05:00
23567c1d0c
[FR] Update Utility Path Computation to use Pathlib ( #3659 )
...
* update
* Updated to pathlib
* Linting
* Add string cast where needed
* Add additional string conversion as needed
* Str conversions to support eql lib
* Attack typo
* Typo in test script
* Updated for more pathlib
* Linting
* Update to convert string to path object
2024-05-21 14:19:20 -04:00
c567d3731a
Refresh Kibana module with API updates ( #3466 )
...
* Refresh Kibana module with API updates
* add import/export commands
* rename repo commands
* add RawRuleCollection and DictRule objects
* save exported rules to files; rule.from_rule_resource
* strip unknown fields in schema
* add remote cli test
* update docs
* bump kibana lib version
---------
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com >
2024-04-26 11:12:50 -06:00
fbb6df506e
Update default ( #3574 )
2024-04-04 20:27:14 -04:00
1566c29bae
[Bug] KQL fails validation on uppercase keywords ( #3568 )
...
* add todo
* Add a normalize_kql_keywords function to utils
* update rule loader to normalize and warn
* optimized loading
* fix linting
* Moved conversion to kql module.
* Updated unit test
* Refactor KQL parser to normalize keywords via flag
* Fix logic typo
* Update detection_rules/utils.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Update lib/kql/kql/__init__.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Updated to fix unit tests and remove warnings
* linting typo
* Added comments
* remove unused imports
* Update kql.parse default
---------
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
2024-04-04 18:03:30 -04:00
d26981f712
[FR] Independently package kql / kibana and bump to py3.12 ( #3514 )
2024-03-14 20:18:32 -05:00
3d2a36be32
Revert "[FR] Independently package kql / kibana and bump to py3.12 ( #3492 )"
...
This reverts commit fc139fc3c2
2024-03-14 19:48:50 -05:00
fc139fc3c2
[FR] Independently package kql / kibana and bump to py3.12 ( #3492 )
2024-03-14 19:14:25 -05:00
6170db6231
[FR] Update _event_sort to use datetime instead of time ( #3375 )
...
* Update _event_sort to use datetime
* remove unused time
* added type hints
2024-01-09 10:59:01 -05:00
9f61ce4923
[FR] Only supporting known compatible rule file types ( #3167 )
...
* Only supporting known compatible file types
* Add --ignore-invalid-files flag
* Added support to ignore invalid rule files
* Update detection_rules/utils.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Update detection_rules/utils.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Update detection_rules/utils.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Update detection_rules/utils.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Update detection_rules/main.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* reverting main
* add punctuation
---------
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
2023-10-11 11:43:42 -04:00
6449cecd08
[FR] Add support for building block rules (BBR) ( #2822 )
...
* added test bbr
* initial implementation
* Added Unit test and exempted bbr from integrations
* fixed linting
* Add schema validation to building block rules
* add separate error messages
* fixed linting
* Add testing bbr validation
* fixed linting
* Add default values
* fixed linting
* added defaults
* fixed linting
* cleaned up test rule
* removed .gitkeep
* read .gitkeep
* Switch to using validates_schema
* addressing some linting
* fixed linting
* Update detection_rules/schemas/definitions.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* add env variable check
* fix skip function
* updated name
* Update detection_rules/schemas/definitions.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Add bbr validation unit test
* Clean up comments
* fix linting
* Move convert time to utils
* Moved to rules_building_block
* Add check for only bbr in bbr dir
* fix linting
* additional linting fix
* Changed to bbr rule loader
* fixed bbr default
* Updated error messages and README
* fixed more linting
* Updating root level README
* Fixed convert_time_span calls
* fixed typo in unit test logic and updated txt
* fixed error message
* updated comment for clarity
* Update detection_rules/rule.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Update detection_rules/rule.py
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
* Updated validation methods for clarity
* fix doctring location
* Fixed typo
* updated error messages.
* removed excess whitespace
* Add per rule bypass
* Add single rule bypass
* Split unit tests
* Update detection_rules/rule.py
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
* Update detection_rules/rule.py
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
---------
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com >
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
2023-06-20 09:00:30 -04:00
450e84ffa2
[FR] Add host family to data path ( #2839 )
...
* add rounding logic
* cleaned up event_sort
* fix linting
* Added host_family to ndjson file path
* linting fix
* Added ability to manually supply host_os_family
* fixed linting
* Update detection_rules/utils.py
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
* Update detection_rules/utils.py
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
* linting updates
---------
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
2023-06-12 16:03:33 -04:00
411ec36ff0
Validate markdown plugin fields ( #2602 )
2023-03-28 09:17:50 -04:00
1784429aa7
[FR] Add Integration Schema Query Validation ( #2470 )
2023-02-02 16:22:44 -05:00
c1dd3c57ad
Adds commands to manage ATT&CK mappings ( #2343 )
...
* add att&ck commands; fix 2 rule mappings
* update message to stdout
* updated date for rule changes
* unrelated click bug fix
* add type hinting
2022-11-01 13:14:40 -06:00
7d973a3b07
add new field related_integrations to the post build ( #2060 )
...
* add new field `related_integrations` to the post build
* add exception for endpoint `integration`
* Skip rules without related integrations
* lint
* refactor related_integrations to TOMLRuleContents class
* update to reflect required_fields updates
* add todo
* add new line for linting
* related_integrations updates, get_packaged_integrations returns list of dictionaries, started work on integrations py
* build_integrations_manifest command completed
* initial test completed for post-building related_integrations
* removed get_integration_manifest method from rule, removed global integrations path
* moved integration related methods to integrations.py and fixed flake issues
* adjustments for PipedQuery from eql sequence rules and packages with no integration
* adjusted github client import for integrations.py
* Update detection_rules/devtools.py
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
* Update detection_rules/devtools.py
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
* added integration manifest schema, made adjustments
* Update detection_rules/integrations.py
* Update detection_rules/rule.py
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
* Update detection_rules/rule.py
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
* Update detection_rules/integrations.py
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
* Update detection_rules/integrations.py
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
* Update detection_rules/integrations.py
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
* Update detection_rules/rule.py
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
* removed get_integrations_package to consolidate code
* removed type list return
* adjusted import flake errors
* Update detection_rules/integrations.py
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
* Update detection_rules/integrations.py
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
* adjusted indentation error
* adjusted rule.get_packaged_integrations to account for kql.ast.OrExpr if event.dataset is not set
* Update detection_rules/devtools.py
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
* Update detection_rules/devtools.py
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
* Update detection_rules/integrations.py
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
* Update detection_rules/integrations.py
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
* adjusted find_least_compatible_version in integrations.py
* Update detection_rules/integrations.py
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
* fixed flake issues
* adjusted get_packaged_integrations
* iterate the ast for literal event.dataset values
* Update detection_rules/integrations.py
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
* Update detection_rules/integrations.py
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
* Update detection_rules/integrations.py
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
* Update detection_rules/integrations.py
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
* made small adjustments to address errors during build manifests command
* addressing integrations.find_least_compatible method to return None instead of raise error only
* Update detection_rules/integrations.py
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com >
Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com >
Co-authored-by: Terrance DeJesus <terrance.dejesus@elastic.co >
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com >
2022-08-08 13:44:36 -04:00
6219fc06b9
Move etc under detection_rules ( #1885 )
...
* Move etc directory under detection_rules
* Prepend original `etc` path with `detection_rules`
* Update docstrings in util and CODEOWNERS
* Add resiliency to tags to account for the old directory structure
* Bug fix: remove unused param caused by commit 6ed1a39efebrokensound77@users.noreply.github.com >
2022-05-02 10:11:21 -04:00
1f216d12aa
Autogenerate docs for integration package releases ( #1567 )
...
* Autogenerate docs for integration package releases
* add parameter to bypass query validation in git loader
* strip space and - from normalized name
2022-01-26 21:19:03 -09:00
5b24eca0bc
[Bug] CLI Fixes ( #1073 )
...
* add support for self-signed certs in es and kibana
* allow Kibana to auth against any providerType
* fix export-rules command
* fix kibana upload-rule command
* fix view-rule command
* fix validate-rule command
* fix search-rules command
* fix dev kibana-diff command
* fix dev package-stats command
* fix dev search-rule-prs command
* fix dev deprecate-rule command
* replace toml with pytoml to fix import-rules command
* use no_verify in get_kibana_client
* use Path for rule-file type in view-rule
* update schemas to resolve additionalProperties type bug
* fix missing unique_fields in package rule filter
* fix github pr loader
* Load gh rules as TOMLRule instead of dict
* remove unnecessary version insertion
2021-09-10 10:06:04 -08:00
17bf3c1e16
Add RuleCollection.load_git_branch ( #1403 )
2021-08-05 01:15:39 -06:00
92937a1ad1
[CI] Fix kibana PR command again ( #1386 )
2021-07-27 16:29:50 -06:00
77c23da1db
[CI] Publish to integrations from on-demand job ( #1340 )
...
* Add command to publish integrations PR
* Add workflow_dispatch job to publish package
* Get working directory dynamically
* Fix the repo settings
* Get the absolute path for local-repo
* Filter out 'main' branch
* Update the description for target_branch
* Fix workflow definition
* Move 'if' into job
* Update ref format
* Remove unnecessary E501 suppression
* Add a link to the full commit hash
* s/partial_args/prefix_args
2021-07-14 16:19:41 -06:00
0ec8d67e78
Refactor experimental ML CLI and code ( #1218 )
...
* move github and ml to their own files
* refactor release and ml commands
* update ML readmes
* add unzip_to_dict function
* prompt for model ID in remove-model
* update experimental rule upload process
* update remove-scripts-pipelines to take multiple options
Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com >
Co-authored-by: Apoorva <appujo@gmail.com >
2021-06-02 20:37:12 -08:00
c0af222e7e
Move Rule into a dataclass ( #1029 )
...
* WIP: Convert Rule to a dataclass
* Fix make release
* Lint fixes
* Remove dead code
* Fix lint and tests
* Use Python 3.8 in GitHub actions
* Update README to 3.8+
* Add Python 3.8 assertion
* Fix is_dirty property
* Remove incorrect pop from contents
* Add mixin with from_dict() and to_dict() methods
* Bypass validation for deprecated rules
* Fix rule_prompt
* Fix dict_hash usage
* Fix rule_event_search
* Switch to definitions.Date
* Fix toml-lint command, ignoring 'unneeded defaults'
* Moved severity Literal to definitions.Severity
* Remove BaseMarshmallowDataclass
* Fix lint and tests
* Add maturity to metadata for rule prompt loop
* Fix typo in devtools
* Use rule loader to load single rule in toml-lint
* Add Schema hint to __schema method
* Add MITREAttackURL definition
* Fix is_dirty to compare sha<-->sha
* Normalize the autoformatted rule output for API and toml-lint
* Make the package hash match
* Make the rule object mutable but not rule contents
* Restore the rules
2021-03-24 10:24:32 -06:00
3fc34b86f2
Update License to Elastic v2 ( #944 )
2021-03-03 22:12:11 -09:00
d57394816f
[FR] Index rules from repo to elasticsearch ( #932 )
...
* Add commands to generate index rules files or index them directly to elasticsearch
* files generated at package creation
* add readme explaining these index files
2021-02-10 10:37:26 -09:00
ad4a2ef0eb
Add test commands to search and survey rule hits ( #485 )
2020-11-17 13:08:00 -09:00
bd680a2bd4
Re-organize commands under more specific click groups ( #356 )
...
* Restructure commands under more specific click groups
* standardize CLI error handling
* add global debug options
* move es and kibana clients into their click groups
* move commands and groups to dedicated files
* distinguish variable names for better env/config parsing
2020-10-07 12:15:33 -08:00
7c1e9c1ed5
Update package summary extras produced during package generation ( #341 )
...
* update summary.txt
* add summary.xlsx
* add changelog entry autogeneration
2020-09-30 14:43:45 -08:00
065bcd8018
Refresh ATT&CK data to v7.2 and expand threat validation ( #330 )
...
* refresh to latest ATT&CK 7.2
* add new unit test to further validate threat mappings
* updated threat mappings in rules to reflect changes
* new func to download and refresh mitre data based on version
2020-09-23 22:03:29 -08:00
8f5ddbb121
Add better CLI support for handling Kibana exported rules ( #83 )
2020-07-27 23:31:19 -05:00
db4f50d4b8
Improve the validation and testing time ( #61 )
...
* Improve the validation and testing time
* Lint fix
* Cache schema validation
2020-07-15 08:05:55 -06:00
e2d97b0a74
Remove unreachable and legacy code
...
Co-Authored-By: Justin Ibarra <brokensound77@users.noreply.github.com >
2020-06-30 10:12:23 -06:00
3b305d3003
Add rule loader and dependencies
...
Co-Authored-By: Justin Ibarra <brokensound77@users.noreply.github.com >
2020-06-29 23:17:42 -06:00
Eric Forte