Commit Graph

41 Commits

Author SHA1 Message Date
Eric Forte a726da5e83 [Bug] [DAC] Custom Rules Filter Discrepancy on Stacks Upgraded to 8.18 (#4945)
* Update Custom Rules KQL

* Bump Patch Version

* Update detection_rules/kbwrap.py

Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com>

* Use or instead of and

* Bump patch version

* Fix results len typo

---------

Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com>
2025-08-05 09:42:25 -04:00
Sergey Polzunov 1fb60d6475 fix: type hinting fixes and additional code checks (#4790)
* first pass

* Adding a dedicated code checking workflow

* Type fixes

* linting config and python version bump

* Type hints

* Drop incorrect config option

* More fixes

* Style fixes

* CI adjustments

* Pyproject fixes

* CI & pyproject fixes

* Proper version bump

* Tests formatting

* Resolve cirtular dependency

* Test fixes

* Make sure the tests are formatted correctly

* Check tweaks

* Bumping python version in CI images

* Pin marshmallow do 3.x because 4.x is not supported

* License fix

* Convert path to str

* Making myself a codeowner

* Missing kwargs param

* Adding a missing kwargs to `set_score`

* Update .github/CODEOWNERS

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* Dropping unnecessary raise

* Dropping skipped test

* Drop unnecessary var

* Drop unused commented-out func

* Disable typehinting for the whole func

* Update linting command

* Invalid type hist on the input param

* Incorrect field type

* Incorrect value used fix

* Stricter values check

* Simpler function call

* Type condition fix

* TOML formatter fix

* Simpligy output conditions

* Formatting

* Use proper types instead of aliases

* MITRE attack fixes

* Using pathlib.Path for an argument

* Use proper method to update a set from a dict

* First round of `ruff` fixes

* More fixes

* More fixes

* Hack against cyclic dependency

* Ignore `PLC0415`

* Remove unused markers

* Cleanup

* Fixing the incorrect condition

* Update .github/CODEOWNERS

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>

* Set explicit default values for optional fields

* Update the guidelines

* Adding None Defaults

---------

Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
Co-authored-by: eric-forte-elastic <eric.forte@elastic.co>
2025-07-01 08:20:55 -05:00
Emmanuel Ferdman 2ad2d68c4a Resolve datetime.utcfromtimestamp deprecation (#4719) 2025-05-19 21:35:07 +05:30
Eric Forte 639d748ec2 [FR] Add check-version-lock dev command (#4650)
* Add check-version-lock dev command

* Bump the version

* Add Check Double Bumps to lock-versions workflow

* Replace return with ctx aware exit

* Bump Version

* Update Double Bump Modulo calculation

* Update if formatting

* Undo formatting typo

* Add logic to process the local file

* Update for descriptiveness

* Allow double bump branch for testing

* Pass github token

* Re-restrict to main

* Patch version bump

* Add comment if no double bumps found

* Bump Version
2025-05-06 13:26:23 -04:00
Eric Forte 62feac3348 [Bug] Update Schema Prompt to include new_terms_fields (#4567)
* Update Schema Prompt to include new_terms_fields

* Version Bump

* Ensure list of strings

* Update utils to support comma deliminated strings

* Also remove excess quotes

* Bump patch version

* Remove Union

* bump version
2025-04-17 10:45:51 -04:00
Eric Forte 47d7a3acaa [DaC] Beta Release (#3889)
Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
Co-authored-by: Mika Ayenson <mika.ayenson@elastic.co>
2024-08-06 18:07:12 -04:00
Eric Forte f43fbfba0d [FR] Update utility path computation to use pathlib (#3699)
* update

* Updated to pathlib

* Linting

* Add string cast where needed

* Add additional string conversion as needed

* Str conversions to support eql lib

* Attack typo

* Typo in test script

* Updated for more pathlib

* Linting

* Update to convert string to path object

* Fix typo
2024-05-23 17:36:51 -04:00
Mika Ayenson 371e24b2ed Revert "[FR] Update Utility Path Computation to use Pathlib (#3659)"
This reverts commit 23567c1d0c.
2024-05-21 16:14:45 -05:00
Eric Forte 23567c1d0c [FR] Update Utility Path Computation to use Pathlib (#3659)
* update

* Updated to pathlib

* Linting

* Add string cast where needed

* Add additional string conversion as needed

* Str conversions to support eql lib

* Attack typo

* Typo in test script

* Updated for more pathlib

* Linting

* Update to convert string to path object
2024-05-21 14:19:20 -04:00
Justin Ibarra c567d3731a Refresh Kibana module with API updates (#3466)
* Refresh Kibana module with API updates
* add import/export commands
* rename repo commands
* add RawRuleCollection and DictRule objects
* save exported rules to files; rule.from_rule_resource
* strip unknown fields in schema
* add remote cli test
* update docs
* bump kibana lib version

---------

Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
2024-04-26 11:12:50 -06:00
Eric Forte fbb6df506e Update default (#3574) 2024-04-04 20:27:14 -04:00
Eric Forte 1566c29bae [Bug] KQL fails validation on uppercase keywords (#3568)
* add todo

* Add a normalize_kql_keywords function to utils

* update rule loader to normalize and warn

* optimized loading

* fix linting

* Moved conversion to kql module.

* Updated unit test

* Refactor KQL parser to normalize keywords via flag

* Fix logic typo

* Update detection_rules/utils.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update lib/kql/kql/__init__.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Updated to fix unit tests and remove warnings

* linting typo

* Added comments

* remove unused imports

* Update kql.parse default

---------

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2024-04-04 18:03:30 -04:00
Mika Ayenson d26981f712 [FR] Independently package kql / kibana and bump to py3.12 (#3514) 2024-03-14 20:18:32 -05:00
Mika Ayenson 3d2a36be32 Revert "[FR] Independently package kql / kibana and bump to py3.12 (#3492)"
This reverts commit fc139fc3c2.
2024-03-14 19:48:50 -05:00
Mika Ayenson fc139fc3c2 [FR] Independently package kql / kibana and bump to py3.12 (#3492) 2024-03-14 19:14:25 -05:00
Eric Forte 6170db6231 [FR] Update _event_sort to use datetime instead of time (#3375)
* Update _event_sort to use datetime

* remove unused time

* added type hints
2024-01-09 10:59:01 -05:00
eric-forte-elastic 9f61ce4923 [FR] Only supporting known compatible rule file types (#3167)
* Only supporting known compatible file types

* Add --ignore-invalid-files flag

* Added support to ignore invalid rule files

* Update detection_rules/utils.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update detection_rules/utils.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update detection_rules/utils.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update detection_rules/utils.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update detection_rules/main.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* reverting main

* add punctuation

---------

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
2023-10-11 11:43:42 -04:00
eric-forte-elastic 6449cecd08 [FR] Add support for building block rules (BBR) (#2822)
* added test bbr

* initial implementation

* Added Unit test and exempted bbr from integrations

* fixed linting

* Add schema validation to building block rules

* add separate error messages

* fixed linting

* Add testing bbr validation

* fixed linting

* Add default values

* fixed linting

* added defaults

* fixed linting

* cleaned up test rule

* removed .gitkeep

* read .gitkeep

* Switch to using validates_schema

* addressing some linting

* fixed linting

* Update detection_rules/schemas/definitions.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* add env variable check

* fix skip function

* updated name

* Update detection_rules/schemas/definitions.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Add bbr validation unit test

* Clean up comments

* fix linting

* Move convert time to utils

* Moved to rules_building_block

* Add check for only bbr in bbr dir

* fix linting

* additional linting fix

* Changed to bbr rule loader

* fixed bbr default

* Updated error messages and README

* fixed more linting

* Updating root level README

* Fixed convert_time_span calls

* fixed typo in unit test logic and updated txt

* fixed error message

* updated comment for clarity

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>

* Updated validation methods for clarity

* fix doctring location

* Fixed typo

* updated error messages.

* removed excess whitespace

* Add per rule bypass

* Add single rule bypass

* Split unit tests

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

---------

Co-authored-by: Justin Ibarra <16747370+brokensound77@users.noreply.github.com>
Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2023-06-20 09:00:30 -04:00
eric-forte-elastic 450e84ffa2 [FR] Add host family to data path (#2839)
* add rounding logic

* cleaned up event_sort

* fix linting

* Added host_family to ndjson file path

* linting fix

* Added ability to manually supply host_os_family

* fixed linting

* Update detection_rules/utils.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* Update detection_rules/utils.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

* linting updates

---------

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>
2023-06-12 16:03:33 -04:00
Justin Ibarra 411ec36ff0 Validate markdown plugin fields (#2602) 2023-03-28 09:17:50 -04:00
Mika Ayenson 1784429aa7 [FR] Add Integration Schema Query Validation (#2470) 2023-02-02 16:22:44 -05:00
Justin Ibarra c1dd3c57ad Adds commands to manage ATT&CK mappings (#2343)
* add att&ck commands; fix 2 rule mappings

* update message to stdout

* updated date for rule changes

* unrelated click bug fix

* add type hinting
2022-11-01 13:14:40 -06:00
Mika Ayenson 7d973a3b07 add new field related_integrations to the post build (#2060)
* add new field `related_integrations` to the post build

* add exception for endpoint `integration`

* Skip rules without related integrations

* lint

* refactor related_integrations to TOMLRuleContents class

* update to reflect required_fields updates

* add todo

* add new line for linting

* related_integrations updates, get_packaged_integrations returns list of dictionaries, started work on integrations py

* build_integrations_manifest command completed

* initial test completed for post-building related_integrations

* removed get_integration_manifest method from rule, removed global integrations path

* moved integration related methods to integrations.py and fixed flake issues

* adjustments for PipedQuery from eql sequence rules and packages with no integration

* adjusted github client import for integrations.py

* Update detection_rules/devtools.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/devtools.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* added integration manifest schema, made adjustments

* Update detection_rules/integrations.py

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/rule.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* removed get_integrations_package to consolidate code

* removed type list return

* adjusted import flake errors

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* adjusted indentation error

* adjusted rule.get_packaged_integrations to account for kql.ast.OrExpr if event.dataset is not set

* Update detection_rules/devtools.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/devtools.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* adjusted find_least_compatible_version in integrations.py

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* fixed flake issues

* adjusted get_packaged_integrations

* iterate the ast for literal event.dataset values

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* Update detection_rules/integrations.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* made small adjustments to address errors during build manifests command

* addressing integrations.find_least_compatible method to return None instead of raise error only

* Update detection_rules/integrations.py

Co-authored-by: Mika Ayenson <Mikaayenson@users.noreply.github.com>

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
Co-authored-by: Terrance DeJesus <terrance.dejesus@elastic.co>
Co-authored-by: Terrance DeJesus <99630311+terrancedejesus@users.noreply.github.com>
2022-08-08 13:44:36 -04:00
Mika Ayenson 6219fc06b9 Move etc under detection_rules (#1885)
* Move etc directory under detection_rules
* Prepend original `etc` path with `detection_rules`
* Update docstrings in util and CODEOWNERS
* Add resiliency to tags to account for the old directory structure
* Bug fix: remove unused param caused by commit 6ed1a39efe

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
2022-05-02 10:11:21 -04:00
Justin Ibarra 1f216d12aa Autogenerate docs for integration package releases (#1567)
* Autogenerate docs for integration package releases
* add parameter to bypass query validation in git loader
* strip space and - from normalized name
2022-01-26 21:19:03 -09:00
Justin Ibarra 5b24eca0bc [Bug] CLI Fixes (#1073)
* add support for self-signed certs in es and kibana
* allow Kibana to auth against any providerType
* fix export-rules command
* fix kibana upload-rule command
* fix view-rule command
* fix validate-rule command
* fix search-rules command
* fix dev kibana-diff command
* fix dev package-stats command
* fix dev search-rule-prs command
* fix dev deprecate-rule command
* replace toml with pytoml to fix import-rules command
* use no_verify in get_kibana_client
* use Path for rule-file type in view-rule
* update schemas to resolve additionalProperties type bug
* fix missing unique_fields in package rule filter
* fix github pr loader
* Load gh rules as TOMLRule instead of dict
* remove unnecessary version insertion
2021-09-10 10:06:04 -08:00
Ross Wolf 17bf3c1e16 Add RuleCollection.load_git_branch (#1403) 2021-08-05 01:15:39 -06:00
Ross Wolf 92937a1ad1 [CI] Fix kibana PR command again (#1386) 2021-07-27 16:29:50 -06:00
Ross Wolf 77c23da1db [CI] Publish to integrations from on-demand job (#1340)
* Add command to publish integrations PR
* Add workflow_dispatch job to publish package
* Get working directory dynamically
* Fix the repo settings
* Get the absolute path for local-repo
* Filter out 'main' branch
* Update the description for target_branch
* Fix workflow definition
* Move 'if' into job
* Update ref format
* Remove unnecessary E501 suppression
* Add a link to the full commit hash
* s/partial_args/prefix_args
2021-07-14 16:19:41 -06:00
Justin Ibarra 0ec8d67e78 Refactor experimental ML CLI and code (#1218)
* move github and ml to their own files
* refactor release and ml commands
* update ML readmes
* add unzip_to_dict function
* prompt for model ID in remove-model
* update experimental rule upload process
* update remove-scripts-pipelines to take multiple options

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
Co-authored-by: Apoorva <appujo@gmail.com>
2021-06-02 20:37:12 -08:00
Ross Wolf c0af222e7e Move Rule into a dataclass (#1029)
* WIP: Convert Rule to a dataclass
* Fix make release
* Lint fixes
* Remove dead code
* Fix lint and tests
* Use Python 3.8 in GitHub actions
* Update README to 3.8+
* Add Python 3.8 assertion
* Fix is_dirty property
* Remove incorrect pop from contents
* Add mixin with from_dict() and to_dict() methods
* Bypass validation for deprecated rules
* Fix rule_prompt
* Fix dict_hash usage
* Fix rule_event_search
* Switch to definitions.Date
* Fix toml-lint command, ignoring 'unneeded defaults'
* Moved severity Literal to definitions.Severity
* Remove BaseMarshmallowDataclass
* Fix lint and tests
* Add maturity to metadata for rule prompt loop
* Fix typo in devtools
* Use rule loader to load single rule in toml-lint
* Add Schema hint to __schema method
* Add MITREAttackURL definition
* Fix is_dirty to compare sha<-->sha
* Normalize the autoformatted rule output for API and toml-lint
* Make the package hash match
* Make the rule object mutable but not rule contents
* Restore the rules
2021-03-24 10:24:32 -06:00
Justin Ibarra 3fc34b86f2 Update License to Elastic v2 (#944) 2021-03-03 22:12:11 -09:00
Justin Ibarra d57394816f [FR] Index rules from repo to elasticsearch (#932)
* Add commands to generate index rules files or index them directly to elasticsearch
* files generated at package creation
* add readme explaining these index files
2021-02-10 10:37:26 -09:00
Justin Ibarra ad4a2ef0eb Add test commands to search and survey rule hits (#485) 2020-11-17 13:08:00 -09:00
Justin Ibarra bd680a2bd4 Re-organize commands under more specific click groups (#356)
* Restructure commands under more specific click groups
* standardize CLI error handling
* add global debug options
* move es and kibana clients into their click groups
* move commands and groups to dedicated files 
* distinguish variable names for better env/config parsing
2020-10-07 12:15:33 -08:00
Justin Ibarra 7c1e9c1ed5 Update package summary extras produced during package generation (#341)
* update summary.txt
* add summary.xlsx
* add changelog entry autogeneration
2020-09-30 14:43:45 -08:00
Justin Ibarra 065bcd8018 Refresh ATT&CK data to v7.2 and expand threat validation (#330)
* refresh to latest ATT&CK 7.2
* add new unit test to further validate threat mappings
* updated threat mappings in rules to reflect changes
* new func to download and refresh mitre data based on version
2020-09-23 22:03:29 -08:00
Justin Ibarra 8f5ddbb121 Add better CLI support for handling Kibana exported rules (#83) 2020-07-27 23:31:19 -05:00
Ross Wolf db4f50d4b8 Improve the validation and testing time (#61)
* Improve the validation and testing time
* Lint fix
* Cache schema validation
2020-07-15 08:05:55 -06:00
Ross Wolf e2d97b0a74 Remove unreachable and legacy code
Co-Authored-By: Justin Ibarra <brokensound77@users.noreply.github.com>
2020-06-30 10:12:23 -06:00
Ross Wolf 3b305d3003 Add rule loader and dependencies
Co-Authored-By: Justin Ibarra <brokensound77@users.noreply.github.com>
2020-06-29 23:17:42 -06:00