Commit Graph

167 Commits

Author SHA1 Message Date
Justin Ibarra 0428e161a8 Refresh ECS/beats schemas up to 8.2 (#1995) 2022-05-25 11:51:43 -08:00
Mika Ayenson e1266a6fd3 Skip previous validation on pre/post load/dump (#1942)
* Build out the dataclasses for a base entry and version lock explicitly
* Ensure previous field does not have a nested previous
* Test validation on version lock for previous fields.
2022-05-25 13:34:03 -04:00
Samirbous 19ff825a91 [New rule] Remote Computer Account DnsHostName Update (#1962)
* [New rule] Remote Computer Account DnsHostName Update

Identifies remote update to a computer account DnsHostName attribute, if the new value is set a valid domain controller DNS hostname and the subject computer name is not a domain controller then it's high likely a preparation step to exploit CVE-2022-26923 in an attempt to elevate privileges from a standard domain user to domain admin privileges :

* added MS ref url

* Update rules/windows/privilege_escalation_suspicious_dnshostname_update.toml

Co-authored-by: Jonhnathan <jonhnathancesar@gmail.com>

* Update rules/windows/privilege_escalation_suspicious_dnshostname_update.toml

Co-authored-by: Jonhnathan <jonhnathancesar@gmail.com>

Co-authored-by: Jonhnathan <jonhnathancesar@gmail.com>
2022-05-11 19:40:34 +02:00
Justin Ibarra 8168551c59 Manually reconciled versions from forked rule package generation bug (#1950) 2022-05-04 10:04:10 -08:00
Justin Ibarra 22679e16d2 Add delta command to determine changes to endpoint rules between tags (#1943)
* update git tag loader to be compatible with lock validation
* add diff command
* default to query for missing rules
2022-05-03 12:30:11 -08:00
Mika Ayenson 6219fc06b9 Move etc under detection_rules (#1885)
* Move etc directory under detection_rules
* Prepend original `etc` path with `detection_rules`
* Update docstrings in util and CODEOWNERS
* Add resiliency to tags to account for the old directory structure
* Bug fix: remove unused param caused by commit 6ed1a39efe

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
2022-05-02 10:11:21 -04:00
Justin Ibarra c803160e4f Validate version lock and deprecation files on load and save (#1884)
* Validate version lock and deprecation files on load and save
* add missing types for previous lock entries
* bump marshmallow_dataclass
2022-04-26 22:17:20 -08:00
Justin Ibarra b3e789a202 Remove deprecated elasticsearch parameter (#1913) 2022-04-12 12:06:11 -08:00
Justin Ibarra ad99c6b489 Update elasticsearch dependency to 8.1 (#1911) 2022-04-06 11:52:22 -08:00
Justin Ibarra 6bdfddac8e Expand timestamp override tests (#1907)
* Expand timestamp_override tests
* removed timestamp_override from eql sequence rules
* add config entry for eql rules with beats index and t_o
* add timestamp_override to missing fields
2022-04-01 15:27:08 -08:00
Terrance DeJesus 648daf1237 Prep for Creation of 8.3 Branch (#1906)
* updating with changes for 8.3 prep
* adding updates
* adjusted version in packages.yml
2022-04-01 13:33:18 -08:00
Terrance DeJesus e72031a71a added comprehensive timeline template definitions (#1905) 2022-04-01 08:51:54 -08:00
Justin Ibarra 5214209f8d reset evasion rules (#1902) 2022-03-29 15:47:48 -08:00
Justin Ibarra 17ef6c558c [Bug] Fix bug in version_lock.py (#1880) 2022-03-24 15:41:16 -08:00
Justin Ibarra 11ec9c230e Prevent changes to rule type for locked rules (#1855)
* add rule type to the rule lock_info
* add check in VersionLock; add type to version.lock
* print changes only on save
2022-03-24 11:56:27 -08:00
Justin Ibarra f4c94af994 [Bug] Version bump with previous (#1870)
* save changes to top level for route C; verbose prints
* update top level on forked rule without overriding min_stack_version
* add check to ensure previous version !> current
2022-03-24 11:12:06 -08:00
Mika Ayenson 1f015ebe85 1554 update eql schemas to fail validation on text fields (#1866)
* Ensure kql2eql conversion doesnt support `text` fields

* Add unit test cases for`text` not supported in eql

* test `field not recognized` in the rule_validator and output a verbose message.

* use elasticsearch_type_family to lookup text mappings

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
2022-03-23 16:22:26 -04:00
Mika Ayenson 84b7ce6582 update beats master branch ref to main (#1853)
* update beats master branch ref to main

* update filename of master beat schema to main

* delete old main beats schema

* rebuilt main beats archive
2022-03-18 10:06:34 -04:00
Justin Ibarra 6653acb21c [Github Workflows] Only generate navigator files on push to main (#1814)
* [Github Workflows] Only generate navigator files on push to main

* fix workflow logic syntax
2022-03-04 09:55:11 -09:00
Justin Ibarra bb105a3c43 Replace * in navigator filenames (#1813) 2022-03-04 08:45:55 -09:00
Justin Ibarra 254b4eb23f Generate ATT&CK navigator layer files and links (#1787)
* Generate attack layer files and build with package
* add update-navigator-gists command
* add workflow to update navigator gists on pushes to main
* Add coverage readme
* fix keys for links
* update navigator layer names
* purge gist files prior to update; add badge
* Update how the navigator links are displayed
* moved navigator code to dedicated and refactored to dataclasses
* convert gist links to permalink versions
* alphabetize; catch 404 for gist update
2022-03-04 08:20:44 -09:00
Justin Ibarra a5eb02ac28 Refresh ATT&CK to v10.1 (#1791) 2022-02-24 16:37:23 -09:00
Justin Ibarra d373db7659 Ensure github module is installed before running PR commands (#1777)
* Ensure github module is installed before running PR commands

* move go and elastic-package assertions to top of command

* update error msg for missing pkg

* remove redundant github assertion

Co-authored-by: Colson Wilhoit <48036388+DefSecSentinel@users.noreply.github.com>
2022-02-24 14:49:01 -09:00
Mika Ayenson 0aeb7399d4 [Bug] Fix toml-lint ordering of Mitre metadata #1249 (#1774)
* Order the MITRE metadata by recursively sorting the rule object before writing.

* Refactor order_rule into the rule_formatter module.

* sort test_toml.json according to rule_formatter spec

* rename var to obj since this will traverse all data in the rule

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
2022-02-22 13:57:49 -05:00
Colson Wilhoit e0dda91f26 Prep for creation of 8.2 branch (#1762) 2022-02-08 18:43:55 -09:00
Justin Ibarra 2828633919 [Bug] Fix AttributeError in RuleCollection dupe check (#1747) 2022-01-31 15:57:46 -09:00
Justin Ibarra 1f216d12aa Autogenerate docs for integration package releases (#1567)
* Autogenerate docs for integration package releases
* add parameter to bypass query validation in git loader
* strip space and - from normalized name
2022-01-26 21:19:03 -09:00
Justin Ibarra e26374cb40 Update base branch in integrations-pr command (#1733) 2022-01-26 20:52:24 -09:00
Justin Ibarra 2e78da5c9a Prepare for creation of 8.1 branch (#1700) 2022-01-25 18:11:59 -09:00
Justin Ibarra d753ecb8d8 Add pattern for "name" in rule schema (#1669) 2022-01-25 12:03:27 -09:00
Justin Ibarra bd9e33e761 [bug] Current stack version in deprecation lock missing parens (#1618)
The function was not being properly called, leading to `null` values
2021-11-16 00:18:27 -09:00
Justin Ibarra 76503e8bcd Fix kibana-pr command (#1616) 2021-11-15 23:55:05 -09:00
Justin Ibarra 95d7e9b6f5 Prepare for creation of 7.16 release branch (#1611) 2021-11-15 09:39:34 -09:00
Justin Ibarra 0efae3a52e Move version lock code to object for portability (#1553)
* Move version lock code to object for portability
* use cached_property to bypass frozen dataclass and set property
* replace load_versions function
2021-11-15 08:46:12 -09:00
Justin Ibarra 5e6a58ebab Add index as a required field to rule_prompt (#1595) 2021-11-14 17:05:42 -09:00
Justin Ibarra d12c04761f Add support for eql-wildcard and kql-match_only_text (#1583)
* Add support for eql-wildcard and kql-match_only_text
* bump kql version
* lookup elasticsearch type family prior to getting type hint
Co-authored-by: David French <56409778+threat-punter@users.noreply.github.com>
2021-10-28 08:57:43 -05:00
Justin Ibarra ab17dfcc28 [Bug] Tighten definitions validation patterns (#1396)
* [Bug] Anchor validation patterns
* Deprecate rule with invalid rule_id and duplicate as new one

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2021-10-26 10:26:20 -05:00
Justin Ibarra 5a69ceb0c5 Add test for improper rule demotion (released production -> development) (#1555) 2021-10-19 21:47:36 -08:00
Justin Ibarra 5bdf70e72c Add min_stack_comments to metadata schema (#1573)
* Add min_stack_comments to metadata schema
2021-10-19 20:52:53 -08:00
Justin Ibarra 7179942be3 Allow CLi config to be multiple formats (#1485) 2021-09-15 20:12:39 -08:00
Justin Ibarra 5b24eca0bc [Bug] CLI Fixes (#1073)
* add support for self-signed certs in es and kibana
* allow Kibana to auth against any providerType
* fix export-rules command
* fix kibana upload-rule command
* fix view-rule command
* fix validate-rule command
* fix search-rules command
* fix dev kibana-diff command
* fix dev package-stats command
* fix dev search-rule-prs command
* fix dev deprecate-rule command
* replace toml with pytoml to fix import-rules command
* use no_verify in get_kibana_client
* use Path for rule-file type in view-rule
* update schemas to resolve additionalProperties type bug
* fix missing unique_fields in package rule filter
* fix github pr loader
* Load gh rules as TOMLRule instead of dict
* remove unnecessary version insertion
2021-09-10 10:06:04 -08:00
David French 90aa65aed3 Generate detection rule to alert on traffic to typosquatting/homonym domains (#1199)
* create new cli commands

* add kibana object to create_dnstwist_rule

* Adding code for index-dnstwist-results

* Changed es to es_client

* Tested. it works!

* flake8-ed

* Adding timestamps

* use eql.utils.load_dump to load json file

* rename data to dnstwist_data

* start working on create-dnstwist-rule command

* add print statements for user

* tweak formatting for line length

* add template threat match rule file

* continue working on threat match rule creation

* create rule using TomlRuleContents

* save rule to toml file

* Moving rule creation to eswrap.py

* Moving create dnstwist rule stuff to eswrap

* Fixed imports

* flake8 fixes

* More flake8 fixes

* fix usage of @add_client('kibana')

* use ctx.invoke to upload rule

* cleanup record assembly and use bulk api

* swap order of notes in `note` for sample rule

* small modifications

* move command to root click group

* remove unused click group

* Update detection_rules/main.py

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

* remove rule upload and convert template to ndjson

* Adding docs for typosquatting rule

* renaming the file

* Adding a note

* separate index and rule prep commands

* Final changes

Co-authored-by: Apoorva <appujo@gmail.com>
Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com>
Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>
2021-09-03 13:35:59 -07:00
Justin Ibarra 7710e2b798 Add DeprecatedCollection to RuleCollection to bypass validation (#1454)
* Add DeprecatedCollection to RuleCollection to bypass validation
* use DeprecatedRule properties in RuleCollection
* use RuleCollection filter for max/min filtering in Package
2021-09-01 15:29:53 -08:00
Justin Ibarra 9d10458be4 [Bug] RuleTOMLContents.to_dict serialize with proper schema (#1460) 2021-08-31 21:06:14 -08:00
Ross Wolf 7b8b18cb20 Update main to point to 7.16 (#1457)
* Update main to point to 7.16
* Add 7.16 -> 7.15 migration
* Update stack-schema-map
* Update conditions.kibana.version
2021-08-26 14:23:55 -06:00
Ross Wolf 4adad703fc [CI] Add GitHub actions workflow to lock versions across branches (#1456)
* Start job to lock versions
* Update lock-versions workflow
* Call lock-multiple script
* Fix script
* Add the lock file to staging
* pass branches to the job
* Fetch all branches and tags
* Push the branch first
* Push with upstream
* Change PR params
* Remove protections machine token
* Add 7.14.0 to the lock for min_stack_version=7.14.0
* Fix branch prefix
* Add trailing newline
* Trailing newline
* Restrict to main branch
2021-08-26 14:17:34 -06:00
Ross Wolf 0d47cb324a Track multiple stacks in lock (#1434)
* Save the stack versions in the lock file
* Support tracking of multiple stacks in the lock
* Update the version locking logic
* Fix bugs and test lock file
* Restore version lock
* Fix lint errors
* Call both click.echo and verbose echo separately
* Change when the change_rules message is output
2021-08-24 16:56:11 -06:00
Ross Wolf 11c443ba26 Fix encoding of 'Any' type in jsonschema (#1438) 2021-08-19 10:15:21 -06:00
Justin Ibarra 2d517432e3 Bump package versions (#1418)
* Bump package versions
* Add 7.14 migration; use master schema map if one does not exist
* add test to ensure an entry exists in the stack-schema-map for the current package version

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2021-08-18 21:25:53 -08:00
Ross Wolf d647c7b809 Skip etc/packages.yml from backport: auto (#1437) 2021-08-18 16:55:21 -06:00