security-tools/atomic-red-team
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Atomic Red Team - JSON Schema Validation CI (#2303)

Browse Source 
* feat: Adding atomic-red-team JSON Schema defintions

* feat: Adding validate.py script to validate all atomics against the defined schema

* feat: Adding validate-schema GitHub Workflow action to validate on every push to the repo

* ci: Updated the validate-schema workflow to support and use Ruby instead of python

* fix: Updated schema to remove schema draft version (not necessarily needed) and update to remove elevation_required as a required defined property

* fix: Removed the yaml schema version

* docs: Adding start of README

* fix: Adding an updated/better version of the python validation but may ultimately be removed

* feat: Adding Ruby version of validate.rb script

* fix: Removing files not needed since we are changing to github action and using the new validation code

* fix: Adding the yaml schema file back and removed the json version

* docs: Updated README with documentation

* fix: Updating schema to use new format validator

* fix: Updated validate.rb to verify that the Technique IDs are in the correct format.

* fix: Upating validate.rb to raise execptions so that failures flow up to the GitHub Action workflow

* fix: Updated all tests that have input_arguments not conformaing to schema defintion for type value of path

* fix: Updating the Validaton README for typos

* fixL: Minor updates to the schema

* minor schema changes

* github actions fix

* schema changes

---------

Co-authored-by: MSAdministrator <MSAdministrator@users.noreply.github.com>
Co-authored-by: Carrie Roberts <clr2of8@gmail.com>
Co-authored-by: Hare Sudhan <code@0x6c.dev>
This commit is contained in:
Josh Rickard
2023-05-12 16:33:47 -05:00
committed by GitHub
parent 9a70b81004
commit 284886292b
26 changed files with 495 additions and 86 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bin/validate-atomics.rb
-31
View File
@@ -1,31 +0,0 @@
#! /usr/bin/env ruby
$LOAD_PATH << "#{File.dirname(File.dirname(__FILE__))}/atomic_red_team" unless $LOAD_PATH.include? "#{File.dirname(File.dirname(__FILE__))}/atomic_red_team"
require 'yaml'
require 'atomic_red_team'
ATOMIC_RED_TEAM = AtomicRedTeam.new
ATOMIC_TEST_TEMPLATE = "#{File.dirname(File.dirname(__FILE__))}/atomic_red_team/atomic_test_template.yaml"
USED_GUIDS_FILE = "#{File.dirname(File.dirname(__FILE__))}/atomics/used_guids.txt"
oks = []
fails = []
unique_guid_array = []
ATOMIC_RED_TEAM.atomic_test_paths.each do |path|
begin
print "Validating #{path}..."
AtomicRedTeam.new.validate_atomic_yaml!(YAML.load_file(path), USED_GUIDS_FILE, unique_guid_array)
oks << path
puts "OK"
rescue => ex
fails << path
puts "FAIL\n#{ex}\n"
# puts "FAIL\n#{ex}\n#{ex.backtrace.join("\n")})"
end
end
puts
puts "#{oks.count + fails.count} techniques, #{fails.count} failures"
exit fails.count
bin/validate/README.md
+167
View File
@@ -0,0 +1,167 @@
# Validaton
We provide validation of each defined Atomic Red Team test in the form of a [JSON Schema](https://json-schema.org/). This schema defines the structure and format of an Atomic test.
- [Validaton](#validaton)
- [Validation Requirements](#validation-requirements)
- [atomic\_tests](#atomic_tests)
- [input\_arguments](#input_arguments)
- [dependencies](#dependencies)
- [dependency\_executor\_name](#dependency_executor_name)
- [executor](#executor)
- [Tooling \& Usage](#tooling--usage)
- [Error Messages (Ruby Version)](#error-messages-ruby-version)
- [Error Messages (Python version)](#error-messages-python-version)
We use this schema to validate the format of Atomics using a [GitHub Action](.../.github/workflows/validate-schema.yml) which runs on every push to the repository. If an Atomic fails validation, it is not allowed to be merged into the main branch.
```
📦atomics
┣ 📂T1234
┃ ┣ 📂T1234.md
┃ ┗ 📂T1234.yaml <-- This is where all the atomic tests live
┃ ┣ 📂src
┃ ┃ ┣ 📜payload1.sct <-- A paload file needed by one of the T1234 atomics (human readable)
┃ ┃ ┣ 📜payload2.dll <-- Another payload file needed by one of the T1234 atomics (binary)
```
In general, a set of atomic tests for a technique should never depend on payloads or supporting files from other atomic directories. We want to keep things nice and close. Use git symlinks if you really need to share files between techniques.
Atomic tests should be fully automated whenever possible, requiring no continued interaction. Include any needed options to execute the commands seamlessly, for example SysInternal's -accepteula option or any -q or -quiet modes.
## Validation Requirements
To explain the requirements around validation, we have broken down each main component.
Each yaml specification requires the following main level entities:
* attack_techniques - A Mitre ATT&CK Technique or Sub-Technique ID with a capital T.
* display_name - Name of the technique or sub-technique as defined by ATT&CK.
* atomic_tests - One or more Atomic tests for a technique / sub-technique.
### atomic_tests
Each `atomic_test` object must have the following fields defined:
|Property Name|Description|Data Type|Accepted Values|
|-------------|-----------|---------|---------------|
|name |The name of the test.|String|Any|
|description |A description about the test|String|Any|
|supported_platforms|One or more supported operating system platforms for this test. This is a list of supported_platforms and each must be unique.|List[String]|windows, macos, linux, office-365, azure-ad, google-workspace, saas, iaas, containers, iaas:gcp, iaas:azure, iaas:aws|
### input_arguments
Each defined test can supply one or more `input_arguments`. Please note that input arguments are not required and only optional. If you do provide a `input_argument` then each must be unique and contain a unique named property as well as sub-properties.
If your argument requires a String or null value you can use the following properties.
|Property Name|Description|Data Type|Accepted Values|
|-------------|-----------|---------|---------------|
|{unique_name}|A unique name for the input argument that will be referenced in commands|String|`^[a-zA-Z0-9_-]+$`|
|description |A description about the the input argument property|String|Any|
|type |The data type of the value for this property|String|Path, Url, String (please note the capitalization)|
|default |The default value for the argument|String or Null|Any|
If your argument requires a integer or float you can use the following properties.
|Property Name|Description|Data Type|Accepted Values|
|-------------|-----------|---------|---------------|
|{unique_name}|A unique name for the input argument that will be referenced in commands|String|`^[a-zA-Z0-9_-]+$`|
|description |A description about the the input argument property|String|Any|
|type |The data type of the value for this property|String|Integer, Float (please note the capitalization)|
|default |The default value for the argument|Number or Null|Any|
### dependencies
A list of dependies that must be met to successfully run this atomic. This is optional but if provided you must provide the following values for that dependency.
> You can supply more than 1 dependency. The `dependencies` property takes a list of dependencies.
|Property Name|Description|Data Type|Accepted Values|
|-------------|-----------|---------|---------------|
|description |A description about the the input argument property|String|Any|
|prereq_command|Commands to check if prerequisites for running this test are met. For the "command_prompt" executor, if any command returns a non-zero exit code, the pre-requisites are not met. For the "powershell" executor, all commands are run as a script block and the script block must return 0 for success.|String|Any|
|get_prereq_command|Commands to meet this prerequisite or a message describing how to meet this prereq|String|Any|
### dependency_executor_name
The executor for the prereq commands, defaults to the same executor used by the attack commands. This field is optional but must be one of the following values:
* command_prompt
* powershell
* sh
* bash
* manual
### executor
The `executor` propery contains a list of unique executors for each environment that the test belongs to or has defined definitions for.
Each defined `executor` can define the following properties.
|Property Name|Description|Data Type|Accepted Values|
|-------------|-----------|---------|---------------|
|name |The name of the executor to use to execute this command sequence.|String|command_prompt, sh, bash, powershell, aws, az, gcloud, kubectl|
|command |The command string to execute.|String|Any|
Each executor can also have the following fields, but these are only specified when needed.
|Property Name|Description|Data Type|Accepted Values|
|-------------|-----------|---------|---------------|
|elevation_required|indicates whether command must be run with admin privileges.|Bool|true or false|
|cleanup_command |The command string to execute to cleanup the system after executing the command above.|String|Any|
You can also specify the executor type of `manual`. The manual executor requires another field called `steps` which is a list of manual steps that the user must take to perform an action.
## Tooling & Usage
There are two main entrypoints to validate atomics. You can do so manually by cloning the repository and running the [validate.rb](validate.rb).
```ruby
ruby ./bin/validate/validate.rb
```
Additionally, the validation script will run on each push to the repository using the provided GitHub Action.
## Error Messages (Ruby Version)
TODO
## Error Messages (Python version)
A typical error message when validation fails looks like the following:
```bash
{'description': 'Daily scheduled task execution time', 'type': 'string', 'default': '07:45'} is not valid under any of the given schemas
Failed validating 'anyOf' in schema['properties']['atomic_tests']['items']['properties']['input_arguments']['patternProperties']['^[a-zA-Z0-9]*$']:
{'anyOf': [{'properties': {'default': {'type': ['string', 'null']},
'description': {'type': 'string'},
'type': {'enum': ['Path', 'Url', 'String'],
'type': 'string'}},
'type': 'object'},
{'properties': {'default': {'type': ['number', 'null']},
'description': {'type': 'string'},
'type': {'enum': ['Integer', 'Float'],
'type': 'string'}},
'required': ['description', 'type', 'default'],
'type': 'object'}],
'type': 'object'}
On instance['atomic_tests'][6]['input_arguments']['time']:
{'default': '07:45',
'description': 'Daily scheduled task execution time',
'type': 'string'}
```
With this error, it may be unclear exactly why the validation failed. That is why we have formatted the output to parse this error to make it more readable. For example, the parsed version is
```bash
Error occurred with ./atomics/T1053.005/T1053.005.yaml.
Each of the following are why it failed:
'string' is not one of ['Path', 'Url', 'String']
The JSON Path is $.atomic_tests[6].input_arguments.time
```
bin/validate/atomic-red-team.schema.yaml
+151
View File
@@ -0,0 +1,151 @@
$id: https://json-schema.org/draft/2020-12/schema
title: Atomic Schema
description: A schema for atomics within the atomic-red-team project
type: object
properties:
attack_technique:
description: A MITRE ATT&CK Technique ID with a capital T
type: string
format: technique_id
pattern: T[\.\d]{4,8}
display_name:
description: Name of the technique as defined by ATT&CK.
type: string
atomic_tests:
description: One or more Atomic tests for a technique
type: array
items:
$ref: "#/$defs/test"
minItems: 1
uniqueItems: true
$defs:
test:
type: object
required:
- name
- description
- supported_platforms
- executor
properties:
name:
type: string
description: The name of the test.
auto_generated_guid:
type: string
description: A unique test GUID
description:
type: string
description: A description about the test
supported_platforms:
type: array
description: One or more supported operating system platforms for this test
uniqueItems: true
items:
type: string
enum:
- windows
- macos
- linux
- office-365
- azure-ad
- google-workspace
- saas
- iaas
- containers
- iaas:gcp
- iaas:azure
- iaas:aws
input_arguments:
type: object
additionalProperties: false
properties:
"/": {}
patternProperties:
"^[\\w-]+$":
type: integer
type: object
required:
- description
properties:
description:
type: string
anyOf:
- required:
- type
properties:
type:
type: string
enum:
- integer
- float
default:
type:
- number
- "null"
- required:
- type
properties:
type:
type: string
enum:
- path
- url
- string
default:
type:
- string
- "null"
dependency_executor_name:
type: string
enum:
- command_prompt
- powershell
- sh
- bash
- manual
dependencies:
type: array
unique: true
items:
type: object
properties:
description:
type: string
prereq_command:
type: string
get_prereq_command:
type: string
required:
- description
- prereq_command
- get_prereq_command
executor:
type: object
required:
- name
properties:
name:
type: string
enum:
- command_prompt
- powershell
- sh
- bash
- manual
oneOf:
- required:
- command
properties:
elevation_required:
type: boolean
command:
type: string
cleanup_command:
type:
- string
- "null"
- required:
- steps
properties:
steps:
type: string
bin/validate/validate.py
+40
View File
@@ -0,0 +1,40 @@
"""Validates atomics based on JSON Schema."""
import glob
import os.path
import sys
import yaml
from jsonschema import validate
from jsonschema.exceptions import ValidationError
is_exception = False
with open(f"{os.path.dirname(os.path.abspath(__file__))}/atomic-red-team.schema.yaml", "r") as f:
schema = yaml.safe_load(f)
for item in glob.glob("./atomics/T*/T*.yaml"):
with open(item, 'r') as file:
data = yaml.safe_load(file)
try:
validate(
instance=data,
schema=schema
)
except ValidationError as ve:
print(f"Error occurred with {item}")
print("Each of the following are why it failed:")
if (context := ve.context) and len(context) > 0:
print(f"\n\t{context[0].message}\n")
else:
print(f"\n\t{ve}\n")
print(f"The JSON Path is {ve.json_path}")
is_exception = True
except Exception as e:
print(f"Error occurred with {item}")
print("Each of the following are why it failed:")
print(f"\n\t{e}\n")
is_exception = True
if is_exception:
print("Validation Failed")
sys.exit(1)
else:
print("Validation Successful")