# Command Line Interface (CLI) This covers more advanced CLI use cases and workflows. To [get started](README.md#getting-started) with the CLI, reference the [README](README.md). Basic use of the CLI such as [creating a rule](CONTRIBUTING.md#creating-a-rule-with-the-cli) or [testing](CONTRIBUTING.md#testing-a-rule-with-the-cli) are referenced in the [contribution guide](CONTRIBUTING.md). ## Using a config file or environment variables CLI commands which are tied to Kibana and Elasticsearch are capable of parsing auth-related keyword args from a config file or environment variables. If a value is set in multiple places, such as config file and environment variable, the order of precedence will be as follows: * explicitly passed args (such as `--user joe`) * environment variables * config values * prompt (this only applies to certain values) #### Setup a config file In the root directory of this repo, create the file `.detection-rules-cfg.json` and add relevant values Currently supported arguments: * elasticsearch_url * kibana_url * cloud_id * *_username (kibana and es) * *_password (kibana and es) #### Using environment variables Environment variables using the argument format: `DR_` will be parsed in commands which expect it. EX: `DR_USER=joe` Using the environment variable `DR_BYPASS_NOTE_VALIDATION_AND_PARSE` will bypass the Detection Rules validation on the `note` field in toml files. Using the environment variable `DR_BYPASS_BBR_LOOKBACK_VALIDATION` will bypass the Detection Rules lookback and interval validation on the building block rules. Using the environment variable `DR_BYPASS_TAGS_VALIDATION` will bypass the Detection Rules Unit Tests on the `tags` field in toml files. ## Importing rules into the repo You can import rules into the repo using the `create-rule` or `import-rules` commands. Both of these commands will require that the rules are schema-compliant and able to pass full validation. The biggest benefit to using these commands is that they will strip[*](#note) additional fields[**](#note-2) and prompt for missing required fields. Alternatively, you can manually place rule files in the directory and run tests to validate as well. \* Note: This is currently limited to flat fields and may not apply to nested values.
\** Note: Additional fields are based on the current schema at the time the command is used. #### `create-rule` ```console Usage: detection_rules create-rule [OPTIONS] PATH Create a detection rule. Options: -c, --config FILE Rule or config file --required-only Only prompt for required fields -t, --rule-type [machine_learning|query|threshold] Type of rule to create -h, --help Show this message and exit. ``` This command will allow you to pass a rule file using the `-c/--config` parameter. This is limited to one rule at a time and will accept any valid rule in the following formats: * toml * json * yaml (yup) * ndjson (as long as it contains only a single rule and has the extension `.ndjson` or `.jsonl`) #### `import-rules` ```console Usage: detection_rules import-rules [OPTIONS] [INPUT_FILE]... Import rules from json, toml, or Kibana exported rule file(s). Options: -d, --directory DIRECTORY Load files from a directory -h, --help Show this message and exit. ``` The primary advantage of using this command is the ability to import multiple rules at once. Multiple rule paths can be specified explicitly with unlimited arguments, recursively within a directory using `-d/--directory`[*](#note-3), or a combination of both. In addition to the formats mentioned using `create-rule`, this will also accept an `.ndjson`/`jsonl` file containing multiple rules (as would be the case with a bulk export). This will also strip additional fields and prompt for missing required fields. \* Note: This will attempt to parse ALL files recursively within a specified directory. ## Commands using Elasticsearch and Kibana clients Commands which connect to Elasticsearch or Kibana are embedded under the subcommands: * es * kibana These command groups will leverage their respective clients and will automatically use parsed config options if defined, otherwise arguments should be passed to the sub-command as: `python -m detection-rules kibana -u -p upload-rule <...>` ```console python -m detection_rules es -h Usage: detection_rules es [OPTIONS] COMMAND [ARGS]... Commands for integrating with Elasticsearch. Options: -et, --timeout INTEGER Timeout for elasticsearch client -ep, --es-password TEXT -eu, --es-user TEXT --cloud-id TEXT -e, --elasticsearch-url TEXT -h, --help Show this message and exit. Commands: collect-events Collect events from Elasticsearch. ``` Providers are the name that Elastic Cloud uses to configure authentication in Kibana. When we create deployment, Elastic Cloud configures two providers by default: basic/cloud-basic and saml/cloud-saml (for SSO). ```console python -m detection_rules kibana -h █▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄ █ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄ █▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█ Usage: detection_rules kibana [OPTIONS] COMMAND [ARGS]... Commands for integrating with Kibana. Options: --ignore-ssl-errors TEXT --space TEXT Kibana space --provider-name TEXT For cloud deployments, Elastic Cloud configures two providers by default: cloud-basic and cloud-saml (for SSO) --provider-type TEXT For cloud deployments, Elastic Cloud configures two providers by default: basic and saml (for SSO) -ku, --kibana-user TEXT --kibana-url TEXT -kp, --kibana-password TEXT -kc, --kibana-cookie TEXT Cookie from an authed session --cloud-id TEXT ID of the cloud instance. Defaults the cloud provider to cloud-basic if this option is supplied -h, --help Show this message and exit. Commands: search-alerts Search detection engine alerts with KQL. upload-rule Upload a list of rule .toml files to Kibana. ``` ## Searching Kibana for Alerts Alerts stored in Kibana can be quickly be identified by searching with the `search-alerts` command. ```console python -m detection_rules kibana search-alerts -h Kibana client: Options: --ignore-ssl-errors TEXT --space TEXT Kibana space --provider-name TEXT For cloud deployments, Elastic Cloud configures two providers by default: cloud-basic and cloud-saml (for SSO) --provider-type TEXT For cloud deployments, Elastic Cloud configures two providers by default: basic and saml (for SSO) -ku, --kibana-user TEXT --kibana-url TEXT -kp, --kibana-password TEXT -kc, --kibana-cookie TEXT Cookie from an authed session --cloud-id TEXT ID of the cloud instance. Defaults the cloud provider to cloud-basic if this option is supplied Usage: detection_rules kibana search-alerts [OPTIONS] [QUERY] Search detection engine alerts with KQL. Options: -d, --date-range ... Date range to scope search -c, --columns TEXT Columns to display in table -e, --extend If columns are specified, extend the original columns -h, --help Show this message and exit. ``` Running the following command will print out a table showing any alerts that have been generated recently. `python3 -m detection_rules kibana --provider-name cloud-basic --kibana-url --kibana-user --kibana-password search-alerts` ```console █▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄ █ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄ █▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█ =================================================================================================================================== host rule hostname name @timestamp =================================================================================================================================== stryker-malwares-MacBook-Pro.local Sudo Heap-Based Buffer Overflow Attempt 2022-06-21T14:08:34.288Z stryker-malwares-MacBook-Pro.local Suspicious Automator Workflows Execution 2022-06-21T13:58:30.857Z stryker-malwares-MacBook-Pro.local Privilege Escalation Enumeration via LinPEAS 2022-06-21T13:33:18.218Z stryker-malwares-MacBook-Pro.local Privilege Escalation Enumeration via LinPEAS 2022-06-21T13:28:14.685Z stryker-malwares-MacBook-Pro.local Potential Reverse Shell Activity via Terminal 2022-06-21T12:53:00.234Z stryker-malwares-MacBook-Pro.local Potential Reverse Shell Activity via Terminal 2022-06-21T12:53:00.237Z stryker-malwares-MacBook-Pro.local Potential Kerberos Attack via Bifrost 2022-06-20T20:33:53.810Z stryker-malwares-MacBook-Pro.local Potential Kerberos Attack via Bifrost 2022-06-20T20:33:53.813Z stryker-malwares-MacBook-Pro.local Potential Privilege Escalation via Root Crontab File Modification 2022-06-20T20:23:50.557Z stryker-malwares-MacBook-Pro.local Download and Execution of JavaScript Payload 2022-06-20T20:18:46.211Z =================================================================================================================================== ``` ## Uploading rules to Kibana Toml formatted rule files can be uploaded as custom rules using the `kibana upload-rule` command. To upload more than one file, specify multiple files at a time as individual args. This command is meant to support uploading and testing of rules and is not intended for production use in its current state. ```console python -m detection_rules kibana upload-rule -h Kibana client: Options: --space TEXT Kibana space -kp, --kibana-password TEXT -ku, --kibana-user TEXT --cloud-id TEXT -k, --kibana-url TEXT Usage: detection_rules kibana upload-rule [OPTIONS] Upload a list of rule .toml files to Kibana. Options: -f, --rule-file FILE -d, --directory DIRECTORY Recursively export rules from a directory -id, --rule-id TEXT -r, --replace-id Replace rule IDs with new IDs before export -h, --help Show this message and exit. (detection-rules-build) (base) ➜ detection-rules git:(rule-loader) ✗ ``` Alternatively, rules can be exported into a consolidated ndjson file which can be imported in the Kibana security app directly. ```console Usage: detection_rules export-rules [OPTIONS] Export rule(s) into an importable ndjson file. Options: -f, --rule-file FILE -d, --directory DIRECTORY Recursively export rules from a directory -id, --rule-id TEXT -o, --outfile FILE Name of file for exported rules -r, --replace-id Replace rule IDs with new IDs before export --stack-version [7.8|7.9|7.10|7.11|7.12] Downgrade a rule version to be compatible with older instances of Kibana -s, --skip-unsupported If `--stack-version` is passed, skip rule types which are unsupported (an error will be raised otherwise) -h, --help Show this message and exit. ``` _*To load a custom rule, the proper index must be setup first. The simplest way to do this is to click the `Load prebuilt detection rules and timeline templates` button on the `detections` page in the Kibana security app._ ## Converting between JSON and TOML [Importing rules](#importing-rules-into-the-repo) will convert from any supported format to toml. Additionally, the command `view-rule` will also allow you to view a converted rule without importing it by specifying the `--rule-format` flag. To view a rule in JSON format, you can also use the `view-rule` command with the `--api-format` flag, which is the default. (See the [note](#a-note-on-version-handling) on the JSON formatted rules and versioning) ## A note on version handling The rule toml files exist slightly different than they do in their final state as a JSON file in Kibana. The files are white space stripped, normalized, sorted, and indented, prior to their json conversion. Everything within the `metadata` table is also stripped out, as this is meant to be used only in the context of this repository and not in Kibana.. Additionally, the `version` of the rule is added to the file prior to exporting it. This is done to restrict version bumps to occur intentionally right before we create a release. Versions are auto-incremented based on detected changes in rules. This is based on the hash of the rule in the following format: * sorted json * serialized * b64 encoded * sha256 hash As a result, all cases where rules are shown or converted to JSON are not just simple conversions from TOML. ## Debugging Most of the CLI errors will print a concise, user friendly error. To enable debug mode and see full error stacktraces, you can define `"debug": true` in your config file, or run `python -m detection-rules -d `. Precedence goes to the flag over the config file, so if debug is enabled in your config and you run `python -m detection-rules --no-debug`, debugging will be disabled. ## Using `transform` in rule toml A transform is any data that will be incorporated into _existing_ rule fields at build time, from within the `TOMLRuleContents.to_dict` method. _How_ to process each transform should be defined within the `Transform` class as a method specific to the transform type. ### CLI support for investigation guide plugins This applies to osquery and insights for the moment but could expand in the future. ``` (venv38) ➜ detection-rules-fork git:(2597-validate-osquery-insights) python -m detection_rules dev transforms -h █▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄ █ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄ █▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█ Usage: detection_rules dev transforms [OPTIONS] COMMAND [ARGS]... Commands for managing TOML [transform]. Options: -h, --help Show this message and exit. Commands: guide-plugin-convert Convert investigation guide plugin format to toml guide-plugin-to-rule Convert investigation guide plugin format to toml ``` `guide-plugin-convert` will print out the formatted toml. ``` (venv38) ➜ detection-rules-fork git:(2597-validate-osquery-insights) python -m detection_rules dev transforms guide-plugin-convert █▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄ █ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄ █▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█ Enter plugin contents []: !{osquery{"query":"SELECT description, display_name, name, path, pid, service_type, start_type, status, user_account FROM services\nWHERE NOT (user_account LIKE \"%LocalSystem\" OR user_account LIKE \"%LocalService\" OR user_account LIKE \"%NetworkService\" OR user_account == null)","label":"label2","ecs_mapping":{"labels":{"field":"description"},"agent.build.original":{"value":"fast"}}}} [transform] [[transform.osquery]] query = "SELECT description, display_name, name, path, pid, service_type, start_type, status, user_account FROM services\nWHERE NOT (user_account LIKE \"%LocalSystem\" OR user_account LIKE \"%LocalService\" OR user_account LIKE \"%NetworkService\" OR user_account == null)" label = "label2" [transform.osquery.ecs_mapping] [transform.osquery.ecs_mapping.labels] field = "description" [transform.osquery.ecs_mapping."agent.build.original"] value = "fast" ``` The easiest way to _update_ a rule with existing transform entries is to use `guide-plugin-convert` and manually add it to the rule.