diff --git a/docs/experimental-machine-learning/host-risk-score.md b/docs/experimental-machine-learning/host-risk-score.md index 55154e217..32abbafc8 100644 --- a/docs/experimental-machine-learning/host-risk-score.md +++ b/docs/experimental-machine-learning/host-risk-score.md @@ -1,105 +1,192 @@ -# Host Risk Score- What is it? +# Host Risk Score -The Host Risk Score package consists of all the artifacts required to stand up the host risk scoring framework in your environment. This framework leverages transforms and visualizations in Kibana to identify the most suspicious hosts in your environment, based on alert activity on the hosts. -To deploy this framework in your environment, follow the steps outlined below. +Host Risk Score is an experimental feature that assigns risk scores to hosts in a given Kibana space. Risk scores are calculated for each host by utilizing transforms on the alerting indices. The transform runs hourly to update the score as new detection rule alerts are generated. The Host Risk Score [package](https://github.com/elastic/detection-rules/releases) contains all of the required artifacts for setup. The Host Risk Score feature provides drilldown Lens dashboards and additional Kibana features such as the **Host Risk Score Card** on the Overview page of the Elastic Security app, and the **Host Risk Keyword** on the Alert details flyout for an enhanced experience. -# Detailed steps +### Notes + - **Host name collision**: Hosts are identified by the `host.name` field in alerts. There may be some edge cases where different hosts use the same name. [details](#host-name-collision-details) -#### 1. Unzip the release bundle +## Setup Instructions -Navigate to the latest GitHub [release](https://github.com/elastic/detection-rules/releases)), with the tag `ML-HostRiskScore-YYYMMDD-N`. From under `Assets`, download the zip file named `ML-HostRiskScore-YYYMMDD-N.zip` and unzip it. New releases may contain updated artifacts. + 1. [Obtain artifacts](#obtain-artifacts) + 2. [Upload scripts](#upload-scripts) + 3. [Upload ingest pipeline](#upload-ingest-pipeline) + 4. [Upload and start the `pivot` transform](#upload-start-pivot) + 5. [Create the Host Risk Score index](#host-risk-index) + 6. [Upload and start the `latest` transform](#upload-start-latest) + 7. [Import dashboards](#import-dashboards) + 8. [Enable Kibana features](#enable-kibana) -#### 2. Modify artifacts to reflect Kibana space +

1. Obtain artifacts

-For security reasons, we require that you restrict the Host Risk Score application to a specific Kibana space. In order to do so, run the `ml_hostriskscore_generate_scripts.py` script in the directory that is prefixed with `ML-HostRiskScore` with your Kibana space as the argument. +The Host Risk Score functionality is space aware for privacy. Downloaded artifacts must be modified with the desired space before they can be used. -Eg: + - Download the latest release [bundle](https://github.com/elastic/detection-rules/releases) with the tag `ML-HostRiskScore-YYYMMDD-N`. + - Unzip the contents of `ML-HostRiskScore-YYYMMDD-N.zip`. + - Run `ml_hostriskscore_generate_scripts.py` script in the unzipped directory with your Kibana space as the argument. +
+Example of modifying artifacts for the default space + 
python ml_hostriskscore_generate_scripts.py --space default
+
-``` -python ml_hostriskscore_generate_scripts.py --space default -``` + - Find a new folder named after your space in the unzipped directory. **You will be using the scripts within this directory for the next steps.** -This will create a folder named ``, consisting of all the required artifacts, modified for your Kibana space. **For the steps that follow, please use the scripts from this folder.** +

2. Upload scripts

-#### 3. Navigate to the Dev Tools console in Kibana +- Navigate to `Management / Dev Tools` in Kibana. +- Upload the contents of `ml_hostriskscore_levels_script.json`, `ml_hostriskscore_map_script.json` and `ml_hostriskscore_reduce_script.json` using the Script API with the following syntax. +- Ensure that your space name (such as `default`) replaces `` in the script names below. -You will now upload all the Host Risk Score artifacts to Kibana. Remember to suffix the names of all the scripts, ingest pipelines, transforms etc. with your Kibana space, while uploading them. -#### 4. Uploading required scripts +
+uploading scripts + 

+PUT _scripts/ml_hostriskscore_levels_script_<your-space-name>
+{contents of ml_hostriskscore_levels_script.json file}
+
-Upload the contents of `ml_hostriskscore_levels_script.json`, `ml_hostriskscore_map_script.json` and `ml_hostriskscore_reduce_script.json` as individual scripts, using the Script API. - -Eg: - -``` -PUT _scripts/ml_hostriskscore_levels_script_ -{content of the ml_hostriskscore_levels_script.json file} -``` - -#### 5. Upload required ingest pipelines - -Upload the ingest pipeline in `ml_hostriskscore_ingest_pipeline.json` using the following API call: +
+ 

+PUT _scripts/ml_hostriskscore_map_script_<your-space-name>
+{contents of ml_hostriskscore_map_script.json file}
+
-``` -PUT _ingest/pipeline/ml_hostriskscore_ingest_pipeline_ -{content of the ml_hostriskscore_ingest_pipeline.json file} -``` - -#### 6. Upload and start the `pivot` transform - -Upload the `pivot` transform in `ml_hostriskscore_pivot_transform.json` using the following API call. This transform calculates the risk level per hour for each host in the Kibana space specified in Step 2: +
+ 

+PUT _scripts/ml_hostriskscore_reduce_script_<your-space-name>
+{contents of ml_hostriskscore_reduce_script.json file}
+
-``` -PUT _transform/ml_hostriskscore_pivot_transform_ -{content of the ml_hostriskscore_pivot_transform.json file} -``` +

3. Upload ingest pipeline

-* Navigate to `Transforms` under `Management` -> `Stack Management`. For the transform with the ID `ml_hostriskscore_pivot_transform_`, under `Actions`, click `Start`. -* Verify that the Transform started as expected by ensuring that documents are appearing in the destination index of the Transform, eg: using the Search/Count APIs: +- Upload the contents of `ml_hostriskscore_ingest_pipeline.json` using the Ingest API with the following syntax. +- Ensure that your space name (such as `default`) replaces `` below. + +
+uploading ingest pipeline + 
PUT _ingest/pipeline/ml_hostriskscore_ingest_pipeline_<your-space-name>
+{contents of ml_hostriskscore_ingest_pipeline.json file}
+
-``` -GET ml_host_risk_score_/_search (or _count) -``` -#### 7. Create the space-aware `ml_host_risk_score_latest_` index with appropriate mappings +

4. Upload and start the pivot transform

-``` -PUT ml_host_risk_score_latest_ +This transform calculates the risk level every hour for each host in the Kibana space specified. + +- Upload the contents of `ml_hostriskscore_pivot_transform.json` using the Transform API with the following syntax. +- Ensure that your space name (such as `default`) replaces `` below. + +
+uploading pivot transform + 
PUT _transform/ml_hostriskscore_pivot_transform_<your-space-name>
+{contents of ml_hostriskscore_pivot_transform.json file}
+
+ +- Navigate to `Transforms` under `Management / Stack Management` in Kibana. Find the transform with the ID `ml_hostriskscore_pivot_transform_`. Open the `Actions` menu on the right side of the row, then click `Start`. +- Confirm the transform is working as expected by navigating to `Management / Dev Tools` and ensuring the target index exists. + +
+sample test query + 
GET ml_host_risk_score_<your-space-name>/_search
+
+ +

5. Create the Host Risk Score index

+ +- Navigate to `Management / Dev Tools` in Kibana. +- Create the Host Risk Score index (`ml_host_risk_score_latest_`) with the following mappings. +- Ensure that your space name (such as `default`) replaces `` below. + +
+creating the Host Risk Score index + 
PUT ml_host_risk_score_latest_<your-space-name>
 {
-    "mappings" : {
-            "properties" : {
-                "host.name" : { "type" : "keyword" }
-            }
-        }
+  "mappings":{
+    "properties":{
+      "host.name":{
+        "type":"keyword"
+      }
+    }
+  }
 }
-```
+
-#### 8. Upload the `latest` transform +

6. Upload and start the latest transform

-Upload the `latest` transform in `ml_hostriskscore_latest_transform.json` using the following API call. This transform gets the most current risk levels for all the hosts in the Kibana space specified in Step 2: +This transform recurringly calculates risk levels for all hosts in the Kibana space specified. +- Upload the contents of `ml_hostriskscore_latest_transform.json` using the Transform API with the following syntax. +- Ensure that your space name (such as `default`) replaces `` below. + +
+uploading latest transform + 
PUT _transform/ml_hostriskscore_latest_transform_<your-space-name>
+{contents of ml_hostriskscore_latest_transform.json file}
+
+ +- Navigate to `Transforms` under `Management / Stack Management` in Kibana. Find the transform with the ID `ml_hostriskscore_latest_transform_`. Open the `Actions` menu on the right side of the row, and click `Start`. +- Confirm the transform is working as expected by navigating to `Management / Dev Tools` and ensuring the target index exists. You should see documents starting to appear in the index if there is ongoing alerting activity associated with hosts. + +
+sample test query + 
GET ml_host_risk_score_latest_<your-space-name>/_search
+
+ +

7. Import dashboards

+ +- Navigate to `Management / Stack Management / Kibana / Saved Objects` in Kibana. +- Click on `Import` and import the `ml_hostriskscore_dashboards.ndjson` file. +- Navigate to `Analytics / Dashboard`. +- Confirm you can see a dashboard named `Current Risk Scores for Hosts`, which displays the current list (Top 20) of suspicious hosts in your environment. +- Confirm you can see a dashboard named `Drilldown of Host Risk Score`, which allows you to further drill down into details of the risk associated with a particular host of interest. + +

8. Enable Kibana features

+ +There are two features in Kibana related to Host Risk Scoring. + +_Host Risk Score card on the Overview page_ +![Host Risk Score card](./images/0a_host_risk_score_card.png) + +_Host Risk Keyword on Alert Details Flyout_ +![Host Risk Keyword](./images/0b_alert_summary.png) + +To enable the _Host Risk Score card on the Overview page_, and the _Host Risk Keyword on Alert Details Flyout_, you will need to add the following configuration to `kibana.yml`. ``` -PUT _transform/ml_hostriskscore_latest_transform_ -{content of the ml_hostriskscore_latest_transform.json file} +xpack.securitySolution.enableExperimental: ['riskyHostsEnabled'] ``` -* Navigate to `Transforms` under `Management` -> `Stack Management`. For the transform with the ID `ml_hostriskscore_latest_transform_`, under `Actions`, click `Start`. -* Verify that the Transform started as expected by ensuring that documents are appearing in the destination index of the Transform, eg: using the Search/Count APIs: +#### Instructions to modify `kibana.yml` on Elastic Cloud + +1. Navigate to your deployment on the cloud +![Navigate to deployment](./images/1_create_deployment.png) -``` -GET ml_host_risk_score_latest_/_search (or _count) -``` +2. Click on Kibana on the sidebar and click on Edit configuration on your Kibana instance +![Edit Kibana config](./images/2_edit_configuration.png) -#### 9. Import the dashboards -* Navigate to `Management` -> `Stack Management` -> `Kibana` -> `Saved Objects` -* Click on `Import` and import the `ml_hostriskscore_dashboards.ndjson` file -* Navigate to `Analytics` -> `Dashboard`. You should see two dashboards- `Current Risk Scores for Hosts`, which displays the current list (Top 20) of suspicious hosts in your environment, and `Drilldown of Host Risk Score`, which allows you to further drill down into details of the risk associated with a particular host of interest. +3. Click on Edit user settings +![Edit user settings](./images/3_edit_user_settings.png) -# About hostnames and host IDs -The Host Risk Score app currently uses host names (`host.name`), not the `host.id` field, for both searching and displaying hosts. There may be some edge cases where hosts use the same name. Physical Windows clients - desktops and laptops - in an Active Directory forest, are unlikely to have name collisions, as their computer accounts and distinguished names should be unique. Non-domain member servers, desktops and laptops, in a Windows workgroup, may occasionally have name collisions. Macs are often not managed by a directory service and may have name collisions. Virtual servers, that are created from templates or cloning processes may have hostname collisions. +4. Modify Kibana configuration by adding `xpack.securitySolution.enableExperimental: ['riskyHostsEnabled']` +![Modify Kibana configuration](./images/4_add_flag.png) + + +5. Save updated Kibana settings +![Save updated Kibana settings](./images/5_save_settings.png) + + +6. Confirm activity finished +![Confirm activity finished](./images/6_confirm_activity_finished.png) + + +7. View Host Risk Score Card on the Overview page +![Host Risk Score card](./images/0a_host_risk_score_card.png) + +
+ +##### Host name collision details + +Physical Windows clients - desktops and laptops - in an Active Directory forest are unlikely to have name collisions, as their computer accounts and distinguished names should be unique. Non-domain member servers, desktops and laptops, in a Windows workgroup, may occasionally have name collisions. Macs are often not managed by a directory service and may have name collisions. Virtual servers, that are created from templates or cloning processes may have hostname collisions. \ No newline at end of file diff --git a/docs/experimental-machine-learning/images/0a_host_risk_score_card.png b/docs/experimental-machine-learning/images/0a_host_risk_score_card.png new file mode 100644 index 000000000..13a0a3524 Binary files /dev/null and b/docs/experimental-machine-learning/images/0a_host_risk_score_card.png differ diff --git a/docs/experimental-machine-learning/images/0b_alert_summary.png b/docs/experimental-machine-learning/images/0b_alert_summary.png new file mode 100644 index 000000000..f985340ff Binary files /dev/null and b/docs/experimental-machine-learning/images/0b_alert_summary.png differ diff --git a/docs/experimental-machine-learning/images/1_create_deployment.png b/docs/experimental-machine-learning/images/1_create_deployment.png new file mode 100644 index 000000000..e4c574daa Binary files /dev/null and b/docs/experimental-machine-learning/images/1_create_deployment.png differ diff --git a/docs/experimental-machine-learning/images/2_edit_configuration.png b/docs/experimental-machine-learning/images/2_edit_configuration.png new file mode 100644 index 000000000..27071ee5a Binary files /dev/null and b/docs/experimental-machine-learning/images/2_edit_configuration.png differ diff --git a/docs/experimental-machine-learning/images/3_edit_user_settings.png b/docs/experimental-machine-learning/images/3_edit_user_settings.png new file mode 100644 index 000000000..76443e4e4 Binary files /dev/null and b/docs/experimental-machine-learning/images/3_edit_user_settings.png differ diff --git a/docs/experimental-machine-learning/images/4_add_flag.png b/docs/experimental-machine-learning/images/4_add_flag.png new file mode 100644 index 000000000..9d5600553 Binary files /dev/null and b/docs/experimental-machine-learning/images/4_add_flag.png differ diff --git a/docs/experimental-machine-learning/images/5_save_settings.png b/docs/experimental-machine-learning/images/5_save_settings.png new file mode 100644 index 000000000..bc9c888bc Binary files /dev/null and b/docs/experimental-machine-learning/images/5_save_settings.png differ diff --git a/docs/experimental-machine-learning/images/6_confirm_activity_finished.png b/docs/experimental-machine-learning/images/6_confirm_activity_finished.png new file mode 100644 index 000000000..8c1abe21e Binary files /dev/null and b/docs/experimental-machine-learning/images/6_confirm_activity_finished.png differ