Adding docs for URL Spoofing (#1400)

* Adding docs for urlspoof

* Fixing typo in readme

* Editing documentation to reflect rule upload process

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
This commit is contained in:
dishadasgupta
2021-08-04 17:13:10 -07:00
committed by GitHub
parent d31ea6253e
commit 7be58b7b09
2 changed files with 98 additions and 0 deletions
@@ -6,6 +6,7 @@ This repo contains some additional information and files to use experimental[*](
* [DGA](DGA.md)
* [ProblemChild](problem-child.md)
* [HostRiskScore](host-risk-score.md)
* [URLSpoof](url-spoof.md)
* [experimental detections](experimental-detections.md)
## Releases
@@ -14,6 +15,7 @@ There are separate [releases](https://github.com/elastic/detection-rules/release
* DGA: `ML-DGA-*`
* ProblemChild: `ML-ProblemChild-*`
* Host Risk Score: `ML-HostRiskScore-*`
* URL Spoof: `ML-URLSpoof-*`
* experimental detections: `ML-experimental-detections-*`
Releases will use the tag `ML-TYPE-YYYMMDD-N`, which will be needed for uploading the model using the CLI.
@@ -0,0 +1,96 @@
# URL Spoofing Detection in the Elastic Stack
With the introduction of the ***URL Spoofing*** framework, you can now detect and monitor potentially malicious URLs in your environment.
The framework leverages supervised machine learning methods, threat intelligence enrichments, and customized detection rules to create an alert when you interact with a potentially malicious URL.
*Note: In order to use these ML features, you must have a platinum or higher [subscription](https://www.elastic.co/subscriptions). This is an **experimental** detection capability that currently works with `Packetbeat` data or any index with a corresponding `url.full` field should you choose to use your own index.*
## Detailed Workflow
### 1. Setup enrichment policy
You will first need to setup an enrichment policy to indicate where to get enrichments from.
You can do this by running the following command in your *Dev Tools* console:
```
PUT /_enrich/policy/url_spoofing_enrichment_policy
{
"match": {
"indices": "filebeat-*",
"query": {"match": {"event.dataset": "threatintel.abuseurl"}},
"match_field": "threatintel.indicator.url.domain",
"enrich_fields": ["threatintel.indicator.url.domain"]
}
}
```
*Note: This enrichment pulls in threat intelligence data from `Filebeat`. You must have `Filebeat` data and a corresponding `filebeat-*` index/index pattern.*
### 2. Execute enrichment policy
After setting up the enrichment policy, you will need to execute the policy in order to add enrichments to incoming documents.
Run the following command in your *Dev Tools* console:
```
PUT /_enrich/policy/url_spoofing_enrichment_policy/_execute
```
*Note: You will need to periodically re-execute the enrichment policy to ensure your documents are being enriched with the latest threat intelligence data. To do so, simply re-run the execution script from **Step 2**. Do **NOT** re-run the script from **Step 1**.*
### 3. Upload model and dependencies
Run the following CLI command:
```
python -m detection_rules es <args_or_config> experimental ml setup -t <release-tag>
```
If updating a new model, you should first uninstall any existing models using *remove-model*.
### 4. Update index pipeline configuration
You will need to update your index settings to point to the *URL Spoofing* pipeline.
You can do this by running the following command in your *Dev Tools* console:
```
PUT your-index-pattern/_settings
{
"index": {
"default_pipeline": ml_urlspoof_inference_pipeline
}
}
```
Run the following command in your *Dev Tools* console to stop adding enrichments from the *URL Spoofing* framework to your documents:
```
PUT your-index-pattern/_settings
{
"index": {
"default_pipeline": null
}
}
```
### 5. Refresh your indexes (Optional)
You can optionally choose to refresh your index mapping from within Kibana:
- Navigate to Stack Management > (Kibana) Index Patterns
- Select the appropriate indexes
- Click refresh field list
### 6. Upload detection rule(s)
You can upload the rules associated with the *URL Spoofing* framework using the instructions provided [here](https://github.com/elastic/detection-rules/blob/main/docs/experimental-machine-learning/experimental-detections.md)
And that's it! You should now be alerted whenever you interact with a predicted malicious URL in your environment.