Integrations - Threat Intelligence Feed Description
Overview
This document is intended for a technical audience and provides a detailed description of a Threat Intelligence Feed with the ThreatConnect Platform. The general purpose of the Threat Intelligence Feed integration is to bring new, external threat intelligence information into the ThreatConnect Platform.
Integration Attributes
Data Directionality: Inbound - Data maintained in an external system is brought into the ThreatConnect Platform typically within a unique source.
ThreatConnect Technologies:
Runtime (Job) Apps - Typically used due to data volumes.
Execution:
Schedule - The Runtime App is executed at a specified interval (usually at least once a day but this may depend on business needs). Information is retrieved from the external system, normalized into the ThreatConnect data model, and stored within the Platform.
Example Integration Diagram
The diagram below provides a simple success-based example of this integration scenario.
Diagram Explanation
A timer event will initiate the execution of this routine based on a schedule.
The Runtime App initiates an HTTP client call to a remote API to gather data.
The remote API receives the request.
The remote API processes the request according to any parameters provided.
The remote API formulates the response payload.
The remote API sends the response back to the HTTP client.
The HTTP client receives this response and parses the data into a usable format.
The Runtime App creates a TcEx Batch to ingest the relevant data provided by the remote API.
The Runtime App submits the TcEx Batch.
(Optionally) The Runtime App will execute any logic to delete stale (invalid) indicators within its scope.
The intel feed is now considered up-to-date.
Integration Key Points
Development Environment
This integration is developed external to the platform using a Python 3.6 environment and the TcEx Framework. Please see the Getting Started section for specific details of how to configure your environment.
The latest version of the TcEx v2.0 framework should be used. Specifically, your requirements.txt should look like this:
tcex>=2.0.0,<2.1.0
See this set of examples for the TcEx v2.0 framework.
Architecture
The template
job_batch
from the TcEx Framework should be used as a starting point for this integration. In particular, the examples in the template for how to submit data, log messages, and other configurations should be used.Incoming indicator feeds must make use of the TcEx Batch module for data manipulation.
See this error guide for common errors you might see using the Batch module.
Supporting differential data updates is optional but desired. Differential data updates is the process whereby only changes are sent regularly after an initial synchronization.
See References - Persistent Input Values for details on how to store a timestamp from run-to-run in your integration.
The property self.tcex.session_external should be used to retrieve a Python requests object configured with the ThreatConnect proxy settings.
This should be used for external HTTP communication to ensure that calls work properly regardless of the environment. The standard Python requests library should not be used.
For TcEx v1: When using this method, it is recommended that you specify a ‘user-agent’ header in order to identify your integration on remote systems. This should include the name and current version of your app.
You can make use of the self.tcex.ij.display_name and self.tcex.ij.program_version parameters to dynamically populate this field.
Configuration
The owner of the data being imported must be configurable. This input parameter must be named
tc_owner
.When you’re testing in the PartnerStage environment, you must use your Source for this field. This should be
<Organization> Source
where<Organization>
is your company name. For example, if your company name is SecuLast, this would beSecuLast Source
.
The log-level for the job must be configurable. The input parameter must be named
tc_log_level
and should be a choice. See References - tc_log_level Input Parameter Configuration for an example.A Feed Deployer configuration should be included in the Runtime App after your initial build is provided to your Solutions Engineer for in-platform testing.
You cannot include Feed Deployer in your initial build for in-platform testing because the steps required to create the Feed Deployer configuration require having a built in-platform already.
See References - Feed Deployer Configuration Example for specific steps.
Data Mapping
Follow the guidance provided in our References - Threat Intelligence Data Mapping document.
Only standard indicator types should be used in this feed. You may not implement custom Indicator types of any kind.
NOTE: The PartnerStage environment does not contain any custom indicators.
Threat Rating and Confidence must be set according to the ThreatConnect best-practices. If the incoming data source does not provide this information, these values must remain configurable and should default to 0.
NOTE: We recommend against putting a blanket confidence or threat rating on all indicators from a source. Instead, provide a configuration parameter for users to configure this behavior only if necessary.
The tag scheme used should only contain the bare words associated with the tag and should not reference the name of the source providing this information.
NOTE: The name of the Owner for the source will provide the appropriate name context.
Victim and Victim Asset data is not a candidate for contribution via Threat Intelligence Feed at this time. See Integrations - Low-Volume Alerts Processing Description for an alternative.
Miscellaneous
Calls for a timestamp should refer to the utilities provided within the TcEx framework.
See References - Application Logging Guidelines for specific details around logging.
The deletion of invalid indicators is optional but desired. If this logic is not available within the integration itself, recommendations on the appropriate Indicator Deprecation rules must be supplied with this integration.
An example of this configuration can be provided.
The management of differential data (deltas) should be done within your integration on the ThreatConnect platform.
This allows an easy reset of this control mechanism if this is desired without requiring a change in a remote system which could lead to unexpected results.
See References - Persistent Input Values for how to store data run-to-run.
The management of any subscription-specific parameters for premium intelligence must be managed by the Partner’s API. This is typically done through unique API keys. The ThreatConnect Platform does not perform any third-party subscription management for feeds.
When making use of the TcEx Framework, it is not necessary to implement any batch controls to limit the number of indicators in a TcEx Batch. The framework will automatically chunk, submit, and monitor batches on your behalf.
NOTE: When you anticipate large amounts of data, it is recommended to implement the Batch.save() method after each object in order to temporarily write data to the disk and reduce memory use.
Batch submissions should be checked for fatal errors. See this link for an example.
NOTE: You must check your integration for errors and remediate any problems found prior to submitting your integration for vetting.
Exit codes should follow Job App Exit Codes.
Distribution
This integration will be distributed by GitHub. The Partner should publish the integration assets to their own GitHub. Once published, the Partner can submit a pull request for the ThreatConnect GitHub to have the asset published in this repository.
Please create a folder name that reflects your integration’s name (based on the Integration Naming guidelines).
Installation
The Threat Intelligence Feed deliverable is installed on the ThreatConnect Platform using the TcExchange Settings page. During testing, this step is performed by your ThreatConnect Solutions Engineer on your behalf as part of our vetting process.
Please see Videos - Threat Intelligence Feed Installation Process for a look at how this process works for the end-user.
Getting Started
Once you’ve reviewed the guidelines above, follow these high-level steps to get started with your project:
Ensure that you’ve installed and are using a Python 3.6 interpreter. This is important to ensure that you match the oldest version of Python used in the ThreatConnect Platform. Python 3.6.8 is recommended.
Install the latest version of the tcex module:
pip3 install 'tcex[development]'
Create a project directory on your system.
NOTE: If you’re using an IDE, do not initialize this directory with your IDE until after you’ve initialized it with the appropriate template in the next step. Otherwise, you’ll receive an error that the directory is not empty.
Change directories into the project directory and prepare a template:
tcinit --template job_batch
Modify the code in app.py. Specifically, your code belongs in the App.run() method.
Ensure that your other project configuration files are up-to-date:
install.json - See this link for reference on this file. Most default values can remain. Key points:
Ensure that your displayName is configured properly per our guidelines.
Ensure that each of your input parameters are defined properly.
tcex.json - Key points:
Ensure that the package > app_name is version of your package name without spaces. Use the _ to substitute for spaces.
For TcEx v1: The package > app_version field will be appended to the package name and doesn’t actually reflect the version embedded in the project.
Any files you add to the project for development but that should not be shipped in the deliverable should be added to the package > excludes array.
args.py - Key Points:
Each argument you add to your app should be included here. You do not need to add any of the pre-defined arguments such as tc_log_level.
requirements.txt - Key Points:
Each package you require for any portion of your app should be specified here.
Prepare the project libraries from the project directory:
tclib
Prepare a run profile/script to test your code. Use the following parameters for your profile/script:
Execute the run.py with the working directory of your project. Use the following arguments:
--tc_api_path
- Set this tohttps://<instance URL>/api
If you’re using the PartnerStage environment, this would be
https://partnerstage.threatconnect.com/api
.If you’re using the ThreatConnect Public Cloud (you access the UI using
https://app.threatconnect.com
), this would behttps://api.threatconnect.com/
.
--api_access_id
- Set this to your API Access ID--api_secret_key
- Set this to your API Secret KeyIf you run this from a
bash
orzsh
command-line, you must single-quote your API Secret Key or it will not work properly (you will get an API 400 response code saying it can’t find the indicator types).If you run this in PyCharm using a run profile, you must double-quote your API Secret Key or it may produce unexpected results inside the interpreter.
--tc_log_path
- Set this to.
to generate the app.log in your working directorySpecify another directory if you desire. All of the exceptions will be captured in this log and will not be printed to the screen.
--tc_log_level
- Set this toDEBUG
for your testing purposes--tc_owner
- Set this to the name of your Source in PartnerStage. This is typically going to be<Organization> Source
as the name. If your company name is SecuLast, this would beSecuLast Source
.This value is only for testing purposes. In the Production environment, you’ll accept a configurable name here in your project (configuration provided in the template).
Also include any other arguments that are defined in the args.py file
For example, if the args.py file looks like the following:
Full run example:
Execute testing against your project to ensure that your code works properly against the description in your Solution Design as well as the guidelines for your integration type.
Ensure that your project is stored in your code repository.
Package the application using ‘tcpackage’. The output will be in ./target by default (a .tcx file).
TCEX Sample Project
Creating a brand new ThreatConnect Job can be overwhelming at first glance and so the Technology Partners Team has created a sample project for reference.
Sample Project Link:
threatconnect-jobs/apps/Malc0de Threat Intelligence at master · ThreatConnect-Inc/threatconnect-jobs
The sample project has a .tcx file which will contain the integration itself and a .pdf which is the user guide.
User Guide
The pdf user guide can be used as a template and the sections include:
Introduction
High level introduction of the integration and products used.
Release Notes
Version of each app release, the release dates and release details
Data Mapping
The Data mapping of used for data and meta data coming into the ThreatConnect platform
Configuration Requirements
Items needed to run the integration
Job App Installation
Installation instructions on how to install the integration on the ThreatConnect Platform
ThreatConnect Job App Configuration
Specific configurations for the job app such as API keys, IDs, parameters, etc…
Using the Integration
Screenshots of how to use the integration inside the ThreatConnect Platform
Support
Contact info for the vendor supporting the integration
TCX Package
The .tcx file is really just a zip file with out special extension. Uncompress it with your preferred zip tool. Inside the .tcx file you will see the following files:
app.py
This sample project will showcase how to utilize many of the common features of the batch module. The main file that most of the code for an integration usually resides in is app.py. The app.py file shown in the sample project will display how to implement certain features of the batch module as shown below:
Setting the owner of indicators/groups:
app.py line 284
Setting indicator threat rating and confidence:
app.py line 288-289
Logging to the app log file using different log levels:
app.py line 294,309
Setting an exit message and how to exit the app:
app.py line 317-318
Getting the current UTC iso8601 time:
app.py line 336-338
Generating a unique XID:
app.py line 340
Creating a group:
app.py line 341
Adding an attribute to a group:
app.py line 342-347
Submitting all indicators/groups to the platform using batch:
app.py line 363-364
Batch Error handling example:
app.py line 362-395
Creating an indicator:
app.py line 411
Adding attributes to an indicator :
app.py line 427
Associating indicators to a threat group:
app.py line 431
Development References