This document is intended for a technical audience and provides a detailed description of Low-Volume Alerts Processing with the ThreatConnect Platform. Low-volume alerts (less than 1000/alerts/day) processing integrations make use of the in-platform Playbooks by way of a Playbook Template to model data handling. Typically, these integrations ingest alerts from remote systems, enrich the provided alert information using the ThreatConnect Platform, and make a decision on whether extended data is stored and users notified for review.
Data Directionality: Inbound - An alert configured in a remote system is fired via Webhook or other mechanism to an incoming listener configured on the ThreatConnect Platform.
ThreatConnect Technologies:
Playbooks (Templates)
Playbook Components
Playbook Apps (App Service)
Execution:
The Playbook Template is typically configured with an HTTP trigger to ingest incoming data.
Deliverables:
Playbook Template (to describe your entire alert-processing scenario)
Playbook App or Component (to provide assistance in processing alert data if necessary)
An event is created in a remote platform.
Event data is appropriately filtered in the remote platform such that the volume fired at the ThreatConnect Platform remains specific and low-volume.
The event payload is prepared in the remote system.
The event payload is fired as a Webhook towards a receiver in the ThreatConnect Platform.
This receiver could be an HTTP Trigger Playbook (Template) or an App Service (which is part of a Playbook Template). The decision on what is used here is best discussed with your ThreatConnect Solutions Engineer.
The event is received in the ThreatConnect Platform as part of a Playbook Template.
Data is processed from the event (JMES Path, etc).
(Optionally) Additional enrichment is requested (such as requesting a document or other body of data regarding the received event).
This is typical when the full context of an event cannot be supplied via the trigger itself.
The work for the additional enrichment should be completed as a separate On-Demand Enrichment integration.
(Optionally) The enrichment request is received by an enrichment source.
(Optionally) The payload of enrichment data is prepared.
(Optionally) The enrichment response is provided.
All enrichment data is processed and typically a decision is made about whether or not to proceed based on known information.
For example, do we have any reason now to continue with these indicators based on other data that we have? If not, we should discontinue. If so, we’ll proceed.
We store the enriched data in the platform (Indicators along with Groups as necessary and according to data modeling).
We fire an in-platform notification to notify ThreatConnect users of information that should be reviewed.
The Playbook logic ends.
Development Environment
Playbooks Templates or Components: This integration is a codeless path and is developed in-platform using the Playbook Designer. Please see this KB article on Playbooks for more information.
Playbook Apps: This integration is developed in-platform using the App Builder. Please see this KB article on App Builder for more information.
It is possible to develop an implementation outside of the platform using a traditional development environment. Please discuss this with your Solutions Engineer if you feel this appropriate for your project.
The latest version of the TcEx v2.0 framework should be used. Specifically, your requirements.txt should look like this: tcex>=2.0.0,<2.1.0
See this set of examples for the TcEx v2.0 framework.
Architecture
General:
Output variable names should follow the schema defined in the References - Playbook Output Variable Naming Schema article.
Playbook Apps provided with this type of integration are specifically designed as App Services triggers.
If additional enrichment is necessary to complete the Playbook Template scenario for your integration, a separate but similar integration of On-Demand Enrichment should be planned.
A main goal of this integration is to show the ability to enrich data using the ThreatConnect Platform. Therefore, CAL or other enrichment data from the ThreatConnect Platform should be used in Playbook Templates that are provided.
Data should be filtered/processed prior to being shipped to the ThreatConnect Platform. The ThreatConnect Platform should not be used for bulk data processing.
This integration type is intended for low-volume applications where less than 1000 alerts per-day are expected on a regular basis.
As part of your overall solution, you may be developing multiple integrations. Your path for work should complete items in the order listed below:
Develop all Playbook Apps
This is important to ensure that the structure of the app is consistent and to ensure that a release version of your app is used in Playbook Templates.
This treats the Playbook App as a single unit and can be released on its own.
Develop all Playbook Components
This is important to ensure that the structure of your component is consistent when used in Playbook Templates.
Develop all Playbook Templates
You must use release versions of Apps in your final work and therefore those must be completed.
Playbook Apps:
The property self.tcex.session_external should be used to retrieve a Python requests object configured with the ThreatConnect proxy settings. This should be used for external HTTP communication to ensure that calls work properly regardless of the environment. The standard Python requests library should not be used.
For TcEx v1: When using this method, it is recommended that you specify a ‘user-agent’ header in order to identify your integration on remote systems. This should include the name and current version of your app.
You can make use of the self.tcex.ij.display_name and self.tcex.ij.program_version parameters to dynamically populate this field.
Configuration
General:
Your integration should accept variable inputs for all parameters (as opposed to any hard-coding of inputs).
Playbook Components and Apps:
Errors: By default, an error in a Playbook will result in the orange path being followed as an error. A non-zero exit code from a Playbook app is considered a failure. Providing an input of Fail on Error allows the opportunity to continue the blue path with a fail and success output count is an alternative and allows the Playbook creator to control the exit behavior.
The exit code 1 should be used when exiting due to a failure.
Fail on No Results: By default, returning no results should result in the orange path being followed as an error. Providing an input of Fail on No Results allows the opportunity to continue the blue path with an output count of how many items were returned is an alternative and allows the Playbook creator to control the exit behavior.
Verifying SSL Certificates: By default, all SSL certificates should be validated. Providing an input of Verify SSL Cert allows the opportunity for the user to decide to disable validation if required for private APIs only. If this option is used, documentation must also clearly state that using validation is a best practice and that disabling this is only appropriate during testing.
NOTE: This option cannot be used for services that hit public APIs as these must be secured appropriately.
Data Mapping
Data brought into the ThreatConnect Platform should be mapped in an exercise with your ThreatConnect Solutions Engineer.
Data should not be stored using Playbook Apps. Storage should only be part of the Playbook Template or Component functionality.
Indicators that are ingested as potentially malicious (but subject to further investigation) should be inserted into the platform with the following characteristics:
Inserted into a specific owner for housing this data
NOTE: This should not be shared with an owner that also contains malicious Indicator records.
A Threat Rating of 1 skull (to indicate that it is suspicious)
A Confidence of 50 (to indicate that it is possibly malicious)
Miscellaneous
General:
You must check your integration for errors and remediate any problems found prior to submitting your integration for vetting.
It is possible to generate in-platform HTML reports via Playbooks using this guide.
Playbook Apps:
Make use of the Labels feature to provide a reasonable number of search terms to help people find your integration within the Platform. These labels might be terms associated with your functionality but not included in the name of your integration. These labels should be specific in nature.
Calls for a timestamp should refer to the utilities provided within the TcEx framework.
See the References - Application Logging Guidelines for specific details around logging.
See the documentation on some of the built-in decorators that we provide to simplify processing in your code.
Make use of the App Note section in the project Metadata tab in App Builder to provide general context for your integration. This should also include a link to your User Documentation as it is displayed in-platform when your component is used.
Make use of the Help Note fields for each input to describe the purpose and expected format(s) for the input.
In particular, the Help Note for your Action field should contain a description of each of the actions at a high-level.
Keep all in-platform notes/descriptions as concise as possible. Reserve extended explanation to your User Documentation.
Playbook Components:
Make use of the Description section in the Component definition to provide context for your integration. This should also include a link to your User Documentation as it is displayed in-platform when your component is used.
Playbook Components and Templates:
Distribution
This integration will be distributed by GitHub. The Partner should publish the integration assets to their own GitHub. Once published, the Partner can submit a pull request for the ThreatConnect GitHub to have the asset published in this repository.
Please create a folder name that reflects your integration’s name (based on the https://threatconnect-techpartners.atlassian.net/wiki/spaces/DP/pages/4587531/Integrations+-+Packaging+and+Solution+Delivery#Integration-Naming guidelines).