Overview

This document is intended for a technical audience and provides a detailed description of On-Demand Enrichment with the ThreatConnect Platform. On-demand enrichment integrations make use of the in-platform Playbooks by way of Components or Playbook Apps. Typically, these integrations are used to enrich information in the platform by communicating with an external system to gather additional detail about a piece of information already stored in the ThreatConnect Platform. This gathered information is then returned as outputs to be used by other Playbook elements.

Integration Attributes

• Data Directionality: Inbound - Data maintained in an external system is typically brought into the ThreatConnect Platform to enrich an existing piece of information.

• ThreatConnect Technologies:

  • Playbooks (Templates)

  • Playbook Components

  • Playbook Apps

• Execution:

  • The Component or App is dropped into a Playbook as part of a custom-designed workflow that may be initiated manually or automatically.

Example Integration Diagram

Open

Diagram Explanation

1. The Component or App (Integration) is triggered as part of a Playbook workflow. Inputs are defined to provide the necessary information to perform the desired action.

2. The Integration initiates an HTTP client call to a remote API to gather data.

3. The remote API receives the request.

4. The remote API processes the request according to any parameters provided.

5. The remote API formulates the response payload.

6. The remote API sends the response back to the HTTP client.

7. The HTTP client receives this response.

8. The Integration parses the response into results appropriate for output.

9. The Integration registers the outputs for the operation.

10. The outputs are made available to downstream items within the originating Playbook’s workflow. Specifically, only a Playbook or Playbook Template should be storing data in the platform. The Playbook App should only be writing outputs for use in this flow.

Integration Key Points

• Development Environment

  • Playbooks Components: This integration is a codeless path and is developed in-platform using the Playbook Designer. Please see this KB article on Playbooks for more information.

  • Playbook Apps: This integration is developed in-platform using the App Builder. Please see this KB article on App Builder for more information.

    • NOTE: App Builder offers the ability to perform a Live Debug of your app. You must ensure that you toggle the Live Debug off once you have completed your debugging session to avoid resource issues on the Sandbox Environment.

    • It is possible to develop an implementation outside of the platform using a traditional development environment. Please discuss this with your Solutions Engineer if you feel this appropriate for your project.

  • The latest version of the TcEx v2.0 framework should be used. Specifically, your requirements.txt should look like this: tcex>=2.0.0,<2.1.0

    • See this set of examples for the TcEx v2.0 framework.

• Architecture

  • General:

    • Output variable names should follow the schema defined in the https://threatconnect-techpartners.atlassian.net/wiki/spaces/DP/pages/4522033 article.

    • For flexibility, the response data from a remote source should be included as an output variable. For example, when retrieving data from a RESTful web service, the raw JSON should be offered back as an output.

    • As part of your overall solution, you may be developing multiple assets. Your path for work should complete items in the order listed below:

      • Develop all Playbook Apps

        • This is important to ensure that the structure of the app is consistent and to ensure that a release version of your app is used in Playbook Templates.

        • This treats the Playbook App as a single unit and can be released on its own.

      • Develop all Playbook Components

        • This is important to ensure that the structure of your component is consistent when used in Playbook Templates.

      • Develop all Playbook Templates

        • You must use release versions of Apps in your final work and therefore those must be completed.

  • Playbook Apps:

    • It is strongly encouraged that you implement Layouts functionality in your app in order to minimize any changes you may have in the future as you add to your integration. See https://threatconnect-techpartners.atlassian.net/wiki/spaces/DP/pages/4554815 for details on how to implement this functionality.

      • NOTE: If your App provides multiple functions, you must use the Layouts functionality in order to structure your inputs and outputs within the Playbooks interface.

    • The property self.tcex.session_external should be used to retrieve a Python requests object configured with the ThreatConnect proxy settings. This should be used for external HTTP communication to ensure that calls work properly regardless of the environment. The standard Python requests library should not be used.

      • For TcEx v1: When using this method, it is recommended that you specify a ‘user-agent’ header in order to identify your integration on remote systems. This should include the name and current version of your app.

        • You can make use of the self.tcex.ij.display_name and self.tcex.ij.program_version parameters to dynamically populate this field.

• Configuration

  • General:

    • Your integration should accept variable inputs for all parameters (as opposed to any hard-coding of inputs).

    • Errors: By default, an error in a Playbook will result in the orange path being followed as an error. A non-zero exit code from a Playbook app is considered a failure. Providing an input of Fail on Error allows the opportunity to continue the blue path with a fail and success output count is an alternative and allows the Playbook creator to control the exit behavior.

      • The exit code 1 should be used when exiting due to a failure.

      • NOTE: This should be evaluated for error conditions that related to the overall success of the response you’re providing from your app. Conditions that prevent your app from working properly (input validation failures, API credential failures) should still produce a failure with an exit code of 1.

    • Fail on No Results: By default, returning no results should result in the orange path being followed as an error. Providing an input of Fail on No Results allows the opportunity to continue the blue path with an output count of how many items were returned is an alternative and allows the Playbook creator to control the exit behavior.

    • Verifying SSL Certificates: By default, all SSL certificates should be validated. Providing an input of Verify SSL Cert allows the opportunity for the user to decide to disable validation if required for private APIs only. If this option is used, documentation must also clearly state that using validation is a best practice and that disabling this is only appropriate during testing.

      • NOTE: This option cannot be used for services that hit public APIs as these must be secured appropriately.

• Data Mapping

  • Data brought into the ThreatConnect Platform should only be provided as an output to your integration and should not be stored directly as Indicators or other objects in most cases.

    • NOTE: Data storage, such as creating Indicators, should take place within the originating Playbook, not the integration. This allows for the most flexibility for an end-user.

      • If you intend to provide an example of data storage, this should be done within a Playbook Template, not within the integration itself.

  • Host and URL type data should be converted into IDNA when you provide data as these output types from your integration.

• Miscellaneous

  • General:

    • Make use of the Labels feature to provide a reasonable number of search terms to help people find your integration within the Platform. These labels might be terms associated with your functionality but not included in the name of your integration. These labels should be specific in nature.

    • NOTE: You must check your integration for errors and remediate any problems found prior to submitting your integration for vetting.

    • It is possible to generate in-platform HTML reports via Playbooks using this guide.

  • Playbook Apps:

    • Calls for a timestamp should refer to the utilities provided within the TcEx framework.

    • See the Application Logging Guidelines for specific details around logging.

    • See the documentation on some of the built-in decorators that we provide to simplify processing in your code.

    • Make use of the App Note section in the project Metadata tab in App Builder to provide general context for your integration. This should also include a link to your User Documentation as it is displayed in-platform when your component is used.

    • Make use of the Help Note fields for each input to describe the purpose and expected format(s) for the input.

      • In particular, the Help Note for your Action field should contain a description of each of the actions at a high-level.

    • Keep all in-platform notes/descriptions as concise as possible. Reserve extended explanation to your User Documentation.

    • Exit codes should follow Playbook App Exit Codes.

  • Playbook Components:

    • Make use of the Description section in the Component definition to provide context for your integration. This should also include a link to your User Documentation as it is displayed in-platform when your component is used.

• Distribution

  • This integration will be distributed by GitHub. The Partner should publish the integration assets to their own GitHub. Once published, the Partner can submit a pull request for the ThreatConnect GitHub to have the asset published in this repository.

    • Please create a folder name that reflects your integration’s name (based on the Integration Naming guidelines).