References - Feed Deployer Configuration Example

Owned by John Hall (Unlicensed)
Last updated: Apr 05, 2021
7 min read

Overview

This document is intended for the technical audience building an integration with the ThreatConnect Platform. This document is most applicable to partners building a Threat Intelligence Feed integration and provides specific instructions for how to implement the Feed Deployer functionality.

The Feed Deployer has three main functions:

  • Configuring and creating a feed (source) inside the ThreatConnect Platform that will contain all of the threat intelligence data from the integration

  • Configuring and creating the default ThreatConnect job that will run the integration

  • Create any custom attributes that the integration needs

Follow the steps in this document in the order provided in order to implement this configuration within your integration.

If the application is being deployed for the first time, it is recommended to not include the feed deployer configuration.

Feeds Configuration

Add a new section to the install.json to account for the feed provided (this block goes within the root node):

"feeds": [ { "attributesFile": "attributes.json", "documentStorageLimitMb": <default value>, "enableBulkJson": true, "indicatorLimit": 2000000, "jobFile": "<name>-intel-job.json", "sourceCategory": "Premium", "sourceDescription": "<brief description>", "sourceName": "<feed name>" } ],

Make the appropriate substitutions above as noted below:

  • <default value> should be replaced with the default document storage limit you would expect for your source. If you provide no documents, 10 is a safe default value. Otherwise, you must determine an appropriate value for the information you provide.

  • <name> should be replaced with a lowercase name of your company and/or product name. This filename will be used in later steps and will be distributed with your package.

  • <brief description> should be a 2-3 sentence brief description of the data provided in your feed.

  • <feed name> should be the name of your feed product as it will appear in the ThreatConnect interface.

“attributesFile” and the value are only required if you require custom attributes for your data.

See this documentation for more details on each of these parameters within this section.

Feed Deployer Parameters

Within your install.json file, update each parameter you have with the parameter “feedDeployer” and a value of true for each parameter that should appear within the Feed Deployer wizard screen. An example parameter is provided below:

"params": [ { "label": "Feed Type", "name": "feed_type", "note": "The type of feed you want to ingest into your ThreatConnect platform.", "required": true, "type": "MultiChoice", "feedDeployer": true, "validValues": [ "C2Addresses", "C2Hosts", "Files" ] },

Do not mark the parameter tc_owner as true for Feed Deployer as you will be hard-coding this value in several places.

Creating the Default Job Configuration

Once your app has been installed for testing (without Feed Deployer components), manually create a job with all of the parameters that you expect to be provided. Follow these steps to create your job:

  1. From the top menu of the system, select the Cogwheel icon and select Org Settings.

  2. On the Org Settings screen that appears, select Apps from the available tabs:

  3. On the Apps screen, select the Plus icon on the right-hand side:

  4. You will be presented with the Add Job wizard. Complete the parameters in the wizard with the default values that should be used for production executions of your feed with this guidance:

    1. Most jobs should run once per day (don’t worry about the time as this will be overridden when the job is deployed).

    2. If your job requires running more frequently, you can configure it to run every X hours. The maximum frequency that your job should run is every 2 hours. This should be based on how frequently your data sources are updated.

  5. Once the job has been completed, execute the job as a test by clicking the Play icon in the job listing next to the job. It is not necessary to activate the job in order to manually run it.

  6. If your job successfully executes, remove any notification parameters you may have configured and then export the job file using the dots menu and the Export Job function:

  7. Save the file that is produced as the name you specified earlier in your install.json file (<name>-intel-job.json). Include this file in your package alongside your install.json file.

Within the example you created, edit the file and update the value of the owner parameter with the same name of your feed as defined in the sourceName field for your feeds block within install.json. Add the parameter “preventUpdates” and a value of true.

An example of this configuration is below for reference only:

{ "programName": "SecuLast Malware Threat Intelligence v1.0", "programVersion": "1.0.2", "jobName": "Do-Not-Copy-This-Example-File", "allowOnDemand": true, "publishAuth": false, "scheduleType": "Daily", "scheduleCronFormat": "0 0 19-23/2,0-18/2 ? * *", "scheduleStartDate": 1571756361381, "enableNotifications": false, "notifyOnFailure": false, "notifyOnPartialFailure": false, "notifyOnComplete": false, "notifyEmail": "", "notifyIncludeLogFiles": false, "params": [ { "name": "tc_owner", "preventUpdates": true, "default": "SecuLast Malware Threat Intelligence" }, { "name": "feed_type", "preventUpdates": false, "default": "C2Addresses|C2Hosts|Files" }, { "name": "api_key", "encrypt": true, "preventUpdates": false }, { "name": "tc_log_level", "preventUpdates": false, "default": "warning" } ] }

Custom Attributes

Follow the steps in the References - Creating Custom Attributes document in order to create a new custom attributes JSON file to include with your integration. This file must be named attributes.json when included in your integration and should live in the same folder as the install.json file.