Outputs

An output is an implementation to send the processed event to an external system, like a database, back into Kafka or for example a HTTP endpoint.

Outputs must only process metrics and not perform any processing steps to change the data. Outputs are designed to work with any handler type so it doesn't matter what vendor or data you are ingesting, the output will work regardless.

Output Architecture

Supported Outputs

TimescaleDB

Time-series based database built around PostGRES. This is our preferred output when storing long lived data due to the compression techniques used to be able to store the most amount of logs and is a validated design.

outputs:
  - type: "TimeScaleDBOutput"
    options:
      address: "tsdb"
      port: 5432
      username: "cgnat"
      password: "password123"
      database: "cgnat"
      batch_size: 30000

Name	Description	Default Value
address	TimescaleDB host to connect to.
port	TCP port to connect to.
username	Username for authentication.
password	Password for authentication.
database	Database to connect to.	cgnat
batch_size	How many metrics to batch in a bulk insert.	10000

HTTP

HTTP based webhook to send a HTTP request to a configured endpoint.

outputs:
  - type: "HTTPWebhookOutput"
    options:
      url: "http://webhook/metrics"
      headers:
        x-api-key: default-change-me
      timeout: 10

Name	Description	Default Value
url	URL to send payload to.
headers	Extra headers to add to the HTTP request
timeout	requests Session timeout in seconds	5

Kafka

Kafka Producer Output to send events back into the existing CGN-EC Kafka or a different Kafka after processing the metrics.

outputs:
- type: "KafkaOutput"
    options:
      bootstrap_servers: "localhost:9094"
      topic: "cgnat.events"
      default_topic: "sometopic.all"
      topic_event_map:
        session-mapping: "sometopic.for.sessionmapping"
        address-mapping: "sometopic.for.addressmapping"
      key_field: "x_ip"
      producer_extra_config: {}

Name	Description	Default Value
bootstrap_servers	Kafka Bootstrap Servers.
topic	If populated, all events get sent to this specific topic, can be empty.
default_topic	When used with topic_event_map, any events not captured in the map will be sent to this topic.	cgnat.events
topic_event_map	Used to map specific events (eg. `session-mapping`) to a specific topic.
key_field	Checks if field exist in the metric, and sets the key for the produced event. Typical options are `x_ip`, `x_port`, `src_ip`, `dst_ip` and `dst_port`.
producer_extra_config	Extra configuration to pass into the Kafka Producer (eg. SSL configuration).

Redis

Use Redis Cache to store events and update them in real-time. Typically src_ip is the best field to track in a cache.

outputs:
  - type: "RedisOutput"
    options:
      host: redis
      port: 6379
      key_field: src_ip
      key_ttl: 900
      key_event_map:
        session-mapping: cgnat:events:sessionmapping
      redis_extra_config:
        password: example

| Name | Description | Default Value | | --- | --- | | host | Redis Host. | | port | Redis Port. | 6379 | | key_field | Field to use in the redis key. | | key_ttl | Sets expiration for the key. | | key_event_map | Used to prepend to the key for specific events (eg. session-mapping), by default: cgnat:events:<event_type>:<key_field>. | | redis_extra_config | Extra config to pass into the Redis class. Eg. Auth/TLS/etc |

AMQP

AMQP Producer Output allows you to interact with servers implementing the AMQP protocol, for example like RabbitMQ.

outputs:
  - type: "AMQPOutput"
    options:
      host: "10.4.21.133"
      port: 5672
      username: cgnec
      password: cgnec

Name	Description	Default Value
host	AMQP Host.
port	AMQP Port.	5672
virtual_host	AMQP Virtual Host.
username	AMQP userid (for authentication).
password	AMQP password (for authentication).
exchange	Exchange to use.	cgnat.events
exchange_type	Exchange Type on the server.	topic
routing_key	Routing key to add to the messages.	Uses the `routing_key_event_map` to separate messages by routing keys
default_routing_key	Any NAT event that doesn't match the `routing_key_event_map` will have this routing key assigned.
routing_key_event_map	Used to attach routing keys to the different event types (`NATEventEnum`)
connection_extra_config	Extra config passed into `Connection` class.

Preprocessors

Preprocessors can be attached to an output to perform logic such as filtering specific keys or checking a certain key/value pair exist. Here are some scenarios where you could apply a preprocessor to a relevant output:

You need to filter a few fields and specifically target a src_ip of 100.64.22.13 and the dst_port field must exist, then send this to a HTTP API for further processing. In this scenario you can use the HTTPWebhookOutput output combined with the filter_keys, match_kvs and key_exists preprocessors.
You need to send events for a specific src_ip to a Kafka queue for further processing. In this scenario you can use the KafkaOutput output combined with the match_kvs preprocessor.
You need to temporarily drop any metrics with a specific dst_ip and dst_port. You can use the BlackholeOutput output combined with the match_kvs preprocessor.

Preprocessor config example

outputs:
  - type: "HTTPWebhookOutput"
    options:
      url: "http://webhook/metrics"
      headers:
        x-api-key: default-change-me
      timeout: 10
    preprocessors:
      - name: filter_keys
        arguments:
          keys:
            - src_ip
            - src_port
            - x_ip
            - x_port
            - timestamp
            - dst_ip
            - dst_port
            - event
      - name: match_kvs
        arguments:
          kvs:
            src_ip: 192.168.3.30
      - name: key_exists
        arguments:
          key: dst_port
          ignore_none: true

You can view more details on preprocessors here.