SysFlow 0.4.0

February 18, 2022

Today we announce the release of SysFlow 0.4.0!

What’s Changed?

This release packages new features, many performance and documentation improvements, new TTP tagging rules, and exciting extensions to the SysFlow plugin system.

Binary packages!

Now SysFlow can be deployed directly on debian- and rpm-based hosts using its binary packages!

Binary packages are especially useful in hardened and/or closed environments where visibility is needed but it’s not desirable to run a container orchestrator just to support the monitoring agent. For that reason, we have automated the creation of these packages and systemd service specifications to make controlling the lifecycle of a SysFlow agent (installing, starting, stopping, and removing) easy.

As a quick primer, to obtain the SysFlow 0.4.0 binary packages for a debian-based distro, run:

wget https://github.com/sysflow-telemetry/sf-collector/releases/download/0.4.0/sfcollector-0.4.0-x86_64.deb \
     https://github.com/sysflow-telemetry/sf-processor/releases/download/0.4.0/sfprocessor-0.4.0-x86_64.deb

Then, run:

dpkg -i sfcollector-0.4.0-x86_64.deb sfprocessor-0.4.0-x86_64.deb

This will install the binaries, the systemd service, and the base configuration to run SysFlow (which can be found and adjusted in /etc/sysflow).

Finally, to start the systemd service, run:

sysflow start

Built on Falco Libs

We completed the migration of the SysFlow Collector to build on the Falco libs and drivers. Falco is an incubating CNCF project that boasts an engaged community of developers and adopters who seek to address pressing observability challenges in cloud-native runtime security. We are also active members in this awesome community (you can often find us answering questions in the CNCF slack channel), and we have big plans to contribute back some of the assets we have built in the SysFlow project.

Plugin System Extensions

The SysFlow processor supports custom plugins that can be dynamically loaded into the processing pipeline via shared libraries using the golang plugin package. Custom plugins enable easy extension of the processor and the creation of custom pipelines tailored to specific use cases.

In this release, actions integrate SysFlow’s plugin system, which now supports four types of user-defined plugins:

drivers: enable the ingestion of different telemetry sources into the processor pipeline.
processors: enable the creation of custom data processing and analytic plugins to extend sf-processor pipelines.
handlers: enable the creation of custom SysFlow record handling plugins.
actions: enable the creation of custom action plugins to extend sf-processor’s policy engine.

Actions are pluggable extensions users can create to augment the semantics of rules in the policy engine. Actions are identified by their names and perform additional processing on matched records. They are implemented via the golang plugin mechanism. Implementing a new action is simple, as shown in this complete example. All you need to do is implement the interface below.

// Action interface for user-defined actions
type Action interface {
        GetName() string
        GetFunc() ActionFunc
}

Actions have a name and an action function. Within a single policy engine instance, action names must be unique. User-defined actions cannot re-declare built-in actions. Reusing names of user-defined actions overwrites previously registered actions. Note that the action function receives the current record as an argument and thus has access to all record attributes. The action result can be stored in the record context via the context modifier methods.

This release also introduces a plugin builder image for building and testing user-defined plugins. This plugin builder helps creating consistent and distributable plugins for SysFlow, matching the exact versions of libraries used in the official SysFlow Processor packages. For example, to build an action plugin, you can use docker to run the following command (set $TAG to the desired SysFlow release, edge, or dev).

docker run --rm \
    -v $(pwd)/plugins:/go/src/github.com/sysflow-telemetry/sf-processor/plugins \
    -v $(pwd)/resources:/go/src/github.com/sysflow-telemetry/sf-processor/resources \
    sysflowtelemetry/plugin-builder:$TAG \
    make -C /go/src/github.com/sysflow-telemetry/sf-processor/plugins/actions/example

Note the two volume mounts, which bind the custom-defined plugins sources and resources directories inside the container. Once the docker run command is executed, the plugin is built and the resulting shared object is writted inside the resources directory. Then, to test the plugin, run your pipeline using sf-processor:

docker run --rm \
    -v $(pwd)/plugins:/usr/local/sysflow/plugins \
    -v $(pwd)/resources:/usr/local/sysflow/resources \
    -w /usr/local/sysflow/bin \
    --entrypoint=/usr/local/sysflow/bin/sfprocessor \
    sysflowtelemetry/sf-processor:$TAG \
    -log=quiet -config=../plugins/actions/example/pipeline.actions.json ../resources/traces/tcp.sf

For this example, you will observe that the records matching the rule Action example defined in the policy referenced in pipeline.actions.json will be tagged with the now_in_nanos timestamp generated by the user-defined action.

{
  "version": 4,
  "endts": 0,
  "opflags": [
    "EXEC"
  ],
  ...
  "policies": [
    {
      "id": "Action example",
      "desc": "user-defined action example",
      "priority": 0
    }
  ],
  "tags": [
    "now_in_nanos:1645409122055957900"
  ]
}

Contributing

SysFlow is a young project but we enjoy seeing it mature into a fully-featured ecosystem for cloud-native telemetry and security analytics. We are an inclusive project and welcome contributions from the community!

If you enjoy coding and designing scalable software, and likes the project’s vision, read our contribution guideline and CoC, and consider becoming a contributor. Look for open or help wanted tags in our issue tracker. Commenting and helping us triage git issues are also great ways to help. We are also looking for a documentation czar for the project. Our projects are written in C++, Go, and Python, so there is room for everyone!

If you are using SysFlow in your projects, academic or otherwise, please shoot us a message on Slack. We would love to hear about it! If you published a research paper in which you used SysFlow, please let us know or send a PR to include the citation in our talks and publications list ;)

Please file bug reports using our git issue tracker.

Connect with us on Slack! If you have any questions about SysFlow, we will be there to help.

Cheers!

– The SysFlow Team ♥

Frederico Araujo

What’s Changed?

Binary packages!

Built on Falco Libs

Plugin System Extensions

Contributing