SysFlow 0.5.0

October 19, 2022

Today we announce the release of SysFlow 0.5.0!

What’s Changed?

This is our largest release since the introduction of the processor. This release packages new features, bug fixes, many performance and documentation improvements, k8s metadata and event collection, and a new library for consuming SysFlow records!

LibSysFlow

LibSysFlow is a self-contained C++ library for creating SysFlow consumers. It defines a concise API that returns SysFlow records and manages access to the underlying telemetry driver. The main interface accepts a config object in which a callback function can be set to process SysFlow records. The config option sets optimal defaults that can be customized by the consumer.

Below is a minumum example of a SysFlow consumer that uses LibSysFlow. A more complete example can be found here. The SysFlow Collector also uses LibSysFlow and serves as a reference implementation for library consumers.

// consumer-defined callback function
void process_sysflow(sysflow::SFHeader* header, sysflow::Container* cont, sysflow::Process* proc, sysflow::File* f1, sysflow::File* f2, sysflow::SysFlow* rec) {
    // your switch block here
}

// example consumer
int main(int argc, char **argv) {
    SysFlowConfig* config = sysflowlibscpp::InitializeSysFlowConfig();
    config->callback = process_sysflow;
    sysflowlibscpp::SysFlowDriver *driver = new sysflowlibscpp::SysFlowDriver(config);
    driver->run();
}

The library is packaged as a static (.a) library and distributed as an rpm/deb/tgz artifact with sf-collector releases (both glibc and musl flavors are available). Additionally, libsysflow performs the checks to verify that the Falco driver is loaded and outputs an exception otherwise. Consumers load the Falco libs driver prior to running their main entrypoint, following the typical entrypoint recipe/script used by Falco and the SysFlow Collector. The library also exposes a set of configurable collection modes that enables the consumer to specialize the types of collected system events and meet CPU and memory requirements.

LibSysFlow 0.5.0 builds on Falco Libs 0.9.0.

Musl support

This release includes a build pipeline and LibSysFlow packages glibc and musl libc. The musl packages enables the creation of portable, static SysFlow agents without dependencies with shared libraries and external packages. We currently support x86 and are working on s390x support. Stay tuned!

K8s Metadata and Event Collection

Starting with version 0.5.0, SysFlow records include k8s pod information and k8s events when monitoring Kubernetes environments.

Extended SysFlow Format

To record this additional information, the SysFlow specification was extended with a Pod object that records pod metadata for containers, denoted by the new set of pod.* attributes, and a new k8s event type, which captures and exposes k8s events in the new set of k8s.* attributes.

specification

Policy Engine Extensions

In addition to attributes of the scalar types bool, int(64), and string, the SysFlow Processor now has attributes of type json, which contain structured information in form of stringified JSON records. The policy language allows access to subfields inside such JSON records via GJSON jsonpath expressions. The jsonpath expression must be specified as a suffix to the attribute enclosed in square brackets. Examples of such terms are:

sf.pod.services[0.clusterip.0]: the first cluster IP address of the first service associated with a pod
sf.ke.message[items.0.namespace]: the namespace of the first item in a KE message attribute

See the GJSON path synax for more details. The result of applying a jsonpath expression to a json attribute is always of type string.

Contributing

SysFlow is maturing into a fully-featured ecosystem for cloud-native telemetry and security analytics. We are an inclusive project and welcome contributions from the community! Let’s grow it together!

If you enjoy coding and designing scalable software, and likes the project’s vision, read our contribution guideline and CoC, and consider becoming a contributor. Look for open or help wanted tags in our issue tracker. Commenting and helping us triage git issues are also great ways to help. Help with documentation is welcome. Our projects are written in C++, Go, and Python, so there is room for everyone!

If you are using SysFlow in your projects, academic or otherwise, please shoot us a message on Slack. We would love to hear about it! If you published a research paper or talk in which you used SysFlow, please let us know or send a PR to include the citation in our talks and publications list ;)

Please file bug reports using our git issue tracker.

Connect with us on Slack! If you have any questions about SysFlow, we will be there to help.

Cheers!

– The SysFlow Team ♥

Frederico Araujo