Installation of SiLK from the ground up on Centos 7

Here we go with more network analysis tool building. Not content with the Wireshark on Centos post that seems quite popular, I’ve decided to look at some NetFlow . I looked around for good, well supported and documented software.. And I find it in SiLK.

Silk is the System for internet Level Knowledge, and is designed to support large datasets. It provides tools which allow folks like me to analyse, query and summarise network traffic flows. The NetSA team do this from a security perspective. I do it from a requirement to understand network traffic flows in new environments.

The documentation is phenomenal, with a good manual, “Using SiLK for Network Traffic Analysis” as a 171-page colour PDF file. And the Installation Handbook (which I’ve heavily cribbed from) is clear and detailed. It’s built by the Network Situational Awareness (NetSA) team that are part of the US CERT organisation, based out of Carnegie Mellon University.

Preparation

You can find the software yourself at the NetSA SiLK downloads page. I grabbed version 3.16.0, as it was the latest, and I didn’t really want to muck about with the 4.0 beta version. (I’d like my time focused on outputs, and not on debugging). I need NetFlow v9 records, as I’m looking at some IPv6 work using the toolkit as well as .

Looking through that comprehensive documentation I notice in Section 2.3.2 of the Installation Handbook:

“… Many SiLK applications support collecting, storing, and querying flow records that contain IPv6 addresses. Because of the overhead of storing IPv6 addresses, this capability is disabled by default. To enable this behavior, specify the –enable-ipv6 on the configure command line. If you want SiLK to be able to collect IPv6 flow records, you must include support for libfixbuf …”

libfixbuf

We’re going to need some more code.. firstly libfixbuf, but we’re also going to need some other common libraries. But let’s do libfixbuf first..

Firstly we’ll take that downloaded file, extract it, and run ./configure
Now we’ll move a copy of the download file to ~/rpmbuild/SOURCES and then rpmbuild -ba libfixbuf.spec

Now we need to install these local libraries, and build SiLK, and then add these to the that will be on the Internet after this process.

Firstly you’ll need to do an sudo rpm --install ~/rpmbuild/RPMS/x86_64/libfixbuf-devel-1.7.1-1.el7.centos.x86_64.rpm

You’ll need the libfixbuf-devel libraries, as we’re going to be building SiLK which needs access to the headers, as well as the library itself. On final devices, you’ll just need libfixbuf, and not the header information.

Now let’s move on to SiLK

We’ll need to add some libraries for TLS connections between nodes (I’m hoping not to need this, but I’ll build it encase I have a paranoid overlords), and a compression library to shrink the filesizes for and transfer.

So we’ll need to:

sudo  install gnutls-devel lzo-devel

Now to build that SiLK stuff, but first we need to configure it correctly.

Let’s add IPv6 flow support, enable the output file compression, and set the file compatibility to use a format that will store IPv4 and IPv6 information.

Configuring SiLK

The configure command below defaults compression to lzox1 now we’ve added the library, and this applies even to the files output raw from the collector. This is good, as I’m expecting lots of records in the final system.

./configure --enable-ipv6 --enable-ipset-compatibility=3.16.0 --enable-output-compression

...

 * Configured : SiLK 3.16.0
 * Host type: x86_64-pc-linux-gnu
 * Source files ($top_srcdir): .
 * Install directory: /usr/local
 * Root of packed data tree: /data
 * Packing logic: via run-time plugin
 * Timezone support: UTC
 * Default compression method: SK_COMPMETHOD_LZO1X
 * IPv6 network connections: YES
 * IPv6 flow record support: YES
 * IPset file compatibility: SiLK 3.14.0 (record-version=5)
 * IPFIX collection support: YES (-pthread -L/usr/local/lib -lfixbuf -lpthread -lgthread-2.0 -lglib-2.0)
 * NetFlow9 collection support: YES
 * sFlow collection support: YES
 * Fixbuf compatibility: libfixbuf-1.7.1 >= 1.7.0
 * Transport encryption support: YES (-lgnutls)
 * IPA support: NO
 * ZLIB support: YES (-lz)
 * LZO support: YES (-llzo2)
 * SNAPPY support: YES (-lsnappy)
 * LIBPCAP support: YES (-lpcap)
 * C-ARES support: YES (-lcares)
 * ADNS support: NO
 * Python interpreter: /usr/bin/python
 * Python support: NO
 * Build analysis tools: YES
 * Build packing tools: YES
 * Compiler (CC): gcc
 * Compiler flags (CFLAGS): -I$(srcdir) -I$(top_builddir)/src/include -I$(top_srcdir)/src/include -DNDEBUG -D_ALL_SOURCE=1 -D_GNU_SOURCE=1 -Wall -W -Wmissing-prototypes -Wformat=2 -Wdeclaration-after-statement -Wpointer-arith -fno-strict-aliasing -O3
 * Linker flags (LDFLAGS): 
 * Libraries (LIBS): -lsnappy -llzo2 -lz -ldl -lm

Now to create the RPM files (and the supporting repodata directory, as I’m going to publish this too). Make sure you have a copy SiLK downloaded file in ~/rpmbuild/SOURCES

Building SiLK

rpmbuild -ba silk-3.16.0.spec
cd ~/rpmbuild/RPMS/x86_64
rpmsign --resign *
createrepo --update .

I’ve created a SiLK.repo file, which you can download from the repository directory. You’ll need to add this to your /etc/yum.repos.d/ directory. Now one last command and we’ll get it added to our system.

yum install silk-rwflowpack silk-analysis

Yea, an installed bit of code.. Now to get it to accept data.. but that’s a post for later.