sysdig

Sysdig captures system calls and other system level events using a linux kernel facility called tracepoints, providing a rich set of real-time, system-level information.

Sysdig “packetizes” this information, so that you can do things like save it into trace files and easily filter it, a bit like you would do with tcpdump. This makes it very flexible to explore what processes are doing.

Sysdig instruments your physical and virtual machines at the OS level by installing into the Linux kernel and capturing system calls and other OS events. Then, using sysdig’s command line interface, you can filter and decode these events in order to extract useful information. Sysdig can be used to inspect systems live in real-time, or to generate trace files that can be analyzed at a later stage.

####Networking * See the top processes in terms of network bandwidth usage > sysdig -c topprocs_net

  • Show the network data exchanged with the host 192.168.0.1
    > as binary:
    > sysdig -s2000 -X -c echo_fds fd.cip=192.168.0.1
    as ASCII:
    > sysdig -s2000 -A -c echo_fds fd.cip=192.168.0.1

  • See the top local server ports
    > in terms of established connections:
    > sysdig -c fdcount_by fd.sport “evt.type=accept”
    > in terms of total bytes:
    > sysdig -c fdbytes_by fd.sport

  • See the top client IPs
    > in terms of established connections
    > sysdig -c fdcount_by fd.cip “evt.type=accept”
    > in terms of total bytes
    > sysdig -c fdbytes_by fd.cip

  • List all the incoming connections that are not served by apache. > sysdig -p”%proc.name %fd.name” “evt.type=accept and proc.name!=httpd”

####Containers * View the CPU usage of the processes running inside the wordpress1 container > sudo sysdig -pc -c topprocs_cpu container.name=wordpress1

  • View the network bandwidth usage of the processes running inside the wordpress1 container > sudo sysdig -pc -c topprocs_net container.name=wordpress1

  • View the processes using most network bandwidth inside the wordpress1 container > sudo sysdig -pc -c topprocs_net container.name=wordpress1

  • View the top files in terms of I/O bytes inside the wordpress1 container > sudo sysdig -pc -c topfiles_bytes container.name=wordpress1

  • View the top network connections inside the wordpress1 container > sudo sysdig -pc -c topconns container.name=wordpress1

  • Show all the interactive commands executed inside the wordpress1 container > sudo sysdig -pc -c spy_users container.name=wordpress1

####Application * See all the GET HTTP requests made by the machine > sudo sysdig -s 2000 -A -c echo_fds fd.port=80 and evt.buffer contains GET

  • See all the SQL select queries made by the machine > sudo sysdig -s 2000 -A -c echo_fds evt.buffer contains SELECT

  • See queries made via apache to an external MySQL server happening in real time > sysdig -s 2000 -A -c echo_fds fd.sip=192.168.30.5 and proc.name=apache2 and evt.buffer contains SELECT

####Disk I/O * See the top processes in terms of disk bandwidth usage > sysdig -c topprocs_file

  • List the processes that are using a high number of files > sysdig -c fdcount_by proc.name “fd.type=file”

  • See the top files in terms of read+write bytes > sysdig -c topfiles_bytes

  • Print the top files that apache has been reading from or writing to > sysdig -c topfiles_bytes proc.name=httpd

  • Basic opensnoop: snoop file opens as they occur > sysdig -p “%12user.name %6proc.pid %12proc.name %3fd.num %fd.typechar %fd.name” evt.type=open

  • See the top directories in terms of R+W disk activity > sysdig -c fdbytes_by fd.directory “fd.type=file”

  • See the top files in terms of R+W disk activity in the /tmp directory > sysdig -c fdbytes_by fd.filename “fd.directory=/tmp/”

  • Observe the I/O activity on all the files named ‘passwd’ > sysdig -A -c echo_fds “fd.filename=passwd”

  • Display I/O activity by FD type
    > sysdig -c fdbytes_by fd.type

####Processes and CPU usage * See the top processes in terms of CPU usage > sysdig -c topprocs_cpu

  • See the top processes for CPU 0 > sysdig -c topprocs_cpu evt.cpu=0

  • Observe the standard output of a process > sysdig -s4096 -A -c stdout proc.name=cat

####Performance and Errors

  • See the files where most time has been spent > sysdig -c topfiles_time

  • See the files where apache spent most time > sysdig -c topfiles_time proc.name=httpd

  • See the top processes in terms of I/O errors > sysdig -c topprocs_errors

  • See the top files in terms of I/O errors > sysdig -c topfiles_errors

  • See all the failed disk I/O calls > sysdig fd.type=file and evt.failed=true

  • See all the failed file opens by httpd > sysdig “proc.name=httpd and evt.type=open and evt.failed=true”

  • See the system calls where most time has been spent > sysdig -c topscalls_time

  • See the top system calls returning errors > sysdig -c topscalls “evt.failed=true”

  • snoop failed file opens as they occur > sysdig -p “%12user.name %6proc.pid %12proc.name %3fd.num %fd.typechar %fd.name” evt.type=open and evt.failed=true

  • Print the file I/O calls that have a latency greater than 1ms: > sysdig -c fileslower 1

####Security

  • Show the directories that the user “root” visits > sysdig -p”%evt.arg.path” “evt.type=chdir and user.name=root”

  • Observe ssh activity > sysdig -A -c echo_fds fd.name=/dev/ptmx and proc.name=sshd

  • Show every file open that happens in /etc > sysdig evt.type=open and fd.name contains /etc

  • Show the ID of all the login shells that have launched the “tar” command > sysdig -r file.scap -c list_login_shells tar

  • Show all the commands executed by the login shell with the given ID > sysdig -r trace.scap.gz -c spy_users proc.loginshellid=5459

  • Applied use of sysdig for forensics analysis:

User Guide: https://github.com/draios/sysdig/wiki/Sysdig%20User%20Guide

Source code: https://github.com/draios/sysdig

Support website: http://www.sysdig.org/

comments powered by Disqus