Troubleshooting with lsof

Often I am asked to help troubleshoot various application running in a client’s environment that I have never run before. One of my front pocket tools is lsof.

What process is listening on port 443?

$ sudo lsof -s TCP:LISTEN -i :443
COMMAND  PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
nginx   1520     root    8u  IPv4  34742      0t0  TCP *:https (LISTEN)
nginx   1521 www-data    8u  IPv4  34742      0t0  TCP *:https (LISTEN)
nginx   1522 www-data    8u  IPv4  34742      0t0  TCP *:https (LISTEN)
nginx   1523 www-data    8u  IPv4  34742      0t0  TCP *:https (LISTEN)
nginx   1524 www-data    8u  IPv4  34742      0t0  TCP *:https (LISTEN)

In this case -s TCP:LISTEN limits the request to only those files that are open and listening TCP sockets, -i specifies which host:port we should be attempting to discover. If the host is unspecified (similar to above) lsof will return any and all hosts.

What connections does a machine have outgoing?

$ sudo lsof -s TCP:ESTABLISHED  -i :22
COMMAND   PID  USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
ssh     28574 chris    3u  IPv4 3220641      0t0  TCP tamarin:46054->gorilla:ssh (ESTABLISHED)
ssh     28669 chris    3u  IPv4 3205758      0t0  TCP tamarin:42238->69-55-236-12.in-addr.arpa.johncompanies.com:ssh (ESTABLISHED)
ssh     28949 chris    3u  IPv4 3220810      0t0  TCP tamarin:47984->ec2-54-67-72-149.us-west-1.compute.amazonaws.com:ssh (ESTABLISHED)
ssh     29245 chris    3u  IPv4 3237031      0t0  TCP tamarin:46240->gorilla:ssh (ESTABLISHED)
sshd    29248  root    4u  IPv4 3236182      0t0  TCP tamarin:ssh->gorilla:27862 (ESTABLISHED)
sshd    29357 chris    4u  IPv4 3236182      0t0  TCP tamarin:ssh->gorilla:27862 (ESTABLISHED)

Here we are looking for any established connections going out to any ssh port as well as any incoming connections. Please note the commands outgoing are ssh and incoming are sshd

Where is this process connecting to for data?

$ sudo lsof -p 15674 -a -i 4 -s TCP:ESTABLISHED

COMMAND     PID  USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
thunderbi 15674 chris   71u  IPv4 2875192      0t0  TCP tamarin:48396->pj-in-f109.1e100.net:imaps (ESTABLISHED)
thunderbi 15674 chris   95u  IPv4 2877297      0t0  TCP tamarin:52020->gorilla:imaps (ESTABLISHED)
thunderbi 15674 chris   96u  IPv4 2875702      0t0  TCP tamarin:47828->wj-in-f16.1e100.net:imaps (ESTABLISHED)
thunderbi 15674 chris   98u  IPv4 2883720      0t0  TCP tamarin:51966->gorilla:imaps (ESTABLISHED)
thunderbi 15674 chris   99u  IPv4 2883721      0t0  TCP tamarin:51968->gorilla:imaps (ESTABLISHED)
thunderbi 15674 chris  101u  IPv4 2875703      0t0  TCP tamarin:47830->wj-in-f16.1e100.net:imaps (ESTABLISHED)
thunderbi 15674 chris  105u  IPv4 2875825      0t0  TCP tamarin:51982->gorilla:imaps (ESTABLISHED)
thunderbi 15674 chris  116u  IPv4 2877300      0t0  TCP tamarin:52026->gorilla:imaps (ESTABLISHED)

You can see here that my Thunderbird process (pid: 15674) is connecting to both gorilla and Gmail’s secure IMAP servers. -a ands together the states of TCP established connections and -i 4 limits us to IPv4 connections

Where is this application logging?

$ sudo lsof -p 1520 |grep log
nginx   1520 root    2w   REG               8,21        0 11667029 /var/log/nginx/error.log
nginx   1520 root    4w   REG               8,21  1075419 11665462 /var/log/nginx/access.log
nginx   1520 root    5w   REG               8,21        0 11667029 /var/log/nginx/error.log

Here we supply the PID of the application and list all open files for that specific process and simply grep for log. Running this without the grep will show all files open by this application.

Conclusion

Once you are able to discern where an unknown application is receiving data from (established connections) and where the application may be writing log files, it should be much easier to troubleshoot after using lsof.