Log management: Using syslog-ng

Introduction

Goals

Notes


Exercises

Please find your classmates that are using the same router as you. Get in to a group and do the following exercise together. That is, pick one person who will log in to your group's router, but all of you should assist with the actual configuration.

Configure your routers to send syslog messages to your shared server (srv1-gY):

Configure network devices below to send logs to srv1-gY.lab.workalaya.net (100.68.Y.254)

You will SSH to your group's network devices and do the following:

lab@vmX-gY:~$ ssh lab@rtr1-gY.lab.workalaya.net rtr1-gY> enable rtr1-gY# config terminal

Repeat the next command "logging 100.68.Y.254" for each network device in your campus group.

rtr1-gY(config)# logging 100.68.Y.254 rtr1-gY(config)# logging facility local0 rtr1-gY(config)# logging userinfo rtr1-gY(config)# exit rtr1-gY# write memory

Now run 'show logging' to see the summary of the logging configuration.

rtr1-gY# show logging

Logout from the router (exit)

rtr1-gY# exit

That's it. The network device should now be sending UDP SYSLOG packets to your shared server (srv1-gY) on port 514. To verify this log in on your shared server:

lab@vmX-gY:~$ ssh lab@srv1-gY.lab.workalaya.net

and do the following:

rancid@srv1-gY:~$ sudo tcpdump -s0 -nv -i eth0 port 514

Then have one person in your group log back in on the network device and do the following:

$ ssh lab@rtr1-gY.lab.workalaya.net rtr1-gY> enable rtr1-gY# config terminal rtr1-gY(config)# exit rtr1-gY# exit

You should see some output on your shared server's (srv1-gY) screen from TCPDUMP. It should look something like:

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 14:05:02.620767 IP (tos 0x0, ttl 254, id 43, offset 0, flags [none], proto UDP (17), length 215) 100.68.6.1.62222 > 100.68.6.254.514: [udp sum ok] SYSLOG, length: 187 Facility local0 (16), Severity notice (5) Msg: 466: Feb 22 14:05:01.545: %SSH-5-SSH2_USERAUTH: User 'lab' authentication for SSH2 Session from 100.64.0.241 (tty = 2) using crypto cipher 'aes128-cbc', hmac 'hmac-sha1' Succeeded

Now you can configure the logging software on your shared server (srv1) to receive this information and log it to a new set of files.


Install syslog-ng using ansible

Now login back to ansible control host.

Create a new ansible playbook named syslog.yml to install syslog-ng and configure syslog-ng

This ansilbe playbook will perform:

(venv) vmX-gY@ansible-host:~/ansible-playbook$ vi syslog.yml
- hosts: syslog_hosts become: true tasks: - name: ensure package cache is up to date apt: update_cache=yes cache_valid_time=3600 tags: install - name: install syslog-ng package: name: "{{ item }}" state: present with_items: - syslog-ng - syslog-ng-core tags: install - name: create /var/log/remote-syslog directory file: path: /var/log/remote-syslog state: directory tags: config - name: enable remote syslog settings copy: src: files/syslog-ng/remote-log.conf dest: /etc/syslog-ng/conf.d/ notify: restart syslog-ng tags: config handlers: - name: restart syslog-ng service: name: syslog-ng state: restarted

update inventory/hosts as following

(venv) vmX-gY@ansible-host:~/ansible-playbook$ vi inventory/hosts
[nagios_hosts] vmX-gY.lab.workalaya.net [snmp_hosts] vmX-gY.lab.workalaya.net [smokeping_hosts] vmX-gY.lab.workalaya.net [rancid_hosts] vmX-gY.lab.workalaya.net [nfsen_hosts] vmX-gY.lab.workalaya.net srv1-gY.lab.workalaya.net [syslog_hosts] srv1-gY.lab.workalaya.net

create syslog-ng config file name remote-log.conf inside file/syslog-ng

(venv) vmX-gY@ansible-host:~/ansible-playbook$ mkdir -p files/syslog-ng (venv) vmX-gY@ansible-host:~/ansible-playbook$ vi files/syslog-ng/remote-log.conf

copy and paste folowing content and save

source s_net { udp(ip(0.0.0.0) port(514)); tcp(ip(0.0.0.0) port(514)); }; destination remote-log { file("/var/log/remote-syslog/${HOST}/${YEAR}/${MONTH}/${DAY}/${FACILITY}.log" owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes) ); }; log { source(s_net); destination(remote-log); };

Now run ansible playbook to install syslog-ng

(venv) vmX-gY@ansible-host:~/ansible-playbook$ ansible-playbook syslog.yml PLAY [syslog_hosts] ***************************************************************************************************************************************** TASK [Gathering Facts] ************************************************************************************************************************************** ok: [vmX-gY.lab.workalaya.net] TASK [ensure package cache is up to date] ******************************************************************************************************************* [WARNING]: Could not find aptitude. Using apt-get instead changed: [vmX-gY.lab.workalaya.net] TASK [install syslog-ng] ************************************************************************************************************************************ changed: [vmX-gY.lab.workalaya.net] => (item=syslog-ng) ok: [vmX-gY.lab.workalaya.net] => (item=syslog-ng-core) TASK [create /var/log/remote-syslog directory] ************************************************************************************************************** changed: [vmX-gY.lab.workalaya.net] TASK [enable remote syslog settings] ************************************************************************************************************************ changed: [vmX-gY.lab.workalaya.net] RUNNING HANDLER [restart syslog-ng] ************************************************************************************************************************* changed: [vmX-gY.lab.workalaya.net] PLAY RECAP ************************************************************************************************************************************************** vmX-gY.lab.workalaya.net : ok=6 changed=5 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Test syslog

To be sure there are some logging messages log back in to the network device, and run some "config" commands, then logout. e.g.

$ ssh lab@rtr1-gY.lab.workalaya.net rtr1-gY> enable rtr1-gY# config terminal rtr1-gY(config)# exit rtr1-gY# exit

Be sure you log out of the network device. If too many people log in without logging out then others cannot gain access to the network device.


On your shared server (srv1), See if messages are starting to appear under /var/log/network/remote-log/

lab@srv1-g1:~$ cd /var/log/remote-syslog/ lab@srv1-g1:/var/log/remote-syslog$ ls 100.68.1.1 100.68.1.21 100.68.1.22 100.68.1.23 ... above will show you the directory showing all remote hosts lab@srv1-g1:/var/log/remote-syslog$ cd 100.68.1.1/ lab@srv1-g1:/var/log/remote-syslog/100.68.1.1$ ls 2019 ... above will show you the directory for the year lab@srv1-g1:/var/log/remote-syslog/100.68.1.1$ cd 2019/ lab@srv1-g1:/var/log/remote-syslog/100.68.1.1/2019$ ls 10 11 ... above will show you the directory for the month lab@srv1-g1:/var/log/remote-syslog/100.68.1.1/2019$ cd 11 lab@srv1-g1:/var/log/remote-syslog/100.68.1.1/2019/11$ ls 22 23 24 ... above will show you the day of the month lab@srv1-g1:/var/log/remote-syslog/100.68.1.1/2019/11$ cd 24 lab@srv1-g1:/var/log/remote-syslog/100.68.1.1/2019/11/24$ ls local0.log local7.log ... above will show you the router's log

You can view the resulting log file by using a pager program such as less, more, cat, etc...

You will see the logging is capture like below;

lab@srv1-g1:/var/log/remote-syslog/100.68.1.1/2019/11/24$ less local0.log Nov 24 18:49:32 100.68.1.1 %SYS-5-CONFIG_I: Configured from console by lab on vty0 (100.68.1.21) Nov 24 18:51:06 100.68.1.1 %SYS-5-CONFIG_I: Configured from console by lab on vty0 (100.68.1.21) Nov 24 18:53:59 100.68.1.1 %SSH-5-SSH2_SESSION: SSH2 Session request from 100.68.100.250 (tty = 1) using crypto cipher '', hmac '' Failed Nov 24 18:53:59 100.68.1.1 %SSH-5-SSH2_CLOSE: SSH2 Session from 100.68.100.250 (tty = 1) for user '' using crypto cipher '', hmac '' closed Nov 24 18:58:59 100.68.1.1 %SSH-5-SSH2_SESSION: SSH2 Session request from 100.68.100.250 (tty = 1) using crypto cipher '', hmac '' Failed Nov 24 18:58:59 100.68.1.1 %SSH-5-SSH2_CLOSE: SSH2 Session from 100.68.100.250 (tty = 1) for user '' using crypto cipher '', hmac '' closed Nov 24 19:01:12 100.68.1.1 %SSH-5-SSH2_CLOSE: SSH2 Session from 100.68.1.21 (tty = 0) for user 'lab' using crypto cipher 'aes128-cbc', hmac 'hmac-sha1' closed

Troubleshooting

If no files are appearing under the /var/log/remote-syslog directory, then another command to try while logged into the router, in config mode, is to shutdown / no shutdown a Loopback interface, for example:

$ ssh lab@rtr1-gY.lab.workalaya.net rtr1-gY> enable rtr1-gY# config terminal rtr1-gY(config)# interface Loopback 999 rtr1-gY(config-if)# shutdown

wait a few seconds

rtr1-gY(config-if)# no shutdown

Then exit, and save the config ("write mem"):

rtr1-gY(config-if)# exit rtr1-gY(config)# exit rtr1-gY# write memory rtr1-gY# exit

Check the logs under /var/log/remote-syslog

lab@srv1-g1:~$ cd /var/log/remote-syslog/ lab@srv1-g1:/var/log/remote-syslog$ ls

...follow the directory trail

Still no logs?

Try the following command to send a test log message locally:

lab@srv1-g1:~$logger -p local0.info "Hello World\!"

If a file has not been created yet under /var/log/remote-syslog, then check your configuration for typos. Don't forget to restart the syslog-ng service each time you change the configuration.

What other commands can you think of that you can run on the router (BE CAREFUL!) that will trigger syslog messages? You could try logging in on the router and typing an incorrect password for "enable".

Be sure that you do an "ls" command in your logging directory to see if a new log file has been created at some point.