Log management: Using Tenshi - Part 2

Introduction

Goals

• Learn how to use tenshi for log monitoring

Notes

• Commands preceded with "$" imply that you should execute the command as a general user - not as root.
• Commands preceded with "#" imply that you should be working as root.
• Commands with more specific command lines (e.g. "rtr1-gY>" or "mysql>") imply that you are executing commands on remote equipment, or within another program.

Exercises

First make sure that your network devices are configured to send logs to your shared srv1-gY server instance (this should have been done in the previous exercise).

Install and configure Tenshi

As a group work on this exercise together.

We will update our ansible playbook file named syslog.yml file to perform following tasks

• configure syslog-ng to save consolidated syslog file from remote source
• log rotation of consolidated syslog file
• install tenshi
• configure tenshi

first update syslog-ng config file named remote-log.conf located on files/syslog-ng as:

(venv) vmX-gY@ansible-host:~/ansible-playbook$ mkdir -p files/syslog-ng
(venv) vmX-gY@ansible-host:~/ansible-playbook$ vi files/syslog-ng/remote-log.conf

copy and paste folowing content and save

source s_net {
    udp(ip(0.0.0.0) port(514));
    tcp(ip(0.0.0.0) port(514));
};

destination remote-log {
    file("/var/log/remote-syslog/${HOST}/${YEAR}/${MONTH}/${DAY}/${FACILITY}.log"
        owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes)
    );
};

log {
    source(s_net);
    destination(remote-log);
};

filter f_routers { facility(local0); };

destination consolidated {
    file("/var/log/remote-syslog/consolidated.log", owner(root) group(root) perm(0644));
};

log {
    source(s_src);
    filter(f_routers);
    destination(routers);
};

and Create a daily automated script to truncate the consolidated.log log file so it doesn't grow too big as (COPY and PASTE):

(venv) vmX-gY@ansible-host:~/ansible-playbook$ vi files/syslog-ng/syslog-consolidated

/var/log/remote-syslog/consolidated.log {
  daily
  copytruncate
  rotate 1
  postrotate
    /etc/init.d/tenshi restart
  endscript
}

next create tenshi config file as:

(venv) vmX-gY@ansible-host:~/ansible-playbook$ mkdir -p templates/tenshi
(venv) vmX-gY@ansible-host:~/ansible-playbook$ vi templates/tenshi/network

set logfile /var/log/remote-syslog/consolidated.log
set queue network_alarms tenshi@localhost lab@localhost [*/1 * * * *] Log check

group_host ^100\.68\.
network_alarms SYS-5-CONFIG_I
network_alarms PRIV_AUTH_PASS
network_alarms LINK
group_end

Now update ansible playbook named syslog.yml file to install tenshi and make required changes as

(venv) vmX-gY@ansible-host:~/ansible-playbook$ vi syslog.yml

- hosts: syslog_hosts
  become: true
  tasks:
    - name: ensure package cache is up to date
      apt: update_cache=yes cache_valid_time=3600
      tags: install

    - name: install syslog-ng
      package:
        name: "{{ item }}"
        state: present
      with_items:
        - syslog-ng
        - syslog-ng-core
      tags: install

    - name: create /var/log/remote-syslog directory
      file:
        path: /var/log/remote-syslog
        state: directory
      tags: config

    - name: enable remote syslog settings
      copy:
        src: files/syslog-ng/remote-log.conf
        dest: /etc/syslog-ng/conf.d/
      notify: restart syslog-ng
      tags: config, tenshi

    - name: updatae log rotation config
      copy:
        src: files/syslog-ng/syslog-consolidated
        dest: /etc/logrotate.d/
      tags: config, tenshi

    - meta: flush_handlers

    - name: install tenshi
      package:
        name: "{{ item }}"
        state: present
      with_items:
        - tenshi
      tags: install, tenshi

    - name: configure tenshi
      copy:
        src: templates/tenshi/network
        dest: /etc/tenshi/includes-available/
      notify: restart tenshi
      tags: config, tenshi

    - name: activate tenshi to load config file
      file:
        src: /etc/tenshi/includes-available/network
        dest: /etc/tenshi/includes-active/network
        state: link
      notify: restart tenshi
      tags: config, tenshi

  handlers:
    - name: restart syslog-ng
      service:
        name: syslog-ng
        state: restarted

    - name: restart tenshi
      systemd:
        name: tenshi
        state: restarted

finally run ansilbe playbook named syslog.yml to install tenshi and configure it as

(venv) vmX-gY@ansible-host:~/ansible-playbook$ ansible-playbook syslog.yml

PLAY [syslog_hosts] *****************************************************************************************************************************************

TASK [Gathering Facts] **************************************************************************************************************************************
ok: [srv1-gY.lab.workalaya.net]

TASK [ensure package cache is up to date] *******************************************************************************************************************
[WARNING]: Could not find aptitude. Using apt-get instead

changed: [srv1-gY.lab.workalaya.net]

TASK [install syslog-ng] ************************************************************************************************************************************
ok: [srv1-gY.lab.workalaya.net] => (item=syslog-ng)
ok: [srv1-gY.lab.workalaya.net] => (item=syslog-ng-core)

TASK [create /var/log/remote-syslog directory] **************************************************************************************************************
ok: [srv1-gY.lab.workalaya.net]

TASK [enable remote syslog settings] ************************************************************************************************************************
changed:: [srv1-gY.lab.workalaya.net]

TASK [updatae log rotation config] **************************************************************************************************************************
changed:: [srv1-gY.lab.workalaya.net]

TASK [install tenshi] ***************************************************************************************************************************************
changed:: [srv1-gY.lab.workalaya.net] => (item=tenshi)

TASK [configure tenshi] *************************************************************************************************************************************
changed:: [srv1-gY.lab.workalaya.net]

TASK [activate tenshi to load config file] ******************************************************************************************************************
changed:: [srv1-gY.lab.workalaya.net]

RUNNING HANDLER [restart tenshi] ****************************************************************************************************************************
changed: [srv1-gY.lab.workalaya.net]

PLAY RECAP **************************************************************************************************************************************************
srv1-gY.lab.workalaya.net  : ok=10    changed=7    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Testing Tenshi

Log in to your router, and run some "config" commands (example below):

lab@vmX-gY:~$ ssh lab@rtr1-gY.lab.workalaya.net
rtr1-gY> enable
Password: <password>
rtr1-gY# config terminal
rtr1-gY(config)# int FastEthernet0/0
rtr1-gY(config-if)# description Description Change for FastEthernet0/0 for Tenshi
rtr1-gY(config-if)# exit
rtr1-gY(config)# exit
rtr1-gY# write memory

Don't exit from the router yet. Just as in the previous syslog-ng exercises, attempt to shutdown / no shutdown loopback interface:

rtr1-gY# conf t
rtr1-gY(config)# interface Loopback 999
rtr1-gY(config-if)# shutdown

wait a few seconds

rtr1-gY(config-if)# no shutdown

Then exit, and save the config ("write mem"):

rtr1-gY(config-if)# exit
rtr1-gY(config)# exit
rtr1-gY# write memory
rtr1-gY# exit

Verify that you are receiving emails to the sysadm user from Tenshi. A quick check is to look in the mail directory:

lab@srv1-gY:~$ ls -l /var/mail

Note: Tenshi checks /var/log/remote-syslog/consolidated.log once a minute, so you may have to wait up to a minute for the email to arrive to the lab user.

Make sure you are logged in as lab (not root). Either open a new session to your virtual machine, or exit from the root user (exit). Then do:

root@srv1-gY:~# su - lab

After becoming the lab user, run the "mutt" command to read the email

lab@srv1-gY:~$ mutt

Scroll up/down to select a message from "tenshi@localhost", then press ENTER to view it, and q to quit and 'q' again to quit mutt.

If mails are not arriving, then check the following:

• Are logs arriving in the file /var/log/remote-syslog/consolidated.log?

  lab@srv1-gY:~$ tail /var/log/remote-syslog/consolidated.log
  

• Do these logs show a hostname like 'rtr1-gY', or possibly an IP like 100.68.x.x ? Remember that the way we have configured tenshi, it only looks at hostnames or IP addresses matching the pattern 'rtr1-gY' or '100.68' (depending on how you configured tenshi).

• Check your tenshi configuration file. Restart tenshi if you change it.

• If you are still stuck ask an instructor or a neighbor for help.

Optional: Add a new Tenshi rule

See if you can figure out how to add a rule to Tenshi so that an email is sent if someone enters an incorrect enable password on your router.

Hints:

• "PRIV_AUTH_FAIL" is the Cisco IOS log message in such cases.
• To test your new rule log in to your router, type "enable" and then enter an incorrect enable password.
• You will need to restart the Tenshi service once you have made your configuration changes.