Nagios and Request Tracker Integration - Creating Tickets

Introduction

Goals

• Learn how to configure Nagios to raise tickets in RT

Notes

• Commands preceded with "$" imply that you should execute the command as a general user - not as root.
• Commands preceded with "#" imply that you should be working as root.
• Commands with more specific command lines (e.g. "rtr1-gY>" or "mysql>") imply that you are executing commands on remote equipment, or within another program.

Exercises

Configure Nagios for Alerts

To configure RT and Nagios so that alerts from Nagios automatically create tickets requires a few steps:

• Create a proper contact entry for Nagios in /etc/nagios3/conf.d/contacts.cfg on your host.
• Update either vms.cfg or an individual host entry to use the new contact group.

Now login back to ansible control host.

Update ansible playbook named nagios.yml

(venv) vmX-gY@ansible-host:~/ansible-playbook$ vi nagios.yml

- hosts: nagios_hosts
  become: true
  tasks:
    - name: ensure package cache is up to date
      apt: update_cache=yes cache_valid_time=3600
      tags: install

    - name: install Nagios Version 3
      package:
        name: "{{ item }}"
        state: present
      with_items:
        - nagios3
        - nagios3-doc
      tags: install

    - name: Check nagios Users
      stat:
        path: /etc/nagios3/htpasswd.users
      ignore_errors: true
      register: nagios_user_pwfile_exists
      tags: configure

    - name: Create empty password file
      command: touch /etc/nagios3/htpasswd.users
      args:
        creates: /etc/nagios3/htpasswd.users
      when: not nagios_user_pwfile_exists
      tags: configure

    - name: Create nagios admin user
      htpasswd:
        path: /etc/nagios3/htpasswd.users
        name: nagiosadmin
        password: "{{ class_password }}"
        state: present
      ignore_errors: true
      tags: configure

    - name: Create nagios guest user
      htpasswd:
        path: /etc/nagios3/htpasswd.users
        name: "{{ item.username }}"
        password: "{{ item.password }}"
        state: present
      ignore_errors: true
      with_items:
        - { username: 'guest', password: 'guest' }
      tags: add_guest

    - name: Configure nagios.cgi to allow guest access
      lineinfile:
        dest: "/etc/nagios3/cgi.cfg"
        regexp: "^{{ item.property | regex_escape() }}="
        line: "{{ item.property }}={{ item.value }}"
      with_items:
        - { property: 'authorized_for_system_information', value: 'nagiosadmin,guest' }
        - { property: 'authorized_for_configuration_information', value: 'nagiosadmin,guest' }
        - { property: 'authorized_for_all_services', value: 'nagiosadmin,guest' }
        - { property: 'authorized_for_all_hosts', value: 'nagiosadmin,guest' }
      tags: add_guest
      notify: verify config

    - name: Update nagios.cfg to Enable External commands
      lineinfile:
        dest: "/etc/nagios3/nagios.cfg"
        regexp: "^{{ item.property | regex_escape() }}="
        line: "{{ item.property }}={{ item.value }}"
      with_items:
        - { property: 'check_external_commands', value: '1' }
      register: update_directory_permission
      tags: external_command
      notify: verify config

    - name: change directory permissions
      shell: "dpkg-statoverride --update --add {{ item.user }} {{ item.group }} {{ item.permission }} {{ item.dir }}"
      with_items:
        - { user: 'nagios', group: 'www-data', permission: '2710', dir: '/var/lib/nagios3/rw' }
        - { user: 'nagios', group: 'nagios', permission: '751', dir: '/var/lib/nagios3' }
      when: update_directory_permission.changed
      tags: external_command
      notify: restart nagios3

    - name: Generate the nagios monitoring templates
      template:
        src: ./templates/nagios/{{ item }}
        dest: /etc/nagios3/conf.d
        backup: yes
      with_items:
        - routers.cfg
        - vms.cfg
        - servicegroups.cfg
        - contacts.cfg
        - noc.cfg
      tags: update_config
      notify: verify config

  handlers:
    - name: verify config
      shell: nagios3 -v /etc/nagios3/nagios.cfg
      notify: restart nagios3

    - name: restart nagios3
      service: name=nagios3 state=restarted

create new contacts.cfg as

(venv) vmX-gY@ansible-host:~/ansible-playbook$ vi templates/nagios/contacts.cfg

define contact{
        contact_name                    net
        alias                           RT Alert Queue
        service_notification_period     24x7
        host_notification_period        24x7
        service_notification_options    c
        host_notification_options       d
        service_notification_commands   notify-service-by-email
        host_notification_commands      notify-host-by-email
        email                           net@srv1-g{{ class_group }}.lab.workalaya.net
        }

define contactgroup{
        contactgroup_name       tickets
        alias                   email to ticket system for RT
        members                 net,root
        }

create new noc.cfg as

(venv) vmX-gY@ansible-host:~/ansible-playbook$ vi templates/nagios/noc.cfg

define hostgroup {
    hostgroup_name noc
    alias          NOC Group
}

define host {
    use             generic-host
    host_name       noc
    alias           Main NOC server for NMMA Workshop
    address         noc1.lab.workalaya.net
    contact_groups  tickets,admins
}

Now run ansible playbook to update changes

(venv) vmX-gY@ansible-host:~/ansible-playbook$ ansible-playbook nagios.yml -t update_config

PLAY [nagios_hosts] ****************************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************************************
ok: [vmX-gY.lab.workalaya.net]

TASK [Generate the nagios monitoring templates] ************************************************************************************************************************
ok: [vmX-gY.lab.workalaya.net] => (item=routers.cfg)
ok: [vmX-gY.lab.workalaya.net] => (item=vms.cfg)
ok: [vmX-gY.lab.workalaya.net] => (item=servicegroups.cfg)
changed: [vmX-gY.lab.workalaya.net] => (item=contacts.cfg)
changed: [vmX-gY.lab.workalaya.net] => (item=noc.cfg)

RUNNING HANDLER [verify config] ****************************************************************************************************************************************
changed: [vmX-gY.lab.workalaya.net]

RUNNING HANDLER [restart nagios3] **************************************************************************************************************************************
changed: [vmX-gY.lab.workalaya.net]

PLAY RECAP *************************************************************************************************************************************************************
vmX-gY.lab.workalaya.net      : ok=4    changed=3    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

See Nagios Tickets in RT

• It will take a bit (up to 5 minutes) for Nagios to report that SSH is "critical". Then you must wait for a total of 4 checks before the SSH service is deemed to be down "hard". At that point Nagios will send an email to net@srv1-gY.lab.workalaya.net and a ticket will be created in RT.

• Remember to see this go to http://srv1-gY.lab.workalaya.net/rt/ and log in as Username "vmX-gY" with the password you used for your host. The new ticket should appear in the "10 newest unowned tickets" box in the main net queue on the main login page in RT.