Emergency Incident - Management Process
This Agents-Only article details the process taken by staff when an incident occurs.
Process Summary
Key Contacts
Incident Contacts
| Role | Name | Contact Number | Proxy | Contact Number |
| Major Incident Manager |
Carmelo Pellizerri
Justin Corlis
Elly Gray
Sean Lofts
| | n/a | |
| Incident Lead | Koen Turenkens | | Brendan Newell | |
| Comms Manager | Kelly Sofia | | Wade Smith | |
SMS Contacts
The following staff members are registered to receive an SMS message when the SMS Alert is triggered from MS Teams.
- Anthony
- Brendan
- Koen
- David
- Kelly
- Elly
- Justin
- Sean
- Carmelo
Incident Stages
The Stages below are used when referring to an Incident. These stages will be used by the Communications Manager when posting updates on the Knowledge Base
| Stage | Step |
| New |
Incident Detection
|
|
Open Comms
|
| Under Investigation |
Assessing
|
|
Communicating
|
| Resolved - Monitoring |
Fix Implemented
|
|
Monitoring
|
| Resolved |
Confirmed Resolved
|
Emergency Process
Support Team
When a Client Support Specialist detects an issue, the following process should be followed:
- Incident detected: An incident is detected either by our technology, customer reports, or personnel. Whoever detects the incident (the first Client Support Specialist aware of the issue) is responsible for logging the incident in our Zendesk ticketing system, and assigning the priority level URGENT.
- Problem ticket: Once an incident ticket is created, a Problem ticket is created and assigned to the development team with the priority level URGENT. All Incident tickets created should be assigned to the Problem ticket.
- Post in the Performance & Bugs channel: Post details of the issue and the Problem Ticket Number in the Performance & Bugs channel (mark the post as Important, so it's red with an ! icon). Tag the Major Incident Managers and the entire group (via @channel name) in the item so that everyone is alerted.
If there is no response from the Major Incident manager within 5mins, please try the next in the list. If you're unable to contact anyone follow the directions in the "Send SMS" step below.
- Send SMS (Weekend or Non-responsive only): If the incident occurs on a weekend, or you're not getting a response from a Major Incident Manager, an SMS will need to be sent. This will alert a list of key contacts and Major Incident Managers via SMS. Use the weekend phone to send a group text.
- Manage Incoming Queries: manage any further incoming queries from customers. Advise that we are investigating, and will provide further updates. If the Communications Manager has advised the Status Page is live, direct the customers to the Status Page on the Knowledge Base for further updates.
- Link Incidents to Problem Tickets: each incident ticket should be linked to the Problem Ticket created in Step 2. This will allow for future reporting on the issue and bulk updates to customers once the issue has been resolved.
If the ticket was created by Voice (aka a phone call) ensure the requester's email address has been added as a cc to the ticket to ensure the customer receives future email updates.
- Monitor Teams for updates: Updates from the Major Incident Managers will be posted to teams periodically to advise the team of any changes. Continue to manage incoming queries and await further instruction until the incident has been resolved.
Back to top
Major Incident Manager
Once alerted to an incident in progress, the designated Major Incident Manager should follow the process below. The Major Incident Manager for the incident may vary depending on who is available for the day, however the steps taken should always be as per the below.
- Perform initial assessment: After being alerted to the issue, perform an initial assessment of the Incident Data. If an escalation is not warranted at this time, the Help Desk or Incident Team will manage these events as Incidents, and monitor for any changes that may occur which would warrant further escalation.
- Post in Active Incidents Channel: If an incident exists, the Major Incident Manager invokes the Major Incident Handling process. Refer the Incident to the Incident Lead and Business Recovery Team (BRT, the development team) for further review by posting in the Active Incidents channel. Mark the post as Important and tag the channel by using the @channel name. This will alert the Comms team.
- Weekends - Create a post in the "Active Incidents" Microsoft Teams channel and tag @SMSAlert. Note there will be an initial response that it was unsuccessful (disregard), followed by a successful notification. The successful notification can take up to 5 minutes.
- Alert Key Personnel: Depending on the severity based on the initial assessment, the Major Incident Manager may need to alert the sales team and Aaron of the incident in progress. An email is sent to sales@ to advise of the Incident.
- Dev/Support Liaison for info: Provide the BRT team with facts about the issue. Contact the Support Team for further information if required.
- Provide Support Team updates: Post updates to the Performance/Bugs channel as appropriate i.e. when the issue is being investigated, identified, resolution implemented and monitoring, resolved. This will allow the Client Support Specialists to provide additional information to customers still continuing to log tickets.
- Issue Resolved: Once the issue has been resolved, notify key personnel and the support team.
- Close Problem Ticket: The Problem Ticket in Zendesk is closed to confirm the issue has been resolved. When closing the Problem ticket, a copy of the public notes will be sent to all linked Incident Tickets so it's essential the notes are written in a customer-friendly format.
Back to top
Communications Manager
Once alerted to an incident, the Comms Manager should follow the process below:
- Comms alerted: Once alerted to an incident an initial assessment should take place. Review the existing content of the issue to determine the severity and to gain a full understanding of the issue. The Comms Manager should be aware of the symptoms of the issue, any temporary solutions available (refreshing cache etc) or alternative steps users should take.
- Confirm with Incident Manager: Approach the Major Incident Manager to confirm the issue and initial assessment. Together, determine which comms will be required for customers: Status Page, Back Office Banner or Zendesk Talk (or a combination of all three).
- Post Comms Updates: Follow the steps in the Emergency Incident - Content Management article to update the variety of mediums required e.g. status page update, enable the emergency message on the phone system, etc.
- Regular updates: Depending on the severity of the issue, regular updates should be performed. As a guide, critical issues should be updated every 10mins, less severe can be 1-2 a day as confirmed by the Incident Manager. If there is no status change, add a line to the Status Page "No further updates at this stage".
- Issue Resolved: Once the issue has been resolved follow the Recovery Checklist in the Emergency Incident - Content Management article. Await further instruction from the Incident Manager as to whether any follow-up communications will be required for customers i.e. EDM.