Documenting and communicating policy and system changes in your organization can be an arduous task. But the effort becomes more manageable when you have a plan in place before an emergency.
This is especially important when it comes to system changes, as auditors want to see that you have detailed logs of what’s happening in your environment, that the changes are properly documented and communicated across your organization, and that you can effectively debug problems after a change is made. All of these requirements and expectations are defined in your change management policy.
Here are four considerations when writing your change management policy:
Define the changes that will be in scope
Initially, it can be intimidating to get into the mindset of tracking system changes. “Do I have to track everything?” is a natural question. The short answer is no, but the better question to ask yourself might be, “What changes in my environment are large enough to initiate procedures for system changes?” This would include changes such as bug fixes, feature releases, and hotfixes.
Create an engineering review board
Once you’ve more clearly defined the type of changes that will be tracked, the next step is to create an engineering review board. This board will be a cross-functional team - usually with representation from engineering, operations, planning, and upper-level management. The board should meet whenever a major systems change needs to be made - and again, it’s up to you to decide what changes call for a review board. Once the proposed changes are discussed, the board will assess if the change is being made securely and then either approve or disapprove it accordingly.
Communicate clearly with customers
As important as it is to communicate clearly with your internal teams, it’s arguably more essential to have a solid plan to communicate changes to your customers. In general, it’s a good idea to lean on the side of over communicating. However, with the overwhelming amount of calls, emails, tweets and texts the average person gets, you need to be careful in choosing what to communicate - and how often.
Your strategy could be formed based on your customer type. Fortune 1000 customers, for example, might expect to be notified of a change months in advance. Smaller companies will be used to hearing about changes in real time. Whatever you decide is the right communication approach, make sure you honor the time commitments you define.
Emergency change procedure
It’s inevitable that at some point your team will need to push out an urgent code change to fix a major bug. In the heat of the moment, it's easy to get heads-down into the code and focus on nothing else except solving the problem, so you probably won’t be able to follow an entire change review procedure. However, you should still be following some procedure.
First, define what qualifies as an emergency. During the planning process, ask yourself, “Will the organization be at significant risk if this change is not applied immediately?” Then, define who will take on the decision-making role during the emergency, and ensure this person or group has administrative permission to conduct and approve an emergency change. You will also need to define the change process if it is not established already. After the change is made, conduct a code review.
Don’t forget to keep your internal teams up to date during an emergency change. If you have a ticketing system, it can be used to keep everyone current on the state of the emergency. Slack can also work well as a “live” data collection by creating a dedicated, temporary channel to have a deep discussion of the issue. Your team can post a continuous stream of updates, screenshots, helpful links, logs, and other artifacts so everyone stays focused - while creating a valuable timeline of evidence in the process. Having this type of war room is especially important if you have employees spread across multiple work sites.
Once you are out of the heat of the emergency, conduct a post-incident discussion. What were the important lessons learned? What could be handled better next time? Does it make sense to do a policy review? Are there network security changes that need to be made now that the incident is closed? You may ultimately decide it makes sense to consult with a managed service provider (or directly with primary vendors such as Microsoft) to have them conduct an impact assessment and/or be on retainer for future emergencies.
Once the emergency change is done, it’s easy to move on with business as usual. Instead, take the time to review how the change affected your existing policies and procedures. If you identified any policy issues during the change, make time to complete any necessary policy changes and policy updates. This type of documentation analysis should become standard practice in your security program activities.
Documenting and communicating policy and system changes in your organization can be an arduous task. But the effort becomes more manageable when you have a plan in place before an emergency. Know who will lead the charge when the incident begins, give your team the tools they need to successfully communicate system changes, and put some parameters around which changes will be communicated to your customers. Having this kind of plan in place - especially before an emergency - can pay dividends in customer loyalty. Ideally, future incidents can become success stories and not disasters.
