The changing nature of software development controls in a DevOps environment

This article is a part of Risk Update 1. 

---

Organisations large and small are rapidly transforming their IT functions to adopt DevOps and take their software development efforts to the next level of agility.

Introduction

A decade ago, DevOps was primarily the domain of technology start-ups, but as the speed, productivity and flexibility benefits have become apparent, adoption has spread to large organisations across a range of industries.

One often underappreciated benefit of DevOps is its impact on software development controls.  Teams leading DevOps transformations and auditors who place reliance on traditional software development controls aren’t always speaking the same language, leading to friction and misunderstanding on both sides.

What is DevOps?

In the traditional software development model, teams develop software in relative isolation from their supporting operations teams. Once ready to release, development teams hand over technical documentation and conduct knowledge transfer sessions to transition support to the operations teams.

DevOps, a portmanteau of ‘development’ and ‘operations’, brings together these two separate teams into one cross-functional team that manages a backlog of activities including both support and development tasks for a software platform.  They work to define, build, test, deploy and maintain software in short iterative sprints lasting a few weeks at most.

The intent is that these self-contained teams can deliver working software with minimal dependency on other teams which, when coupled with automation, enables rapid releases into production (e.g. weekly or even daily).  Critical to this change is not just the re-organisation of teams but also the use of proper tools and disciplines.

“Teams leading DevOps transformations and auditors aren’t always speaking the same language.”

What are the traditional software development controls?

In traditional software development, the practice of handing over large releases from development to operational teams provides a number of natural control points:

• Verifying adequate testing has been completed and any defects addressed;
• Assessing and accepting residual security risks;
• Confirming resiliency requirements are addressed and disaster recovery plans in place;
• Enforcing strict governance over changes through change approval boards and documented approval records; and
• Segregating the duties of development and operational staff to (in theory) minimise the likelihood of fraud from developers releasing malicious code into production.

Managing the handoff between teams using these controls has an overhead and may introduce waste and longer release cycles (e.g. quarterly).  The resulting rush to meet ever tighter deadlines brings with it an increased risk that some controls are not in place or adequately enforced.

Maintaining control with DevOps

In a well implemented DevOps model, with the right use of tools and disciplines in place, change related risks should be lower.  It’s far easier to identify and troubleshoot problems with code or adapt to feedback from users when changes are small, incremental and frequent. 

A poorly implemented DevOps model without the right tools and disciplines in place poses a potentially significant risk as there is neither the benefit of frequent production releases nor the controls of the traditional software development model.

1. Key Tooling Controls

Continuous Integration Continuous Deployment: Continuous Integration Continuous Deployment (CICD) pipeline tools manage the merging of code from multiple developers, execution of automated testing and the promotion of code between environments.  Ideally, production deployment should only be possible through CICD tools with production access, otherwise strictly controlled for emergencies only.

Automated testing: Automated testing, including functional testing (e.g. unit, regression and integration) and non-functional (e.g. performance and load) testing are used to the maximum extent possible.  Deployment by CICD tools should only be possible if testing has passed successfully.

Static and dynamic code scanning: Static and dynamic code scanning identifies common security vulnerabilities in source code before it is deployed.  Again, this should be enforced as a pre-requisite for deployment by the CICD tools.

Real-time production monitoring: Real-time production monitoring at both the infrastructure and application level can provide early warning signs of performance issues or errors, which can be easily traced back to recent changes if required.  It may not be possible to do proper stress testing for every release, so this can help trace the proverbial ‘straw that breaks the camel’s back’.

2. Key Discipline Controls

Peer review: Peer review involves developers review of each others’ code contributions, including automated testing code and infrastructure configurations.  For higher risk settings this should be system enforced so that new code cannot be deployed until it has been reviewed.

Training in secure coding practices: Training in secure coding practices ensures developers understand common vulnerabilities and incorporate good practices into their development efforts.

Living support documentation: Living support documentation should be ritually updated as part of every development sprint. Some of this documentation may exist within the code itself.

Cross-skilling: Cross-skilling of the team minimises key person reliance. Ideally, all members of the team should be able to write user stories, code, develop automated tests, and write technical documentation etc.

An empowered product owner: An empowered product owner is a key role that should be assigned to an individual with the capacity and capability to make decisions on requirements on behalf of the business. Without this, development teams may struggle to arrive at decisions or prioritise effectively.

Closing Thoughts

We have explored a number of impacts DevOps is having on the control environment.

Previously, formalised change controls, comprehensive test reports and change approval boards governed changes, giving way to automated build tools which test and deploy small changes in a ‘hands-off’ approach.

The traditional segregation of duties between people developing changes and deploying them has given way to system enforced peer review of source code.

Rather than relying on penetration testing to identify security defects, developers must be familiar with secure coding practices and build tools should enforce secure code scanning.

Finally, teams leading DevOps transformations need to take care to incorporate these new controls, while risk and audit professionals need to familiarise themselves with these new ways of working.

Then the only thing left is to get everyone reading from the same page.