Security 101: Four eye principle
I believe every single time I flagged this during an audit I had some people of a team go nearly ballistic. How could I only dare to suggest wasting precious time in such a stupid way? Do I not know that this prevents the team from getting things done? It surely has to be an anti pattern for an agile scrum waterfall team! And yet… it is one of the things that will regularly come up during an audit when people start looking at your processes and procedures.
People who had this discussion will likely point out that this is relevant for the whole company and not engineering specifically. It's also obviously part of the change management process. And they are right.
But it would not be a 101 if I would not try to deliver this in small, digestible chunks which do not make you hate a potential auditor before they even enter the office. So we will focus on engineering this time.
For many teams it is not even a discussion if there will be a review of a merge request (or pull request, depending on which centralised hosting provider you chose for your decentralised version control system), because in the end this is best practice, right? And it is a perfect example of the four eye principle. One engineer is implementing changes, the other is verifying that they will not blow up the whole system the moment they are being pushed to production.
But the readiness to commit to the four eye principle often ends right there. How are SQL queries being approved before they run against your database? How do you ensure infrastructure changes are not destructive? And obviously: How can you ensure that none of the two examples above causes a vulnerability or data leak?
While the avid reader might now hold their breath and wait for me to explain the one and only true solution to solve all of those problems, the more experience reader likely sheds a tear - knowing that this is a hard problem without a good solution.
You obviously want the process to be as light as possible (so you are not waiting days for approval to run
SELECT COUNT(id) FROM users). You also want to make sure you have some form of log in place in case something goes wrong so you can track what happened. Obviously you need exceptions during an emergency, but you still need the log so you can audit all steps taken after the incident was resolved. And those three are only the bare essential requirements to not fail an audit immediately. Thankfully those are also the most important things in case of less regulated domains.
While there might not be a silver bullet, there is at least some advice that works will enough for most early stage startups: Track everything in your version control system.
This might not sound very sophisticated. It surely requires discipline since you do not have any safeguards in place to prevent people from just ignoring this part of the process. But it is better than nothing. A lot better actually. So how does this look in practice?
One way to approach this is by creating a new repository with one file per technical domain. Infrastructure, database, data warehouse,… and simply open merge requests against the file. Approval works the same as for code reviews. In case of an incident you just copy and paste what you enter into the file and open the merge request after the incident is resolved. You should obviously pair with someone when you are working on a critical piece of your stack to have another pair of eyes on what you are doing.
I would argue this is not a lot of work, but you might rightfully ask what it buys you beside some points during a potential audit. Well: It buys you a safety net. We all do something stupid from time to time. We all make mistakes. Having another pair of eyes on the problem means we have a chance to catch them before they become a problem. Having partially reconstructed a database from backups at two in the morning - yay for whitelisting people to punch through DnD - I can tell you that waiting five minutes to execute a command in production is a lot less painful than the consequences of one single mistake.
Except being a safety net this is a huge compliance factor for a good number of certifications and accreditations. In many cases it will likely not be sufficient though. Having single individuals being able to do whatever they want without controls and approval can become a problem at some point. Especially when those individuals have access to customer data and are about to leave the company because they feel like they were treated badly or someone else is willing to pay well for the information.
Consider four eyes on everything a practice that scales well with your team and enables you to fulfil new requirements and enable additional controls when time comes - without large scale changes to your overall workflow.
If you want to follow this article series you can either subscribe to the general RSS feed or to the tag specific one if you only care about startup security posts.