SecDevOps: What is it, and Do You Need it in Your Organization?

In the realm of software development and IT operations, the dynamics are continually shifting. Security, once an afterthought in the development process, has now become an integral part of it. The concept of SecDevOps, a fusion of security into the popular DevOps model, has emerged as a key player in this new era. In this blog post, we delve into the fundamentals of SecDevOps, exploring its definition and its two main components: Security as Code (SaC) and Infrastructure as Code (IaC). We also have the privilege of hosting Michael Oberlaender, a globally renowned security executive, as a special guest. Michael shares his expertise and insights on various aspects of SecDevOps, providing valuable recommendations and highlighting its significance in today's security landscape.

DevOps has been great. Where the operations teams were previously far drawn from the software development, DevOps has now cemented their place in development. Prior to DevOps, operations teams were often isolated from the software development process and seen as a separate entity responsible for maintaining systems after software was released. This often resulted in slow and unreliable software delivery, as well as miscommunication and finger-pointing between the two teams.

With DevOps, operations teams are now integrated into the development process, working collaboratively with development teams to ensure smooth delivery of software. This helps to create a shared sense of ownership and responsibility for the success of software projects.

The result of this collaboration is faster and more reliable software delivery, with greater quality and stability. Operations teams can provide insight and support to development teams, and development teams can ensure that the software they create is optimized for the operational environment.

While this ecosystem has been super good for a long time, there is an emerging element that needs urgent incorporation into development: SECURITY. Why?

Most of the cyber threats that companies are grappling with today have their origin in software vulnerabilities. A weak code here and there, a missed patch somewhere, and the attackers will have a field day.

In order to ensure that products coming out of the development process are secure, it’s increasingly becoming important to prioritize security in development. And this is how we end up with SecDevOps.

Michael Oberlaender is a globally renowned security executive. His amazing credentials include MS, CISSP, CISM, CGEIT, CRISC, CISA, ACSE, GSNA, TOGAF 9, CNSS-4016, CDPP, CDPSE.

Michael is the author of the highly successful book «C(I)SO — And Now What – How to Successfully Build Security by Design». He sits at the advisory boards of many successful corporations. His work has been published in leading journals.

Enough of our attempt at introducing Michael :) Let him him do the real introduction by himself:

Joseph Harisson: Hey Michael, Welcome. As you can expect our readers will be happy to hear your story from you personally. Please proceed:

Michael Oberlaender: Hey Joseph, thanks for having me here – well, I think it’s fair to say that I am a global industry leader in the security field – I have been doing this now for almost 3 decades and have held 8 prior CSO / CISO roles in large global enterprises and different verticals.

Having worked in these many leadership roles around the world and with global exposure where you had to manage multiple teams and sub teams etc. in different parts of the world, while protecting organizations against external (and seldomly, but sometimes) against internal threats and that on a 24x7 schedule is not for the faint of heart, and is providing you experience and learning that you won’t want to miss.

Over the decades I have always shared my knowledge and know-how, by either writing articles in the leading trade journals, or speaking at conferences, or then even writing books.

My first book is quite famous, «C(I)SO – And Now What». It has been read by many CISOs and those that want to become one. It was published in 2013, and is still valid today. That was the skeleton. My new book «Global CISO – Strategy, Tactics, and Leadership» (published exactly 3 years ago) is the «flesh to the bone» – where I really showcase a lot of the items and provide many examples, what to learn from them, how to build a security program, organization, and the step by step approach to the many items on the path there. I’ve published these books to help other CISOs and newcomers to help us secure the planet and make this a safer world.

That has always been my vision and it’s worthwhile doing it. There is no alternative, or our kids and grandkids won’t be able to do anything securely. Coming from a German culture and background, I have been educated to 100% focus on the problems and solve them head-on.

Let me also share that here in the US there’re a lot of misconceptions of security – and privacy. Hiring a CISO is not the last step, rather the first one. Then comes the necessary investments, and changes of process, culture, and increased maturity. Privacy is still in its infancy, despite some industry regulations as we’re missing a federal privacy and security law that the EU has adapted. Guess why the CJEU (European Court of Justice) has stricken down several instances of the US/EU data transfer agreements, regardless if they were called «Safe Harbor», or «Privacy Shield» or whatever else is next. We need to solve the problem, not sugar coating it and announcing some executive order. In other areas, the US is ahead of the EU, for example, we’re taking better steps now to protect our critical infrastructure – but it took us decades to understand this weakness.

Overall I can say, that I have enjoyed my early on decision to focus on security – I saw it coming, I couldn’t tell exactly when or how, but I could foresee that the current state we’re in, where you read / hear / see in the media, including TV, about hacks and breaches daily, would come. And here we are. I still enjoy working in this field, and there is work for the next hundred years to be done.

Joseph Harisson: Fantastic introduction, and what a colorful career, Michael!

We have also posed a number of questions to Michael as we prepared this article, and you will have an opportunity to find his thoughts infused within various sections throughout the article.

Let’s start by defining SecDevOps.

What is the definition of SecDevOps?

SecDevOps is a software development methodology that prioritizes security in the software development and deployment life cycle. The approach integrates security into every stage of the software development process, rather than treating it as an after-thought or a separate step. This allows for the identification and mitigation of potential security risks early on, making the software more secure and reducing the risk of security breaches.

In SecDevOps, security is not just dependent on the tools used, but is integrated into every stage of the development process and supported by the tools. As DevOps is about quick iterations and releases, SecDevOps is about ensuring that security checks and balances are integrated right from early on and becomes part and parcel of the fast DevOps process.

There are two main components in SecDevOps, namely SaC (Security as Code) and IaC (Infrastructure as Code)

Further Reading: Product Development Lifecycle

Security as Code

Security as Code (SaC) refers to the integration of security considerations into the software development process through the use of code and automation.

In SaC, security is treated as just another aspect of software development, like functionality or performance, and is automated and integrated into the DevOps pipeline. Each piece of code must be scanned, including every time it undergoes changes.

Infrastructure as Code

Infrastructure as Code refers to the use of specific tools to configure and manage the infrastructure items.

That’s to say the managing and provisioning of the IT infrastructure is through code.

We asked Michael Oberlaender to give the top two best practices he would recommend for ensuring confidentiality, integrity, and availability when utilizing IaC and SaC.

Michael Oberlaender: It’s important to first explain the concept:

IaC means Infrastructure as Code – and it means that companies / developers have over the years used the DevOps build a concept / methodology where you can always create a new build from the latest development trunks – you get your build environment from a repository that automates the process, and then “compiles” the code and is ready to be executed.

Security as Code re-uses that concept even farther and lets you build your security controls and tools into that development / build / run environment – and nowadays in the cloud. You need your (virtual) firewalls, zero trust, network segmentation, CASB, IDM and IAM, and encryption at least (if not more) as much as when you had it on-prem, in your own data centers or controlled perimeters.

It is of utmost importance to ensure confidentiality – so you CANNOT store credentials in the code – no «hard coded credentials» in the code. This is oftentimes overlooked and people don’t understand or underestimate the potential impact. Storing access tokens/keys to your cloud environment in the code or a repository is literally providing the proverbial keys to the kingdom to the cyber crooks.

Another one is the integrity of the source code – you should take great care to ensure both it's confidentiality (as this is most likely your IP / Intellectual Property), and the integrity of the committed code – that no one can plant a backdoor or can leverage a vulnerability at runtime to make the code do something unwanted.

Code-signing with only authorized certs and keys and its validation, publishing rules and libraries (3rd party especially) that are considered «safe to be used» and enforcing developers to adhere to these rules is a key requirement there.

On the availability side – make backups of your code and IaC/SaC, encrypt it, AND store the encryption keys OUTSIDE of it in a secure and safe place that only very limited people have access to.

Joseph Harisson: What should teams consider when selecting IaC tools for their projects?

Michael Oberlaender: I can’t speak specifically as this depends on their needs – BUT, it is important to have the following:

• Your requirements understood, defined, and listed;
• Automation in mind and in practice as much as possible, so ensure that the process is well designed and the tools help automate it;
• Integration across the various tools is key – it doesn’t help if they all require additional means to talk to each other;
• Visibility in mind – you always want to see where your pipeline is at, where code issues pop up, and have a positive feedback loop ensuring you avoid errors as soon as possible in the build, so the tools at the rear end of the pipeline should be able to communicate with those in the front.

Do you need SecDevOps in your organization, and why?

As SecDevOps is an emerging concept, organizations and even some developers might question why exactly they need SecDevOps. After all, isn't there supposed to be a security component in every organization that is concerned with matters of security? Why invest in an additional layer to DevOps?

Well, the simplest answer to these questions and more is that SecDevOps significantly reduces the software risk levels at source. Instead of deploying software that is swarmed with all manner of vulnerabilities, would it not make sense to integrate a process that will flush out the vulnerabilities during development? Isn't proactive always superior to reactive?

Here is a much more detailed breakdown of the reasons, or let’s just call them benefits that should compel you to consider moving to SecDevOps:

1. SecDevOps puts security at the top of everyone's mind

It’s impossible to ignore what’s on top of our minds. And this is exactly what SecDevOps does. By prioritizing security in development, its importance rises to the top of minds of all those involved, including stakeholders.

This means that every time a developer is writing code, for example, the question of security is going through their mind. When you combine this with the fact that SecDevOps provides critical tools plus automated checks, you begin to see why even the tiniest of loopholes will be easily sealed.

Joseph Harisson: What’s the best strategy that project managers should adopt to ensure that security remains at the top of teams' minds throughout the development lifecycle? Michael gave very good insights into this:

Michael Oberlaender:Great question – this requires multiple components:

1. You start with awareness about the issue, so that developers understand that code is NOT inherently safe — most of them never got taught in school/college/university how to write secure code).
2. After initial awareness, make sure you provide them the learning environments and tools to deep dive into the subject and make this mandatory.
3. Once the foundational understanding and knowledge is there, make sure that your design process has key steps like threat modeling, secure user stories, and always authorization and authentication as part of it (literally, checklist!).
4. Add a person that is code proof-reading (second pair of eyes) and then for your sprints ensure the sprint master is checking all these steps have been done.
5. Sprint reviews always provide positive feedback loops to improve this further.

Overall, security and privacy must be design goals. If there is no security / privacy sprint goal, then don’t start the sprint. If teams have done well in regards to security and privacy, provide them kudos and publish internally some nice accolades about the best / most secure code ever. 😊

2. SecDevOps saves costs

Security happens to be one of the key factors contributing to the rising cost of software development these days. Firstly, the cost of deploying and managing security controls can add up quickly. As security threats become more sophisticated, teams must deploy more complex and comprehensive security controls, which can increase the cost of software development.

Secondly, the cost of recovering from security incidents can be significant. If teams do not have robust security controls in place, they are more likely to experience data loss or other security incidents, which can be costly to recover from.Teams must also factor in the cost of compliance when considering the cost of software development. As regulatory requirements become more stringent, teams must make sure that their software is compliant with these requirements, which can add to the cost of development.

Joseph Harisson: Michael, how does SecDevOps help to bring down the software development costs that are related to security?

Michael Oberlaender: For years, if not decades, security was ignored and never designed into the code. Then, when it became obvious, infrastructure solutions (such as firewalls or WAFs) were added (bolted on). A lot of work and cost at the end of the overall SDLC.

It is well understood that a code change in production is the costliest and most complicated – the costs are hundredfold higher than in the early code development stages.

By bringing in the security concepts, designs, coding, validation, automated security testing, and security sign-off into the SDLC (so now, it is indeed a S-SDLC) you shift the problem to the left, where it is better to be solved by design, from the get-go, avoiding bolted on solutions and other crutches.

If you take the time of patch development into consideration, getting SecDevOps established actually reduces the overall development costs.

3. SecDevOps quickens development

Yes it does. In fact if you are SecDevOps ambassador facing a hard time convincing DevOps teams to adopt SecDevOps, just simplify things and tell them that this methodology will quicken production.

After all, speed is the central worry for most DevOps teams — they are always thinking along the lines of speed. So any suggestion of a framework that speeds production even more will easily get their buy-in.

According to Michael Oberlaender, this is how SecDevOps can quicken production:

Michael Oberlaender: Similarly as above – once you have overcome initial resistance, initial hurdles of adaption, and the more you automate the necessary security steps in the pipeline, the faster the overall process becomes – no more bottle neck, waiting on security sign-off or even back to the drawing board.

Instead, you have security (and privacy) by design and default – so the pipeline just keeps running and the new and secure code can be released into production. This time we know it’s much, much safer (I say this because anything done by humans, especially software, always has bugs).

4. Higher responsibility

Ever been a victim of «passing the buck»? It’s normal not just in software development circles but in many other professions as well.

Truth be said, no one likes to take the blame. Even those that are directly involved in mistakes may often want to find a place to direct the responsibility — it’s human nature. But must it always be like this? Sometimes it’s costly.

Luckily with SecDevOps, project managers can easily infuse security into each role. So every team member understands what’s expected of them as far as security is concerned. No room for blame games!

Having been involved at the highest level of decision making, Michael understands well the importance of accountability when it comes to security and how SecDevOps can help. Here are his insights:

Michael Oberlaender: It’s best to clearly understand and state this to the developer teams as well as everyone involved: security is everybody’s responsibility – not just the security department, the security champions, the security lead, or similar – everyone has to do their piece, and it starts with them, they need to adhere to the process, embrace the security policies, standards, procedures, and guidelines, and help to enhance the security deliverable step by step. The accountability (the overall A in the RACI matrix if you want) rests of course with one person, the highest ranking person in that group, department, function, or division.

Keeping leadership accountable is key: in case they set unrealistic deadlines for functionality and features, without giving the security goals the same weight, guess what the outcome will be.

So in companies with a product management function and an engineering function, both sides are accountable: the product management side to ensure security is a design goal, has enough developer time (percentage wise, like 20-25%), and is promoted both visibly and inherently in the process; and the engineering / development side needs to ensure their developers are trained (see above), follow the rules, guardrails and provided security controls, and by putting security front and center in their development lifecycle.

SecDevOps enables this by design and is therefore the right framework or methodology. To use the old and well-known statement by Peter Drucker: «you cannot manage what you don’t measure» – so it’s key to define security metrics (examples: #of security defects per lines of code, #of code review(s/ers) per production code snippet or library, %of time developers have worked on security designs, etc.) versus overall development time.

There is plenty of room, start small and increase/improve those over time to get smarter, better, and safer).

5. High customer lock in

This applies more to software development companies. Obviously, no customer wants to deal with insecure software when they have already spent a fortune to get it done. As a software development company keen to grow your customer base, SecDevOps is your friend.

Releasing software with as few vulnerabilities as possible (or none at all, if you can manage it) will make your current customers happy with your services — and, as is usually the case with good services, they'll want to recommend you to their peers, translating into more business for your company.

More customers will come in while the current ones stay locked in, exactly what every software developer wants.

SecDevOps challenges

Before we get into the common challenges that organizations are likely to encounter when implementing and using the SecDevOps framework, it’s important to mention that challenges can and often do vary from project to project and even from organization to organization.

We also sought to find out from Michael whether organizations should seriously think about and prepare for possible challenges or they should approach SecDevOps with an open mind and deal with challenges as they come?

Michael Oberlaender: Well, of course you need to think things through and prepare for challenges. Grasp the idea, understand the paradigm change, help encode/automate/facilitate it.

When challenges occur, use an open mind to analyze. What is the issue/challenge, and why is it there? Is it systemic – what can be done in the design system approach to solve. Is it a one off that can be solved without adding to technical debt. Is it a fundamental change issue that warrants a deeper analysis and problem-solving strategy etc.

On the other hand, don’t let issues and challenges make you throw away the overall concept just because it’s not always easy. Instead, use your problem-solving hat on and try to invent new pathways, new approaches.

You won’t get SecDevOps overnight, and DevOps wasn’t done overnight either. SecDevOps adoption and implementation is an iterative process that requires introspection, lessons learned, phased approaches, and adaptation.

Not all companies are equal, not all processes or tools are the same. The core questions that need to be validated from time to time are: «does this make the code more secure, do we save the money and time we expected, do we win over more customers because our product is more secure and doesn’t end up in the news all the time etc.».

Here are the common SecDevOps challenges:

1. Culture shift

SecDevOps requires buy-in from both security and development teams, as well as senior leaders. It also requires a significant shift in the way teams view security — from seeing it as an afterthought to viewing it as an integral part of the development process. Traditionally, security has been viewed as a roadblock to innovation and speed, but in the world of SecDevOps, security must be seen as an enabler of both.

For security teams, this shift means embracing a more proactive and collaborative approach to security. They must work closely with development teams to understand their needs and priorities, and to help identify and mitigate security risks before they become major problems.

For development teams, this shift means embracing a more secure mindset from the beginning of the development process. They must consider security as a critical component of their work and take steps to ensure that their code is secure.

Getting this shift right requires significant involvement, and there could be costs to it. Organizations must, for example, invest in training, education programs and cyber security certifications to help build the skills and knowledge necessary to support SecDevOps.

2. Compliance

Industry regulations such as PCI DSS, HIPAA, and GDPR are ever changing fast, so SecDevOps must match them.

Keeping up with these constantly changing security regulations and industry standards can be a challenge, especially in a fast-paced DevOps environment where updates and changes are made quickly.

3. Balancing speed and security

The need for speed in DevOps can often conflict with the need for security, making it difficult to find the right balance.

It is important to prioritize security, but it should not impede the speed and agility of the software development process.

4. Managing multiple tools

Organizations often use a variety of tools and technologies to support SecDevOps, from code repositories to tools for vulnerability scanning and more.

Managing these tools and ensuring they are properly integrated into the development pipeline can be a challenge, and organizations must have processes in place for regular reviews.

5. Managing third-party dependencies

In a fast-paced SecDevOps environment, organizations may rely on third-party components and libraries to support their applications.

Managing the security of these components and ensuring they are regularly updated can be a challenge. There is also the need to have processes in place for regularly reviewing and addressing vulnerabilities in third-party dependencies.

6. Shortage of talent

The field of SecDevOps is relatively new and naturally you can expect a shortage of skilled professionals who are proficient in both security and DevOps.

This can make it challenging for organizations to find the talent they need to build and maintain a secure DevOps pipeline.

Organizations must invest in training and development programs to build their own internal talent or look to outside resources, such as consulting services, to help fill the gap. Additionally, organizations must create a supportive work environment that attracts and retains top talent in the field of SecDevOps.

Preparing to Shift to SecDevOps

To wrap it up, we asked Michael to highlight the top 3 items that teams and organizations should get right when preparing to shift to SecDevOps

• Define your security requirements (like, principals).
• Define, simplify and improve your development process (the time to optimize is now, don’t automate a bad process) pipeline, and how to ingrain the security controls there.
• Automate and validate, improve and mature. Measure your metrics and keep them improving.

As a bonus, we also convinced him to share his future outlook.

Joseph Harisson: How do you see SecDevOps evolving going forward? Any big impact trends that we should be paying attention to?

Michael Oberlaender:Well I can’t predict the future, I can just make an educated guess.

It’s a bit too early to really say this but I think recent improvements in AI (artificial intelligence) and ML (machine learning) will help automate this process chain further and also validate more before code is being put into production. It’s a little bit how chess computers are programmed – you analyze so many «half-moves» to think ahead / through, then you give the outcome «values» by rating it versus the goal, and then go for the best path with the highest ratings (most secure code snippets).

But so far, I have not seen a machine completely programming a machine. Without security / safety controls, this could get quickly out of hand, and then the robots would start killing us. So, the security (safety) control and principle #1 always MUST be: «do no harm to humans – neither directly nor indirectly».

Joseph Harisson: Thank you Michael for your time. It’s been our honor and pleasure to host and share your insightful thoughts. We look forward to having you again.

Thank you readers as well. We hope that these insights from Michael Oberlaender give you a good picture of SecDevOps.

Anything you would like us to feature again on this topic? Please let us know in the comments.

Here are the links to Michael’s books and his notes regarding each book:

1. My new book (2020) is “GLOBAL CISO — STRATEGY, TACTICS, & LEADERSHIP: How to Succeed in InfoSec and CyberSecurity”, ISBN-13: 979-8604917756.
2. My first book (2013) is "C(I)SO — And Now What?: How to Successfully Build Security by Design ", ISBN-13: 978-1480237414; ISBN-10: 1480237418.