Today’s enterprise security staff cannot keep up with the fast moving DevOps software development model using their old waterfall-like gatekeeping control structures. At work I am heavily involved in that shift-left-of-security-transformation, which lead to this collection of notes.

The content is taken from numerous papers and talks, but mostly from Tips and Tricks For Scaling Security (aka SecDevOps Best Practices) - Clint Gibler and Practical Tips for Defending Web Applications in the Age of DevOps - Zane Lackey.

Agile / DevOps

First, let us summarize what Agile and DevOps mean for us in Development and IT Operation. Most obvious fact is that changes happen much more faster: release daily / weekly instead of quarterly / yearly. You can define infrastructure in code instead of configuring it manually (eg. Ansible / Terraform / Cloudformation). We have now a decentralized ownership of deployment (from Dev-Test-QA-Sec-Ops-Prod to Dev-Ops). The talks with security become opt-in instead of a mandated task.

The main benefit of being agile is to empower developers to push features / changes faster into production and allow faster feedback loops regarding user experience, code quality and security. From a feedback loop of some month it now gets down to some minutes in blue/green deployment.

But there are numerous challenges on the way to working in the agile DevOps mode. First of all traditional processes must be changed because of internal private repos, manual deployment by outsourced ops that takes weeks, different platforms between dev & prod, no central monitoring, no CI/CD that can be used by everyone [3] and so on. There are most of the times more than 100 developers / operators for 1 security expert. That makes it impossible to review all code and infrastructure changes because security cannot be the blocker - a fixed quality gateway like in waterfall is impossible.

Manual test and review of applications and the infrastructure is not feasible anymore. The new task of security teams is to scale security to support growing demand of the engineers.

How do you scale security for software development?

What are the possibilities to scale security with software development?

create secure libraries and systems that are easy to use
automated static and dynamic checks (test driven security)

Security can no longer be “outsourced” to the security team, but rather that the security team’s mission changes to providing the resources for teams to be security self-sufficient. [4].

How can be upper management brought on board?

cost savings (get more things done with the same headcount)
scale security teams along dev teams
maintain security standard across current and future services

What are the reasons for security experts to change?

automate boring recurring tasks
spend more time on higher leveraged, interesting problems

What can motivate the developers to get in touch with security?

new security processes add low to no friction for developers
new tools are as easy to adopt as possible and has close to 0% false positives
secure libraries work as drop-in replacement for existing ones
finding / fixing bugs increases also robustness and code quality
performance metrics motivates and makes work transparent
centralized tools for logging and monitoring makes remote SSH access unnecessary [5]

Security solutions with the best Return on Investment:

The best return on investment for security automation have libraries and tools wich are secure by default and can be used easily by the developers. Those are for example libraries for secret management, crypto, authentication, authorization, sql, file IO, battle tested user facing libraries (views, data model wrappers) and ready to use infrastructure blueprints.

What can be implemented on developer workstations?

The ultimate criteria for developer software is that it doesn’t interfere with their workflow. So everything must works out of the box, must have nearly zero false positives and must be integrated in the testsuite.

Simple make scripts can execute grep-like checks for DB access via unapproved API / library, SQL calls with concatenated strings, dangerous functions (e.g.: exec, eval, new Function()) and commits that seem to include secret keys (API keys, passwords, usernames).

What can be implemented in the code pipeline (CI/CD)?

To do so developers must use a central code repository and codepipeline (e.g.: Gitlab, Jenkins).

The most simple checks are basic grep-like examples from above. It is also really easy to block banned or dangerous functions, which improves code quality over time by blocking new additions of anti-patterns. You can also detect security-relevant code additions that alert the security team so that they can review and start discussion on single commits. This delivers direct feedback about demand on security functions to security engineers.

You can also define sensitive files / packages and alert on file changes. That pings security team which can investigate changes on eg. firewall configurations, infrastructure code, authentication libraries, etc. This again brings security topics to the attention of security experts, which can assist developers when they need it.

Another check can analyze out-of-date / vulnerable dependencies. It can also sets a policy for update process on time or severity. Newly added dependencies can be analyzed in regards of their trustworthiness based on downloads / maturity / included dangerous function calls.

What are some tools that can assist security testing?

Netflix Scumblr - is performing periodic syncs of data sources (such as Github repositories and URLs) and performing analysis (such as static analysis, dynamic checks, and metadata collection) on the identified results
Salesforce Providence - is a system for code commit & bug system monitoring. It is deployed within an organization to monitor code commits for security (or other) concerns, via customizable plugins. Slides
Gitrob - is a tool to help find potentially sensitive files in git
OWASP Zap Baseline Scan - Require baseline security on all websites (CSP, Secure Cookies, TLS Only, …)

When should we use a static software analysis tool (SAST)?

Static software analysis tools have a bad reputation of beeing an heavyweight process which runs weekly that generates extensive reports, and takes weeks to remediate. Because we want to defend ourself against the most critical attacks first we have to focus on single vulnerabilities to generate a low number of false positives. The acceptance of such tool depends highly on used technologies, development processes and the company culture.

You should invest in a commercial SAST solution if you have a large complex legacy codebase with a static typed language (C, C++, Java) and an underfunded security group which cannot assist everywhere. It would make also sense to invest in SAST if you can predict the high probability of finding a lot of bugs with static code analysis.

Usage of dynamic software analysis tools (DAST) [2]:

Those are great if you want to ensure a continous security baseline for new services by enforcing security policies (eg. only HTTPS comm., security headers, closed ports). You can also implement regression tests for known vulnerabilities to be sure that they won’t be introduced into the system in the future. State of the art is fuzzy testing where you send a really high number of request against an API to detect unknown problems. That increases the application resilience a lot.

Keep the following in mind when building your own scanning tool:

You should focus on finding one class of vulnerability well with very low false positives! That increases the acceptance and adoption. Report directly to devs because that builds trust with developers. If that is successful, choose another bug class and repeat. If one check returns high number of false positives - deactivate it because it doesn’t generates value.

What changes for the logging and monitoring in DevOps?

The traditional logging and monitoring consists of isolated collections of bugs, logs, customer service reports and outage information. The information is not shared across the company because of historic secrecy or technical interoperability.

The goal of modern security monitoring is to break down silos of data isolation so that everyone has context and visibility that they need! Share logs / tickets / issues across organization.

For traditional IT annual pentests only answer the question: Do I have bugs? That would be sufficient if a product is released annually. However today software gets deployed monthly, weekly and also hourly. Point-in-time pentest cannot fulfill the security needs. Therefor basic pentesting tasks must be included using bug bounty programs. The combination of in-depth pentests and continuous security bug bounty testing brings the security.

Summary

Most important for todays application defense is combination of continuous feedback plus visibility. Key question is: “How do I know when my attackers are being successful?” Living DevOps with security gives your devops-teams the ability to build secure-by-default products, detect attackers as early as possibility and to continuously refine the vulnerability response procedures them self.