Security, CI/CD, and continuous assurance

The idea of Continuous Integration / Continuous Deployment (CI/CD) tends to make an old-school security practitioner a bit nervous. ("You want to automatically do what with our code / infrastructure? What about penetration testing? What about code review? What about the pre-release checklist? What about security signoff?")

But I'd argue that embracing CI/CD practices & tools, as part of a broader automation push, is - overall - a net positive for security.

While the Arivale consumer program has sadly closed down (long live scientific wellness!), over the past few years we'd built out a pretty tidy CI/CD toolchain with various embedded security elements.

We used both Circle CI & AWS CodePipeline/CodeBuild. Security-specific elements included:

• Commits to our Python repositories (mostly backend services) caused the resultant build to be scanned for vulnerable dependencies using safety.

• Commits to our JavaScript repositories (mostly frontend web/mobile apps, some backend) triggered an npm audit run.

• Security-specific unit tests and integration tests targetted access controls (mostly authorization) and input handling (testing various malformed/malicious inputs). Again, these were triggered on every commit.

• Artifacts were generated that included build manifests, so for each build we had a record of which components/versions had been included.

The primary benefit here was the tight feedback loop. Engineers received near-immediate feedback on security-related elements of their work, at the same time as other elements (results of broader unit tests, etc). Security issues were able to be addressed as part of the standard development process.

Beyond the standard per-commit CI/CD flow, we had a collection of recurring jobs (hourly, daily, etc) that were intended to provide additional continuous visibility:

• Python scripts (AWS Lambda parallelized w/ gevent) attempted to access public & non-public HTTP/HTTPS endpoints, tracked variances in results, and alerted on unexpected behavior wrt. encryption, authentication, and authorization.

• terraform plan was run to ensure that our AWS environment configured as defined, and identify variances.

• Linux instances and containers were spun up & standard distribution packaging/update mechanisms (yum --security check-update, apt-get update && apt-get upgrade --dry-run --show-upgraded --verbose-versions | grep ^Inst) used to check for any available updates.

• Bandit was run against the Python repositories and changes in report output investigated.

• Scans against metadata endpoints built in to each backend service allowed for a second level of checking for the existence of outdated/vulnerable dependencies.

• Scans against external network ranges identified any changes (see ndiff) to network-level configuration.

Output from all these tools were aggregated into a simple variance-tracking backend, with higher-priority items causing alerts to be pushed into a queue (and in our case, Slack :tada: :100:).

This may not all be relevant to other environments, and your milage may vary, but CI/CD & automated security checks more generally can definitely be a security net positive.