ProTI
Automated Unit Testing of Infrastructure as Code

Fully Automated

ProTI automatically mocks all resource definitions and quickly tests a Pulumi TypeScript program in many different configurations.

Custom Specifications

ProTI provides ad-hoc specification syntax to augment test case generation and validation with application-specific values and checks.

Extensible Plugins

ProTI features an extensible plugin mechanism for test generators and oracles, enabling reuse, exchange, and research on novel strategies for IaC program testing.

ProTI is open source: Find out more below, read the publications and try it out.

Talk to us if you are interested or have ideas!

IaC Programs

With Programming Languages Infrastructure as Code (PL-IaC) solutions like Pulumi, developers implement IaC programs in a general-purpose programming language like TypeScript or Python. The execution of such an imperative program constructs the declarative target state of the deployment, which is then set up by the deployment engine. The target state is a directed, acyclic graph where nodes are resources with their configuration, and arcs are dependencies between the resources. A node is defined by calling the constructor of the resource’s class with the intended input configuration. Once defined, the deployment engine deploys the resource and returns its output configuration, which is accessible in the remainder of the program execution on the resource’s object. An arc in the target state is defined explicitly by referencing another resource object in a resource’s configuration or implicitly by defining the resource in a code block that depends on the output configuration of another resource. Once defined, resource nodes and dependency arcs are immutable, yielding that the target state graph is append-only.

With Programming Languages Infrastructure as Code (PL-IaC) solutions like Pulumi, developers implement IaC programs in a general-purpose programming language like TypeScript or Python. The execution of such an imperative program constructs the declarative target state of the deployment, which is then set up by the deployment engine. The target state is a directed, acyclic graph where nodes are resources with their configuration, and arcs are dependencies between the resources. A node is defined by calling the constructor of the resource’s class with the intended input configuration. Once defined, the deployment engine deploys the resource and returns its output configuration, which is accessible in the remainder of the program execution on the resource’s object. An arc in the target state is defined explicitly by referencing another resource object in a resource’s configuration or implicitly by defining the resource in a code block that depends on the output configuration of another resource. Once defined, resource nodes and dependency arcs are immutable, yielding that the target state graph is append-only.

Random word webpage deployment illustration

As an example, we demonstrate the random word webpage, a simple static website hosted in an AWS S3 bucket that displays one randomly selected word from the words array defined in line 5. Running the Pulumi TypeScript program constructs the target state graph we show next to the code. Lines 6–8 define the AWS S3 Bucket node, lines 10–11 the RandomInteger node, and lines 14–19 the BucketObject node index. The arc from index to bucket is defined by referencing bucket in the index’ input configuration in line 15. The arc from index to the random integer is defined implicitly by defining the bucket object in the apply callback in lines 13–20, which is a code block that depends on the output configuration of the random integer. Specifically, line 13 defines that the callback is run on the result output of the rng resource object, which is the randomly drawn number after the deployment of the random integer resource. The drawn number is provided as the first parameter wordId to the callback and used to configure the index’ content in line 18.

Random word webpage deployment illustration
Webpage deployment resource graph 
 1 import * as pulumi from "@pulumi/pulumi";
 2 import * as aws from "@pulumi/aws";
 3 import * as random from "@pulumi/random";
 4
 5 const words = ["software", "is", "great"];
 6 const bucket = new aws.s3.Bucket("website", {
 7     website: { indexDocument: "index.html" }
 8 });
 9
10 const rng = new random.RandomInteger("word-id", {
11     min: 0, max: words.length
12 });
13 rng.result.apply((wordId) => {
14     new aws.s3.BucketObject("index", {
15         bucket: bucket, key: "index.html",
16         contentType: "text/html; charset=utf-8",
17         content: "<!DOCTYPE html>" +
18                  words[wordId].toLowerCase()
19     });
20 });
21
22 export const url = bucket.websiteEndpoint;
Webpage deployment as Pulumi TypeScript IaC program and the resource graph it defines.

Testing Problem

The correctness of IaC programs is crucial. Faults can cause the entire deployment to malfunction. Even worse, a faulty IaC program can also yield a working deployment with severe security vulnerabilities. As IaC programs are implemented in popular general-purpose programming languages, they are eligible for the respective testing tools. Still, unit testing – despite being a common practice in traditional software – is barely applied to IaC programs. We found in August 2022 that less than 1% of all public Pulumi projects on GitHub implemented a unit test. Instead, developers solely rely on integration testing, which is slow and resource-intensive in the case of IaC programs.

Why do developers skip unit testing IaC programs? We conjecture it is the high development effort compared to the IaC programs themselves. IaC programs often do not implement much complex logic, e.g., there is no algorithmic code in the random word webpage example above. Typically, most of the code is instantiating resource objects. Each of these object instantiations integrates with the cloud, which directly receives the input configuration, creates the resource, and returns the resource’s output configuration that can afterward be accessed on the resource’s object in the remaining program execution. In a unit test, this cloud integration has to be mocked.

Mocking all resource instantiations is simple. For instance, Pulumi’s runtime provides a mocking feature, which enables developers to mock all resource instantiations with a simple mock in less than 20 lines of code. However, unit tests with such a simplistic mock are useless. Why? The mocks have two crucial roles. First, they return the resources’ output configuration, which is test input for the remainder of the execution. In our example above, the mock of the random integer must return a realistic value for rng.result to test the callback in lines 14–19. Thus, the mocks must implement a good test generator. Second, to provide insight, the mocks have to validate all input configurations. Hence, they need to implement good test oracles.

A suitable unit test for the random word webpage example has to implement at least a test generator that provides values for rng.result and bucket.websiteEndpoint. Further, the mock should validate all provided input configurations. As the image shows, such a test is roughly 50 lines long for our example – more than double the size of the IaC program. And even worse, if the IaC program grows, the unit test is likely to grow even faster. Also, the test reimplements most of the logic of the IaC program and even some logic of the cloud. Hence, it is comparatively big and tightly coupled, causing high development effort and slowing down future changes in the IaC program itself.

The random word webpage IaC program next to a unit test for it, which implements a suitable test generator and oracle.
The random word webpage IaC program next to a unit test for it, which implements a suitable test generator and oracle.

Approach: Automated Configuration Testing

To solve the problem of testing IaC programs, we propose the Automated Configuration Testing (ACT) framework. ACT automatically mocks the IaC program under test. Its mock receives all resource input configurations and returns for each realistic output configuration. To obtain the output configurations, it employs an interface for generator plugins. Such an interface enables the development and reuse of generalized test generation strategies as exchangeable plugins. Similarly, ACT has an interface for oracle plugins, implementing reusable input configuration validation strategies. ACT fully automates mocking with test generation and validation, allowing to automatically test the IaC program in many different configurations.

ACT moves the major effort of mocking IaC programs for unit testing from the development of an individual IaC program to the community, just like Pulumi’s providers do for resource type classes. Once the community developed the framework and a set of suitable generator and oracle plugins, new and existing IaC programs can easily be unit tested, often without writing a single line of testing code.

Overview of the Automated Configuration Testing (ACT) approach.
Overview of our Automated Configuration Testing (ACT) approach.

Reusable ACT test generator and oracle plugins implement generalized and reusable testing strategies and models. Still, there may be cases where developers can leverage deployment-specific knowledge for more precise test generation and validation. For these cases, we propose inline specifications, where developers can annotate more precise test generation and validation hints directly in the program. During regular execution, such inline specifications are ignored. During testing, they guide the test generator and oracles. For instance, this code is an extract of the random word webpage example with added ACT inline specifications. In line 1, we surrounded rng.result with ps.generate(...).with(ps.integer(rngRange)) to specify a more precise value generator for rng.result. In lines 7 and 8, we surrounded words[wordId].toLowerCase() with ps.expect(...).to((s) => s.length > 0) to add a check that the index’ content is not empty.

1  ps.generate(rng.result).with(ps.integer(rngRange))
2    .apply((wordId) => {
3      new aws.s3.BucketObject("index", {
4          bucket: bucket, key: "index.html",
5         contentType: "text/html; charset=utf-8",
6          content: "<!DOCTYPE html>" +
7              ps.expect(words[wordId].toLowerCase())
8                .to((s) => s.length > 0)
9      });
10 });
Exerpt of the example with ACT inline specifications.

Solution: ProTI

We present ProTI, our implementation of the Automated Configuration Testing framework for Pulumi TypeScript IaC programs. ProTI builds upon the popular JavaScript testing framework Jest, implementing Jest runner, test-runner, and reporter plugins that jointly achieve automated IaC program testing. The respective NPM packages are @proti-iac/runner, @proti-iac/test-runner, and @proti-iac/reporter. @proti-iac/core implements the core mechanisms and abstractions, i.e., module loading, scheduling, plugin interfaces, and utilities. For random-based test abstractions, we reuse the popular JavaScript property-based testing implementation fast-check. @proti-iac/spec implements the inline specifications syntax, which – only when run in ProTI – hooks into ProTI’s central test scheduling and plugin interface mechanism.

Beyond the core infrastructure and abstractions, we implement the first plugins in @proti-iac/pulumi-packages-schema. They are type-based, leveraging input and output configuration type metadata from Pulumi package schemas. By design, these are available for all resource types distributed in Pulumi packages. The generator plugin composes primitive fast-check arbitraries to a complex arbitrary of the shape of the resource type’s output configuration type and draws random output configuration values from such arbitrary. The oracle plugin checks all received input configurations for type compliance with the resource type’s input configuration type.

The focus of future work on ProTI is on the development of more sophisticated generator and oracle plugins. Leveraging advanced test generation and guidance techniques, e.g., symbolic execution, and additional and more precise models of cloud configurations for generation and validation is the essence of further improving the effectiveness of ACT and ProTI.

Overview of the ProTI NPM packages
Overview of the ProTI NPM packages.

Getting Started

To work with ProTI you require an installation of NodeJS with NPM. ProTI is developed with and supports NodeJS 18 LTS.

Using ProTI

  1. Set up Jest in the project. Using NPM and ts-jest for the transpilation, you can run these commands in the project directory: 
    npm install --save-dev jest ts-jest
  2. Install @proti-iac/runner and @proti-iac/test-runner: 
    npm install --save-dev @proti-iac/runner @proti-iac/test-runner
  3. Configure Jest to invoke ProTI. The easiest way to do so is to inherit the ProTI configuration preset from @proti-iac/test-runner. You can configure Jest by creating a jest.config.js file in the root of your project with this content: 
    /**
 * For a detailed explanation regarding each configuration property, visit:
 * https://jestjs.io/docs/configuration
 */
/** @type {import('jest').Config} */
const config = {
 preset: "@proti-iac/test-runner",
};
module.exports = config;

    Add further configuration to the file to augment Jest’s, ts-jest’s, and ProTI’s default configuration. The default configuration configures a simple empty state test generator and an oracle that only checks URN uniqueness across all resources, which are implemented in @proti-iac/core. Most likely, you want to configure more sophisticated generator and generator plugins. Configuring ProTI describes how. Concretely, @proti-iac/pulumi-packages-schema’s README describes how to install and configure our first type-based plugins.

  4. Run ProTI by running Jest on the project: 
    npx jest

Using ProTI’s Inline Specifications

To use ProTI’s inline specification syntax, additionally, install the @proti-iac/spec package as a dependency (not only as a development dependency):

npm install --save @proti-iac/spec

Simply import the package in your IaC program’s code and use the syntax it exports:

import * as ps from "@proti-iac/spec";

As an example of its use, you can have a look at the correct random word website example with ProTI inline specifications.

Detailed Reporting

For detailed reporting in CSV format, additionally install the @proti-iac/reporter package, and configure it as Jest reporter.

Developing ProTI Plugins

ProTI is extended through generator and oracle plugins. Implementing either is simple and demonstrated in @proti-iac/plugins-demo. This package serves as a blueprint for new plugins. Please refer to its code and documentation for further details.

Publications

  1. TSE
    ProTI
    Automated Infrastructure as Code Program Testing

    Daniel Sokolowski, David Spielmann, Guido Salvaneschi
    IEEE Transactions on Software Engineering 50 (6), 2024

    PDF Supp Code Website

    Infrastructure as Code (IaC) enables efficient deployment and operation, which are crucial to releasing software quickly. As setups can be complex, developers implement IaC programs in general-purpose programming languages like TypeScript and Python, using PL-IaC solutions like Pulumi and AWS CDK. The reliability of such IaC programs is even more relevant than in traditional software because a bug in IaC impacts the whole system. Yet, even though testing is a standard development practice, it is rarely used for IaC programs. For instance, in August 2022, less than 1% of the public Pulumi IaC programs on GitHub implemented tests. Available IaC program testing techniques severely limit the development velocity or require much development effort. To solve these issues, we propose Automated Configuration Testing (ACT), a methodology to test IaC programs in many configurations quickly and with low effort. ACT automatically mocks all resource definitions in the IaC program and uses generator and oracle plugins for test generation and validation. We implement ACT in ProTI, a testing tool for Pulumi TypeScript with a type-based generator and oracle, and support for application specifications. Our evaluation with 6081 programs from GitHub and artificial benchmarks shows that ProTI can directly be applied to existing IaC programs, quickly finds bugs where current techniques are infeasible, and enables reusing existing generators and oracles thanks to its pluggable architecture.

  2. ICSE Companion
    ProTI
    Unleashing the Giants: Enabling Advanced Testing for Infrastructure as Code

    Daniel Sokolowski, David Spielmann, Guido Salvaneschi
    In Companion Proceedings of the 46th International Conference on Software Engineering, ICSE Companion, 2024

    PDF

    Infrastructure as Code (IaC) programs are written in imperative programming languages like Python or TypeScript while declaratively defining the target state of software deployments, which the IaC solution then sets up, e.g., Pulumi and AWS CDK. Through a repository mining study and analysis, we noticed that testing IaC programs poses a dilemma: current techniques are either slow and expensive or require prohibitively high development effort. To solve this issue, we introduce Automated Configuration Testing (ACT), enabling efficient testing with low development effort. ACT automates the tedious aspects of unit testing IaC programs and is extensible through a plugin system for test generators and oracles. ACT is already effective with simple type-based plugins, and leveraging existing giants, i.e., advanced test generation and oracle techniques, in new plugins will further boost its effectiveness.

  3. Reliable infrastructure as Code for Decentralized Organizations

    Daniel Sokolowski
    Doctoral dissertation, Universität St. Gallen, 2024

    PDF

    IT must be reliable for organizations to thrive and quickly adaptable for their swift reaction to their environment. Agility is vital, and DevOps achieves these goals by empowering independent cross-functional teams in decentralized organizations and automating the entire software pipeline. Infrastructure as Code (IaC) is the critical tool to automate software operations, including infrastructure provisioning, application deployment, and configuration. Beyond simple IaC scripts, developers implement IaC programs in programming languages like TypeScript and Python. Such IaC programs are software, and their reliability is crucial to the functionality and security of the deployed systems. Still, techniques for the rapid development of reliable IaC programs are missing, limiting organizations’ agility. Specifically, developers lack automation for deployment coordination and updating and quality assurance tools for, e.g., testing and verification.

    We surveyed 134 IT professionals, finding that coordination across deployments is commonly needed and often requires manual coordination, even though IT professionals believe automated coordination yields better agility. However, automated approaches are centralized, limiting team independence and agility in decentralized organizations. To solve this issue, we propose automating coordination across deployments in a decentralized fashion through μs ([mju:z] “muse”), a novel IaC solution. With μs, teams have separate IaC programs, which express and jointly automate the correct order of operations across deployments. We further show how μs enables safe updating through IaC programs, preventing updates from breaking distributed transactions or workflows.

    Beyond automating the coordination of IaC programs, we address the reliability of IaC program code. To unblock studies, we built a dataset of 37 712 public IaC programs. In initial analyses, only a vanishing fraction implements tests. We identified that available testing techniques are either slow and resource-intensive or require excessive development effort. To solve this dilemma, we propose ACT, an extensible automated unit testing approach that enables testing IaC programs quickly in hundreds of configurations, often without writing additional testing code.

    This dissertation studies the coordination and testing of IaC programs. Empirically motivated, we present μs for safe deployment coordination and updating in decentralized setups and ACT for efficient testing of IaC programs. Our contributions nurture future research and reliable deployments in decentralized organizations, ensuring agility.

  4. FIST
    Towards Reliable Infrastructure as Code

    Daniel Sokolowski, Guido Salvaneschi
    In Companion Proceedings of 2023 IEEE 20th International Conference on Software Architecture, ICSA Companion, 2023

    PDF

    Modern Infrastructure as Code (IaC) programs are increasingly complex and much closer to traditional software than to simple configuration scripts. Their reliability is crucial because their failure prevents the deployment of applications, and incorrect behavior can introduce malfunction and severe security issues. Yet, software engineering tools to develop reliable programs, such as testing and verification, are barely used in IaC. In fact, we observed that developers mainly rely on integration testing, a slow and expensive practice that can increase confidence in end-to-end functionality but is infeasible to systematically test IaC programs in various configurations—which is required to ensure robustness. On the other hand, fast testing techniques, such as unit testing, are cumbersome with IaC programs because, today, they require significant coding overhead while only providing limited confidence.

    To solve this issue, we envision the automated testing tool ProTI, reducing the manual overhead and boosting confidence in the test results. ProTI embraces modern unit testing techniques to test IaC programs in many different configurations. Out of the box, ProTI is a fuzzer for Pulumi TypeScript IaC programs, randomly testing the program in many different configurations for termination, configuration correctness, and existing policy compliance. Then developers can add specifications to their program to guide random-based value generation, test additional properties, and add further mocking, making ProTI a property-based testing tool. Lastly, we aim at automatically verifying IaC-specific properties, e.g., access paths between resources.

  5. CONFLANG
    Creed for Speed: Comprehensive Infrastructure as Code Testing

    Daniel Sokolowski, David Spielmann, Guido Salvaneschi
    Presentation at the CONFLANG 2023 workshop, 2023

    PDF

    With Programming Languages Infrastructure as Code (PL-IaC), developers implement imperative IaC programs in one of many general-purpose programming languages, e.g., TypeScript, Python, or Go, to declaratively describe deployments. Using these languages provides access to quality assurance techniques and tools developed for traditional software; however, programmers routinely rely on prohibitively slow integration testing – if they test at all. As a result, even simple bugs are found late, tremendously slowing down the development process.

    To improve the velocity of PL-IaC development, we propose ProTI, an automated unit testing approach that quickly tests PL-IaC programs in many different configurations. ProTI mocks all cloud resources, replacing them with pluggable oracles that validate all resources configurations and a generator for realistic test inputs. We implemented ProTI for Pulumi TypeScript with simple generator and oracle plugins. Our experience of testing with ProTI encourages the exploration of more sophisticated oracles and generators, leading to the early detection of more bugs. ProTI enables programmers to rapidly prototype, explore, and plug in new oracles and generators for efficient PL-IaC program testing.

  6. SPLASH Companion
    Extensible Testing for Infrastructure as Code

    David Spielmann, Daniel Sokolowski, Guido Salvaneschi
    In Companion Proceedings of the 2023 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity, SPLASH Companion, 2023

    PDF

    Developers automate deployments with Programming Languages Infrastructure as Code (PL-IaC) by implementing IaC programs in popular languages like TypeScript and Python. Yet, systematic testing—well established for high-velocity software development—is rarely applied to IaC programs because IaC testing techniques are either slow or require extensive development effort. To solve this dilemma, we develop ProTI, a novel IaC unit testing approach, and implement it for Pulumi TypeScript. Our preliminary experiments with simple type-based test case generators and oracles show that ProTI can find bugs reliably in a short time, often without writing any additional testing code. ProTI’s extensible plugin architecture allows combining, adopting, and experimenting with new approaches, opening the discussion about novel generators and oracles for efficient IaC testing.