Solution: ProTI
We present ProTI,
our implementation of the Automated Configuration Testing framework for Pulumi TypeScript IaC programs.
ProTI builds upon the popular JavaScript testing framework Jest,
implementing Jest runner, test-runner, and reporter plugins that jointly achieve automated IaC program testing.
The respective NPM packages are @proti-iac/runner, @proti-iac/test-runner, and @proti-iac/reporter.
@proti-iac/core implements the core mechanisms and abstractions,
i.e., module loading, scheduling, plugin interfaces, and utilities.
For random-based test abstractions, we reuse the popular JavaScript property-based testing implementation fast-check.
@proti-iac/spec implements the inline specifications syntax,
which – only when run in ProTI – hooks into ProTI’s central test scheduling and plugin interface mechanism.
Beyond the core infrastructure and abstractions,
we implement the first plugins in @proti-iac/pulumi-packages-schema.
They are type-based, leveraging input and output configuration type metadata from Pulumi package schemas.
By design, these are available for all resource types distributed in Pulumi packages.
The generator plugin composes primitive fast-check arbitraries to a complex arbitrary
of the shape of the resource type’s output configuration type
and draws random output configuration values from such arbitrary.
The oracle plugin checks all received input configurations for type compliance with the resource type’s input configuration type.
The focus of future work on ProTI is on the development of more sophisticated generator and oracle plugins. Leveraging advanced test generation and guidance techniques, e.g., symbolic execution, and additional and more precise models of cloud configurations for generation and validation is the essence of further improving the effectiveness of ACT and ProTI.
