CircleCI Pipeline¶
This presents a reference implementation of the Application Pipeline reference architecture. The pipeline is created with CircleCI for building the software and performing testing tasks. All the infrastructure for this reference implementation is defined with AWS Cloud Development Kit. The source code for this reference implementation is available in GitHub for review.
This reference implementation can be utilized by forking this repository and onboarding the project to CircleCI. Once you've forked the repo, you can use the provided CircleCI configuration file to set up your CI/CD pipeline and start automating your builds, tests, and deployments on CircleCI.
Additionally, you will need to create CircleCI contexts to securely store credentials and environment variables. These contexts will allow your pipeline to interact with CircleCI and any third-party services required by the project.
Lastly, establish an OIDC connection between CircleCI and AWS to enable secure, identity-based access. Once the necessary IAM role has been created, update the CircleCI configuration parameters with your AWS account ID and the role’s ARN to complete the setup.
Disclaimer
This reference implementation is intended to serve as an example of how to accomplish the guidance in the reference architecture using CircleCI. The reference implementation has intentionally not followed the following AWS Well-Architected best practices to make it accessible by a wider range of customers. Be sure to address these before using parts of this code for any workloads in your own environment:
- cdk bootstrap with AdministratorAccess - the default policy used for
cdk bootstrapisAdministratorAccessbut should be replaced with a more appropriate policy with least privilege in your account. - TLS on HTTP endpoint - the listener for the sample application uses HTTP instead of HTTPS to avoid having to create new ACM certificates and Route53 hosted zones. This should be replaced in your account with an
HTTPSlistener.
Local Development¶
Developers need fast-feedback for potential issues with their code. Automation should run in their developer workspace to give them feedback before the deployment pipeline runs.
Pre-Commit Hooks
Pre-Commit hooks are scripts that are executed on the developer's workstation when they try to create a new commit. These hooks have an opportunity to inspect the state of the code before the commit occurs and abort the commit if tests fail. An example of pre-commit hooks are Git hooks. Examples of tools to configure and store pre-commit hooks as code include but are not limited to husky and pre-commit.
The following .pre-commit-config.yaml is added to the repository that will build the code with Maven, run unit tests with JUnit, check for code quality with Checkstyle, run static application security testing with PMD and check for secrets in the code with gitleaks.
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.3.0
hooks:
- id: check-yaml
- id: check-json
- id: trailing-whitespace
- repo: https://github.com/pre-commit/mirrors-eslint
rev: v8.23.0
hooks:
- id: eslint
- repo: https://github.com/ejba/pre-commit-maven
rev: v0.3.3
hooks:
- id: maven-test
- repo: https://github.com/zricethezav/gitleaks
rev: v8.12.0
hooks:
- id: gitleaks
Source¶
Application Source Code
The application source code can be found in the src/main/java directory. It is intended to serve only as a reference and should be replaced by your own application source code.
This reference implementation includes a Spring Boot application that exposes a REST API and uses a database for persistence. The API is implemented in FruitController.java:
public class FruitController {
/**
* JPA repository for fruits.
*/
private final FruitRepository repository;
/**
* Logic to map between entities and DTOs
*/
private final FruitMapper mapper;
FruitController(final FruitRepository r, final FruitMapper m) {
this.repository = r;
this.mapper = m;
}
@GetMapping("/api/fruits")
List<FruitDTO> all() {
return repository.findAll()
.stream()
.map(mapper::toDto)
.collect(Collectors.toList());
}
@PostMapping("/api/fruits")
FruitDTO newFruit(@RequestBody final FruitDTO fruit) {
return mapper.toDto(repository.save(mapper.toEntity(fruit)));
}
@GetMapping("/api/fruits/{id}")
FruitDTO one(@PathVariable final Long id) {
return repository.findById(id)
.map(mapper::toDto)
.orElseThrow(() -> new FruitNotFoundException(id));
}
@PutMapping("/api/fruits/{id}")
FruitDTO replaceFruit(
@RequestBody final FruitDTO newFruit,
@PathVariable final Long id) {
newFruit.setId(id);
return mapper.toDto(repository.save(mapper.toEntity(newFruit)));
}
@DeleteMapping("/api/fruits/{id}")
void deleteFruit(@PathVariable final Long id) {
repository.deleteById(id);
}
}
The application source code is stored in the version control system (VCS) you link to CircleCI. To get started quickly, simply fork this repository and connect it to CircleCI to begin setting up your CI/CD pipeline.
Test Source Code
The test source code can be found in the src/test/java directory. It is intended to serve only as a reference and should be replaced by your own test source code.
The reference implementation includes source code for unit, integration and end-to-end testing. Unit and integration tests can be found in src/test/java. For example, FruitControllerWithoutClassificationTest.java performs unit tests of each API path with the JUnit testing library:
public void shouldReturnList() throws Exception {
when(repository.findAll()).thenReturn(Arrays.asList(new Fruit("Mango", FruitClassification.pome), new Fruit("Dragonfruit", FruitClassification.berry)));
this.mockMvc.perform(get("/api/fruits")).andDo(print()).andExpect(status().isOk())
.andExpect(content().json("[{\"name\": \"Mango\"}, {\"name\": \"Dragonfruit\"}]"));
}
Acceptance tests are preformed with SoapUI and are defined in fruit-api-soapui-project.xml. They are executed by Maven using plugins in pom.xml.
Infrastructure Source Code
The infrastructure source code can be found in the infrastructure directory. It is intended to serve as a reference but much of the code can also be reused in your own CDK applications.
Infrastructure source code defines the deployment of the application are stored in infrastructure/ folder and uses AWS Cloud Development Kit.
super(scope, id, props);
const image = new AssetImage('.', { target: 'build' });
const appName = Stack.of(this)
.stackName.toLowerCase()
.replace(`-${Stack.of(this).region}-`, '-');
const vpc = new ec2.Vpc(this, 'Vpc', {
maxAzs: 3,
natGateways: props?.natGateways,
});
new FlowLog(this, 'VpcFlowLog', {
resourceType: FlowLogResourceType.fromVpc(vpc),
});
const dbName = 'fruits';
const dbSecret = new DatabaseSecret(this, 'AuroraSecret', {
username: 'fruitapi',
secretName: `${appName}-DB`,
});
const db = new DatabaseCluster(this, 'Database', {
engine: DatabaseClusterEngine.auroraMysql({
version: AuroraMysqlEngineVersion.VER_3_07_1,
}),
credentials: Credentials.fromSecret(dbSecret),
writer: ClusterInstance.serverlessV2('writer'),
defaultDatabaseName: dbName,
serverlessV2MaxCapacity: 2,
serverlessV2MinCapacity: 0.5,
vpc,
clusterIdentifier: appName,
storageEncrypted: true,
});
const cluster = new ecs.Cluster(this, 'Cluster', {
vpc,
containerInsights: true,
clusterName: appName,
});
const appLogGroup = new LogGroup(this, 'AppLogGroup', {
retention: RetentionDays.ONE_WEEK,
logGroupName: `/aws/ecs/service/${appName}`,
removalPolicy: RemovalPolicy.DESTROY,
});
let deploymentConfig: IEcsDeploymentConfig | undefined = undefined;
if (props?.deploymentConfigName) {
deploymentConfig = EcsDeploymentConfig.fromEcsDeploymentConfigName(
this,
'DeploymentConfig',
props.deploymentConfigName,
);
}
const appConfigEnabled =
props?.appConfigRoleArn !== undefined &&
props.appConfigRoleArn.length > 0;
const service = new ApplicationLoadBalancedCodeDeployedFargateService(
this,
'Api',
{
cluster,
capacityProviderStrategies: [
{
capacityProvider: 'FARGATE_SPOT',
weight: 1,
},
],
minHealthyPercent: 50,
maxHealthyPercent: 200,
desiredCount: 3,
cpu: 512,
memoryLimitMiB: 1024,
taskImageOptions: {
image,
containerName: 'api',
containerPort: 8080,
family: appName,
logDriver: AwsLogDriver.awsLogs({
logGroup: appLogGroup,
streamPrefix: 'service',
}),
secrets: {
SPRING_DATASOURCE_USERNAME: Secret.fromSecretsManager(
dbSecret,
'username',
),
SPRING_DATASOURCE_PASSWORD: Secret.fromSecretsManager(
dbSecret,
'password',
),
},
environment: {
SPRING_DATASOURCE_URL: `jdbc:mysql://${db.clusterEndpoint.hostname}:${db.clusterEndpoint.port}/${dbName}`,
APPCONFIG_AGENT_APPLICATION:
this.node.tryGetContext('workloadName'),
APPCONFIG_AGENT_ENVIRONMENT:
this.node.tryGetContext('environmentName'),
APPCONFIG_AGENT_ENABLED: appConfigEnabled.toString(),
},
},
deregistrationDelay: Duration.seconds(5),
responseTimeAlarmThreshold: Duration.seconds(3),
targetHealthCheck: {
healthyThresholdCount: 2,
unhealthyThresholdCount: 2,
interval: Duration.seconds(60),
path: '/actuator/health',
},
deploymentConfig,
terminationWaitTime: Duration.minutes(5),
apiCanaryTimeout: Duration.seconds(5),
apiTestSteps: [
{
name: 'getAll',
path: '/api/fruits',
jmesPath: 'length(@)',
expectedValue: 5,
},
],
},
);
if (appConfigEnabled) {
service.taskDefinition.addContainer('appconfig-agent', {
image: ecs.ContainerImage.fromRegistry(
'public.ecr.aws/aws-appconfig/aws-appconfig-agent:2.x',
),
essential: false,
logging: AwsLogDriver.awsLogs({
logGroup: appLogGroup,
streamPrefix: 'service',
}),
environment: {
SERVICE_REGION: this.region,
ROLE_ARN: props!.appConfigRoleArn!,
ROLE_SESSION_NAME: appName,
LOG_LEVEL: 'info',
},
portMappings: [{ containerPort: 2772 }],
});
service.taskDefinition.addToTaskRolePolicy(
new PolicyStatement({
actions: ['sts:AssumeRole'],
resources: [props!.appConfigRoleArn!],
}),
);
}
service.service.connections.allowTo(
db,
ec2.Port.tcp(db.clusterEndpoint.port),
);
this.apiUrl = new CfnOutput(this, 'endpointUrl', {
value: `http://${service.listener.loadBalancer.loadBalancerDnsName}`,
});
Notice that the infrastructure code is written in Typescript which is different from the Application Source Code (Java). This was done intentionally to demonstrate that CDK allows defining infrastructure code in whatever language is most appropriate for the team that owns the use of CDK in the organization.
Static Assets
There are no static assets used by the sample application.
Dependency Manifests
All third-party dependencies used by the sample application are define in the pom.xml:
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.liquibase</groupId>
<artifactId>liquibase-core</artifactId>
</dependency>
</dependencies>
Static Configuration
Static configuration for the application is defined in src/main/resources/application.yml:
spring:
application:
name: fruit-api
main:
banner-mode: "off"
jackson:
default-property-inclusion: non_null
springdoc:
swagger-ui:
path: /swagger-ui
appconfig-agent:
environment: alpha
log-level-from:
configuration: operations
Database Source Code
The database source code can be found in the src/main/resources/db directory. It is intended to serve only as a reference and should be replaced by your own database source code.
The code that manages the schema and initial data for the application is defined using Liquibase in src/main/resources/db/changelog/db.changelog-master.yml:
databaseChangeLog:
- changeSet:
id: "1"
author: AWS
changes:
- createTable:
tableName: fruit
columns:
- column:
name: id
type: bigint
autoIncrement: true
constraints:
primaryKey: true
nullable: false
- column:
name: name
type: varchar(250)
- insert:
tableName: fruit
columns:
- column:
name: name
value: Apple
- insert:
tableName: fruit
columns:
- column:
name: name
value: Orange
- insert:
tableName: fruit
columns:
- column:
name: name
value: Banana
- insert:
tableName: fruit
columns:
- column:
name: name
value: Cherry
- insert:
tableName: fruit
columns:
- column:
name: name
value: Grape
- changeSet:
id: "2"
author: AWS
changes:
- addColumn:
tableName: fruit
columns:
- column:
name: classification
type: varchar(250)
constraints:
nullable: true
- update:
tableName: fruit
columns:
- column:
name: classification
value: pome
where: name='Apple'
- update:
tableName: fruit
columns:
- column:
name: classification
value: berry
where: name='Orange'
- update:
tableName: fruit
columns:
- column:
name: classification
value: berry
where: name='Banana'
- update:
tableName: fruit
columns:
- column:
name: classification
value: drupe
where: name='Cherry'
- update:
tableName: fruit
columns:
- column:
name: classification
value: berry
where: name='Grape'
Build¶
Actions in this stage are designed to complete in under 3 minutes, allowing developers to quickly receive feedback and address any issues before moving on to their next task. Each of the actions below is defined as code using the CircleCI configuration file to ensure consistency and repeatability.
Build Code
The Java source code is compiled, unit tested and packaged by Maven. An action is added to the workflow to build and package the source code:
package:
docker:
- image: cimg/openjdk:17.0
working_directory: << pipeline.parameters.working-dir >>
environment:
TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db
steps:
- checkout:
path: ~/aws-deployment-pipeline-reference-architecture
- restore_cache:
keys:
- maven-repo-v1-{{ checksum "pom.xml" }}
- run:
name: Run Maven Verify
command: mvn verify --batch-mode --no-transfer-progress
- save_cache:
paths:
- ~/.m2/repository
key: maven-repo-v1-{{ checksum "pom.xml" }}
- trivy/scan:
scan-type: fs
ignore-unfixed: true
format: spdx-json
output: << pipeline.parameters.working-dir >>/dependency-results.sbom.json
- store_test_results:
path: << pipeline.parameters.working-dir >>/target/surefire-reports
- store_test_results:
path: << pipeline.parameters.working-dir >>/target/soapui-reports
- store_artifacts:
path: << pipeline.parameters.working-dir >>/dependency-results.sbom.json
- store_artifacts:
path: << pipeline.parameters.working-dir >>/target/spotbugsXml.xml
- store_artifacts:
path: << pipeline.parameters.working-dir >>/target/jmeter/results
- store_artifacts:
path: << pipeline.parameters.working-dir >>/target/fruit-api.jar
- persist_to_workspace:
root: << pipeline.parameters.working-dir >>
paths:
- target/fruit-api.jar
Unit Tests
The unit tests are run by Maven simultaneously with the Build Code step. The results of the unit tests are then uploaded to CircleCI Test Insights to track test performance and trends over time. CircleCI Test Insights provides visibility into test results, helping you monitor for any regressions and improve the efficiency of your testing process.
Code Quality
Code quality is enforced through the PMD and Checkstyle Maven plugins:
<plugin>
<artifactId>maven-pmd-plugin</artifactId>
<configuration>
<printFailingErrors></printFailingErrors>
</configuration>
<executions>
<execution>
<phase>test</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-checkstyle-plugin</artifactId>
<configuration>
<printFailingErrors></printFailingErrors>
</configuration>
<executions>
<execution>
<phase>test</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
Additionally, cdk-nag is run against the deployment stack to identify any security issues with the resources being created. The pipeline will fail if any are detected. The following code demonstrates how cdk-nag is called as a part of the build stage. The code also demonstrates how to suppress findings.
import { App, Aspects } from 'aws-cdk-lib';
import { Annotations, Match, Template } from 'aws-cdk-lib/assertions';
import { SynthesisMessage } from 'aws-cdk-lib/cx-api';
import { AwsSolutionsChecks, NagSuppressions } from 'cdk-nag';
import { DeploymentStack } from '../src/deployment';
function synthesisMessageToString(sm: SynthesisMessage): string {
return `${sm.entry.data} [${sm.id}]`;
}
expect.addSnapshotSerializer({
test: (val) => typeof val === 'string' && val.match(/^dummy.dkr.ecr.us-east.1/) !== null,
serialize: () => '"dummy-ecr-image"',
});
expect.addSnapshotSerializer({
test: (val) => typeof val === 'string' && val.match(/^[a-f0-9]+\.zip$/) !== null,
serialize: () => '"code.zip"',
});
describe('cdk-nag', () => {
let stack: DeploymentStack;
let app: App;
beforeAll(() => {
const appName = 'fruit-api';
const workloadName = 'food';
const environmentName = 'unit-test';
app = new App({ context: { appName, environmentName, workloadName } });
stack = new DeploymentStack(app, 'TestStack', {
env: {
account: 'dummy',
region: 'us-east-1',
},
});
Aspects.of(stack).add(new AwsSolutionsChecks());
// Suppress CDK-NAG for TaskDefinition role and ecr:GetAuthorizationToken permission
NagSuppressions.addResourceSuppressionsByPath(
stack,
`/${stack.stackName}/Api/TaskDef/ExecutionRole/DefaultPolicy/Resource`,
[{ id: 'AwsSolutions-IAM5', reason: 'Allow ecr:GetAuthorizationToken', appliesTo: ['Resource::*'] }],
);
// Suppress CDK-NAG for secret rotation
NagSuppressions.addResourceSuppressionsByPath(
stack,
`/${stack.stackName}/AuroraSecret/Resource`,
[{ id: 'AwsSolutions-SMG4', reason: 'Dont require secret rotation' }],
);
// Suppress CDK-NAG for RDS Serverless
NagSuppressions.addResourceSuppressionsByPath(
stack,
`/${stack.stackName}/Database/Resource`,
[
{ id: 'AwsSolutions-RDS6', reason: 'IAM authentication not supported on Serverless v1' },
{ id: 'AwsSolutions-RDS10', reason: 'Disable delete protection to simplify cleanup of Reference Implementation' },
{ id: 'AwsSolutions-RDS11', reason: 'Custom port not supported on Serverless v1' },
{ id: 'AwsSolutions-RDS14', reason: 'Backtrack not supported on Serverless v1' },
{ id: 'AwsSolutions-RDS16', reason: 'CloudWatch Log Export not supported on Serverless v1' },
],
);
NagSuppressions.addResourceSuppressionsByPath(stack, [
`/${stack.stackName}/Api/DeploymentGroup/Deployment/DeploymentProvider/framework-onEvent`,
`/${stack.stackName}/Api/DeploymentGroup/Deployment/DeploymentProvider/framework-isComplete`,
`/${stack.stackName}/Api/DeploymentGroup/Deployment/DeploymentProvider/framework-onTimeout`,
`/${stack.stackName}/Api/DeploymentGroup/Deployment/DeploymentProvider/waiter-state-machine`,
], [
{ id: 'AwsSolutions-IAM5', reason: 'Unrelated to construct under test' },
{ id: 'AwsSolutions-L1', reason: 'Unrelated to construct under test' },
{ id: 'AwsSolutions-SF1', reason: 'Unrelated to construct under test' },
{ id: 'AwsSolutions-SF2', reason: 'Unrelated to construct under test' },
], true);
// Ignore findings from access log bucket
NagSuppressions.addResourceSuppressionsByPath(stack, [
`/${stack.stackName}/Api/AccessLogBucket`,
], [
{ id: 'AwsSolutions-S1', reason: 'Dont need access logs for access log bucket' },
{ id: 'AwsSolutions-IAM5', reason: 'Allow resource:*', appliesTo: ['Resource::*'] },
]);
NagSuppressions.addResourceSuppressionsByPath(stack, [
`/${stack.stackName}/Api/Canary/ServiceRole`,
], [{ id: 'AwsSolutions-IAM5', reason: 'Allow resource:*' }]);
NagSuppressions.addResourceSuppressionsByPath(stack, [
`/${stack.stackName}/Api/CanaryArtifactsBucket`,
], [{ id: 'AwsSolutions-S1', reason: 'Dont need access logs for canary bucket' }]);
NagSuppressions.addResourceSuppressionsByPath(stack, [
`/${stack.stackName}/Api/DeploymentGroup/ServiceRole`,
], [
{ id: 'AwsSolutions-IAM4', reason: 'Allow AWSCodeDeployRoleForECS policy', appliesTo: ['Policy::arn:<AWS::Partition>:iam::aws:policy/AWSCodeDeployRoleForECS'] },
]);
NagSuppressions.addResourceSuppressionsByPath(stack, [
`/${stack.stackName}/Api/DeploymentGroup/Deployment`,
], [
{
id: 'AwsSolutions-IAM4',
reason: 'Allow AWSLambdaBasicExecutionRole policy',
appliesTo: ['Policy::arn:<AWS::Partition>:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'],
},
], true);
NagSuppressions.addResourceSuppressionsByPath(stack, [
`/${stack.stackName}/Api/TaskDef`,
], [
{
id: 'AwsSolutions-ECS2',
reason: 'Allow environment variables for configuration of values that are not confidential',
},
]);
NagSuppressions.addResourceSuppressionsByPath(stack, [
`/${stack.stackName}/Api/LB/SecurityGroup`,
], [
{
id: 'AwsSolutions-EC23',
reason: 'Allow public inbound access on ELB',
},
]);
});
test('Snapshot', () => {
const template = Template.fromStack(stack);
expect(template.toJSON()).toMatchSnapshot();
});
test('cdk-nag AwsSolutions Pack errors', () => {
const errors = Annotations.fromStack(stack).findError(
'*',
Match.stringLikeRegexp('AwsSolutions-.*'),
).map(synthesisMessageToString);
expect(errors).toHaveLength(0);
});
test('cdk-nag AwsSolutions Pack warnings', () => {
const warnings = Annotations.fromStack(stack).findWarning(
'*',
Match.stringLikeRegexp('AwsSolutions-.*'),
).map(synthesisMessageToString);
expect(warnings).toHaveLength(0);
});
});
describe('Deployment without AppConfig', () => {
let stack: DeploymentStack;
let app: App;
beforeAll(() => {
const appName = 'fruit-api';
const environmentName = 'unit-test';
app = new App({ context: { appName, environmentName } });
stack = new DeploymentStack(app, 'TestStack', {
env: {
account: 'dummy',
region: 'us-east-1',
},
});
});
test('Snapshot', () => {
const template = Template.fromStack(stack);
expect(template.toJSON()).toMatchSnapshot();
});
test('taskdef', () => {
const template = Template.fromStack(stack);
template.hasResourceProperties('AWS::ECS::TaskDefinition', {
ContainerDefinitions: [
{
Environment: [{
Name: 'SPRING_DATASOURCE_URL',
}, {
Name: 'APPCONFIG_AGENT_APPLICATION',
}, {
Name: 'APPCONFIG_AGENT_ENVIRONMENT',
Value: 'unit-test',
}, {
Name: 'APPCONFIG_AGENT_ENABLED',
Value: 'false',
}],
},
],
});
});
});
describe('Deployment with AppConfig', () => {
let stack: DeploymentStack;
let app: App;
beforeAll(() => {
const appName = 'fruit-api';
const workloadName = 'food';
const environmentName = 'unit-test';
app = new App({ context: { appName, environmentName, workloadName } });
stack = new DeploymentStack(app, 'TestStack', {
appConfigRoleArn: 'dummy-role-arn',
env: {
account: 'dummy',
region: 'us-east-1',
},
});
});
test('Snapshot', () => {
const template = Template.fromStack(stack);
expect(template.toJSON()).toMatchSnapshot();
});
test('taskdef', () => {
const template = Template.fromStack(stack);
template.hasResourceProperties('AWS::ECS::TaskDefinition', {
ContainerDefinitions: [
{
Environment: [{
Name: 'SPRING_DATASOURCE_URL',
}, {
Name: 'APPCONFIG_AGENT_APPLICATION',
Value: 'food',
}, {
Name: 'APPCONFIG_AGENT_ENVIRONMENT',
Value: 'unit-test',
}, {
Name: 'APPCONFIG_AGENT_ENABLED',
Value: 'true',
}],
},
{
Environment: [{
Name: 'SERVICE_REGION',
Value: 'us-east-1',
}, {
Name: 'ROLE_ARN',
Value: 'dummy-role-arn',
}, {
Name: 'ROLE_SESSION_NAME',
}, {
Name: 'LOG_LEVEL',
Value: 'info',
}],
},
],
});
});
});
Secrets Detection
Trivy is used to scan the source for secrets. The Trivy CircleCI Orb is used in the CircleCI workflow to perform the scan:
trivy:
docker:
- image: cimg/base:2024.11
working_directory: << pipeline.parameters.working-dir >>
steps:
- checkout:
path: ~/aws-deployment-pipeline-reference-architecture
- trivy/scan:
scan-type: fs
ignore-unfixed: true
format: sarif
output: << pipeline.parameters.working-dir >>/report.sarif
scanners: vuln,secret,misconfig,license
- store_artifacts:
path: << pipeline.parameters.working-dir >>/report.sarif
environment:
TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db
Static Application Security Testing (SAST)
The SpotBugs Maven plugin is used along with the Find Security Bugs plugin to identify OWASP Top 10 and CWE vulnerabilities in the application source code.
Package and Store Artifact(s)
CircleCI utilizes workspaces to transfer artifacts between jobs in the pipeline, allowing seamless handoff from one job to the next. For logs, scan reports, and other assets users want to keep for reference, CircleCI’s artifact storage can be used to retain these files.
The AWS Cloud Development Kit (CDK) is then leveraged to use the artifacts generated earlier in the CircleCI pipeline in the deployment jobs. This process includes building Docker images and publishing them to Amazon ECR repositories, ensuring all assets are readily available for deployment across environments.
Software Composition Analysis (SCA)
Trivy is used to scan the source for vulnerabilities in its dependencies. The pom.xml and Dockerfile files are scanned for configuration issues or vulnerabilities in any dependencies. The scanning is accomplished by a CircleCI Orb that runs Trivy CLI:
trivy:
docker:
- image: cimg/base:2024.11
working_directory: << pipeline.parameters.working-dir >>
steps:
- checkout:
path: ~/aws-deployment-pipeline-reference-architecture
- trivy/scan:
scan-type: fs
ignore-unfixed: true
format: sarif
output: << pipeline.parameters.working-dir >>/report.sarif
scanners: vuln,secret,misconfig,license
- store_artifacts:
path: << pipeline.parameters.working-dir >>/report.sarif
environment:
TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db
Trivy is also used within the Dockerfile to scan the image after it is built. The docker build will fail if Trivy finds any vulnerabilities in the final image:
FROM public.ecr.aws/amazoncorretto/amazoncorretto:17-al2022-jdk as build
USER nobody
WORKDIR /app
COPY target/fruit-api.jar /app
HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 CMD /bin/curl --fail --silent localhost:8080/actuator/health | grep UP || exit 1
ENTRYPOINT ["java","-jar","/app/fruit-api.jar"]
# Use multi-stage builds to scan newly created image with Trivy. This second stage 'vulnscan'
# isn't published to Amazon ECR and is never run. It is only used to run the Trivy scan
# against the newly created image in the 'build' stage.
#
# This stage must run as root so Trivy can scan all files in the image, not just
# those accessible by the nobody user. The user is switched back to 'nobody' at
# the end to ensure that even if this image is used for something it is done
# without the 'root' user.
FROM build AS vulnscan
USER root
COPY --from=aquasec/trivy:latest /usr/local/bin/trivy /usr/local/bin/trivy
RUN trivy filesystem --exit-code 1 --no-progress --ignore-unfixed -s CRITICAL /
USER nobody
Software Bill of Materials (SBOM)
Trivy generates an SBOM in the form of a SPDX JSON report. The SBOM is saved as a CircleCI artifact. Trivy supports additional SBOM formats such as CycloneDX, and SARIF.
Immortally, the Trivy scan to generate the SBOM happens in the same job as the building of the application. This ensures that the software and dependencies used to build the artifact are tracked.
package:
docker:
- image: cimg/openjdk:17.0
working_directory: << pipeline.parameters.working-dir >>
environment:
TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db
steps:
- checkout:
path: ~/aws-deployment-pipeline-reference-architecture
- restore_cache:
keys:
- maven-repo-v1-{{ checksum "pom.xml" }}
- run:
name: Run Maven Verify
command: mvn verify --batch-mode --no-transfer-progress
- save_cache:
paths:
- ~/.m2/repository
key: maven-repo-v1-{{ checksum "pom.xml" }}
- trivy/scan:
scan-type: fs
ignore-unfixed: true
format: spdx-json
output: << pipeline.parameters.working-dir >>/dependency-results.sbom.json
- store_test_results:
path: << pipeline.parameters.working-dir >>/target/surefire-reports
- store_test_results:
path: << pipeline.parameters.working-dir >>/target/soapui-reports
- store_artifacts:
path: << pipeline.parameters.working-dir >>/dependency-results.sbom.json
- store_artifacts:
path: << pipeline.parameters.working-dir >>/target/spotbugsXml.xml
- store_artifacts:
path: << pipeline.parameters.working-dir >>/target/jmeter/results
- store_artifacts:
path: << pipeline.parameters.working-dir >>/target/fruit-api.jar
- persist_to_workspace:
root: << pipeline.parameters.working-dir >>
paths:
- target/fruit-api.jar
Test (Beta)¶
Launch Environment
The infrastructure for each environment is defined in AWS Cloud Development Kit:
super(scope, id, props);
const image = new AssetImage('.', { target: 'build' });
const appName = Stack.of(this)
.stackName.toLowerCase()
.replace(`-${Stack.of(this).region}-`, '-');
const vpc = new ec2.Vpc(this, 'Vpc', {
maxAzs: 3,
natGateways: props?.natGateways,
});
new FlowLog(this, 'VpcFlowLog', {
resourceType: FlowLogResourceType.fromVpc(vpc),
});
const dbName = 'fruits';
const dbSecret = new DatabaseSecret(this, 'AuroraSecret', {
username: 'fruitapi',
secretName: `${appName}-DB`,
});
const db = new DatabaseCluster(this, 'Database', {
engine: DatabaseClusterEngine.auroraMysql({
version: AuroraMysqlEngineVersion.VER_3_07_1,
}),
credentials: Credentials.fromSecret(dbSecret),
writer: ClusterInstance.serverlessV2('writer'),
defaultDatabaseName: dbName,
serverlessV2MaxCapacity: 2,
serverlessV2MinCapacity: 0.5,
vpc,
clusterIdentifier: appName,
storageEncrypted: true,
});
const cluster = new ecs.Cluster(this, 'Cluster', {
vpc,
containerInsights: true,
clusterName: appName,
});
const appLogGroup = new LogGroup(this, 'AppLogGroup', {
retention: RetentionDays.ONE_WEEK,
logGroupName: `/aws/ecs/service/${appName}`,
removalPolicy: RemovalPolicy.DESTROY,
});
let deploymentConfig: IEcsDeploymentConfig | undefined = undefined;
if (props?.deploymentConfigName) {
deploymentConfig = EcsDeploymentConfig.fromEcsDeploymentConfigName(
this,
'DeploymentConfig',
props.deploymentConfigName,
);
}
const appConfigEnabled =
props?.appConfigRoleArn !== undefined &&
props.appConfigRoleArn.length > 0;
const service = new ApplicationLoadBalancedCodeDeployedFargateService(
this,
'Api',
{
cluster,
capacityProviderStrategies: [
{
capacityProvider: 'FARGATE_SPOT',
weight: 1,
},
],
minHealthyPercent: 50,
maxHealthyPercent: 200,
desiredCount: 3,
cpu: 512,
memoryLimitMiB: 1024,
taskImageOptions: {
image,
containerName: 'api',
containerPort: 8080,
family: appName,
logDriver: AwsLogDriver.awsLogs({
logGroup: appLogGroup,
streamPrefix: 'service',
}),
secrets: {
SPRING_DATASOURCE_USERNAME: Secret.fromSecretsManager(
dbSecret,
'username',
),
SPRING_DATASOURCE_PASSWORD: Secret.fromSecretsManager(
dbSecret,
'password',
),
},
environment: {
SPRING_DATASOURCE_URL: `jdbc:mysql://${db.clusterEndpoint.hostname}:${db.clusterEndpoint.port}/${dbName}`,
APPCONFIG_AGENT_APPLICATION:
this.node.tryGetContext('workloadName'),
APPCONFIG_AGENT_ENVIRONMENT:
this.node.tryGetContext('environmentName'),
APPCONFIG_AGENT_ENABLED: appConfigEnabled.toString(),
},
},
deregistrationDelay: Duration.seconds(5),
responseTimeAlarmThreshold: Duration.seconds(3),
targetHealthCheck: {
healthyThresholdCount: 2,
unhealthyThresholdCount: 2,
interval: Duration.seconds(60),
path: '/actuator/health',
},
deploymentConfig,
terminationWaitTime: Duration.minutes(5),
apiCanaryTimeout: Duration.seconds(5),
apiTestSteps: [
{
name: 'getAll',
path: '/api/fruits',
jmesPath: 'length(@)',
expectedValue: 5,
},
],
},
);
if (appConfigEnabled) {
service.taskDefinition.addContainer('appconfig-agent', {
image: ecs.ContainerImage.fromRegistry(
'public.ecr.aws/aws-appconfig/aws-appconfig-agent:2.x',
),
essential: false,
logging: AwsLogDriver.awsLogs({
logGroup: appLogGroup,
streamPrefix: 'service',
}),
environment: {
SERVICE_REGION: this.region,
ROLE_ARN: props!.appConfigRoleArn!,
ROLE_SESSION_NAME: appName,
LOG_LEVEL: 'info',
},
portMappings: [{ containerPort: 2772 }],
});
service.taskDefinition.addToTaskRolePolicy(
new PolicyStatement({
actions: ['sts:AssumeRole'],
resources: [props!.appConfigRoleArn!],
}),
);
}
service.service.connections.allowTo(
db,
ec2.Port.tcp(db.clusterEndpoint.port),
);
this.apiUrl = new CfnOutput(this, 'endpointUrl', {
value: `http://${service.listener.loadBalancer.loadBalancerDnsName}`,
});
The CDK deployment is then performed for each environment. The deployment steps are the same for each environment with only the defined parameters changing like environment and region. Using CircleCI's reusable config, you can defined the deployment job once and use that job for other deployments.
Here is the generic deployment job:
deploy:
docker:
- image: cimg/aws:2024.03
parameters:
environment:
type: string
default: Beta
stack-name:
type: string
default: fruit-api
region:
type: string
default: us-east-1
cdk-context:
type: string
default: deploymentConfigurationName=CodeDeployDefault.ECSCanary10Percent5Minutes
working_directory: << pipeline.parameters.working-dir >>
steps:
- checkout:
path: ~/aws-deployment-pipeline-reference-architecture
- attach_workspace:
at: << pipeline.parameters.working-dir >>
- setup_remote_docker:
docker_layer_caching: true
- aws-cli/setup:
profile_name: default
role_arn: << pipeline.parameters.aws-role-arn >>
role_session_name: default
- install-cdk
- run:
name: Set CDK Env Vars
command: |
echo "export CDK_DEFAULT_ACCOUNT=<< pipeline.parameters.aws-account-id >>" >> $BASH_ENV
echo "export CDK_DEFAULT_REGION=<< parameters.region >>" >> $BASH_ENV
- run:
name: Bootstrap CDK Environment
command: cdk bootstrap aws://$CDK_DEFAULT_ACCOUNT/$CDK_DEFAULT_REGION
- run:
name: Deploy CDK Stack << parameters.stack-name >>
command: |
cdk deploy << parameters.stack-name >> \
--region << parameters.region >> \
--outputs-file cdk-output.json \
--context << parameters.cdk-context >> \
--require-approval never
no_output_timeout: 20m
- run:
command: cat cdk-output.json
- store_artifacts:
path: << pipeline.parameters.working-dir >>/cdk-output.json
- persist_to_workspace:
root: << pipeline.parameters.working-dir >>
paths:
- cdk-output.json
For deploying to the Beta environment the follow job is used:
- deploy:
name: Beta Deploy
requires:
- Verify CDK Synth
- Run Trivy Scan
- Build Application
- Scanning for Secrets with GitGuardian
Database Deploy
Spring Boot is configured to run Liquibase on startup. This reads the configuration in src/main/resources/db/changelog/db.changelog-master.yml to define the tables and initial data for the database:
databaseChangeLog:
- changeSet:
id: "1"
author: AWS
changes:
- createTable:
tableName: fruit
columns:
- column:
name: id
type: bigint
autoIncrement: true
constraints:
primaryKey: true
nullable: false
- column:
name: name
type: varchar(250)
- insert:
tableName: fruit
columns:
- column:
name: name
value: Apple
- insert:
tableName: fruit
columns:
- column:
name: name
value: Orange
- insert:
tableName: fruit
columns:
- column:
name: name
value: Banana
- insert:
tableName: fruit
columns:
- column:
name: name
value: Cherry
- insert:
tableName: fruit
columns:
- column:
name: name
value: Grape
- changeSet:
id: "2"
author: AWS
changes:
- addColumn:
tableName: fruit
columns:
- column:
name: classification
type: varchar(250)
constraints:
nullable: true
- update:
tableName: fruit
columns:
- column:
name: classification
value: pome
where: name='Apple'
- update:
tableName: fruit
columns:
- column:
name: classification
value: berry
where: name='Orange'
- update:
tableName: fruit
columns:
- column:
name: classification
value: berry
where: name='Banana'
- update:
tableName: fruit
columns:
- column:
name: classification
value: drupe
where: name='Cherry'
- update:
tableName: fruit
columns:
- column:
name: classification
value: berry
where: name='Grape'
Deploy Software
The Launch Environment action above creates a new Amazon ECS Task Definition for the new docker image and then updates the Amazon ECS Service to use the new Task Definition.
Integration Tests
Integration tests are preformed during the Build Source action. They are defined with SoapUI in fruit-api-soapui-project.xml. They are executed by Maven in the integration-test phase using plugins in pom.xml. Spring Boot is configure to start a local instance of the application with an H2 database during the pre-integration-test phase and then to terminate on the post-integration-test phase. The results of the unit tests are then uploaded to CircleCI Test Insights to track test performance and trends over time.
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<executions>
<execution>
<id>pre-integration-test</id>
<goals>
<goal>start</goal>
</goals>
</execution>
<execution>
<id>post-integration-test</id>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>com.smartbear.soapui</groupId>
<artifactId>soapui-maven-plugin</artifactId>
<version>5.7.0</version>
<configuration>
<junitReport>true</junitReport>
<outputFolder>target/soapui-reports</outputFolder>
<endpoint>${soapui.endpoint}</endpoint>
</configuration>
<executions>
<execution>
<phase>integration-test</phase>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
Acceptance Tests
Acceptance tests are preformed after the Launch Environment and Deploy Software actions:
The tests are defined with SoapUI in fruit-api-soapui-project.xml. They are executed by Maven with the endpoint overridden to the URL from the CloudFormation output. A test job is added after each deployment in the CircleCI workflow to run SoapUI.
Just like the deployment, you can use CircleCI's reusable config, have a generic test job. The test steps are the same for each environment with only the defined parameters changing like environment and region.
Here is the generic test job:
test:
docker:
- image: cimg/openjdk:17.0
parameters:
environment:
type: string
default: Beta
region:
type: string
default: us-east-1
threads:
type: integer
default: 300
duration:
type: integer
default: 300
throughput:
type: integer
default: 6000
working_directory: << pipeline.parameters.working-dir >>
steps:
- checkout:
path: ~/aws-deployment-pipeline-reference-architecture
- attach_workspace:
at: << pipeline.parameters.working-dir >>
- run:
name: Extract endpointUrl from CDK Output
command: |
# Extract the endpointUrl from the CDK output JSON
if [ -f cdk-output.json ]; then
endpoint_url=$(jq -r '.["fruit-api"].endpointUrl' cdk-output.json)
echo "export ENDPOINT_URL=$endpoint_url" >> $BASH_ENV
echo "Endpoint URL extracted: $endpoint_url"
else
echo "CDK output file not found."
exit 1
fi
- restore_cache:
keys:
- maven-repo-v1-{{ checksum "pom.xml" }}
- run:
name: Run SoapUI Tests against << parameters.environment >>
command: mvn --batch-mode --no-transfer-progress soapui:test -Dsoapui.endpoint=${ENDPOINT_URL}
- run:
name: Run JMeter Tests against << parameters.environment >>
command: mvn --batch-mode --no-transfer-progress compile jmeter:jmeter jmeter:results -Djmeter.endpoint=${ENDPOINT_URL} -Djmeter.threads=<< parameters.threads >> -Djmeter.duration=<< parameters.duration >> -Djmeter.throughput=<< parameters.throughput >>
- save_cache:
paths:
- ~/.m2/repository
key: maven-repo-v1-{{ checksum "pom.xml" }}
- store_test_results:
path: << pipeline.parameters.working-dir >>/target/soapui-reports
- store_artifacts:
path: << pipeline.parameters.working-dir >>/target/jmeter/results
For testing to the Beta environment the follow job is used:
- test:
name: Beta Test
environment: Beta
requires:
- "Beta Deploy"
The results of the unit tests are uploaded to CircleCI Test Insights to be tracked over time.
Test (Gamma)¶
Launch Environment
The infrastructure for each environment is defined in AWS Cloud Development Kit:
super(scope, id, props);
const image = new AssetImage('.', { target: 'build' });
const appName = Stack.of(this)
.stackName.toLowerCase()
.replace(`-${Stack.of(this).region}-`, '-');
const vpc = new ec2.Vpc(this, 'Vpc', {
maxAzs: 3,
natGateways: props?.natGateways,
});
new FlowLog(this, 'VpcFlowLog', {
resourceType: FlowLogResourceType.fromVpc(vpc),
});
const dbName = 'fruits';
const dbSecret = new DatabaseSecret(this, 'AuroraSecret', {
username: 'fruitapi',
secretName: `${appName}-DB`,
});
const db = new DatabaseCluster(this, 'Database', {
engine: DatabaseClusterEngine.auroraMysql({
version: AuroraMysqlEngineVersion.VER_3_07_1,
}),
credentials: Credentials.fromSecret(dbSecret),
writer: ClusterInstance.serverlessV2('writer'),
defaultDatabaseName: dbName,
serverlessV2MaxCapacity: 2,
serverlessV2MinCapacity: 0.5,
vpc,
clusterIdentifier: appName,
storageEncrypted: true,
});
const cluster = new ecs.Cluster(this, 'Cluster', {
vpc,
containerInsights: true,
clusterName: appName,
});
const appLogGroup = new LogGroup(this, 'AppLogGroup', {
retention: RetentionDays.ONE_WEEK,
logGroupName: `/aws/ecs/service/${appName}`,
removalPolicy: RemovalPolicy.DESTROY,
});
let deploymentConfig: IEcsDeploymentConfig | undefined = undefined;
if (props?.deploymentConfigName) {
deploymentConfig = EcsDeploymentConfig.fromEcsDeploymentConfigName(
this,
'DeploymentConfig',
props.deploymentConfigName,
);
}
const appConfigEnabled =
props?.appConfigRoleArn !== undefined &&
props.appConfigRoleArn.length > 0;
const service = new ApplicationLoadBalancedCodeDeployedFargateService(
this,
'Api',
{
cluster,
capacityProviderStrategies: [
{
capacityProvider: 'FARGATE_SPOT',
weight: 1,
},
],
minHealthyPercent: 50,
maxHealthyPercent: 200,
desiredCount: 3,
cpu: 512,
memoryLimitMiB: 1024,
taskImageOptions: {
image,
containerName: 'api',
containerPort: 8080,
family: appName,
logDriver: AwsLogDriver.awsLogs({
logGroup: appLogGroup,
streamPrefix: 'service',
}),
secrets: {
SPRING_DATASOURCE_USERNAME: Secret.fromSecretsManager(
dbSecret,
'username',
),
SPRING_DATASOURCE_PASSWORD: Secret.fromSecretsManager(
dbSecret,
'password',
),
},
environment: {
SPRING_DATASOURCE_URL: `jdbc:mysql://${db.clusterEndpoint.hostname}:${db.clusterEndpoint.port}/${dbName}`,
APPCONFIG_AGENT_APPLICATION:
this.node.tryGetContext('workloadName'),
APPCONFIG_AGENT_ENVIRONMENT:
this.node.tryGetContext('environmentName'),
APPCONFIG_AGENT_ENABLED: appConfigEnabled.toString(),
},
},
deregistrationDelay: Duration.seconds(5),
responseTimeAlarmThreshold: Duration.seconds(3),
targetHealthCheck: {
healthyThresholdCount: 2,
unhealthyThresholdCount: 2,
interval: Duration.seconds(60),
path: '/actuator/health',
},
deploymentConfig,
terminationWaitTime: Duration.minutes(5),
apiCanaryTimeout: Duration.seconds(5),
apiTestSteps: [
{
name: 'getAll',
path: '/api/fruits',
jmesPath: 'length(@)',
expectedValue: 5,
},
],
},
);
if (appConfigEnabled) {
service.taskDefinition.addContainer('appconfig-agent', {
image: ecs.ContainerImage.fromRegistry(
'public.ecr.aws/aws-appconfig/aws-appconfig-agent:2.x',
),
essential: false,
logging: AwsLogDriver.awsLogs({
logGroup: appLogGroup,
streamPrefix: 'service',
}),
environment: {
SERVICE_REGION: this.region,
ROLE_ARN: props!.appConfigRoleArn!,
ROLE_SESSION_NAME: appName,
LOG_LEVEL: 'info',
},
portMappings: [{ containerPort: 2772 }],
});
service.taskDefinition.addToTaskRolePolicy(
new PolicyStatement({
actions: ['sts:AssumeRole'],
resources: [props!.appConfigRoleArn!],
}),
);
}
service.service.connections.allowTo(
db,
ec2.Port.tcp(db.clusterEndpoint.port),
);
this.apiUrl = new CfnOutput(this, 'endpointUrl', {
value: `http://${service.listener.loadBalancer.loadBalancerDnsName}`,
});
The CDK deployment is then performed for each environment. Since the deployment is the same as Beta, but with the environment and region parameters changed, you can refer back to the Beta deployment for the CircleCI config.
Here is the unique Gamma deployment job:
- deploy:
name: "<< matrix.environment >> << matrix.region>> Deploy"
matrix:
parameters:
environment: [Gamma]
region: [us-east-1, us-west-2]
requires:
- "Beta Test"
filters:
branches:
only:
- main
Database Deploy
Spring Boot is configured to run Liquibase on startup. This reads the configuration in src/main/resources/db/changelog/db.changelog-master.yml to define the tables and initial data for the database:
databaseChangeLog:
- changeSet:
id: "1"
author: AWS
changes:
- createTable:
tableName: fruit
columns:
- column:
name: id
type: bigint
autoIncrement: true
constraints:
primaryKey: true
nullable: false
- column:
name: name
type: varchar(250)
- insert:
tableName: fruit
columns:
- column:
name: name
value: Apple
- insert:
tableName: fruit
columns:
- column:
name: name
value: Orange
- insert:
tableName: fruit
columns:
- column:
name: name
value: Banana
- insert:
tableName: fruit
columns:
- column:
name: name
value: Cherry
- insert:
tableName: fruit
columns:
- column:
name: name
value: Grape
- changeSet:
id: "2"
author: AWS
changes:
- addColumn:
tableName: fruit
columns:
- column:
name: classification
type: varchar(250)
constraints:
nullable: true
- update:
tableName: fruit
columns:
- column:
name: classification
value: pome
where: name='Apple'
- update:
tableName: fruit
columns:
- column:
name: classification
value: berry
where: name='Orange'
- update:
tableName: fruit
columns:
- column:
name: classification
value: berry
where: name='Banana'
- update:
tableName: fruit
columns:
- column:
name: classification
value: drupe
where: name='Cherry'
- update:
tableName: fruit
columns:
- column:
name: classification
value: berry
where: name='Grape'
Deploy Software
The Launch Environment action above creates a new Amazon ECS Task Definition for the new docker image and then updates the Amazon ECS Service to use the new Task Definition.
Application Monitoring & Logging
Amazon ECS uses Amazon CloudWatch Metrics and Amazon CloudWatch Logs for observability by default.
Synthetic Tests
Amazon CloudWatch Synthetics is used to continuously deliver traffic to the application and assert that requests are successful and responses are received within a given threshold. The canary is defined via CDK using the @cdklabs/cdk-ecs-codedeploy construct:
const service = new ApplicationLoadBalancedCodeDeployedFargateService(this, 'Api', {
...
apiCanaryTimeout: Duration.seconds(5),
apiTestSteps: [{
name: 'getAll',
path: '/api/fruits',
jmesPath: 'length(@)',
expectedValue: 5,
}],
Performance Tests
Apache JMeter is used to run performance tests against the deployed application. The tests are stored in src/test/jmeter and added to the CircleCI workflow:
Just like the deployment, you can use CircleCI's reusable config, have a generic test job. Since the testing is the same as Beta, but with the environment and region parameters changed, you can refer back to the Beta testing for the CircleCI config.
The test steps are the same for each environment with only the defined parameters changing like environment and region.
Here is the unique Gamma deployment job:
- test:
name: "<< matrix.environment >> << matrix.region>> Test"
matrix:
parameters:
environment: [Gamma]
region: [us-east-1, us-west-2]
requires:
- << matrix.environment >> << matrix.region>> Deploy
filters:
branches:
only:
- main
Resilience Tests
Not Implemented
Dynamic Application Security Testing (DAST)
Not Implemented
Prod¶
Manual Approval
For manual approvals, you can use CircleCI’s hold job, which pauses the pipeline and waits for someone to approve it before proceeding. In this reference architecture, manual approvals are used to gate deployments to production. If code is pushed to a development or non-production branch, the pipeline will run all jobs up to the Gamma deployment stage. For production branches (in this case, main), the pipeline will deploy to Gamma, and if successful, a manual approval gate will activate. Once a designated individual or group approves this job, the pipeline will continue to deploy to production.
Database Deploy
Spring Boot is configured to run Liquibase on startup. This reads the configuration in src/main/resources/db/changelog/db.changelog-master.yml to define the tables and initial data for the database:
databaseChangeLog:
- changeSet:
id: "1"
author: AWS
changes:
- createTable:
tableName: fruit
columns:
- column:
name: id
type: bigint
autoIncrement: true
constraints:
primaryKey: true
nullable: false
- column:
name: name
type: varchar(250)
- insert:
tableName: fruit
columns:
- column:
name: name
value: Apple
- insert:
tableName: fruit
columns:
- column:
name: name
value: Orange
- insert:
tableName: fruit
columns:
- column:
name: name
value: Banana
- insert:
tableName: fruit
columns:
- column:
name: name
value: Cherry
- insert:
tableName: fruit
columns:
- column:
name: name
value: Grape
- changeSet:
id: "2"
author: AWS
changes:
- addColumn:
tableName: fruit
columns:
- column:
name: classification
type: varchar(250)
constraints:
nullable: true
- update:
tableName: fruit
columns:
- column:
name: classification
value: pome
where: name='Apple'
- update:
tableName: fruit
columns:
- column:
name: classification
value: berry
where: name='Orange'
- update:
tableName: fruit
columns:
- column:
name: classification
value: berry
where: name='Banana'
- update:
tableName: fruit
columns:
- column:
name: classification
value: drupe
where: name='Cherry'
- update:
tableName: fruit
columns:
- column:
name: classification
value: berry
where: name='Grape'
Progressive Deployment
Progressive deployment is implemented with AWS CodeDeploy for ECS. CodeDeploy performs a linear blue/green by deploying the new task definition as a new task with a separate target group and then shifting 10% of traffic every minute until all traffic is shifted. A CloudWatch alarm is monitored by CodeDeploy and an automatic rollback is triggered if the alarm exceeds the threshold.
Implementation of this type deployment presents challenges due to the following limitations:
- aws/aws-cdk #19163 - CDK Pipelines aren't intended to be used with CodeDeploy actions.
- AWS CloudFormation User Guide - The use of
AWS::CodeDeploy::BlueGreenhooks andAWS::CodeDeployBlueGreenrestricts the types of changes that can be made. Additionally, you can't use auto-rollback capabilities of CodeDeploy. - aws/aws-cdk #5170 - CDK doesn't support defining CloudFormation rollback triggers. This rules out CloudFormation based blue/green deployments.
The solution was to use the @cdklabs/cdk-ecs-codedeploy construct from the Construct Hub which addresses aws/aws-cdk #1559 - Lack of support for Blue/Green ECS Deployment in CDK.
const service = new ApplicationLoadBalancedCodeDeployedFargateService(this, 'Api', {
cluster,
capacityProviderStrategies: [
{
capacityProvider: 'FARGATE_SPOT',
weight: 1,
},
],
minHealthyPercent: 50,
maxHealthyPercent: 200,
desiredCount: 3,
cpu: 512,
memoryLimitMiB: 1024,
taskImageOptions: {
image,
containerName: 'api',
containerPort: 8080,
family: appName,
logDriver: AwsLogDriver.awsLogs({
logGroup: appLogGroup,
streamPrefix: 'service',
}),
secrets: {
SPRING_DATASOURCE_USERNAME: Secret.fromSecretsManager( dbSecret, 'username' ),
SPRING_DATASOURCE_PASSWORD: Secret.fromSecretsManager( dbSecret, 'password' ),
},
environment: {
SPRING_DATASOURCE_URL: `jdbc:mysql://${db.clusterEndpoint.hostname}:${db.clusterEndpoint.port}/${dbName}`,
},
},
deregistrationDelay: Duration.seconds(5),
responseTimeAlarmThreshold: Duration.seconds(3),
healthCheck: {
healthyThresholdCount: 2,
unhealthyThresholdCount: 2,
interval: Duration.seconds(60),
path: '/actuator/health',
},
deploymentConfig,
terminationWaitTime: Duration.minutes(5),
apiCanaryTimeout: Duration.seconds(5),
apiTestSteps: [{
name: 'getAll',
path: '/api/fruits',
jmesPath: 'length(@)',
expectedValue: 5,
}],
});
this.apiUrl = new CfnOutput(this, 'endpointUrl', {
value: `http://${service.listener.loadBalancer.loadBalancerDnsName}`,
});
```rrideLogicalId('DeploymentId');
Deployments are rolled out incrementally across regions in waves using CircleCI’s matrix and parallelism. Each wave consists of a set of regions to deploy to in parallel, allowing you to deploy to multiple regions simultaneously within each wave. Once a wave completes, the pipeline will proceed to the next wave, ensuring a controlled, staggered deployment across regions.
Synthetic Tests
Amazon CloudWatch Synthetics is used to continuously deliver traffic to the application and assert that requests are successful and responses are received within a given threshold. The canary is defined via CDK using the @cdklabs/cdk-ecs-codedeploy construct:
const service = new ApplicationLoadBalancedCodeDeployedFargateService(this, 'Api', {
...
apiCanaryTimeout: Duration.seconds(5),
apiTestSteps: [{
name: 'getAll',
path: '/api/fruits',
jmesPath: 'length(@)',
expectedValue: 5,
}],
Frequently Asked Questions¶
What operating models does this reference implementation support?
This reference implementation can accommodate any operation model with minor updates:
- Fully Separated - Restrict the role that CDK uses for CloudFormation execution to only create resources from approved product portfolios in AWS Service Catalog. Ownership of creating the products in Service Catalog is owned by the Platform Engineering team and operational support of Service Catalog is owned by the Platform Operations team. The Platform Engineering team should publish CDK constructs internally that provision AWS resources through Service Catalog. Update the CDK app in the
infrastructure/directory to use CDK constructs provided by thePlatform Engineeringteam. Use a CODEOWNERS file to require all changes to theinfrastructure/directory be approved by the Application Operations team. Additionally, restrict permissions to the Manual Approval action to only allow members of the Application Operations to approve. - Separated AEO and IEO with Centralized Governance - Restrict the role that CDK uses for CloudFormation execution to only create resources from approved product portfolios in AWS Service Catalog. Ownership of creating the products in Service Catalog is owned by the Platform Engineering team and operational support of Service Catalog is owned by the Platform Engineering team. The Platform Engineering team should publish CDK constructs internally that provision AWS resources through Service Catalog. Update the CDK app in the
infrastructure/directory to use CDK constructs provided by thePlatform Engineeringteam. - Separated AEO and IEO with Decentralized Governance - The Platform Engineering team should publish CDK constructs internally that provision AWS resources in manner that achieve organizational compliance. Update the CDK app in the
infrastructure/directory to use CDK constructs provided by thePlatform Engineeringteam.