Automate Spring Runtime Configuration with AWS Lambda, CDK, and CloudWatch Alarms
By
In this article I present an peculiar use case to demonstrate how one could piece together various serverless cloud computing resources and services to automate the runtime configuration (and behavior) of a Spring Boot app deployed with Spring Actuator. The Spring Boot app is built as a containerized application then deployed to Elastic Container Service (ECS) on the Serverless Fargate Container Runtime within a Virtual Private Cloud (VPC) and exposed to client apps through an Application Load Balancer (ALB) using the Cloud Development Kit (CDK).
Once deployed a CloudWatch log filter based Alarm is deployed to trigger into Alarm State on the presence of ERROR in the logs which then is broadcasted to an AWS Lambda which securely calls the Spring Actuator loggers endpoint and modifies the runtime logging levels to verbose logging. The CloudWatch Alarm eventually undergoes state transition back to OK and broadcasts the transition to another AWS Lambda that again makes a secure request to the Actuator endpoint reducing the verbosity of the logging level.
The source code for this demo project is available on GitHub.
Building the CDK Project
As mentioned previously I'll be deploying my Spring Boot app to ECS on Fargate using AWS CDK so, I must first create a new CDK project using the CDK CLI. If you wish to code along, please have the AWS CDKv2 installed as described in the AWS CDK docs.
Create a new directory to work in named cdk-springboot-actuator-automation.
mkdir cdk-springboot-actuator-automation
cd cdk-springboot-actuator-automationInitialize a new CDK project (I'll be using Typescript).
cdk init app --language typescriptRun a quick build to ensure everything is setup correctly.
npm run buildIf you get an error like this.
node_modules/@types/prettier/index.d.ts:41:54 - error TS2315: Type 'IsTuple' is not generic.
41 type IndexProperties<T extends { length: number }> = IsTuple<T> extends true
~~~~~~~~~~
node_modules/@types/prettier/index.d.ts:53:6 - error TS2456: Type alias 'IsTuple' circularly references itself.Install this version of a dev dependency called prettier then rebuild (problem should go away).
npm i @types/prettier@2.6.0 --save-devMost of the CDK code for this demo will live in the lib/cdk-springboot-actuator-automation-stack.ts which I'll introduce the updates to it incrementially describing the functionality as I go. To start I use the Secret construct to generate a random password to go with a username of awslambda and be stored securely in AWS Secrets Manager. After that I provision a VPC spanning 2 AZs complete with a pair of public & private subnets which is then utilized to deploy a Fargate ECS Cluster.
export class CdkSpringbootActuatorAutomationStack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
const authSecret = new secretsmgr.Secret(this, "auth-secret", {
secretName: "/greeter/actuator/auth-creds",
generateSecretString: {
secretStringTemplate: JSON.stringify({ username: "awslambda" }),
generateStringKey: "password"
}
});
const vpc = new ec2.Vpc(this, "vpc", {
maxAzs: 2
});
const cluster = new ecs.Cluster(this, "cluster", {
vpc
});
}Next I add a Task Role to provide it granular permissions read the previously generated secret which will be granted to the ECS Task running the Spring Boot app giving it acces to the secret. After this I add a log group for capturing logs from my Spring Boot app then utilize the ApplicationLoadBalancedFargateService high level construct to deploy the app.
export class CdkSpringbootActuatorAutomationStack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
// ... omitting previous code for brevity
const taskRole = new iam.Role(this, "task-role", {
assumedBy: new iam.ServicePrincipal("ecs-tasks.amazonaws.com")
});
authSecret.grantRead(taskRole);
const logGroup = new logs.LogGroup(this, "app-logs", {
logGroupName: "greeter-loggroup",
retention: logs.RetentionDays.THREE_DAYS,
removalPolicy: RemovalPolicy.DESTROY
});
const fargateSvc = new ecs_patterns.ApplicationLoadBalancedFargateService(this, "fargate-svc", {
cluster,
taskImageOptions: {
image: ecs.ContainerImage.fromAsset("actuator-aws-automation"),
containerName: "app",
containerPort: 8080,
environment: {
AWS_ACTUATOR_SECRET: authSecret.secretName
},
logDriver: ecs.LogDriver.awsLogs({
logGroup,
streamPrefix: "app-"
}),
taskRole
},
taskSubnets: {
subnetType: ec2.SubnetType.PRIVATE_WITH_NAT
}
});
fargateSvc.node.addDependency(authSecret);
fargateSvc.targetGroup.configureHealthCheck({ path: "/actuator/health" });
new CfnOutput(this, "greeter-endpoint", {
value: `http://${fargateSvc.loadBalancer.loadBalancerDnsName}/greet/joker`
});
}The ApplicationLoadBalancedFargateService construct is configured to be deployed to the private subnets, expose port 8080 into the container which is expected to be built from a Dockerfile (to be added later) in the actuator-aws-automation directory. Perhaps the most important thing to point out is how an environment variable representing the name of the secret is being injected into the container at runtime. I also configure a path to the actuator health endpoint to be used by the ALB to route traffic to healthy instances of the app.
Next I add a couple of AWS Systems Manager String List Parameters for externalizing some logging level configurations which will be consumed by AWS Lambda functions and pushed to the Spring Boot app through the REST interfaces exposed by Actuator.
export class CdkSpringbootActuatorAutomationStack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
// ... omitting previous code for brevity
const verboseLogsParam = new ssm.StringListParameter(this, "verbose-logs", {
parameterName: "/greeter/verbose-logs",
stringListValue: [
"org.springframework.security:DEBUG",
"com.thecodinginterface.actuatorawsautomation:DEBUG"
]
});
const tidyLogsParam = new ssm.StringListParameter(this, "tidy-logs", {
parameterName: "/greeter/tidy-logs",
stringListValue: [
"org.springframework.security:WARN",
"com.thecodinginterface.actuatorawsautomation:ERROR"
]
});
const errorAlarmTransitionFn = new lambdaNodeJs.NodejsFunction(this, "error-alarm-fn", {
entry: path.resolve(__dirname, "handlers", "loglevel-transition-handler.ts"),
handler: "handler",
logRetention: logs.RetentionDays.THREE_DAYS,
environment: {
AWS_ACTUATOR_SECRET: authSecret.secretName,
LOGS_PARAM: verboseLogsParam.parameterName,
SERVICE_ENDPOINT: `http://${fargateSvc.loadBalancer.loadBalancerDnsName}`
}
});
const okAlarmTransitionFn = new lambdaNodeJs.NodejsFunction(this, "ok-alarm-fn", {
entry: path.resolve(__dirname, "handlers", "loglevel-transition-handler.ts"),
handler: "handler",
logRetention: logs.RetentionDays.THREE_DAYS,
environment: {
AWS_ACTUATOR_SECRET: authSecret.secretName,
LOGS_PARAM: tidyLogsParam.parameterName,
SERVICE_ENDPOINT: `http://${fargateSvc.loadBalancer.loadBalancerDnsName}`
}
});
errorAlarmTransitionFn.node.addDependency(authSecret, verboseLogsParam, fargateSvc);
okAlarmTransitionFn.node.addDependency(authSecret, tidyLogsParam, fargateSvc);
authSecret.grantRead(errorAlarmTransitionFn);
authSecret.grantRead(okAlarmTransitionFn);
verboseLogsParam.grantRead(errorAlarmTransitionFn);
tidyLogsParam.grantRead(okAlarmTransitionFn);
}
Here I've used the NodejsFunction construct because it makes it incredibly simple to bundle and deploy a Typescript based NodeJS AWS Lambda function just by pointing it to a typescript file which it then handles the building and compilation to NodeJS compliant code. First I add a couple more dependencies that I'll utilize in the Lambda based automation.
npm i @aws-sdk/signature-v4-crt @aws-sdk/client-ssm @aws-sdk/client-secrets-manager \
@types/aws-lambda aws-lambda esbuild axios@^0.27.2 --save-devNow I can put these dependencies to work to whip up the AWS Lambda source in a Typescript file at lib/handlers/loglevel-transition-handler.ts which fetches the logging level configuration from Systems Manager Parameter Store. Next I fetch the credentials securing the Spring Actuator /loggers endpoint from Secrets Manager before using them to update the runtime logging level configuration.
import { SNSEvent } from "aws-lambda";
import { SecretsManagerClient, GetSecretValueCommand } from "@aws-sdk/client-secrets-manager";
import { SSMClient, GetParameterCommand } from "@aws-sdk/client-ssm";
import axios from "axios";
const secretsMgr = new SecretsManagerClient({ region: process.env.AWS_REGION });
const ssm = new SSMClient({ region: process.env.AWS_REGION });
interface LoggerLevel {
readonly name: string;
readonly configuredLevel: string;
}
export async function handler(event: SNSEvent) {
const ssmResponse = await ssm.send(new GetParameterCommand({
Name: process.env.LOGS_PARAM
}));
console.log(`ssmResponse = ${JSON.stringify(ssmResponse, null, 2)}`);
const loggerLevels = ssmResponse.Parameter?.Value?.split(",").map<LoggerLevel>(item => {
const [name, configuredLevel] = item.split(":");
return { name, configuredLevel };
});
console.log(`loggerLevels=${JSON.stringify(loggerLevels, null, 2)}`);
if (!loggerLevels) {
return;
}
const secretsMgrResponse = await secretsMgr.send(new GetSecretValueCommand({
SecretId: process.env.AWS_ACTUATOR_SECRET
}));
const creds = JSON.parse(secretsMgrResponse.SecretString!);
for (const loggerLevel of loggerLevels) {
const axiosResponse = await axios.post(
`${process.env.SERVICE_ENDPOINT}/actuator/loggers/${loggerLevel.name}`,
{ configuredLevel: loggerLevel.configuredLevel },
{ auth: { ...creds } }
);
console.log(`axiosResponse status=${axiosResponse.status} for ${loggerLevel.name} set to ${loggerLevel.configuredLevel}`);
}
}
What remains, over in lib/cdk-springboot-actuator-automation-stack.ts, is to establish a filter metric based CloudWatch Alarm on the application log triggering on the presence of the ERROR term broadcasted through SNS topics which are subscribed to by the previously described AWS Lambda Functions.
export class CdkSpringbootActuatorAutomationStack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
// ... omitting previous code for brevity
const filter = logGroup.addMetricFilter("error-filter", {
filterPattern: logs.FilterPattern.anyTerm("ERROR"),
metricName: "greeter-log-errors",
metricNamespace: "GREETER",
defaultValue: 0
});
const errorAlarmMetric = filter.metric({ statistic: "sum" });
const errorAlarm = errorAlarmMetric.createAlarm(this, "error-alarm", {
threshold: 1,
evaluationPeriods: 1
});
const errorTopic = new sns.Topic(this, "error-topic", {
topicName: "greeter-error-topic",
displayName: "Integrates CloudWatch Alarm to Lambda on Alarm State Transition"
});
const okTopic = new sns.Topic(this, "ok-topic", {
topicName: "greeter-ok-topic",
displayName: "Integrates CloudWatch Alarm to Lambda on Ok State Transition"
});
errorTopic.addSubscription(new snsSubs.LambdaSubscription(errorAlarmTransitionFn));
okTopic.addSubscription(new snsSubs.LambdaSubscription(okAlarmTransitionFn));
errorAlarm.addAlarmAction(new cwActions.SnsAction(errorTopic));
errorAlarm.addOkAction(new cwActions.SnsAction(okTopic));
}
At this point I've completed my setup of the CDK project and I can move on to creating my Spring Boot app.
Building the Spring Boot App
Like most of the Spring Boot community I'll be heading over to Spring Initializr website to generate a new Spring Boot app that uses Spring Web to build a REST API, Spring Boot Actuator to manage the runtime configuration, Spring Security to secure the app, and Lombok to reduce boilerplate code.
I move the starter project to the same directory as the CDK project I initialized earlier expand it and open it in an editor. First things first I rename the application.properties file converting it to YAML as application.yml. Next I add the following configuration to expose the loggers and health endpoints in made available by Spring Actuator, then configure logging levels as well as the applicaiton name and my target AWS region.
management:
endpoints:
web:
exposure:
include:
- loggers
- health
endpoint:
loggers:
enabled: true
logging:
level:
root: ERROR
org:
springframework:
security: WARN
spring:
application:
name: greeter
aws:
# change to your target aws region
region: us-east-1
After this I update the build.gradle file to pull in dependencies for the AWS Java SDK to access credentials from AWS Secrets Manager to secure the applciation with externalized credentials available to other AWS Services like AWS Lambda for secure integration.
repositories {
mavenCentral()
}
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-actuator'
implementation 'org.springframework.boot:spring-boot-starter-security'
implementation 'org.springframework.boot:spring-boot-starter-web'
implementation platform('software.amazon.awssdk:bom:2.15.0')
implementation 'software.amazon.awssdk:secretsmanager:2.15.0'
compileOnly 'org.projectlombok:lombok'
annotationProcessor 'org.projectlombok:lombok'
testImplementation 'org.springframework.boot:spring-boot-starter-test'
testImplementation 'org.springframework.security:spring-security-test'
}I add an exclude to the @SpringBootApplication annotation so that I can later control the password to be used to protect the actuator endpoint which I'll source from a randomly generated password provisioned and stored in AWS Secrets Manager via the CDK. To do this I must exclude the SecurityAutoConfiguration class from using autoconfiguration to auto generate the password. I also expose a /greet/{name} endpoint that logs messages at various levels allowing me to introduce varying log statements including the INFO, WARN, and ERROR keywords.
@RestController
@SpringBootApplication(exclude = { SecurityAutoConfiguration.class })
public class ActuatorAwsAutomationApplication {
final static Logger logger = LoggerFactory.getLogger(ActuatorAwsAutomationApplication.class);
public static void main(String[] args) {
SpringApplication.run(ActuatorAwsAutomationApplication.class, args);
}
@GetMapping("greet/{name}")
public Map<String, String> greet(@PathVariable String name) {
logger.info("Greeting {}", name);
if (name.equalsIgnoreCase("two-face")) {
logger.warn("Be careful greeting sketchy characters");
}
if (name.equalsIgnoreCase("joker")) {
logger.error("Report criminal to authorities.");
}
return Map.of("greetings", "Hello " + name);
}
}I add a new file to the project named SpringConfig.java to configure Spring Security to fetch credentials as username and password from AWS Secrets Manager to be used to protect the Actuator Endpoints with the logger runtime configuration capabilities which incidentially is accomplished by exposing the health and greet endpoints leaving all others secured.
@Configuration
@EnableWebSecurity
public class SecurityConfig {
static final Logger logger = LoggerFactory.getLogger(SecurityConfig.class);
@Value("${aws.actuator.secret}")
public String actuatorSecret;
@Value("${aws.region}")
public String awsRegion;
public ActuatorAuthCreds actuatorAuthCreds() throws JsonProcessingException {
if (actuatorSecret == null || !actuatorSecret.startsWith("/greeter")) {
return new ActuatorAuthCreds("actuator", "Develop3r");
}
var secretsMgr = makeSecretsManagerClient();
GetSecretValueResponse response = secretsMgr.getSecretValue(
GetSecretValueRequest.builder().secretId(actuatorSecret).build()
);
ObjectMapper objMapper = new ObjectMapper();
try {
return objMapper.readValue(response.secretString(), ActuatorAuthCreds.class);
} finally {
secretsMgr.close();
}
}
public SecretsManagerClient makeSecretsManagerClient() {
return SecretsManagerClient.builder()
.region(Region.of(awsRegion))
.build();
}
@Bean
public UserDetailsService userDetailsService(PasswordEncoder passwordEncoder) {
ActuatorAuthCreds authCreds = null;
try {
authCreds = actuatorAuthCreds();
} catch (JsonProcessingException e) {
logger.error("failed fetching actuator creds", e);
}
UserDetails actuator = User.builder()
.username(authCreds.getUsername())
.password(passwordEncoder.encode(authCreds.getPassword()))
.roles("ACTUATOR", "ADMIN", "USER")
.build();
return new InMemoryUserDetailsManager(actuator);
}
@Bean
public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {
http.csrf().disable().authorizeRequests()
.antMatchers("/actuator/health").anonymous()
.antMatchers("/greet/**").anonymous()
.anyRequest()
.authenticated()
.and()
.httpBasic();
return http.build();
}
@Bean
public PasswordEncoder passwordEncoder() {
PasswordEncoder encoder = PasswordEncoderFactories.createDelegatingPasswordEncoder();
return encoder;
}
@Data
@AllArgsConstructor
@NoArgsConstructor
static final public class ActuatorAuthCreds {
private String username;
private String password;
}
}
The last piece to finish up that is specific to building out the Spring Boot app is adding a Dockerfile to package the app and its runtime up into a Docker image.
# syntax=docker/dockerfile:experimental
FROM eclipse-temurin:11-jdk-alpine AS build
WORKDIR /workspace/app
COPY . .
RUN --mount=type=cache,target=/root/.gradle ./gradlew clean build -x test
RUN mkdir -p build/dependency && cp build/libs/*-SNAPSHOT.jar build/dependency/app.jar
FROM eclipse-temurin:11-jdk-alpine
VOLUME /tmp
RUN apk --no-cache add curl
ARG DEPENDENCY=/workspace/app/build/dependency
COPY --from=build /workspace/app/build/dependency/app.jar /app.jar
ENTRYPOINT ["java", "-jar", "./app.jar"]
Deploy to AWS Cloud and Test the Functionality
At this point I can deploy the project using NPM and the CDK CLI.
npm run cdk synth
npm run cdk deployAfter the deploy is successful there will be an output specifying the Greet endpoint exposed through the Application Load Balancer which I can use to generate an Error.
curl http://some-letters-and-numbers.region.elb.amazonaws.com/greet/jokerThis will trigger the CloudWatch Alarm to go into ALARM state which then through SNS invokes the first AWS Lambda function to fetch the verbose logging settings and apply then to the Spring Boot app through the protected Actuator /actuator/loggers management endpoint.
I can the inspect the CLoudWatch Logs for the Spring Boot app deployed to ECS and I can see that shortly after the ERROR term is first seen in the logs the log level shifts to DEBUG.
As long as I don't generate another Joker greeting the CloudWatch Alarm will eventually (in about 5 mins) transition back to the OK state initiating the chain of events leading the the second AWS Lambda function restoring the logging levels of the Spring Boot app to original levels.
When I'm done playing around with my app I destroy the environment to cease the charges I'm incurring in my AWS Account.
npm run cdk destroyConclusion
In this article I demonstrated how one can deploy a Spring Boot app the AWS Cloud with ECS and Fargate then used the Spring Boot Actuator project along with CloudWatch and AWS Lambda to modify runtime configuration in an event driven automated fashion.