Project no.8

Goals

Reduce fear of implementing resiliency testing by providing examples in common development and scripting languages
Illustrate how AWS Fault Injection Simulator (FIS) can implement chaos testing using AWS native tooling and integrations
Show how failure injection fits into the context of Chaos Engineering
Demonstrate resilience testing of EC2 instances
Demonstrate resilience testing of RDS Multi-AZ instances
Demonstrate resilience testing using Availability Zones failures
Demonstrate resilience testing of application failures.
Demonstrate resilience testing of S3 objects
Learn how to implement resiliency using those tests
Learn how to think about what a failure will cause within your infrastructure
Learn how common AWS services can reduce mean time to recovery (MTTR)

Material source credit: https://wellarchitectedlabs.com/

Architecture diagram of the demo

Simulated a complete failure of the RecommendationService (DynamoDB). The servers were not returning actual http error codes, rather they failed to return any data at all. To fix this issue, updated server code was deployed. Now, the Website is working again, but in a degraded capacity since it is no longer serving personalized recommendations. While this is less than ideal, it is much better than when it was failing with http 502 errors. The RecommendationService is not available, so the app instead returns a static response (the default recommendation) instead of the data it would have obtained from RecommendationService.

The ELB health check is configured to return healthy when it receives an http 200 response on the /healthcheck path. Since the healthcheck code simply always returns http 200, the bad server still returns http 200 and is seen as healthy.

Update server code to add a deep health check response, and create and configure a new health check that will include a check on whether the server can access its dependency.
This is a deep health check – it checks the actual function of the server including the ability to call service dependencies

The ELB has identified the us-east-2c server as unhealthy and will not route traffic to it

Project no.8

Back to Projects page