Don’t touch AWS resources you haven’t created!

In this blog post we want to share with you a golden rule we have confirmed recently: don’t touch resources created by other services.

Or to be more concrete about our today subject: don’t touch S3 buckets that appear in your console, but which have been created by other services.

AWS S3 is one of the main pillars of the entire AWS ecosystem and it’s integrated in various uses cases:

  • where snapshots are saved: DynamoDB, RDS, etc.
  • a place where artifacts are persisted: Beanstalk, CloudFormation, etc.
  • input from where to read data: Redshift, Kinesis Firehose, etc.

Recently a good friend called us complaining that his Elastic Beanstalk service suddenly ceased to work. That was an exciting challenge. We have assumed and we decided to investigate deeper to see what happened.

It took us more than 1 day to find the root cause, but in the end we were able to find the root cause: a lifecycle rule was added to the S3 bucket created by AWS Beanstalk. That rule deletes all files in that bucket that is older than 30 days. So, if no deployment is done in the last 30 days, then when Beanstalk try to spin up a new instance, it cannot find the artifacts to install on it and, consequently, it fails with an error like this:

No manifest found. (ElasticBeanstalk::ManifestDownloadError)

The solution was simple: upload again the artifacts and deploy them. But the time spent to find the root cause plus losses caused by this incident cannot be neglected.

So, to conclude, the idea is very simple: even we speak about a S3 bucket, a SNS topic, a security group or whatever other resource created by other service on behalf of you, it’s better not to touch them if you are not 300% sure you don’t make any mistake!

Happy AWS week!