A collection of bash shell scripts for automating various tasks withAmazon Web Servicesusing theAWS CLIandjq.
Why does this project exist?
This repository is intended to make some of the more difficult DevOps tasks commonly required to maintain complex hosting infrastructure in AWS simple, quick and easy. This is my attempt to automate and expedite difficult, repetitive, tedious and time consuming processes into a simple shell script that gets the job done as cleanly as possible. These scripts were developed out of frustration in clicking around on the same things over and over again in the web console every day, week, month when they could easly be done in seconds in a script that uses the AWS CLI. I've tried to keep everything applicable to as many use cases across regions and across as many different AWS accounts as possible. I run many of these scripts myself, mostly on a Mac or in Linux and do periodic usability and bug checking, making updates for any changes to the CLI. I hope this collection of tools helps you too, and if you use these please hit the Star/Fork button and if you have any suggestions please open an Issue or PR!
The AWS CLI is an open source tool built on top of the AWS SDK for Python (Boto) that provides commands for interacting with AWS services.
Installing the AWS Command Line Interface
Requirements:
- Python 2 version 2.6.5+ or Python 3 version 3.3+
- macOS, Linux, or Unix
If you already have pip and a supported version of Python, you can install the AWS CLI with the following command:
$ pip install awscli --upgrade --user
For general use, the aws configure command is the fastest way to set up your AWS CLI installation.
$ aws configure
The AWS CLI will prompt you for four pieces of information. AWS Access Key ID and AWS Secret Access Key are your account credentials.
The AWS CLI supports named profiles stored in the config and credentials files. You can configure additional profiles by usingaws configurewith the--profile optionor by adding entries to the config and credentials files.
$ aws configure --profile example
jq is a lightweight and flexible command-line JSON processor.
OS X: UseHomebrewto install jq:
$ brew install jq
Linux: jq is in the officialAmazon Linux AMI,DebianandUbunturepositories.
Amazon Linux AMI, RHEL, CentOS:
$ sudo yum install jq
Debian/Ubuntu:
$ sudo apt-get install jq
- cloudfront-inprogress-status.shChecks CloudFront Distributions with In-Progress status to complete and become Deployed
- cloudfront-invalidation-status.shChecks CloudFront Distributions for cache invalidation status to complete
- cloudwatch-create-alarms.shCreate CloudWatch alarms for EC2, RDS, Load Balancer environments
- cloudwatch-create-alarms-statuscheckfailed.shCreate CloudWatch StatusCheckFailed Alarms with Recovery Action for all running EC2 Instances in all regions available
- cloudwatch-create-alarms-unhealthyhost.shCreate CloudWatch UnhealthyHost Alarms for all ALB and ELB Elastic Load Balancers in all regions available
- cloudwatch-logs-cleanup.shDelete all CloudWatch Log Groups with a Last Event that is older than the Retention Policy
- cloudwatch-logs-delete-groups.shQuickly delete all CloudWatch Log Groups with a specified prefix in all regions available
- cloudwatch-logs-search.shSearch CloudWatch Logs for any string across all regions and log groups
- cloudwatch-logs-retention-policy.shSet CloudWatch Logs Retention Policy for all log groups in all regions available
- ec2-ami-encrypted-ebs-boot-volume.shCreate an AMI with an encrypted EBS boot volume from the latest Amazon Linux AMI
- ec2-associate-elastic-ip.shReassign a previously allocated Elastic IP to the instance which runs this script
- ec2-classic-import-network-acl.shImport CIDR IP list to AWS EC2 Classic ACL rules and deny access
- ec2-ebs-create-snapshots.shCreate a snapshot of each EC2 EBS volume that is tagged with the backup flag
- ec2-ebs-delete-snapshots.shDeletes snapshots for each EC2 EBS volume that is tagged with the backup flag and matches the specified date
- ec2-elb-export-template.shExport an ELB to a JSON template file for version control, duplication or recreation
- ec2-elb-upload-ssl-cert.shUpload an SSL Certificate to AWS for use in setting up an ELB
- elastic-beanstalk-set-hostname-within-instance.shSet the hostname on Elastic Beanstalk servers from within the instance with their EB environment name and public IP address
- elastic-beanstalk-update-hostnames.shUpdates the hostname on Elastic Beanstalk servers with their environment name and IP address
- iam-create-s3-users.shCreate the S3 IAM user, generate IAM keys, add to IAM group, generate user policy
- route53-export-zones.shUsescli53to export the zone file for each Hosted Zone domain in Route 53
- s3-buckets-local-backup.shBackup all contents of all S3 buckets in AWS account locally
- s3-buckets-security-audit.shExport S3 bucket ACL, CORS, Policy and Website as JSON for auditing security of all buckets
- s3-buckets-total-file-size.shCount total size of all data stored in all S3 buckets usings3api
- s3-fix-content-type-metadata.shSafely fix invalid content-type metadata on AWS S3 bucket website assets for some common filetypes
- s3-open-bucket-policy.shSet an S3 bucket policy to allow GetObject requests from any IP address (publicly accessible website)
- s3-remove-glacier-objects.shDelete all Glacier storage type objects in a single S3 bucket
- s3-restrict-bucket-policy.shSet an S3 bucket policy to only allow GetObject requests from an IP whitelist file named iplist
- s3-set-cache-control-max-age.shSet Cache-Control max-age value on AWS S3 bucket website assets for all filetypes
- s3-setup-buckets.shCreate S3 buckets, set CORS config and tag bucket with client name
- vpc-eni-monitor.shGenerate an HTML page to monitor the number of AWS VPC Elastic Network Interfaces currently in use and upload it to an S3 bucket website
- vpc-sg-import-rules.shCreate an AWS VPC Security Group with rules to allow access to each IP at the port specified
- vpc-sg-import-rules-cloudflare.shCreate VPC Security Group withCloudflareIP ranges
- vpc-sg-import-rules-cloudfront.shCreate VPC Security Group with CloudFront IP ranges
- vpc-sg-import-rules-pingdom.shCreate or Update VPC Security Groups withPingdomprobe server IP ranges
- vpc-sg-merge-groups.shMerge two existing VPC Security Groups together into one single group
- vpc-sg-rename-group.shRename an existing VPC Security Group by creating an identical new group
- waf-export-ip-sets.shExport each AWS WAF IP set match condition to a JSON file for backup
- waf-import-ip-set-facebook.shImport list of currentFacebookcrawl server IPs into AWS WAF IP Set - work in progress, currently not possible to execute
- waf-web-acl-pingdom.shManage WAF Web ACL to allow currentPingdomprobe server IPs by creating or updating AWS WAF IP Addresses Set, Rules and Web ACLs
- convert-iplist-cidr-json-array.shConverts an IPv4 iplist to CIDR block notation and JSON array format, sorting and de-duplicating IPs
- install-awscli.shInstall and configure AWS CLI
- install-s3cmd.shInstall and setups3cmdfrom the GitHub Repo
- terraform-redact-iam-secrets.shReplaces AWS IAM Secret Keys and IAM SES SMTP Passwords with "REDACTED" inTerraformstate files
Have a bug or a feature request? Theissue trackeris the preferred channel for bug reports, feature requests and submitting pull requests. If your problem or idea is not addressed yet,please open a new issue.
Shawn Woodford
Code and Documentation Copyright 2012-2018 Shawn Woodford. Code released under theApache License 2.0.