Rails – Custom CloudWatch Application Logs on Elastic Beanstalk
I recently ran into a scenario where I was hosting a Ruby on Rails application on AWS Elastic Beanstalk, and I wanted to view a live stream of the application’s log from within CloudWatch.
For some reason, Elastic Beanstalk doesn’t enable this functionality by default. When you enable CloudWatch streaming on your Elastic Beanstalk environment, it only sends the basic eb-activity.log, passenger.log, and your nginx access/error logs (which aren’t particularly useful).
I did a lot of googling, and cobbled together a solution to the problem by piecing together information from probably around a hundred different pages/forum posts/documentation pages. I have to say, it was rather annoying how difficult it was to figure out something that should have been so simple!
In this scenario, I have two separate environments (one named “production”, and another named “staging”). Each environment has an environment variable called “RAILS_ENV” with its own environment name set on it. The server is configured by Elastic Beanstalk so that it symlinks the Rails application log to “/var/app/support/logs/{environment_name}.log”, and Elastic Beanstalk automatically rotates those logs.
To enable log streaming for the Rails application log, I created a config file in my project at “.ebextensions/configure_logging.config” with these contents:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 |
# Note: For this to work, your instance will need permission to push to cloudwatch. # Create a policy like this, and attach it to the "aws-elasticbeanstalk-ec2-role" role: # { # "Version": "2012-10-17", # "Statement": [ # { # "Sid": "CloudWatchLogsAccess", # "Action": [ # "logs:CreateExportTask", # "logs:CreateLogGroup", # "logs:CreateLogStream", # "logs:DescribeDestinations", # "logs:DescribeExportTasks", # "logs:DescribeLogGroups", # "logs:FilterLogEvents", # "logs:PutDestination", # "logs:PutDestinationPolicy", # "logs:PutLogEvents", # "logs:PutMetricFilter" # ], # "Effect": "Allow", # "Resource": [ # "arn:aws:logs:*:*:log-group:*" # ] # } # ] # } # option_settings: - namespace: aws:elasticbeanstalk:cloudwatch:logs option_name: StreamLogs value: true - namespace: aws:elasticbeanstalk:cloudwatch:logs option_name: DeleteOnTerminate value: false - namespace: aws:elasticbeanstalk:cloudwatch:logs option_name: RetentionInDays value: 7 # Enable custom logging files: "/etc/awslogs/config_disabled/beanstalklogs_production.conf": mode: "000644" owner: root group: root content: | [/var/app/support/logs/production.log] log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/app/support/logs/production.log"]]}` log_stream_name = {instance_id} file = /var/app/support/logs/production.log* "/etc/awslogs/config_disabled/beanstalklogs_staging.conf": mode: "000644" owner: root group: root content: | [/var/app/support/logs/staging.log] log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/app/support/logs/staging.log"]]}` log_stream_name = {instance_id} file = /var/app/support/logs/staging.log* # Files in the location /opt/elasticbeanstalk/addons/logstreaming/hooks/ # are executed after the app has been deployed & the logstreaming addon has been configured "/opt/elasticbeanstalk/addons/logstreaming/hooks/config/99-restart-service.sh": content : | #!/bin/bash service awslogs restart mode : "000755" owner : root group : root # This copies and enables the custom logging config file that matches the current RAILS_ENV environment variable, # and it also restarts the awslogs service, once the config file is in place. commands: 01_enable_logging_config: command: '. /opt/elasticbeanstalk/support/envvars && cp -f "/etc/awslogs/config_disabled/beanstalklogs_$RAILS_ENV.conf" "/etc/awslogs/config/beanstalklogs_$RAILS_ENV.conf"' ignoreErrors: true 02_check_config: command: chkconfig awslogs on 03_restart_awslogs: command: service awslogs restart |
Thanks a lot for the summary of your research (I know how painful this is quite often!). I used it for my Python environment, the path for the environment variables is different (/opt/python/current/env) and also the location of my Django logs (/opt/python/log/django.log), but the rest I could use as is. AWS gave me a warning when I created the policy regarding “logs:PutLogEvents”, but I see the same policy was used in several other descriptions for making specific logs available in CloudWatch