Install Aegir Master Slave with Pressflow Platform on CentOS 5 in EC2

Most of the documentation found on this subject are targeted towards Ubuntu. I had quite a time rounding up data for a complete CentOS configuration and ended up writing two scripts for handling this task.

I am currently setting this up in Amazon EC2, complete with auto scaling. I will post more of the advanced configuration when I have that auto scaling portion of this project complete.

The first script is one that can be run on a base CentOS 5.4 ami and will need to be run on both the master and slave instances. Changing this to accomodate CentOS 5.6 or 6.0 is as easy as modifying a few of the paths in the below scripts.

I am using Zend Server Enterprise for my PHP stack, and am therefore pointing to the appropriate php.ini in my scripts as well as the Zend php binaries for path creation.

Upon execution of these scripts on a CentOS system, you will have the following:

- Apache Server
- Zend Server Enterprise Edition
- Varnish
- Aegir Master
- Aegir Slave(s)



Application Server Setup Script

- For this to work properly, you will need to first create an rsa key pair and store it on Amazon S3. You will need to update the last portion of the below script to match your S3 repo.
- You will also need the keys from your Amazon account in order to download these keys automagically to this instance.

#!/bin/sh
# Aegir-Base-Install.sh
#
#
# Created by Shawn LoPresto on 9/1/11.


export WEBHOME=/var/aegir


#Verify that this is a RedHat system
if ! [ -s /etc/redhat-release ] #assuming this is sufficient
then
echo " ERR: This is not a Redhat based distribution. Quitting"
exit1
fi


#Global Master/Slave Setup#
#Aegir#
#CentOS 5.4#


#setup aegir user with new home dir and permissions
echo " INFO: User creation"
useradd --home-dir $WEBHOME aegir
gpasswd -a aegir apache
chmod -R 755 $WEBHOME
! [ -d $WEBHOME ] && mkdir $WEBHOME
chown -R aegir:apache $WEBHOME


# Install Zend-Server
#Add the following repo
cat >> /etc/yum.repos.d/zend.repo <> /etc/profile < /dev/null
if ! [ $? -eq 0 ]
then
echo "alias apachectl='/usr/local/zend/bin/apachectl'" >> $WEBHOME/.bashrc
fi
. $WEBHOME/.bashrc

source $WEBHOME/.bashrc

#enable apache at boot
chkconfig httpd on

#setup aegir user with new home dir and permissions
echo " INFO: User creation"
useradd --home-dir $WEBHOME aegir
gpasswd -a aegir apache
chmod -R 755 $WEBHOME
! [ -d $WEBHOME ] && mkdir $WEBHOME
chown -R aegir:apache $WEBHOME

#grant permission to apachectl for aegir user
grep aegir /etc/sudoers > /dev/null
if ! [ $? -eq 0 ]
then
echo "aegir ALL=NOPASSWD: /usr/sbin/apachectl" >> /etc/sudoers
sed -i 's/^Defaults requiretty/#Defaults requiretty/g' /etc/sudoers
fi

if ! [ -d /etc/httpd/conf.d/aegir.conf ]
then
ln -s $WEBHOME/config/apache.conf /etc/httpd/conf.d/aegir.conf
fi

######################
###POSTFIX CONFIG#####
######################

#Setup postfix relay
cat >> /etc/postfix/main.cf <> /etc/postfix/sasl_passwd < websmtp.myserver.com:587 myemailuser:mypassword
EOF

postmap /etc/postfix/sasl_passwd
/etc/init.d/postfix restart

######################
##Aegir User SSH Key##
######################

su - aegir
mkdir ~/.ssh
cd ~/.ssh
s3cmd --configure
s3cmd get s3://Aegir_Setup/aegir_keys/id_rsa ~/.ssh/id_rsa
s3cmd get s3://Aegir_Setup/aegir_keys/id_rsa.pub ~/.ssh/authorized_keys
chmod 700 /var/aegir/.ssh
chmod 600 /var/aegir/.ssh/*

exit

Varnish



** Do not waste time trying to make Varnish cache your https pages.  HTTPS requests should be sent directly to your web server, where the certificate can be verified.

#!/bin/sh

#  Varnish.sh

#  

#

#  Created by Shawn LoPresto on 9/3/11.

#make sure varnish is installed

yum -y install varnish

# Create varnish config

echo " Creating Varnish Config"

cat >> /etc/varnish/default.vcl <<EOF

backend default {

.host = "127.0.0.1";

.port = "8082";

.connect_timeout = 600s;

.first_byte_timeout = 600s;

.between_bytes_timeout = 600s;

}

sub vcl_recv {

// Remove has_js and Google Analytics __* cookies.

set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js)=[^;]*", "");

// Remove a ";" prefix, if present.

set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");

// Remove empty cookies.

if (req.http.Cookie ~ "^\s*$") {

unset req.http.Cookie;

}

// Skip the Varnish cache for install, update, and cron

if (req.url ~ "install\.php|update\.php|cron\.php") {

return (pass);

}

// Cache all requests by default, overriding the

// standard Varnish behavior.

// if (req.request == "GET" || req.request == "HEAD") {

//   return (lookup);

// }

}

sub vcl_hash {

if (req.http.Cookie) {

set req.hash += req.http.Cookie;

}

}

EOF

#Edit default port in /etc/sysconfig/varnish

sed -i 's/DAEMON_OPTS="-a\ :6081/DAEMON_OPTS="-a\ :80/' /etc/sysconfig/varnish

##Should read

#DAEMON_OPTS="-a :80 \

#-T localhost:6082 \

#-f /etc/varnish/default.vcl \

#-u varnish -g varnish \

##Next switch limits size of the cache

#-s file,/var/lib/varnish/varnish_storage.bin,1G"

##Change Apache default listening ports

sed -i 's/Listen\ 80/Listen\ 8082/' /etc/httpd/conf/httpd.conf

#Listen 8082

chkconfig varnish on

chkconfig httpd on

/etc/init.d/httpd restart

/etc/init.d/varnish restart

#####Aegir Specific#######

##Each Aegir server will need to be configured with a default port of 8082 for http (within the UI configuration).##

##########################

###Remainder of configuration is done on Aegir Master and requires making some additions to the settings.php#########

Aegir Master

The below script will require some user input for the initial Aegir configuration, but will walk you through the complete install.

#!/bin/sh

#  Aegir Master Install

#  

#

#  Created by Shawn LoPresto on 9/1/11.

######################

#####MASTER ONLY######

######################

#Setup Profile Variables

echo " INFO: Switching to aegir user"

su -c -l aegir '

export DRUSH_VERSION=7.x-4.5

export WEBHOME=/var/aegir

export HOME=$WEBHOME

export drush="$HOME/bin/drush/drush" 

export DRUPAL_VER=6.x

export AEGIR_DB_HOST=aegir-db.mydomain.com ##Update with your DB host FQDN

export AEGIR_HOST=aegir-master.mydomain.com ##Update with the FQDN for the Aegir master

export AEGIR_DB_USER=aegirmaster ###update with privileged MySQL admin account

mkdir ~/bin/

cd $HOME/bin

wget http://ftp.drupal.org/files/projects/drush-$DRUSH_VERSION.tar.gz 

gunzip -c drush-$DRUSH_VERSION.tar.gz | tar -xf -

rm -rf drush-$DRUSH_VERSION.tar.gz

mkdir $WEBHOME/make-files

#Create Pressflow Make file

cat >> $WEBHOME/make-files/pressflow.make <<EOF

; DRUPAL VERSION

core = 6.x

; CORE MODULES

projects[pressflow][type] = "core"

projects[pressflow][download][type] = "get"

projects[pressflow][download][url] = "http://files.pressflow.org/pressflow-6-current.tar.gz"

; DEVELOPMENT

projects[devel][subdir] = "contrib"

projects[backup_migrate][subdir] = "contrib"

; PERFORMANCE

projects[memcache][subdir] = "contrib"

projects[varnish][subdir] = "contrib"

; UTILITY

projects[apachesolr][subdir] = "contrib"

projects[libraries][subdir] = "contrib"

projects[jquery_ui][subdir] = "contrib"

projects[modalframe][subdir] = "contrib"

; UI

projects[admin][subdir] = "contrib"

projects[admin][version] = "2.0-beta3"

projects[vertical_tabs][subdir] = "contrib"

projects[wysiwyg][subdir] = "contrib"

; KEY MODULES

projects[cck][subdir] = "contrib"

projects[views][subdir] = "contrib"

projects[token][subdir] = "contrib"

projects[pathauto][subdir] = "contrib"

; FILE/IMAGE HANDLING

projects[filefield][subdir] = "contrib"

projects[imagefield][subdir] = "contrib"

projects[imagecache][subdir] = "contrib"

projects[imageapi][subdir] = "contrib"

projects[transliteration][subdir] = "contrib"

; THEMES

projects[tao][location] = http://code.developmentseed.org/fserver

projects[rubik][location] = http://code.developmentseed.org/fserver

; OTHER FILES

; ApacheSolr

libraries[SolrPhpClient][download][type] = "get"

libraries[SolrPhpClient][download][url] = "http://solr-php-client.googlecode.com/files/SolrPhpClient.r22.2009-11-09.tgz"

libraries[SolrPhpClient][directory_name] = "SolrPhpClient"

libraries[SolrPhpClient][destination] = "modules/contrib/apachesolr"

; jQuery UI

libraries[jquery_ui][download][type] = "get"

libraries[jquery_ui][download][url] = "http://jquery-ui.googlecode.com/files/jquery.ui-1.6.zip"

libraries[jquery_ui][directory_name] = "jquery.ui"

libraries[jquery_ui][destination] = "modules/contrib/jquery_ui"

; TinyMCE 

libraries[tinymce][download][type] = "get"

libraries[tinymce][download][url] = "http://downloads.sourceforge.net/project/tinymce/TinyMCE/3.2.7/tinymce_3_2_7.zip"

libraries[tinymce][directory_name] = "tinymce"

EOF

#set up the alias for drush

grep drush $HOME/.bashrc > /dev/null

if ! [ $? -eq 0 ]

then

echo "alias drush='/var/aegir/bin/drush/drush'" >> $HOME/.bashrc

fi

. $HOME/.bashrc

source $HOME/.bashrc

cd $HOME

#get the latest version of drush before starting 

$drush self-update

echo " INFO: Installing Drupal $DRUPAL_VER"

$drush --destination=$HOME dl drupal-$DRUPAL_VER

echo " INFO: Installing drupal module : provision"

$drush dl --destination=$HOME/.drush provision-6.x

echo  " INFO: Running hostmaster install"

$drush --aegir_db_host=$AEGIR_DB_HOST --aegir_db_user=$AEGIR_DB_USER hostmaster-install 

#install update module and run updates

echo " INFO: Updating Drupal installation"

##enable the update module

$drush -y -r /var/aegir/hostmaster-6.x-1.3 -l http://$AEGIR_HOST enable update

##do the update

$drush -y -r /var/aegir/hostmaster-6.x-1.3 -l http://$AEGIR_HOST up

#update drush make

echo " INFO: Updating drush make:

cd /var/aegir/.drush

wget http://ftp.drupal.org/files/projects/drush_make-6.x-2.3.tar.gz

tar xzvf drush_make-6.x-2.3.tar.gz

rm -f drush_make-6.x-2.3.tar.gz

echo " INFO: Installation complete. Please visit http://$AEGIR_HOST to verify"

'

That is it for the install. Now go to your browser and hit the newly created URL to begin configuring Aegir.

I will create another post with the additional changes needed to actually utilize Varnish for your Drupal sites.

Webistrano on CentOS

CentOS installs an old version of Ruby and I found that Webistrano was failing due to an issue reading from the gem path. To resolve, I just removed the yum installed ruby version and compiled 1.9.2 from source. Below are the basic instructions.

Additionally, I found myself having to search for information on using git versus subversion, so I will notate a few items that are worth note on that as well.

Setup Ruby Environment

1. Run the following

sudo yum erase ruby ruby-libs ruby-mode ruby-rdoc ruby-irb ruby-ri ruby-docs

cd /tmp

sudo yum install readline-devel zlib zlib-devel

sudo yum groupinstall 'Development Tools'
$ sudo yum install readline-devel
$ cd /usr/local/src
$ wget ftp://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.1-p376.tar.gz
$ tar xzvf ruby-1.9.1-p376.tar.gz
$ cd ruby-1.9.1-p376
$ ./configure && make
$ sudo make install

Setup Webistrano

Capistrano is an open source, command line, deployment tool that provides all of these features. It's written in Ruby. You leverage a variety of built in "recipes" (Capistrano's term for a deployment script) that execute certain procedures to deploy an app. Out-of-the-box it's ideally built to deploy a Rails app. However, after some minor tweaks it can deploy most anything and do it well. It can restart servers, update symlinks, change permissions - pretty much anything. It assumes you access your POSIX compliant server via SSH via the same password (or have ssh keys setup).

Webistrano is an open source web front-end for Capistrano. It's a convenience layer that abstracts the command line away and provides an interface to perform the same tasks. This interface shows history as well as providing a convenient GUI for creating new deployment projects, stages, and recipes. Highly recommended.

Let's get down to business. This post makes a few assumptions about things you've already installed and used previously.

• Ruby 1.8.5+
• RubyGems
• MySQL 5.0+
• Apache 2+
• Subversion and a repository containing the "production" branch of your app.

Installing Capistrano

Well, this is an easy one (you probably want to do this as root):

gem install capistrano

Installing Webistrano

Also fairly easy, with a little splash of configuration.

# wget http://labs.peritor.com/webistrano/attachment/wiki/Download/webistrano-1.4.zip
# unzip webistrano-1.4.zip
# mv webistrano-1.4 /path/to/where/you/want/webistrano

Setup the database tables and create a new webistrano user (obviously be conscious of your security preferences for access to your database in the host and password portions):

# mysql
mysql> CREATE DATABASE `webistrano`;
mysql> CREATE USER 'webistrano'@'localhost' IDENTIFIED BY 'password';
mysql> GRANT ALL PRIVILEGES ON `webistrano`.* TO 'webistrano'@'localhost' WITH GRANT OPTION;

Now, in the directory where you placed webistrano you're going to want to copy config/database.yml.sampleto config/database.yml. Edit this file, in the production area, to match your database settings. By default the file expects a socket to connect, you can chase this by specifying host: and port:. (Keep in mind Webistrano is simply a Rails app).

You should now be able to have Rails migrate the new database you created. In the webistrano directory:

# RAILS_ENV=production rake db:migrate

Finally, copy config/webistrano_config.rb.sample to config/webistrano_config.rb and edit according to your preferred mail settings.

We can now test to see if webistrano is working properly by serving it via mongrel:

# ruby script/server -d -e production -p 3000

This starts a single mongrel daemon, using the production environment, listening on port 3000. You should now be able to hit http://127.0.0.1:3000/ and get the Webistrano login prompt. If this is working, kill that mongrel instance.

For longer term serving I decided to go with Phusion Passenger (essentially mod_rails for Apache). It's a nearly zero configuration solution for serving a rails app and will feel at home to anyone with experience serving PHP apps via Apache and mod_php.

Installing Phusion Passenger

Again, as root:

# gem install passenger
# passenger-install-apache2-module

The second command will invoke an installer which compiled Passenger and provides instructions on integrating it into your Apache config. Essentially, edit your httpd.conf as follows (these were specific to my install, make sure to use the ones provide by the installer for you):

LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-2.2.9/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-2.2.9
PassengerRuby /usr/bin/ruby

Now you can simply add VirtualHost entries to your httpd.conf for any of your Rails apps. Let's add one for Webistrano:

<VirtualHost *:80>
ServerName webistrano.mydomain.com
DocumentRoot /path/to/webistrano/public
</VirtualHost>

Yes, Passenger makes it that simple. Add configuration directives as needed for your environment.

Now Webistrano should be serving from the VirtualHost you specified, seamlessly, via Passenger.

Deploying A Non-Rails App

Now the fun stuff.

Capistrano breaks things down into projects, stages, and recipes. Each app you want managed by capistrano should be it's own project. Each project should have a stage for at least production and optionally staging and development.

Hosts are added globally and form the targets of a deploy for any given project. Hosts can include web, app, and database servers.

Deployments in Capistrano are done to a child directory under "releases" named via the date and time of the deployment. By default 5 releases are kept and available to rollback to. Upon successful deployment a symlink (default is called "current" and can be modified via the current_path configuration variable) is updated to that release directory. It is this symlink that should be targeted by your webserver (your DocumentRoot in Apache).

Capistrano also creates a "shared" directory that is symlinked to in each release useful for storing logs and other data that should be maintained through each deployment.

For non-rails apps you'll use the "Pure File" project type when creating your new project. Upon project creation you can add configuration variables specific to your project. I recommend using :export instead of:checkout for deploy_via for production subversion deployments as this doesn't expose .svn directories. Use an SSH user that has enough permissions to create directories where your deploy will occur or, specifyuse_sudo to true and create a new configuration variable admin_runner and set it to the same user asrunner.

Add a stage to your new project for "production". In the "Manage Hosts" page add a new host for each of your application servers. Then add each host as a target of your "production" stage of your project.

At this point you should be able to execute the "Setup" task for your "production" stage. This is a one time task that simply creates the directories.

Assuming this went successfully, try doing a "Deploy" and see if that finishes without error. You might have to play around with permissions and other minor issues - post a comment if you have any specific questions.

For my PHP framework there are a couple specific tasks I wanted to run in addition to the default Capistrano tasks. You do this by creating custom recipes in the "Manage Recipes" page in Webistrano. Recipes are simply procedures written in ruby. Here's what my recipe looks like:

01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19

namespace :deploy do     task :setup, :except => { :no_release => true } do         dirs = [deploy_to, releases_path, shared_path]         dirs += shared_children.map { |d| File.join(shared_path, d) }         run "#{try_sudo} mkdir -p #{dirs.join(' ')} && #{try_sudo} chmod g+w #{dirs.join(' ')}"         run "chmod 777 #{shared_path}/log"     end     task :finalize_update, :except => { :no_release => true } do         run "mkdir -p #{latest_release}/app/tmp"         run "chmod -R 777 #{latest_release}/app/tmp"         run "rm -rf #{latest_release}/app/logs"         run "ln -s #{shared_path}/log #{latest_release}/app/logs"         run "cp #{latest_release}/public_html/.htaccess-production #{latest_release}/public_html/.htaccess"         run "cp #{latest_release}/app/config/config-production.php #{latest_release}/app/config/config.php"         run "cp #{latest_release}/app/config/db-default.php #{latest_release}/app/config/db.php"         run "cp #{latest_release}/app/config/memcache-default.php #{latest_release}/app/config/memcache.php"     end end

If you're not familiar with Ruby - what this code is essentially doing is overwriting two tasks in the :deploy namespace with my custom code.

The first, :setup, simply duplicates the base :setup functionality discussed above (creating the releases and shared directories) and chmods the shared log directory to be writable.

The second, :finalize_update, performs a variety of configuration tasks for a PHP app built with my framework. Also, you'll notice that I'm removing my app's logs directory and symlinking to the shared log directory. This way all releases will log to the same directory, consistently.

In my case all of these procedures are command line instructions. Alternatively, you can do a variety of things leveraging the full breadth of the Ruby language and any gem you'd like to introduce. Things such as accessing your CDN API to clear image, JS, or CSS caching, etc.

Deploying Django Apps

First off it's worth noting that I serve my Django apps via mod_wsgi. To make the deployment process easier here's what my app.wsgi script looks like:

1 2 3 4 5 6 7 8 9

import os import sys appdir = os.path.normpath(os.path.join(os.path.realpath(os.path.dirname(__file__)), '..')) sys.path.insert(0, appdir) os.environ['DJANGO_SETTINGS_MODULE'] = 'settings' os.environ['PYTHON_EGG_CACHE'] = os.path.join(appdir, '.python-eggs') import django.core.handlers.wsgi application = django.core.handlers.wsgi.WSGIHandler()

This code allows us to avoid having to hardcode paths in the wsgi script (and thus avoid having to change them when we deploy). It assumes the following directory structure:

.python-eggs (egg cache)
apps (apps path is added to python system path in settings.py)
public (where your .wsgi script resides)
site_media
templates
settings.py
settings-production.py (used for deploy)
urls.py
...

If you follow this convention, the following Capistrano recipe works great:

01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16

namespace :deploy do     task :setup, :except => { :no_release => true } do         dirs = [deploy_to, releases_path, shared_path]         dirs += shared_children.map { |d| File.join(shared_path, d) }         run "#{try_sudo} mkdir -p #{dirs.join(' ')} && #{try_sudo} chmod g+w #{dirs.join(' ')}"         run "chmod 777 #{shared_path}/log"     end     task :finalize_update, :except => { :no_release => true } do         run "rm -rf #{latest_release}/logs"         run "ln -s #{shared_path}/log #{latest_release}/logs"         run "cp #{latest_release}/settings-production.py #{latest_release}/settings.py"         run "mkdir -p #{latest_release}/.python-eggs"         run "chmod 777 #{latest_release}/.python-eggs"     end end

Fin