Chef Configuration with Multiple Websites & Multiple Apps

Chef makes it easy to provision servers and deploy apps. Jason Grimes wrote an excellent three part tutorial including Vagrant and EC2 details.

This post assumes you have a successfully running project as instructed by Jason. Building upon Jason’s excellent work, let’s look at configuring multiple websites and apps using Chef.

As always, your feedback is valued so please leave additional tips and suggestions in the comments.

Chef Directory Structure for Multiple Apps

1. Within your chef-repo directory (~/path/to/chef-repo/), create a new sub-directory called projects.
   Projects will contain Chef configurations for multiple projects.
2. Copy the contents of your chef-repo directory except for the projects directory and move them to ~/path/to/chef-repo/projects/original_project/
3. Access the new location and confirm your original nodes are listed.
   cd ~/path/to/chef-repo/projects/original_project
knife node list
   If your original nodes are listed, you are ready to proceed.

Add a Second Project to Chef

Create the directory structure to support a new project. We will call the new project new_project.

mkdir ~/path/to/chef-repo/projects/new_project
mkdir ~/path/to/chef-repo/projects/new_project/.chef

Create directories normally found in a chef configuration (certificates, config, cookbooks, data_bags, environments, roles) although these may vary based on your configuration.

Create a New Organization at Opscode

Web developers are often working on multiple projects for multiple clients. Often web developers are working on side projects or start ups as well. It is best to create an Opscode account for each client and/or business.

Keeping client & project files separate is also why we chose to use a directory based approach as opposed to simply adding nodes and roles to a single configuration. Perhaps the node approach can be examined in a future post.

1. Create a new account at Opscode Hosted Chef.
2. While logged in, download your Chef Organization Validation Key and your knife config file (knife.rb).
3. Download your User Private Key. This file is <username>.pem where <username> is replaced with your Chef username. If you don’t have this file for your NEW account, go to Opscode and click “Reset User Key”.
4. Move the 3 downloaded files (knife.rb, <organization_name>-validation.pem and <username>.pem) to ~/path/to/chef-repo/projects/new_project/.chef
5. cd ~/path/to/chef-repo/projects/new_project
knife client list
   and confirm your new project is listed. You should see <organization-name&rt;-validator.

What Happens With Cookbooks in a Shared Chef Configuration?

The easiest way to share cookbooks between web projects is to use a tool such as Berkshelf or Librarian.

There are times when sharing is not possible even though cookbooks are required by 2 apps. In an non-DRY approach, copy the applicable cookbooks from ~/chef-repo/projects/original_project/cookbooks to ~/chef-repo/projects/new_project/cookbooks. NOTE: You won’t want to copy the original_project.rb cookbook and other original_project specific cookbooks.

In the ~/chef-repo/projects/new_project/ directory, upload the cookbooks.
knife cookbook upload --all

Similarly, continue to setup your environments, roles, users & databag secrets. If you need assistance, check out Jason Grimes’s Excellent tutorial.

Pay special attention to your data bag items. Each project should have unique passwords.

VagrantFile for 2 Websites – Multiple Virtual Machines

Vagrant supports Multi-VM Environments. Replace your existing VagrantFile with the configuration details for each project-specific VM.


app_servers = {
 :original_project => {
  :appname => "original_app",
  :orgname => "org1",
  :ipaddress => "10.0.0.20",
  :fwdport => "8080",
  :hostname => "originalapp-vm",
  :vmname => "ORIGINAL APP VM"
 },
 :new_project => {
  :appname => "new_app",
  :orgname => "org2",
  :ipaddress => "10.0.0.21",
  :fwdport => "8081",
  :hostname => "newapp-vm",
  :vmname => "NEW APP VM"
 }
}


Be sure to edit the values with your project specific values.

Append the code below to the end of your new VagrantFile.


Vagrant::Config.run do | config |

 app_servers.each do | app_server_name, app |
  config.vm.define app_server_name do |app_config|
   app_config.vm.box = "precise64"
   app_config.vm.forward_port 80, app[:fwdport].to_i
   app_config.vm.customize [
    "modifyvm", :id,
    "--name", "#{app[:vmname]}",
    "--memory", "2048"
   ]

   # Assign this VM to a host-only network IP
   app_config.vm.network :hostonly, app[:ipaddress]
   app_config.vm.host_name = "#{app[:hostname]}"

   node = ENV['NODE']
   node ||= "vagrant-#{app[:orgname]}"

   app_config.vm.share_folder("v-root", "/home/vagrant/apps", ".", :nfs => false)
   # remove nfs inside the parentheses for non-mac / non-linux hosts

   app_config.vm.provision :chef_client do |chef|
    chef.chef_server_url = "https://api.opscode.com/organizations/#{app[:orgname]}"
    chef.validation_key_path = "#{ENV['HOME']}/chef-repo/projects/#{app[:appname]}/.chef/#{app[:orgname]}-validator.pem"
    chef.validation_client_name = "#{app[:orgname]}-validator"
    chef.encrypted_data_bag_secret_key_path = "#{ENV['HOME']}/chef-repo/projects/#{app[:appname]}/.chef/encrypted_data_bag_secret"
    chef.node_name = "#{node}"
    chef.provisioning_path = "/etc/chef"
    chef.log_level = :debug
    chef.environment = "dev"
    chef.add_role("base")
    chef.add_role("db_master")
    chef.add_role("webserver")
   end
  end
 end
end

Chef Error with Mysql Installation

A quick sidenote: if you are receiving an error during the MySql portion of provisioning, check your password length. On my Vagrant installs, I ran into issues with passwords longer than 12 characters.

If you know why this error is occurring, please share in the comments below.

Update Hosts File

Use the Hosts file to add a human-friendly hostname for your project. Obtain the IP address for each project from your VagrantFile.

Add the following lines to your host file where you substitute the “x”s for the appropriate IP address.

127.0.0.1 localhost
xx.x.x.xx original_app.dev
xx.x.x.xy new_app.dev


Your new app should now be accessible at http://new_app.dev. Check for the Apache “It Worked” message. Same for http://original_app.dev. Obviously you will only be able to access these websites from your workstation.

Chef is Flexible Depending on Your Project

This approach separates each website / app into unique project directories. This approach is best for multiple clients and/or unrelated websites / apps.

Another, perhaps more efficient but not always applicable, method is to separate the websites into multiple nodes. The multiple nodes approach is best when you have related websites / apps such as an e-commerce store front, a blog and a brochureware site all on sub-domains.

How Can This Chef Configuration Be Improved?

Share with us in the comments below. Help all of us get better!