Install MySQL using Ansible, using an idempotent script


This Ansible role will install MySQL on a *nix and may be run multiple times without failure, even though root’s password is changed when running it.
The order is important and here are some tips:

  • The ‘etc.my.cnf’ template does not include user and password entries
  • The ‘.my.cnf’ template only includes user and password entries and is copied to root’s home directory (since my script runs as root), not the deploy’s home directory.
  • Root’s password is set for security reasons
  • Deploy’s only granted access to the application’s databases. I use db1 and db2 as examples here.

Put the below section in your /tasks/main.yml file.

  - name: Install MySQL packages
    apt: pkg={{item}} state=installed
    with_items:
      - bundler
      - mysql-server-core-5.5
      - mysql-client-core-5.5
      - libmysqlclient-dev
      - python-mysqldb
      - mysql-server
      - mysql-client
      - build-essential

- name: Remove the MySQL test database
action: mysql_db db=test state=absent

- name: Create global my.cnf
template: src=etc.my.cnf dest=/etc/mysql/my.cnf

- name: Create databases
mysql_db: name={{item}} state=present collation=utf8_general_ci encoding=utf8
with_items:
- db1
- db2

- name: Add deploy DB user and allow access to news_* databases
mysql_user: name={{user}} password={{password}} host="%" priv=db1.*:ALL/db2.*:ALL,GRANT state=present

- name: Set root password
mysql_user: name=root password={{password}} host="{{item}}" priv=*.*:ALL,GRANT state=present
with_items:
- "{{ansible_hostname}}"
- 127.0.0.1
- ::1
- localhost

- name: Create local my.cnf for root user
template: src=my.cnf dest=/root/.my.cnf owner=root mode=0600

- name: Restart the MySQL service
action: service name=mysql state=restarted enabled=true

From Zero to Deployment: Vagrant, Ansible, Capistrano 3 to deploy your Rails Apps to DigitalOcean automatically (part 0)


tl;dr

Use Cucumber to start us off on our Infrastructure as code journey.

 

Introduction

Part 1 of this blog series demonstrates some Ansible playbooks to create a VM ready for Rails deployment using Vagrant. This is a prequel in the sense that, as a staunch believer in all that’s xDD, I should have started this blog with some Cucumber BDD!
Please forgive my misbehaving and accept my apologies with a few Cucumber scenarios as penance. Hey, it’s never too late to write tests…

The Cucumber Scenarios

As BDD artefacts, they should speak for themselves; write to me if they don’t as it means they were not clear enough!

 

Feature: App deploys to a VM
Background:
Given I have a vm with ip "33.33.33.33"
Scenario: Building the VM
When I provision users on it
Then I can log on to it as the "deploy" user
And I can log on to it as the "root" user
And I can log on to it as the "vagrant" user
Then I remove the VM
Scenario: Adding Linux dependencies
When I provision users on it
When I run the "webserver" ansible playbook
And I log on as "deploy", there is no "ruby"
But "gcc" is present
Then I remove the VM
Scenario: Installing mySQL
When I provision users on it
When I run the "dbserver" ansible playbook
Then I log on as "deploy", then "mysql" is installed
And I can log on as "deploy" to mysql
Then I remove the VM

The Cucumber Steps

Given(/^I have a vm with ip "(.*?)"$/) do |ip|
@ip = ip
output=`vagrant up`
assert $?.success?
end
When(/^I provision users on it$/) do
output=`vagrant provision web`
assert $?.success?
end
Then(/^I can log on to it as the "(.*?)" user$/) do |user|
output=`ssh "#{user}@#{@ip}" exit`
assert $?.success?
end
When(/^I run the "(.*?)" ansible playbook$/) do |playbook|
output=`ansible-playbook devops/"#{playbook}".yml -i devops/webhosts`
assert $?.success?
end
When(/^I log on as "(.*?)", there is no "(.*?)"$/) do |user, program|
@user = user
output = run_remote(user, program)
assert !$?.success?
end
When(/^"(.*?)" is present$/) do |program|
output = run_remote(@user, program)
assert $?.success?
end
Then(/^I log on as "(.*?)", then "(.*?)" is installed$/) do |user, program|
output = run_remote(user, program)
assert $?.success?
end
Then(/^I remove the VM$/) do
output=`vagrant destroy -f`
assert $?.success?
end
Then(/^I can log on as "(.*?)" to mysql$/) do |user|
`ssh "#{user}@#{@ip}" 'echo "show databases;" | mysql -u "#{user}" -praindrop'`
end
def run_remote(user, program)
`ssh "#{user}@#{@ip}" '"#{program}" --version'`
end

From Zero to Deployment: Vagrant, Ansible, Capistrano 3 to deploy your Rails Apps to DigitalOcean automatically (part 1)


update: please refer to the prequel that sets the stage with Cucumber scenarios as a BDD exercise.

tl;dr

In this post, I would like to share that my anxiety about setting up a new server to host an application reminded me why I like being in IT: automation. I attempt to avoid snowflake servers and deploy a Rails application to a VM using idempotent scripts with the help of Ansible and Capistrano.

This entry is a step-by-step guide to get a VM up and running with a Rails app deployed to it. I describe the steps needed to be taken with VagrantAnsible and Capistrano to deploy to a local VM while leaving deployment to DigitalOcean for part two.

the problem

Writing code comes easy to you. As a developer, you develop and test your code with a certain ease and enjoyment . To a certain extent, you may not even think much about the production phase of your project as you may already have an environment set up. However, you might only have a certain idea of what your prod environment looks like as you may have set it up, say, a year or two ago? Maybe your development environment is out-of-sync? Maybe you have to rely on other people (sys-admins) to take care of that “stuff”? That requires A HandOff Ceremony, something we want to avoid on planet DevOps.

In summary, it would be nice to have an automated, testable, repeatable way of provisioning hosts for testing and deployment uses. Obviously, scripts and scripting systems exist for that, and after mucking around with Chef and Puppet, I opted for Ansible.

a solution

In my mind, Ansible is to shell, what CoffeeScript is to JavaScript. I can express what I want to do at a high level (given there’s a module for it) and not worry about the details. In the case of Ansible, I don’t have to worry about idempotence either. So I settled on a way to provision virtual machines (VMs) using Vagrant and Ansible.

While I do not claim to be an expert in any systems herein mentioned, I do declare that “it worked for me”. Please leave comments, tips and tricks if you see any aberrations or more elegant ways of doing things with these tools.

I’d like to credit my friend and colleague Jefferson Girao from ThoughtWorks for having introduced me to Ansible in the first place, and mention that he’s on a similar journey to optimising Rails deployment, with the goal of using Ansible only. I am taking a more conservative approach and will stick with good-old Capistrano for the Rails part.
 

0: punt on windows, linux.

The demo is on a Mac, but feel free to try to adapt it to other platforms.

 

1: Install VirtualBox, Vagrant and Anisble

Here we install stuff, not a lot. 

Get VirtualBox here, or by following the vagrant guide and then install the vagrant gem:

gem install vagrant

 Now let’s install Ansible by the command:

brew install ansible

That assumed you had brew installed. If you don’t have it, I recommend installing it as it makes Mac OS X installations easy. If you prefer not to use brew, do it the hard way

 

2: Prepare to build the machine

Here we create a sub-drectory that will contain our Vagrant file and later on, our Rails app. We’ll keep the Vagrant file near our source code so we can say that we’re compatible with the idea of “Infrastructure As Code” (we’ll get to that in a future chapter).

mkdir app
cd app
vagrant init

This will create an initial Vagrantfile. Replace it with this one: https://gist.github.com/ihassin/7968349

In summary, when run (don’t run it yet, it will fail), this Vagrant script will spin up an Ubuntu Precise 64 instance, make its home on your private network on IP 33.33.33.33 and will invoke the Ansible provisioner to run the user.yml playbook.

 

Intermission

Before we can run the above Vagrantfile, we need to create the ‘user.yml’ file in the devops directory, or elsewhere, if you care to change the  ‘anisble.playbook’  line in Vagrantfile.

I’d like to pause and explain what that user.yml playbook will do so you don’t freak out when you see me moving rsa keys all over the place.

On one hand, I’d like to set up a machine with all needed dependencies. This will require making some apt-get and other calls that will need root rights. That’s fine. We’ll have root (later on, when talking to DigiitalOcean), but for the moment, we’ve the default privileged ‘vagrant’ account for that, which is fine. I would like, however, to run my Rails stuff under the ‘deploy’ account, which would be better off being a regular account. So now we have two accounts, ‘vagrant’ (built-in) and ‘deploy’. I care less about the vagrant user since we’ll throw it away when we provision to DigitalOcean. I do care about the deploy account though:

That ‘deploy’ account will later be used to connect to an external git host, such as bitbucket or github and it will need keys to do so. I will be using that account to log into the instance, so it would be nice if it had my key too. For the scm related issue, I generated a key pair and posted the public portion to bitbucket and github under my account, so they will allow it git operations.

So take a deep breath and step through ‘devops/user.yml’ by reading the task names.

 

3: Playbook: set up a user on the VM

At the app folder root, do this:

mkdir -p devops
 

Copy the following text into ‘user.yml’: https://gist.github.com/ihassin/7968371

The names of the tasks document sufficiently what they do. Note the following however:

1. I send over a known_hosts file that includes bitbucket’s URL.
2. I send a config file that contains bitbucket’s into the deploy user’s .ssh directory so that the first git operation does not hang forever.

OK, if you’re eager to run this playbook, you’ll need the vars.yml file:

Create vars.yml in the same directory as the user.yml file and paste this into it: https://gist.github.com/ihassin/7968378

Replace the text in red with your own values:

1. Running crypt on “secret” with “SaltAndPepper” will create a password token that you place in the password variable. That is the password for the deploy user created on your VM. It’s neat that we don’t have to keep clear text passwords in YAML files.
2. repo holds the git repo you’re application resides in (for a later step).

And you’ll need the templates folder with the following files in it:

Create the templates folder:

mkdir -p templates

Inside it:

1. Copy your public key into a file named ‘your.pub’
2. Copy bitbucket’s RSA signature to a file named known_hosts, thus:

bitbucket.org,207.223.240.181 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAubiN81eDcafrgMeLzaFPsw2kNvEcqTKl/VqLat/MaB33pZy0y3rJZtnqwR2qOOvbwKZYKiEO1O6VqNEBxKvJJelCq0dTXWT5pbO2gDXC6h6QDXCaHo6pOHGPUy+YBaGQRGuSusMEASYiWunYN0vCAI8QaXnWMXNMdFP3jHAJH0eDsoiGnLPBlBp4TNm6rYI74nMzgz3B9IikW4WVK+dc8KZJZWYjAuORU3jc1c/NPskD2ASinf8v3xnfXeukU0sJ5N6m5E8VLjObPEO+mN2t/FZTMZLiFqPWc/ALSqnMnnhwrNi2rbfg/rd/IpL8Le3pSBne8+seeFVBoGqzHM9yXw==

3. Copy your deploy’s private key into a file named deploy_rsa
4. Copy your deploy’s public key into a file named deploy_rsa.pub
5. Copy this to a file named ssh_config:

Hostname bitbucket.com
  IdentityFile /home/deploy/.ssh/deploy_rsa
  StrictHostKeyChecking no

This will make some security people cringe – I’m bypassing checking on bitbucket. Yeah.

6. Create a copy of your sudoers file and add the following line to it:

deploy  ALL=(ALL:ALL) ALL

Then place it in the templates directory as well.

That’s all that’s needed as templates for now. 

You need an inventory file too: Create a file called webhosts and paste the following into it:

[webservers]
33.33.33.33

To run this playbook, enter this at the command prompt:

vagrant up web
vagrant provision web

The first line wakes up vagrant. If it’s the first time you’re trying to access Precise64, this step can take quite a bit of time – Vagrant will download the Precise64 box over your internet connection. Time to brew and drink some coffee.
The second line will be cute to watch, Ansible will light up your screen like a disco, at the end of which you’ll have a VM with Ubuntu installed as well as a login for deploy, using your own ssh key.

You can access this VM via any of the following commands:

1. vagrant ssh
2. ssh vagrant@33.33.33.33
3. ssh deploy@33.33.33.33

If it does not work, it’s either this blog is buggy or it’s a case of PEBKAC. Please check and let me know.

If it works, have some fun with your new free VM, something that would have otherwise cost you a few hundred dollars at your retail PC store.

By the way, adventurous developers can try to provision directly from Ansible:

vagrant up web
ansible-playbook devops/user.yml -i devops/webhosts
 

4: Playbook: get some linux

 

The playbook will give us a real Linux to allow us to move forward with our provisioning (Ruby, Rails)

 

Create a file called webserver.yml and paste this into it: https://gist.github.com/ihassin/7968389 

Play it by issuing the following command:

ansible-playbook devops/webserver.yml -i devops/webhosts


5. Playbook: get some mySQL

The playbook will install mySQL on the provisioned VM. Create a file called dbserver.yml and paste this into it: https://gist.github.com/ihassin/8106956

It will install the needed packages for mySQL and then:

  • Start the service
  • Remove the test database
  • Create a ‘deploy’ user
  • Remove anonymous users from the DB
  • Set up a my.cnf file
  • Change root password
While a great idea to change the root password, this feature renders this playbook non idempotent.


6: Playbook: get some Ruby

The playbook will install the current Ruby 2.0 version. This edition of the blog does not use RVM as it is hell to deal with non-interactive terminals, I am saving the setup of RVM with Ansible for a later post.

Create a file called virtual_ruby.yml and paste this into it: https://gist.github.com/ihassin/7968406

Play it by issuing the following command:

ansible-playbook devops/virtual_ruby.yml -i devops/webhosts
 

7: Playbook: get the project’s ruby and install bundler

The playbook will install the project’s ruby in under the deploy user and install bundler to be used later on.

Create a file called project.yml and paste this into it: https://gist.github.com/ihassin/8004746

Play it by issuing the following command:

ansible-playbook devops/project.yml -i devops/webhosts
 

8: Using Capistrano 3 to deploy the Rails app

This is not a playbook, of course, but a Capistrano 3 recipe.

Install Capistrano 3 following their instructions and replace the deploy.rb file with this one: https://gist.github.com/ihassin/8106917.

Replace the contents of config/deploy/production.rb file with this: https://gist.github.com/ihassin/8107048.

Deploy the app by issuing the following command:

cap production deploy 

9: Have some fun with your new scripts. See the disco colours!

You can repeat these commands to provision, re-provision or just test Ansible’s idempotence:
vagrant up web
vagrant provision web
ansible-playbook devops/user.yml -i devops/webhosts -vvvvv
ansible-playbook devops/webserver.yml -i devops/webhosts -vvvvv
ansible-playbook devops/dbserver.yml -i devops/webhosts -vvvvv
ansible-playbook devops/virtual_ruby.yml -i devops/webhosts -vvvvv
ansible-playbook devops/project.yml -i devops/webhosts -vvvvv
cap production deploy

In the next post, we’ll push the Rails project to a DigitalOcean VM instead of a local one and it run.

Please comment and send feedback about the form and content.

Happy provisioning!

output

Puppet book review


This book is an excellent Puppet book for beginners and professionals alike.

I manage a software team and have read this book cover-to-cover in order to study Puppet for our team’s use on a daily basis.

Despite step-by-step instructions for the initial installation, I needed some tinkering since different OSs have slightly different distributions, but once I had a server and agent running on two different VMs (Ubuntu) – there was an “Aha!” moment when the agent had emacs automatically installed on it! Getting past the initial installation phase allowed me to really enjoy the rest of the book as well as enjoy Puppet itself.

Puppet is not trivial, but the book covers its concepts very clearly and one “gets” it quite early on (especially if you get your hands dirty and follow along the examples).

The book then expertly guides the reader to its “pro” section detailing use of Puppet with configuration management tools such as git and db-based storage.

It then goes on to detail how to use AMQ with Puppet for scaling. I doubt I will use such a robust configuration, but was thrilled to see how flexible and extensible Puppet is by use of load-balancers and integration with Apache/Passenger.

Overall, the book is well written, and I would highly recommend it as a *text book* for Puppet. This is a readable text book on the subject – not a reference manual, although it has countless links to the reference manuals.

I always wanted to learn Puppet, and this book certainly is the one to read if you’re dealing with configuration management whether as a developer or a DevOps person.