Returns Driven Development

The premise of all the “DD” acronyms is to minimise technical debt in one way or another and otherwise drive us to being lean.

The motivation for this article is “writing the minimum amount of code” in the spirit of Agile in general and TDD/BDD specifically. As someone who has developed code for more than a quarter of a century, I have learned that anything I write as code will be used against me as long as the software is in use. I don’t want to write more code than I need to in order to justify my reward. In this case, my reward is to have the RDD monitor set off an alert that serves as feedback to knowledgable people to make decisions about the product such that I will continue to be rewarded.
So, what is RDD?

TDD instructs us to write as little code as we can to assure a passing set of tests.
BDD instructs us to write as little code as we can to assure a valuable set of features.
I’d like to extend these guides to a methodology that instructs us to write as little code as we can to assure a specific level of business returns (i.e. ROI). I’ll call it RDD for fun. Returns Driven Development (thanks to my fellow ThoughtWorker Kyle for coming up with the name!).

In most cases, there is an underlying business case for creating or modifying software. Of those, some are justified by a business plan that shows how much more money the business would make if only the requested features were implemented. Of those, only a few are borne of a real market analysis. In the rest of the cases, the primary motivation is the product manager’s intuition that it would be nice to have these new features.

I wanted a way for the product owner to convey her ideas about the modifications, without regard to her motivation. RDD is a way to describe software feature requests without having to make up financial data to justify the requests. It’s also a way to validate the intuition of the product team.

Some examples:

Our customer acquisition rate will increase if we made signup easier.

Our salespeople will sell more licenses if they could demonstrate the software at trade shows with preloaded customer accounts.

Our sales will increase if we exposed our B2B services to the public Internet.

All these sound valid points for a product manager to present as justification in embarking on a technical investment in creating or modifying existing software.

The only change I would make to the above examples is to add a quantity. Acquisition rate will increase by 30%; we’d have 25% more sales etc.

This is the starting point of RDD: in order to assure the growth of this business, we need to increase sales by X%.

Now that we have that statement, it will be scrutinised by the company’s board and a decision will be made regarding its implementation. If action were to be taken, RDD is now charged by proving those statements.

RDD assures statement validation by providing business feedback to the product owner that the course charted is indeed driving towards that stated goal. The sooner and more precise the feedback, better decisions will be made to adjust the statement or the course of action.

RDD proposes to set up the monitors first and develop minimal software to satisfy them. The monitors will provide a baseline of the current situation and, prior to development, will indicate whether the premise was indeed factual and worthwhile.

As an example, an RDD monitor will state:

Generate an alert if the number of the daily sales of licenses is below 30 or is in decline more than 5% week over week.

Generate an alert if the number of B2B API calls originates from more than 10% of our customers.

Primarily, the alerts will indicate movement in their business domains and will set a baseline of alert frequency. They can also serve as indicators that something is not functioning on a technical level, but that’s secondary as other IT alerts exist for that purpose.

Now comes the fun, DD, part:

The monitor’s premise is dependent on much more than meets the eye at first reading.
The data for the alarm may not exist. The transaction table may or many not exist, depending on the state of the product that the alarm is set to monitor. If this is a new product, a transaction data source may actually have to be created just for the monitor. That alone is an excellent improvement to the organisation.
Following that scenario, a data table without data is not much use; enter BDD. Enter TDD. Soon, you may have developed the app from scratch. A whole system may have been created as the result of a Returns statement made by the product manager, yet we invested the minimal amount of development needed to satisfy the monitor. Feedback is guaranteed as the monitor was implemented up front even before the software was.
RDD is also effective when extending existing software as well, while assuring that the minimum code was written to satisfy the monitor’s goal.
The claim that a simple sign up form will boost customer acquisitions will soon be proven right or wrong. The monitor will raise an alert if signups have increased week-over-week. If it does not, we may need another monitor that observes another aspect of the product that questions its value to the users.

So, the next time you are involved in a product’s inception or new feature, start with business monitors!

Try asking for a business returns monitor from the operations group. At first, their mouths will open and close but without words coming out. Soon after, they will realise that it is nothing but another monitor. You then employ DDD/BDD/TDD To develop it and the system that feeds it information. You then sit back and wait for the product owners to request new monitors or features as they attempt to regulate the reported data to prove their original claims either right or wrong, or a little of both.

Drobo will not mount in Finder

I have a Drobo with 4x2GB disks installed to hold all my stuff. ALL my stuff.

The other day, I connected to Drobo in order to execute a TimeMachine backup. Drobo came up green yet Finder did not mount it. I heard rumbling noises and knew that something was messing with it. I just was not sure whether it was Finder or the Drobo’s firmware itself.

DiskUtils could not access the disk. Big panic. Called Drobo support and was surprised by their shallow “do a repair” reply.

I stumbled upon the following procedure by poking around and taking chances. I was “this far” from writing off 3TB of photos and projects, so I felt I had nothing to lose. Please note that in no way can I guarantee that this will help you, but it worked for me:

1. The rumbling noises that I heard when I connected the Drobo to the Mac were due to Finder trying to mount the disk but with running FSCK prior to doing so. The rumbling was probably due to a screwed up MBS.

Disk Utils was hampered as long as Finder was messing with Drobo, so I killed it:

2. Looked for any disk-related tasks by ‘ps ax’ in terminal.

3. Killed those tasks using ‘sudo kill -9′

4. That left the Drobo in the exact state I wanted it: Unmounted and left alone

5. Ran DiskUtils and chose to Repair the disk. I got a reasonable progress bar (as opposed the infinite one when I ran DiskUtils while Finder had stuff going on). It took over 12 hours to repair. Be patient!

6. Used the Mount icon in DiskUtils to mount Drobo.

7. All good, happy.

So now what? A backup of a backup on S3? Is there no end to this cycle?

Command & Control Management – The Party Killer

I was asked why some developers don’t have parties or late night coding sessions. I do not think it was meant literally, since organising a party is a trivial activity and does not warrant a discussion.

I understood the question to be why wouldn’t they be as involved with their projects as others might be elsewhere. After all, celebrating success or staying late to meet a deadline is the result of being engaged, involved and caring about the projects one is working on. Consequently, not celebrating success may be a symptom of not being engaged, nor involved nor caring about those projects.

I propose that they do not have parties because their management style is “Command and Control”. They have a hierarchy and teams are told what to do. Teams have leaders that enforce C&C upon their members. There is a separation of duties and expectations across teams and the relationship between all teams is defined by their relative roles in the project’s lifecycle. A “food chain” emerges that is defined by “suppliers” and “customers”. A team’s role in the project is either to serve another team’s needs or is entitled to another team’s services.
This vision is well suited to the C&C management style, which clearly defines the roles and behaviour for the participating teams. The teams, however, rarely have a say in defining the goals of the product nor a say in the overall strategy of achieving those goals. Value is skewed and variances from it are not tolerated thus creating more problems for future business and technological change.

Handoffs between teams are mandated, rarely with any multilateral conversations, and the handoff of requirements is basically synonymous with “Shut up, this is what we want, what’s the estimate?”

C&C stifles independent thought and is inconsistent with excellent programming, which demands intelligence and creativity.

No one would think of ordering the sax player to play certain notes in a Jazz session. Developing products is closer in structure and dynamics to Jazz sessions rather than to orchestrated classical music.

The C&C approach distances people from the product so much that moving the project to the next station in the workflow is met with a sigh of relief …not with the joy we all want to feel when creating something of value. No one in the C&C production chain feels as though they own the product. No one thinks to throw a party because they do not have shared values to celebrate.

C&C demotivates creativity, teamwork, and the drive it takes to work long hours or over the weekend. Under C&C, people wait to be told what to do while the list of backlog tasks becomes a black hole of client frustration.

Another reason is that developers are often described as “resources”. Just by that outlook, we’ll fail. We are human beings with names, abilities and skills. Any project plan will fail if we do not address our people on a personal level, taking their strengths and weaknesses into account. When that is not a part of management practices, teams are manage like conference rooms – available or busy.
A happy, engaged, concentrated developer will produce quality software, all else being equal. A resource is acquired, used, then relinquished. Have we ever seen conference rooms get together for a party? Resources don’t have parties, humans do.

Product Management, Marketing, PMOs, Developers and QA should all meet and brainstorm throughout the project’s lifecycle. Without collaboration, there is little creativity and even less ownership. No one is motivated to pitch in to make better quality products. Teams will not feel ownership, have parties or mark the occasion of software releases if they aren’t invited to actively participate at the beginning. Because of separation, developers are never present at business meetings, and don’t have the opportunity to fully understand the client’s needs. Instead, they are ordered to write code (quickly).

I doubt that we will ever see self-organising teams or a true sense of ownership as long as C&C and segregation is instilled in management’s culture. We need ownership and team collaboration. Alas, teams find themselves against each other in a game of politics. There is scarcely any collaboration between them, only downstream C&C. In some companies, PM does not stand for Project Manager, but for Political Manager.

As a consequence, creativity and communication have been replaced by stale, boring, incomplete and sub-standard power-point presentations. They present lies: the business unit grossly exaggerates the product’s value and the developers exaggerate the cost of its implementation. Everyone is scared.

Another symptom of C&C is that we do not share goals. Since we are broken into segregated teams, each tends to develop their own set of goals and priorities.

Those goals are then presented (barked) as imperatives to the other teams. Product Management’s goal is to have something available in the market. The developers’ goal is to have something adhering to current best practices and quality standards. Project Management’s goal is to satisfy Product Management and so forth. The lack of shared goals divides the teams, creates the need to run interference, and justifies still greater C&C.

I don’t know what value the other units extract from such fiascos, but developers do manage to extract experience and technical problem solving skills, not the least of which are getting around SysAdmin and Network Engineering obstacles that diminish their productivity.
So theirs is not a total loss, but how could anyone expect any team to rejoice at the release of such products? The feeling is of regret, if anything, at having been forced to write rushed code for a perceived meaningless business case. No parties there.

On the other hand, in startup restaurants cooks also take out the garbage, and the owner also sweeps the floors. In software development, developers take QA’s testability needs into account when writing code without being asked to. The business analyst works with the product manager in optimising the process before feature requests are discussed.

I propose that we form project-teams from all disciplines that would report to the project itself, to give all the participants a sense of ownership. I’ll argue that this would lead to self organising teams and that it would also lead to parties, that there would be no more such questions and that I would not have needed to spend so much time writing this apology for bad management practices.

Ugh, what a dismal end to this article. So on a happier note:

Come to planet agile and enjoy the party!

Quickie – ssh dynamic port forwarding to avoid unsecured public networks

You’re in an airport, and there’s free wifi (you’re obviously not in the US…). You want to connect but are worried about someone sniffing your connection. You’re rich, so you have a remote box with ssh access to it.

The solution is to ssh into your remote box and forward all your traffic to it. It will be your secure proxy for your session.

Easy to do:

Open a terminal and issue:

ssh -D 8888 remote-host

This will start port dynamic port forwarding to the remote-host machine.

Then, set up a proxy on your local machine to proxy all localhost traffic to port 8888.

On the Mac, it looks like this:

Presto, as long as the terminal is open with the ssh -D command running, all your internet communications will pass through to the remote-host using the secure socket connection.

How to reconnect to a database when its connection was lost

One of my projects has a long-running task that constantly needs information from the database. I needed a mechanism to assure that the task will automatically reconnect to the database if and when that connection was broken.

I came up with this scheme using a trick with rescue blocks (code abbreviated for clarity) in this gist.

def my_task

    while(true) do

      begin

        database_access_here

      rescue Exception => ex

        begin

          ActiveRecord::Base.connection.reconnect!

        rescue

          sleep 10

          retry # will retry the reconnect

        else

          retry # will retry the database_access_here call

        end

      end

    end

  end

Here’s a line-by-line explanation:

Line 4: This is where your application’s database access logic would be.

Line 5: Catch a database access exception here

Here is where it gets interesting:

Line 7: Open a new block and retry the connection.

Line 10: This retry will retry the reconnect method and will loop as long as the database connection is still down.

Line 11: The else clause will execute if _no_ exception happened in line 10, and will retry the original database call in line 4.

In my case and example, I am not counting retries because I don’t care that I’ve failed – I must continue to retry. You may want to use “retry if retries < 3″ as a break mechanism.

I have also removed some mailer code that notifies me when the reconnect fails so I can (manually) see what happened to the connection. The moment the connection is re-established, life goes on as normal within the infinite while loop.

Weekend warrior – MacRuby and rSpec, Mac OS X Lion, Xcode V4.3.2

Inspired by the recent buzz over RubyMotion, of which I am a proud licensee, I wanted to play a little with MacRuby just to get into the swing of things.

After deciding that doing so was more worthwhile than to mow the lawn, I set out to see what it took to start a project in MacRuby with rSpec support as a basis to start work.

MacRuby’s article got me started, but did not work because the test target could not find the framework that I wanted tested. I don’t know why, since I (sort of) follwed the instructions there. I say “sort of” since the article shows screen-shots of an older Xcode, and even though I thought I set things correctly in my version (Xcode V4.3.2), it still would not build. Also, I am on Mac OS X Lion and that may have had something to do with it.

After realising that if I did not continue trying, a certain member of the household would make me mow that lawn, Google found another article here by Steve Madsen.

It too looked promising, but again, needed tweaking to get working in my environment. It’s thanks to Steve’s post that I managed to get it working.

Here were my steps:
a. Create a new project in Xcode (or use an existing one that you want to rSpec)
b. Install MacRuby
c. Follow Steve Madsen’s instructions

At that stage it still did not work for me, but that was because of a misunderstaning that was clarified quickly enough:

Steve’s screen-shot for the scheme settings on the Specs framework is cut off and does not show the “Expand Variables Based On” setting, so $(SRCROOT) was never expanded for me. I replaced it with an absolute path (ugh) and it worked, so I knew something was not picking up that macro. The solution was to give a value to that drop-down, as shown in the screen-shot below.

If, like me, you’re on Xcode V4.3.2, you might find the following screen-shots useful (just refer to them as you follow Steve’s post):

a. Build settings:

b. Scheme settings:

You cannot imagine the joy of seeing Ruby code drive an Objective-C framework testing session using rSpec in Xcode.

Now to that mower…

DDD – Document Driven Development

We rarely document. We are used to being handed a set of PowerPoint slides that describe, on a very high level, the business need for software. We roll our eyes at the slides, and get to work, asking questions, clarifying the needs, hope to understand them and start imagining features and how we can deliver the implementation within the requested timeline.

If we follow the Agile framework, we’ll translate the transformed slides into stories. We do so and derive tasks from them. If we’re lucky, we might be able to condition the business to accept deliverable milestones that are aligned with those stories.

Using BDD, we’ll transcribe the stories into Gherkin and using TDD, we’ll start coding tests at that time (rSpec, Cucumber).

As development gets under way, we cycle through iterations and we deliver collaboratively.

After the celebrations, all the good things mentioned above (stories, milestones, BDD, TDD) evaporate as the project starts gliding at low altitude as the business moves to new territories. We’re left with mundane maintenance and tickets are opened for small bug fixes and minor enhancements. Stories are no longer written as “it’s not worth it” and small changes are never fully documented.

The project stops being documented and over time, as the team members rotate and business rules change, people no longer remember why we check-off the ‘accept contract’ terms after signup and not on the page where the user enters their email address. It so happens that there will be a major impact on the back-end provisioning system if we change that.

I think the pattern is clear – If we don’t use our documents, the whole eco-system of our product degrades to entropy and will ultimately lead us to revival by rewrite, or at least by going through the analysis again and likely to some re-engineering. Time wasted.

What I would love to see is a system whereby the development and maintenance is driven by documentation and that the documentation drives the deliverables.

The pieces are there, we just need to use them:
Participate in the requirements phases, translate them to stories, deliver story implementions. Always, recurringly. Never stopping this cycle.

Months from now, anyone reading your stories will fully understand why the system behaves the way it does – people like to read stories and will understand the system on their own terms. New hires in the business will use them as a guidline on how to perform their jobs. New developers to the team will have a standard to meet when fixing bugs or evaulating new or changed requirements.

We will end up with a document-driven system, accumulating a library of living documents that drove our software development effort. Any new contradictory story will violate the automated validations for previous generations of stories and will stop us in our tracks, showing us exactly where the business flow will break if we add that new feature. No one actually needs to know this in advance: Let the business tell new stories and see how the system reacts. It’ll tell us whether we’re in violation of any existing processes and alert us automatically.

If you’re using Gherkin and Cucumber already, put them front and center of your development workflow and don’t let go of them!

 

The tip of the (good) iceberg

Recently, a “perfect storm” situation occurred when we realised that there was a convergence of a new business need with an old technical need.

We have a technical issue that we wanted to deal with for a long time and yet never got to it because of high-priority tasks requests for the business. Our issue has to do with overhauling data structures and internal SOA processes to be more flexible and to be able to support business requirements in the future. The tasks and migrations were analyzed and estimated at “medium” and “hard” complexity levels and felt like an elephant in every meeting concerning the project, which is a central one in our business.

The “perfect storm” appeared when the business unit requested a feature that was solved by our internal analysis as part of the overhaul, but the key factor was that they asked for an initial implementation where only 20% of the customers be effected.
We were confronted with a situation where both parties had the same goal – we had technical justifications to make what we saw as needed changes and the business unit had a market-driven justification for asking for changes. This is a perfect situation to be in as both units are aligned and there is no conflict of interest.

The beauty of the situation is in the “20%”. By requesting that only a selected 20% of the customer base be affected, we could now picture the technical scope of the project with a different mindset – one of depth of development for a limited breath of the customers. By this I mean that we are able to plan for the “most value” for the business unit – producing working software to solve for the 20%, while back-filling the rest of our technical debt towards this project by the current processes till more is developed for the next segment of customers. True, there will be “production support” till we achieve 100% customer base, yet solving for 20% economically justifies that cost and effort.

The result is that the business unit will see only the tip of the iceberg of this project with immediate value, while we work on the invisible part that will cover the subsequent market segments that will be addressed sequentially over time.

The convergence of the business goals with ours makes it possible for us to succeed with this project and introduce it to the market in small segments. If only all business and technical requirements were so well aligned!

Dolly – The case against deploy drama

These are the slides from a presentation I made concerning deployments and how we can avoid downtime and drama.

Dolly – Deploying

 

Oh, the places you’ll go…

Inspired from the Practicing Ruby entry, I somewhat clarified the code a little (for my taste) and learned that the call stack in Ruby is:

0) Undefined method resolution
1) Methods defined in the object’s singleton class (i.e. the object itself)
2) Modules mixed into the singleton class in reverse order of inclusion
3) Methods defined by the object’s class
4) Modules included into the object’s class in reverse order of inclusion
5) Methods defined by the object’s superclass, i.e. inherited methods

module ModuleA
 def foo
   "- Mixed in method defined by ModuleA\n" + super
 end
end  
module ModuleB
  def foo
   "- Mixed in method defined by ModuleB\n" + super
  end
end  
module ModuleC
  def foo
   "- Extended in method defined by ModuleC\n" + super
 end
end  
module ModuleD
  def foo
   "- Extended in method defined by ModuleD\n" + super
 end
end  
class A
 def foo
   "- Instance method defined by A\n"
 end
end  
class B < A
 include ModuleA
 include ModuleB
 def foo
    "- Instance method defined by B\n" + super
 end  
  def method_missing(method)
   puts "- method_missing (#{method}) on b. Redeirecting to b.foo\n"
   foo
 end
end  
b = B.new
b.extend(ModuleC)
b.extend(ModuleD)
def b.foo
 "- Method defined directly on an instance of B\n" + super
end
def b.method_missing(method)
 "- method_missing (#{method}) on b. Calling super\n" + super
end
puts "Calling 'bar' on b of type #{b.class}:\n"
puts b.bar

Which gives:

~/projects/ita/ruby$ ruby test.rb

Calling ‘bar’ on b of type B:

- method_missing (bar) on b. Redeirecting to b.foo
- method_missing (bar) on b. Calling super
- Method defined directly on an instance of B
- Extended in method defined by ModuleD
- Extended in method defined by ModuleC
- Instance method defined by B
- Mixed in method defined by ModuleB
- Mixed in method defined by ModuleA
- Instance method defined by A

Follow the conversation on Stack Overflow.

Puppet book review

This book is an excellent Puppet book for beginners and professionals alike.

I manage a software team and have read this book cover-to-cover in order to study Puppet for our team’s use on a daily basis.

Despite step-by-step instructions for the initial installation, I needed some tinkering since different OSs have slightly different distributions, but once I had a server and agent running on two different VMs (Ubuntu) – there was an “Aha!” moment when the agent had emacs automatically installed on it! Getting past the initial installation phase allowed me to really enjoy the rest of the book as well as enjoy Puppet itself.

Puppet is not trivial, but the book covers its concepts very clearly and one “gets” it quite early on (especially if you get your hands dirty and follow along the examples).

The book then expertly guides the reader to its “pro” section detailing use of Puppet with configuration management tools such as git and db-based storage.

It then goes on to detail how to use AMQ with Puppet for scaling. I doubt I will use such a robust configuration, but was thrilled to see how flexible and extensible Puppet is by use of load-balancers and integration with Apache/Passenger.

Overall, the book is well written, and I would highly recommend it as a *text book* for Puppet. This is a readable text book on the subject – not a reference manual, although it has countless links to the reference manuals.

I always wanted to learn Puppet, and this book certainly is the one to read if you’re dealing with configuration management whether as a developer or a DevOps person.

Setting up a Rails server on a GoDaddy VPS

I thought my experience with setting up a Centos 5 box from scratch with Rails 3.1 would be helpful to some readers.

1. Get a VM – this one is on GoDaddy, just for kicks.

Demo config:

Operating System: CentOS 5
RAM: 1 GB
Disk Space: 20 GB
And it costs $30 a month. Not too bad.

2. Get some tools

Become root for that: “$ su -”

Then issue:

# yum groupinstall ‘Development Tools’
# yum groupinstall ‘Development Libraries’
# exit

3. Install your ssh key for logins

Copy your key to ~/.ssh/authorized_keys

chmod go-w ~
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

4. Install node.js

Become root for that: “$ su -”
Then issue:

# cd /root
# wget http://nodejs.org/dist/node-v0.4.11.tar.gz
# gunzip node-v0.4.11.tar.gz
# tar -xf node-v0.4.11.tar
# cd node-v0.4.11
# ./configure
# make
# make install
# exit

5. Install Git

Become root for that: “$ su -”
Then issue:

# yum install gettext-devel expat-devel curl-devel zlib-devel openssl-devel
# yum install zlib-devel
# yum install openssl-devel
# yum install perl
# yum install cpio
# yum install expat-devel
# yum install gettext-devel

# wget http://www.codemonkey.org.uk/projects/git-snapshots/git/git-latest.tar.gz
# tar xzvf git-latest.tar.gz
# cd git-{date}
# autoconf
#./configure –with-curl=/usr/local
# make
# make install
# exit

6. Install RVM

$ bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)

then add
[[ -s "/home/your-user/.rvm/scripts/rvm" ]] && source “/home/your-user/.rvm/scripts/rvm”
to the end of .bash_profile

8. Install readine

$ rvm pkg install readline

9. Install ruby

$ rvm install 1.9.2 –with-readline-dir=$rvm_path/usr

10. Create a gemset

$ rvm gemset create rails3.1

$ rvm –default use 1.9.2@rails3.1

11. Load Rails3.1

$ export LC_CTYPE=en_US.UTF-8

$ export LANG=en_US.UTF-8

$ gem install rails 3.1

12. Create ssh key for git repo

$ ssh-keygen -t rsa

13. Upload the public key to your repo

Make sure the end of the key file has a newline

Test access by issuing
$ git clone ssh://git@yourepo.com/xxx.git

14. Install bundler

$ gem install bundler

Test bundler by running ‘bundle install’ in the directory created by (6)

15. Install mysql

Become root for that: “$ su -”

Then issue:

# yum install mysql
# /etc/init.d/mysqld start
# exit

16. Get a copy of your project

From git by cloning the repo and run ‘rake spec’ to see that everything is installed and running correctly
This assumes you use rSpec, else run ‘rake test’, or whatever testing framework you use.

17. Install passenger

$ gem install passenger
$ passenger start

18. Test it out…

Navigate to http://xxxx:3000 to see your app!

Hello world!

This seems to be a nice place to organise my thoughts. Please don’t think that I presume that those are at all worthy to be published just because they appear here. I’ll use this shoebox as a platform to solicit constructive criticism in order to chop away at my ignorance.