I enjoy writing code using BDD by starting from a user story and writing a cucumber feature to define what I want to see on the page. From there I drop down several levels from Cucumber to RSpec and back again as I go.

When creating a new cucumber feature using declarative steps, it’s normal to frequently have undefined steps. It’d be helpful if our editor could show us which steps are already matched (and thus defined in our step definitions) and which steps are undefined as a sort of progress bar to track feature step implementation.

This is possible with Vim and Syntastic. I’ll provide details about how to get this all working in the context of a Rails 4 application with Ruby 2.2.1.

Step 1

Alter your cucumber.yml file to include a syntastic profile and use the -r flag to require the files contained within the step_definitions directory.

1
2
3
4
5
6
7
8
9

<%
rerun = File.file?('rerun.txt') ? IO.read('rerun.txt') : ""
rerun_opts = rerun.to_s.strip.empty? ? "--format #{ENV['CUCUMBER_FORMAT'] || 'progress'} features" : "--format #{ENV['CUCUMBER_FORMAT'] || 'pretty'} #{rerun}"
std_opts = "--format #{ENV['CUCUMBER_FORMAT'] || 'pretty'} --tags ~@wip --snippet-type percent"
%>
syntastic: -r features/step_definitions <%= std_opts %> CUCUMBER_PROFILE='syntastic'
default:   -r features                  <%= std_opts %>
wip:       -r features                  --tags @wip:3 --wip
rerun:     -r features                  <%= rerun_opts %> --format rerun --out rerun.txt --strict --tags ~@wip --snippet-type percent

Note: if your feature files are in a non-standard folder (features/steps perhaps), be sure to use that folder name instead of features/step_definitions

Step 2

Setup Vim to use the newly added syntastic cucumber profile in your .vimrc:

1
2
3
4
5

let g:syntastic_mode_map = { 'mode': 'active' }
let g:syntastic_ruby_checkers = ['mri', 'rubocop']
let g:syntastic_cucumber_cucumber_args="--profile syntastic"
let g:syntastic_warning_symbol = "⚠"
" let g:syntastic_debug = 1

syntastic_ruby_checkers specifies both mri and rubocop, which I highly recommend. If you don’t want to use rubocop, just remove it from the array.

The last option is commented out, but is useful if syntastic isn’t working properly. Just uncomment it, then save a feature file, and check the messages output from vim in Command-line mode with :messages. This will show you the cucumber command that syntastic just ran. Executing the command that syntastic tried to run yourself at the command line is a great troubleshooting resource if something isn’t working right. Here’s what syntastic should be trying to do at this point when running a single feature file.

1

cucumber --profile syntastic --dry-run --quiet --strict --format pretty features/creating_a_new_company.feature

If you don’t use any gems that define methods that need to be available in step definitions then you’re done!

Working with gems that define methods that implement regex patterns

If you use the pickle gem or another gem that defines methods to help with regex matching in step definitions, then you’ll need to add a few more tweaks to get proper undefined step status reported from syntastic.

Notice the env var specified in the above cucumber.yml file, CUCUMBER_PROFILE='syntastic'. This is so that we can conditionally require some extra code in a few step definition files that need methods defined from within the pickle gem to be available.

The “highlight undefined steps” feature of syntastic takes advantage of the ability for cucumber to do a ‘dry run’ where it doesn’t actually execute any of the code inside of the steps. It will, however, let you know if a step is undefined. One caveat with --dry-run is that it doesn’t load the rails environment, so the pickle steps that need to reach into the gem for some regex matching will be shown as undefined. Initially I took the path of just defining these methods inline in the file like so:

1
2
3
4
5
6
7
8
9
10
11
12
13

if ENV['PROFILE'] == 'syntastic'
  # allow cucumber --dry-run to execute (env.rb isn't loaded), so these methods
  # defined in the pickle gem won't be found. Define them here with a very
  # generic capture-all regex. Doing this also requires the --guess flag be
  # added to the syntastic profile since these matchers make things a litte
  # more generic than the actual gem versions
  def capture_email;          '(.+)'; end
  def capture_model;          '(.+)'; end
  def capture_fields;         '(.+)'; end
  def capture_plural_factory; '(.+)'; end
  def capture_predicate;      '(.+)'; end
  def capture_value;          '(.+)'; end
end

However, the --guess flag is required for cucumber with this setup. That proved to be problematic and I had a lot of false positives that matched when they shouldn’t have.

Since the rails environment isn’t loaded we need to manually require the necessary files to allow our pickle steps to call their gem’s methods.

Step 3

Create a features/support/pickle_dry_run.rb with the following:

1
2
3
4
5
6
7
8
9
10

module PickleDryRun
  pickle_path = Bundler.rubygems.find_name('pickle').first.full_gem_path
  require "#{pickle_path}/lib/pickle"
  require "#{pickle_path}/lib/pickle/parser/matchers"
  require "#{pickle_path}/lib/pickle/email/parser"
  include Pickle
  def config; Pickle.config; end
  include Pickle::Parser::Matchers
  include Pickle::Email::Parser
end

Step 4

In the pickle_steps.rb file add this to the top:

1
2
3
4

if ENV['CUCUMBER_PROFILE'] == 'syntastic'
  require_relative '../../support/pickle_dry_run'
  include PickleDryRun
end

This will conditionally require and includes the PickleDryRun module during a syntastic check so that the steps in pickle_steps.rb can call out to the methods contained within the pickle gem.

If you generated the pickle email steps, then we need to add something similar. The top of my email_steps.rb looks like this:

1
2
3
4
5
6
7
8
9
10
11

if ENV['CUCUMBER_PROFILE'] == 'syntastic'
  require_relative '../../support/pickle_dry_run'
  include PickleDryRun
else
  ActionMailer::Base.delivery_method = :test
  ActionMailer::Base.perform_deliveries = true

  Before do
    ActionMailer::Base.deliveries.clear
  end
end

Step 5

To allow for factory_girl to work properly with pickle inside of a dry-run triggered by syntastic, we need to execute cucumber with spring.

Luckily, the ability to change the syntastic cucumber executable exists and is easily configurable. Just beneath the syntastic_cucumber_cucumber_args specify this in your .vimrc:

1

let g:syntastic_cucumber_cucumber_exe='bin/cucumber'

This will use the cucumber binstub which relies on spring having already loaded the rails environment. The first request can be slow-ish, since spring has to spin up rails (if it hasn’t yet), but subsequent executions are very fast.

Just make sure that there is a cucumber binstub in the bin folder at the root of the rails app. If there isn’t, be sure to check out spring’s setup guide which mentions the need to run bundle exec spring binstub --all. The spring-commands-cucumber gem is also required as noted in the spring readme.

With all of this in place, things should be working properly. As with most things vim though, closing completely out re-launching vim is usually necessary.

Now if you turn syntastic debug mode on, it’ll show cucumber executing from the binstub. Nice!

1

bin/cucumber --profile syntastic --dry-run --quiet --strict --format pretty features/creating_a_new_company.feature

Next time you create an undefined step and save your feature file, it’ll look like this!

Today I re-learned a valuable lesson about making a test fail before letting it pass. I was working on an old codebase that wasn’t using the site prism gem and so I was doing some DOM checking by hand in Cucumber, something I hadn’t done in a while. In a step definition I expected that if I returned false, the step would fail. This, however, is not the case!

I feel like I have known this for a long time but simply forgot. The rest of the steps that check things use rspec to do so with syntax like:

1

expect(page.find('.some_class').text).to eq('some value')

But I didn’t remember or notice that at first. Here’s the step definition I thought would fail properly, and fail at the correct line, giving a “good error message” to boot.

1
2
3
4
5
6

Then %r{I should see "(.+)" for each entry in the applications table$} do |val|
  page.find('td.industry').text == val
  page.find('td.goal').text     == val
  page.find('td.profit').text   == val
  page.find('td.revenue').text  == val
end

It passed the first time and I almost moved on, but I decided to change the default value for the last table cell and make sure it falied. In this app the UI shows a default value of “TBD” if a user logs in to check their application status and that field isn’t yet populated. Thank goodness I checked because it continued to pass. And then I couldn’t for the life of me make it fail!

1
2
3

Then %r{I should see "(.+)" for each entry in the applications table$} do |val|
  1==2
end

passed…

1
2
3

Then %r{I should see "(.+)" for each entry in the applications table$} do |val|
  false
end

passed…

I was getting very concerned. If I issued a raise statement it would error out… so why weren’t my checks that evaluated to false causing cucumber to fail?!?!

As it turns out, it doesn’t matter what you return in a step definition. You have to raise an error if you want the step to fail. I must have never fully groked that since I previously always remembered to just use rspec when evaluating outcomes in a step definition.

So after 10 minutes of wondering if I was going to be filing a bug against cucumber, I finally ended up with this step def failing properly:

1
2
3
4
5
6
7
8
9

Then %r{I should see "(.+)" for each entry in the applications table$} do |val|
  vals = []
  vals << page.find('td.industry').text == val
  vals << page.find('td.goal').text     == val
  vals << page.find('td.profit').text   == val
  vals << page.find('td.revenue').text  == val

  expect(vals.flatten.uniq).to eq([val])
end

After changing one of the table columns to have the text ‘FOO’ instead of ‘TBD’ the failures generated out of this step definition now work, but it’s not a very obvious failure. Something that, again, I wouldn’t have noticed if I haden’t taken the time to watch the test fail.

1
2
3
4
5
6
7
8

And I should see "TBD" for each entry in the applications table

  expected: ["TBD"]
       got: ["TBD", "FOO"]

  (compared using ==)
   (RSpec::Expectations::ExpectationNotMetError)
  ./features/step_definitions/entrepreneur_application_steps.rb:8:in `/I should see "(.+)" for each entry in the applications table$/'

The problem here is that the line rspec points out is the final line that actually does the assertion. This doesn’t help us track down which css class has the incorrect text.

So the final refactored version, while somewhat less DRY, is actually a net win when the failure crops up 6 months down the road. The failure message tells me exactly what css class has the text not matching my expectation.

1
2
3
4
5
6

Then %r{I should see "(.+)" for each entry in the applications table$} do |val|
  expect(page.find('td.industry').text).to eq(val)
  expect(page.find('td.goal').text).to eq(val)
  expect(page.find('td.profit').text).to eq(val)
  expect(page.find('td.revenue').text).to eq(val)
end

1
2
3
4
5
6
7
8

And I should see "TBD" for each entry in the applications table

  expected: "TBD"
       got: "FOO"

  (compared using ==)
   (RSpec::Expectations::ExpectationNotMetError)
  ./features/step_definitions/entrepreneur_application_steps.rb:5:in `/I should see "(.+)" for each entry in the applications table$/'

This makes it very clear that line 5 is the offending line, and I can go check td.revenue to see what’s up.

A lot of the FUD you read about Cucumber involves teams abandoning the tool because they think it adds a layer of extra work to the testing process. However, teams that go that route are missing a big benefit of plain text user stories: executable documentation that is readable by “the business”. The ultimate goal is to have the business (aka the stakeholders) actually collaborate on user stories so that a shared understanding of what needs to be built can be developed across the whole team.

In order to achieve that goal, however, Cucumber scenaios need to be written in a certain way. They need to read as conversational requirements, not a QA test plan check list. This style of writing has a name: declarative.

“Possibly the most common sentence type in the English language, declarative
sentences are used when you want to make a statement. Whether it’s a bold
statement or a simple fact, the sole purpose of a declarative sentence is to
give information. It always ends with a simple period. And if you’d like to see
an example of a declarative sentence, you don’t need to look any further.
Actually, every sentence in this paragraph is a declarative sentence.”

The most important part of that last paragraph is this: “the sole purpose of a declarative sentence is to give information”. This is why Cucumber is best suited to this style of writing. A user story’s entire purpose is also to give information.

Perhaps the easiest way to show a good user story style is by first showing an antipattern. Here is a user story written in an imperative style. If you have been using Cucumber for a long time, you might recgonize these steps as being matched by the now defunkt features/support/web_steps.rb file.

1
2
3
4
5
6
7
8

Given I am on the home page
 When I fill in "Username" with: "jondkinney"
  And I fill in "Password" with: "SuperSecret123"
  And I check "Remember me"
  And I press "Log in"
 Then a user session should be persisted
  And I should be on my dashboard
  And I should see "You have successfully logged in."

Sometimes this style can be useful. It can help tease out an implementation detail that the only documented way to log in is clicking the ‘Log in’ button, for instance. However, I think stories that are structured in a declarative method overall provide more value more of the time.

A declarative version of the story above might read like this:

1
2
3
4

Given I am on the home page 
 When I login as an admin
 Then I should be on my dashboard 
  And I should see "You have successfully logged in."

The implementation details are largely gone from the story, and it reads a lot better.

Another example where an admin is merging two user accounts in an asset management system might read something like this:

1
2
3
4

Given two users with matching addresses and different spellings for full name
 When I edit one of the user names to match the other and submit the form
 Then the assets for both user records should exist under a single user 
  And the duplicate user record should no longer exist

Here’s another shorter example:

1
2
3

Given I am logged in as an admin 
 When I unlock a user account
 Then the user should receive an email with a link to reset their password

Other resources on declarative user stories

This style isn’t a new idea anymore. However, many of the teams we work with still fall into the imperative trap when a deadline looms or it’s unclear how to extract several steps into a more declarative one. For some more inspiration on the subject, take a look at a few posts from other experts on testing and BDD.

• Imperative vs declarative scenarios in user stories
• Declarative vs imperative gherkin scenarios for cucumber

And from the author of cucumber itself: The training wheels came off

We’ve had good luck with this more declarative style at Faster Agile. Stakeholders can better read and understand the requirements in a user story, and with this conversational and plain language tool to describe how the system is supposed to work, they also help write the stories more often too!

Taking it a step further

After conquering the verbosity and repetativeness of imperative stories, there’s still a nagging issue with the resulting step defintions. Using bare capybara as the underlying mechanism to interact with the DOM can lead to mutliple ways of targeting elements to fill-in and click. Even if a project tries to standardize on targeting only IDs or only label text, that code is still repeated in many places leading to a brittle test suite that is locked pretty heavily to the existing ID/class structure and/or label text.

Stay tuned for a future post where we’ll show how Faster Agile uses the Site Prism gem and the concept of “page objects” to encapsulate the DOM for any given page making it much easier to keep a Cucumber suite more decoupled from the DOM it’s driving.