Today I re-learned a valuable lesson about making a test fail before letting it pass. I was working on an old codebase that wasn’t using the site prism gem and so I was doing some DOM checking by hand in Cucumber, something I hadn’t done in a while. In a step definition I expected that if I returned false, the step would fail. This, however, is not the case!

I feel like I have known this for a long time but simply forgot. The rest of the steps that check things use rspec to do so with syntax like:

1

expect(page.find('.some_class').text).to eq('some value')

But I didn’t remember or notice that at first. Here’s the step definition I thought would fail properly, and fail at the correct line, giving a “good error message” to boot.

1
2
3
4
5
6

Then %r{I should see "(.+)" for each entry in the applications table$} do |val|
  page.find('td.industry').text == val
  page.find('td.goal').text     == val
  page.find('td.profit').text   == val
  page.find('td.revenue').text  == val
end

It passed the first time and I almost moved on, but I decided to change the default value for the last table cell and make sure it falied. In this app the UI shows a default value of “TBD” if a user logs in to check their application status and that field isn’t yet populated. Thank goodness I checked because it continued to pass. And then I couldn’t for the life of me make it fail!

1
2
3

Then %r{I should see "(.+)" for each entry in the applications table$} do |val|
  1==2
end

passed…

1
2
3

Then %r{I should see "(.+)" for each entry in the applications table$} do |val|
  false
end

passed…

I was getting very concerned. If I issued a raise statement it would error out… so why weren’t my checks that evaluated to false causing cucumber to fail?!?!

As it turns out, it doesn’t matter what you return in a step definition. You have to raise an error if you want the step to fail. I must have never fully groked that since I previously always remembered to just use rspec when evaluating outcomes in a step definition.

So after 10 minutes of wondering if I was going to be filing a bug against cucumber, I finally ended up with this step def failing properly:

1
2
3
4
5
6
7
8
9

Then %r{I should see "(.+)" for each entry in the applications table$} do |val|
  vals = []
  vals << page.find('td.industry').text == val
  vals << page.find('td.goal').text     == val
  vals << page.find('td.profit').text   == val
  vals << page.find('td.revenue').text  == val

  expect(vals.flatten.uniq).to eq([val])
end

After changing one of the table columns to have the text ‘FOO’ instead of ‘TBD’ the failures generated out of this step definition now work, but it’s not a very obvious failure. Something that, again, I wouldn’t have noticed if I haden’t taken the time to watch the test fail.

1
2
3
4
5
6
7
8

And I should see "TBD" for each entry in the applications table

  expected: ["TBD"]
       got: ["TBD", "FOO"]

  (compared using ==)
   (RSpec::Expectations::ExpectationNotMetError)
  ./features/step_definitions/entrepreneur_application_steps.rb:8:in `/I should see "(.+)" for each entry in the applications table$/'

The problem here is that the line rspec points out is the final line that actually does the assertion. This doesn’t help us track down which css class has the incorrect text.

So the final refactored version, while somewhat less DRY, is actually a net win when the failure crops up 6 months down the road. The failure message tells me exactly what css class has the text not matching my expectation.

1
2
3
4
5
6

Then %r{I should see "(.+)" for each entry in the applications table$} do |val|
  expect(page.find('td.industry').text).to eq(val)
  expect(page.find('td.goal').text).to eq(val)
  expect(page.find('td.profit').text).to eq(val)
  expect(page.find('td.revenue').text).to eq(val)
end

1
2
3
4
5
6
7
8

And I should see "TBD" for each entry in the applications table

  expected: "TBD"
       got: "FOO"

  (compared using ==)
   (RSpec::Expectations::ExpectationNotMetError)
  ./features/step_definitions/entrepreneur_application_steps.rb:5:in `/I should see "(.+)" for each entry in the applications table$/'

This makes it very clear that line 5 is the offending line, and I can go check td.revenue to see what’s up.