Behaviour Driven Development for Lua.
Specl is testing tool for Lua, providing a Behaviour Driven Development framework in the vein of RSpec.
specl command verifies that the behaviour of your software meets
the specifications encoded in one or more spec-files. A spec-file is
a YAML structured file, laid out as groups of nested plain-English
descriptions of specifications, with associated snippets of Lua
code that verify whether the software behaves as described.
A tiny spec-file outline follows:
The first significant line of any specification is the plain-English
description of the first example group, ending with a
Underneath that are two examples, each starting with
-_ (minus, space)
and separated by a
: (colon) into a description of some desired
behaviour, and the associated Lua code to demonstrate it.
The descriptions above follow the RSpec convention of using
describe as the first word of a group description, and it as the
first word of an example description. Specl doesn’t enforce them,
they are conventions after all, but
specl output tends to look much
better if you follow them. There are more conventions regarding the
choice of first word in a description under various other circumstances,
which we’ll cover shortly.
A fuller spec-file will contain several example groups, similar to the
one above, each typically followed by dozens of individual examples.
To easily keep track of what specifications go with what parts of your
implementation, it’s good practice to put all your specs in a
subdirectory, with one spec named after each file being specified. For
example, your application might have a
src/stack.lua class, along
specs/stack_spec.yaml file that contains all the matching
All of those specifications eventually boil down to lists of behaviour descriptions and example code, all indented as prescribed by the YAML file-format.
YAML makes for a very readable specification file-format, and allows embedded Lua code right within the standard, as you saw in the last section. However, there are some rules to follow as you write your spec-files in order to maintain valid YAML format that Specl can load correctly.
Indenting with TAB characters is a syntax error, because the YAML parser uses indentation columns to infer nesting. It’s easiest just to avoid putting TAB characters in your spec files entirely.
Some punctuation is not allowed in an unquoted YAML string, so you
will need to force the parser to read the description as a string by
surrounding it with
" (double-quote mark) if you want to put any
punctuation in the description text:
Indentation of the code following an example description must be at
least one column further in than the first letter of the description
text above, because YAML counts the leading
-_ (minus, space) as
part of the indentation whitespace.
Specl treats everything following the
: (colon) as a Lua code:
By default YAML removes indentation and line-breaks from the example
code following the
: separator, so that by the time Lua receives
the code, it’s all on a single line. More often than not, this isn’t a
problem, because the Lua parser is not overly fussy about placement
of line-breaks, but sometimes (to make sure there is a newline to
terminate an embedded comment, for example) you’ll need to prevent
YAML from giving Lua everything on a single line. Use the
literal block marker
_| (space, pipe) after the
: separator for
You also have to be careful about commenting within a spec-file. YAML
comments begin with
_# (space, hash) and extend to the end of the line.
You can use these anywhere outside of a Lua code block, including any
lines immediately following a description before any actual Lua code.
Lua comments don’t work outside of a lua block, and YAML comments
don’t work inside a Lua block, so you have to pick the right comment
character, depending where in the hierarchy it will go.
You can further sub-divide your example groups by context. In addition to listing examples in each group, list items can also be contexts, which in turn list more examples of their own:
By convention, the context descriptions start with the word “context”, but Specl doesn’t enforce that tradition, so you should just try to write a description that makes the output easy to understand (see Command Line).
Actually, description naming conventions aside, there is no difference between an example group and a context: Each serves to describe a group of following examples, or nested contexts.
Specl doesn’t place any restrictions on how deeply you nest your contexts: 2 or 3 is very common, though you should seriously consider splitting up a spec if you are using more than 4 or 5 levels of nesting in a single file.
At the innermost nesting of all those context and example group entries, you will ultimately want to include one or more actual examples. These too are best written with readable names in plain-English, as shown in the sample from the previous section, but (unlike contexts) they are followed by the associated example code in Lua, rather than containing more nested contexts.
Traditionally, the example descriptions start with the words “it”, “example” or “specify”, but again, Lua really doesn’t mind what you call them.
Each of your examples lists a series of expectations that Specl runs
to determine whether the specification for that part of your project is
being met. Inside the Lua part of each example, you should write a
small block of code that checks that the example being described meets
your expectations. Specl gives you a new
expect command to check
that each example evaluates as it should:
The call to expect is almost like English: “Expect size of stack to be zero.”
Behind the scenes, when you evaluate a Lua expression with expect, it’s
passed to a matcher method (
.to_be in this example), which is
used to check whether that expression matched its expected evaluation.
There are quite a few matchers already implemented in Specl, and you
can easily add new ones if they make your expectations more expressive.
The next section describes the built in matchers in
more detail. The
specl.shell module provides some specialist matchers
for checking whether shell commands have behaved according to
specifications; see the section on shell commands for
more details on those.
Often, you’ll think of a useful expectation or behaviour that you don’t have time to implement right now. Noting it off-line somewhere, or even adding a commented out example is likely to lead to it being forgotten. Better to add it to your spec-file as a pending example while it is still on your mind, so that Specl can remind you that it needs finishing – but without contributing a full-blown failing expectation or specification.
The simplest kind of pending example is an example description with no associated Lua example code:
The built in formatters will show this as a non-failing unimplemented (pending!) example when the spec-file is checked, so you can keep track of specifications for code you have yet to write.
Alternatively, if you have written a suitable specification, only to
realise that you are specifying an unimplemented behaviour, just add
a call to
pending () somewhere near the beginning of the example
to disable following expectations, without removing or commenting out
This prevents Specl from counting the
expect result as a failure,
but crucially also allows Specl to inform you when the expectation
begins passing to remind you to remove stale
pending () calls from
Sometimes, it’s useful to add some metadata to a pending example that
you want to see in the summary report. Pass a single string parameter
pending function call like this:
Running Specl now shows the string in the pending summary report:
There is no need for a formal
skip command in Specl; the example
execution engine will not display the descriptions of examples that have
no live expectations.
When running this spec-file with the Lua 5.2 interpreter, the hash example expectation fires, and its result is reported:
However, with Lua 5.1 (which does not respect the
of tables with the hash operator), the
if expression will be false, and
expect command is not executed. In that case, formatters do
not report that result, effectively skipping the example:
Note that, like
pending examples, the example Lua code is
executed and needs to be well formed, it is just ignored by formatters
when reporting the results from a spec-file.
expect looks up a matcher to validate an expectation, the
to_ part is just syntactic sugar to make the whole line read more
clearly when you say it out loud. The idea is that the code for the
specification should be self-documenting, and easily understood by
reading the code itself, rather than having half of the lines in the
spec-file be comments explaining what is going on, and needing to be
kept in sync with the code being described.
The matchers themselves are stored by just the root of their name (
in this case). See
Inverting a Matcher with not, for more
about why that is.
The matchers built in to Specl are listed below.
This matches only when the result of
expect is the exact same object
as the matcher argument. For example, Lua interns strings as they
are compiled, so this expectation passes:
Conversely, Lua constructs a new table object every time it reads one from the source, so this expectation fails:
While the tables look the same, and have the same contents, they are still separate and distinct objects.
To get around that problem when comparing tables, or std.object derived
objects, use the
equal matcher, which does a recursive element by
element comparison of the contents of the expectation arguments. The
following expectations all pass:
equal, this matcher is also useful for comparing tables, or
std.object derived objects, and usually gives the same results.
copy will fail if the result of
expect is the exact same
object as the matcher argument. The following example:
is equivalent to:
When comparing strings, you might not want to write out the entire
contents of a very long expected result, when you can easily tell with
just some substring whether
expect has evaluated as specified:
expect evaluates to a table, this matcher will
succeed if any element or key of that table matches the expectation
string. The comparison is done with
equal, so table elements or
keys can be of any type.
A final convenience is that
contain will use the
metamethod of any lua-stdlib
std.object derived objects to coerce
a table to test for matching keys or values in the expectation.
expect passes anything other than a string, table or
derivative to this matcher, Specl aborts with an error; use
tostring or similar if you need to.
When a simple substring search is not appropriate,
match will compare
the expectation against a Lua pattern:
Specifications for error conditions are a great idea! And this matcher
checks both that an
error was raised and that the subsequent error
message contains the supplied substring, if any.
raise above, but instead of searching for a substring, this
matcher checks against a Lua pattern:
Oftentimes, in your specification you need to check that an expectation
does not match a particular outcome, and Specl has you covered
there too. Rather than implement another set of matchers to do that
though, you can just insert
not_ right in the matcher method name.
You can write
not_ either before or after
to_, whichever you find
most readable. Some people are annoyed by split infinitives, but
Specl is not as grumpy as that, and will happily accept
not_to_ as entirely equivalent.
Specl will still call the matcher according to the root name (see Matchers), but inverts the result of the comparison before reporting a pass or fail:
Note that the last
not_to_raise example doesn’t pass the error
message substring that to not matches, because it is never checked,
but you can pass the string if it makes an expectation clearer.
In addition to using matchers for straight one-to-one comparisons
between the result of an
expect and the argument provided to the
matcher, Specl has some shortcuts that can intercept the arguments
and adapt the comparison sequence. These shortcuts are called
When you want to check whether an expectation matches among a list of
alternatives, Specl supports an
any_of adaptor for any matcher:
The expectation above succeeds if
ctermid () output matches any of
the patterns in the table argument to
Conversely, as you might expect, when you combine
an expectation succeeds only if none of the alternatives match:
When you need to ensure that several matches succeed, Specl provides
This expectation succeeds if the
split method produces a table that
contains each of the strings in the argument to
all_of; note that it
does not fail if there are elements other than those specified - the
example above will succeed even though there is (presumably!) an
"5" element in the table returned by this
all_of can surely be combined with
not, but the
resulting expression is hard to understand, so I recommend that you
don’t use it. Try running the following to see whether it behaves as
you expect, and notice how carefully you have to think about it
compared to the usual English inspired syntax of Specl
If you want to assert that an expectation does not contain any of the supplied elements, it is far better to use:
While Specl makes every effort to maintain ordering of elements in
the tables (and objects) it uses, there are times when you really want
to check the contents of an inherently unordered expectation - say,
pairs returns all the elements of a set containing functions
which can’t be guaranteed to have the same sort order on every run.
In this example, sorting
elements before comparing them is dangerous,
because we can’t know what order the addresses of the functions it
contains will have been assigned by Lua, but using
here guarantees that
elements contains the same elements as
irrespective of order.
Prior to the introduction of
all_of was the nearest
equivalent functionality - but
all_of will not complain if
has even more elements than what it is supposed
to_contain at the time
Just like the built in matchers described above, you can use the
Matcher factory object from
specl.matchers to register additional
custom matchers to make your spec files easier to understand. The
minimum required is a predicate method, which is then called by
Specl to determine whether the result of an
matches the contents of the
This is exactly how the
be matcher is implemented, where Specl
actual result from the expectation and the
value from the
to_ argument – and considers the expectation as
a whole to have passed if they are both the same according to a Lua
Of course, our custom
be matcher reimplementation is not available
to spec files until it has been registered in Specls matcher table.
You can do this in a
before block, or your
Separating Helper Functions).
Note that the
matchers table needs to do some work to fully install
be_again, and so checks that the assignment is the result
Matcher factory call. Trying to assign anthing else won’t
work - although nothing stops you from cloning the
prototype to set default fields and methods prior to assignment.
If you try to use
be_again as it stands, you’ll discover that it
doesn’t display the results from failed expectations as nicely as the
be matcher - missing the defining “exactly” text in the output.
To implement additional formatting around the
expected message, add
an implementation for the optional
format_expect method to the
Notice the use of
matchers.stringify to coerce the
parameter to a nicely formatted and quoted string.
less useful here than it is in the other formatting method slot,
Both of these methods are passed all of the arguments that are
generated in the code wrapped in
expect that eventually leads to
the custom matcher, though they are not useful in this particular
example, the full prototypes are:
specl.shell custom matchers use this feature if you want to see
an example of how it can be useful.
Usually, you’ll also need to provide nicely formatted messages when
any_of calls fail. Not surprisingly, to do that, you define another
method in the
When constructed without a specific
Matcher uses the default format, similarly to how
behaves with “, “ separators, except that the final separator is always
the string “ or “, and the individual entries are stringified first. If
you want to make use of that format in your own matchers, it is
concat. Again examples of this, and the
more complicated shell output formatter (
available in the source code, from
One final feature of the
Matcher constructor is that you can have it
enforce a particular type (or types) for the
actual parameter, by
actual_types to a list of acceptable types. For example,
the built in
contain matcher handles matching against both Lua string
types and Lua tables:
Valid values for this list include any of the core Lua types as
returned by the Lua
type function, but also any extended types
implemented as a table with a
type field, such as the
command objects defined by the
specl.shell extensions, or anything
else you care to build using the
specl.std.Object base type (such
Matcher factory object used throughout this section of the
Adding custom matcher with this API automatically handles lookups
to_ and inverting matchers with the
When you create a custom matcher, it can often improve the expressiveness of your spec files to allow additional custom adaptors that are specific to a particular Matcher object (and other Matchers cloned from it).
Matcher based object method named with a trailing question-mark
will be called automatically if that matcher is invoked with an
equivalent adaptor name. For example, the built in adaptors,
any_of are implemented as methods called
on the base
To add a custom adaptor to
be_again, we simply define the custom
adaptor method in the same way. For consistency with the built in
adaptors, I strongly recommend that you perform type checks against the
The utility function
type_check checks that the types of each element
of the table in argument 2 match one of the corresponding type names
from argument 3, or else raise an error for mismatched arguments using
the name given in argument 1. So the first call to
actual, the argument to
"expect", matches one of the
types listed in the object’s
actual_type field; and the next call
expected, the argument to
"the_same_size_as", is a
non-empty table. See the API documentation for more details of how to
To make this adaptor work properly with Specl, it must return a
boolean decribing whether the adaptor matched successfully, followed by
an error message that specl will use if the overall expectation failed
(which can happen even when we return
true, if the expectation uses
not_to_). Again, we use the
Matcher object’s format functions to
ensure that any specialisations of this particular object will continue
to behave properly with custom
format_ functions too.
There is nothing sacred about the built in matchers, so feel free to add
additional adaptors to the existing
Matcher objects too:
And then Specl will support expectations such as:
Some adaptors (such as the
any_of built in adaptor) need access to the
match function normally used by a plain matcher (i.e. without an
adaptor) to compare the result of the
expect call (
each of the alternatives in a table passed to the adaptor (
That function is stored as a matcher method that can be accessed from
the adaptor method with
It’s important that every example be evaluated from a clean slate, both to prevent the side effects of one example affecting the start conditions of another, and in order to focus on a given example without worrying what the earlier examples might have done when debugging a specification.
Specl achieves this by initialising a completely new environment in which to execute each example, then tearing it down afterwards to build another clean environment for executing the next example, and so on.
To keep examples as readable and concise as possible, it’s best not to have too much code in each. For example, it’s inefficient to repeat a few lines of set up and clean up around each expectation.
Much like RSpec, Specl supports the use of before and after
functions to isolate that repeated code. A
before is executed prior to
every example, just after the new environment is initialised, and
after is executed immediately after the example has
finished, just prior to tearing the environment down. Since we don’t
need any fancy long descriptions for
their table keys are just a bare
Note that, unlike normal Lua code, we don’t declare everything with
local scope, since the environment is reset before each example, so no
state leaks out. And, eliding all the redundant
local keywords makes
for more concise example code in the specification.
If you have used RSpec, you’ll already know that it supports
before(:all), and equivalents for
then goes to some lengths to warn that if you initialise any mutable
before(:all), then you’ve provided a way to let one
example leave side effects that could affect the behaviour of following
before is equivalent to RSpec’s
and it has no
before(:all) analogue (and likewise for
However, Specl does support nested contexts, which are mainly useful
for grouping, but also allow you to write a
before function outside of
a group, where it will behave as if it were a
before(:all) inside the
before placement aside, it’s always a good idea to organize
large spec files in example groups, and the best way to do that is with
a nested context (and write the description starting with the word
“context” rather than “describe” if you are a traditionalist!).
Oftentimes, spec files can become crowded with so much setup code that
the actual specifications can get lost in the noise. In this case, it
helps the clarity of the specification files, and the helper code too,
if you move as much of it as appropriate into a separate file, usually
Specl automatically loads the
spec_helper.lua file from the same
directory as the spec file being loaded. Thus, any global symbols set by
spec_helper.lua are available to all the spec files it shares a
Almost always, there is a
spec_helper.lua that sets up the Lua
package.cpath to the relative paths from the
top-level project directory so that you can run
specl directly from
the command line in that directory without needing a wrapper script or
make rules to set them on each invocation:
Furthermore, Specl automatically loads the
spec_helper.lua file in
the function environment of the outer-most
before block (a virtual
before is created at runtime if necessary), so it has access to all
of Specl’s Lua extensions, such as
expect and matchers. See
Programmatic Specifications for some
examples of how to take advantage of this.
As Specl executes examples and tests the expectations of a specification, it can displays its progress using a formatter.
Specl comes with two formatters already implemented, though you can write your own very easily if the format of the built in formatters doesn’t suit you.
The default formatter simply displays Specl’s progress by writing a
single period for every expectation that is met, or an
F instead if an
expectation is not met. Once all the expectations have been evaluated,
a one line summary follows:
In verbose mode (see Command Line), a longer description of any pending or failing examples is displayed, along with the file and line location of each.
The other built in formatter writes out the specification descriptions in an indented list in an easy to read format, followed by a slightly more detailed summary.
Failing and pending expectations are annotated inline, and again with more detail in the summary footer.
In verbose mode (see Command Line), the inline annotations are expanded to the more detailed summary format, and both inline and summary reports give the file and line number of each, making a particular example easy to find within a large spec-file.
A formatter is just a table of functions that Specl can call as it runs your specifications, so provided you supply the table keys that Specl is expecting, you can write your own formatters:
footer are called before any expectations,
and after all expectations, respectively.
stats argument to
footer is a table containing:
You can use this to print out statistics at the end of the formatted
output. Note that
starttime may be the result of an earlier call to
os.time (), or if
luaposix is installed where Specl can find it,
it will be the result of an earlier call to
In either case, you can pass it to
specl.util.timesince (earlier) to
turn it into a printable time elapsed since
earlier (with the best
accumulated argument to
footer is a string made by concatenating
all the returned strings, if any, from other calls to the formatter API
functions. This is useful, for example, to return failure reports from
expectations and then display a summary report from
footer, like the
built in formatters.
Instead of accumulating string returns and concatentating them into a
single long string to pass back into
footer, a table of named strings
can be returned by your
expectations functions, in which
case the accumulation of those keys is passed back to
example, if each call to
expectations returns a table with these two
footer will be passed a similar table, but with each entry being
the accumulation of every non-empty value returned with that key prior
footer being called. See the built in formatters for more
spec is called with a table of each of the descriptions
that the calling specification or context (the headers with descriptions
that typically begin with either
context) is nested
And finally, the function
expectations is called after each example
has been run, passing in a table with the format shown below, with
one expectation entry for each
expect call in that example, along with
a similar table of nested descriptions as were passed to
line fields hold the filename and line-number from
which the example being reported came.
ispending field will be set to
true if the entire example
is pending - that is, if it has no example code, save perhaps a call to
pending () function.
pending field in one of the
expectations elements is true, then
a call was made to
expect () from a pending example. The two are
necessary so that formatters can diagnose an unexpected
status == true
in a pending example, among other things.
The standard Specl formatters in the
directory of the sources show how these functions can be used to
display progress using an output format of your choice.
See the next section for details of how to get Specl to load your custom formatter.
Given a spec-file or two, along with the implementation of the code
being checked against those specifications, you run Specl inside the
project directory using the provided
specl command expects spec-files to be kept in a top-level
specs/, and to have names ending in
long as you follow that format, invoking
specl will find and check
all the matching spec-files automatically.
Often, after adding the new examples for a feature, you’re left with
a block of failing tests that scroll off the screen when the following
passing and failing examples are reported, even though you want to
work on the first failure first – because you’ve sensibly ordered your
examples with the fundamental features earlier than the later examples
that depend on those earlier ones. Use the
--fail-fast option to stop
checking examples as soon as the first failure has been reported.
Once you have accumulated a large collection of spec-files, you might
only need to check a selection of specs relevent to the files you are
working on. As long as you follow the best practice of putting specs
for, say, a source file named
foo/bar/baz.lua in a spec-file named
specs/foo/bar/baz_spec.yaml, you can list just the specs that are
named for the list of files you’re working on, like this:
The output will display results using the default
To use the
report formatter instead, add the
option to the command line above.
For finer grained selection of a subset of examples than by file,
Specl accepts any number of filters to match against the full nested
YAML path to each example, using the
Given the following spec-file:
The full name of an example is made by starting at the nearest top level
YAML description field, and concatenating all of the nested
descriptions that lead to the example itself, but leaving off the very
first word of each. For example, you can tell
specl to check the
first two examples, named
module group one passes and
one hasn't decided yet like this:
Specl will run all examples that match any one (or more) of the
-e) arguments you give it. Those arguments are
interpreted as Lua patterns, so you must be careful to escape any
pattern meta-characters with an additional
% (percent) character.
Other than that, each argument is matched against the concatenated
description path leading to each example with respect to pattern anchors
and the like, so you could include the final example in addition to the
first group selected above as follows:
When invoked with
display pending and failing examples with a
filename is the name of the spec file containing the non-passing
NN is the line-number of the first line of the non-passing
example in that file, and
EE is the ordinal expectation within that
example. If you need to recheck just that example, you can cut and
filename:NN:EE directly into your next
Actually, the final
:EE is always ignored, because there’s no way
for Specl to tell what parts of the Lua code in a given example
are relevant to one
expect statement or another, so it always checks
the entire example. You can omit the
:EE when you type at the command
If you want to check more than a single non-passing example, without
rechecking all of the specifications in a given file, Specl also
+ prefixed line numbers prior to the file name argument:
If you prefer to format the results of your specification examples with
a custom formatter, you should make sure your formatter is visible on
LUA_PATH, and use the
--formatter=BASENAME option to load it.
Note that, for security reasons, Specl removes the current directory from the system search path, so if you want to load a formatter in the current directory, you will need to explicitly re-enable loading Lua code from the current directory:
Otherwise you can load a formatter from the existing
name, including built in formatters, like this:
--help option for help and brief documentation on usage of the
remaining available options.
In addition to writing example code to specify the behaviour of Lua objects and modules, Specl is extremely useful for specifying the behaviour of other command-line programs. In fact, Specl has a growing collection of specifications for itself in the form of spec-files!
When a program can be executed from the shell, Specl provides the
specl.shell module for running the program with specified options,
arguments and standard input, and capturing the exit status and standard
output and error streams.
To use this module in your specifications you must require it, either in
spec_helper.lua file, or directly in the spec-file:
This function constructs a
Command object from it’s parameters and
then executes it using Lua’s
io.popen function. When called with a
string, that string is the shell command that will be executed by
io.popen returns only the process output end of the
command pipeline, so
spawn (ab)uses shell redirections and temporary
files to provide input, and collect output from the
Command. If you
try to call
spawn with a string that has its own redirections, that
might defeat the redirections added by
spawn, with unspecified results.
In practice, this doesn’t seem to be a real limitation. If it turns out
that this implementation has problems, then a future release of Specl
will use luaposix or alien for finer control over the
pipeline creation process.
spawn accepts a command specification table with the
command words in the array part of the table, with optional standard
stdin and environment variable settings in
spawn in this example first sets
- in the shell
environment of the
Command object, and primes standard input with the
foo, ultimately executing
cat -, which then copies standard
foo) to standard output.
The result of a successful execution of a
Command object is a
Process object that can be compared against expectations using any of
the following matchers.
expect does not contain a
Process object (either from a call to
spawn or otherwise), when using these matchers, Specl will raise an
error reporting what type was received instead.
This matcher succeeds when the
Process captured by
expect has the
given exit status:
Naturally, the expectation fails if the exit status does not match.
Often, specifying an exact exit status is not as clear as determining
Process exited normally or not:
This is exactly equivalent to:
This matcher fails if the exit status is non-zero.
Conversely, it’s often useful to determine that the
abnormally in an expectation:
This is entirely equivalent to:
This matcher fails only when the exit status is zero.
Of course, you won’t get very far when all you can specify is the exit
status of a
Process. This matcher is for specifying the entire
standard output a
Process should write throughout its execution:
output matcher works with
Process output in a similar
way to how the
be matcher works with raw strings, this
contain_output matcher is equivalent to
contain_output matcher works with
Process output in a
similar way to how the
contain matcher works with raw
match_output matcher is equivalent to
This matcher checks both that program exits normally, and that the
Process outputs exactly the given text:
succeed_with will fail if either the output differs, or
Process exits abnormally.
Similarly, you can check that the program exited normally, and that
Process contains the given text as a substring:
Process exited abnormally, or the given text is not found
anywhere in its output, then
You can also check that the program exited normally, and that the
Process output matches the given Lua pattern with
Although it is useful to write specifications that check for expected
Process output and exit status, you may have noticed that when those
matchers fail, they will also show the content of standard error (if
When you need to properly specify the content of standard error, use this matcher:
Note that the previous example is likely to fail
/bin/sh outputs a different error message
prefix, or if the error message text itself is slightly different on
the host machine, or on a system with internationalised messages etc.
contain_output, you can specify a
substring to find in standard error with:
Or similarly with Lua patterns using the
When you need to ensure that the
Process exited abnormally, in
addition to producing exactly the given error output:
Or specifying a substring in addition to the abnormal exit:
Or more precisely still, an abnormal exit and a standard error that matches the given Lua pattern:
At times, especially with large or complex specifications, it can feel as though YAML forces you to write a lot of boiler plate Lua over and over again across related examples. Actually, YAML was chosen as the file format for spec-files purely to help organise and separate examples as your specifications are fleshed out; but, when it starts to get in the way, you can usually move boiler plate code into a helper function and simply call that function each time you need to use it.
However, that doesn’t work when the boiler plate crosses examples:
What we really want to do here is skip the entire group on systems that
do not support that particular feature. We could write a wrapper
function to call
expect, and sometimes that’s a useful idiom; but
here, we can use a programmatic
examples inside the outer
group, and then wrap the whole example in an
if, like so:
As you can see, we can call
examples with a one element table using
the description string as a key, followed by a thunk for a value. A
programmatic specificiation written this way interacts with
formatters in precisely the same fashion as a separate
it description and example in that position would have done.
For even further simplification of repetitive boiler-plate code you
can write the examples calls in a function in your
file, which adds that same series of parameterized examples when called
from an example block in any spec-file:
No support for mocks in the current version.