bulloak

A Solidity test generator based on the Branching Tree Technique.

• Installation
  • VSCode
• Usage
  • bulloak scaffold
  • bulloak check
    • Rules
  • Compiler Errors
• Trees
  • Terminology
  • Spec
• Output
• Contributing
• Publishing
• Supported By
• License

Warning

Note thatbulloakis still0.*.*,so breaking changes may occur at any time.If you must depend onbulloak,we recommend pinning to a specific version, i.e.,=0.y.z.

Installation

cargo install bulloak

VSCode

The following VSCode extensions are not essential but they are recommended for a better user experience:

• Solidity Inspector- syntax highlighting for.treefiles
• Ascii Tree Generator: convenient way to generate ASCII trees

Usage

bulloakimplements two commands:

• bulloak scaffold
• bulloak check

Scaffold Solidity Files

Say you have afoo.treefile with the following contents:

FooTest
└── When stuff is called // Comments are supported.
└── When a condition is met
└── It should revert.
└── Because we shouldn't allow it.

You can usebulloak scaffoldto generate a Solidity contract containing modifiers and tests that match the spec described infoo.tree.The following will be printed tostdout:

// $ bulloak scaffold foo.tree
// SPDX-License-Identifier: UNLICENSED
pragma solidity0.8.0;

contractFooTest{
modifierwhenStuffIsCalled() {
_;
}

functiontest_RevertWhen_AConditionIsMet()externalwhenStuffIsCalled {
// It should revert.
// Because we shouldn't allow it.
}
}

You can use the-woption to write the generated contracts to the file system. Say we have a bunch of.treefiles in the current working directory. If we run the following:

$ bulloak scaffold -w./**/*.tree

bulloakwill create a.t.solfile per.treefile and write the generated contents to it.

If a.t.solfile's title matches a.treein the same directory, then bulloakwill skip writing to that file. However, you may override this behaviour with the-fflag. This will forcebulloakto overwrite the contents of the file.

$ bulloak scaffold -wf./**/*.tree

Note all tests are showing as passing when their body is empty. To prevent this, you can use the-S(or--vm-skip) option to add avm.skip(true);at the beginning of each test function. This option will also add an import for forge-std'sTest.soland all test contracts will inherit from it.

You can skip emitting the modifiers by passing the-m(or--skip--modifiers) option. This way, the generated files will only include the test functions.

Check That Your Code And Spec Match

You can usebulloak checkto make sure that your Solidity files match your spec. For example, any missing tests will be reported to you.

Say you have the following spec:

HashPairTest
├── It should never revert.
├── When first arg is smaller than second arg
│ └── It should match the result of `keccak256(abi.encodePacked(a,b))`.
└── When first arg is bigger than second arg
└── It should match the result of `keccak256(abi.encodePacked(b,a))`.

And a matching Solidity file:

pragma solidity0.8.0;

contractHashPairTest{
functiontest_ShouldNeverRevert()external{
// It should never revert.
}

functiontest_WhenFirstArgIsSmallerThanSecondArg()external{
// It should match the result of `keccak256(abi.encodePacked(a,b))`.
}
}

This Solidity file is missing the tests for the branch When first arg is bigger than second arg,which would be reported after runningbulloak check tests/scaffold/basic.tree,like so:

warn: function "test_WhenFirstArgIsBiggerThanSecondArg" is missing in.sol
+ fix: run `bulloak check --fix tests/scaffold/basic.tree`
--> tests/scaffold/basic.tree:5

warn: 1 check failed (run `bulloak check --fix <.tree files>` to apply 1 fix)

As you can see in the above message,bulloakcan fix the issue automatically. If we run the command with the--stdoutflag, the output is:

-->tests/scaffold/basic.t.sol
pragma solidity0.8.0;

contractHashPairTest{
functiontest_ShouldNeverRevert()external{
// It should never revert.
}

functiontest_WhenFirstArgIsSmallerThanSecondArg()external{
// It should match the result of `keccak256(abi.encodePacked(a,b))`.
}

functiontest_WhenFirstArgIsBiggerThanSecondArg()external{
// It should match the result of `keccak256(abi.encodePacked(b,a))`.
}
}
<--

success:1issue fixed.

Running the command without the--stdoutflag will overwrite the contents of the solidity file with the fixes applied. Note that not all issues can be automatically fixed, and bulloak's output will reflect that.

warn: 13 checks failed (run `bulloak check --fix <.tree files>` to apply 11 fixes)

You can skip checking that the modifiers are present by passing the-m (or--skip--modifiers) option. This way,bulloakwill not warn when a modifier is missing from the generated file.

Rules

The following rules are currently implemented:

• A Solidity file matching the spec file must exist and be readable.
  • The spec and the Solidity file match if the difference between their names is only.tree&.t.sol.
• There is a contract in the Solidity file and its name matches the root node of the spec.
• Every construct, as it would be generated bybulloak scaffold,is present in the Solidity file.
• The order of every construct, as it would be generated bybulloak scaffold, matches the spec order.
  • Any valid Solidity construct is allowed and only constructs that would be generated bybulloak scaffoldare checked. This means that any number of extra functions, modifiers, etc. can be added to the file.

Compiler Errors

Another feature ofbulloakis reporting errors in your input trees.

For example, say you have a buggyfoo.treefile, which is missing a└ character. Runningbulloak scaffold foo.treewould report the error like this:

•••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••
bulloak error: unexpected `when` keyword

── when the id references a null stream
^^^^

--- (line 2, column 4) ---
file: foo.tree

Trees

bulloak scaffoldscaffolds Solidity test files based on.treespecifications that follow the Branching Tree Technique.

Currently, there is on-going discussionon how to handle different edge-cases to better empower the Solidity community. This section is a description of the current implementation of the compiler.

Terminology

• Condition:when/givenbranches of a tree.
• Action:itbranches of a tree.
• Action Description:Children of an action.

Spec

Eachtreefile should describe at least one function under test. Trees follow these rules:

• The first line is the root tree identifier, composed of the contract and function names which should be delimited by a double colon.
• bulloakexpects you to use├and└characters to denote branches.
• If a branch starts with eitherwhenorgiven,it is a condition.
  • whenandgivenare interchangeable.
• If a branch starts withit,it is an action.
  • Any child branch an action has is called an action description.
• Keywords are case-insensitive:itis the same asItandIT.
• Anything starting with a//is a comment and will be stripped from the output.
• Multiple trees can be defined in the same file to describe different functions by following the same rules, separating them with two newlines.

Take the following Solidity function:

functionhashPair(bytes32a,bytes32b)privatepurereturns(bytes32) {
returna<b?hash(a, b):hash(b, a);
}

A reasonable spec for the above function would be:

HashPairTest
├── It should never revert.
├── When first arg is smaller than second arg
│ └── It should match the result of `keccak256(abi.encodePacked(a,b))`.
└── When first arg is bigger than second arg
└── It should match the result of `keccak256(abi.encodePacked(b,a))`.

There is a top-level action that will generate a test to check the function invariant that it should never revert.

Then, we have the two possible preconditions:a < banda >= b.Both branches end in an action that will makebulloak scaffoldgenerate the respective test.

Note the following things:

• Actions are written with ending dots but conditions are not. This is because actions support any character, but conditions don't. Since conditions are transformed into modifiers, they have to be valid Solidity identifiers.
• You can have top-level actions without conditions. Currently,bulloakalso supports actions with sibling conditions, but this might get removed in a future version per this discussion.
• The root of the tree will be emitted as the name of the test contract.

Suppose you have additional Solidity functions that you want to test in the same test contract, sayUtilswithinutils.t.sol:

functionmin(uint256a,uint256b)privatepurereturns(uint256) {
returna<b?a:b;
}

functionmax(uint256a,uint256b)privatepurereturns(uint256) {
returna>b?a:b;
}

The full spec for all the above functions would be:

Utils::hashPair
├── It should never revert.
├── When first arg is smaller than second arg
│ └── It should match the result of `keccak256(abi.encodePacked(a,b))`.
└── When first arg is bigger than second arg
└── It should match the result of `keccak256(abi.encodePacked(b,a))`.


Utils::min
├── It should never revert.
├── When first arg is smaller than second arg
│ └── It should match the value of `a`.
└── When first arg is bigger than second arg
└── It should match the value of `b`.


Utils::max
├── It should never revert.
├── When first arg is smaller than second arg
│ └── It should match the value of `b`.
└── When first arg is bigger than second arg
└── It should match the value of `a`.

Note the following things:

• Contract identifiers must be present in all roots.
• Contract identifiers that are missing from subsequent trees, or otherwise mismatched from the first tree root identifier, will causebulloakto error. This violation is not currently fixable withbulloak check --fixso will need to be manually corrected.
• Duplicate conditions between separate trees will be deduplicated when transformed into Solidity modifiers.
• The function part of the root identifier for each tree will be emitted as part of the name of the Solidity test (e.g.test_MinShouldNeverRevert).

Output

There are a few things to keep in mind about the scaffolded Solidity test:

• The contract filename is the same as the.treebut with a.t.sol extension. E.g.test.treewould correspond totest.t.sol.
• Tests are emitted in the order their corresponding actions appear in the .treefile.
• We generate one modifier per condition, except for leaf condition nodes.
• Test names follow Foundry's best practices.

Examples

You can find practical examples of using BTT here:

• BTT Examples by Paul Berg
• CreateX

Contributing

Please refer toCONTRIBUTING.md.

Publishing

These are the current steps taken to publishbulloak:

• Bump the version field inCargo.toml.
• Update theCHANGELOG.mdfile with git cliff -o CHANGELOG.md.This step includes setting the proper header for the latest tag.
• Commit the changes.
• Runcargo publish --dry-runto make sure that everything looks good.
• Create the corresponding git tag named after the version.
• Push to origin.
• Runcargo publish.

Supported By

This project has been possible thanks to the support of:

• Sense Finance
• Sablier

License

This project is licensed under either of:

• Apache License, Version 2.0, (LICENSE-APACHEor https://www.apache.org/licenses/LICENSE-2.0).
• MIT license (LICENSE-MITor https://opensource.org/licenses/MIT).