Table of Contents
- Sorbet user-facing design principles
- Quickstart
- Learning how Sorbet works
- Building Sorbet
- Running Sorbet
- Running the tests
- Testing Sorbet against pay-server
- Writing tests
- Debugging
- Writing docs
- Editor and environment
Quickstart
Install the dependencies
brew install bazel autoconf coreutils parallel
Clone this repository
git clone https://github.com/sorbet/sorbet.git
cd sorbet
Build Sorbet
./bazel build //main:sorbet --config=dbg
Run Sorbet!
bazel-bin/main/sorbet -e "42 + 'hello'"
Learning how Sorbet works
We’ve documented the internals of Sorbet in a separate doc. Cross-reference between that doc and here to learn how Sorbet works and how to change it!
There is also a talk online that describes Sorbet’s high-level architecture and the reasons why it’s fast:
Building Sorbet
There are multiple ways to build sorbet
. This one is the most common:
./bazel build //main:sorbet --config=dbg
This will build an executable in bazel-bin/main/sorbet
(see “Running Sorbet”
below). There are many options you can pass when building sorbet
:
--config=dbg
- Most common build config for development.
- Good stack traces, runs all ENFORCEs.
--config=sanitize
- Link in extra sanitizers, in particular: UBSan and ASan.
- Catches most memory and undefined-behavior errors.
- Substantially larger and slower binary.
--config=debugsymbols
- (Included by
--config=dbg
) debugging symbols, and nothing else.
- (Included by
--config=forcedebug
- Use more memory, but report even more sanity checks.
--config=static-libs
- Forcibly use static linking (Sorbet defaults to dynamic linking for faster build times).
- Sorbet already uses this option in release builds (see below).
--config=release-mac
and--config=release-linux
- Exact release configuration that we ship to our users.
Independently of providing or omitting any of the above flags, you can turn on optimizations for any build:
-c opt
- Enables
clang
optimizations (i.e.,-O2
)
- Enables
These args are not mutually exclusive. For example, a common pairing when debugging is
--config=dbg --config=sanitize
In .bazelrc
you can find out what all these options (and others) mean.
Install command line tools
xcode-select –install
Ensure that the system finds command line tools in an active Xcode directory
sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
Clear bazel’s cache, which may contain files generated from a previous
version of Xcode command line tools.
bazel clean –expunge
**(Mac) `fatal error: 'math.h' file not found`** (or some other system header)
This error can happen on Macs when the `/usr/include` folder is missing. The
solution is to install macOS headers via the following package:
macOS Mojave:
```shell
open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg
macOS Catalina:
sudo ln -s /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/* /usr/local/include/
Running Sorbet
Run Sorbet on an expression:
bazel-bin/main/sorbet -e "1 + false"
Run Sorbet on a file:
bazel-bin/main/sorbet foo.rb
Running bazel-bin/main/sorbet --help
will show lots of options. These are
the common ones for contributors:
-p <IR>
- Asks sorbet to print out any given intermediate representation.
- See
--help
for available values of<IR>
.
--stop-after <phase>
- Useful when there’s a bug in a later phase, and you want to quit early to debug.
-v
,-vv
,-vvv
- Show
logger
output (increasing verbosity)
- Show
--max-threads=1
- Useful for determining if you’re dealing with a concurrency bug or not.
--wait-for-dbg
- Will freeze Sorbet on startup and wait for a debugger to attach
- This is useful when you don’t have control over launching the process (LSP)
Running the tests
To run all the tests:
bazel test //... --config=dbg
(The //...
literally means “all targets”.)
To run a subset of the tests curated for faster iteration and development speed, run:
bazel test test --config=dbg
Note that in bazel terms, the second test is an alias for //test:test
, so we’re being a bit cute here.
By default, all test output goes into files. To also print it to the screen:
bazel test //... --config=dbg --test_output=errors
If any test failed, you will see two pieces of information printed:
1. //test:test_testdata/resolver/optional_constant
2. /private/var/tmp/.../test/test_testdata/resolver/optional_constant/test.log
- the test’s target (in case you want to run just this test again with
bazel test <target>
) - a (runnable) file containing the test’s output
To see the failing output, either:
- Re-run
bazel test
with the--test_output=errors
flag - Copy/paste the
*.log
file and run it (the output will open inless
)
Writing tests
We write tests by adding files to subfolders of the test/
directory.
Individual subfolders are “magic”; each contains specific types of tests.
We aspire to have our tests be fully reproducible.
C++ note: In C++, hash functions are only required to produce the same result for the same input within a single execution of a program.
Thus, we expect all user-visible outputs to be explicitly sorted using a key stable from one run to the next.
There are many ways to test Sorbet, some “better” than others. We’ve ordered them below in order from most preferable to least preferable. And we always prefer some tests to no tests!
test_corpus tests
The first kind of test can be called either test_corpus tests or testdata tests, based on the name of the test harness or the folder containing these tests, respectively.
To create a test_corpus test, add any file <name>.rb
to test/testdata
, in
any folder depth. The file must either:
- type check entirely, or
- throw errors only on lines marked with a comment (see below).
To mark that a line should have errors, append # error: <message>
(the
<message>
must match the raised error message). In case there are multiple
errors on this line, add an # error: <message>
on its own line just below.
Error checks can optionally point to a range of characters rather than a line:
1 + '' # error: `String` doesn't match `Integer`
rescue Foo, Bar => baz
# ^^^ error: Unable to resolve constant `Foo`
# ^^^ error: Unable to resolve constant `Bar`
You can run this test with:
bazel test //test:test_PosTests/testdata/path/to/<name>
Expectation tests
Each test_corpus test can be turned into an expectation test by optionally
creating any number of <name>.rb.<phase>.exp
files (where <name>
matches the
name of the ruby file for this test). These files contain pretty printed
representations of internal data structures, according to what would be printed
by -p <phase>
. The snapshot must exactly match the output generated by running
sorbet -p <phase> <name>.rb
for the test to pass.
You can run this test with:
bazel test //test:test_PosTests/testdata/path/to/<name>
Files that begin with a prefix and __
will be run together. For example,
foo__1.rb
and foo__2.rb
will be run together as test foo
. If such sets of
files have *.exp
files associated with them, the *.exp
files must instead
follow the pattern <name>.<phase>.exp
, where <name>
does not include the
__*.rb
suffix. So foo__1.rb
and foo__2.rb
would have an exp file like
foo.<pass>.exp
.
Another exception: for package-tree
exp tests, the filename is always
pass.package-tree.exp
, no matter the name of the test.
CLI tests
Any folder <name>
that is added to test/cli/
becomes a test.
This folder should have a file test.sh
that is executable.
When run, its output will be compared against test.out
in that folder.
Our bazel setup will produce two targets:
bazel run //test/cli:test_<name>
will execute the.sh
filebazel test //test/cli:test_<name>
will execute the.sh
and check it against what’s in the.out
file.
The scripts are run inside Bazel, so they will be executed from the top of the
workspace and have access to source files and built targets using their path
from the root. In particular, the compiled sorbet binary is available under
main/sorbet
.
LSP tests
Most LSP tests are expectation tests with additional LSP-specific annotations.
They are primarily contained in test/testdata/lsp
, but all files in test/testdata
are tested in LSP mode. You can run a test test/testdata/lsp/<name>.rb
like so:
bazel test //test:test_LSPTests/testdata/lsp/<name>
^ def: a 1
a = 20
Testing “Go to Type Definition”
This is somewhat similar to “Find Definition” above, but also slightly different because there’s no analogue of “Find All Type Definitions.”
class A; end
# ^ type-def: some-label
aaa = A.new
# ^ type: some-label
The type: some-label
assertion says “please simulate a Go to Type Definition
here, named some-label
” and the type-def: some-label
assertion says “assert
that the results for some-label
are exactly these locations.”
That means if the type definition could return multiple locs, the assertions will have to cover all results:
class A; end
# ^ type-def: AorB
class B; end
# ^ type-def: AorB
aaa = T.let(A.new, T.any(A, B))
# typed: false
class A; end
aaa = A.new
# ^ def: (nothing)
Testing hover
LSP tests can also assert the contents of hover responses with hover
assertions:
a = 10
# ^ hover: Integer(10)
If a location should report the empty string, use the special label (nothing)
:
a = 10
# ^ hover: (nothing)
Assert the contents of a specific line of the hover response with hover-line
assertions:
a = 10
# ^ hover-line: 1 Integer(10)
Testing completion
LSP tests can also assert the contents of completion responses with completion
assertions.
class A
def self.foo_1; end
def self.foo_2; end
foo
# ^ completion: foo_1, foo_2
end
The ^
corresponds to the position of the cursor. So in the above example, it’s
as if the cursor is like this: foo│
. If the ^
had been directly under the
last o
, it would have been like this: fo|o
. Only the first ^
is used. If
you use ^^^
in the assertion, the test harness will send a completion
assertion at the position of the first caret.
You can also write a test for a partial prefix of the completion results:
class A
def self.foo_1; end
def self.foo_2; end
foo
# ^ completion: foo_1, ...
end
Add the , ...
suffix to the end of a partial list of completion results, and
the test harness will ensure that the listed identifiers match a prefix of the
completion items. This prefix must still be listed in order.
If a location should report zero completion items, use the special message
(nothing)
:
class A
def self.foo_1; end
def self.foo_2; end
zzz
# ^ completion: (nothing)
end
To write a test for the snippet that would be inserted into the document if a particular completion item was selected, you can make two files:
# -- test/testdata/lsp/completion/mytest.rb --
class A
def self.foo_1; end
end
A.foo_
# ^ apply-completion: [A] item: 0
The apply-completion
assertion says “make sure the file mytest.A.rbedited
contains the result of inserting the completion snippet for the 0th completion
item into the file.”
# -- test/testdata/lsp/completion/mytest.A.rbedited --
class A
def self.foo_1; end
end
A.foo_1${0}
# ^ apply-completion: [A] item: 0
As you can see, the fancy ${...}
(tabstop placeholders) show up verbatim in
the output if they were sent in the completion response.
It’s not currently possible to test these parts of a completion response:
- completion kind
- documentation
- detail
For these, your best bet is to test manually in VS Code / your preferred editor and verify that you’re seeing your changes. For documentation specifically, nearly all the code paths are shared with hover, so you can alternatively write a hover test.
Testing workspace symbols (symbol search)
LSP tests can assert that a specific item appears in a symbol search (the
textDocument/workspaceSymbols
request) using the symbol-search
assertion:
class Project::Foo
# ^^^ symbol-search: "Foo"
end
The symbol-search
can optionally specify how that item should appear in
search results:
class Project::Foo
# ^^^ symbol-search: "Foo", name = "Foo", container = "Project"
end
In the above, container
can also be the special string "(nothing)"
to
indicate that the item has no container.
symbol-search
can also specify the item’s relative rank in the ordered
search results:
class Project::Foo
# ^^^ symbol-search: "Foo", rank = 1
end
Testing “Go to Implementation”
Testing the “Go to Implementation” feature is really similar to the testing techniques of the “Go to Type Definition”.
module A
# ^ find-implementation: A
extend T::Sig
extend T::Helpers
interface!
end
class B
#^^^^^^^ implementation: A
extend T::Sig
include A
# ^ find-implementation: A
end
There are two types of assertions:
find-implementation: <symbol>
means make a “Go to Implementation” request here.<symbol>
marks the symbol name we are looking for.implementation: <symbol>
marks the location which should be returned for the “Go to Implementation” call for a given<symbol>
If the request returns multiple locations, you should mark all of them with implementation: <symbol>
Testing rename constant
To write a test for renaming constants, you need to make at least two files:
# -- test/testdata/lsp/refactor/mytest.rb --
# typed: true
# frozen_string_literal: true
class Foo
class Foo
end
end
foo = Foo.new
# ^ apply-rename: [A] newName: Bar
The apply-rename
assertion here says “simulate a user starting a rename from
the position of this caret.” You’ll need to add an .rbedited
file that reflects
what the result of the changes should look like. In this case, the file would look
like this:
# -- test/testdata/lsp/refactor/mytest.A.rbedited --
# typed: true
# frozen_string_literal: true
class Bar
class Foo
end
end
foo = Bar.new
# ^ apply-rename: [A] newName: Bar
You can test that invalid renames aren’t applied by adding invalid: true
to your
test, like so:
# -- test/testdata/lsp/refactor/mytest.rb --
# typed: true
# frozen_string_literal: true
class Foo
class Foo
end
end
foo = Foo.new
# ^ apply-rename: [A] newName: foo invalid:true
To test for a specific error message, add an expectedErrorMessage
argument to the test:
# typed: true
# frozen_string_literal: true
require_relative './constant__class_definition.rb'
sig { params(foo: Foo::Foo).returns(Foo::Foo) }
def foo(foo); end
class Baz
#### Testing incremental type checking
In LSP mode, Sorbet runs file updates on a *fast path* or a *slow path*. It checks the structure of the
file before and after the update to determine if the change is covered under the fast path. If it is,
it performs further processing to determine the set of files that need to be type checked.
LSP tests can define file updates in `<name>.<version>.rbupdate` files which contain the contents of `<name>.rb`
after the update occurs. For example, the file `foo.1.rbupdate` contains the updated contents of `foo.rb`.
If the test contains multiple files by using a `__` suffixed prefix, then all rbupdates with the same version will
be applied in the same update. For example, `foo__bar.1.rbupdate` and `foo__baz.1.rbupdate` will be applied
simultaneously to update `foo__bar.rb` and `foo__baz.rb`.
Inside `*.rbupdate` files, you can assert that the slow path ran by adding a line with `# assert-slow-path: true`.
You can assert that the fast path ran on `foo__bar.rb` and `foo__baz.rb` with
`#assert-fast-path: foo__bar.rb,foo__baz.rb`.
Note that the default behavior when testing multi-file updates (e.g.,
`*__1.1.rbupdate` + `*__2.1.rbupdate`) is to include all the files in the file
update that is created and sent to the LSP server. When testing changes that
assert whether the right files were typechecked on the fast path with
`assert-fast-path`, you also likely want to declare which files **should not**
be included in the file edit, leaving Sorbet to figure out the subset of files
to be typechecked. **But** regardless of whether a file was included in the
update set, you likely want to assert that error occur at certain points inside
the file. For this, you can use `# exclude-from-file-update: true` inside an
`rbupdate` file. Note that when using this, the act of adding the
`exclude-from-file-update` assertion in the `rbupdate` will have the effect of
shifting all the `error` assertions off by one line compared to where the LSP
server will be reporting those errors. To work around this, you should leave a
spacer line in the previous file, so that the `exclude-from-file-update`
assertion replaces the spacer line, instead of being inserted into the file as a
completely new line. Search for `spacer` in some of the `fast_path` tests to see
an example.
To craft an update to an RBI file, use `.rbiupdate` instead of `.rbupdate`,
unless you mean to simulate the effect of converting an RBI file to an RB file.
### LSP recorded tests
It is possible to record an LSP session and use it as a test. We are attempting to move away from this form of
testing, as these tests are hard to update and understand. If at all possible, try to add your test case as a
regular LSP test.
Any folder `<name>` that is added to `test/lsp/` will become a test.
This folder should contain a file named `<folderName>.rec` that contains a
recorded LSP session.
- Lines that start with "Read:" will be sent to sorbet as input.
- Lines that start with "Write:" will be expected from sorbet as output.
### Updating tests
Frequently when a test is failing, it's because something inconsequential
changed in the captured output, rather than there being a bug in your code.
To recapture the traces, you can run
tools/scripts/update_exp_files.sh
You will probably want to look through the changes and `git checkout` any files
with changes that you believe are actually bugs in your code and fix your code.
`update_exp_files.sh` updates every snapshot file kind known to Sorbet. This can
be slow, depending on what needs to be recompiled and updated. Some faster
commands:
```bash
# Only update the `*.exp` files in `test/testdata`
tools/scripts/update_testdata_exp.sh
# Only update the `*.exp` files in `test/testdata/cfg`
tools/scripts/update_testdata_exp.sh test/testdata/cfg
# Only update a single exp file's test:
tools/scripts/update_testdata_exp.sh test/testdata/cfg/next.rb
# Only update the `*.out` files in `test/cli`
bazel test //test/cli:update
Debugging
In general,
- to debug a normal build of sorbet?
lldb bazel-bin/main/sorbet -- <args> ...
- (Consider using
--config=static-libs
for better debug symbols) - If you see weird Python errors on macOS, try
PATH=/usr/bin lldb
.
- to debug an existing Sorbet process (i.e., LSP)
- launch Sorbet with the
--wait-for-dbg
flag lldb -p <pid>
- set breakpoints and then
continue
- launch Sorbet with the
Also, it’s good to get in the practice of fixing bugs by first adding an
ENFORCE
(assertion) that would have caught the bug before actually fixing the
bug. It’s far easier to fix bugs when there’s a nice error message stating what
invariant you’ve violated. ENFORCE
s are free in the release build.
Writing docs
The sources for Sorbet’s documentation website live in the
website/
folder. Specifically, the docs live in
website/docs/
, are all authored with Markdown, and are built
using Docusaurus.
^ See here for how to work with the documentation site locally.
Editor and environment
The .bazelrc.local will live in the sorbet repo so it doesn’t interfere with
other bazel-based repos you have.
echo “build –disk_cache=\(HOME/.cache/sorbet/bazel-cache" >> ./.bazelrc.local echo "test --disk_cache=\)HOME/.cache/sorbet/bazel-cache” >> ./.bazelrc.local mkdir -p “$HOME/.cache/sorbet/bazel-cache”
### Multiple git worktrees
Sometimes it can be nice to have [multiple working trees] in Git. This allows
you to have multiple active checkouts Sorbet, sharing the same `.git/` folder.
To set up a new worktree with Sorbet:
```shell
tools/scripts/make_worktree.sh <worktree_name>
Shell
Many of the build commands are very long. You might consider shortening the common ones with shell aliases of your choice:
# mnemonic: 's' for sorbet
alias sb="bazel build //main:sorbet --config=dbg"
alias st="bazel test //... --config=dbg --test_output=errors"
Formatting files
We ensure that C++ files are formatted with clang-format
and that Bazel BUILD
files are formatted with buildifier
. To avoid inconsistencies between
different versions of these tools, we have scripts which download and run these
tools through bazel
:
tools/scripts/format_cxx.sh
tools/scripts/format_build_files.sh
CI will fail if there are any unformatted files, so you might want to set up your files to be formatted automatically with one of these options:
- Set up a pre-commit / pre-push hook which runs these scripts.
- Set up your editor to run these scripts. See below.