Test files within Roblox

[Anaminus]

Options:

• Use rbxmk to build a place file containing each test.
  • Depends on rbxfile, a differing implementation.
  • Reencoding will invalidate certain tests.
• Stream objects from a server via HttpService.
  • Cannot deserialize received objects.
• Copy tests into local content folder, load with game:GetObjects(). (WINNER)
  • Combine with HttpService and server for automated testing.

[Anaminus]

General

Units

Each test file may contain a number of units. A unit is an instance within the file that tests a particular behavior. Not all instances are units, so a way is needed to indicate when an instance is a unit.

Add #+unit directive. Each use of +unit specifies a path to an instance within the file, and declares that the instance is a unit that must exist. For example, if the file contained a unit Part within a Model within the Workspace, then the directive would be #+unit: Workspace.Model.Part. This selects the first match, though units should be unique anyway. Ancestors are automatically expected, and cannot be unexpected.

Also add #-unit directive. This is like +unit, but rather than expecting an instance, the lack of instance is expected. Ancestors are automatically expected, and cannot be unexpected.

Server

Locate current install

The server must be able to locate the current installation of Roblox Studio, in order to find the content folder.

Synchronize tests

When requested, copy data directory to {studio}/content/tests. Ideally, this would work like rsync -a --delete, but a simple delete-then-copy is probably good enough for now.

Request endpoints

The host is localhost (127.0.0.1) on port 4866.

http://127.0.0.1:4866

_{Why 4866?}

("t"):byte()*1 +
("e"):byte()*2 +
("s"):byte()*3 +
("t"):byte()*4 +
("f"):byte()*5 +
("i"):byte()*6 +
("l"):byte()*7 +
("e"):byte()*8 +
("s"):byte()*9

GET /ping

Ping the server to see if it's running.

Response	Description
OK	Always. Failure indicated by lack of response.

GET /sync

Cause the server to copy tests to the content directory.

Response	Description
OK	On success.
ERROR	On failure.

GET /list

Receive a list of test files and units from the server.

Response	Description
LIST	On success.
ERROR	On failure.

POST /diff?file=path/to/test.rbxmx

The plugin sends a golden file as the body, and asks the server to compare it with the golden file corresponding to the given test file. This avoids having to implement a differ in Lua.

Response	Description
DIFF	On success.
ERROR	On failure.

Response format

The server responds to a plugin request using the X-Type header.

X-Type: OK

Generic indicator that a request succeeded. No body.

X-Type: ERROR

Indicates that an error occurred. The body is the error message.

X-Type: LIST

The response to a List request. The body is JSON:

[
	{
		"Path": "path/to/test.rbxmx",
		"Units": [
			{
				"Expected": true,
				"Path": ["ExpectedUnit"]
			},
			{
				"Expected": false,
				"Path": ["Path", "To", "UnexpectedUnit"]
			}
		]
	}
]

X-Type: DIFF

The response to a Diff request. The body is the diff result. Empty indicates equality.

Plugin

The plugin creates a button that toggles a window. The window contains various commands that communicate with the server to perform actions. If the server is ever unavailable, then the plugin cancels the command and notifies the user.

When the window is opened, the plugin pings the server to verify that it is running. Until success, it periodically retries while the window is open.

Golden files

Must implement a golden file generator in Lua.

Display

The plugin displays each test in a table. The first column contains test files and units, structured as a tree.

The second column displays the results of each test. Coloring is red/green/gray (or blue/yellow/gray) for failure/success/undetermined. If a test fails, a message explaining why can be displayed here.

Commands

Update

Display the tests that are available.

1. /list
2. Display with undetermined results.

Run

Run the tests.

1. /sync
2. /list
3. For each test file, run game:GetObjects("rbxasset://tests/$PATH").
4. If an error occurs, mark the whole file as failed.
   • Check if GetObjects will throw the error; we may have to monitor LogService instead.
5. Traverse through each unit. Mark any unexpected units as failed.
6. Generate a golden file.
7. For each test, request /diff. Mark tests with non-empty results as failed.
8. Update display to reflect results.

Settings

• Halt on failure: Stop running tests on the first failure. Defaults to false.