Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion documentation/ErrorCodes.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
A list of `replica` error codes:

| Error Code | Meaning |
|:----------:|:--------|
| :---------: | :------ |
| 0 | Success |
| 1 to 127 | Number of failing tests (wrong expectation or test error) |
| 128 | More than 127 failing tests |
Expand Down
21 changes: 10 additions & 11 deletions documentation/TestSpecification.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,24 +19,23 @@ Here is the list of available fields:
<!-- markdownlint-disable MD013 MD033 -->
| Field name | JSON Type | Dhall Type | Mandatory | Default | Description |
| :--------- | --------- | ---------- | :-------: | ------- | ----------- |
| `description` | String | Optional Text | | Use when you display text info |
| `command` | String | Text | Yes | | The tested command. |
| `workingDir` | String | Optional Text | No | `.` | The directory where the test is executed |
| `description` | String | Optional Text | No | | Use when you display text info |
| `command` | String | Text | Yes | | The tested command. |
| `workingDir` | String | Optional Text | No | `.` | The directory where the test is executed |
| `beforeTest` | Array String | List Text | No | `[]` | A list of command to execute before the test. <br /> It is ran in a separated shell and thus you can't declare environment variables needed for the test here. |
| `afterTest` | Array String | List Text | No | `[]` | A list of command to execute after the test. <br /> It is ran in a separated shell and thus you can't access the environment variables declared in the test here. |
| `input` | String | Optional Text | No | | Text that is sent to the test command as standard input |
| `require` | Array String | List Text | No | `[]` | A list of tests that must succeed before this one can be triggered
| `require` | Array String | List Text | No | `[]` | A list of tests that must succeed before this one can be triggered |
| `tags` | Array String | List Text | No | `[]` | Used to classify tests |
| `pending` | Boolean | Bool | No | `False` | Pending tests won't be executed |
| `succeed` | Boolean | Optional Bool | No | | If set, REPLica will check the value returned by the command |
| `spaceSensitive` | Boolean | Bool | No | `True` | If set, the spaces are normalized before comparing the given and expected output: each chunk of space-like character are replaced by a single space and empty-lines are not considered |
| `stdOut` | Anything but an integer | Optional Expectation | No | True | set the expectation for `stdOut`, see [Expectation](#expectation) |
| `stdErr` | Anything but an integer | Optional Expectation | No | False | set the expectation for `stdErr`, see [Expectation](#expectation) |
| `files` | Object | Map Text Expectation | No | | List the files to check, and set the
corresponding expectation, see [Expectation](#expectation) |
| `files` | Object | Map Text Expectation | No | | List the files to check, and set the corresponding expectation, see [Expectation](#expectation) |
<!-- markdownlint-enable MD013 MD033 -->

The default value are infered in JSON.
The default value are inferred in JSON.
In Dhall, you need to use `Replica.Test` a schema that populate
a test record with the default values.

Expand All @@ -48,7 +47,7 @@ which respectively set the `succeed` value to `Some True` and `Some False`.

By default the behaviour of REPLica is to wait for a golden value
for `stdOut` to be saved
(generaly thanks to `replica run --interactive`)
(generally thanks to `replica run --interactive`)
and then to compare the output of the next runs with this _golden value_.

However, users may wants to inline their own expectations directly in the test.
Expand All @@ -58,7 +57,7 @@ The semantic of an `expectation` depends on the type of its value.

### JSON

There is three types of values that are supported for expactations:
There is three types of values that are supported for expectations:

- **Booleans.** If true, use a golden value.
If false, explicitly skip this source.
Expand All @@ -71,7 +70,7 @@ There is three types of values that are supported for expactations:
the result of the command must contain each member of the array.
If `spaceSensitive` is set to false, both the output and the
expectations are normalized before comparison.
- **An object**: allow the defitinion of several requirements that must all be satisfied.
- **An object**: allow the definition of several requirements that must all be satisfied.
The recognised fields are:

- `generated`: A boolean that indicates whether or not we use a golden value
Expand All @@ -85,7 +84,7 @@ There is three types of values that are supported for expactations:

### Dhall

The corresponding specification in dhall is the following:
The corresponding specification in Dhall is the following:

```dhall
let Replica.Expectation
Expand Down
126 changes: 37 additions & 89 deletions flake.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

16 changes: 4 additions & 12 deletions flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,16 @@
description = "Golden tests for command-line interfaces.";

inputs = {
flake-utils.url = github:numtide/flake-utils;
flake-utils.url = "github:numtide/flake-utils";
idris = {
url = "github:idris-lang/Idris2";
inputs.nixpkgs.follows = "nixpkgs";
inputs.flake-utils.follows = "flake-utils";
};
papers = {
url = "github:idris-lang/Idris2?dir=libs/papers";
flake = false;
};
pre-commit-hooks.url = "github:cachix/pre-commit-hooks.nix";
replicadhall.url = "github:ReplicaTest/replica-dhall";
};
outputs = { self, nixpkgs, idris, papers, flake-utils, pre-commit-hooks, replicadhall }:
outputs = { self, nixpkgs, idris, flake-utils, pre-commit-hooks, replicadhall }:
flake-utils.lib.eachDefaultSystem (system:
let
npkgs = import nixpkgs { inherit system; };
Expand All @@ -27,14 +23,12 @@
dhall-json;

version = import ./version.nix;
idrisPkgs = papers;

callPackage = lib.callPackageWith (npkgs // packages);

packages = {
inherit version;
buildIdris = idris.buildIdris.${system};
papersLib = callPackage ./nix/papersLib.nix { inherit papers; };
replica_dhall = replicadhall.packages.${system}.default;
buildReplica = callPackage ./nix/buildReplica.nix { };
replica = callPackage ./nix/replica.nix { };
Expand All @@ -43,9 +37,7 @@

inherit (packages)
replica
replicaTest
replica_dhall
papersLib;
replicaTest;

dockerImage = npkgs.dockerTools.buildImage {
name = "replica";
Expand Down Expand Up @@ -77,7 +69,7 @@
};

devShells.default = npkgs.mkShell {
packages = [ idris2 papersLib npkgs.rlwrap dhall dhall-json ];
packages = [ idris2 npkgs.rlwrap dhall dhall-json ];
shellHook = ''
alias idris2="rlwrap -s 1000 idris2 --no-banner"
${self.checks.${system}.pre-commit-check.shellHook}
Expand Down
4 changes: 2 additions & 2 deletions nix/buildReplica.nix
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ buildIdris, papersLib }:
{ buildIdris }:
buildIdris {
ipkgName = "replica";
src = ../.;
idrisLibraries = [ papersLib ];
idrisLibraries = [ ];
}
2 changes: 1 addition & 1 deletion nix/online-tests.nix
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,5 @@
make test RUN="-t online"
'';
pass_filenames = false;
stages = [ "push" ];
stages = [ "pre-push" ];
}
Loading