Skip to content

chore(test): Clean up testing guide and Makefile testing dependency checks #9611

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
matthewmcneely merged 3 commits into from
Mar 2, 2026
Merged

chore(test): Clean up testing guide and Makefile testing dependency checks #9611

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
31e6f7d
docs(testing): add make examples and remove completed improvements
mlwelles Feb 25, 2026
d2adb32
docs(testing): add three-column reference table for all test targets
mlwelles Feb 25, 2026
fdb73a7
chore(test): rename deps → check-deps and restructure dependency mana…
mlwelles Feb 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,16 +84,16 @@ uninstall: ## Uninstall dgraph binary
dgraph-installed:
$(MAKE) install

.PHONY: deps
deps: ## Check test dependencies (pass AUTO_INSTALL=true to auto-install missing ones)
$(MAKE) -C t deps
.PHONY: check-deps
check-deps: ## Check test dependencies (pass AUTO_INSTALL=true to auto-install missing ones)
$(MAKE) -C t check-deps

.PHONY: setup
setup: ## Install all test dependencies automatically
$(MAKE) deps AUTO_INSTALL=true
$(MAKE) check-deps AUTO_INSTALL=true

.PHONY: test
test: dgraph-installed local-image ## Run tests (default: integration + integration2)
test: check-deps dgraph-installed local-image ## Run tests (default: integration + integration2)
ifdef TAGS
@echo "Running tests with tags: $(TAGS)"
go test -v --tags="$(TAGS)" \
Expand Down Expand Up @@ -177,7 +177,7 @@ test-all: ## Every test: all t/ suites + integration2 + upgrade + fuzz
$(MAKE) test-fuzz

.PHONY: test-benchmark
test-benchmark: ## Go benchmarks (i.e. 'go test -bench')
test-benchmark: check-deps ## Go benchmarks (i.e. 'go test -bench')
go test -bench=. -benchmem $(if $(PKG),./$(PKG)/...,./...)

.PHONY: local-image
Expand Down
200 changes: 136 additions & 64 deletions TESTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,24 +142,28 @@ auto-install them:
make setup

# Check dependencies without installing (reports what's missing)
make deps
make check-deps

# Same as 'make deps' but auto-installs anything missing
make deps AUTO_INSTALL=true
# Same as 'make check-deps' but auto-installs anything missing
make check-deps AUTO_INSTALL=true
```

The check scripts validate:

- Go version (1.21+)
- Docker and Docker Compose versions and memory allocation
- Go version (matches go.mod requirement)
- Docker and Docker Compose versions
- Docker available memory (warns if < 8GB, with auto-fix on macOS)
- gotestsum installation
- ack installation
- Cross-compiler for non-Linux hosts (macOS)
- protoc installation (Linux only)
- Dgraph binary existence and correct architecture

### Required Tools

> **Note:** You don't need to install these manually. Running `make setup` from the repo root
> automatically installs missing dependencies. The commands below are listed for reference.
> **Note:** You do **not** need to install these manually. Running `make setup` or
> `make check-deps AUTO_INSTALL=true` from the repo root automatically checks and installs all
> missing dependencies. The commands below are listed only as reference for what gets installed.

#### 1. Go (1.21+)

Expand Down Expand Up @@ -273,6 +277,9 @@ Use `go test` to run one easy test on types package:

```bash
go test -v ./types/... -run TestConvert

# Or with make (runs all unit tests, not just one)
make test-unit
```

**Expected output:**
Expand All @@ -294,6 +301,9 @@ ok github.com/dgraph-io/dgraph/v25/types (cached)

```bash
cd t && go build . && ./t --test=TestGQLSchema

# Or with make
make test TEST=TestGQLSchema
```

If both pass, you're ready to run all test types!
Expand All @@ -304,28 +314,27 @@ If both pass, you're ready to run all test types!

### Using Make Targets

The simplest way to run tests:

```bash
# Run default tests (~30 min): integration suite + integration2
make test

# Run every test in the repo (all suites + all tag-based tests + fuzz)
make test-all

# Common shortcuts (run 'make help' for full list)
make test-unit # True unit tests only — no Docker, no build tags
make test-integration # Integration tests via t/ runner with Docker (SUITE=integration)
make test-integration-heavy # All heavy tests: systest-heavy + ldbc + load
make test-core # Core tests (i.e. 'make test SUITE=core')
make test-systest # All systest packages: systest-baseline + systest-heavy
make test-vector # Vector search tests (i.e. 'make test SUITE=vector')
make test-integration2 # Integration2 tests via dgraphtest (i.e. 'make test TAGS=integration2')
make test-upgrade # Upgrade tests (i.e. 'make test TAGS=upgrade')
make test-fuzz # Fuzz tests (i.e. 'make test FUZZ=1')
make test-all # Every test: all t/ suites + integration2 + upgrade + fuzz
make test-benchmark # Go benchmarks (i.e. 'go test -bench')
```
The simplest way to run tests is `make test` (default: `integration` suite + `integration2`). Each
`test-*` target is a shortcut for `make test` with specific arguments. The table below shows all
three ways to run each test type.

| Target | `make test` equivalent | Without make |
| ----------------------------- | ----------------------------------------- | ----------------------------------------------------------------------------- |
| `make test` | _(default)_ | `cd t && ./t --suite=integration` then `go test -v --tags=integration2 ./...` |
| `make test-unit` | `make test SUITE=unit` | `cd t && ./t --suite=unit` |
| `make test-integration` | `make test SUITE=integration` | `cd t && ./t --suite=integration` |
| `make test-core` | `make test SUITE=core` | `cd t && ./t --suite=core` |
| `make test-systest` | `make test SUITE=systest` | `cd t && ./t --suite=systest` |
| `make test-vector` | `make test SUITE=vector` | `cd t && ./t --suite=vector` |
| `make test-integration-heavy` | `make test SUITE=systest-heavy,ldbc,load` | `cd t && ./t --suite=systest-heavy,ldbc,load` |
| `make test-integration2` | `make test TAGS=integration2` | `go test -v --tags=integration2 ./...` |
| `make test-upgrade` | `make test TAGS=upgrade` | `go test -v --tags=upgrade ./...` |
| `make test-fuzz` | `make test FUZZ=1` | `go test -v -fuzz=Fuzz -fuzztime=300s ./dql/...` |
| `make test-benchmark` | _(no equivalent)_ | `go test -bench=. -benchmem ./...` |
| `make test-all` | _(no equivalent)_ | Runs `SUITE=all` + `integration2` + `upgrade` + fuzz sequentially |

> **Tip:** All targets accept `PKG=`, `TEST=`, and `TIMEOUT=` variables. For example:
> `make test-systest PKG=systest/plugin TEST=TestPasswordReturn TIMEOUT=60m`

Run `make help` to see all available targets, variables, and dynamically discovered SUITE/TAGS
values.
Expand Down Expand Up @@ -501,6 +510,19 @@ go test ./types/...
go test -v ./types/... -run TestConvert
```

**With make:**

```bash
# Run all unit tests (no Docker, no build tags)
make test-unit

# Run unit tests for a specific package
make test-unit PKG=types

# Run a specific unit test
make test-unit PKG=types TEST=TestConvert
```

### Identifying a unit test

- No `//go:build` tag at the top of the file = unit test
Expand Down Expand Up @@ -586,6 +608,19 @@ cd t && go build .
./t -r
```

**With make:**

```bash
# Run a suite
make test SUITE=core

# Run specific package
make test SUITE=integration PKG=systest/export

# Run single test
make test TEST=TestExportAndLoadJson
```

### Key Flags

| Flag | Description |
Expand Down Expand Up @@ -621,6 +656,16 @@ cd t && go build .
./t --pkg=systest/export --keep
```

**With make:**

```bash
# Run all tests in a package (make builds the runner automatically)
make test SUITE=integration PKG=systest/export

# Run single test
make test TEST=TestExportAndLoadJson
```

### Method 2: Manual Cluster + go test

For fine-grained control, manually start a cluster and run tests against it.
Expand Down Expand Up @@ -693,6 +738,19 @@ Using `t/` runner:
./t --suite=systest
```

**With make:**

```bash
# Run all systest packages
make test-systest

# Run specific systest package
make test SUITE=systest PKG=systest/export

# Run specific test by name
make test TEST=TestExportAndLoadJson
```

### Key Environment Variables

| Variable | Purpose | Set by |
Expand Down Expand Up @@ -738,6 +796,19 @@ go test -v --tags=integration2 ./systest/integration2/
go test -v --tags=integration2 --run '^TestName$' ./pkg/
```

**With make:**

```bash
# Run all integration2 tests
make test-integration2

# Run integration2 tests for a specific package
make test TAGS=integration2 PKG=systest/vector

# Run a specific integration2 test
make test TAGS=integration2 PKG=systest/vector TEST=TestVectorSearch
```

### Version & Binary Management

**Automatic version handling:**
Expand Down Expand Up @@ -948,6 +1019,19 @@ go test -v --tags=upgrade ./worker/
go test -v --tags=upgrade -run '^TestUpgradeName$' ./pkg/
```

**With make:**

```bash
# Run all upgrade tests
make test-upgrade

# Run upgrade tests for a specific package
make test TAGS=upgrade PKG=acl

# Run a specific upgrade test
make test TAGS=upgrade PKG=acl TEST=TestACL
```

### Where Upgrade Tests Live

| Package | Tests |
Expand Down Expand Up @@ -1222,6 +1306,19 @@ go test -v --tags=integration --run 'TestPluginTestSuite/TestPasswordReturn/subt
go test -v --tags=upgrade --run 'TestPluginTestSuite/TestPasswordReturn' ./systest/plugin/
```

**With make:**

```bash
# Run the plugin systest package via t/ runner
make test SUITE=systest PKG=systest/plugin

# Run a specific test
make test SUITE=systest PKG=systest/plugin TEST=TestPluginTestSuite/TestPasswordReturn

# Run in upgrade mode
make test TAGS=upgrade PKG=systest/plugin TEST=TestPluginTestSuite/TestPasswordReturn
```

**When NOT to use:**

- Simple one-off tests → use regular `func TestX(t *testing.T)`
Expand Down Expand Up @@ -1258,6 +1355,16 @@ go test -v ./dql -fuzz=Fuzz -fuzztime=5m
go test -v ./dql -fuzz=Fuzz -fuzztime=300s -fuzzminimizetime=120s
```

**With make:**

```bash
# Run all fuzz tests (default 300s per package)
make test-fuzz

# Fuzz a specific package with custom duration
make test FUZZ=1 PKG=dql FUZZTIME=5m
```

### CI Workflow

- `ci-dgraph-fuzz.yml` (runs on PRs)
Expand All @@ -1276,41 +1383,6 @@ go test -v ./dql -fuzz=Fuzz -fuzztime=300s -fuzzminimizetime=120s

## Future Improvement Ideas

### ✅ Completed Improvements

The following items from the original wishlist have been implemented:

- **✅ OS detection and automatic binary handling:** The Makefile now detects the host OS at runtime
and automatically builds the correct binaries. On macOS, `make install` builds both native and
Linux binaries without manual intervention.

- **✅ Automatic binary path management:** The `LINUX_GOBIN` environment variable is automatically
set based on OS. Docker Compose files use `${LINUX_GOBIN:-$GOPATH/bin}` to mount the correct
binary.

- **✅ No manual setup scripts required:** The `make test` target now depends on `dgraph-installed`
which automatically builds binaries if missing. Dependency checking scripts in `t/scripts/` can
auto-install missing tools with `AUTO_INSTALL=true`.

- **✅ Prerequisites handled automatically:** Running `make test` validates dependencies and builds
required binaries before running tests.

- **✅ Unified test interface:** A single `make test` entry point that accepts arguments to run any
test type (unit, integration, integration2, upgrade, fuzz) with environment variables for control.
The default (`make test` with no args) runs `integration` suite plus `integration2` for a fast
feedback loop (~30 min). Use `make test-all` to run every test.

- **✅ Example commands that "just work":** The following now work as expected:

```bash
make test SUITE=systest
make test FUZZ=1 PKG=dql
make test TAGS=upgrade PKG=acl
make test SUITE=systest PKG=systest/plugin
```

### Remaining Ideas

The following improvements could still enhance the developer experience:

- **Extend t/ runner:** Have the `t/` runner also handle unit and integration2 tests, providing a
Expand Down
Loading
Loading