Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .devcontainer/setup_env.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ set -euo pipefail
if [ ! -f test.env ]; then
cp test.env.sample test.env
fi

sudo apt-get update
sudo apt-get install -y libltdl7 libkrb5-3 libgssapi-krb5-2

export UV_VENV_CLEAR=1
uv venv .venv
uv pip install --python .venv/bin/python -r dev_requirements.txt
Expand Down
40 changes: 38 additions & 2 deletions .github/workflows/patched-integration-tests-sqlserver.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
sqlserver:
image: ghcr.io/${{ github.repository }}:server-${{ matrix.sqlserver_version }}
env:
ACCEPT_EULA: 'Y'
ACCEPT_EULA: "Y"
SA_PASSWORD: 5atyaNadella
DBT_TEST_USER_1: DBT_TEST_USER_1
DBT_TEST_USER_2: DBT_TEST_USER_2
Expand All @@ -45,4 +45,40 @@ jobs:
DBT_TEST_USER_1: DBT_TEST_USER_1
DBT_TEST_USER_2: DBT_TEST_USER_2
DBT_TEST_USER_3: DBT_TEST_USER_3
SQLSERVER_TEST_DRIVER: 'ODBC Driver ${{ matrix.msodbc_version }} for SQL Server'
SQLSERVER_TEST_DRIVER: "ODBC Driver ${{ matrix.msodbc_version }} for SQL Server"

integration-tests-sql-server-mssql-python:
name: mssql-python
runs-on: ubuntu-latest
permissions:
contents: read
packages: read
container:
image: ghcr.io/${{ github.repository }}:CI-3.13-msodbc18
credentials:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
services:
sqlserver:
image: ghcr.io/${{ github.repository }}:server-2022
env:
ACCEPT_EULA: "Y"
SA_PASSWORD: 5atyaNadella
DBT_TEST_USER_1: DBT_TEST_USER_1
DBT_TEST_USER_2: DBT_TEST_USER_2
DBT_TEST_USER_3: DBT_TEST_USER_3
COLLATION: SQL_Latin1_General_CP1_CS_AS
steps:
- uses: actions/checkout@v4

- name: Install dependencies
run: pip install -r dev_requirements.txt

- name: Run functional tests with mssql-python
run: pytest -ra -v tests/functional --profile "ci_sql_server"
env:
DBT_TEST_USER_1: DBT_TEST_USER_1
DBT_TEST_USER_2: DBT_TEST_USER_2
DBT_TEST_USER_3: DBT_TEST_USER_3
SQLSERVER_TEST_DRIVER: "ODBC Driver 18 for SQL Server"
SQLSERVER_TEST_USE_MSSQL_PYTHON: "True"
70 changes: 35 additions & 35 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
default_language_version:
python: python3
repos:
- repo: 'https://github.com/pre-commit/pre-commit-hooks'
- repo: "https://github.com/pre-commit/pre-commit-hooks"
rev: v4.6.0
hooks:
- id: check-yaml
args:
- '--unsafe'
- "--unsafe"
- id: check-json
- id: end-of-file-fixer
- id: trailing-whitespace
Expand All @@ -20,79 +20,79 @@ repos:
- id: fix-byte-order-marker
- id: mixed-line-ending
- id: check-docstring-first
- repo: 'https://github.com/adrienverge/yamllint'
- repo: "https://github.com/adrienverge/yamllint"
rev: v1.35.1
hooks:
- id: yamllint
args:
- '-d {extends: default, rules: {line-length: disable, document-start: disable}}'
- '-s'
- repo: 'https://github.com/MarcoGorelli/absolufy-imports'
- "-d {extends: default, rules: {line-length: disable, document-start: disable}}"
- "-s"
- repo: "https://github.com/MarcoGorelli/absolufy-imports"
rev: v0.3.1
hooks:
- id: absolufy-imports
- repo: 'https://github.com/hadialqattan/pycln'
- repo: "https://github.com/hadialqattan/pycln"
rev: v2.5.0
hooks:
- id: pycln
args:
- '--all'
- repo: 'https://github.com/pycqa/isort'
- "--all"
- repo: "https://github.com/pycqa/isort"
rev: 5.13.2
hooks:
- id: isort
args:
- '--profile'
- "--profile"
- black
- '--atomic'
- '--line-length'
- '99'
- '--python-version'
- '39'
- repo: 'https://github.com/psf/black'
- "--atomic"
- "--line-length"
- "99"
- "--python-version"
- "39"
- repo: "https://github.com/psf/black"
rev: 24.8.0
hooks:
- id: black
args:
- '--line-length=99'
- '--target-version=py39'
- "--line-length=99"
- "--target-version=py39"
- id: black
alias: black-check
stages:
- manual
args:
- '--line-length=99'
- '--target-version=py310'
- '--check'
- '--diff'
- repo: 'https://github.com/pycqa/flake8'
- "--line-length=99"
- "--target-version=py310"
- "--check"
- "--diff"
- repo: "https://github.com/pycqa/flake8"
rev: 7.1.1
hooks:
- id: flake8
args:
- '--max-line-length=99'
- "--max-line-length=99"
- id: flake8
args:
- '--max-line-length=99'
- "--max-line-length=99"
alias: flake8-check
stages:
- manual
- repo: 'https://github.com/pre-commit/mirrors-mypy'
- repo: "https://github.com/pre-commit/mirrors-mypy"
rev: v1.11.1
hooks:
- id: mypy
args:
- '--show-error-codes'
- '--ignore-missing-imports'
- '--explicit-package-bases'
files: '^dbt/adapters'
- "--show-error-codes"
- "--ignore-missing-imports"
- "--explicit-package-bases"
files: "^dbt/adapters"
- id: mypy
alias: mypy-check
stages:
- manual
args:
- '--show-error-codes'
- '--pretty'
- '--ignore-missing-imports'
- '--explicit-package-bases'
files: '^dbt/adapters'
- "--show-error-codes"
- "--pretty"
- "--ignore-missing-imports"
- "--explicit-package-bases"
files: "^dbt/adapters"
22 changes: 22 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,15 +32,35 @@ The functional tests require a running SQL Server instance. You can easily spin
make server
```

The default development flow uses the existing ODBC-based path. If you want to develop or test the optional `mssql-python` backend instead, make sure the package is installed in your environment before running tests.

```shell
pip install mssql-python
```

On Debian/Ubuntu-based environments, `mssql-python` may also require these system libraries:

```shell
sudo apt-get install -y libltdl7 libkrb5-3 libgssapi-krb5-2
```

This will use Docker Compose to spin up a local instance of SQL Server. Docker Compose is now bundled with Docker, so make sure to [install the latest version of Docker](https://docs.docker.com/get-docker/).

Next, tell our tests how they should connect to the local instance by creating a file called `test.env` in the root of the project.
You can use the provided `test.env.sample` as a base and if you started the server with `make server`, then this matches the instance running on your local machine.

If you are testing the optional `mssql-python` backend, also enable its profile flag in `test.env` so the adapter selects that implementation instead of the legacy driver-based one.

```shell
cp test.env.sample test.env
```

When using the optional `mssql-python` backend, update `test.env` with:

```shell
SQLSERVER_TEST_USE_MSSQL_PYTHON=True
```

You can tweak the contents of this file to test against a different database.

Note that we need 3 users to be able to run tests related to the grants.
Expand All @@ -57,6 +77,8 @@ make unit
make functional
```

This remains the documented test procedure for both connection backends. When the `mssql-python` flag is enabled, run the same commands after installing `mssql-python` and setting `SQLSERVER_TEST_USE_MSSQL_PYTHON=True` in `test.env`.

## CI/CD

We use Docker images that have all the things we need to test the adapter in the CI/CD workflows.
Expand Down
67 changes: 66 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,12 @@ Join us on the [dbt Slack](https://getdbt.slack.com/archives/CMRMDDQ9W) to ask q

## Installation

By default this adapter uses the Microsoft ODBC driver.

This adapter requires the Microsoft ODBC driver to be installed:
[Windows](https://docs.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver16#download-for-windows) |
[macOS](https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/install-microsoft-odbc-driver-sql-server-macos?view=sql-server-ver16) |
[Linux](https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver16)
[Linux](https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-sql-server?view=sql-server-ver16)

<details><summary>Debian/Ubuntu</summary>
<p>
Expand All @@ -45,6 +47,18 @@ Latest pre-release: ![GitHub tag (latest SemVer pre-release)](https://img.shield
pip install -U --pre dbt-sqlserver
```

### Optional: `mssql-python` backend

This adapter can also use the `mssql-python` driver behind a feature flag.

Install it explicitly when you want to use that backend:

```shell
pip install -U mssql-python
```

When this backend is enabled, the adapter does not require the ODBC driver-based connection path for that profile.

## Changelog

See [the changelog](CHANGELOG.md)
Expand All @@ -55,6 +69,8 @@ See [the changelog](CHANGELOG.md)

- `dbt_sqlserver_use_default_schema_concat`: *(default: `false`)* Controls schema name generation when a [custom schema](https://docs.getdbt.com/docs/build/custom-schemas) is set on a model.

- `use_mssql_python`: *(default: `false` in the profile)* Switches the connection backend from the legacy ODBC / `pyodbc` path to the `mssql-python` driver for that target profile.

| Flag value | `custom_schema_name` | Result |
|---|---|---|
| `false` (default, legacy) | *(none)* | `target.schema` |
Expand All @@ -74,6 +90,55 @@ See [the changelog](CHANGELOG.md)

> **Note:** If you want to permanently customise schema generation and avoid any future deprecation of this flag, override the `sqlserver__generate_schema_name` macro directly in your project.

### `mssql-python` feature flag usage

Enable the backend per target in your `profiles.yml`:

```yaml
your_profile:
target: dev
outputs:
dev:
type: sqlserver
host: your-server
port: 1433
database: your-database
schema: dbo
user: your-user
password: your-password
encrypt: true
trust_cert: false
use_mssql_python: true
```

#### Notes

- `use_mssql_python: true` is a profile-level feature flag.
- When enabled, the adapter uses `mssql-python` instead of the legacy `pyodbc` connection path.
- The legacy ODBC driver setting is only needed for profiles that continue to use the ODBC backend.
- If you enable `use_mssql_python`, make sure the `mssql-python` package is installed in the environment running dbt.
- This path is intended to fail fast when required dependencies or unsupported settings are missing.

#### Testing

For local development and validation, use the documented adapter workflow from `CONTRIBUTING.md`:

```shell
make dev
make server
cp test.env.sample test.env
make unit
make functional
```

To exercise the `mssql-python` backend in tests, configure the profile or environment so that the target under test sets:

```yaml
use_mssql_python: true
```

If you are testing in the devcontainer, ensure the `mssql-python` package is installed in that environment before running the unit or functional suite.



## Contributing
Expand Down
Loading
Loading