SARIF reporter

SARIF is a unified file format used to exchange information between
static analysis tools (like pylint) and various types of formatters,
meta-runners, broadcasters / alert system, ...

This implementation is ad-hoc, and non-validating.

Spec v Github
-------------

Turns out Github both doesn't implement all of SARIF (which makes
sense) and requires a bunch of properties which the spec considers
optional. The [official SARIF validator][] (linked to by both oasis
and github) was used to validate the output of the reporter, ensuring
that all the github requirements it flags are fulfilled, and
fixing *some* of the validator's pet issues.

As of now the following issues are left unaddressed:

- azure requires `run.automationDetails`, looking at the spec I don't
  think it makes sense for the reporter to inject that, it's more up
  to the CI
- the validator wants a `run.versionControlProvenance`, same as above
- the validator wants rule names in PascalCase, lol
- the validator wants templated result messages, but without pylint
  providing the args as part of the `Message` that's a bit of a chore
- the validator wants `region` to include a snippet (the flagged
  content)
- the validator wants `physicalLocation` to have a `contextRegion`
  (most likely with a snippet)

On URIs
-------

The reporter makes use of URIs for artifacts (~files). Per ["guidance
on the use of artifactLocation objects"][3.4.7], `uri` *should*
capture the deterministic part of the artifact location and
`uriBaseId` *should* capture the non-deterministic part. However as
far as I can tell pylint has no requirement (and no clean way to
require) consistent resolution roots: `path` is just relative to the
cwd, and there is no requirement to have project-level files to use
pylint. This makes the use of relative uris dodgy, but absolute uris
are pretty much always broken for the purpose of *interchange* so
they're not really any better.

As a side-note, Github [asserts][relative-uri-guidance]

> While this [nb: `originalUriBaseIds`] is not required by GitHub for
> the code scanning results to be displayed correctly, it is required
> to produce a valid SARIF output when using relative URI references.

However per [3.4.4][] this is incorrect, the `uriBaseId` can be
resolved through end-user configuration, `originalUriBaseIds`,
external information (e.g. envvars), or heuristics.

It would be nice to document the "relative root" via
`originalUriBaseIds` (which may be omitted for that purpose per
[3.14.14][], but per the above claiming a consistent project root is
dodgy.

We *could* resolve known project files (e.g. pyproject.toml, tox.ini,
etc...) in order to find a consistent root (project root, repo root,
...) and set / use that for relative URIs but that's a lot of
additional complexity which I'm not sure is warranted at least for a
first version.

Fixes #5493

[3.4.4]: https://docs.oasis-open.org/sarif/sarif/v2.1.0/csprd01/sarif-v2.1.0-csprd01.html#_Toc10540869
[3.4.7]:
https://docs.oasis-open.org/sarif/sarif/v2.1.0/csprd01/sarif-v2.1.0-csprd01.html#_Toc10540872
[3.14.14]: https://docs.oasis-open.org/sarif/sarif/v2.1.0/csprd01/sarif-v2.1.0-csprd01.html#_Toc10540936
[relative-uri-guidance]:
https://docs.github.com/en/code-security/code-scanning/integrating-with-code-scanning/sarif-support-for-code-scanning#relative-uri-guidance-for-sarif-producers
[official SARIF validator]: https://sarifweb.azurewebsites.net/

Author: xmo-odoo

pylint/lint/base_options.py

@@ -104,7 +104,8 @@ def _make_linter_options(linter: PyLinter) -> Options:
                 "group": "Reports",
                 "help": "Set the output format. Available formats are: 'text', "
                 "'parseable', 'colorized', 'json2' (improved json format), 'json' "
-                "(old json format), msvs (visual studio) and 'github' (GitHub actions). "
+                "(old json format), msvs (visual studio), 'github' (GitHub actions), "
+                "and 'sarif'. "
                 "You can also give a reporter class, e.g. mypackage.mymodule."
                 "MyReporterClass.",
                 "kwargs": {"linter": linter},

pylint/reporters/json_reporter.py

@@ -7,16 +7,22 @@
 from __future__ import annotations
 
 import json
-from typing import TYPE_CHECKING, TypedDict
+import sys
+from textwrap import shorten
+from typing import TYPE_CHECKING, TypedDict, TextIO
 
+from pylint.constants import MSG_TYPES
+
+import pylint
 from pylint.interfaces import CONFIDENCE_MAP, UNDEFINED
 from pylint.message import Message
 from pylint.reporters.base_reporter import BaseReporter
-from pylint.typing import MessageLocationTuple
+from pylint.typing import MessageLocationTuple, MessageTypesFullName
 
 if TYPE_CHECKING:
     from pylint.lint.pylinter import PyLinter
     from pylint.reporters.ureports.nodes import Section
+    from pylint.reporters import sarif_types
 
 # Since message-id is an invalid name we need to use the alternative syntax
 OldJsonExport = TypedDict(
@@ -196,6 +202,106 @@ def serialize_stats(self) -> dict[str, str | int | dict[str, int]]:
         }
 
 
+class SARIFReporter(BaseReporter):
+    name = "sarif"
+    extension = "sarif"
+    linter: PyLinter
+
+    def display_reports(self, layout: Section) -> None:
+        """Don't do anything in this reporter."""
+
+    def _display(self, layout: Section) -> None:
+        """Do nothing."""
+
+    def display_messages(self, layout: Section | None) -> None:
+        """Launch layouts display."""
+        output: sarif_types.Log = {
+            "version": "2.1.0",
+            "$schema": "https://docs.oasis-open.org/sarif/sarif/v2.1.0/cs01/schemas/sarif-schema-2.1.0.json",
+            "runs": [{
+                "tool": {
+                    "driver": {
+                        "name": "pylint",
+                        "fullName": f"pylint {pylint.__version__}",
+                        "version": pylint.__version__,
+                        # should be versioned but not all versions are kept so...
+                        "informationUri": "https://pylint.readthedocs.io/",
+                        "rules": [
+                            {
+                                "id": m.msgid,
+                                "name": m.symbol,
+                                "deprecatedIds": [msgid for msgid, _ in m.old_names],
+                                "deprecatedNames": [name for _, name in m.old_names],
+                                # per 3.19.19 shortDescription should be a
+                                # single sentence which can't be guaranteed,
+                                # however github requires it...
+                                "shortDescription": {'text': m.description.split(".", 1)[0]},
+                                # github requires that this is less than 1024 characters
+                                "fullDescription": {'text': shorten(m.description, 1024, placeholder="...")},
+                                "help": {"text": m.format_help()},
+                                "helpUri": f"https://pylint.readthedocs.io/en/stable/user_guide/messages/{MSG_TYPES[m.msgid[0]]}/{m.symbol}.html"
+                                # handle_message only gets the formatted message,
+                                # so to use `messageStrings` we'd need to
+                                # convert the templating and extract the args
+                                # out of the msg
+                            }
+                            for checker in self.linter.get_checkers()
+                            for m in checker.messages
+                            if m.symbol in self.linter.stats.by_msg
+                        ]
+                    }
+                },
+                "results": [self.serialize(message) for message in self.messages],
+            }]
+        }
+        json.dump(output, self.out)
+
+    @staticmethod
+    def serialize(message: Message) -> sarif_types.Result:
+        region: sarif_types.Region = {
+            "startLine": message.line,
+            "startColumn": message.column + 1,
+            "endLine": message.end_line or message.line,
+            "endColumn": (message.end_column or message.column) + 1,
+        }
+
+        location: sarif_types.Location = {
+            "physicalLocation": {
+                "artifactLocation": {
+                    "uri": message.path.replace('\\', '/'),
+                },
+                "region": region,
+            },
+        }
+        if message.obj:
+            logical_location: sarif_types.LogicalLocation = {
+                "name": message.obj,
+                "fullyQualifiedName": f"{message.module}.{message.obj}",
+            }
+            location["logicalLocations"] = [logical_location]
+
+        return {
+            "ruleId": message.msg_id,
+            "message": {"text": message.msg},
+            "level": CATEGORY_MAP[message.category],
+            "locations": [location],
+            "partialFingerprints": {
+                # encoding the node path seems like it would be useful to dedup alerts?
+                "nodePath/v1": "",
+            }
+        }
+
+CATEGORY_MAP: dict[str, sarif_types.ResultLevel] = {
+    "convention": "note",
+    "refactor": "note",
+    "statement": "note",
+    "info": "note",
+    "warning": "warning",
+    "error": "error",
+    "fatal": "error",
+}
+
 def register(linter: PyLinter) -> None:
     linter.register_reporter(JSONReporter)
     linter.register_reporter(JSON2Reporter)
+    linter.register_reporter(SARIFReporter)

pylint/reporters/sarif_types.py

@@ -0,0 +1,128 @@
+"""
+SARIF is a pretty sprawling and deeply nested format, which makes it quite
+verbose to express in a relatively classic type system like Python's.
+
+As such, this module provides the subset of the SARIF schema necessary for
+pylint's output, translated to Python types.
+"""
+from __future__ import annotations
+
+from typing import TypedDict, Literal
+
+
+class Run(TypedDict):
+    tool: Tool
+    # invocation parameters / environment for the tool
+    # invocation: list[Invocations]
+    results: list[Result]
+    # originalUriBaseIds: dict[str, ArtifactLocation]
+
+
+Log = TypedDict(
+    "Log",
+    {
+        "version": Literal["2.1.0"],
+        "$schema": Literal["https://docs.oasis-open.org/sarif/sarif/v2.1.0/cs01/schemas/sarif-schema-2.1.0.json"],
+        "runs": list[Run],
+    }
+)
+
+
+class Tool(TypedDict):
+    driver: Driver
+
+
+class Driver(TypedDict):
+    name: Literal["pylint"]
+    # optional but azure wants it
+    fullName: str
+    version: str
+    informationUri: str  # not required but validator wants it
+    rules: list[ReportingDescriptor]
+
+
+class ReportingDescriptorOpt(TypedDict, total=False):
+    deprecatedIds: list[str]
+    deprecatedNames: list[str]
+    messageStrings: dict[str, MessageString]
+
+
+class ReportingDescriptor(ReportingDescriptorOpt):
+    id: str
+    # optional but validator really wants it (then complains that it's not pascal cased)
+    name: str
+    # not required per spec but required by github
+    shortDescription: MessageString
+    fullDescription: MessageString
+    help: MessageString
+    helpUri: str
+
+
+class MarkdownMessageString(TypedDict, total=False):
+    markdown: str
+
+
+class MessageString(MarkdownMessageString):
+    text: str
+
+
+ResultLevel = Literal["none", "note", "warning", "error"]
+
+
+class ResultOpt(TypedDict, total=False):
+    ruleId: str
+    ruleIndex: int
+
+    level: ResultLevel
+
+
+class Result(ResultOpt):
+    message: Message
+    # not required per spec but required by github
+    locations: list[Location]
+    partialFingerprints: dict[str, str]
+
+
+class Message(TypedDict, total=False):
+    # needs to have either text or id but it's a PITA to type
+
+    #: plain text message string (can have markdown links but no other formatting)
+    text: str
+    #: formatted GFM text
+    markdown: str
+    #: rule id
+    id: str
+    #: arguments for templated rule messages
+    arguments: list[str]
+
+
+class Location(TypedDict, total=False):
+    physicalLocation: PhysicalLocation  # actually required by github
+    logicalLocations: list[LogicalLocation]
+
+
+class PhysicalLocation(TypedDict):
+    artifactLocation: ArtifactLocation
+    # not required per spec, required by github
+    region: Region
+
+class ArtifactLocation(TypedDict, total=False):
+    uri: str
+    #: id of base URI for resolving relative `uri`
+    uriBaseId: str
+    description: Message
+
+
+class LogicalLocation(TypedDict, total=False):
+    name: str
+    fullyQualifiedName: str
+    #: schema is `str` with a bunch of *suggested* terms, of which this is a subset
+    kind: Literal['function', 'member', 'module', 'parameter', 'returnType', 'type', 'variable']
+
+
+class Region(TypedDict):
+    # none required per spec, all required by github
+    startLine: int
+    startColumn: int
+    endLine: int
+    endColumn: int

tests/reporters/unittest_json_reporter.py

@@ -13,11 +13,11 @@
 
 import pytest
 
-from pylint import checkers
+from pylint import checkers, __version__
 from pylint.interfaces import HIGH, UNDEFINED
 from pylint.lint import PyLinter
 from pylint.message import Message
-from pylint.reporters.json_reporter import JSON2Reporter, JSONReporter
+from pylint.reporters.json_reporter import JSON2Reporter, JSONReporter, SARIFReporter
 from pylint.reporters.ureports.nodes import EvaluationSection
 from pylint.typing import MessageLocationTuple
 
@@ -263,3 +263,63 @@ def test_json2_result_with_broken_score() -> None:
     reporter.display_messages(None)
     report_result = json.loads(output.getvalue())
     assert "division by zero" in report_result["statistics"]["score"]
+
+def test_simple_sarif():
+    output = StringIO()
+    reporter = SARIFReporter(output)
+    linter = PyLinter(reporter=reporter)
+    checkers.initialize(linter)
+    linter.config.persistent = 0
+    linter.open()
+    linter.set_current_module("0123")
+    linter.add_message(
+        "line-too-long",
+        line=1,
+        args=(1, 2),
+        end_lineno=1,
+        end_col_offset=4
+    )
+    reporter.display_messages(None)
+    assert json.loads(output.getvalue()) == {
+        "version": "2.1.0",
+        "$schema": "https://docs.oasis-open.org/sarif/sarif/v2.1.0/cs01/schemas/sarif-schema-2.1.0.json",
+        "runs": [{
+            "tool": {
+                "driver": {
+                    "name": "pylint",
+                    "fullName": f"pylint {__version__}",
+                    "version": __version__,
+                    "informationUri": "https://pylint.readthedocs.io/",
+                    "rules": [{
+                        "id": "C0301",
+                        "deprecatedIds": [],
+                        "name": "line-too-long",
+                        "deprecatedNames": [],
+                        "shortDescription": {"text": "Used when a line is longer than a given number of characters"},
+                        "fullDescription": {"text": "Used when a line is longer than a given number of characters."},
+                        "help": {"text": ":line-too-long (C0301): *Line too long (%s/%s)*\n  Used when a line is longer than a given number of characters."},
+                        "helpUri": "https://pylint.readthedocs.io/en/stable/user_guide/messages/convention/line-too-long.html",
+                    }],
+                },
+            },
+            "results": [{
+                "ruleId": "C0301",
+                "message": {"text": "Line too long (1/2)"},
+                "level": "note",
+                "locations": [{
+                    "physicalLocation": {
+                        "artifactLocation": {
+                            "uri": "0123",
+                        },
+                        "region": {
+                            "startLine": 1,
+                            "startColumn": 1,
+                            "endLine": 1,
+                            "endColumn": 5,
+                        },
+                    },
+                }],
+                "partialFingerprints": {"nodePath/v1": ""},
+            }]
+        }]
+    }