Skip to content

Consolidate Asset/Dandiset models; gate publish validation on datePublished#419

Draft
candleindark wants to merge 2 commits into
masterfrom
consolidate-models
Draft

Consolidate Asset/Dandiset models; gate publish validation on datePublished#419
candleindark wants to merge 2 commits into
masterfrom
consolidate-models

Conversation

@candleindark

@candleindark candleindark commented Jun 8, 2026

Copy link
Copy Markdown
Member

Summary

Make every model class's schemaKey value equal its class name, so that an eventual LinkML translation can use schemaKey as its type designator (designates_type). Three classes violated this (BareAsset"Asset", PublishedAsset"Asset", PublishedDandiset"Dandiset"), and fixing them takes two distinct changes:

  1. Consolidate the publication-specific variants into their base classes: PublishedDandiset merges into Dandiset and PublishedAsset into Asset. Publication requirements now fire via a datePublished-gated validator instead of separate classes.
  2. Align BareAsset.schemaKey to "BareAsset"

Stored schemaKey values in archive data are unchanged, and the old names remain as deprecated aliases so dandi-cli/dandi-archive imports keep working. Adopting this release does, however, require one lockstep dandi-cli change (see Follow-ups).

What changed (all in dandischema/models.py)
  • Merge PublishedDandisetDandiset, PublishedAssetAsset. Publication-only fields (doi, publishedBy, datePublished, releaseNotes on Dandiset; publishedBy, datePublished on Asset) become optional, and the publication requirements move into a datePublished-gated check_publication_status model validator:

    • datePublished is None (draft) → publication-only fields must be absent;
    • datePublished set (published) → enforce the former Published* rules (publishedBy/url/doi presence, stricter id/url patterns, check_filesbytes, digest_sha256check).

    dandi-archive injects datePublished before validating, so the gated checks fire exactly as the Published* classes did.

  • Per-field validation errors. The publication validator accumulates all violations and raises them via pydantic_core.ValidationError.from_exception_data(...), so each error carries its own field loc (e.g. ('assetsSummary',)) plus the usual "Value error, …" message — rather than one model-level error with an empty loc.

  • Keep BareAsset (Asset still inherits it) with schemaKey: Literal["BareAsset"]; Asset overrides it to Literal["Asset"] (a localized # type: ignore[assignment] covers the narrowed override), and ensure_schemakey pins each instance's value to its class name.

  • Remove the Publishable mixin; add PublishedDandiset/PublishedAsset as deprecated aliases.

  • Simplify ensure_schemakey to a plain schemaKey == class-name check.

  • to_datacite asserts the published precondition (datePublished set) the PublishedDandiset type used to guarantee.

Why gate on datePublished rather than an explicit (context-triggered) validator

The publish-only checks could instead be triggered explicitly by the caller (e.g. a Pydantic validation-context flag passed by the publish flow) rather than by the presence of datePublished. We chose data-driven gating because:

  • It can be expressed in LinkML; an explicit context cannot. Gating on a real field is a conditional constraint on the data, which is exactly what a LinkML rule expresses (gen-json-schema emits it as if/then). A caller-supplied validation mode has no LinkML equivalent — LinkML describes only the data instance, not how validation is invoked — so it would have to live as hand-written Python forever, outside the source of truth, working against the migration this change is meant to enable.
  • It travels with the data. The gated checks fire through both validate() and direct construction (Dandiset(**data) / model_validate). A context flag only rides model_validate(..., context=...): the terse PublishedDandiset(**meta) form (used by to_datacite and in tests) cannot pass one, so it would silently skip the publish checks. Beyond the call-site rewrites that would force, a constructor whose enforcement quietly depends on how it is invoked is a lasting programming hazard.
  • It matches the domain. datePublished present ⟺ published is a real invariant: a draft never carries it, and the archive injects it only when building a publishable version. "Published" is a state of the record, not a mode a caller opts into.
  • Published-ness stays introspectable. With gating, "is this published?" is just obj.datePublished is not None; with a context flag, whether an object was validated as published is ephemeral to the call and recorded nowhere on the object.
Compatibility
  • Generated JSON Schema (verified by before/after diff): draft Dandiset/Asset gain only optional, readOnly publication properties — nothing newly required, no pattern tightened — so the dandi-archive Meditor is unaffected (it hides readOnly props). The published-*.json schemas become the relaxed versions, with publication strictness enforced by the gated validator.
  • PublishedDandiset/PublishedAsset/BareAsset imports continue to work in dandi-cli/dandi-archive.
Follow-ups (separate PRs)
  • dandi-cli (lockstep with adopting this release): set schemaKey = "Asset" on bare metadata at upload, since BareAsset.schemaKey is now "BareAsset" and the server does not normalize it.
  • dandi-archive _encode_pydantic_error robustness_encode_pydantic_error raises IndexError on validation errors with empty loc dandi-archive#2851. This PR's per-field errors avoid the empty-loc case at the source, but that fix is still worth having as defense-in-depth.
  • Later: remove the deprecated aliases once consumers adopt the consolidated names (and update dandi-archive's views/schema.py model mapping).

Test plan

  • tox -e py3 (269 passed, 17 skipped — skips are environment-only: no DOI_PREFIX / instance name not "DANDI")
  • tox -e lint,typing
  • Generated JSON Schema before/after diff reviewed
  • Added tests for the datePublished-gated publish requirements and the coherence invariant
  • Confirm against dandi-archive's publish/validation flow once it adopts this release

@codecov

codecov Bot commented Jun 8, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 35.35354% with 64 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.33%. Comparing base (d752738) to head (30cf2fd).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
dandischema/models.py 0.00% 63 Missing ⚠️
dandischema/datacite/__init__.py 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #419      +/-   ##
==========================================
+ Coverage   48.31%   48.33%   +0.02%     
==========================================
  Files          19       19              
  Lines        2434     2464      +30     
==========================================
+ Hits         1176     1191      +15     
- Misses       1258     1273      +15     
Flag Coverage Δ
unittests 48.33% <35.35%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…Published

Collapse the publication-specific model variants into their base classes so
that each class's schemaKey value matches its class name, which is what an
eventual LinkML translation needs to use schemaKey as a type designator
(designates_type). The three classes whose schemaKey differed from their class
name (BareAsset -> "Asset", PublishedAsset -> "Asset", PublishedDandiset ->
"Dandiset") were the blockers.

Changes (all in dandischema/models.py):

- Merge PublishedDandiset into Dandiset and PublishedAsset into Asset. The
  publication-only fields (doi, publishedBy, datePublished, releaseNotes on
  Dandiset; publishedBy, datePublished on Asset) become optional, and the
  publication requirements move into a datePublished-gated
  `check_publication_status` model validator on each class:
    - when datePublished is None (a draft), the publication-only fields must be
      absent;
    - when datePublished is set (published), enforce the former Published*
      requirements (publishedBy/url/doi presence, the stricter id/url patterns,
      check_filesbytes, digest_sha256check). All violations are reported
      together in one error.
  dandi-archive's publish flow injects datePublished before validating, so the
  gated checks fire exactly as the Published* classes did before.

- Keep BareAsset as a distinct class (Asset still inherits from it) but align
  its schemaKey to Literal["BareAsset"], so both BareAsset and Asset are
  schemaKey-aligned. The client (dandi-cli) is responsible for setting
  schemaKey to "Asset" when uploading bare metadata as an Asset.

- Remove the Publishable mixin; add PublishedDandiset/PublishedAsset as
  deprecated aliases of Dandiset/Asset for backward compatibility (dandi-cli,
  dandi-archive). These will be removed in a follow-up once consumers migrate.

- Simplify DandiBaseModel.ensure_schemakey to a plain schemaKey == class-name
  check now that no class intentionally diverges.

to_datacite now asserts the published precondition (datePublished set) that the
PublishedDandiset type used to guarantee. Tests updated for the merged models:
the published variants report the same missing fields as their base classes,
and the publication requirements are exercised on complete, datePublished
instances; new tests cover the publication-coherence invariant.

Generated JSON Schema diff: the draft schemas (Dandiset, Asset) gain only
optional, readOnly publication properties (nothing newly required, no pattern
tightened), so the dandi-archive Meditor is unaffected; the published-*.json
schemas become the relaxed versions, with publication strictness now enforced
by the gated Pydantic validator.

Co-Authored-By: Claude Code 2.1.161 / Claude Opus 4.8 claude-opus-4-8 <[email protected]>
Comment thread dandischema/models.py
if self.datePublished is None:
for name in ("doi", "publishedBy", "releaseNotes"):
if getattr(self, name) is not None:
err(name, f"{name} is not allowed unless datePublished is set")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please express this model/check as linkml "rule" to validate Dandiset published and not, and also compare to how it would look like if we made it keyed on having ANY of the publish-related fields set. Also how errors would look like in both cases.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worked this up as self-contained, runnable LinkML demos (linkml 1.11: linkml-validate / gen-json-schema / gen-pydantic), and the headline is reassuring: LinkML rules can make one field's required-or-forbidden status depend on whether another field is set, so the publication-status dependency this PR enforces translates directly to LinkML. By "publication-status dependency" I mean what the PR's check_publication_status implements: the publication fields must be set together, present when the record is published and absent when it is a draft. Nothing here suggests changing the model-validator in this PR.

Two ways to express it, and they turn out to be logically equivalent:

Case 1: keyed on datePublished (matches this PR)

Both directions of the PR's check_publication_status become rules:

rules:
  # published: datePublished set => publishedBy and doi required
  - preconditions:  { slot_conditions: { datePublished: { value_presence: PRESENT } } }
    postconditions: { slot_conditions: { publishedBy: { required: true }, doi: { required: true } } }
  # draft: datePublished absent => publishedBy and doi absent  (one rule per field, see gotcha below)
  - preconditions:  { slot_conditions: { datePublished: { value_presence: ABSENT } } }
    postconditions: { slot_conditions: { publishedBy: { value_presence: ABSENT } } }
  - preconditions:  { slot_conditions: { datePublished: { value_presence: ABSENT } } }
    postconditions: { slot_conditions: { doi: { value_presence: ABSENT } } }

Case 2: keyed on ANY of the fields being set

rules:
  - preconditions:
      any_of:
        - slot_conditions: { datePublished: { value_presence: PRESENT } }
        - slot_conditions: { publishedBy:   { value_presence: PRESENT } }
        - slot_conditions: { doi:           { value_presence: PRESENT } }
    postconditions:
      slot_conditions:
        datePublished: { required: true }
        publishedBy:   { required: true }
        doi:           { required: true }

They are logically equivalent

Both admit exactly "all three present, or all three absent". Validating all 8 presence combinations against each gives identical pass/fail in every case:

present set Case 1 Case 2
{} pass pass
{datePublished} / {publishedBy} / {doi} fail fail
any two of the three fail fail
all three pass pass

So the choice of keying is an implementation detail: in LinkML (Case 1 vs Case 2), and equally in the Python model-validator, which could key on datePublished or on "any publication field is set" to the same effect. The design in this PR does not need to change on account of LinkML. That said, keying on datePublished is conceptually cleaner than keying on "any publication field is set", regardless of the encoding:

  • Unambiguous intent. datePublished is not None is the single, authoritative test for "this is a published version", a real domain invariant (the archive sets datePublished exactly when it builds a publishable version). Keying on "any publication field is set" instead treats an accidentally-populated field as a declaration that the record is published, when it may just be a mistake.
  • Cleaner docs and user code. Downstream code and documentation can define published/draft off one field instead of "any of N".
  • More directional error messages. Because the datePublished-keyed Python validator knows which way the dependency is violated, it can say either "doi is not allowed unless datePublished is set" (a stray field on a draft) or "doi is required for a published Dandiset" (a missing field on a published record). Keyed on "any publication field is set", the validator has only one move: demand the missing siblings. (This concerns the hand-written Python messages. The LinkML CLI messages shown below are unambiguous in both keyings, but being derived solely from JSON Schema validation errors, they can't carry the finer-grained, directional information a Pydantic validator can.)

How the errors look

These are the messages from linkml-validate, the LinkML CLI validator, which by default validates an instance against the JSON Schema that gen-json-schema derives from the rules (they become if/then). Full matrix below.

Published-but-incomplete ({datePublished: ...}), identical under both keyings:

[ERROR] [.../0] 'publishedBy' is a required property in /
[ERROR] [.../0] 'doi' is a required property in /

The distinguishing input, a stray field with no datePublished ({doi: ...}):

  • Case 1 (keyed on datePublished) flags the stray field:
    [ERROR] [.../0] {'doi': '...'} should not be valid under {'required': ['doi']} in /
    
  • Case 2 (keyed on any) instead demands the missing siblings:
    [ERROR] [.../0] 'datePublished' is a required property in /
    [ERROR] [.../0] 'publishedBy' is a required property in /
    

Both are unambiguous messages; they just point at different fixes, matching each schema's semantics. One caveat: the structured loc is the object root (in /), not the offending field, because JSON Schema required attaches at the object level. The field name does still appear in the message text, it is just not in the machine-readable loc, which is coarser than the per-field loc the Pydantic from_exception_data validator in this PR produces.

Note for the migration: gen-pydantic does NOT enforce these rules

The rules only produce enforcement through gen-json-schema. gen-pydantic emits them as inert linkml_meta metadata with no validator, so the generated Pydantic model happily constructs every one of the invalid instances above (verified for both keyings, matrix below). This is specific to cross-field rules: gen-pydantic does still enforce field-level constraints (types, required, pattern, enums) as ordinary Pydantic field validators.

This isn't a blocker, just a decision about where validation lives. Rather than keeping a hand-written model_validator forever, dandischema could delegate validation to LinkML (validating against the generated JSON Schema via the linkml validation path) and use the generated Pydantic models for loading and manipulating data. In that split, field-level constraints are still checked by Pydantic on load, and the cross-field publication dependency is enforced by the LinkML rules. In this PR, which predates the migration, that enforcement lives in the model_validator as before.

Aside: a gotcha writing the draft-direction rule (likely a LinkML bug; will file separately)

The natural way to forbid the fields on a draft, a single rule with two value_presence: ABSENT postconditions, is wrong:

  - preconditions:  { slot_conditions: { datePublished: { value_presence: ABSENT } } }
    postconditions: { slot_conditions: { publishedBy: { value_presence: ABSENT }, doi: { value_presence: ABSENT } } }

gen-json-schema compiles that then to {"not": {"required": ["publishedBy", "doi"]}}, i.e. ¬(publishedBy ∧ doi) = "not both present", so {publishedBy} or {doi} alone slips through. Splitting into one rule per forbidden field yields two separate {"not": {"required": ["publishedBy"]}} / ["doi"], which correctly AND to "neither present". I'll file this against LinkML.

Reproduce

Everything above was produced with linkml 1.11 (linkml/linkml-runtime 1.11.0). The Pydantic models were generated with:

$ gen-pydantic case1_datePublished_keyed.yaml > case1_models.py
$ gen-pydantic case2_any_field_keyed.yaml     > case2_models.py

JSON Schema validation of a metadata file against a schema, and inspecting the generated if/then:

$ linkml-validate -s case1_datePublished_keyed.yaml -C Dandiset metadata/<file>.yaml
$ gen-json-schema --top-class Dandiset case1_datePublished_keyed.yaml

The full set of files follows.

case1_datePublished_keyed.yaml (correct Case 1: draft direction split into one rule per field)
id: https://example.org/case1_datePublished_keyed
name: case1_datePublished_keyed
prefixes:
  linkml: https://w3id.org/linkml/
imports:
  - linkml:types
default_range: string
classes:
  Dandiset:
    slots: [datePublished, publishedBy, doi]
    rules:
      - preconditions:
          slot_conditions:
            datePublished: { value_presence: PRESENT }
        postconditions:
          slot_conditions:
            publishedBy: { required: true }
            doi: { required: true }
      - preconditions:
          slot_conditions:
            datePublished: { value_presence: ABSENT }
        postconditions:
          slot_conditions:
            publishedBy: { value_presence: ABSENT }
      - preconditions:
          slot_conditions:
            datePublished: { value_presence: ABSENT }
        postconditions:
          slot_conditions:
            doi: { value_presence: ABSENT }
slots:
  datePublished:
  publishedBy:
  doi:
case2_any_field_keyed.yaml
id: https://example.org/case2_any_field_keyed
name: case2_any_field_keyed
prefixes:
  linkml: https://w3id.org/linkml/
imports:
  - linkml:types
default_range: string

# Case 2: the same fields, but required conditioned on ANY of them being set.
# If any of datePublished / publishedBy / doi is present, all three are required.

classes:
  Dandiset:
    slots:
      - datePublished
      - publishedBy
      - doi
    rules:
      - preconditions:
          any_of:
            - slot_conditions:
                datePublished:
                  value_presence: PRESENT
            - slot_conditions:
                publishedBy:
                  value_presence: PRESENT
            - slot_conditions:
                doi:
                  value_presence: PRESENT
        postconditions:
          slot_conditions:
            datePublished:
              required: true
            publishedBy:
              required: true
            doi:
              required: true

slots:
  datePublished:
  publishedBy:
  doi:
case1_naive_coherence_bug.yaml (buggy single-rule draft direction, kept to demonstrate the gotcha)
id: https://example.org/case1_naive_coherence_bug
name: case1_naive_coherence_bug
prefixes:
  linkml: https://w3id.org/linkml/
imports:
  - linkml:types
default_range: string

# Case 1: fields required conditioned on ONE designated field being set,
# plus the coherence direction: when datePublished is absent, the other two
# must be absent too.
# When datePublished is present, publishedBy and doi become required.
# When datePublished is absent, publishedBy and doi must be absent.

classes:
  Dandiset:
    slots:
      - datePublished
      - publishedBy
      - doi
    rules:
      - preconditions:
          slot_conditions:
            datePublished:
              value_presence: PRESENT
        postconditions:
          slot_conditions:
            publishedBy:
              required: true
            doi:
              required: true
      - preconditions:
          slot_conditions:
            datePublished:
              value_presence: ABSENT
        postconditions:
          slot_conditions:
            publishedBy:
              value_presence: ABSENT
            doi:
              value_presence: ABSENT

slots:
  datePublished:
  publishedBy:
  doi:
Test metadata (8 instances, metadata/*.yaml)
# all_absent__valid_draft.yaml  -> valid under both keyings
{}

# all_present__valid_published.yaml  -> valid under both keyings
datePublished: "2021-01-01T00:00:00"
publishedBy: "urn:uuid:publisher"
doi: "10.48324/dandi.000001/0.210101.0000"

# only_datePublished__published_missing_publishedBy_and_doi.yaml  -> invalid under both
datePublished: "2021-01-01T00:00:00"

# datePublished_and_publishedBy__published_missing_doi.yaml  -> invalid under both
datePublished: "2021-01-01T00:00:00"
publishedBy: "urn:uuid:publisher"

# datePublished_and_doi__published_missing_publishedBy.yaml  -> invalid under both
datePublished: "2021-01-01T00:00:00"
doi: "10.48324/dandi.000001/0.210101.0000"

# only_publishedBy__stray_no_datePublished.yaml  -> the distinguishing case
publishedBy: "urn:uuid:publisher"

# only_doi__stray_no_datePublished.yaml  -> the distinguishing case
doi: "10.48324/dandi.000001/0.210101.0000"

# publishedBy_and_doi__stray_no_datePublished.yaml  -> invalid under both correct keyings
publishedBy: "urn:uuid:publisher"
doi: "10.48324/dandi.000001/0.210101.0000"
case1_models.py (from gen-pydantic case1_datePublished_keyed.yaml)

Note the rules survive only as an inert linkml_meta ClassVar; nothing enforces them.

from __future__ import annotations

import re
import sys
from datetime import (
    date,
    datetime,
    time
)
from decimal import Decimal
from enum import Enum
from typing import (
    Any,
    ClassVar,
    Literal,
    Optional,
    Union
)

from pydantic import (
    BaseModel,
    ConfigDict,
    Field,
    RootModel,
    SerializationInfo,
    SerializerFunctionWrapHandler,
    field_validator,
    model_serializer
)


metamodel_version = "1.11.0"
version = "None"


class ConfiguredBaseModel(BaseModel):
    model_config = ConfigDict(
        serialize_by_alias = True,
        validate_by_name = True,
        validate_assignment = True,
        validate_default = True,
        extra = "forbid",
        arbitrary_types_allowed = True,
        use_enum_values = True,
        strict = False,
    )


class LinkMLMeta(RootModel):
    root: dict[str, Any] = {}
    model_config = ConfigDict(frozen=True)

    def __getattr__(self, key:str):
        return getattr(self.root, key)

    def __getitem__(self, key:str):
        return self.root[key]

    def __setitem__(self, key:str, value):
        self.root[key] = value

    def __contains__(self, key:str) -> bool:
        return key in self.root


linkml_meta = LinkMLMeta({'default_prefix': 'https://example.org/case1_datePublished_keyed/',
     'default_range': 'string',
     'id': 'https://example.org/case1_datePublished_keyed',
     'imports': ['linkml:types'],
     'name': 'case1_datePublished_keyed',
     'prefixes': {'linkml': {'prefix_prefix': 'linkml',
                             'prefix_reference': 'https://w3id.org/linkml/'}},
     'source_file': 'case1_datePublished_keyed.yaml'} )


class Dandiset(ConfiguredBaseModel):
    linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({'from_schema': 'https://example.org/case1_datePublished_keyed',
         'rules': [{'postconditions': {'slot_conditions': {'doi': {'name': 'doi',
                                                                   'required': True},
                                                           'publishedBy': {'name': 'publishedBy',
                                                                           'required': True}}},
                    'preconditions': {'slot_conditions': {'datePublished': {'name': 'datePublished',
                                                                            'value_presence': 'PRESENT'}}}},
                   {'postconditions': {'slot_conditions': {'publishedBy': {'name': 'publishedBy',
                                                                           'value_presence': 'ABSENT'}}},
                    'preconditions': {'slot_conditions': {'datePublished': {'name': 'datePublished',
                                                                            'value_presence': 'ABSENT'}}}},
                   {'postconditions': {'slot_conditions': {'doi': {'name': 'doi',
                                                                   'value_presence': 'ABSENT'}}},
                    'preconditions': {'slot_conditions': {'datePublished': {'name': 'datePublished',
                                                                            'value_presence': 'ABSENT'}}}}]})

    datePublished: Optional[str] = Field(default=None, json_schema_extra = { "linkml_meta": {'domain_of': ['Dandiset']} })
    publishedBy: Optional[str] = Field(default=None, json_schema_extra = { "linkml_meta": {'domain_of': ['Dandiset']} })
    doi: Optional[str] = Field(default=None, json_schema_extra = { "linkml_meta": {'domain_of': ['Dandiset']} })


# Model rebuild
# see https://pydantic-docs.helpmanual.io/usage/models/#rebuilding-a-model
Dandiset.model_rebuild()
case2_models.py (from gen-pydantic case2_any_field_keyed.yaml) — only the Dandiset class differs
class Dandiset(ConfiguredBaseModel):
    linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({'from_schema': 'https://example.org/case2_any_field_keyed',
         'rules': [{'postconditions': {'slot_conditions': {'datePublished': {'name': 'datePublished',
                                                                             'required': True},
                                                           'doi': {'name': 'doi',
                                                                   'required': True},
                                                           'publishedBy': {'name': 'publishedBy',
                                                                           'required': True}}},
                    'preconditions': {'any_of': [{'slot_conditions': {'datePublished': {'name': 'datePublished',
                                                                                        'value_presence': 'PRESENT'}}},
                                                 {'slot_conditions': {'publishedBy': {'name': 'publishedBy',
                                                                                      'value_presence': 'PRESENT'}}},
                                                 {'slot_conditions': {'doi': {'name': 'doi',
                                                                              'value_presence': 'PRESENT'}}}]}}]})

    datePublished: Optional[str] = Field(default=None, json_schema_extra = { "linkml_meta": {'domain_of': ['Dandiset']} })
    publishedBy: Optional[str] = Field(default=None, json_schema_extra = { "linkml_meta": {'domain_of': ['Dandiset']} })
    doi: Optional[str] = Field(default=None, json_schema_extra = { "linkml_meta": {'domain_of': ['Dandiset']} })
Full JSON Schema validation matrix (linkml-validate)
# case1_datePublished_keyed.yaml  = keyed on datePublished, coherence split one-rule-per-field (CORRECT)
# case2_any_field_keyed.yaml      = keyed on ANY of the three fields being present
# case1_naive_coherence_bug.yaml  = keyed on datePublished, coherence as a SINGLE rule (buggy translation)

================================================================
SCHEMA: case1_datePublished_keyed.yaml
================================================================
--- all_absent__valid_draft.yaml
No issues found

--- all_present__valid_published.yaml
No issues found

--- datePublished_and_doi__published_missing_publishedBy.yaml
[ERROR] [.../0] 'publishedBy' is a required property in /

--- datePublished_and_publishedBy__published_missing_doi.yaml
[ERROR] [.../0] 'doi' is a required property in /

--- only_datePublished__published_missing_publishedBy_and_doi.yaml
[ERROR] [.../0] 'publishedBy' is a required property in /
[ERROR] [.../0] 'doi' is a required property in /

--- only_doi__stray_no_datePublished.yaml
[ERROR] [.../0] {'doi': '10.48324/dandi.000001/0.210101.0000'} should not be valid under {'required': ['doi']} in /

--- only_publishedBy__stray_no_datePublished.yaml
[ERROR] [.../0] {'publishedBy': 'urn:uuid:publisher'} should not be valid under {'required': ['publishedBy']} in /

--- publishedBy_and_doi__stray_no_datePublished.yaml
[ERROR] [.../0] {'publishedBy': 'urn:uuid:publisher', 'doi': '...'} should not be valid under {'required': ['publishedBy']} in /
[ERROR] [.../0] {'publishedBy': 'urn:uuid:publisher', 'doi': '...'} should not be valid under {'required': ['doi']} in /

================================================================
SCHEMA: case2_any_field_keyed.yaml
================================================================
--- all_absent__valid_draft.yaml
No issues found

--- all_present__valid_published.yaml
No issues found

--- datePublished_and_doi__published_missing_publishedBy.yaml
[ERROR] [.../0] 'publishedBy' is a required property in /

--- datePublished_and_publishedBy__published_missing_doi.yaml
[ERROR] [.../0] 'doi' is a required property in /

--- only_datePublished__published_missing_publishedBy_and_doi.yaml
[ERROR] [.../0] 'publishedBy' is a required property in /
[ERROR] [.../0] 'doi' is a required property in /

--- only_doi__stray_no_datePublished.yaml
[ERROR] [.../0] 'datePublished' is a required property in /
[ERROR] [.../0] 'publishedBy' is a required property in /

--- only_publishedBy__stray_no_datePublished.yaml
[ERROR] [.../0] 'datePublished' is a required property in /
[ERROR] [.../0] 'doi' is a required property in /

--- publishedBy_and_doi__stray_no_datePublished.yaml
[ERROR] [.../0] 'datePublished' is a required property in /

================================================================
SCHEMA: case1_naive_coherence_bug.yaml
================================================================
--- all_absent__valid_draft.yaml
No issues found

--- all_present__valid_published.yaml
No issues found

--- datePublished_and_doi__published_missing_publishedBy.yaml
[ERROR] [.../0] 'publishedBy' is a required property in /

--- datePublished_and_publishedBy__published_missing_doi.yaml
[ERROR] [.../0] 'doi' is a required property in /

--- only_datePublished__published_missing_publishedBy_and_doi.yaml
[ERROR] [.../0] 'publishedBy' is a required property in /
[ERROR] [.../0] 'doi' is a required property in /

--- only_doi__stray_no_datePublished.yaml
No issues found          <-- WRONG: stray doi slips through

--- only_publishedBy__stray_no_datePublished.yaml
No issues found          <-- WRONG: stray publishedBy slips through

--- publishedBy_and_doi__stray_no_datePublished.yaml
[ERROR] [.../0] {'publishedBy': 'urn:uuid:publisher', 'doi': '...'} should not be valid under {'required': ['publishedBy', 'doi']} in /
Pydantic construction matrix (both generated models) — nothing is rejected
# gen-pydantic does NOT emit validators for rules -> rule violations are NOT caught.

================================================================
MODEL: case1_datePublished_keyed
================================================================
all_absent__valid_draft.yaml                                  -> constructed OK
all_present__valid_published.yaml                             -> constructed OK
datePublished_and_doi__published_missing_publishedBy.yaml     -> constructed OK
datePublished_and_publishedBy__published_missing_doi.yaml     -> constructed OK
only_datePublished__published_missing_publishedBy_and_doi.yaml-> constructed OK
only_doi__stray_no_datePublished.yaml                         -> constructed OK
only_publishedBy__stray_no_datePublished.yaml                 -> constructed OK
publishedBy_and_doi__stray_no_datePublished.yaml              -> constructed OK

================================================================
MODEL: case2_any_field_keyed
================================================================
(identical: all 8 instances constructed OK)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An issue has been filed related to the bug we discovered in LinkML in the last response.

The `check_publication_status` model-validators on `Dandiset`/`Asset`
previously raised a single combined `ValueError`, which pydantic reports with
an empty `loc` (`()`), losing the per-field association and tripping consumers
that index `error["loc"][0]` (e.g. dandi-archive's `_encode_pydantic_error`).

Build a `list[InitErrorDetails]` instead — one entry per failing check, each
with its own `loc=(field,)`, `type="value_error"`, and the message carried in
`ctx={"error": ...}` — and raise it via
`pydantic_core.ValidationError.from_exception_data(...)`. This keeps the single
accumulating model-validator design (all violations reported together) while
giving every publication error a proper field `loc`, and preserves the exact
"Value error, <message>" text the former field-validators produced.

test_dandimeta_1 now asserts the per-field locs rather than one combined
message.

Co-Authored-By: Claude Code 2.1.198 / Claude Opus 4.8 claude-opus-4-8 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants