Consolidate Asset/Dandiset models; gate publish validation on datePublished#419
Consolidate Asset/Dandiset models; gate publish validation on datePublished#419candleindark wants to merge 2 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #419 +/- ##
==========================================
+ Coverage 48.31% 48.33% +0.02%
==========================================
Files 19 19
Lines 2434 2464 +30
==========================================
+ Hits 1176 1191 +15
- Misses 1258 1273 +15
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
e632eee to
a15e2e9
Compare
…Published
Collapse the publication-specific model variants into their base classes so
that each class's schemaKey value matches its class name, which is what an
eventual LinkML translation needs to use schemaKey as a type designator
(designates_type). The three classes whose schemaKey differed from their class
name (BareAsset -> "Asset", PublishedAsset -> "Asset", PublishedDandiset ->
"Dandiset") were the blockers.
Changes (all in dandischema/models.py):
- Merge PublishedDandiset into Dandiset and PublishedAsset into Asset. The
publication-only fields (doi, publishedBy, datePublished, releaseNotes on
Dandiset; publishedBy, datePublished on Asset) become optional, and the
publication requirements move into a datePublished-gated
`check_publication_status` model validator on each class:
- when datePublished is None (a draft), the publication-only fields must be
absent;
- when datePublished is set (published), enforce the former Published*
requirements (publishedBy/url/doi presence, the stricter id/url patterns,
check_filesbytes, digest_sha256check). All violations are reported
together in one error.
dandi-archive's publish flow injects datePublished before validating, so the
gated checks fire exactly as the Published* classes did before.
- Keep BareAsset as a distinct class (Asset still inherits from it) but align
its schemaKey to Literal["BareAsset"], so both BareAsset and Asset are
schemaKey-aligned. The client (dandi-cli) is responsible for setting
schemaKey to "Asset" when uploading bare metadata as an Asset.
- Remove the Publishable mixin; add PublishedDandiset/PublishedAsset as
deprecated aliases of Dandiset/Asset for backward compatibility (dandi-cli,
dandi-archive). These will be removed in a follow-up once consumers migrate.
- Simplify DandiBaseModel.ensure_schemakey to a plain schemaKey == class-name
check now that no class intentionally diverges.
to_datacite now asserts the published precondition (datePublished set) that the
PublishedDandiset type used to guarantee. Tests updated for the merged models:
the published variants report the same missing fields as their base classes,
and the publication requirements are exercised on complete, datePublished
instances; new tests cover the publication-coherence invariant.
Generated JSON Schema diff: the draft schemas (Dandiset, Asset) gain only
optional, readOnly publication properties (nothing newly required, no pattern
tightened), so the dandi-archive Meditor is unaffected; the published-*.json
schemas become the relaxed versions, with publication strictness now enforced
by the gated Pydantic validator.
Co-Authored-By: Claude Code 2.1.161 / Claude Opus 4.8 claude-opus-4-8 <[email protected]>
a15e2e9 to
1298b26
Compare
| if self.datePublished is None: | ||
| for name in ("doi", "publishedBy", "releaseNotes"): | ||
| if getattr(self, name) is not None: | ||
| err(name, f"{name} is not allowed unless datePublished is set") |
There was a problem hiding this comment.
please express this model/check as linkml "rule" to validate Dandiset published and not, and also compare to how it would look like if we made it keyed on having ANY of the publish-related fields set. Also how errors would look like in both cases.
There was a problem hiding this comment.
I worked this up as self-contained, runnable LinkML demos (linkml 1.11: linkml-validate / gen-json-schema / gen-pydantic), and the headline is reassuring: LinkML rules can make one field's required-or-forbidden status depend on whether another field is set, so the publication-status dependency this PR enforces translates directly to LinkML. By "publication-status dependency" I mean what the PR's check_publication_status implements: the publication fields must be set together, present when the record is published and absent when it is a draft. Nothing here suggests changing the model-validator in this PR.
Two ways to express it, and they turn out to be logically equivalent:
Case 1: keyed on datePublished (matches this PR)
Both directions of the PR's check_publication_status become rules:
rules:
# published: datePublished set => publishedBy and doi required
- preconditions: { slot_conditions: { datePublished: { value_presence: PRESENT } } }
postconditions: { slot_conditions: { publishedBy: { required: true }, doi: { required: true } } }
# draft: datePublished absent => publishedBy and doi absent (one rule per field, see gotcha below)
- preconditions: { slot_conditions: { datePublished: { value_presence: ABSENT } } }
postconditions: { slot_conditions: { publishedBy: { value_presence: ABSENT } } }
- preconditions: { slot_conditions: { datePublished: { value_presence: ABSENT } } }
postconditions: { slot_conditions: { doi: { value_presence: ABSENT } } }Case 2: keyed on ANY of the fields being set
rules:
- preconditions:
any_of:
- slot_conditions: { datePublished: { value_presence: PRESENT } }
- slot_conditions: { publishedBy: { value_presence: PRESENT } }
- slot_conditions: { doi: { value_presence: PRESENT } }
postconditions:
slot_conditions:
datePublished: { required: true }
publishedBy: { required: true }
doi: { required: true }They are logically equivalent
Both admit exactly "all three present, or all three absent". Validating all 8 presence combinations against each gives identical pass/fail in every case:
| present set | Case 1 | Case 2 |
|---|---|---|
{} |
pass | pass |
{datePublished} / {publishedBy} / {doi} |
fail | fail |
| any two of the three | fail | fail |
| all three | pass | pass |
So the choice of keying is an implementation detail: in LinkML (Case 1 vs Case 2), and equally in the Python model-validator, which could key on datePublished or on "any publication field is set" to the same effect. The design in this PR does not need to change on account of LinkML. That said, keying on datePublished is conceptually cleaner than keying on "any publication field is set", regardless of the encoding:
- Unambiguous intent.
datePublished is not Noneis the single, authoritative test for "this is a published version", a real domain invariant (the archive setsdatePublishedexactly when it builds a publishable version). Keying on "any publication field is set" instead treats an accidentally-populated field as a declaration that the record is published, when it may just be a mistake. - Cleaner docs and user code. Downstream code and documentation can define published/draft off one field instead of "any of N".
- More directional error messages. Because the datePublished-keyed Python validator knows which way the dependency is violated, it can say either "doi is not allowed unless datePublished is set" (a stray field on a draft) or "doi is required for a published Dandiset" (a missing field on a published record). Keyed on "any publication field is set", the validator has only one move: demand the missing siblings. (This concerns the hand-written Python messages. The LinkML CLI messages shown below are unambiguous in both keyings, but being derived solely from JSON Schema validation errors, they can't carry the finer-grained, directional information a Pydantic validator can.)
How the errors look
These are the messages from linkml-validate, the LinkML CLI validator, which by default validates an instance against the JSON Schema that gen-json-schema derives from the rules (they become if/then). Full matrix below.
Published-but-incomplete ({datePublished: ...}), identical under both keyings:
[ERROR] [.../0] 'publishedBy' is a required property in /
[ERROR] [.../0] 'doi' is a required property in /
The distinguishing input, a stray field with no datePublished ({doi: ...}):
- Case 1 (keyed on
datePublished) flags the stray field:[ERROR] [.../0] {'doi': '...'} should not be valid under {'required': ['doi']} in / - Case 2 (keyed on any) instead demands the missing siblings:
[ERROR] [.../0] 'datePublished' is a required property in / [ERROR] [.../0] 'publishedBy' is a required property in /
Both are unambiguous messages; they just point at different fixes, matching each schema's semantics. One caveat: the structured loc is the object root (in /), not the offending field, because JSON Schema required attaches at the object level. The field name does still appear in the message text, it is just not in the machine-readable loc, which is coarser than the per-field loc the Pydantic from_exception_data validator in this PR produces.
Note for the migration: gen-pydantic does NOT enforce these rules
The rules only produce enforcement through gen-json-schema. gen-pydantic emits them as inert linkml_meta metadata with no validator, so the generated Pydantic model happily constructs every one of the invalid instances above (verified for both keyings, matrix below). This is specific to cross-field rules: gen-pydantic does still enforce field-level constraints (types, required, pattern, enums) as ordinary Pydantic field validators.
This isn't a blocker, just a decision about where validation lives. Rather than keeping a hand-written model_validator forever, dandischema could delegate validation to LinkML (validating against the generated JSON Schema via the linkml validation path) and use the generated Pydantic models for loading and manipulating data. In that split, field-level constraints are still checked by Pydantic on load, and the cross-field publication dependency is enforced by the LinkML rules. In this PR, which predates the migration, that enforcement lives in the model_validator as before.
Aside: a gotcha writing the draft-direction rule (likely a LinkML bug; will file separately)
The natural way to forbid the fields on a draft, a single rule with two value_presence: ABSENT postconditions, is wrong:
- preconditions: { slot_conditions: { datePublished: { value_presence: ABSENT } } }
postconditions: { slot_conditions: { publishedBy: { value_presence: ABSENT }, doi: { value_presence: ABSENT } } }gen-json-schema compiles that then to {"not": {"required": ["publishedBy", "doi"]}}, i.e. ¬(publishedBy ∧ doi) = "not both present", so {publishedBy} or {doi} alone slips through. Splitting into one rule per forbidden field yields two separate {"not": {"required": ["publishedBy"]}} / ["doi"], which correctly AND to "neither present". I'll file this against LinkML.
Reproduce
Everything above was produced with linkml 1.11 (linkml/linkml-runtime 1.11.0). The Pydantic models were generated with:
$ gen-pydantic case1_datePublished_keyed.yaml > case1_models.py
$ gen-pydantic case2_any_field_keyed.yaml > case2_models.pyJSON Schema validation of a metadata file against a schema, and inspecting the generated if/then:
$ linkml-validate -s case1_datePublished_keyed.yaml -C Dandiset metadata/<file>.yaml
$ gen-json-schema --top-class Dandiset case1_datePublished_keyed.yamlThe full set of files follows.
case1_datePublished_keyed.yaml (correct Case 1: draft direction split into one rule per field)
id: https://example.org/case1_datePublished_keyed
name: case1_datePublished_keyed
prefixes:
linkml: https://w3id.org/linkml/
imports:
- linkml:types
default_range: string
classes:
Dandiset:
slots: [datePublished, publishedBy, doi]
rules:
- preconditions:
slot_conditions:
datePublished: { value_presence: PRESENT }
postconditions:
slot_conditions:
publishedBy: { required: true }
doi: { required: true }
- preconditions:
slot_conditions:
datePublished: { value_presence: ABSENT }
postconditions:
slot_conditions:
publishedBy: { value_presence: ABSENT }
- preconditions:
slot_conditions:
datePublished: { value_presence: ABSENT }
postconditions:
slot_conditions:
doi: { value_presence: ABSENT }
slots:
datePublished:
publishedBy:
doi:case2_any_field_keyed.yaml
id: https://example.org/case2_any_field_keyed
name: case2_any_field_keyed
prefixes:
linkml: https://w3id.org/linkml/
imports:
- linkml:types
default_range: string
# Case 2: the same fields, but required conditioned on ANY of them being set.
# If any of datePublished / publishedBy / doi is present, all three are required.
classes:
Dandiset:
slots:
- datePublished
- publishedBy
- doi
rules:
- preconditions:
any_of:
- slot_conditions:
datePublished:
value_presence: PRESENT
- slot_conditions:
publishedBy:
value_presence: PRESENT
- slot_conditions:
doi:
value_presence: PRESENT
postconditions:
slot_conditions:
datePublished:
required: true
publishedBy:
required: true
doi:
required: true
slots:
datePublished:
publishedBy:
doi:case1_naive_coherence_bug.yaml (buggy single-rule draft direction, kept to demonstrate the gotcha)
id: https://example.org/case1_naive_coherence_bug
name: case1_naive_coherence_bug
prefixes:
linkml: https://w3id.org/linkml/
imports:
- linkml:types
default_range: string
# Case 1: fields required conditioned on ONE designated field being set,
# plus the coherence direction: when datePublished is absent, the other two
# must be absent too.
# When datePublished is present, publishedBy and doi become required.
# When datePublished is absent, publishedBy and doi must be absent.
classes:
Dandiset:
slots:
- datePublished
- publishedBy
- doi
rules:
- preconditions:
slot_conditions:
datePublished:
value_presence: PRESENT
postconditions:
slot_conditions:
publishedBy:
required: true
doi:
required: true
- preconditions:
slot_conditions:
datePublished:
value_presence: ABSENT
postconditions:
slot_conditions:
publishedBy:
value_presence: ABSENT
doi:
value_presence: ABSENT
slots:
datePublished:
publishedBy:
doi:Test metadata (8 instances, metadata/*.yaml)
# all_absent__valid_draft.yaml -> valid under both keyings
{}
# all_present__valid_published.yaml -> valid under both keyings
datePublished: "2021-01-01T00:00:00"
publishedBy: "urn:uuid:publisher"
doi: "10.48324/dandi.000001/0.210101.0000"
# only_datePublished__published_missing_publishedBy_and_doi.yaml -> invalid under both
datePublished: "2021-01-01T00:00:00"
# datePublished_and_publishedBy__published_missing_doi.yaml -> invalid under both
datePublished: "2021-01-01T00:00:00"
publishedBy: "urn:uuid:publisher"
# datePublished_and_doi__published_missing_publishedBy.yaml -> invalid under both
datePublished: "2021-01-01T00:00:00"
doi: "10.48324/dandi.000001/0.210101.0000"
# only_publishedBy__stray_no_datePublished.yaml -> the distinguishing case
publishedBy: "urn:uuid:publisher"
# only_doi__stray_no_datePublished.yaml -> the distinguishing case
doi: "10.48324/dandi.000001/0.210101.0000"
# publishedBy_and_doi__stray_no_datePublished.yaml -> invalid under both correct keyings
publishedBy: "urn:uuid:publisher"
doi: "10.48324/dandi.000001/0.210101.0000"case1_models.py (from gen-pydantic case1_datePublished_keyed.yaml)
Note the rules survive only as an inert linkml_meta ClassVar; nothing enforces them.
from __future__ import annotations
import re
import sys
from datetime import (
date,
datetime,
time
)
from decimal import Decimal
from enum import Enum
from typing import (
Any,
ClassVar,
Literal,
Optional,
Union
)
from pydantic import (
BaseModel,
ConfigDict,
Field,
RootModel,
SerializationInfo,
SerializerFunctionWrapHandler,
field_validator,
model_serializer
)
metamodel_version = "1.11.0"
version = "None"
class ConfiguredBaseModel(BaseModel):
model_config = ConfigDict(
serialize_by_alias = True,
validate_by_name = True,
validate_assignment = True,
validate_default = True,
extra = "forbid",
arbitrary_types_allowed = True,
use_enum_values = True,
strict = False,
)
class LinkMLMeta(RootModel):
root: dict[str, Any] = {}
model_config = ConfigDict(frozen=True)
def __getattr__(self, key:str):
return getattr(self.root, key)
def __getitem__(self, key:str):
return self.root[key]
def __setitem__(self, key:str, value):
self.root[key] = value
def __contains__(self, key:str) -> bool:
return key in self.root
linkml_meta = LinkMLMeta({'default_prefix': 'https://example.org/case1_datePublished_keyed/',
'default_range': 'string',
'id': 'https://example.org/case1_datePublished_keyed',
'imports': ['linkml:types'],
'name': 'case1_datePublished_keyed',
'prefixes': {'linkml': {'prefix_prefix': 'linkml',
'prefix_reference': 'https://w3id.org/linkml/'}},
'source_file': 'case1_datePublished_keyed.yaml'} )
class Dandiset(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({'from_schema': 'https://example.org/case1_datePublished_keyed',
'rules': [{'postconditions': {'slot_conditions': {'doi': {'name': 'doi',
'required': True},
'publishedBy': {'name': 'publishedBy',
'required': True}}},
'preconditions': {'slot_conditions': {'datePublished': {'name': 'datePublished',
'value_presence': 'PRESENT'}}}},
{'postconditions': {'slot_conditions': {'publishedBy': {'name': 'publishedBy',
'value_presence': 'ABSENT'}}},
'preconditions': {'slot_conditions': {'datePublished': {'name': 'datePublished',
'value_presence': 'ABSENT'}}}},
{'postconditions': {'slot_conditions': {'doi': {'name': 'doi',
'value_presence': 'ABSENT'}}},
'preconditions': {'slot_conditions': {'datePublished': {'name': 'datePublished',
'value_presence': 'ABSENT'}}}}]})
datePublished: Optional[str] = Field(default=None, json_schema_extra = { "linkml_meta": {'domain_of': ['Dandiset']} })
publishedBy: Optional[str] = Field(default=None, json_schema_extra = { "linkml_meta": {'domain_of': ['Dandiset']} })
doi: Optional[str] = Field(default=None, json_schema_extra = { "linkml_meta": {'domain_of': ['Dandiset']} })
# Model rebuild
# see https://pydantic-docs.helpmanual.io/usage/models/#rebuilding-a-model
Dandiset.model_rebuild()case2_models.py (from gen-pydantic case2_any_field_keyed.yaml) — only the Dandiset class differs
class Dandiset(ConfiguredBaseModel):
linkml_meta: ClassVar[LinkMLMeta] = LinkMLMeta({'from_schema': 'https://example.org/case2_any_field_keyed',
'rules': [{'postconditions': {'slot_conditions': {'datePublished': {'name': 'datePublished',
'required': True},
'doi': {'name': 'doi',
'required': True},
'publishedBy': {'name': 'publishedBy',
'required': True}}},
'preconditions': {'any_of': [{'slot_conditions': {'datePublished': {'name': 'datePublished',
'value_presence': 'PRESENT'}}},
{'slot_conditions': {'publishedBy': {'name': 'publishedBy',
'value_presence': 'PRESENT'}}},
{'slot_conditions': {'doi': {'name': 'doi',
'value_presence': 'PRESENT'}}}]}}]})
datePublished: Optional[str] = Field(default=None, json_schema_extra = { "linkml_meta": {'domain_of': ['Dandiset']} })
publishedBy: Optional[str] = Field(default=None, json_schema_extra = { "linkml_meta": {'domain_of': ['Dandiset']} })
doi: Optional[str] = Field(default=None, json_schema_extra = { "linkml_meta": {'domain_of': ['Dandiset']} })Full JSON Schema validation matrix (linkml-validate)
# case1_datePublished_keyed.yaml = keyed on datePublished, coherence split one-rule-per-field (CORRECT)
# case2_any_field_keyed.yaml = keyed on ANY of the three fields being present
# case1_naive_coherence_bug.yaml = keyed on datePublished, coherence as a SINGLE rule (buggy translation)
================================================================
SCHEMA: case1_datePublished_keyed.yaml
================================================================
--- all_absent__valid_draft.yaml
No issues found
--- all_present__valid_published.yaml
No issues found
--- datePublished_and_doi__published_missing_publishedBy.yaml
[ERROR] [.../0] 'publishedBy' is a required property in /
--- datePublished_and_publishedBy__published_missing_doi.yaml
[ERROR] [.../0] 'doi' is a required property in /
--- only_datePublished__published_missing_publishedBy_and_doi.yaml
[ERROR] [.../0] 'publishedBy' is a required property in /
[ERROR] [.../0] 'doi' is a required property in /
--- only_doi__stray_no_datePublished.yaml
[ERROR] [.../0] {'doi': '10.48324/dandi.000001/0.210101.0000'} should not be valid under {'required': ['doi']} in /
--- only_publishedBy__stray_no_datePublished.yaml
[ERROR] [.../0] {'publishedBy': 'urn:uuid:publisher'} should not be valid under {'required': ['publishedBy']} in /
--- publishedBy_and_doi__stray_no_datePublished.yaml
[ERROR] [.../0] {'publishedBy': 'urn:uuid:publisher', 'doi': '...'} should not be valid under {'required': ['publishedBy']} in /
[ERROR] [.../0] {'publishedBy': 'urn:uuid:publisher', 'doi': '...'} should not be valid under {'required': ['doi']} in /
================================================================
SCHEMA: case2_any_field_keyed.yaml
================================================================
--- all_absent__valid_draft.yaml
No issues found
--- all_present__valid_published.yaml
No issues found
--- datePublished_and_doi__published_missing_publishedBy.yaml
[ERROR] [.../0] 'publishedBy' is a required property in /
--- datePublished_and_publishedBy__published_missing_doi.yaml
[ERROR] [.../0] 'doi' is a required property in /
--- only_datePublished__published_missing_publishedBy_and_doi.yaml
[ERROR] [.../0] 'publishedBy' is a required property in /
[ERROR] [.../0] 'doi' is a required property in /
--- only_doi__stray_no_datePublished.yaml
[ERROR] [.../0] 'datePublished' is a required property in /
[ERROR] [.../0] 'publishedBy' is a required property in /
--- only_publishedBy__stray_no_datePublished.yaml
[ERROR] [.../0] 'datePublished' is a required property in /
[ERROR] [.../0] 'doi' is a required property in /
--- publishedBy_and_doi__stray_no_datePublished.yaml
[ERROR] [.../0] 'datePublished' is a required property in /
================================================================
SCHEMA: case1_naive_coherence_bug.yaml
================================================================
--- all_absent__valid_draft.yaml
No issues found
--- all_present__valid_published.yaml
No issues found
--- datePublished_and_doi__published_missing_publishedBy.yaml
[ERROR] [.../0] 'publishedBy' is a required property in /
--- datePublished_and_publishedBy__published_missing_doi.yaml
[ERROR] [.../0] 'doi' is a required property in /
--- only_datePublished__published_missing_publishedBy_and_doi.yaml
[ERROR] [.../0] 'publishedBy' is a required property in /
[ERROR] [.../0] 'doi' is a required property in /
--- only_doi__stray_no_datePublished.yaml
No issues found <-- WRONG: stray doi slips through
--- only_publishedBy__stray_no_datePublished.yaml
No issues found <-- WRONG: stray publishedBy slips through
--- publishedBy_and_doi__stray_no_datePublished.yaml
[ERROR] [.../0] {'publishedBy': 'urn:uuid:publisher', 'doi': '...'} should not be valid under {'required': ['publishedBy', 'doi']} in /
Pydantic construction matrix (both generated models) — nothing is rejected
# gen-pydantic does NOT emit validators for rules -> rule violations are NOT caught.
================================================================
MODEL: case1_datePublished_keyed
================================================================
all_absent__valid_draft.yaml -> constructed OK
all_present__valid_published.yaml -> constructed OK
datePublished_and_doi__published_missing_publishedBy.yaml -> constructed OK
datePublished_and_publishedBy__published_missing_doi.yaml -> constructed OK
only_datePublished__published_missing_publishedBy_and_doi.yaml-> constructed OK
only_doi__stray_no_datePublished.yaml -> constructed OK
only_publishedBy__stray_no_datePublished.yaml -> constructed OK
publishedBy_and_doi__stray_no_datePublished.yaml -> constructed OK
================================================================
MODEL: case2_any_field_keyed
================================================================
(identical: all 8 instances constructed OK)
There was a problem hiding this comment.
An issue has been filed related to the bug we discovered in LinkML in the last response.
The `check_publication_status` model-validators on `Dandiset`/`Asset`
previously raised a single combined `ValueError`, which pydantic reports with
an empty `loc` (`()`), losing the per-field association and tripping consumers
that index `error["loc"][0]` (e.g. dandi-archive's `_encode_pydantic_error`).
Build a `list[InitErrorDetails]` instead — one entry per failing check, each
with its own `loc=(field,)`, `type="value_error"`, and the message carried in
`ctx={"error": ...}` — and raise it via
`pydantic_core.ValidationError.from_exception_data(...)`. This keeps the single
accumulating model-validator design (all violations reported together) while
giving every publication error a proper field `loc`, and preserves the exact
"Value error, <message>" text the former field-validators produced.
test_dandimeta_1 now asserts the per-field locs rather than one combined
message.
Co-Authored-By: Claude Code 2.1.198 / Claude Opus 4.8 claude-opus-4-8 <[email protected]>
1ba0cc5 to
30cf2fd
Compare
Summary
Make every model class's
schemaKeyvalue equal its class name, so that an eventual LinkML translation can useschemaKeyas its type designator (designates_type). Three classes violated this (BareAsset→"Asset",PublishedAsset→"Asset",PublishedDandiset→"Dandiset"), and fixing them takes two distinct changes:PublishedDandisetmerges intoDandisetandPublishedAssetintoAsset. Publication requirements now fire via adatePublished-gated validator instead of separate classes.BareAsset.schemaKeyto"BareAsset"Stored
schemaKeyvalues in archive data are unchanged, and the old names remain as deprecated aliases sodandi-cli/dandi-archiveimports keep working. Adopting this release does, however, require one lockstepdandi-clichange (see Follow-ups).What changed (all in
dandischema/models.py)Merge
PublishedDandiset→Dandiset,PublishedAsset→Asset. Publication-only fields (doi,publishedBy,datePublished,releaseNotesonDandiset;publishedBy,datePublishedonAsset) become optional, and the publication requirements move into adatePublished-gatedcheck_publication_statusmodel validator:datePublishedisNone(draft) → publication-only fields must be absent;datePublishedset (published) → enforce the formerPublished*rules (publishedBy/url/doipresence, stricterid/urlpatterns,check_filesbytes,digest_sha256check).dandi-archive injects
datePublishedbefore validating, so the gated checks fire exactly as thePublished*classes did.Per-field validation errors. The publication validator accumulates all violations and raises them via
pydantic_core.ValidationError.from_exception_data(...), so each error carries its own fieldloc(e.g.('assetsSummary',)) plus the usual"Value error, …"message — rather than one model-level error with an emptyloc.Keep
BareAsset(Assetstill inherits it) withschemaKey: Literal["BareAsset"];Assetoverrides it toLiteral["Asset"](a localized# type: ignore[assignment]covers the narrowed override), andensure_schemakeypins each instance's value to its class name.Remove the
Publishablemixin; addPublishedDandiset/PublishedAssetas deprecated aliases.Simplify
ensure_schemakeyto a plainschemaKey == class-namecheck.to_dataciteasserts the published precondition (datePublishedset) thePublishedDandisettype used to guarantee.Why gate on
datePublishedrather than an explicit (context-triggered) validatorThe publish-only checks could instead be triggered explicitly by the caller (e.g. a Pydantic validation-context flag passed by the publish flow) rather than by the presence of
datePublished. We chose data-driven gating because:gen-json-schemaemits it asif/then). A caller-supplied validation mode has no LinkML equivalent — LinkML describes only the data instance, not how validation is invoked — so it would have to live as hand-written Python forever, outside the source of truth, working against the migration this change is meant to enable.validate()and direct construction (Dandiset(**data)/model_validate). A context flag only ridesmodel_validate(..., context=...): the tersePublishedDandiset(**meta)form (used byto_dataciteand in tests) cannot pass one, so it would silently skip the publish checks. Beyond the call-site rewrites that would force, a constructor whose enforcement quietly depends on how it is invoked is a lasting programming hazard.datePublishedpresent ⟺ published is a real invariant: a draft never carries it, and the archive injects it only when building a publishable version. "Published" is a state of the record, not a mode a caller opts into.obj.datePublished is not None; with a context flag, whether an object was validated as published is ephemeral to the call and recorded nowhere on the object.Compatibility
Dandiset/Assetgain only optional,readOnlypublication properties — nothing newly required, no pattern tightened — so the dandi-archive Meditor is unaffected (it hidesreadOnlyprops). Thepublished-*.jsonschemas become the relaxed versions, with publication strictness enforced by the gated validator.PublishedDandiset/PublishedAsset/BareAssetimports continue to work indandi-cli/dandi-archive.Follow-ups (separate PRs)
schemaKey = "Asset"on bare metadata at upload, sinceBareAsset.schemaKeyis now"BareAsset"and the server does not normalize it._encode_pydantic_errorrobustness —_encode_pydantic_errorraises IndexError on validation errors with emptylocdandi-archive#2851. This PR's per-field errors avoid the empty-loccase at the source, but that fix is still worth having as defense-in-depth.views/schema.pymodel mapping).Test plan
tox -e py3(269 passed, 17 skipped — skips are environment-only: noDOI_PREFIX/ instance name not "DANDI")tox -e lint,typingdatePublished-gated publish requirements and the coherence invariant