Repository navigation
RefResolver (or its successor) should allow customizing deserialization, not only scheme / retrieval #4
Description
I'm using currently handling schemas written in YAML (for human-readability) but validated by jsonschema. I parse the schema in YAML (and make sure I subset my YAML to stay JSON-compatible) and then run the parsed dictionary through jsonschema. It is working nicely with one issue: When I want to use file:// includes, the files are implicitly assumed to be json.
@Julian I would like to add support for these files being YAML and am wondering if this is something you would support. If so, what is your recommendation for the cleanest way to add it? If it's not something you support, what would you recommend as the cleanest way to add the support for my own code without hacking/forking jsonschema? I thought about adding custom resolver as is done in python-jsonschema/jsonschema#225, but it sounds like RefResolver is not intended to be API-stable, so subclassing is liable to break in the future. Is there another way?
Thanks,
Martin
Activity
martingkelly commentedon Jun 5, 2018
Upon further code reading, I found the
handlersargument toRefResolver. If I attach ayaml://handler (or similar), I can make this work. That said, it feels hacky, in thatyaml://is not really a location (likefile://orhttp://) and it would be better to decouple the file format (YAML/JSON) from the location (including the logic for handling base URIs and $id).robherring commentedon Jun 5, 2018
@martingkelly I'm doing the same thing with YAML and have $ref's working. See here: https://github.com/robherring/yaml-bindings/blob/master/dtschema.py#L212
martingkelly commentedon Jun 5, 2018
Thanks @robherring, I ended up doing the same thing. I think it's not ideal in that by implementing your own handler, you lose the more general caching logic, and you lose the ability to do JSON http... ideally the document format and transport would be better decoupled.
Btw, I really like your device-tree YAML project; I think it's an excellent idea :).
handrews commentedon Jun 5, 2018
You can see one approach to this problem in JavaScript in the AutoExtensionDereferencer function of JSON Schema Tools. Although the "dereferencer" part is not necessarily relevant here, just the automatic file extension search approach, which also understands overriding the URI scheme to load from a file as if it were a
file://URI.martingkelly commentedon Jun 5, 2018
Yes, something like that would work. @Julian any thoughts?
Julian commentedon Jun 7, 2018
Have to respond in a bit more length -- something a bit more general than that (that doesn't use file extensions, just fully allows someone to inject in what to deserialize to what) is what I'd lean to, though this also strikes me as the sort of thing I wish existed in an external spec -- i.e. within HTTP, if you were able to say
yaml+<URL>and that'd mean "what lives there at that URL should be treated as YAML".To some extent that's
Accept/Content-Type, but it's not used that way by client libraries right, so something that says "I'm JSON" -- there's no HTTP client (for Python) that will then say "OK I'll automatically know to call json.loads on that body" that I know of (and to allow you to basically hook into any possibleContent-Typevalue).martingkelly commentedon Jun 7, 2018
I agree with that ideally being in spec but don't know enough to say where that would fit in, if at all. @handrews any opinion on integrating this sort of thing into spec?
Julian commentedon Jun 7, 2018
Julian commentedon Jun 7, 2018
martingkelly commentedon Jun 7, 2018
I realize that would be more ideal, but I'm guessing it's out of scope and would take years to modify HTTP or URI specs. I'm wondering if there's any place for jsonschema to graft something in, even if not ideal.
handrews commentedon Jun 9, 2018
@martingkelly it really does not fit in the JSON Schema spec. For one thing, it's much better to allow implementations flexibility in how they manage this. Some implementations will be able to support rather complex loading/parsing features, others will be resource- or performance-constrained and will only want to support JSON.
Additionally, this is a pretty easy problem to solve outside of implementations- write your schemas in whatever you want, and have a build step that converts them to JSON.
martingkelly commentedon Jun 11, 2018
Although a build step that converts JSON to YAML is technically possible, I don't think it's a great solution, as all the file references have to somehow be faked out and turned into JSON files temporarily, or you have to copy the build directory somewhere else and convert the files into JSON (but keep the file extension of
.yamleven though the file is JSON). If would much prefer the code itself natively supporting YAML. I can add this as ajsonschemaextension using ahandler, but it has the drawbacks I mentioned above.Julian commentedon Jun 11, 2018
Still haven't gotten a chance to elaborate here, but just on that last message -- just in case you weren't aware, all JSON is essentially valid YAML.
martingkelly commentedon Jun 11, 2018
I do know that :). In this case, I'm writing schemas in YAML (json-compatible subset) and parsing them with
jsonschema. I like this approach because I find YAML much more readable than JSON, but I findjsonschemato be a useful tool.handrews commentedon Jun 12, 2018
@martingkelly why would you try to do anything other than edit the files in YAML? The build step would happen before you try to load anything, so you shouldn't need to refer directly to the YAML paths.
3 remaining items
handrews commentedon Jun 12, 2018
The spec already makes a point of saying that it interprets instances according to a data model rather than as JSON text. This holds true when the schema is an instance (with respect to the meta-schema).
So the spec already does not care how you get something into the data model, although it only defines the mapping for JSON.
What the spec avoids doing is placing requirements on such support. There are many reasons that an implementation may not want to support other formats. JSON was designed to be simple and fast to parse (which, as I understand it, is why there are some weird corner cases like duplicate properties when parsing- leaving them undefined allows a faster implementation than requiring detecting the error and taking action on it).
For that matter, an Internet of Things implementation might not even load JSON, it might only load CBOR.
martingkelly commentedon Jun 12, 2018
I understand JSON schema is not unique to JSON (despite the name), but since it's not specified how a parser should interpret a given
$ref, each library will invent their own syntax for specifying content type. That seems non-ideal to me.martingkelly commentedon Jun 12, 2018
I stupidly didn't see
contentMediaTypebefore in the spec, but what if we check for that key and, if specified, assume that a given$refis not JSON but is instead the specified media type?handrews commentedon Jun 12, 2018
contentMediaTypeis for media types encoded as JSON strings, so I don't think it helps here?Per section 8.3.1,
$refvalues are identifiers, and do not say anything at all about how the identified schema is loaded.If it is being loaded over a network, then the network protocol (probably HTTP or CoAP) should be used to understand and handle the available media type(s). There is no need for JSON Schema to say anything about that process- it is how the web works already. For something like
file://URIs, while it is less clear how to detect and manage media types, that problem is not at all specific to JSON Schema.If it has been pre-loaded (which is a common approach, as it separates parsing concerns from referencing concerns), then presumably it was parsed into an in-memory representation already and exactly how that happens and what formats are supported is intentionally left to implementations. Different implementations serve different needs, and can balance flexibility vs performance/resource consumption.
handrews commentedon Jun 12, 2018
The idea of treating
$refas an RFC 8288 web link and allowing target hints has come up at least once before (with OpenAPI), but in a somewhat different context. There is https://github.com/json-schema-org/json-schema-spec/issues/141, which gets into talking about the type of reference/pointer targets (initially just for JSON Pointer, but the discussion expanded along the way). That might be a place to look if you want to advocate for something like this in the spec. We shouldn't hijack @Julian's project for spec proposals :-)martingkelly commentedon Jun 12, 2018
Yes, in my case I'm dealing with only local file references, no network at all. I think having some way to specify a mime-type or similar would be useful... I frankly don't care if it's in the spec or not as long as it solves my problem :). It's perhaps hacky, but having an optional
$mime-typekey in parallel with$refwould solve the problem. You could then pair a parser to a mime type.[-]Handling YAML $refs[/-][+]RefResolver 9[/+][-]RefResolver 9[/-][+]RefResolver (or its successor) should allow customizing deserialization, not only scheme / retrieval[/+]Squashed 'json/' changes from ea415537..86f52b87
Julian commentedon Feb 23, 2023
Hello there!
This, along with many many other
$ref-related issues, is now finally being handled in python-jsonschema/jsonschema#1049 with the introduction of this new referencing library which is fully compliant and has APIs which I hope are a lot easier to understand and customize. (At some point recently I'd transferred this issue to the issue tracker for referencing, in case that wasn't clear).The next release of
jsonschema(v4.18.0) will contain a merged version of that PR, and should be released shortly in beta, and followed quickly by a regular release, assuming no critical issues are reported.Dereferencing YAML, TOML, or whatever else is definitely doable using
referencing. If you still care to, I'd love it if you tried out the beta once it is released, or certainly it'd be hugely helpful to immediately install the branch containing this work (https://github.com/python-jsonschema/jsonschema/tree/referencing) and confirm. You can in the interim find documentation for the change in a preview page here. A section specifically on resolving YAML is here and can be combined of course with the example shown there on retrieving remote resources over HTTP if so desired.I'm going to close this given it indeed seems like it is addressed by python-jsonschema/jsonschema#1049, but feel free to follow up with any comments. Sorry for the delay in getting to these, but hopefully this new release will bring lots of benefit!