Some general improvements / discussion

[cpaulik]

This adds three things

1. Allows validation of extensions for collections.
2. Improve error reporting by using best_match to select the error.
3. Add a schema map argument similar to the node-stac-validator schemaMap to allow validating against local copies of schemas during development.

I'm happy to open separate MRs if one of the features is not wanted.
I'm also happy to add tests once we agree that a feature is going to be included.

[cpaulik]

Force pushed a fix for the Mypy issues and updated existing tests as well.

[jonhealy1]

Hi @cpaulik all good ideas here :)

[cpaulik]

Thanks. Tests should hopefully pass after the latest push. I forgot a breakpoint() in the code. 🤦

[cpaulik]

I'll add tests for the schema_map feature next. Still need to figure out if additional tests for the collection extensions functionality are needed. At least one of the tests I adapted seems to do that already.

[cpaulik]

I've added another test for making sure references to sub-schemas also are replaced by the schema_map correctly.
So from my side this is done. Looking forward to the review.

[jonhealy1]

Hi @cpaulik. Looks really good. Just a couple of things. 1. An explanation and an example of a schema map, how to use them, to the readme. 2. An entry in the changelog, highlighting these changes. The changes would be added under the Unreleased section for now. Thanks!

[cpaulik]

Thanks, I added both. Let me know if that is what you had in mind.

[cpaulik]

I did just discover a small bug - stac-validator doesn't seem to validate extensions for collections if recursive mode is used. I'll need to look into that

[cpaulik]

That was just a matter of not keeping all the schemas in the message. That' fixed with the latest commit and handled similar to how it it done with item during recursive validation

[cpaulik]

I did just find another issue. It seems in recursive mode errors are not correctly reported since the default_validator doesn't actually raise a jsonschema.exceptions.ValidationError but just returns a message with valid_stac set to False.

I'll fix that as well.

[cpaulik]

The last commit should fix recursive reporting of errors for collections.

I do have a question about error reporting in the recursive case.
At the moment stac-validator reports all the messages. Wouldn't it be more useful to only show the messages that have valid_stac set to False in case of error?

At the moment in recursive mode we could print out 100s of messages that are valid which makes it hard to find the ones that actually cause the error.

[jonhealy1]

@cpaulik You are right about recursive messaging. People may want some feedback so they can see the validation is working. An optional verbose mode for recursive would probably be the best.

[cpaulik]

the last commit does the following in recursive mode

• if an error is found -> only show the items with errors and set exitcode 1
• if an error is found and verbose mode is on -> show all items and set exitcode 1
• if no error is found show all items and set exitcode 0

Exitcode is unchanged and verbose mode with no errors is also unchanged.

[cpaulik]

I found another bug when validating a catalog with multiple child collections. Previously the StacValidate.valid was not set correctly if the last child collection was valid. I've fixed this and also added a test for it.

[jonhealy1]

Hi. Great work here. What do you think about a config json (maybe containing key value pairs) that could be used instead of specifying numerous --schema-map pairs. It wouldn't replace what you are doing here, it would just be another option.

[jonhealy1]

Maybe there should be a -s option for example, to make it a little easier.

[cpaulik]

Done.

[jonhealy1]

It would be nice to still show the schema here that was validated against,

[cpaulik]

I'm not sure actually. This is an interesting case. This test runs with extensions = True which means only validate extensions if I understand correctly.

The json file we validate against here has an empty list in stac_extensions so in this case it seems correct to not validate against anything.

I've changed this to fall back on the core-validator in this case but I would argue that at least a warning should be logged in this case.

[jonhealy1]

If only extensions are to be validated you are correct. Otherwise, every item validates against the relevant item spec schema by default. You can change this back if you would like.

[jonhealy1]

--extensions is for extensions only, so we should change this back before we merge

[cpaulik]

What do you think about a config json (maybe containing key value pairs) that could be used instead of specifying numerous --schema-map pairs. It wouldn't replace what you are doing here, it would just be another option.

I think it could be useful but I would want to make that a separate MR. We could take some inspiration from the config file from stac-node-validator

[jonhealy1]

Really great work here, thank you!

[jonhealy1]

The config file from stac-node-validator looks really useful. A separate PR would be better.

[cpaulik]

Thanks for merging and making a release.