Can't import SendGrid OpenAPI JSON: error converting YAML to JSON: yaml: control characters are not allowed

[BenjaminNolan]

When we attempt to import the SendGrid OpenAPI spec at https://raw.githubusercontent.com/sendgrid/sendgrid-oai/main/oai_v3_stoplight.json using openapi3.NewLoader().LoadFromURI("https://raw.githubusercontent.com/sendgrid/sendgrid-oai/main/oai_v3_stoplight.json"), we get the error in the title. This seems to be caused by the � (U+009F) character at position 14/15 within the string value of the city property of the following JSON object:

{
  "id": "db637d33-bce1-462c-ae9c-91ec4f761de6",
  "email": "t7N5TjDmKhC0@gfdifW.ua",
  "alternate_emails": [
      "gQol@Xcfilli.hc",
      "n4K7OdaVQh@YfsnF.ie",
      "TdnvS3nMStREn@miFjGzNDCPZWhiswJNxrFnOYdUAZEpesQ.yxpu",
      "xRzGDTTzzbYK@eJ.wpgb",
      "iI1rOpx2ct@aZhuYGZBxJLZ.phr"
  ],
  "first_name": "ea et eu",
  "last_name": "velit Ut laborum ipsu",
  "address_line_1": "labore",
  "address_line_2": "non",
  "city": "Ĕȸą�¸ȠɏbɄ",
  "state_province_region": "deserunt dolore",
  "postal_code": -95171713,
  "country": "do",
  "list_ids": [
      "c712288b-2300-4069-bef4-2e05b5948ec3",
      "9003ef29-5eb7-4951-898b-1b102e490d6e"
  ],
  "custom_fields": {
      "custom_field_name1": "amet deserunt mollit",
      "custom_field_name2": "minim consequat id"
  }
}

This is perfectly valid JSON in this form, however the kin-openapi parser errors in func yaml_parser_update_buffer(parser *yaml_parser_t, length int) bool at yaml.v3@v3.0.1/readerc.go:387 and I don't think it should.

If you need any further details, please let me know.

[BenjaminNolan]

I've just done a bit of further testing after submitting this, and it looks like it's rejecting the entire 0x9* range, as the Sendgrid OpenAPI json includes multiple characters in this range, and if I add them as cases in the aforementioned function, I get a new error for each non-handled character.

[BenjaminNolan]

Sorry for multiple comments quickly, but I just realised the receiver of the return at the end of this switch-default doesn't actually print out the offset or int value that are provided, as we had to add an extra fmt.Println() prior to the return to figure out which characters were causing the problem.

switch {
case value == 0x09:
case value == 0x0A:
case value == 0x0D:
case value >= 0x20 && value <= 0x7E:
case value == 0x85:
case value >= 0xA0 && value <= 0xD7FF:
case value >= 0xE000 && value <= 0xFFFD:
case value >= 0x10000 && value <= 0x10FFFF:
default:
  return yaml_parser_set_reader_error(parser,
    "control characters are not allowed",
    parser.offset, int(value))
}

[BenjaminNolan]

Last one, if we add

case value >= 0x90 && value <= 0x9F:

the file imports correctly.

[fenollp]

Hi! I've seen this before and pinned go-yaml/yaml#737 as the issue to track.
Note that kin-openapi uses github.com/invopop/yaml which wraps around go-yaml so this may be easier to fix this issue (I'm thinking using a go mod replace statement until that's upstream).

[BenjaminNolan]

Ah, my bad, I should've noticed that it was in the upstream package, not kin-openapi directly! Thanks for the quick response and the workaround!

[fenollp]

That's alright :)

I don't really know how to move this forward, so if you can help getting this upstream that would be much easier to maintain. I'm thinking a message on the PR or on this issue: go-yaml/yaml#737 (comment)

In the meantime I'll finish the kin-openapi PR.