Runtime error - Can't post issue, so here it is

[sarimarton]

Hey guys,

There's a runtime error occurring in yq, and I can't post an issue because the 'Submit' button is disabled, I don't have time to figure out. I guess you're interested in runtime errors, otherwise sorry for bothering ¯_(ツ)_/¯

So here is the issue:

Describe the bug

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x28 pc=0x1048e4d60]

goroutine 1 [running]:
container/list.(*List).Len(...)
        container/list/list.go:66
github.com/mikefarah/yq/v4/pkg/yqlib.variableLoopSingleChild(0x14000158ea0?, {0x140002bad20, 0x140002badb0, 0x0, {0x0, 0x0}}, 0x140000ad728)
        github.com/mikefarah/yq/v4/pkg/yqlib/operator_variables.go:85 +0x730
github.com/mikefarah/yq/v4/pkg/yqlib.variableLoop(0x140000e3018?, {0x140002ba150, 0x140002bacc0, 0x0, {0x0, 0x0}}, 0x140000e2fb8?)
        github.com/mikefarah/yq/v4/pkg/yqlib/operator_variables.go:43 +0x268
github.com/mikefarah/yq/v4/pkg/yqlib.pipeOperator(0x14000158ea0?, {0x140002ba150, 0x140002bacc0, 0x0, {0x0, 0x0}}, 0x0?)
        github.com/mikefarah/yq/v4/pkg/yqlib/operator_pipe.go:6 +0x1d4
github.com/mikefarah/yq/v4/pkg/yqlib.(*dataTreeNavigator).GetMatchingNodes(0x104a2b820?, {0x140002ba150, 0x140002bacc0, 0x0, {0x0, 0x0}}, 0x140000ad728)
        github.com/mikefarah/yq/v4/pkg/yqlib/data_tree_navigator.go:37 +0x1e0
github.com/mikefarah/yq/v4/pkg/yqlib.variableLoopSingleChild(0x14000158ea0?, {0x140002ba150, 0x140002ba1e0, 0x0, {0x0, 0x0}}, 0x140000ad740)
        github.com/mikefarah/yq/v4/pkg/yqlib/operator_variables.go:84 +0x6e4
github.com/mikefarah/yq/v4/pkg/yqlib.variableLoop(0x140000e36d8?, {0x140002ba0c0, 0x0, 0x0, {0x0, 0x0}}, 0x140000e3688?)
        github.com/mikefarah/yq/v4/pkg/yqlib/operator_variables.go:43 +0x268
github.com/mikefarah/yq/v4/pkg/yqlib.pipeOperator(0x14000158ea0?, {0x140002ba0c0, 0x0, 0x0, {0x0, 0x0}}, 0x0?)
        github.com/mikefarah/yq/v4/pkg/yqlib/operator_pipe.go:6 +0x1d4
github.com/mikefarah/yq/v4/pkg/yqlib.(*dataTreeNavigator).GetMatchingNodes(0x0?, {0x140002ba0c0, 0x0, 0x0, {0x0, 0x0}}, 0x140000ad740)
        github.com/mikefarah/yq/v4/pkg/yqlib/data_tree_navigator.go:37 +0x1e0
github.com/mikefarah/yq/v4/pkg/yqlib.(*streamEvaluator).Evaluate(0x140002c9c80, {0x16b84f37b, 0x1}, {0x104a81828?, 0x1400009eea0?}, 0x104925579?, {0x104a83c20, 0x1400009ede0}, {0x104a82ad8, 0x140002a5490})
        github.com/mikefarah/yq/v4/pkg/yqlib/stream_evaluator.go:108 +0x28c
github.com/mikefarah/yq/v4/pkg/yqlib.(*streamEvaluator).EvaluateFiles(0x104926134?, {0x16b84eeda, 0x4a0}, {0x140000a0e50, 0x1, 0x10491dddc?}, {0x104a83c20, 0x1400009ede0}, {0x104a82ad8, 0x140002a5490})
        github.com/mikefarah/yq/v4/pkg/yqlib/stream_evaluator.go:65 +0x108
github.com/mikefarah/yq/v4/cmd.evaluateSequence(0x14000160300?, {0x140000a0e40, 0x2, 0x4})
        github.com/mikefarah/yq/v4/cmd/evalute_sequence_command.go:135 +0x6c0
github.com/spf13/cobra.(*Command).execute(0x14000160300, {0x140000a0e00, 0x4, 0x4})
        github.com/spf13/cobra@v1.6.1/command.go:916 +0x5c4
github.com/spf13/cobra.(*Command).ExecuteC(0x14000160000)
        github.com/spf13/cobra@v1.6.1/command.go:1044 +0x340
github.com/spf13/cobra.(*Command).Execute(0x140000969d0?)
        github.com/spf13/cobra@v1.6.1/command.go:968 +0x1c
main.main()
        github.com/mikefarah/yq/v4/yq.go:22 +0x1a4
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x28 pc=0x100a00d60]

Version of yq: 4.31.2
Operating system: mac
Installed via: homebrew

Description:

yq produces a runtime error with a certain input. Please find the cmd line below, note that if date= is a valid past date (e.g. date=2023-03-10'), it works. Obviously with a wrong date the webpage will be out of the desired shape, but it shouldn't (and up until recently didn't) produce a runtime error.

Input command:

curl --silent "https://www.xe.com/currencytables/?from=HUF&date=asdf" \
| sed -n '1h;1!H;${;g;s/<head.*<\/head>//g;p;}' \
| yq -p xml '
  (
    "EUR|USD"
    | split("|")
  ) as $ccys
  | ($ccys | map({ "key": ., "value": null }) | from_entries) as $ccysDefaultMap
  | .html.body.div[0].div[1].div[2].section.div[1].div.table as $table
  | $table.tbody.tr
' -

It also produces a runtime error if I don't reference a declared variable, no matter of what the input is:

curl --silent "https://www.xe.com/currencytables/?from=HUF&date=2023-03-10" \
| sed -n '1h;1!H;${;g;s/<head.*<\/head>//g;p;}' \
| yq -p xml '
  (
    "EUR|USD"
    | split("|")
  ) as $ccys
  | ($ccys | map({ "key": ., "value": null }) | from_entries) as $ccysDefaultMap
  | .html.body.div[0].div[1].div[2].section.div[1].div.table as $table
' -

[mikefarah]

Hi - thanks for raising this - not sure why you can't submit an issue!

Found the bug - it's caused by a log statement when handling another error; will fix in the next release.

The error yq is trying to show you is:

Error: must use variable with a pipe, e.g. `exp as $x | ...`

I recently updated variable handling to work more like jq - so you will need to add | . to proceed with your expression:

| sed -n '1h;1!H;${;g;s/<head.*<\/head>//g;p;}' \
| yq -p xml '
  (
    "EUR|USD"
    | split("|")
  ) as $ccys
  | ($ccys | map({ "key": ., "value": null }) | from_entries) as $ccysDefaultMap
  | .html.body.div[0].div[1].div[2].section.div[1].div.table as $table | .' -

[mikefarah]

Created an issue here to track: #1600

[mikefarah]

Fixed in 4.32.1