General refs/planner

[johanfylling]

Fixes: #5995

[netlify]

✅ Deploy Preview for openpolicyagent ready!

Name	Link
🔨 Latest commit	b59b673
🔍 Latest deploy log	https://app.netlify.com/sites/openpolicyagent/deploys/6514151cd5dbb50008ea87f5
😎 Deploy Preview	https://deploy-preview-6235--openpolicyagent.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[srenatus]

There's a potential follow-up wrt the CallDynamic optimization:

With the following rego,

package x
import future.keywords

p.a.b := true
p.d.e := 12

querying echo b | EXPERIMENTAL_GENERAL_RULE_REFS=true opa eval -fpretty -I -d foo.rego 'data.x.p[input].b'
gives us this plan (removed some cruft):

| | *ir.Plan Plan eval (1 blocks)
| | | *ir.Block Block (3 statements)
| | | | *ir.IsDefinedStmt &{Source:Local<0> Location:{File:0 Col:10 Row:1 file:<query> text:input}}
| | | | *ir.AssignVarStmt &{Source:{Value:Local<0>} Target:Local<2> Location:{File:0 Col:10 Row:1 file:<query> text:input}}
| | | | *ir.BlockStmt BlockStmt (1 blocks) &{0 1 1 <query> data.x.p[input].b}
| | | | | *ir.Block Block (5 statements)
| | | | | | *ir.BlockStmt BlockStmt (1 blocks) &{0 0 0  }
| | | | | | | *ir.Block Block (2 statements)
| | | | | | | | *ir.BlockStmt BlockStmt (2 blocks) &{0 0 0  }
| | | | | | | | | *ir.Block Block (2 statements)
| | | | | | | | | | *ir.CallDynamicStmt &{Args:[Local<0> Local<1>] Result:Local<3> Path:[{Value:String<1>} {Value:String<2>} {Value:String<3>} {Value:Local<2>} {Value:String<4>}] Location:{File:0 Col:1 Row:1 file:<query> text:data.x.p[input].b}}
| | | | | | | | | | *ir.BreakStmt &{Index:1 Location:{File:0 Col:1 Row:1 file:<query> text:data.x.p[input].b}}
| | | | | | | | | *ir.Block Block (6 statements)
| | | | | | | | | | *ir.DotStmt &{Source:{Value:Local<1>} Key:{Value:String<2>} Target:Local<4> Location:{File:0 Col:1 Row:1 file:<query> text:data.x.p[input].b}}
| | | | | | | | | | *ir.DotStmt &{Source:{Value:Local<4>} Key:{Value:String<3>} Target:Local<5> Location:{File:0 Col:1 Row:1 file:<query> text:data.x.p[input].b}}
| | | | | | | | | | *ir.DotStmt &{Source:{Value:Local<5>} Key:{Value:Local<2>} Target:Local<6> Location:{File:0 Col:1 Row:1 file:<query> text:data.x.p[input].b}}
| | | | | | | | | | *ir.DotStmt &{Source:{Value:Local<6>} Key:{Value:String<4>} Target:Local<7> Location:{File:0 Col:1 Row:1 file:<query> text:data.x.p[input].b}}
| | | | | | | | | | *ir.AssignVarStmt &{Source:{Value:Local<7>} Target:Local<3> Location:{File:0 Col:1 Row:1 file:<query> text:data.x.p[input].b}}
| | | | | | | | | | *ir.BreakStmt &{Index:1 Location:{File:0 Col:1 Row:1 file:<query> text:data.x.p[input].b}}
| | | | | | | | *ir.BreakStmt &{Index:2 Location:{File:0 Col:0 Row:0 file: text:}}
| | | | | | *ir.AssignVarStmt &{Source:{Value:Local<3>} Target:Local<8> Location:{File:0 Col:1 Row:1 file:<query> text:data.x.p[input].b}}
| | | | | | *ir.MakeObjectStmt &{Target:Local<9> Location:{File:0 Col:10 Row:1 file:<query> text:input}}
| | | | | | *ir.ObjectInsertStmt &{Key:{Value:String<0>} Value:{Value:Local<8>} Object:Local<9> Location:{File:0 Col:10 Row:1 file:<query> text:input}}
| | | | | | *ir.ResultSetAddStmt &{Value:Local<9> Location:{File:0 Col:10 Row:1 file:<query> text:input}}

So since input is a known variable, we dispatch the function call dynamically, with it as part of the Path of CallDynamic,

Path:[{Value:String<1>} {Value:String<2>} {Value:String<3>} {Value:Local<2>} {Value:String<4>}]
                                                            ^^^^^^^^^^^^^^^^

Now, if we make the rego file

package x
import future.keywords

p.a.b := true
p.d.e := 12
p[y].c if y := "b"

we get this plan,

| | *ir.Plan Plan eval (1 blocks)
| | | *ir.Block Block (9 statements)
| | | | *ir.IsDefinedStmt &{Source:Local<0> Location:{File:0 Col:10 Row:1 file:<query> text:input}}
| | | | *ir.AssignVarStmt &{Source:{Value:Local<0>} Target:Local<2> Location:{File:0 Col:10 Row:1 file:<query> text:input}}
| | | | *ir.CallStmt &{Func:g0.data.x.p Args:[{Value:Local<0>} {Value:Local<1>}] Result:Local<3> Location:{File:0 Col:1 Row:1 file:<query> text:data.x.p[input].b}}
| | | | *ir.DotStmt &{Source:{Value:Local<3>} Key:{Value:Local<2>} Target:Local<4> Location:{File:0 Col:1 Row:1 file:<query> text:data.x.p[input].b}}
| | | | *ir.DotStmt &{Source:{Value:Local<4>} Key:{Value:String<2>} Target:Local<5> Location:{File:0 Col:1 Row:1 file:<query> text:data.x.p[input].b}}
| | | | *ir.AssignVarStmt &{Source:{Value:Local<5>} Target:Local<6> Location:{File:0 Col:1 Row:1 file:<query> text:data.x.p[input].b}}
| | | | *ir.MakeObjectStmt &{Target:Local<7> Location:{File:0 Col:10 Row:1 file:<query> text:input}}
| | | | *ir.ObjectInsertStmt &{Key:{Value:String<0>} Value:{Value:Local<6>} Object:Local<7> Location:{File:0 Col:10 Row:1 file:<query> text:input}}
| | | | *ir.ResultSetAddStmt &{Value:Local<7> Location:{File:0 Col:10 Row:1 file:<query> text:input}}

where g0.data.x.p has three bodies, one per rule body underneath data.x.p.

Now, it's absolutely non-obvious how to make that happen, so this -- for now -- might just be something we need to be aware of and keep in mind; not a blocker. What do you think?

[srenatus]

👏 But let's also remove the comments 😉

[johanfylling]

Oh, I totally forgot about those 😅. Will remove 👍.

[srenatus]

👏 Let's get this merged.

[johanfylling]

where g0.data.x.p has three bodies, one per rule body underneath data.x.p.

In previous versions of OPA, you see the same behavior with partial object "overrides":

package x
import future.keywords

p.a := true
p.d := 12
p[y] if y := "b"

-->

| *ir.Funcs Funcs (1 funcs)
| | *ir.Func g0.data.x.p (2 params: [Local<0> Local<1>], 5 blocks, path: [g0 x p])
| | | *ir.Block Block (1 statements)
| | | | *ir.MakeObjectStmt &{Target:Local<2> Location:{File:0 Col:1 Row:6 file:module-0.rego text:p[y]}}
| | | *ir.Block Block (4 statements)
| | | | *ir.ResetLocalStmt &{Target:Local<3> Location:{File:0 Col:1 Row:6 file:module-0.rego text:p[y]}}
| | | | *ir.CallStmt &{Func:assign Args:[{Value:Local<4>} {Value:String<0>}] Result:Local<5> Location:{File:0 Col:9 Row:6 file:module-0.rego text:y := "b"}}
| | | | *ir.NotEqualStmt &{A:{Value:Local<5>} B:{Value:Bool<false>} Location:{File:0 Col:9 Row:6 file:module-0.rego text:y := "b"}}
| | | | *ir.ObjectInsertOnceStmt &{Key:{Value:Local<4>} Value:{Value:Bool<true>} Object:Local<2> Location:{File:0 Col:1 Row:6 file:module-0.rego text:p[y]}}
| | | *ir.Block Block (4 statements)
| | | | *ir.ResetLocalStmt &{Target:Local<3> Location:{File:0 Col:1 Row:5 file:module-0.rego text:p.d := 12}}
| | | | *ir.NotEqualStmt &{A:{Value:Bool<true>} B:{Value:Bool<false>} Location:{File:0 Col:8 Row:5 file:module-0.rego text:12}}
| | | | *ir.MakeNumberRefStmt &{Index:2 Target:Local<4> Location:{File:0 Col:1 Row:5 file:module-0.rego text:p.d := 12}}
| | | | *ir.ObjectInsertOnceStmt &{Key:{Value:String<1>} Value:{Value:Local<4>} Object:Local<2> Location:{File:0 Col:1 Row:5 file:module-0.rego text:p.d := 12}}
| | | *ir.Block Block (3 statements)
| | | | *ir.ResetLocalStmt &{Target:Local<3> Location:{File:0 Col:1 Row:4 file:module-0.rego text:p.a := true}}
| | | | *ir.NotEqualStmt &{A:{Value:Bool<true>} B:{Value:Bool<false>} Location:{File:0 Col:8 Row:4 file:module-0.rego text:true}}
| | | | *ir.ObjectInsertOnceStmt &{Key:{Value:String<3>} Value:{Value:Bool<true>} Object:Local<2> Location:{File:0 Col:1 Row:4 file:module-0.rego text:p.a := true}}
| | | *ir.Block Block (1 statements)
| | | | *ir.ReturnLocalStmt &{Source:Local<2> Location:{File:0 Col:1 Row:6 file:module-0.rego text:p[y]}}

There might be an opportunity to do some optimization here, though. Conflict resolution might still force us to always execute blocks for the rules with variables in their ref (we want to error whenever x.p[y] produces a y equal to e.g. "a"); but if you query for data.x.p.a, maybe we only need to evaluate the blocks for x.p.a and x.p[y], and can skip x.p.d 🤔 . In topdown, pre-unification of the ref would "skip" any conflicts in x.p[y] with x.p.d, right? So if I'm not completely making a fool out of myself, such an optimization wouldn't even be a cause of behavioral conflict between the IR and topdown ..

[srenatus]

In topdown, pre-unification of the ref would "skip" any conflicts in x.p[y] with x.p.d, right? So if I'm not completely making a fool out of myself, such an optimization wouldn't even be a cause of behavioral conflict between the IR and topdown ..

It's been more than a week since I've had a look, so I don't know 😅

BTW it's a good point, I actually messed this up when introducing string prefixes 👍