EqualExportedValues: Handle nested pointer, slice and map fields #1379

HaraldNordgren · 2023-04-24T09:30:07Z

The current implementation of ObjectsExportedFieldsAreEqual does not handle (nested) pointer or slices fields. This means it doesn't work for comparing e.g. any Protobuf struct if there is any of these on the Protobuf message.

Here, I am hoisting the reflect.Kind() checking from inside the loop to earlier in the logic. This allows me to take better advantage of the recursiveness of the function for checking equality for different types of structs. All while still maintaining the original functionality, as seen by the fact that the original tests are not altered.

I added a lot of tests cove different cases. It may seems like a lot of tests for this feature, but I used them when developing and since the reflection logic is pretty brittle in general, they were quite necessary for me to be able to refactor with any kind of confidence.

Thanks a bunch to @mchlp for the original implementation, this function is very needed for using Testify together with Protobuf! 🙌🏻

HaraldNordgren · 2023-04-25T11:34:51Z

Ping @boyan-soubachov @mchlp @feidtmb

mchlp

Just a question about handling of map. Looks good otherwise!

assert/assertions.go

feidtmb

I like this approach! I left a few nits but it looks good to me otherwise.

assert/assertions.go

HaraldNordgren · 2023-04-28T10:41:46Z

@mchlp @feidtmb I added support for maps now, and I think I address all of your comments as well! 🤗

feidtmb

It doesn't look like ObjectsExportedFieldsAreEqual is actually used anymore beyond tests, so we might want to remove it. Other than that though ✅ . (And thanks for adding map support!)

edit: I think a data structure that references itself would cause an infinite loop (example test case from a different PR). IMO this is acceptable for a test helper package like testify, but I think it would be worth noting in the function comment on EqualExportedValues.

mchlp

Looks good! Thanks for adding support for maps - I think this covers all of the composite data types.

HaraldNordgren · 2023-04-29T07:12:52Z

Thanks again!

@feidtmb, I added a comment about infinite recursion to your PR: #1373

HaraldNordgren · 2023-05-05T07:15:02Z

Hi again @boyan-soubachov, would you have some time to look at this?

boyan-soubachov · 2023-05-09T10:30:20Z

assert/assertions.go

+		result := reflect.New(expectedType).Elem()
+		for i := 0; i < expectedType.NumField(); i++ {
+			field := expectedType.Field(i)
+			isExported := field.PkgPath == "" // should use field.IsExported() but it's not available in Go 1.16.5


We probably shouldn't support Go 1.16 any more since it's not officially supported anyways.

Feel free to change the build to only support 1.20, 1.19, and 1.18 -- you can then change this code to not consider Go 1.16

@boyan-soubachov Updated!

Be aware the the auto-formatting rules seem to have changed for Go versions >= 1.19. I'm re-formatting the files and committing the new ones as well.

And we are now skipping that part of the build job for 1.18 because it otherwise fails because of formatting issues.

boyan-soubachov · 2023-05-09T20:51:04Z

.github/workflows/main.yml

-      - run: ./.ci.gogenerate.sh    
-      - run: ./.ci.gofmt.sh
+      - name: Go generate
+        if: ${{ matrix.go_version != 1.18 }}


Why are we excluding gogenerate.sh and gofmt.sh from running if it's Go 1.18?

It's because it failed. I wrote a comment about it here: #1379 (comment)

If we don't want this extra logic, we could also drop support for 1.18, maybe it's old enough at this point?

That's fair -> https://endoflife.date/go

We can drop 1.18 support.

@boyan-soubachov Updated to drop 1.18!

Co-authored-by: Michael Pu <michael.pu123@gmail.com>

boyan-soubachov

Thank you :)

nars1 · 2023-05-15T15:31:10Z

@HaraldNordgren : Hi. We have a Go application that uses the stretchry/testify package. It was working fine on various systems with different go versions. But for the past 4 days, our nightly pipelines have been failing when run with go 1.13.8.

$ go version
go version go1.13.8 linux/amd64

The failure symptom is the following (pasted from https://gitlab.com/YottaDB/Lang/YDBGo/-/jobs/4268493955)

$ go test -short $(go list ./... | grep -v /vendor/)
# github.com/stretchr/testify/assert
/go/src/github.com/stretchr/testify/assert/assertions.go:94:23: field.IsExported undefined (type reflect.StructField has no field or method IsExported)

The same application also fails on a system with go version go1.15.15 linux/amd64 but passes on a system with go version go1.18.1 linux/amd64.

4c93d8f changed assertions.go to add the field.isExported call. But looks like that call is only supported by recent versions of go.

Is this seeming regression expected? Thanks.

HaraldNordgren · 2023-05-15T15:54:58Z

Hi @nars1!

In general I would definitely advise against using the 'master' build of any library. You are better off pinning the version to testify@v1.8.2 and you would avoid this issue.

This repo has been updated to only support Go versions >=1.18.

Be aware the the Go versions you are using are not officially supported anymore: https://endoflife.date/go. It would also be a good idea to upgrade your Go versions 🤗

nars1 · 2023-05-15T16:09:08Z

@HaraldNordgren : Will pin the version like you suggest and try to upgrade our Go versions. Thank you for the prompt response.

nars1 · 2023-05-15T21:28:24Z

@HaraldNordgren : How do I pin the version like you suggest. My project go.mod file looks as follows.

module lang.yottadb.com/go/yottadb

go 1.13

require github.com/stretchr/testify v1.4.0

How do I say don't go more than 1.8.2? I tried searching for pinning but could not find any references. Hoping you can point me in the right direction. Thanks.

@HaraldNordgren

…hr/testify/assert package) Background ---------- * Below is pasted from stretchr/testify#1379 (comment) where I had asked a question on a recent failure we saw in the YDBGo pipelines (and in the YDBTest/go/unit_test subtest failures in in-house systems which had go 1.13 and 1.15). @HaraldNordgren : Hi. We have a Go application that uses the `stretchry/testify` package. It was working fine on various systems with different go versions. But for the past 4 days, our nightly pipelines have been failing when run with go 1.13.8. ```sh $ go version go version go1.13.8 linux/amd64 ``` The failure symptom is the following (pasted from https://gitlab.com/YottaDB/Lang/YDBGo/-/jobs/4268493955) ```sh $ go test -short $(go list ./... | grep -v /vendor/) # github.com/stretchr/testify/assert /go/src/github.com/stretchr/testify/assert/assertions.go:94:23: field.IsExported undefined (type reflect.StructField has no field or method IsExported) ``` The same application also fails on a system with `go version go1.15.15 linux/amd64` but passes on a system with `go version go1.18.1 linux/amd64`. 4c93d8f changed `assertions.go` to add the `field.isExported` call. But looks like that call is only supported by recent versions of go. Is this seeming regression expected? Thanks. * And below was the response (pasted from stretchr/testify#1379 (comment)) In general I would definitely advise against using the 'master' build of any library. You are better off pinning the version to testify@v1.8.2 and you would avoid this issue. This repo has been updated to only support Go versions >=1.18. Be aware the the Go versions you are using are not officially supported anymore: https://endoflife.date/go. It would also be a good idea to upgrade your Go versions. Fix --- * I tried pinning the version like was suggested. I tried updating go.mod as follows. ```diff $ git diff go.mod @@ -4,2 +4,3 @@ go 1.13 -require github.com/stretchr/testify v1.4.0 +require github.com/stretchr/testify v1.8.2 +replace github.com/stretchr/testify => github.com/stretchr/testify v1.8.2 ``` But the YDBGo pipeline still failed as before when it used go 1.13 or go 1.15. * I then tried the following change. ```diff $ git diff go.mod @@ -4,2 +4,3 @@ go 1.13 -require github.com/stretchr/testify v1.8.2 +require github.com/stretchr/testify v1.4.0 +replace github.com/stretchr/testify => github.com/stretchr/testify v1.8.2 ``` This one failed the same way as well. * Finally decided to bump the go version required by the pipeline to 1.18 as suggested above. Searched for all references of `1.13` in the YDBGo repository in the `develop` branch and changed all of them to be `1.18`. In some cases, used `1.18.10` as that was the latest golang version in the 1.18 series. And regenerated `go.mod` using the following steps. This auto-generated go.mod file is included in this commit. ```sh $ cd YDBGo $ echo "module lang.yottadb.com/go/yottadb" > go.mod $ go get -t go: added github.com/davecgh/go-spew v1.1.1 go: added github.com/pmezard/go-difflib v1.0.0 go: added github.com/stretchr/testify v1.8.2 go: added gopkg.in/yaml.v3 v3.0.1 ``` * And in the in-house systems that have go 1.13 (Ubuntu 20.04) and go 1.15 (Debian 11), decided to install the latest Go version using snap instead of using the old go version from the package manager. The install worked fine on x86_64, AARCH64 and ARM32 architectures. With those changes, the `go/unit_tests` subtest passes fine on all those architectures when it previously reliably failed.

SuperQ · 2023-06-02T06:27:58Z

This change has set the Go minimum version hint to 1.20 in go.mod, but continues to test with 1.19. This is confusing for downstream users.

HaraldNordgren · 2023-06-02T13:58:29Z

Hi @SuperQ, it’s testing both for 1.19 and 1.20.

SuperQ · 2023-06-02T14:14:03Z

Fix for go.mod: #1394

This was referenced Apr 24, 2023

ObjectsExportedFieldsAreEqual: Handle pointer and slice fields #1378

Closed

Compare public elements of struct #1309

Merged

HaraldNordgren changed the title ~~EqualExportedValues: Handle pointer and slice fields~~ EqualExportedValues: Handle nested pointer and slice fields Apr 24, 2023

HaraldNordgren mentioned this pull request Apr 25, 2023

EqualExportedValues: Support nested structs and time.Time fields #1373

Closed

mchlp reviewed Apr 26, 2023

View reviewed changes

assert/assertions.go Show resolved Hide resolved

assert/assertions.go Outdated Show resolved Hide resolved

feidtmb reviewed Apr 27, 2023

View reviewed changes

assert/assertions.go Outdated Show resolved Hide resolved

assert/assertions.go Outdated Show resolved Hide resolved

HaraldNordgren changed the title ~~EqualExportedValues: Handle nested pointer and slice fields~~ EqualExportedValues: Handle nested pointer, slice and map fields Apr 28, 2023

HaraldNordgren requested review from mchlp and feidtmb April 28, 2023 10:42

feidtmb previously approved these changes Apr 28, 2023

View reviewed changes

mchlp previously approved these changes Apr 28, 2023

View reviewed changes

boyan-soubachov reviewed May 9, 2023

View reviewed changes

HaraldNordgren dismissed stale reviews from mchlp and feidtmb via 255f82e May 9, 2023 18:04

HaraldNordgren force-pushed the nested_pointer_fields_format_diffs_correctly branch 12 times, most recently from da2f494 to b39be5d Compare May 9, 2023 18:43

HaraldNordgren requested a review from boyan-soubachov May 9, 2023 18:50

boyan-soubachov reviewed May 9, 2023

View reviewed changes

HaraldNordgren force-pushed the nested_pointer_fields_format_diffs_correctly branch from b39be5d to 0a7ed06 Compare May 10, 2023 06:38

HaraldNordgren and others added 6 commits May 10, 2023 08:42

EqualExportedValues: Handle pointer and slice fields

f1a7f27

Update assert/assertions.go

5b2f55b

Co-authored-by: Michael Pu <michael.pu123@gmail.com>

Reduce redundant calls to 'copyExportedFields'

e0e8d46

Update comments

90f0df9

Add support for maps

ca398e3

Update Go version support to 1.19 and onward

74fa1ec

HaraldNordgren force-pushed the nested_pointer_fields_format_diffs_correctly branch from 0a7ed06 to ae7aab5 Compare May 10, 2023 06:43

Re-generate after rebasing

9c3edea

HaraldNordgren force-pushed the nested_pointer_fields_format_diffs_correctly branch from ae7aab5 to 9c3edea Compare May 10, 2023 06:43

HaraldNordgren requested a review from boyan-soubachov May 10, 2023 07:10

boyan-soubachov approved these changes May 10, 2023

View reviewed changes

boyan-soubachov merged commit 4c93d8f into stretchr:master May 10, 2023
2 checks passed

HaraldNordgren deleted the nested_pointer_fields_format_diffs_correctly branch May 11, 2023 11:24

maranqz mentioned this pull request May 19, 2023

Bump github.com/stretchr/testify from 1.8.2 to 1.8.3 avito-tech/go-transaction-manager#64

Closed

HaraldNordgren mentioned this pull request Jul 30, 2023

assert: add NotZeroThenSetZero and NotNilThenSetNil #1440

Closed

zrbecker mentioned this pull request Sep 25, 2023

assert.EqualExportedValues: fix handling of arrays #1473

Merged

lunar-renovate mentioned this pull request Mar 22, 2024

fix(deps): update external go packages lunarway/release-manager-bot#93

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EqualExportedValues: Handle nested pointer, slice and map fields #1379

EqualExportedValues: Handle nested pointer, slice and map fields #1379

HaraldNordgren commented Apr 24, 2023 •

edited

HaraldNordgren commented Apr 25, 2023

mchlp left a comment

feidtmb left a comment

HaraldNordgren commented Apr 28, 2023

feidtmb left a comment •

edited

mchlp left a comment

HaraldNordgren commented Apr 29, 2023

HaraldNordgren commented May 5, 2023

boyan-soubachov May 9, 2023

boyan-soubachov May 9, 2023

HaraldNordgren May 9, 2023 •

edited

boyan-soubachov May 9, 2023

HaraldNordgren May 9, 2023

boyan-soubachov May 9, 2023

HaraldNordgren May 10, 2023

boyan-soubachov left a comment

nars1 commented May 15, 2023

HaraldNordgren commented May 15, 2023 •

edited

nars1 commented May 15, 2023

nars1 commented May 15, 2023

SuperQ commented Jun 2, 2023

HaraldNordgren commented Jun 2, 2023

SuperQ commented Jun 2, 2023

EqualExportedValues: Handle nested pointer, slice and map fields #1379

EqualExportedValues: Handle nested pointer, slice and map fields #1379

Conversation

HaraldNordgren commented Apr 24, 2023 • edited

HaraldNordgren commented Apr 25, 2023

mchlp left a comment

Choose a reason for hiding this comment

feidtmb left a comment

Choose a reason for hiding this comment

HaraldNordgren commented Apr 28, 2023

feidtmb left a comment • edited

Choose a reason for hiding this comment

mchlp left a comment

Choose a reason for hiding this comment

HaraldNordgren commented Apr 29, 2023

HaraldNordgren commented May 5, 2023

boyan-soubachov May 9, 2023

Choose a reason for hiding this comment

boyan-soubachov May 9, 2023

Choose a reason for hiding this comment

HaraldNordgren May 9, 2023 • edited

Choose a reason for hiding this comment

boyan-soubachov May 9, 2023

Choose a reason for hiding this comment

HaraldNordgren May 9, 2023

Choose a reason for hiding this comment

boyan-soubachov May 9, 2023

Choose a reason for hiding this comment

HaraldNordgren May 10, 2023

Choose a reason for hiding this comment

boyan-soubachov left a comment

Choose a reason for hiding this comment

nars1 commented May 15, 2023

HaraldNordgren commented May 15, 2023 • edited

nars1 commented May 15, 2023

nars1 commented May 15, 2023

SuperQ commented Jun 2, 2023

HaraldNordgren commented Jun 2, 2023

SuperQ commented Jun 2, 2023

HaraldNordgren commented Apr 24, 2023 •

edited

feidtmb left a comment •

edited

HaraldNordgren May 9, 2023 •

edited

HaraldNordgren commented May 15, 2023 •

edited