go-rod panics with iFrames due to null pointer dereference

[NuLL3rr0r]

Rod Version: v0.114.3

I am getting the following null pointer dereference from go-rod:

ERROR 2023/09/29 19:36:19 error value: "invalid memory address or nil pointer dereference"
goroutine 7 [running]:
runtime/debug.Stack()
	/usr/lib/go/src/runtime/debug/stack.go:24 +0x5e
github.com/go-rod/rod.Try.func1()
	/path/to/my/project/cmd/tw-login/vendor/github.com/go-rod/rod/utils.go:213 +0x3d
panic({0x8758c0?, 0xf32550?})
	/usr/lib/go/src/runtime/panic.go:914 +0x21f
github.com/go-rod/rod.(*Page).getJSCtxID(0xc0002b8790)
	/path/to/my/project/cmd/tw-login/vendor/github.com/go-rod/rod/page_eval.go:349 +0x113
github.com/go-rod/rod.(*Page).ensureJSHelper(0x20?, 0xf36fc0)
	/path/to/my/project/cmd/tw-login/vendor/github.com/go-rod/rod/page_eval.go:248 +0x32
github.com/go-rod/rod.(*Page).formatArgs(0x17?, 0x180?)
	/path/to/my/project/cmd/tw-login/vendor/github.com/go-rod/rod/page_eval.go:233 +0x1f1
github.com/go-rod/rod.(*Page).evaluate(0xc0004236f8?, 0xc0002a69c0)
	/path/to/my/project/cmd/tw-login/vendor/github.com/go-rod/rod/page_eval.go:149 +0x2f
github.com/go-rod/rod.(*Page).Evaluate(0xc0002b8790, 0xc0002a69c0)
	/path/to/my/project/cmd/tw-login/vendor/github.com/go-rod/rod/page_eval.go:128 +0x4a
github.com/go-rod/rod.(*Page).ElementByJS.func2()
	/path/to/my/project/cmd/tw-login/vendor/github.com/go-rod/rod/query.go:172 +0x9c
github.com/go-rod/rod/lib/utils.Retry({0xc77210, 0xc00039a2d0}, 0xc00039a300, 0xc0004237a0)
	/path/to/my/project/cmd/tw-login/vendor/github.com/go-rod/rod/lib/utils/sleeper.go:139 +0x31
github.com/go-rod/rod.(*Page).ElementByJS(0xc0002b8790, 0xc0002a69c0)
	/path/to/my/project/cmd/tw-login/vendor/github.com/go-rod/rod/query.go:167 +0xba
github.com/go-rod/rod.(*Page).ElementX(0xc0002b86e0?, {0x951e40?, 0x10?})
	/path/to/my/project/cmd/tw-login/vendor/github.com/go-rod/rod/query.go:155 +0x19e
babaei.net/MamadouArchivesBackend/internal/bot.Bot.Unlock.func4()
	/path/to/my/project/cmd/tw-login/vendor/babaei.net/MamadouArchivesBackend/internal/bot/lib.go:504 +0x48
github.com/go-rod/rod.Try(0xc0001e56b0?)
	/path/to/my/project/cmd/tw-login/vendor/github.com/go-rod/rod/utils.go:217 +0x57
babaei.net/MamadouArchivesBackend/internal/bot.Bot.Unlock({{0xc0000c2900, 0xe}, {0xc0000c290e, 0x18}, {0xc0000c2926, 0x20}, {{0xc0000c2970, 0xd}, 0x50, {0xc0000c2981, ...}, ...}, ...})
	/path/to/my/project/cmd/tw-login/vendor/babaei.net/MamadouArchivesBackend/internal/bot/lib.go:502 +0x5f2
main.loginBot({{0xc0000c2900, 0xe}, {0xc0000c290e, 0x18}, {0xc0000c2926, 0x20}, {{0xc0000c2970, 0xd}, 0x50, {0xc0000c2981, ...}, ...}, ...})
	/path/to/my/project/cmd/tw-login/main.go:125 +0x238
main.loginBots.func1()
	/path/to/my/project/cmd/tw-login/main.go:101 +0x1fe
created by main.loginBots in goroutine 1
	/path/to/my/project/cmd/tw-login/main.go:89 +0xdf

This is the kind of code that triggers it:

	var arkoseFrame *rod.Element = nil
	err = rod.Try(func() {
		arkoseFrame, err = bot.Page.Timeout(time.Second * 8).
			ElementX(`//iframe[@id='arkose_iframe']`)
	})
	if errors.Is(err, context.DeadlineExceeded) {
		jww.WARN.Println("Getting the Arkose frame has timed out!")
	} else if err != nil {
		jww.ERROR.Println(err)
		return false
	} else if arkoseFrame != nil {
		err = arkoseFrame.WaitStable(time.Second * 4)
		if err != nil {
			jww.ERROR.Println(err)
			return false
		}
	}

	var arkosePage *rod.Page = nil
	if arkoseFrame != nil {
		arkosePage, err = arkoseFrame.Frame()
		if err != nil {
			jww.ERROR.Println(err)
			return false
		}
	}

	if arkosePage == nil {
		jww.WARN.Println(fmt.Sprintf(
			"Failed to get the Arkose Labs iframe for '%s'...", bot.ScreenName))
		return false
	}

	var verificationChallengeFrame *rod.Element = nil
	err = rod.Try(func() {
		verificationChallengeFrame, err = arkosePage.Timeout(time.Second * 8).
			ElementX(`//iframe[@title='Verification challenge']`)
	})
	if errors.Is(err, context.DeadlineExceeded) {
		jww.WARN.Println("Getting the verificationChallenge frame has timed out!")
	} else if err != nil {
		jww.ERROR.Println(err)
		return false
	} else if verificationChallengeFrame != nil {
		err = verificationChallengeFrame.WaitStable(time.Second * 4)
		if err != nil {
			jww.ERROR.Println(err)
			return false
		}
	}

Line 504 in internal/bot/lib.go:504 +0x48 is:

			ElementX(`//iframe[@title='Verification challenge']`)

in my code which is the last line from my code before the panic from go-rod.

[github-actions]

Please fix the format of your markdown:

105 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```"]

_{generated by check-issue}

[achsal2]

i'm having the same issue , simple as :

browser := launchBrowser()

page := stealth.MustPage(browser)

page.MustNavigate(pageURL)

frame := page.MustElement("iframe:not([data-hcaptcha-response])").MustFrame()

println(frame.HTML())

throws this error :

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x10 pc=0x100b333]

goroutine 1 [running]:
github.com/go-rod/rod.(*Page).getJSCtxID(0xc000216210)
        C:/Users/hp/go/pkg/mod/github.com/go-rod/rod@v0.114.3/page_eval.go:349 +0x113
github.com/go-rod/rod.(*Page).ensureJSHelper(0x106182e?, 0x156d040)
        C:/Users/hp/go/pkg/mod/github.com/go-rod/rod@v0.114.3/page_eval.go:248 +0x32 
github.com/go-rod/rod.(*Page).formatArgs(0x16?, 0xc0?)
        C:/Users/hp/go/pkg/mod/github.com/go-rod/rod@v0.114.3/page_eval.go:233 +0x1f1
github.com/go-rod/rod.(*Page).evaluate(0xc0001cfd58?, 0xc0002a0440)
        C:/Users/hp/go/pkg/mod/github.com/go-rod/rod@v0.114.3/page_eval.go:149 +0x2f 
github.com/go-rod/rod.(*Page).Evaluate(0xc000216210, 0xc0002a0440)
        C:/Users/hp/go/pkg/mod/github.com/go-rod/rod@v0.114.3/page_eval.go:128 +0x4a 
github.com/go-rod/rod.(*Page).ElementByJS.func2()
        C:/Users/hp/go/pkg/mod/github.com/go-rod/rod@v0.114.3/query.go:172 +0x9c
github.com/go-rod/rod/lib/utils.Retry({0x131e528, 0xc0000d6190}, 0xc000486c90, 0xc0001cfe00)
        C:/Users/hp/go/pkg/mod/github.com/go-rod/rod@v0.114.3/lib/utils/sleeper.go:139 +0x31
github.com/go-rod/rod.(*Page).ElementByJS(0xc000216210, 0xc0002a0440)
        C:/Users/hp/go/pkg/mod/github.com/go-rod/rod@v0.114.3/query.go:167 +0xba
github.com/go-rod/rod.(*Page).Element(0xc000216210?, {0x112336e?, 0x0?})
        C:/Users/hp/go/pkg/mod/github.com/go-rod/rod@v0.114.3/query.go:143 +0x19e
github.com/go-rod/rod.(*Page).HTML(0xc000486660?)
        C:/Users/hp/go/pkg/mod/github.com/go-rod/rod@v0.114.3/page.go:106 +0x1f
main.main()
        C:/work/projects/go_scraper/scraper.go:164 +0x73
exit status 2

it does get the right element with just MustElement but once i use MustFrame to be able to get nested elements it throws the error

[achsal2]

works in headless mode but panics with show

[NuLL3rr0r]

Ah, thanks for mentioning that. I was also developing so, I disabled the headless mode. Probably, I could use the screenshot feature to see what's going on.

[ysmood]

I think it's related to #548

[achsal2]

yes i think its the same issue, tho i saw you confirm the issue happens in puppeteer too but in puppeteer it worked fine for me, im rewriting my nodejs scraper in go in the nodejs it worked fine

[zplzpl]

i'm having the same issue.

[achsal2]

for anyone still having issue, what seems to work right now is headless mode, sucks that you're forced to debug anything post iframe with console logs and traces but at least that works