Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

promtail: journal scrape not working for aarch64 / rpi4 #3522

Closed
AngellusMortis opened this issue Mar 21, 2021 · 3 comments
Closed

promtail: journal scrape not working for aarch64 / rpi4 #3522

AngellusMortis opened this issue Mar 21, 2021 · 3 comments

Comments

@AngellusMortis
Copy link

Describe the bug
I am unable to enable the journal scrape on an aarch64 install of Linux on a RPI (archlinuxarm).

Cross-posted to archlinuxarm forum as well since I am not sure where the fix would exist at:
https://archlinuxarm.org/forum/viewtopic.php?f=15&t=15235

To Reproduce
Steps to reproduce the behavior:

  1. Install archlinuxarm (aarch64) on RPI4: https://archlinuxarm.org/platforms/armv8/broadcom/raspberry-pi-4
  2. Install promtail: pacman -S promtail
  3. Add journal scrape config
  4. Started Promtail (v2.2.0 / a27c799)

Expected behavior
Promtail start as expected and sends logs for journald.

Environment:

  • RPI4 + archlinuxarm (aarch64)

Screenshots, Promtail config, or terminal output

Mar 20 15:29:16 alarm promtail[243566]: level=info ts=2021-03-20T19:29:16.163114189Z caller=main.go:110 msg="Starting Promtail" version="(version=2.1.0, branch=, revision=)"
Mar 20 15:29:17 alarm promtail[243566]: unexpected fault address 0x6c62767862ea9f
Mar 20 15:29:17 alarm promtail[243566]: fatal error: fault
Mar 20 15:29:17 alarm promtail[243566]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x6c62767862ea9f pc=0xaaaaaec746d0]
Mar 20 15:29:17 alarm promtail[243566]: goroutine 225 [running]:
Mar 20 15:29:17 alarm promtail[243566]: runtime.throw(0xaaaaaffa7175, 0x5)
Mar 20 15:29:17 alarm promtail[243566]:         runtime/panic.go:1117 +0x54 fp=0x40008f38d0 sp=0x40008f38a0 pc=0xaaaaae848b24
Mar 20 15:29:17 alarm promtail[243566]: runtime.sigpanic()
Mar 20 15:29:17 alarm promtail[243566]:         runtime/signal_unix.go:741 +0x230 fp=0x40008f3910 sp=0x40008f38d0 pc=0xaaaaae8603a0
Mar 20 15:29:17 alarm promtail[243566]: github.com/golang/snappy.encodeBlock(0x4000f80003, 0x6b614, 0x6b614, 0x4000db8000, 0x10000, 0x5c08b, 0x400089c138)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/golang/snappy@v0.0.2/encode_arm64.s:666 +0x360 fp=0x40008fb9b0 sp=0x40008f3920 pc=0xaaaaaec746d0
Mar 20 15:29:17 alarm promtail[243566]: github.com/golang/snappy.Encode(0x4000f80000, 0x6b617, 0x6b617, 0x4000dc8000, 0x4c08b, 0x4c08b, 0x0, 0xaaaaaea2075c, 0x40009a48f0)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/golang/snappy@v0.0.2/encode.go:39 +0x17c fp=0x40008fba00 sp=0x40008fb9b0 pc=0xaaaaaec7336c
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/client.(*batch).encode(0x400095e3c0, 0x1, 0x10, 0x4c, 0x4000af6730, 0x40009a48f0, 0x400007dbd8)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/client/batch.go:79 +0x94 fp=0x40008fba60 sp=0x40008fba00 pc=0xaaaaaefaecc4
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/client.(*client).sendBatch(0x400044c900, 0x0, 0x0, 0x400095e3c0)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/client/client.go:241 +0x30 fp=0x40008fbc90 sp=0x40008fba60 pc=0xaaaaaefafe40
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/client.(*client).run(0x400044c900)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/client/client.go:233 +0x46c fp=0x40008fbfd0 sp=0x40008fbc90 pc=0xaaaaaefafdac
Mar 20 15:29:17 alarm promtail[243566]: runtime.goexit()
Mar 20 15:29:17 alarm promtail[243566]:         runtime/asm_arm64.s:1130 +0x4 fp=0x40008fbfd0 sp=0x40008fbfd0 pc=0xaaaaae87fda4
Mar 20 15:29:17 alarm promtail[243566]: created by github.com/grafana/loki/pkg/promtail/client.New
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/client/client.go:170 +0x320
Mar 20 15:29:17 alarm promtail[243566]: goroutine 1 [chan receive]:
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/server.(*noopServer).Run(0x40009bc2b8, 0x6, 0x400098c3c0)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/server/server.go:260 +0x8c
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail.(*Promtail).Run(0x40003569b0, 0x4000998b00, 0x4)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/promtail.go:98 +0xe8
Mar 20 15:29:17 alarm promtail[243566]: main.main()
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/cmd/promtail/main.go:113 +0x48c
Mar 20 15:29:17 alarm promtail[243566]: goroutine 19 [chan receive]:
Mar 20 15:29:17 alarm promtail[243566]: k8s.io/klog.(*loggingT).flushDaemon(0xaaaab1a35920)
Mar 20 15:29:17 alarm promtail[243566]:         k8s.io/klog@v1.0.0/klog.go:1010 +0x70
Mar 20 15:29:17 alarm promtail[243566]: created by k8s.io/klog.init.0
Mar 20 15:29:17 alarm promtail[243566]:         k8s.io/klog@v1.0.0/klog.go:411 +0x11c
Mar 20 15:29:17 alarm promtail[243566]: goroutine 69 [select]:
Mar 20 15:29:17 alarm promtail[243566]: go.opencensus.io/stats/view.(*worker).start(0x400049e100)
Mar 20 15:29:17 alarm promtail[243566]:         go.opencensus.io@v0.22.4/stats/view/worker.go:276 +0x98
Mar 20 15:29:17 alarm promtail[243566]: created by go.opencensus.io/stats/view.init.0
Mar 20 15:29:17 alarm promtail[243566]:         go.opencensus.io@v0.22.4/stats/view/worker.go:34 +0x70
Mar 20 15:29:17 alarm promtail[243566]: goroutine 70 [chan receive]:
Mar 20 15:29:17 alarm promtail[243566]: k8s.io/klog/v2.(*loggingT).flushDaemon(0xaaaab1a35a00)
Mar 20 15:29:17 alarm promtail[243566]:         k8s.io/klog/v2@v2.3.0/klog.go:1131 +0x70
Mar 20 15:29:17 alarm promtail[243566]: created by k8s.io/klog/v2.init.0
Mar 20 15:29:17 alarm promtail[243566]:         k8s.io/klog/v2@v2.3.0/klog.go:416 +0x11c
Mar 20 15:29:17 alarm promtail[243566]: goroutine 148 [chan receive]:
Mar 20 15:29:17 alarm promtail[243566]: go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0x4000111728)
Mar 20 15:29:17 alarm promtail[243566]:         go.etcd.io/etcd@v0.5.0-alpha.5.0.20200520232829-54ba9589114f/pkg/logutil/merge_logger.go:173 +0x2dc
Mar 20 15:29:17 alarm promtail[243566]: created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
Mar 20 15:29:17 alarm promtail[243566]:         go.etcd.io/etcd@v0.5.0-alpha.5.0.20200520232829-54ba9589114f/pkg/logutil/merge_logger.go:91 +0x7c
Mar 20 15:29:17 alarm promtail[243566]: goroutine 187 [chan receive]:
Mar 20 15:29:17 alarm promtail[243566]: go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0x400000e840)
Mar 20 15:29:17 alarm promtail[243566]:         go.etcd.io/etcd@v0.5.0-alpha.5.0.20200520232829-54ba9589114f/pkg/logutil/merge_logger.go:173 +0x2dc
Mar 20 15:29:17 alarm promtail[243566]: created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
Mar 20 15:29:17 alarm promtail[243566]:         go.etcd.io/etcd@v0.5.0-alpha.5.0.20200520232829-54ba9589114f/pkg/logutil/merge_logger.go:91 +0x7c
Mar 20 15:29:17 alarm promtail[243566]: goroutine 190 [chan receive]:
Mar 20 15:29:17 alarm promtail[243566]: go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0x400000eab0)
Mar 20 15:29:17 alarm promtail[243566]:         go.etcd.io/etcd@v0.5.0-alpha.5.0.20200520232829-54ba9589114f/pkg/logutil/merge_logger.go:173 +0x2dc
Mar 20 15:29:17 alarm promtail[243566]: created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
Mar 20 15:29:17 alarm promtail[243566]:         go.etcd.io/etcd@v0.5.0-alpha.5.0.20200520232829-54ba9589114f/pkg/logutil/merge_logger.go:91 +0x7c
Mar 20 15:29:17 alarm promtail[243566]: goroutine 258 [select]:
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/positions.(*positions).run(0x4000356a00)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/positions/positions.go:153 +0xbc
Mar 20 15:29:17 alarm promtail[243566]: created by github.com/grafana/loki/pkg/promtail/positions.New
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/positions/positions.go:95 +0x144
Mar 20 15:29:17 alarm promtail[243566]: goroutine 263 [select]:
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd/sdjournal.(*JournalReader).Follow(0x40009bc288, 0x4000984300, 0xaaaab0b1aaf8, 0xaaaab1a64c00, 0x0, 0x0)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/coreos/go-systemd@v0.0.0-20191104093116-d3cd4ed1dbcf/sdjournal/read.go:238 +0x130
Mar 20 15:29:17 alarm promtail[243566]: github.com/grafana/loki/pkg/promtail/targets/journal.journalTargetWithReader.func1(0x40001ae400, 0x4000984300)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/targets/journal/journaltarget.go:184 +0x54
Mar 20 15:29:17 alarm promtail[243566]: created by github.com/grafana/loki/pkg/promtail/targets/journal.journalTargetWithReader
Mar 20 15:29:17 alarm promtail[243566]:         github.com/grafana/loki/pkg/promtail/targets/journal/journaltarget.go:182 +0x310
Mar 20 15:29:17 alarm promtail[243566]: goroutine 275 [syscall]:
Mar 20 15:29:17 alarm promtail[243566]: os/signal.signal_recv(0x0)
Mar 20 15:29:17 alarm promtail[243566]:         runtime/sigqueue.go:168 +0xcc
Mar 20 15:29:17 alarm promtail[243566]: os/signal.loop()
Mar 20 15:29:17 alarm promtail[243566]:         os/signal/signal_unix.go:23 +0x20
Mar 20 15:29:17 alarm promtail[243566]: created by os/signal.Notify.func1.1
Mar 20 15:29:17 alarm promtail[243566]:         os/signal/signal.go:151 +0x44
Mar 20 15:29:17 alarm promtail[243566]: goroutine 203 [syscall]:
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd/sdjournal._Cfunc_my_sd_journal_wait(0xffff5af0eb80, 0xaaaadb8b6200, 0x186a0, 0x0)
Mar 20 15:29:17 alarm promtail[243566]:         _cgo_gotypes.go:554 +0x44
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd/sdjournal.(*Journal).Wait.func1(0xffff5af0eb80, 0x40001b47d0, 0x186a0, 0x0)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/coreos/go-systemd@v0.0.0-20191104093116-d3cd4ed1dbcf/sdjournal/journal.go:1014 +0x80
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd/sdjournal.(*Journal).Wait(0x40001b47d0, 0x5f5e100, 0x0)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/coreos/go-systemd@v0.0.0-20191104093116-d3cd4ed1dbcf/sdjournal/journal.go:1014 +0xa8
Mar 20 15:29:17 alarm promtail[243566]: github.com/coreos/go-systemd/sdjournal.(*JournalReader).Follow.func1(0x40009bc288, 0x4000982150, 0x40001b4830)
Mar 20 15:29:17 alarm promtail[243566]:         github.com/coreos/go-systemd@v0.0.0-20191104093116-d3cd4ed1dbcf/sdjournal/read.go:233 +0x38
Mar 20 15:29:17 alarm promtail[243566]: created by github.com/coreos/go-systemd/sdjournal.(*JournalReader).Follow
Mar 20 15:29:17 alarm promtail[243566]:         github.com/coreos/go-systemd@v0.0.0-20191104093116-d3cd4ed1dbcf/sdjournal/read.go:232 +0xe8
Mar 20 15:29:17 alarm systemd[1]: promtail.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
@cyriltovena
Copy link
Contributor

It doesn't seems related to journald, apparently it's failing to encode in snappy.

Sounds like I can just re-vendor snappy.

golang/snappy#56

@cyriltovena
Copy link
Contributor

cyriltovena commented Mar 22, 2021 •
edited

Weird we are up to date are you sure you're using promtail 2.2 ?

ha the logs says you're not.

Mar 20 15:29:16 alarm promtail[243566]: level=info ts=2021-03-20T19:29:16.163114189Z caller=main.go:110 msg="Starting Promtail" version="(version=2.1.0, branch=, revision=)"

Update and it should all be fine.

@AngellusMortis
Copy link
Author

Thanks for the help. I actually had my RPIs mixed up. This was my Manjaro ARM RPI, not my archlinuxarm one. I think this might be a good enough excuse to get rid of the rest of my Manjaro ARM RPIs, they seem to constantly break/have/issues.

@snowicepeng snowicepeng mentioned this issue May 23, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@AngellusMortis @cyriltovena