Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nats Timeout Error While Sending Message To JetStream subject #4954

Open
niknikhil025 opened this issue Jan 15, 2024 · 17 comments
Open

Nats Timeout Error While Sending Message To JetStream subject #4954

niknikhil025 opened this issue Jan 15, 2024 · 17 comments
Labels
defect Suspected defect such as a bug or regression

Comments

@niknikhil025
Copy link

Observed behavior

Nats client timesout when trying to connect using python. Resources are available on the server.
There are some 20-30 streams.
One of the streams has some 200K Subscribers 200K subjects

Expected behavior

Sending message without any timeouts

Server and client version

nats-server: v2.10.3
0.1.1

Host environment

Ubuntu
64 GB Ram
16 CPU

Steps to reproduce

No response
@niknikhil025 niknikhil025 added the defect Suspected defect such as a bug or regression label Jan 15, 2024
@niknikhil025
Copy link
Author

RECENT LOGS

[2453688] 2024/01/15 06:16:05.273300 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 35755_766'
[2453688] 2024/01/15 06:16:06.713819 [ERR] 60.241.109.18:55447 - cid:9720059 - authentication error
[2453688] 2024/01/15 06:16:15.879093 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 32469'
[2453688] 2024/01/15 06:16:22.658564 [ERR] 59.100.219.94:60489 - cid:9720267 - authentication error
[2453688] 2024/01/15 06:16:28.886593 [ERR] 60.241.109.18:55501 - cid:9720364 - authentication error
[2453688] 2024/01/15 06:16:35.083712 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 24277'
[2453688] 2024/01/15 06:16:43.361248 [ERR] 60.241.109.18:55535 - cid:9720514 - authentication error
[2453688] 2024/01/15 06:16:46.933300 [ERR] 60.241.109.18:55547 - cid:9720557 - authentication error
[2453688] 2024/01/15 06:16:52.383097 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 35029'
[2453688] 2024/01/15 06:16:54.977942 [WRN] Internal subscription on "$JS.API.CONSUMER.INFO.agent_stream.11324" took too long: 2.481408455s
[2453688] 2024/01/15 06:17:00.975760 [ERR] 60.241.109.18:55573 - cid:9720765 - authentication error
[2453688] 2024/01/15 06:17:03.731495 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 35756_766'
[2453688] 2024/01/15 06:17:10.940334 [ERR] 60.241.109.18:55595 - cid:9720903 - authentication error
[2453688] 2024/01/15 06:17:26.853885 [ERR] 60.241.109.18:55627 - cid:9721165 - authentication error
[2453688] 2024/01/15 06:17:30.388837 [ERR] 59.100.219.94:60503 - cid:9721229 - authentication error
[2453688] 2024/01/15 06:17:35.815126 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 29671'
[2453688] 2024/01/15 06:17:38.327795 [WRN] Internal subscription on "$JS.API.CONSUMER.INFO.agent_stream.1820" took too long: 2.01508203s
[2453688] 2024/01/15 06:17:38.328130 [WRN] Internal subscription on "$JS.API.CONSUMER.INFO.agent_stream.32462" took too long: 2.093218068s
[2453688] 2024/01/15 06:17:38.328169 [WRN] Internal subscription on "$JS.API.CONSUMER.INFO.agent_stream.32462" took too long: 2.093263827s
[2453688] 2024/01/15 06:17:38.328198 [WRN] Internal subscription on "$JS.API.CONSUMER.INFO.agent_stream.32462" took too long: 2.093345423s
[2453688] 2024/01/15 06:17:38.328213 [WRN] 45.77.239.85:57758 - cid:9721318 - Readloop processing time: 2.093386529s
[2453688] 2024/01/15 06:17:38.786262 [ERR] 60.241.109.18:55655 - cid:9721371 - authentication error
[2453688] 2024/01/15 06:17:38.811384 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 35595'
[2453688] 2024/01/15 06:17:50.555080 [ERR] 45.77.239.85:21356 - cid:9721544 - authentication error
[2453688] 2024/01/15 06:17:52.779526 [ERR] 45.77.239.85:21750 - cid:9721594 - authentication error
[2453688] 2024/01/15 06:18:02.840773 [ERR] 60.241.109.18:55705 - cid:9721728 - authentication error
[2453688] 2024/01/15 06:18:08.840660 [ERR] 45.77.239.85:16762 - cid:9721773 - authentication error
[2453688] 2024/01/15 06:18:11.584973 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 35757_591'
[2453688] 2024/01/15 06:18:15.671286 [ERR] 60.241.109.18:55741 - cid:9721832 - authentication error
[2453688] 2024/01/15 06:18:18.952244 [ERR] 60.241.109.18:55752 - cid:9721881 - authentication error
[2453688] 2024/01/15 06:18:33.086818 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 34454'
[2453688] 2024/01/15 06:18:37.020531 [ERR] 60.241.109.18:55787 - cid:9722142 - authentication error
[2453688] 2024/01/15 06:18:39.021977 [ERR] 45.77.239.85:53690 - cid:9722203 - authentication error
[2453688] 2024/01/15 06:18:41.594598 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 19940'
[2453688] 2024/01/15 06:18:51.108818 [ERR] 60.241.109.18:55820 - cid:9722368 - authentication error
[2453688] 2024/01/15 06:18:51.939518 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 28053'
[2453688] 2024/01/15 06:18:58.399012 [ERR] 59.100.219.94:60527 - cid:9722470 - authentication error
[2453688] 2024/01/15 06:18:58.988241 [ERR] 60.241.109.18:55836 - cid:9722479 - authentication error
[2453688] 2024/01/15 06:19:19.093669 [ERR] 60.241.109.18:55881 - cid:9722732 - authentication error
[2453688] 2024/01/15 06:19:22.989240 [ERR] 45.77.239.85:8498 - cid:9722777 - authentication error
[2453688] 2024/01/15 06:19:23.004514 [ERR] 60.241.109.18:55893 - cid:9722787 - authentication error
[2453688] 2024/01/15 06:19:33.824443 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 9689'
[2453688] 2024/01/15 06:19:45.111688 [ERR] 60.241.109.18:55942 - cid:9723052 - authentication error
[2453688] 2024/01/15 06:19:55.129556 [ERR] 60.241.109.18:55970 - cid:9723227 - authentication error
[2453688] 2024/01/15 06:20:21.258709 [ERR] 60.241.109.18:56023 - cid:9723677 - authentication error
[2453688] 2024/01/15 06:20:32.062486 [ERR] 45.77.239.85:25926 - cid:9723812 - authentication error
[2453688] 2024/01/15 06:20:37.036341 [ERR] 45.77.239.85:26304 - cid:9723897 - authentication error
[2453688] 2024/01/15 06:20:45.092098 [ERR] 60.241.109.18:56075 - cid:9724057 - authentication error
[2453688] 2024/01/15 06:20:47.569376 [ERR] 59.100.219.94:60564 - cid:9724113 - authentication error
[2453688] 2024/01/15 06:21:00.963222 [ERR] 60.241.109.18:56111 - cid:9724284 - authentication error
[2453688] 2024/01/15 06:21:03.177331 [ERR] 60.241.109.18:56117 - cid:9724324 - authentication error
[2453688] 2024/01/15 06:21:37.512635 [ERR] 60.241.109.18:56185 - cid:9724723 - authentication error
[2453688] 2024/01/15 06:22:03.875149 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 35724'
[2453688] 2024/01/15 06:22:26.420541 [ERR] 59.100.219.94:60592 - cid:9725448 - authentication error
[2453688] 2024/01/15 06:22:28.592418 [ERR] 45.77.239.85:22292 - cid:9725469 - authentication error
[2453688] 2024/01/15 06:22:29.878025 [ERR] 45.77.239.85:22492 - cid:9725498 - authentication error
[2453688] 2024/01/15 06:22:33.666242 [ERR] 60.241.109.18:56283 - cid:9725562 - authentication error
[2453688] 2024/01/15 06:22:37.901558 [ERR] 60.241.109.18:56293 - cid:9725598 - authentication error
[2453688] 2024/01/15 06:23:04.126502 [ERR] 60.241.109.18:56358 - cid:9725962 - authentication error
[2453688] 2024/01/15 06:23:10.118342 [ERR] 45.77.239.85:32020 - cid:9726069 - authentication error
[2453688] 2024/01/15 06:23:10.141114 [ERR] 60.241.109.18:56370 - cid:9726071 - authentication error
[2453688] 2024/01/15 06:23:25.076866 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 35602'
[2453688] 2024/01/15 06:23:27.470580 [WRN] Internal subscription on "$JS.API.CONSUMER.INFO.agent_stream.5981_2254" took too long: 2.209689337s
[2453688] 2024/01/15 06:23:38.212356 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 34998'
[2453688] 2024/01/15 06:23:40.575457 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 34794'
[2453688] 2024/01/15 06:23:50.325749 [ERR] 60.241.109.18:56449 - cid:9726641 - authentication error
[2453688] 2024/01/15 06:24:00.000793 [ERR] 59.100.219.94:60620 - cid:9726748 - authentication error
[2453688] 2024/01/15 06:24:08.338609 [ERR] 60.241.109.18:56493 - cid:9726820 - authentication error
[2453688] 2024/01/15 06:24:08.696862 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 34725'
[2453688] 2024/01/15 06:24:21.418143 [ERR] 60.241.109.18:56520 - cid:9726966 - authentication error
[2453688] 2024/01/15 06:24:27.986063 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 34703'
[2453688] 2024/01/15 06:24:30.386887 [WRN] Internal subscription on "$JS.API.CONSUMER.INFO.agent_stream.21891" took too long: 2.296905924s
[2453688] 2024/01/15 06:24:30.386921 [WRN] Internal subscription on "$JS.API.CONSUMER.INFO.agent_stream.21891" took too long: 2.296950671s
[2453688] 2024/01/15 06:24:30.386961 [WRN] Internal subscription on "$JS.API.CONSUMER.INFO.agent_stream.21891" took too long: 2.297059527s
[2453688] 2024/01/15 06:24:30.386979 [WRN] 45.77.239.85:30070 - cid:9727083 - Readloop processing time: 2.297115448s
[2453688] 2024/01/15 06:24:40.520083 [ERR] 60.241.109.18:56560 - cid:9727249 - authentication error
[2453688] 2024/01/15 06:25:06.670519 [ERR] 60.241.109.18:56610 - cid:9727687 - authentication error
[2453688] 2024/01/15 06:25:13.045095 [ERR] 59.100.219.94:60633 - cid:9727754 - authentication error
[2453688] 2024/01/15 06:25:22.853887 [ERR] 60.241.109.18:56642 - cid:9727923 - authentication error
[2453688] 2024/01/15 06:25:34.624961 [ERR] 45.77.239.85:53042 - cid:9728070 - authentication error
[2453688] 2024/01/15 06:25:44.753675 [ERR] 60.241.109.18:56692 - cid:9728214 - authentication error
[2453688] 2024/01/15 06:25:55.088414 [ERR] 60.241.109.18:56720 - cid:9728364 - authentication error
[2453688] 2024/01/15 06:26:29.360763 [ERR] 60.241.109.18:56782 - cid:9728783 - authentication error
[2453688] 2024/01/15 06:26:43.409270 [ERR] 45.77.239.85:35042 - cid:9729019 - authentication error
[2453688] 2024/01/15 06:26:47.454019 [ERR] 60.241.109.18:56818 - cid:9729053 - authentication error
[2453688] 2024/01/15 06:26:51.694190 [ERR] 59.100.219.94:60662 - cid:9729100 - authentication error
[2453688] 2024/01/15 06:26:52.965273 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 26776_611'
[2453688] 2024/01/15 06:27:11.546427 [ERR] 60.241.109.18:56866 - cid:9729299 - authentication error
[2453688] 2024/01/15 06:27:12.501972 [ERR] 60.241.109.18:56871 - cid:9729304 - authentication error
[2453688] 2024/01/15 06:27:46.599309 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 6621'
[2453688] 2024/01/15 06:28:24.732532 [ERR] 59.100.219.94:60689 - cid:9730389 - authentication error
[2453688] 2024/01/15 06:28:35.388771 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 21611'
[2453688] 2024/01/15 06:28:45.200439 [ERR] 60.241.109.18:56998 - cid:9730647 - authentication error
[2453688] 2024/01/15 06:29:38.942690 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 35472'
[2453688] 2024/01/15 06:29:52.958708 [ERR] 59.100.219.94:60718 - cid:9731466 - authentication error
[2453688] 2024/01/15 06:30:12.217602 [ERR] 45.77.239.85:10372 - cid:9731706 - authentication error
[2453688] 2024/01/15 06:30:12.242533 [ERR] 60.241.109.18:57114 - cid:9731709 - authentication error
[2453688] 2024/01/15 06:30:34.778013 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 35666'
[2453688] 2024/01/15 06:31:22.677075 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 35759_2472'
[2453688] 2024/01/15 06:31:44.776816 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 24117'
[2453688] 2024/01/15 06:31:51.466006 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 19017'
[2453688] 2024/01/15 06:31:57.092257 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 21510'
[2453688] 2024/01/15 06:31:57.510133 [ERR] 59.100.219.94:60765 - cid:9732825 - authentication error
[2453688] 2024/01/15 06:32:07.249902 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 33963'
[2453688] 2024/01/15 06:32:09.365826 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 35046'
[2453688] 2024/01/15 06:32:09.921427 [ERR] 60.241.109.18:57266 - cid:9732940 - authentication error
[2453688] 2024/01/15 06:32:22.399771 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 35547'
[2453688] 2024/01/15 06:32:31.911595 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 30041'
[2453688] 2024/01/15 06:33:16.633524 [ERR] 45.77.239.85:26574 - cid:9733833 - authentication error
[2453688] 2024/01/15 06:33:19.140954 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 29901'
[2453688] 2024/01/15 06:33:30.786171 [ERR] 59.100.219.94:60795 - cid:9733950 - authentication error
[2453688] 2024/01/15 06:33:31.709130 [ERR] 60.241.109.18:57369 - cid:9733963 - authentication error
[2453688] 2024/01/15 06:34:14.551763 [ERR] 45.77.239.85:50754 - cid:9734598 - authentication error
[2453688] 2024/01/15 06:34:18.579976 [ERR] 60.241.109.18:57440 - cid:9734637 - authentication error
[2453688] 2024/01/15 06:34:18.654323 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 32198'
[2453688] 2024/01/15 06:34:46.615359 [ERR] 60.241.109.18:57501 - cid:9734911 - authentication error
[2453688] 2024/01/15 06:34:47.828878 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 34819'
[2453688] 2024/01/15 06:34:48.758359 [ERR] 45.77.239.85:60526 - cid:9734940 - authentication error
[2453688] 2024/01/15 06:35:04.783206 [ERR] 60.241.109.18:57542 - cid:9735140 - authentication error
[2453688] 2024/01/15 06:35:04.930120 [ERR] 59.100.219.94:60831 - cid:9735142 - authentication error
[2453688] 2024/01/15 06:35:08.789877 [ERR] 45.77.239.85:39170 - cid:9735190 - authentication error
[2453688] 2024/01/15 06:35:08.795704 [ERR] 60.241.109.18:57555 - cid:9735192 - authentication error
[2453688] 2024/01/15 06:35:14.838586 [ERR] 60.241.109.18:57582 - cid:9735254 - authentication error
[2453688] 2024/01/15 06:35:24.865612 [ERR] 60.241.109.18:57602 - cid:9735384 - authentication error
[2453688] 2024/01/15 06:35:30.352558 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 35676'
[2453688] 2024/01/15 06:35:36.926714 [ERR] 45.77.239.85:9866 - cid:9735542 - authentication error
[2453688] 2024/01/15 06:35:36.933445 [ERR] 60.241.109.18:57626 - cid:9735544 - authentication error
[2453688] 2024/01/15 06:36:02.982291 [ERR] 60.241.109.18:57677 - cid:9735837 - authentication error
[2453688] 2024/01/15 06:36:08.997398 [ERR] 45.77.239.85:46406 - cid:9735886 - authentication error
[2453688] 2024/01/15 06:36:14.982648 [ERR] 45.77.239.85:53934 - cid:9735985 - authentication error
[2453688] 2024/01/15 06:36:19.023249 [ERR] 60.241.109.18:57713 - cid:9736022 - authentication error
[2453688] 2024/01/15 06:36:24.662376 [ERR] 59.100.219.94:60851 - cid:9736079 - authentication error
[2453688] 2024/01/15 06:36:32.564611 [ERR] 60.241.109.18:57746 - cid:9736161 - authentication error
[2453688] 2024/01/15 06:36:39.119703 [ERR] 60.241.109.18:57763 - cid:9736247 - authentication error
[2453688] 2024/01/15 06:36:55.259583 [ERR] 60.241.109.18:57799 - cid:9736472 - authentication error
[2453688] 2024/01/15 06:37:00.197009 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 35614'

@Jarema
Copy link
Member

Jarema commented Jan 15, 2024

Which Python client version are you using?
At first glance it seems your client fails to authenticate. Please double check the permissions.
Also it's always good to be upgraded to the last version of the nats-server.

@niknikhil025
Copy link
Author

nats-py==2.6.0

we are authenticating using nkeys

This happens with our golang client too when subscribing or sending messages to jetstream queue.
It timesout when connecting to nats itself. We will upgrade nats-server and check

@niknikhil025
Copy link
Author

authentication is for some clients which do not have permissions updated and that is fine for us. The ones with correct credentials are also not able to communicate.

I will update the status once we update nats-server

@Jarema
Copy link
Member

Jarema commented Jan 15, 2024

By stream having 200k subscribers, do you mean 200k consumers?

@niknikhil025
Copy link
Author

niknikhil025 commented Jan 15, 2024 •
edited

By stream having 200k subscribers, do you mean 200k consumers?

Yes. All are consuming different subjects in the stream. so each subject is subscribed by 1 consumer and these 200K subjects are in a single stream. It is a wildcard channel pattern.

Python Script To Create Stream

import nats
import os
import dotenv
from nats.js import api

dotenv.load_dotenv()

async def create_stream():
    nats_url = os.getenv("NATS_URL")
    nats_creds_path = os.getenv("CRED_PATH")
    nc = await nats.connect(nats_url, user_credentials=nats_creds_path)

    js = nc.jetstream()
    resp = await js.add_stream(
        name='stream_abc', subjects=['abc_channel.>'], 
    )
    print("Add Stream Response", resp)

@niknikhil025
Copy link
Author

This is what happens when we restart our nats cluster after upgrading nats-server to
nats-server: v2.10.9

[1094386] 2024/01/15 08:36:28.759778 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 15489'
[1094386] 2024/01/15 08:36:28.901796 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 951'
[1094386] 2024/01/15 08:36:28.936951 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 33089'
[1094386] 2024/01/15 08:36:28.938780 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 29502_1886'
[1094386] 2024/01/15 08:36:28.939385 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 23881'
[1094386] 2024/01/15 08:36:28.940019 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 21475_11909'
[1094386] 2024/01/15 08:36:28.972722 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 24942_431'
[1094386] 2024/01/15 08:36:28.975580 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 32040'
[1094386] 2024/01/15 08:36:28.984608 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 20579'
[1094386] 2024/01/15 08:36:28.987775 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 28768'
[1094386] 2024/01/15 08:36:28.989340 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 10918'
[1094386] 2024/01/15 08:36:29.075386 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 18099_339'
[1094386] 2024/01/15 08:36:29.104330 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 24543'
[1094386] 2024/01/15 08:36:29.107282 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 34111_2293'
[1094386] 2024/01/15 08:36:29.116342 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 30044'
[1094386] 2024/01/15 08:36:29.162228 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 21821'
[1094386] 2024/01/15 08:36:29.232554 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 12163_3313'
[1094386] 2024/01/15 08:36:29.233163 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 20965'
[1094386] 2024/01/15 08:36:29.242092 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 31962_3180'
[1094386] 2024/01/15 08:36:29.244240 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 29042'
[1094386] 2024/01/15 08:36:29.293956 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 25960'
[1094386] 2024/01/15 08:36:29.316028 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 4207'
[1094386] 2024/01/15 08:36:29.371353 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 27008'
[1094386] 2024/01/15 08:36:29.373606 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 6652'
[1094386] 2024/01/15 08:36:29.447203 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 22622_3119'

@niknikhil025
Copy link
Author

niknikhil025 commented Jan 15, 2024 •
edited

After a while these are the logs

getting this once in a while
[1094386] 2024/01/15 08:57:59.064555 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > abc_stream > 36341'

[1094386] 2024/01/15 08:47:04.551495 [ERR] 60.241.109.18:64737 - cid:171031 - authentication error
[1094386] 2024/01/15 08:47:14.836296 [ERR] 60.241.109.18:64761 - cid:171141 - authentication error
[1094386] 2024/01/15 08:47:23.088930 [ERR] 45.77.239.85:63478 - cid:171234 - authentication error
[1094386] 2024/01/15 08:47:24.230802 [ERR] 59.100.219.94:63691 - cid:171247 - authentication error
[1094386] 2024/01/15 08:47:31.365769 [ERR] 60.241.109.18:64796 - cid:171333 - authentication error
[1094386] 2024/01/15 08:47:35.497491 [ERR] 60.241.109.18:64808 - cid:171384 - authentication error
[1094386] 2024/01/15 08:47:46.561977 [INF] JetStream cluster new consumer leader for 'ADUCTL447Q7CNBNXRFG6YRXZI4YZQBZOSAVVRUH5742SPG2ZGHJO5A3M > agent_stream > 18306'
[1094386] 2024/01/15 08:47:51.705161 [ERR] 60.241.109.18:64845 - cid:171578 - authentication error
[1094386] 2024/01/15 08:48:02.575682 [ERR] 60.241.109.18:64870 - cid:171681 - authentication error
[1094386] 2024/01/15 08:48:03.148824 [WRN] Internal subscription on "$JS.API.CONSUMER.INFO.agent_stream.27483" took too long: 2.02177809s
[1094386] 2024/01/15 08:48:06.983379 [WRN] Internal subscription on "$JS.API.CONSUMER.INFO.agent_stream.10109" took too long: 2.17092731s
[1094386] 2024/01/15 08:48:07.217464 [WRN] 45.77.239.85:38792 - cid:171708 - Readloop processing time: 2.231999009s

@niknikhil025
Copy link
Author

niknikhil025 commented Jan 15, 2024 •
edited

now pushing to stream is working but now we are not able to subscribe to the channel
stream -> abc_stream has channel -> channel_abc.>
there are around 100K consumers right now and when i try to subscribe I get timeouts nothing in nats server logs.

channel_name, durable, stream_name

sub = await js.pull_subscribe(queue_name, str("abc"), stream="abc_stream") # pull

Stream Info

Information for Stream abc_stream created 2024-01-14 17:43:25

              Subjects: abc_channel.>
              Replicas: 1
               Storage: File

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: Old
      Duplicate Window: 2m0s
     Allows Msg Delete: true
          Allows Purge: true
        Allows Rollups: false

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: unlimited
         Maximum Bytes: unlimited
           Maximum Age: unlimited
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

Cluster Information:

                  Name: atl
                Leader: n1-2C-worker-B

State:

              Messages: 1,664,928
                 Bytes: 260 MiB
        First Sequence: 1 @ 2024-01-14 17:43:25 UTC
         Last Sequence: 1,664,928 @ 2024-01-15 10:22:07 UTC
      Active Consumers: 57,643
    Number of Subjects: 46,440

@shankarmn94
Copy link

[4111682] 2024/01/15 12:05:53.932465 [INF] Recovering 1 consumers for stream - 'AAN2QDQ4H4G34UIBTO3IFKZ3TT622XBGCGTXGBZ36XCHOWQP3BZV3JTK > agent_jobdata'
[4111682] 2024/01/15 12:05:54.195044 [INF] Recovering 107049 consumers for stream - 'AAN2QDQ4H4G34UIBTO3IFKZ3TT622XBGCGTXGBZ36XCHOWQP3BZV3JTK > agent_stream'

how much time it will take to restore or recover these 107049 consumers,
nats-server - 2.10.9

@shankarmn94
Copy link

@Jarema @derekcollison

@shankarmn94
Copy link

will nats-server restart automatically in any of the cases, as i observed after updating 2.10.9.

@shankarmn94
Copy link

@Jarema @derekcollison

is there any consumer limit on any sections for a jetstream.

@Jarema
Copy link
Member

Jarema commented Jan 16, 2024

@shankarmn94 There is max_consumers config field on the Stream configuration, and max_consumers on account config too

@shankarmn94
Copy link

@Jarema yes we have set it to unlimited.

Account Usage:

                    Storage: 3.8 GiB
                     Memory: 0 B
                    Streams: 39
                  Consumers: 135,502

Account Limits:

        Max Message Payload: 8.0 MiB

Tier: Default:

  Configuration Requirements:

    Stream Requires Max Bytes Set: false
     Consumer Maximum Ack Pending: Unlimited

  Stream Resource Usage Limits:

                           Memory: 0 B of Unlimited
                Memory Per Stream: Unlimited
                          Storage: 3.8 GiB of Unlimited
               Storage Per Stream: Unlimited
                          Streams: 39 of Unlimited
                        Consumers: 135,502 of Unlimited

@wallyqs wallyqs changed the title Nats Timout Error While Sending Message To Jetstream subject Nats Timeout Error While Sending Message To JetStream subject Feb 7, 2024
@wallyqs
Copy link
Member

wallyqs commented Feb 7, 2024

Some of the behaviors reported in this issue would have been improved by #5022 and other fixes which were part of the v2.10.10 release.

@shankarmn94
Copy link

can i know y am getting context deadline timeout for specifi streams and when i do stream ls

it will list like inaccessible stream section what this tells. @wallyqs

nats consumer report appl
? Select a Stream application_patching
nats: error: context deadline exceeded

nats consumer report agent_assetinfo
? Select a Stream agent_assetinfo
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Consumer report for agent_assetinfo with 3 consumers │
├──────────────────────────┬──────┬────────────┬──────────┬─────────────┬─────────────┬─────────────┬───────────┬──────────────────────┤
│ Consumer │ Mode │ Ack Policy │ Ack Wait │ Ack Pending │ Redelivered │ Unprocessed │ Ack Floor │ Cluster │
├──────────────────────────┼──────┼────────────┼──────────┼─────────────┼─────────────┼─────────────┼───────────┼──────────────────────┤
│ agent_assetinfo_consumer │ Pull │ Explicit │ 30.00s │ 0 │ 0 │ 0 │ 965,553 │ n1-pod-500-worker-3* │
│ golangnats │ Pull │ Explicit │ 30.00s │ 0 │ 0 │ 0 │ 965,553 │ n1-pod-500-worker-3* │
│ security_report_card │ Pull │ Explicit │ 30.00s │ 15 │ 0 │ 0 │ 965,538 │ n1-pod-500-worker-3* │
╰──────────────────────────┴──────┴────────────┴──────────┴─────────────┴─────────────┴─────────────┴───────────┴──────────────────────╯

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
defect Suspected defect such as a bug or regression
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@wallyqs @Jarema @niknikhil025 @shankarmn94