issues after upgrade - lz4 compression stopped working

[thezoggy]

moving issue from nfsen to here and cleaning up issue to focus on just the one problem here.

When upgrading from 9198d94 to latest (564f3a9) code, nfcapd stopped storing files in lz4 compression. Reverting back to 9198d94 and everything works as before.

---

upgraded nfdump from:

nfdump -V
nfdump: Version: 1.7.4-9198d94 Options: ZSTD BZIP2 Date: 2024-03-17 17:52:55 +0100

to

nfdump -V
nfdump: Version: 1.7.4-564f3a9 Options: ZSTD BZIP2 Date: 2024-05-11 10:55:50 +0200

changes between git hashes:
https://github.com/phaag/nfdump/compare/9198d94..564f3a9

I upgraded nfdump at 2024-05-16 ~1600 UTC, and while appeared fine initially but per graphs there you can see it struggled up until a day later box ran out of hdd space..

At first I thought maybe stuff stopped expiring but then later discovered that after the upgrade nfcapd was no longer compressing the data to lz4 like before. Which caused it to use up the rest of the free space on the box until everything broke down.

Looking at one router as example..

before upgrade ~115M a file, then post upgrade 450M a file

-rw-r--r--  1 www-data www-data 115M May 16 15:20 nfcapd.202405161515
-rw-r--r--  1 www-data www-data 115M May 16 15:25 nfcapd.202405161520
-rw-r--r--  1 www-data www-data 115M May 16 15:30 nfcapd.202405161525
-rw-r--r--  1 www-data www-data 115M May 16 15:35 nfcapd.202405161530
-rw-r--r--  1 www-data www-data 115M May 16 15:40 nfcapd.202405161535
-rw-r--r--  1 www-data www-data 115M May 16 15:45 nfcapd.202405161540
-rw-r--r--  1 www-data www-data 115M May 16 15:50 nfcapd.202405161545
-rw-r--r--  1 www-data www-data 115M May 16 15:55 nfcapd.202405161550 <<< taken down for upgrade
-rw-r--r--  1 www-data www-data  67M May 16 15:57 nfcapd.202405161555
-rw-r--r--  1 www-data www-data  80M May 16 16:10 nfcapd.202405161605
-rw-r--r--  1 www-data www-data 456M May 16 16:15 nfcapd.202405161610
-rw-r--r--  1 www-data www-data 448M May 16 16:20 nfcapd.202405161615
-rw-r--r--  1 www-data www-data 472M May 16 16:25 nfcapd.202405161620
-rw-r--r--  1 www-data www-data 468M May 16 16:30 nfcapd.202405161625
-rw-r--r--  1 www-data www-data 475M May 16 16:35 nfcapd.202405161630
-rw-r--r--  1 www-data www-data 472M May 16 16:40 nfcapd.202405161635

so went from lz4 compressed

> nfdump -v nfcapd.202405161540
File       : nfcapd.202405161540
Version    : 2 - lz4 compressed
Created    : 2024-05-16 15:40:02
Created by : nfcapd
nfdump     : f1070400
encryption : no
Appdx blks : 1
Data blks  : 190
Checking data blocks
Checking appendix blocks

Total
Type 3 blocks : 191
Records       : 2650069

to not compressed

> nfdump -v nfcapd.202405161640
File       : nfcapd.202405161640
Version    : 2 - not compressed
Created    : 2024-05-16 16:40:18
Created by : nfcapd
nfdump     : f1070400
encryption : no
Appdx blks : 1
Data blks  : 237
Checking data blocks
Checking appendix blocks

Total
Type 3 blocks : 238
Records       : 2680930

This box is running: Ubuntu 20.04.6 LTS
Have been using it to solely handle nfsen/nfdump as a netflow ingestion box.

The upgrade process:

sudo systemctl stop nfsen

# navigate to nfdump repo, update+ build+make+install
cd ~/nfdump/
git pull
./autogen.sh
./configure --enable-nfprofile --enable-maxmind --enable-readpcap --enable-nfpcapd --enable-sflow
make
sudo make install
sudo ldconfig

# check for os updates, reboot if needed (nfsen service would auto start on boot)
sudo apt-get update 
sudo apt-get upgrade
sudo reboot

# if box wasnt rebooted/didnt need to.. start nfsen back up
sudo systemctl start nfsen

I've done this process numerous times over the years without any issues, and for whatever reason this time you can see that in the config.log when I built out 564f3a9 that it failed to include lz4:

...
configure:12320: $? = 0
configure:12320: result: yes
configure:12345: checking lz4.h usability
configure:12345: gcc -c -g -O3 -std=gnu17 -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wmissing-noreturn -fno-strict-aliasing -pthread  conftest.c >&5
conftest.c:88:10: fatal error: lz4.h: No such file or directory
   88 | #include <lz4.h>
      |          ^~~~~~~
compilation terminated.
configure:12345: $? = 1
...

| #endif
| #include <lz4.h>
configure:12345: result: no
configure:12345: checking lz4.h presence
configure:12345: gcc -E  conftest.c
conftest.c:55:10: fatal error: lz4.h: No such file or directory
   55 | #include <lz4.h>
      |          ^~~~~~~
compilation terminated.
configure:12345: $? = 1
...

during this, you can see it failed check for lz4 but ended up with embedded support for lz4:

> ./configure --enable-nfprofile --enable-maxmind --enable-readpcap --enable-nfpcapd --enable-sflow
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for clang... no
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
...
checking lz4.h usability... no
checking lz4.h presence... no
checking for lz4.h... no
...
checking zstd.h usability... yes
checking zstd.h presence... yes
checking for zstd.h... yes
checking for ZSTD_decompress in -lzstd... yes
...

config.status: executing libtool commands

----------------------------------
 Build Settings for nfdump v1.7.4
----------------------------------
  host type          = linux-gnu
  install dir        = /usr/local
  CC                 = gcc
  CFLAGS             =  -g -O3 -std=gnu17 -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wmissing-noreturn -fno-strict-aliasing -pthread
  CPPFLAGS           =  
  LDFLAGS            =  
  LIBS               = -lresolv -lpthread -latomic  -lbz2 -lzstd
  Enable liblz4      = embedded
  Enable libbz2      = yes
  Enable libzstd     = yes
  Enable ja4         = no
  Build geolookup    = yes
  Build torlookup    = no
  Build sflow        = yes
  Build nfpcapd      = yes
  Build nfprofile    = yes
  Build ft2nfdump    = no
----------------------------------

I looked back and saw that nothing really changed on the box just the normal security patches weeks ago but nothing recently since the last server boot as the box has been working fine upto this point and I see lz4 is still installed.

I thought maybe liblz4-dev needs to be installed now?
Installed and tried rebuilding latest code again, and lz4 this time was built/included:

> ./configure --enable-nfprofile --enable-maxmind --enable-readpcap --enable-nfpcapd --enable-sflow
checking for a BSD-compatible install... /usr/bin/install -c
...
checking lz4.h usability... yes
checking lz4.h presence... yes
checking for lz4.h... yes
checking for LZ4_compress in -llz4... yes
checking bzlib.h usability... yes
checking bzlib.h presence... yes
checking for bzlib.h... yes
checking for BZ2_bzCompressInit in -lbz2... yes
checking zstd.h usability... yes
checking zstd.h presence... yes
checking for zstd.h... yes
checking for ZSTD_decompress in -lzstd... yes
...

----------------------------------
 Build Settings for nfdump v1.7.4
----------------------------------
  host type          = linux-gnu
  install dir        = /usr/local
  CC                 = gcc
  CFLAGS             =  -g -O3 -std=gnu17 -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wmissing-noreturn -fno-strict-aliasing -pthread
  CPPFLAGS           =  
  LDFLAGS            =  
  LIBS               = -lresolv -lpthread -latomic  -llz4 -lbz2 -lzstd
  Enable liblz4      = yes
  Enable libbz2      = yes
  Enable libzstd     = yes
  Enable ja4         = no
  Build geolookup    = yes
  Build torlookup    = no
  Build sflow        = yes
  Build nfpcapd      = yes
  Build nfprofile    = yes
  Build ft2nfdump    = no
----------------------------------

do see:

> ldconfig -p | grep lz4
	liblz4.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/liblz4.so.1

Then also to rule out any gremlins I rebooted the box.

On boot, nfsen started and I saw that some nfcapd files being saved were compressed with lz4.

looking at nfsen status, I do see it is -z=lz4 is being passed to nfcapd

> sudo systemctl status nfsen
● nfsen.service - NfSen Service
     Loaded: loaded (/etc/systemd/system/nfsen.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2024-05-17 23:36:27 UTC; 11s ago
    Process: 2193948 ExecStart=/data/nfsen/bin/nfsen start (code=exited, status=0/SUCCESS)
      Tasks: 945 (limit: 38372)
     Memory: 9.0G
     CGroup: /system.slice/nfsen.service
             ├─2193959 /usr/local/bin/nfcapd -D -p 40054 -u www-data -g www-data -B 200000 -S 1 -P /data/nfsen/var/run/p40054.pid -z=lz4 -t 300 -I router1 -w /data/nfsen/profiles-data/live/router1 -s 5000
             ├─2193966 /usr/local/bin/nfcapd -D -p 40036 -u www-data -g www-data -B 200000 -S 1 -P /data/nfsen/var/run/p40036.pid -z=lz4 -t 300 -I router2 -w /data/nfsen/profiles-data/live/router2 -s 5000

I can confirm one of the uncompressed files can be compressed with nfdump just fine manually:

sudo /usr/local/bin/nfdump -r nfcapd.202405161610 -J lz4
File nfcapd.202405161610 compression changed

:/data/nfsen/profiles-data/live/router9999/2024/05/16> nfdump -v nfcapd.202405161610
File       : nfcapd.202405161610
Version    : 2 - lz4 compressed
Created    : 2024-05-17 23:49:03
Created by : nfcapd
nfdump     : f1070400
encryption : no
Appdx blks : 1
Data blks  : 228
Checking data blocks
Checking appendix blocks

Total
Type 3 blocks : 229
Records       : 2572940

But files again are no longer was being compressed.
I did not see anything in logs to give me any clue why, so thinking it was just a code bug I tried reverting nfdump code to earlier versions (built+install) to try and isolate where it might have been included.

Tried:

> git reset --hard f402ba30c342e187aa8e3eb6e616ead20c22cc11
HEAD is now at f402ba3 Merge branch 'master' of github.com:phaag/nfdump

Same issue, see lz4 being used at first:

> nfdump -v nfcapd.202405180055
File       : nfcapd.202405180055
Version    : 2 - lz4 compressed
Created    : 2024-05-18 00:59:03
Created by : nfcapd
nfdump     : f1070400
encryption : no
Appdx blks : 1
Data blks  : 24
Checking data blocks
Checking appendix blocks
 -
Total
Type 3 blocks : 25
Records       : 263303

Then checking an hour later, compression no longer being used:

:/data/nfsen/profiles-data/live/router27/2024/05/18> nfdump -v nfcapd.202405180150
File       : nfcapd.202405180150
Version    : 2 - not compressed
Created    : 2024-05-18 01:50:20
Created by : nfcapd
nfdump     : f1070400
encryption : no
Appdx blks : 1
Data blks  : 3
Checking data blocks
Checking appendix blocks

Stopped and reverted nfdump to older commit:

> git reset --hard e7cac3ed2286d532f844e1c8e7df1e548c0d212c
HEAD is now at e7cac3e Fix AS range check

Same issue, no compression..

Reverting back to the version I was running before all this started..

git reset --hard 9198d9497101caa8a5d0caef05a0a7f1180410fe

No everything using lz4 compression just fine, even hours/days later:

:/data/nfsen/profiles-data/live/router1/2024/05/18> nfdump -v nfcapd.202405180440
File       : nfcapd.202405180440
Version    : 2 - lz4 compressed
Created    : 2024-05-18 04:40:31
Created by : nfcapd
nfdump     : f1070400
encryption : no
Appdx blks : 1
Data blks  : 106
Checking data blocks
Checking appendix blocks

Total
Type 3 blocks : 107
Records       : 1480155

in case it helps, here are config/service info:

nfdump.conf

> grep -v ^# /usr/local/etc/nfdump.conf
[nfdump]
fmt.geolong = "%ts %td %pr %sc %gsap -> %dc %gdap %flg %pkt %byt %fl"
fmt.pfline = "fmt:%ts %pfact %pfrea  %pfdir on %pfifn %pfrule  %pr %sap -> %dap %pkt %byt"
geodb.path = "/data/bin/mmdb.nf"
maxworkers = 32

[nfcapd]

nfsen.conf

> grep -v ^# /data/nfsen/etc/nfsen.conf
$BASEDIR = "/data/nfsen";
$BINDIR="${BASEDIR}/bin";
$LIBEXECDIR="${BASEDIR}/libexec";
$CONFDIR="${BASEDIR}/etc";
$HTMLDIR    = "/var/www/html/nfsen/";
$DOCDIR="${HTMLDIR}/doc";
$VARDIR="${BASEDIR}/var";

$PROFILESTATDIR="${BASEDIR}/profiles-stat";
$PROFILEDATADIR="${BASEDIR}/profiles-data";
$BACKEND_PLUGINDIR="${BASEDIR}/plugins";
$FRONTEND_PLUGINDIR="${HTMLDIR}/plugins";
$PREFIX  = '/usr/local/bin';
$USER    = "www-data";
$WWWUSER  = "www-data";
$WWWGROUP = "www-data";
$BUFFLEN = 200000;
$SUBDIRLAYOUT = 1;
$ZIPcollected	 = 1;
$ZIPprofiles	 = 1;
$DISKLIMIT = 90;
$PROFILERS = 12;
$low_water = 90;
$syslog_facility = 'local3';

@plugins = (
    # profile    # module
    # [ '*',     'demoplugin' ],
);

%PluginConf = (
	# For plugin demoplugin
	demoplugin => {
		# scalar
		param2 => 42,
		# hash
		param1 => { 'key' => 'value' },
	},
	# for plugin otherplugin
	otherplugin => [
		# array
		'mary had a little lamb'
	],
);

$MAIL_FROM   = 'netflow@localhost';
$SMTP_SERVER = 'localhost';

$MAIL_BODY	 = q{
Alert '@alert@' triggered at timeslot @timeslot@
};

%sources = (
    'router1'    => { 'port' => '40190', 'col' => '#CC0099', 'type' => 'netflow', 'optarg' => '-s 5000' },
    'router2'    => { 'port' => '40194', 'col' => '#3399EE', 'type' => 'netflow', 'optarg' => '-s 5000' },
...
);
1;

nfsen.service

[Unit]
Description=NfSen Service
After=network.target

[Service]
Type=forking
ExecStart=/data/nfsen/bin/nfsen start
ExecStop=/data/nfsen/bin/nfsen stop
Restart=on-abort
TimeoutSec=900

[Install]
WantedBy=multi-user.target

cleaning up

To just document here to help others, due to amount of files (if doing layout 1) for the profile it meant that when you tried to do something like nfdump -r live/ -J lz4 it would run and eventually stop once it hit x files (guessing argument list limit internally or something). So I just segment it to doing it per subfolder in the profile, and could also have it focus on a specific day with something like:

cd /data/nfsen/profiles-data/live
find . -maxdepth 1 -type d -name r\* -exec sh -c "sudo /usr/local/bin/nfdump -r {}/2024/05/16/ -J lz4" \;

During this command, various routers would toss a "appendix offset error" for "nfcapd.202405171645".
I had to delete this nfcapd file before the above command could actually handle any of the files afterwards.
So for me was easy to just delete anything smaller than 8k as we know that would be a 'bad' file:

cd /data/nfsen/profiles-data/live
sudo find . -type f -name 'nfcapd.202405171645' -size -8k -delete

Then just re-run command, and now since it wont get hung up on those files it should be able to convert all the files...

If you find that your files ownership changed, you can fixup by doing: sudo chown <user>:<group> /data/nfsen/profiles-data/live -R

Now that all that is done, rebuild profile:

cd /data/nfsen/profiles-data/
sudo /data/nfsen/bin/nfsen -r live

---

It makes me puzzled of what changes could cause compression to stop working, and oddly for it to work for a few files then stop.. when upgrading to newer code.

[phaag]

It looks like, you did not read my comments at phaag/nfsen#33

Between the two commits, there is no change which breaks any compression whatsoever.

You can test this by yourself by compiling both commits and run a collector:

% mkdir tmp
% src/nfcapd/nfcapd -w tmp -z=lz4

then run in another shell:

% nfreplay -r <any_flow_file>

Finally temintate (cntrl-C) the collector and check the resulting file:

% src/nfcapd/nfcapd -w tmp -z=lz4
Bound to IPv4 host/IP: any, Port: 9995
Init v5/v7: Default sampling: 1
Init v9: Max number of v9 tags enabled: 106, default sampling: 1
Init IPFIX: Max number of ipfix tags enabled: 97, default sampling: 1
Startup nfcapd.
Process_v9: New v9 exporter: SysID: 1, Domain: 1, IP: 127.0.0.1
^CIdent: 'none' Flows: 1009, Packets: 2649, Bytes: 748862, Sequence Errors: 0, Bad Packets: 0, Blocks: 0
Terminating nfcapd.
% nfdump -v tmp/nfcapd.*
File       : tmp/nfcapd.202405201700
Version    : 2 - lz4 compressed
Created    : 2024-05-20 17:02:06
Created by : nfcapd
nfdump     : f1070400
encryption : no
Appdx blks : 1
Data blks  : 1
Checking data blocks
Checking appendix blocks

Total
Type 3 blocks : 2
Records       : 1013

This works likewise for both commits:

% cd nfdump-9198d94/
% src/nfdump/nfdump -V
src/nfdump/nfdump: Version: 1.7.4-9198d94  Date: 2024-03-17 17:52:55 +0100
% cd ../nfdump-564f3a9
% src/nfdump/nfdump -V
src/nfdump/nfdump: Version: 1.7.4-564f3a9  Date: 2024-05-11 10:55:50 +0200

Regarding the missing lz4.h in config.log. As I explained in phaag/nfsen#33

conftest.c:85:10: fatal error: 'lz4.h' file not found
#include <lz4.h>
         ^~~~~~~
1 error generated.
configure:16827: $? = 1

This is a test, if the local lz4 library is installed on the system. If so, it uses and links the system provided lz4 librray,
otherwise it compiles the provided lz4 code. This is also identical in both commits. The compression lz4 is always included in the final binary.

I suspect, that you maybe have some local inconsistencies on your system or in NfSen not the latest version. The compression flags are controlled by NfSen, when starting up the collectors.

However, your case is not an nfdump/nfcapd issue.

[thezoggy]

I understood what you said, however this was not the case.

After upgrading and figuring out that the issue was nfcapd was not compressing the files I went back at the config log and saw the lz4 lib not being included. Which I thought was odd since lz4 was just previously working on previous version and lz4 working fine when I tried it out. So thought hey maybe liblz4-dev was needed, installed that and when building nfdump from latest, I saw lz4 lib was included just fine... However after starting nfsen I saw nfcapd using lz4 compression like before, but shortly after a few files it stopped. Looking at processes I could see -z=lz4 is being passed and puzzled why it could compress just fine if I did it manually but why it was not doing it itself anymore.

I tried reverting nfdump to earlier versions as noted, but each time I kept having the same issue. I start nfsen, see nfcapd store flows with lz4... then shortly afterwards the new files would be uncompressed.. so lz4 no longer used. I was trying to isolate what change and reverted to various versions, but each time the same outcome. I gave up and went back to 9198d94 and that version, compression works reliably.

I honestly have no clue if its some sort of forking/threading issue where settings get loss or what, why I shared the configs/service, and also if there was some lib linking issue which is why I shared the process I follow to do the upgrade.

[phaag]

Again, check your NfSen installation and correct settings as well as using the latest github main branch. The ps output shows the running collectors and their arguments.

It is not an nfdump/nfcapd issue.

[thezoggy]

Again, check your NfSen installation and correct settings as well as using the latest github main branch. The ps output shows the running collectors and their arguments.

It is not an nfdump/nfcapd issue.

I shared what the ps output showed above, the command used then vs now is exactly the same (just different pid). Which i noted I see -z=lz4 being passed even when it was not working. nfsen is latest version, and was untouched when doing the nfdump upgrade. When reverting back to older nfdump the issue went away.

Looking back at logs around may 16th 1600 utc is when i stopped nfsen to upgrade nfdump something does stand out now that I'm digging through this.
When nfsen does the process to update live profile, before upgrade fine, after upgrade exit 1, and then now back to older version back to being fine.

netflow4:/var/log> sudo grep -A5 "Update profile live in group" messages-20240519
...
May 16 15:55:15 netflow4.vm nfsen[810477]: Update profile live in group .
May 16 15:55:24 netflow4.vm nfsen[1297208]: Run expire at Thu May 16 15:55:00 2024
May 16 15:55:24 netflow4.vm nfsen[1297208]: Expire has 220s in this slot!
May 16 15:55:24 netflow4.vm nfsen[1297208]: Expire profile live group . low water mark: 90% 
May 16 15:55:24 netflow4.vm nfsen[1297208]: End expire at Thu May 16 15:55:00 2024
May 16 15:57:22 netflow4.vm systemd[1]: Stopping NfSen Service...
--
May 16 16:00:16 netflow4.vm nfsen[810477]: Update profile live in group .
May 16 16:00:16 netflow4.vm nfcapd[14033]: Ident: 'router25' Flows: 612703, Packets: 4559730000, Bytes: 3714702190000, Sequence Errors: 381, Bad Packets: 0, Blocks: 0
May 16 16:00:16 netflow4.vm nfcapd[14033]: Terminating nfcapd.
May 16 16:00:17 netflow4.vm nfcapd[3847]: Ident: 'router67' Flows: 211020, Packets: 1818595000, Bytes: 1722365545000, Sequence Errors: 237, Bad Packets: 0, Blocks: 0
May 16 16:00:17 netflow4.vm nfcapd[3847]: Terminating nfcapd.
May 16 16:00:18 netflow4.vm nfcapd[16033]: Ident: 'router68' Flows: 4362, Packets: 37765000, Bytes: 29293850000, Sequence Errors: 0, Bad Packets: 0, Blocks: 0
--
May 16 16:07:14 netflow4.vm nfsen[1311241]: Update profile live in group .
May 16 16:07:14 netflow4.vm nfcapd[1310549]: Append new sampler id: -1, algorithm: 1, packet interval: 1, packet space: 4999
May 16 16:07:14 netflow4.vm nfsen[1311241]: Run nfdump failed: Exit: 1, Signal: 0, Coredump: 0
May 16 16:07:14 netflow4.vm nfsen[1311241]: message repeated 2 times: [ Run nfdump failed: Exit: 1, Signal: 0, Coredump: 0]
May 16 16:07:14 netflow4.vm nfcapd[1311235]: Add new sampler id: -2, algorithm: 0, packet interval: 1, packet space: 4999
May 16 16:07:14 netflow4.vm nfcapd[1311235]: Process_ipfix: New ipfix exporter: SysID: 36, Observation domain 589827 from: 129.250.0.48
--
May 16 16:10:15 netflow4.vm nfsen[1311241]: Update profile live in group .
May 16 16:10:16 netflow4.vm nfcapd[1311172]: Append new sampler id: 1, algorithm: 2, packet interval: 1, packet space: 4999
May 16 16:10:17 netflow4.vm systemd[1]: Stopping User Manager for UID 0...
May 16 16:10:17 netflow4.vm systemd[1311990]: Stopped target Main User Target.
May 16 16:10:17 netflow4.vm systemd[1311990]: Stopped target Basic System.
May 16 16:10:17 netflow4.vm systemd[1311990]: Stopped target Paths.
--
May 16 16:15:15 netflow4.vm nfsen[1311241]: Update profile live in group .
May 16 16:15:15 netflow4.vm nfsen[1311241]: Run nfdump failed: Exit: 1, Signal: 0, Coredump: 0
May 16 16:15:17 netflow4.vm nfsen[1311241]: message repeated 262 times: [ Run nfdump failed: Exit: 1, Signal: 0, Coredump: 0]
May 16 16:15:21 netflow4.vm nfsen[1314881]: Run expire at Thu May 16 16:15:00 2024
May 16 16:15:21 netflow4.vm nfsen[1314881]: Expire has 223s in this slot!
May 16 16:15:21 netflow4.vm nfsen[1314881]: Expire profile live group . low water mark: 90% 
...

I wonder if I can correlate that each time it did that it caused nfdump to stop using compression.. as you can see first run fine then 2nd+ exit.. which matches up with what I was seeing.

and just like before currently, working fine:

May 18 23:55:15 netflow4.vm nfsen[322631]: Update profile live in group .
May 18 23:55:20 netflow4.vm nfsen[2640158]: Run expire at Sat May 18 23:55:00 2024
May 18 23:55:20 netflow4.vm nfsen[2640158]: Expire has 224s in this slot!
May 18 23:55:20 netflow4.vm nfsen[2640158]: Expire profile live group . low water mark: 90% 
May 18 23:55:21 netflow4.vm nfsen[2640158]: End expire at Sat May 18 23:55:00 2024

Is there any recent changes to nfdump that would require updates to nfsen to make something play nice again?
As nothing has changed in nfsen-dist.conf in awhile..

When I get some free time this week I'll try to replicate it and see if I can triage it better with your suggestions.