Increase maximum supported fixed args on varargs calls from 3 to 255

[andrew-nowak]

I was encountering an issue where JNA would only support varargs calls when the function contained <= 3 fixed args. Thanks to @matthiasblaesing for pointing me to

jna/src/com/sun/jna/Function.java

Lines 79 to 81 in 54fc662

	/** Mask for number of fixed args (1-3) for varargs calls. */
	@java.lang.annotation.Native
	public static final int USE_VARARGS = 0x180;

which explains that JNA would only report up to 3 fixed args to libffi on a varargs function call, and for suggesting this patch.

I only encountered the issue only when running on a JVM for darwin-arm64, and not on x86_64 or on linux-arm64, which I think is explained by this block in libffi:

jna/native/libffi/src/aarch64/ffi.c

Lines 610 to 628 in 54fc662

	#if defined (__APPLE__)
	/* Perform Apple-specific cif processing for variadic calls */
	ffi_status FFI_HIDDEN
	ffi_prep_cif_machdep_var(ffi_cif *cif, unsigned int nfixedargs,
	unsigned int ntotalargs)
	{
	ffi_status status = ffi_prep_cif_machdep (cif);
	cif->aarch64_nfixedargs = nfixedargs;
	return status;
	}
	#else
	ffi_status FFI_HIDDEN
	ffi_prep_cif_machdep_var(ffi_cif *cif, unsigned int nfixedargs, unsigned int ntotalargs)
	{
	ffi_status status = ffi_prep_cif_machdep (cif);
	cif->flags |= AARCH64_FLAG_VARARG;
	return status;
	}
	#endif /* __APPLE__ */

nfixedargs is only used on the #ifdef __APPLE__ branches, and ignored otherwise (and as far as I can tell ignored entirely on x86/x86_64).

I've built JNA with this patch and can confirm that it fixes both my original problem and my minimal-ish reproduction. I've added a testcase but of course it only demonstrates the fix on architectures that had the problem before.

[matthiasblaesing]

Thank you for looking into this. We looked into this in parallel, so I'd ask you to have a look here:

https://github.com/matthiasblaesing/jna/tree/longvarargs

• Commits dfa9fff and 7071f08 implement a similar approach to yours. The difference is, that I pulled out the shift and mask for the argument count into constants so that they are named and ensured to be identical on both sides of the java <-> native border
• Commit 2032eb7 ensures that the checksum of the native interface also considers the defines. As these need to be identical on both sides, they need to be considered for the interface.
• Commit 88a4f0d updates the checksum and version for the native interface

Please have a look at the changes and consider adapting them into your patch.

There are formatting errors: https://github.com/java-native-access/jna/actions/runs/4216898273/jobs/7320850642

This needs mentioning in the CHANGES.md file under bugfixes. I suggest to add an Important changes (see release 5.12.0) section, that describes, that libjnidispatch must be rebuild as the java<->native interfaces changed.

I'll take care of rebuilding the native libraries after the changes are in.

[andrew-nowak]

Thanks for the suggestions! I've reworked my commit as you suggested, and cherry-picked your checksum changes on top

[matthiasblaesing]

Thank you. Looks sane to me. It'll take a few days to do the native builds, when that is done, I'll create a new PR, that will merge this work, together with the prebuild-library update.

[matthiasblaesing]

You enabled changes to your branch for maintainers when you opened the PR. This makes it possible, that I'll push the updated native libraries through this PR. Please don't be surprised when your branch changes.

[matthiasblaesing]

Build done, github actions happy, appveyor builds happy. Let's get this in @andrew-nowak thanks for working on this.