Unable to compile since 0.26.7 version and above

[alexander-akait]

Bug description

Error:

▌ Whoops, the AssemblyScript compiler has crashed during optimize (converge) :-(
▌ 
▌ Here is the stack trace hinting at the problem, perhaps it's useful?
▌ 
▌ RuntimeError: null function or function signature mismatch
▌     at wasm://wasm/013de5e2:wasm-function[106]:0x10895
▌     at wasm://wasm/013de5e2:wasm-function[4324]:0x2b104a
▌     at A._BinaryenModuleRunPasses (file:///home/akait/IdeaProjects/webpack/node_modules/binaryen/index.js:9:138831)
▌     at yl.runPasses (file:///home/akait/IdeaProjects/webpack/node_modules/assemblyscript/dist/assemblyscript.js:17:35834)
▌     at yl.optimize (file:///home/akait/IdeaProjects/webpack/node_modules/assemblyscript/dist/assemblyscript.js:17:39578)
▌     at Module.t0 (file:///home/akait/IdeaProjects/webpack/node_modules/assemblyscript/dist/assemblyscript.js:298:2304)
▌     at Module.Me (file:///home/akait/IdeaProjects/webpack/node_modules/assemblyscript/dist/asc.js:22212:10339)
▌     at async /home/akait/IdeaProjects/webpack/tooling/generate-wasm-code.js:33:22
▌ 
▌ If you see where the error is, feel free to send us a pull request. If not,
▌ please let us know: https://github.com/AssemblyScript/assemblyscript/issues
▌ 
▌ Thank you!

Steps to reproduce

Input:

// //////////////////////////////////////////////////////////
// xxhash64.h
// Copyright (c) 2016 Stephan Brumme. All rights reserved.
// see http://create.stephan-brumme.com/disclaimer.html
//
// XXHash (64 bit), based on Yann Collet's descriptions, see
// http://cyan4973.github.io/xxHash/
//
// Modified for hash-wasm by Dani Biró
//
// Ported to assemblyscript by Tobias Koppers
// Modifications:
// - seed is always 0
// - update is only called with a multiple of 32
// - final takes the remaining 0 - 31 bytes
//

const Prime1: u64 = 11400714785074694791;
const Prime2: u64 = 14029467366897019727;
const Prime3: u64 = 1609587929392839161;
const Prime4: u64 = 9650029242287828579;
const Prime5: u64 = 2870177450012600261;

let state0: u64;
let state1: u64;
let state2: u64;
let state3: u64;
let totalLength: u64;

function processSingle(previous: u64, input: u64): u64 {
	return rotl(previous + input * Prime2, 31) * Prime1;
}

export function init(): void {
	state0 = Prime1 + Prime2;
	state1 = Prime2;
	state2 = 0;
	state3 = 0 - Prime1;
	totalLength = 0;
}

export function update(length: u32): void {
	if (length == 0) return;

	totalLength += length;

	let dataPtr: u32 = 0;

	let s0 = state0;
	let s1 = state1;
	let s2 = state2;
	let s3 = state3;

	do {
		s0 = processSingle(s0, load<u64>(dataPtr));
		s1 = processSingle(s1, load<u64>(dataPtr + 8));
		s2 = processSingle(s2, load<u64>(dataPtr + 16));
		s3 = processSingle(s3, load<u64>(dataPtr + 24));
		dataPtr += 32;
	} while (dataPtr < length);

	state0 = s0;
	state1 = s1;
	state2 = s2;
	state3 = s3;
}

export function final(length: u32): void {
	// fold 256 bit state into one single 64 bit value
	let result: u64;
	if (totalLength > 0) {
		result =
			rotl(state0, 1) + rotl(state1, 7) + rotl(state2, 12) + rotl(state3, 18);
		result = (result ^ processSingle(0, state0)) * Prime1 + Prime4;
		result = (result ^ processSingle(0, state1)) * Prime1 + Prime4;
		result = (result ^ processSingle(0, state2)) * Prime1 + Prime4;
		result = (result ^ processSingle(0, state3)) * Prime1 + Prime4;
	} else {
		result = Prime5;
	}

	result += totalLength + length;

	let dataPtr: u32 = 0;

	// at least 8 bytes left ? => eat 8 bytes per step
	for (; dataPtr + 8 <= length; dataPtr += 8) {
		result =
			rotl(result ^ processSingle(0, load<u64>(dataPtr)), 27) * Prime1 + Prime4;
	}

	// 4 bytes left ? => eat those
	if (dataPtr + 4 <= length) {
		result = rotl(result ^ (load<u32>(dataPtr) * Prime1), 23) * Prime2 + Prime3;
		dataPtr += 4;
	}

	// take care of remaining 0..3 bytes, eat 1 byte per step
	while (dataPtr !== length) {
		result = rotl(result ^ (load<u8>(dataPtr) * Prime5), 11) * Prime1;
		dataPtr++;
	}

	// mix bits
	result ^= result >> 33;
	result *= Prime2;
	result ^= result >> 29;
	result *= Prime3;
	result ^= result >> 32;

	store<u64>(0, u32ToHex(result >> 32));
	store<u64>(8, u32ToHex(result & 0xffffffff));
}

function u32ToHex(x: u64): u64 {
	// from https://johnnylee-sde.github.io/Fast-unsigned-integer-to-hex-string/

	x = ((x & 0xffff) << 32) | ((x & 0xffff0000) >> 16);
	x = ((x & 0x0000ff000000ff00) >> 8) | ((x & 0x000000ff000000ff) << 16);
	x = ((x & 0x00f000f000f000f0) >> 4) | ((x & 0x000f000f000f000f) << 8);

	const mask = ((x + 0x0606060606060606) >> 4) & 0x0101010101010101;

	x |= 0x3030303030303030;

	x += 0x27 * mask;

	return x;
}

AssemblyScript version

v0.26.7

[CountBleck]

That's really strange. Binaryen is crashing, and the error is completely different for me. Even the function that's called by AS at the crash is different (_BinaryenModuleAllocateAndWrite vs _BinaryenModuleRunPasses). I'll see what I can find.

▌ Whoops, the AssemblyScript compiler has crashed during emitBinary :-(
▌ 
▌ Here is the stack trace hinting at the problem, perhaps it's useful?
▌ 
▌ TypeError: c(...) is not a function
▌     at QB (file:///home/turtle/Projects/as-xxhash-webpack-issue/node_modules/binaryen/index.js:9:155802)
▌     at wasm://wasm/014183e2:wasm-function[93]:0xf678
▌     at wasm://wasm/014183e2:wasm-function[5042]:0x331f42
▌     at DB (file:///home/turtle/Projects/as-xxhash-webpack-issue/node_modules/binaryen/index.js:9:156323)
▌     at wasm://wasm/014183e2:wasm-function[5094]:0x345f17
▌     at A._BinaryenModuleAllocateAndWrite (file:///home/turtle/Projects/as-xxhash-webpack-issue/node_modules/binaryen/index.js:9:138932)
▌     at file:///home/turtle/Projects/as-xxhash-webpack-issue/node_modules/binaryen/index.js:10:674
▌     at w (file:///home/turtle/Projects/as-xxhash-webpack-issue/node_modules/binaryen/index.js:9:162613)
▌     at B.emitBinary (file:///home/turtle/Projects/as-xxhash-webpack-issue/node_modules/binaryen/index.js:10:651)
▌     at Module.Me (/home/turtle/Projects/as-xxhash-webpack-issue/node_modules/assemblyscript/cli/index.js:899:31)
▌ 
▌ If you see where the error is, feel free to send us a pull request. If not,
▌ please let us know: https://github.com/AssemblyScript/assemblyscript/issues
▌ 
▌ Thank you!

[dcodeIO]

Seems to be limited to optimizing a module with -O3, specifically --optimizeLevel 3, which is the default.

[MaxGraey]

Can reproduce in playground. It seems some bug in binaryen. Probably make sense to update their version

[dcodeIO]

I've locally updated to latest Binaryen. There, I am seeing RuntimeError: null function or function signature mismatch as well on the std/map test when optimizing. When running the passes individually, the error appears on inlining-optimizing.

However, when I try with the above snippet with latest Binaryen, I am seeing TypeError: c(...) is not a function upon remove-unused-module-elements.

[CountBleck]

I got an error even without optimizations (-O0).

Is a "memory access out of bounds" error? I reached that error without optimizations, but I'd blame -O0...although -sALLOW_MEMORY_GROWTH is enabled.

[dcodeIO]

Note that this issue was closed automatically via merge. Let's reopen this if it turns out that the workaround doesn't work.

[CountBleck]

was closed automatically via merge

Whoops! Sorry about that.

[mnater]

This might also solve #2670

[dcodeIO]

Updating to the nightly build of Binaryen, which applies the larger stack size, indeed unblocks #2683. Note, though, that this is merely a workaround, whereas the root cause is in Binaryen.

Regarding #2670, see #2683 (comment). Appears that these two issues are only seemingly related, in that both blow the stack, but for different reasons.

[dcodeIO]

Regarding the snippet above, the stack overflow can be shown in action by applying the following diff:

export function final(length: u32): void {
	// fold 256 bit state into one single 64 bit value
	let result: u64;
	if (totalLength > 0) {
		result =
			rotl(state0, 1) + rotl(state1, 7) + rotl(state2, 12) + rotl(state3, 18);
		result = (result ^ processSingle(0, state0)) * Prime1 + Prime4;
		result = (result ^ processSingle(0, state1)) * Prime1 + Prime4;
		result = (result ^ processSingle(0, state2)) * Prime1 + Prime4;
-               result = (result ^ processSingle(0, state3)) * Prime1 + Prime4;
+               // result = (result ^ processSingle(0, state3)) * Prime1 + Prime4;

Commenting out that one line, that would otherwise be optimized into a larger sequence of binary expressions, suppresses the issue as the sequence becomes just short enough to not produce the observed effects. Uncommenting manifests the issue. It's not hard to see how similar sequences can be crafted to force overflowing any stack size.