Transaction lookup using eth_getTransactionReceipt returns the wrong information after reorg

[tobidae-cb]

System information

Geth version: Geth/v1.13.11-stable-8f7eb9cc/linux-amd64/go1.20.7
CL client & version: lighthouse@v4.6.0
OS & Version: Linux

Expected behaviour

eth_getTransactionReceipt returns the right receipt information including block hash, number, e.t.c. Even when a reorg occurs on that block and the transaction is mined in another block.

Actual behaviour

eth_getTransactionReceipt sometimes returns the receipt with information from the reorg'd block. This causes our internal systems to fail. This is especially observed in the sepolia/holesky testnets since 1.13.11 is required for the upcoming hardfork.

Steps to reproduce the behaviour

1. Upgrade to 1.13.11 stable version along with lighthouse 4.6.0
2. Run an archival node with the settings below (although it can possibly happen in full nodes)

exec /app/geth \
    --networkid "17000" \
    --syncmode="full" \
    --gcmode="archival" \
    --snapshot="true" \
    --rpc.txfeecap=0 \
    --rpc.batch-request-limit 100000 \
    --rpc.batch-response-max-size 500000000 \
    --http \
    --http.addr 0.0.0.0 \
    --http.vhosts=* \
    --http.api="engine,eth,web3,net,debug,txpool" \
    --http.corsdomain=* \
    --authrpc.addr 0.0.0.0 \
    --authrpc.vhosts=* \
    --cache "16384" \
    --state.scheme hash

3. When a reorg occurs there's a probability that the cache for eth_getTransactionReceipt stores the wrong receipt information so any subsequent calls to get a receipt returns the wrong information. When querying the block using the hash or number, the right details are contained there

Backtrace

Logs that had reorg

# Specific transaction with observed behavior
curl -X POST -H "Content-Type: application/json" --data '{"method":"eth_getTransactionReceipt","params":["0x36057fedd64685fb69ba23f9d49cbf7b1c0039659d24adaf2b60274b716ef012"],"id":0,"jsonrpc":"2.0"}' http://localhost:8545 | jq

# Actual Result
{
  "jsonrpc": "2.0",
  "id": 0,
  "result": {
    "blockHash": "0xa0056303bb52371aa38b7996f031a9d10bc8dd1d78ce921b5047abb47334ad7a",
    "blockNumber": "0xc78e8",
    "contractAddress": null,
    "cumulativeGasUsed": "0x1c7bac2",
    "effectiveGasPrice": "0x3b9aca00",
    "from": "0x9146a7ece7054c39f2d27dfc45efa8fa8e1d1323",
    "gasUsed": "0xfaa5e",
    "logs": [],
    "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
    "status": "0x1",
    "to": "0x3f5dfd1522a84394390382a9801616bd530a9655",
    "transactionHash": "0x36057fedd64685fb69ba23f9d49cbf7b1c0039659d24adaf2b60274b716ef012",
    "transactionIndex": "0x2d",
    "type": "0x2"
  }
}


# Expected result
{
  "jsonrpc": "2.0",
  "id": 0,
  "result": {
    "blockHash": "0x046495e9e416d899e933c21cd0539aad3d90f6888bd46007246456189c482299",
    "blockNumber": "0xc78e9",
    "contractAddress": null,
    "cumulativeGasUsed": "0xffe36",
    "effectiveGasPrice": "0x3b9aca00",
    "from": "0x9146a7ece7054c39f2d27dfc45efa8fa8e1d1323",
    "gasUsed": "0xfaa5e",
    "logs": [],
    "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
    "status": "0x1",
    "to": "0x3f5dfd1522a84394390382a9801616bd530a9655",
    "transactionHash": "0x36057fedd64685fb69ba23f9d49cbf7b1c0039659d24adaf2b60274b716ef012",
    "transactionIndex": "0x1",
    "type": "0x2"
  }
}


# Logs from problematic deployment
For prod ethereum-0 archival holesky
ethereum-0 geth INFO [01-26|22:08:21.009] Chain head was updated                   number=817,384 hash=a00563..34ad7a root=51418b..2f5c82 elapsed=4.17309ms
ethereum-0 geth INFO [01-26|22:08:28.259] Imported new potential chain segment     number=817,384 hash=319dbf..fec640 blocks=1 txs=55 mgas=29.916 elapsed=128.272ms   mgasps=233.222 snapdiffs=1.35MiB triedirty=0.00B
ethereum-0 geth INFO [01-26|22:08:29.254] Imported new potential chain segment     number=817,385 hash=046495..482299 blocks=1 txs=33 mgas=29.944 elapsed=135.098ms   mgasps=221.645 snapdiffs=1.35MiB triedirty=0.00B
ethereum-0 geth INFO [01-26|22:08:31.895] Chain reorg detected                     number=817,383 hash=2bd818..756b40 drop=1 dropfrom=a00563..34ad7a add=2 addfrom=046495..482299
ethereum-0 geth INFO [01-26|22:08:31.904] Chain head was updated                   number=817,385 hash=046495..482299 root=c7fbc7..da6141 elapsed=9.360949ms
ethereum-0 geth INFO [01-26|22:08:38.883] Imported new potential chain segment     number=817,386 hash=6b13b0..95ed30 blocks=1 txs=23 mgas=9.466  elapsed=43.761ms    mgasps=216.302 snapdiffs=1.35MiB triedirty=0.00B
ethereum-0 geth INFO [01-26|22:08:39.176] Chain head was updated                   number=817,386 hash=6b13b0..95ed30 root=b1f284..f364cd elapsed=4.695815ms

# Logs from expected behavior
ethereum-0 geth INFO [01-26|22:07:39.776] Imported new potential chain segment     number=817,383 hash=2bd818..756b40 blocks=1    txs=0       mgas=0.000    elapsed=3.333ms     mgasps=0.000   snapdiffs=3.18MiB    triedirty=0.00B
ethereum-0 geth INFO [01-26|22:07:40.072] Chain head was updated                   number=817,383 hash=2bd818..756b40 root=c0724c..aab761 elapsed="815.439µs"
ethereum-0 geth INFO [01-26|22:08:14.904] Imported new potential chain segment     number=817,384 hash=a00563..34ad7a blocks=1    txs=49      mgas=29.998   elapsed=146.869ms   mgasps=204.250 snapdiffs=3.18MiB    triedirty=0.00B
ethereum-0 geth INFO [01-26|22:08:20.822] Chain head was updated                   number=817,384 hash=a00563..34ad7a root=51418b..2f5c82 elapsed=3.650328ms
ethereum-0 geth INFO [01-26|22:08:23.798] Imported new potential chain segment     number=817,384 hash=319dbf..fec640 blocks=1    txs=55      mgas=29.916   elapsed=122.690ms   mgasps=243.832 snapdiffs=3.19MiB    triedirty=0.00B
ethereum-0 geth INFO [01-26|22:08:24.836] Imported new potential chain segment     number=817,385 hash=046495..482299 blocks=1    txs=33      mgas=29.944   elapsed=120.422ms   mgasps=248.658 snapdiffs=3.19MiB    triedirty=0.00B
ethereum-0 geth INFO [01-26|22:08:26.823] Chain reorg detected                     number=817,383 hash=2bd818..756b40 drop=1 dropfrom=a00563..34ad7a add=1 addfrom=319dbf..fec640
ethereum-0 geth INFO [01-26|22:08:26.829] Chain head was updated                   number=817,384 hash=319dbf..fec640 root=aa8a16..a2261d elapsed=6.317064ms
ethereum-0 geth INFO [01-26|22:08:27.667] Chain head was updated                   number=817,385 hash=046495..482299 root=c7fbc7..da6141 elapsed=4.855725ms
ethereum-0 geth INFO [01-26|22:08:38.878] Imported new potential chain segment     number=817,386 hash=6b13b0..95ed30 blocks=1    txs=23      mgas=9.466    elapsed=39.822ms    mgasps=237.699 snapdiffs=3.19MiB    triedirty=0.00B
ethereum-0 geth INFO [01-26|22:08:39.170] Chain head was updated                   number=817,386 hash=6b13b0..95ed30 root=b1f284..f364cd elapsed=1.247488ms
ethereum-0 geth INFO [01-26|22:08:49.142] Imported new potential chain segment     number=817,387 hash=a6dd16..dbda80 blocks=1    txs=38      mgas=28.078   elapsed=109.512ms   mgasps=256.391 snapdiffs=3.19MiB    triedirty=0.00B
ethereum-0 geth INFO [01-26|22:08:49.440] Chain head was updated                   number=817,387 hash=a6dd16..dbda80 root=b8bece..4073f3 elapsed=1.507894ms
ethereum-0 geth INFO [01-26|22:09:02.154] Regenerated local transaction journal    transactions=0 accounts=0

When submitting logs: please submit them as text and not screenshots.

[tobidae-cb]

Possibly related bug introduction - #28703

Possible fix - #28865

[lightclient]

Thanks for filing this. I think you are correct and we'll try to get that PR merged in.

[vr-devil]

after upgrade to geth version 1.13.11-stable-8f7eb9cc. i encountered same problems when chain reorged.

geth log.

INFO [01-29|01:58:51.163] Imported new potential chain segment     number=19,109,058 hash=036941..3d0e6a blocks=1          txs=122           mgas=14.757  elapsed=44.882ms     mgasps=328.798 snapdiffs=5.89MiB    triedirty=1.41GiB
INFO [01-29|01:58:51.248] Chain head was updated                   number=19,109,058 hash=036941..3d0e6a root=013716..b3eeb3 elapsed=3.599635ms
INFO [01-29|01:59:02.428] Imported new potential chain segment     number=19,109,058 hash=3b5329..9eb85b blocks=1          txs=190           mgas=20.047  elapsed=58.085ms     mgasps=345.128 snapdiffs=5.94MiB    triedirty=1.41GiB
WARN [01-29|01:59:08.921] Served eth_call                          reqid=437154 duration="233.774µs"   err="execution reverted: arithmetic underflow or overflow" errdata=0x4e487b710000000000000000000000000000000000000000000000000000000000000011
INFO [01-29|01:59:09.017] Chain reorg detected                     number=19,109,057 hash=3e4906..ad730b drop=1 dropfrom=036941..3d0e6a add=1 addfrom=3b5329..9eb85b
INFO [01-29|01:59:09.025] Chain head was updated                   number=19,109,058 hash=3b5329..9eb85b root=8fc057..a4434d elapsed=8.064401ms

my program can't get right receipt, and wrong receipt can't automatically recover. always wrong. now, my program stucked, can't proceed.

DEBU[2024-01-29T10:33:42+08:00] log Block:19109058, BlockHash:0x3b5329221a4116442e4ce8ae0c2a3550af1e93a675667088b7214ee5079eb85b, TxIndex:99, TxHash: 0xb8a926f27d1700683570ad3838166e383fe887353ebe6c59d48749ff9d4c9f58, 
DEBU[2024-01-29T10:33:42+08:00] receipt Block:19109058, BlockHash:0x0369413459669e271b9a8124184c58758271fc03177967f0cfb5a164643d0e6a, TxIndex:85, TxHash: 0xb8a926f27d1700683570ad3838166e383fe887353ebe6c59d48749ff9d4c9f58,

[vr-devil]

update:
geth has cache lead to wrong receipt in period. after 1 hour later, my program can get right receipt。

this is my command arguments.

geth --datadir /var/lib/ethereum/mainnet/execution --datadir.ancient /var/lib/ethereum/mainnet/execution/ancient --mainnet --cache 8192 --history.transactions 0

[Enigmatic331]

Any chance this might see a fix before the Sepolia hardfork? 🙏 Thanks 🙏

[demon-xxi]

I've observed this issue manifesting in various edge cases. It appears to be linked to the current design of the eth_getTransactionReceipt method. The core concern is that its output is not consistently idempotent following blockchain reorganizations. To enhance its reliability and utility, I'd propose extending the method with an optional parameter for the specific block hash. This addition would not only bolster the consistency of the results in line with the caller's expectations but also offer the added benefit of retrieving transaction receipts from uncle blocks. This approach could significantly improve the method's robustness in handling diverse blockchain scenarios.

[rjl493456442]

but also offer the added benefit of retrieving transaction receipts from uncle blocks

There is no guarantee the side blocks along with their data(e.g. receipts) will be stored. We don't really want to provide an API to retrieve something which may or may not exist.

[rjl493456442]

This issue is fixed by #28865, will be shipped by the next release. Sorry for the chaos.

[eldarbogdanov]

@rjl493456442 As a result of this fix, would an Archival node that was previously running v1.13.11, and synced an incorrect txn receipt, fix its history? Or do I need to resync Archival nodes from the moment of upgrading to v1.13.11? 😬