🐛 [cloud_firestore] Still ServerTimestampBehavior crashes #10715

tecbot · 2023-04-03T12:49:54Z

Bug report

Describe the bug
Still crashes related to ServerTimestampBehavior after bug fix #10391 and #10314

cloud_firestore: 4.4.5

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Subtype: KERN_INVALID_ADDRESS at 0x000078facf0484d0

0   libobjc.A.dylib               	0x00000001cc0a1c20 objc_msgSend + 32
1   CoreFoundation                	0x00000001d2cef994 -[__NSCFNumber compare:] + 64 (CFObject.m:1134)
2   CoreFoundation                	0x00000001d2cef938 -[__NSCFNumber isEqualToNumber:] + 24 (CFObject.m:853)
3   CoreFoundation                	0x00000001d2cee4c4 -[__NSDictionaryM objectForKeyedSubscript:] + 164 (NSDictionaryM_Common.h:32)
4   Runner                        	0x0000000100d811ac -[FLTFirebaseFirestoreWriter FIRQuerySnapshot:] + 204 (FLTFirebaseFirestoreWriter.m:199)
5   Runner                        	0x0000000100d805e8 -[FLTFirebaseFirestoreWriter writeValue:] + 832 (FLTFirebaseFirestoreWriter.m:47)

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x00004e9d6e9fc9f0

0  libobjc.A.dylib                0x2820 objc_msgSend + 32
1  CoreFoundation                 0x32ec -[__NSCFNumber compare:] + 64
2  CoreFoundation                 0x3290 -[__NSCFNumber isEqualToNumber:] + 24
3  CoreFoundation                 0x1e24 -[__NSDictionaryM objectForKeyedSubscript:] + 164
4  Runner                         0x478d64 -[FLTFirebaseFirestoreWriter FIRDocumentSnapshot:] + 153 (FLTFirebaseFirestoreWriter.m:153)
5  Runner                         0x478548 -[FLTFirebaseFirestoreWriter writeValue:] + 43 (FLTFirebaseFirestoreWriter.m:43)

EXC_BREAKPOINT 0x0000000196699318

0  CoreFoundation                 0x7e318 CFNumberGetValue + 160
1  CoreFoundation                 0x64170 -[__NSCFNumber unsignedLongLongValue] + 48
2  CoreFoundation                 0x35c0 -[__NSCFNumber compare:] + 124
3  CoreFoundation                 0x3528 -[__NSCFNumber isEqualToNumber:] + 24
4  CoreFoundation                 0xa0dc mdict_removeObjectForKey + 164
5  Runner                         0x478dd8 -[FLTFirebaseFirestoreWriter FIRDocumentSnapshot:] + 159 (FLTFirebaseFirestoreWriter.m:159)
6  Runner                         0x478548 -[FLTFirebaseFirestoreWriter writeValue:] + 43 (FLTFirebaseFirestoreWriter.m:43)

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Subtype: KERN_INVALID_ADDRESS at 0x0000f3b4d44e4bb0 -> 0x00000034d44e4bb0 (possible pointer authentication failure)

0   libobjc.A.dylib               	0x00000001bcb0dc20 objc_msgSend + 32 (:-1)
1   CoreFoundation                	0x00000001c39950dc mdict_removeObjectForKey + 164 (NSDictionaryM_Common.h:32)
2   Runner                        	0x0000000104c84dd8 -[FLTFirebaseFirestoreWriter FIRDocumentSnapshot:] + 252 (FLTFirebaseFirestoreWriter.m:159)
3   Runner                        	0x0000000104c84548 -[FLTFirebaseFirestoreWriter writeValue:] + 672 (FLTFirebaseFirestoreWriter.m:43)

The text was updated successfully, but these errors were encountered:

danagbemava-nc · 2023-04-04T10:30:23Z

Hi @tecbot, can you provide a minimal sample? Can you also check if it reproduces on the latest version of cloud_firestore?

tecbot · 2023-04-04T11:14:38Z

@danagbemava-nc I can't create a minimal sample because this error happen only in production and only for a rare amount of users. There are no relevant code changes between 4.4.5 and 4.5.0 which could fix this issue, therefore im sure this issue still exists in 4.5.0. Are those 3 stack traces not helpful?

danagbemava-nc · 2023-04-05T10:04:15Z

Hi @tecbot, it's standard procedure to request reporters to try on newer versions whenever a bug is reported on an "older" version of a package.

Since this is a production bug with no steps to reproduce, I'll be labeling this based on the crash logs provided above.

cc @Lyokone

Lyokone · 2023-04-05T12:15:28Z

Hello @tecbot,
Are you using ServerTimestampBehavior in your code? Can you at least provide the code used to call Firestore?
Are you also using a background runner that could call native calls simultaneously?

I have a tentative fix in #10728, could you see if it solves your issue?

tecbot · 2023-04-05T12:55:01Z

Hey @Lyokone. No we don't use this feature. We have upgraded from version 4.2.0 to 4.4.5 and the crashes started to occur.
We have 100+ places where we access firestore in our app. Which of them could be the root of this I can't figure out form the crash logs, sorry.

We have some background plugins but none of them accessing firestore.

Regarding the fix in #10728 we could try this out. Approx. I can give you an answer in ~7 days after we have slowed rolled out to customers. However I think the fix also needs to be included in the FIRQuerySnapshot method. Because 1. Stacktrace happen on line 199

Lyokone · 2023-04-06T06:14:52Z

I've added a fix for line 199 as well. If it solves crashes on your end, I'll merge that. I suspect some kind of race conditions between 2 serialisations. I wasn't able to reproduce it with concurrent reads.

markglibres · 2023-04-06T12:14:40Z

We encountered the same problem with intermittent users in production and noticed a similar log file. The issue is happening on a specific screen that was running a query snapshot.

mazoRen · 2023-04-12T01:58:59Z

@Lyokone， I encountered the same problem and modified it according to @Lyokone's method, but it still crashed.

The Log：

Crashed: dev.flutter.firebase.firestore
0 libobjc.A.dylib 0x7974 lookUpImpOrForward + 92
1 libobjc.A.dylib 0x20c4 _objc_msgSend_uncached + 68
2 CoreFoundation 0x2104 -[__NSDictionaryM objectForKeyedSubscript:] + 164
3 Runner 0x48d344 -[FLTFirebaseFirestoreWriter FIRDocumentSnapshot:] + 160 (FLTFirebaseFirestoreWriter.m:160)
4 Runner 0x48cae4 -[FLTFirebaseFirestoreWriter writeValue:] + 43 (FLTFirebaseFirestoreWriter.m:43)
5 Flutter 0x591bd0 (Missing UUID 4c4c44e655553144a165b4c4c8109699)
6 Flutter 0x58e48c (Missing UUID 4c4c44e655553144a165b4c4c8109699)
7 Runner 0x48a094 __63-[FLTFirebaseFirestorePlugin documentGet:withMethodCallResult:]_block_invoke + 483 (FLTFirebaseFirestorePlugin.m:483)
8 Runner 0x21b73c -[FIRDocumentReference wrapDocumentSnapshotBlock:]::Converter::OnEvent(firebase::firestore::util::StatusOrfirebase::firestore::api::DocumentSnapshot) + 440324
9 Runner 0x206c74 firebase::firestore::api::DocumentReference::GetDocument(firebase::firestore::api::Source, std::__1::unique_ptr<firebase::firestore::core::EventListenerfirebase::firestore::api::DocumentSnapshot, std::__1::default_delete<firebase::firestore::core::EventListenerfirebase::firestore::api::DocumentSnapshot > >&&)::ListenOnce::OnEvent(firebase::firestore::util::StatusOrfirebase::firestore::api::DocumentSnapshot) + 355644
10 Runner 0x2078a0 firebase::firestore::api::DocumentReference::AddSnapshotListener(firebase::firestore::core::ListenOptions, std::__1::unique_ptr<firebase::firestore::core::EventListenerfirebase::firestore::api::DocumentSnapshot, std::__1::default_delete<firebase::firestore::core::EventListenerfirebase::firestore::api::DocumentSnapshot > >&&)::Converter::OnEvent(firebase::firestore::util::StatusOrfirebase::firestore::core::ViewSnapshot) + 358760
11 Runner 0x2504e8 void std::__1::__invoke_void_return_wrapper<void, true>::__call<firebase::firestore::core::AsyncEventListenerfirebase::firestore::core::ViewSnapshot::OnEvent(firebase::firestore::util::StatusOrfirebase::firestore::core::ViewSnapshot)::'lambda'()&>(firebase::firestore::core::AsyncEventListenerfirebase::firestore::core::ViewSnapshot::OnEvent(firebase::firestore::util::StatusOrfirebase::firestore::core::ViewSnapshot)::'lambda'()&) + 656816
12 Runner 0x310cd8 firebase::firestore::util::Task::ExecuteAndRelease() + 1445280
13 libdispatch.dylib 0x3fdc _dispatch_client_callout + 20
14 libdispatch.dylib 0xb694 _dispatch_lane_serial_drain + 672
15 libdispatch.dylib 0xc1e0 _dispatch_lane_invoke + 384
16 libdispatch.dylib 0x16e10 _dispatch_workloop_worker_thread + 652
17 libsystem_pthread.dylib 0xdf8 _pthread_wqthread + 288
18 libsystem_pthread.dylib 0xb98 start_wqthread + 8

markglibres · 2023-04-18T15:04:38Z

@Lyokone, after doing some trial-and-error, we were able to stop the crashes by setting firestore persistent storage = false and set snapshot includeMetadataChanges = false....

hope this gets fixed so we can take advantage of reading offline data

Lyokone · 2023-04-19T07:13:52Z

Thanks for your research, it will help to find the origin of the issue.

markglibres · 2023-04-28T02:07:37Z

hey @Lyokone ... any luck on finding the bug? looking forward to use offline mode again.. :)

markglibres · 2023-05-02T05:35:37Z

@Lyokone .. FYI: we upgrade cloud_firestore to version 4.5.3 then enabled the persistence with "includeMetadata" on snapshot and no more crashes. I think this ticket #10579) has fixed the issue

Lyokone · 2023-05-02T09:47:04Z

Good to know, thanks for letting me know; it was in my backlog for this week.

It's probably why it was hard to reproduce, I'll see to add some tests around this 👍

markglibres · 2023-05-24T12:56:25Z

hey @Lyokone, with the latest version of Flutter and Firestore, we have this error but for now can only replicate the error on iPhone 12 with offline mode enabled.. iPhone 14 and others were ok.

hukusuke1007 · 2023-07-19T14:29:24Z

Same issue??

This is log.

#2	0x00000001011f2590 in -[FLTFirebaseFirestoreWriter FIRDocumentSnapshot:] at /Users/shohei/.pub-cache/hosted/pub.dev/cloud_firestore-4.8.2/ios/Classes/FLTFirebaseFirestoreWriter.m:159
#3	0x00000001011f1d74 in -[FLTFirebaseFirestoreWriter writeValue:] at /Users/shohei/.pub-cache/hosted/pub.dev/cloud_firestore-4.8.2/ios/Classes/FLTFirebaseFirestoreWriter.m:43

Package version.

firebase_core: ^2.14.0
cloud_firestore: ^4.8.2

Occurs in the production environment. But also occurs in simulator.

hukusuke1007 · 2023-08-10T14:44:40Z

When will this be fixed? I'm in a lot of trouble.

paulw11 · 2023-08-21T00:50:50Z

We have also seen crashes related to this. Looking at the code, the implementation of serverTimestampMap in FLTFirebaseFirestorePlugin.m is simply not thread-safe. Line 74 contains a previous attempt at a fix, which instantiates a new dictionary if the property is nil, but this has probably made things worse, if anything.

In Objective C you need to consider multi-threading. Neither properties of objects nor structures such as NSDictionary are inherently thread safe. As a static property, serverTimestampMap should probably be created via a dispatch_once block, rather than instantiated lazily by which ever threads encounter a nil.

Updates to the dictionary itself need to be protected via some form of exclusive access mechanism such as a semaphore.

use dispatch_once to instantiate `serverTimestampMap` Fix firebase#10715

Use thread-safe `NSCache` instead of `NSMutableDictionary` Do not bypass getter Consistently use static property USe `dispatch_once` to ensure single instantiation of cache.

paulw11 · 2023-09-18T00:31:10Z

From our internal monitoring this issue has been resolved by #11501

russellwheatley · 2023-09-21T11:44:20Z

Thanks for the update, @paulw11. Could you also confirm, @tecbot? Be sure to install the latest version of cloud_firestore which contains the fix 👍

google-oss-bot · 2023-10-02T01:04:59Z

Hey @tecbot. We need more information to resolve this issue but there hasn't been an update in 7 weekdays. I'm marking the issue as stale and if there are no new updates in the next 7 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

russellwheatley · 2023-10-02T10:41:10Z

Closing out as I believe this issue has been resolved with this fix #11501

tecbot added Needs Attention This issue needs maintainer attention. type: bug Something isn't working labels Apr 3, 2023

danagbemava-nc added the triage Issue is currently being triaged. label Apr 4, 2023

danagbemava-nc added blocked: customer-response Waiting for customer response, e.g. more information was requested. and removed Needs Attention This issue needs maintainer attention. labels Apr 4, 2023

google-oss-bot added Needs Attention This issue needs maintainer attention. and removed blocked: customer-response Waiting for customer response, e.g. more information was requested. labels Apr 4, 2023

danagbemava-nc added plugin: cloud_firestore platform: ios Issues / PRs which are specifically for iOS. type: crash A compile error or crash and removed Needs Attention This issue needs maintainer attention. triage Issue is currently being triaged. labels Apr 5, 2023

Lyokone mentioned this issue Apr 5, 2023

fix(firestore, ios): tentative fix for a crash that could occur during Snapshot serialization #10728

Merged

10 tasks

Lyokone self-assigned this Apr 5, 2023

paulw11 added a commit to paulw11/flutterfire that referenced this issue Aug 21, 2023

Replace NSDictionary with NSCache and

7a1c6c6

use dispatch_once to instantiate `serverTimestampMap` Fix firebase#10715

paulw11 added a commit to paulw11/flutterfire that referenced this issue Aug 21, 2023

Fix firebase#10715

3ebd1b2

Use thread-safe `NSCache` instead of `NSMutableDictionary` Do not bypass getter Consistently use static property USe `dispatch_once` to ensure single instantiation of cache.

paulw11 mentioned this issue Aug 21, 2023

fix(cloud_firestore): Fix crashes on iOS/macOS #11501

Merged

10 tasks

harry-dickson mentioned this issue Aug 30, 2023

🐛 [cloud_firestore] Crash Subscribing to Data at Startup #11528

Closed

russellwheatley added the blocked: customer-response Waiting for customer response, e.g. more information was requested. label Sep 21, 2023

google-oss-bot added the Stale Issue with no recent activity label Oct 2, 2023

russellwheatley closed this as completed Oct 2, 2023

danagbemava-nc added the resolution: fixed A fix has been merged or is pending merge from a PR. label Oct 3, 2023

firebase locked and limited conversation to collaborators Nov 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 [cloud_firestore] Still ServerTimestampBehavior crashes #10715

🐛 [cloud_firestore] Still ServerTimestampBehavior crashes #10715

tecbot commented Apr 3, 2023 •

edited

danagbemava-nc commented Apr 4, 2023

tecbot commented Apr 4, 2023

danagbemava-nc commented Apr 5, 2023

Lyokone commented Apr 5, 2023 •

edited

tecbot commented Apr 5, 2023

Lyokone commented Apr 6, 2023 •

edited

markglibres commented Apr 6, 2023

mazoRen commented Apr 12, 2023 •

edited

markglibres commented Apr 18, 2023 •

edited

Lyokone commented Apr 19, 2023

markglibres commented Apr 28, 2023

markglibres commented May 2, 2023

Lyokone commented May 2, 2023

markglibres commented May 24, 2023 •

edited

hukusuke1007 commented Jul 19, 2023 •

edited

hukusuke1007 commented Aug 10, 2023

paulw11 commented Aug 21, 2023

paulw11 commented Sep 18, 2023

russellwheatley commented Sep 21, 2023

google-oss-bot commented Oct 2, 2023

russellwheatley commented Oct 2, 2023

🐛 [cloud_firestore] Still ServerTimestampBehavior crashes #10715

🐛 [cloud_firestore] Still ServerTimestampBehavior crashes #10715

Comments

tecbot commented Apr 3, 2023 • edited

Bug report

danagbemava-nc commented Apr 4, 2023

tecbot commented Apr 4, 2023

danagbemava-nc commented Apr 5, 2023

Lyokone commented Apr 5, 2023 • edited

tecbot commented Apr 5, 2023

Lyokone commented Apr 6, 2023 • edited

markglibres commented Apr 6, 2023

mazoRen commented Apr 12, 2023 • edited

markglibres commented Apr 18, 2023 • edited

Lyokone commented Apr 19, 2023

markglibres commented Apr 28, 2023

markglibres commented May 2, 2023

Lyokone commented May 2, 2023

markglibres commented May 24, 2023 • edited

hukusuke1007 commented Jul 19, 2023 • edited

hukusuke1007 commented Aug 10, 2023

paulw11 commented Aug 21, 2023

paulw11 commented Sep 18, 2023

russellwheatley commented Sep 21, 2023

google-oss-bot commented Oct 2, 2023

russellwheatley commented Oct 2, 2023

tecbot commented Apr 3, 2023 •

edited

Lyokone commented Apr 5, 2023 •

edited

Lyokone commented Apr 6, 2023 •

edited

mazoRen commented Apr 12, 2023 •

edited

markglibres commented Apr 18, 2023 •

edited

markglibres commented May 24, 2023 •

edited

hukusuke1007 commented Jul 19, 2023 •

edited