RFC: Web Observability with Crashlytics

[abdallahshaban557]

You may have seen some telemetry PRs floating around the firebase repos. The rumors are true, we’re working on web observability in firebase!

The goal of this RFC is to get your feedback on what a great V1 for web error reporting should look like, what metrics matter most, and how you'd expect to see them.

What features are most important to you?

1. If you could only pick THREE features from the list below, which would you choose?
   1. Automatic capture of logs and errors.
   2. A manual API to log non-fatal errors (e.g., Crashlytics.logError(e))
   3. An API to set a User ID (e.g., Crashlytics.setUserId("123"))
   4. An API to set custom keys/tags (e.g., Crashlytics.setCustomKey(...))
   5. Alerting
   6. The ability to mute errors
   7. Breadcrumbs (a log of user events & console messages leading to an error)
2. What is the number 1 feature from that list that, if missing, would make a V1 "unusable" for you?
3. Which frameworks or libraries are you using (Ex: React, NextJS, Vue, Nuxt, etc).
   1. Would you prefer framework-specific error handling methods or low-level features that you can integrate into your framework your own way?
4. Are there any other features we are missing? Please let us know!

What do you want to see in the Firebase Dashboard?

We don't just want to collect errors; we want to present them in a way that is actionable for you.

1. When you open your error-reporting dashboard, what is most critical for you to see first? Total error count over time? Trending issues?
2. How do you define the health of your web app? What are some key metrics you track? The following are some examples we have in mind:
   1. Errors per Session
   2. Core Web Vitals
   3. Custom errors
   4. Something else? (Please tell us!)
3. How would you want to group or filter your web errors to make them triagable? (e.g., By Browser, By OS, By URL/Page, By App Version/Build ID, By Custom Tag, By Client- or Server-side error).

Anything else you want us to know? Drop a comment below. Thanks in advance for sharing your feedback!

[sperochon]

Hello Abdallah, I'm a Flutter dev. My wish is to use the same Flutter code I already use for mobile.

What features are most important to you?

1. A manual API to log fatal errors (not non-fatal) (e.g., FirebaseCrashlytics.instance.recordFlutterFatalError(...))
2. An API to set custom keys/tags (e.g., FirebaseCrashlytics.instance.setCustomKey(...))
3. An API to set a User ID (e.g., FirebaseCrashlytics.instance.setUserIdentifier("123"))

What is the number 1 feature from that list that, if missing, would make a V1 "unusable" for you?

1. A manual API to log fatal errors (not non-fatal) (e.g., FirebaseCrashlytics.instance.recordFlutterFatalError(...))

Which frameworks or libraries are you using (Ex: React, NextJS, Vue, Nuxt, etc).
Flutter

Would you prefer framework-specific error handling methods or low-level features that you can integrate into your framework your own way?
framework-specific error handling methods (= through official firebase_crashlytics package managed by firebase team)

Are there any other features we are missing? Please let us know!

1. A manual API to log fatal errors (not non-fatal) (e.g., FirebaseCrashlytics.instance.recordFlutterFatalError(...))
2. Managing the deobfuscation of the stack trace (symbols) thing (https://firebase.google.com/docs/crashlytics/get-deobfuscated-reports?platform=flutter)

What do you want to see in the Firebase Dashboard?
No more than the same as android and iOS Firebase Dashboards (it can be just a few things to start). Priority is to be able to report fatals errors and see deobfuscated stack traces, to fix them asap.

When you open your error-reporting dashboard, what is most critical for you to see first?
The number of errors of each day of the selected period (= the Trending issues)
Then the number of events and the number of concerned users per issue (= the issues table)

How do you define the health of your web app? What are some key metrics you track?
Not having an issue that concerns a lot of users, or that happens often (= number of events).

How would you want to group or filter your web errors to make them triagable?
Filter By App Version/Build ID, search by userId

Thanks!

[abdallahshaban557]

Hi @sperochon - thank you for this feedback! Do you often run into native crashes when using Crashlytics with Flutter? Or are they mostly Dart related exceptions that you mostly run into?

[sperochon]

Well, I don't know yet on web ;-) but on mobile, they are mostly (only?) Dart related exceptions.

[abdallahshaban557]

Ok great! I should have clarified I meant on mobile, but seems like you got what I was trying to ask. Thank you so much!

[fredrikj31]

Hey Abdallah. Thanks for looking into this missing feature. I have started to use Firebase in many of my web apps, and I'm loving the dev experience a lot. Would be great to get crashlytics for the web.

What features are most important to you?

If you could only pick THREE features from the list below, which would you choose?

1. A manual API to log non-fatal errors (e.g., Crashlytics.logError(e))
2. An API to set custom keys/tags (e.g., Crashlytics.setCustomKey(...))
3. Breadcrumbs (a log of user events & console messages leading to an error)

What is the number 1 feature from that list that, if missing, would make a V1 "unusable" for you?

• Alerting

Which frameworks or libraries are you using (Ex: React, NextJS, Vue, Nuxt, etc).

• React.js

What do you want to see in the Firebase Dashboard?

When you open your error-reporting dashboard, what is most critical for you to see first?

I think you already have some good metrics you can present in the dashboard. I think the most important ones are:

• Total error count over time
• Trending issues
• Errors per session
• Which pages does the errors happen on

How do you define the health of your web app? What are some key metrics you track?

• Core web vitals
• Errors per session

How would you want to group or filter your web errors to make them triagable?

I think by grouping the errors by the most common attributes from the browser could be a standard like these:

• User Agent
• Browser
• Country
• URL

But then it could also be nice to be able to search for a specific key/tag, and group by them.

[abdallahshaban557]

Hi @fredrikj31 - this is great - thank you for this feedback!

[jwolfson-goog]

Hey @fredrikj31 - awesome feedback, quick follow up: How do you use core web vitals?

Do you use them when deciding which error to investigate? To understand impact on SEO? Or another use case?

[fredrikj31]

Hi @jwolfson-goog

I normally use the Core Web Vitials to identify slow loading pages, or where some page can use some optimizations.

I normally run a lighthouse test on the specific test, and then investigating the numbers and bad metrics I get from that report. The reason why I'm using the Core Web Vitals is both to improve SEO, but also just in general the overall user experierence. We can see that our users the site much more when they have a nice experience exploring the pages.

[LanderBeeuwsaert]

Hi, we're a 10-year long user of firebase and because of the lack of observability until now, we have built our own solution and improved upon it over the years. So some feedback below on what we think would be useful, both as an ultimate solution for all users as well as for us:

Automatic capture of logs and errors:
less important for us because we would do manual logging. But yes certainly important for other users.

A manual API to log non-fatal errors (e.g., Crashlytics.logError(e)):
Priority 1 for us, in combination with the breadcrumbs / clarity

An API to set a User ID (e.g., Crashlytics.setUserId("123")):
Sure, but we would prefer a much more deep auditing/review functionality (see proposal of clarity)

An API to set custom keys/tags (e.g., Crashlytics.setCustomKey(...)):
Sure, but we would prefer a much more deep auditing/review functionality (see proposal of clarity)

Alerting:
less important for us because we already do our alerting, but yes would be relatively important for such a platform.
But would probably only be best on certain conditions. And the conditions can be complex.

The ability to mute errors:
Certainly, but would probably only be best on certain conditions. And the conditions can be complex.
(don't log this error if the user is on safari < version X)
(don't log this error if the stacktrace contains the word "googleMapsAPI")
(don't log this error if the user logged out in the last 3 seconds)

Breadcrumbs (a log of user events & console messages leading to an error):
PRIORITY 1 for us.
This, in combination with logging of errors.
However, the breadcrumbs should ideally actually be a detailed description of what the user has been doing.
Ideally it would be a sort of technology like: https://clarity.microsoft.com/projects
That would be the ultimate solution, that we can visually understand the last 10 seconds before the crash happened what the user was doing.

[amal]

A manual API to log non-fatal errors (e.g., Crashlytics.logError(e)) is the most important for us. Saving the last log entries with the error - secondary. Custom keys/tags/userId - tertiary.

[abdallahshaban557]

Thank you @LanderBeeuwsaert and @amal for the feedback - that makes sense to me!

@LanderBeeuwsaert - it sounds like you would like to get session recordings as well? or are heatmaps more important? I'd be grateful if you can walk me through an example of how you would use either of these features.

[LanderBeeuwsaert]

@abdallahshaban557 heatmaps wouldn't interest us honestly. That would be useful to understand how many times a certain feature is used. But for that we have other tools (google analytics, ...)

Session recordings: 1000%.
If an error happens, what we (but I think this counts universally) want to do is to understand what the user was doing before the crash.
What were the specific actions the user did, what was the data the user was viewing & interacting with, what browser is the user on, ...
So ideally you want something like clarity session recordings and/or something like what playwright offers where you can review a recording actually see the clicks happening and the dom & browser.
That would make debugging a SAAS super super super easy.
Does what I say make sense?

[abdallahshaban557]

Yes, totally makes sense! Appreciate you getting back to me so quickly @LanderBeeuwsaert!

[WarrenMoore]

Hey @abdallahshaban557 thanks for opening up this RFC! I'm an architect working mostly on HTML-based (or WebGL-based) streaming applications for Smart TVs, games consoles, and set-top boxes, so our needs/wishlist might be different from others!

In short what we would be interested in is something that can track crashes for us, and provide some context/stack trace around what happened (many of the devices we support don't even expose a console).

Existing solutions like Sentry.io are extremely bulky and from a performance perspective are just a non-starter, so we're really looking for a strong alternative in environments where performance is tight but reliability is critical.

What features are most important to you?

• Automatic capture of logs and errors.
• A manual API to log non-fatal errors (e.g., Crashlytics.logError(e))
• An API to set a User ID (e.g., Crashlytics.setUserId("123"))

What is the number 1 feature from that list that, if missing, would make a V1 "unusable" for you?
Nothing would make it unusable I think, I think as long as it didn't massively impact on the performance of the device we would definitely be able to derive some value from even just some of these features.

Which frameworks or libraries are you using (Ex: React, NextJS, Vue, Nuxt, etc).
We're using RDK's Lightning framework.

Would you prefer framework-specific error handling methods or low-level features that you can integrate into your framework your own way?
I wouldn't expect any framework-specific error handling for us, so low-level would be more than enough.

Are there any other features we are missing? Please let us know!
You mentioned "alerting", we don't necessarily need to have specific alerting functionality, but the ability to get data out that we can run our own analysis on would be a nice-to-have. 🙂

What do you want to see in the Firebase Dashboard?

• Grouping of errors so we can notice when errors are really indicative of an ongoing incident.
• High-level error trends over time (spikes, regressions).
• Breakdown by device type/browser/environment (especially important for low-powered environments like STBs and older TVs).
• Clear surfacing of newly introduced errors vs. known/recurrent ones.
• Integration with release versions or build identifiers so we can correlate crashes to deployments.

When you open your error-reporting dashboard, what is most critical for you to see first?
Probably what is impacting users right now. So things around frequency, or maybe number of affected users if we end up being able to set user IDs, that we can use to determine the severity of issues. Some sort of intelligent grouping/noise reduction here would be great.

How do you define the health of your web app? What are some key metrics you track?
I guess things like:

• Error-free session rate (% of sessions without a fatal or critical error).
• Frequency of recoverable vs. non-recoverable errors (e.g. video playback recoveries vs. fatal crashes).
• Impact on core user flows (launch, login, playback start).

How would you want to group or filter your web errors to make them triagable?

• By release/build version.
• By device class (desktop browser vs. STB vs. TV model).
• By geography/region (sometimes CDN or entitlement issues are region-specific).
• By severity (blocking crash vs. background recoverable error).

[abdallahshaban557]

Hi Warren,

Thank you for this detailed response! I have some questions for you:

1. Are you able to use Sentry with RDK today?
2. Can you elaborate on existing solutions being bulky. Do you mean that it results in a large bundle size - or is there another concern here perhaps?
3. How would you define fatal vs non-fatal errors?

cc @jwolfson-goog