Presenting CrashProbe

2014-05-23

There are many crash reporting services and all claim to provide the best crash reports. So we started wondering which service actually holds up to their claims?

CrashProbe is a crash reporting test suite presented by HockeyApp that benchmarks and compares popular iOS and Mac OS X crash reporting services. In our effort to ensure the best possible and most accurate crash reports, we created a set of test cases that we use to verify PLCrashReporter and improve HockeyApp.

The set contains 21 test cases that we ran on all architectures for iOS (armv7, armv7s, and arm64) and on an iMac (64-bit). If the test result is 100% accurate, it’s marked as green. If some information were missing or the stack trace showed wrong class names, methods, or line numbers, it’s marked as neutral. And, last but not least, if the crash report was not created at all or did not include the method that crashes, it’s marked as red. You can view the details for all test cases on CrashProbe.com.

Here are the results:

CrashProbe for iOS

CrashProbe for OS X

To understand the difference in the results, two components need to be considered:

  1. A crash reporting framework that can be embedded into apps to capture crashes and exceptions.

  2. A server-side process that symbolicates the crash reports to add details like class names, filenames, line numbers, and additional information such as the reason.

Both components in this process are important, so even if a service is using the same framework as another service, the crash report can still be quite different. The teams behind PLCrashReporter and HockeyApp have been working very hard to ensure the best results possible, so we’re proud to share these results with you.

CrashProbe is open source, so you can perform all tests yourself with any crash reporting framework. Compare the quality of the symbolicated crash reports yourself on iOS and OS X. The source code is available under the MIT license on GitHub.