Core Dumping for fun and profit

Intro

The modern iOS developer has access to an array of tools. Starting with Apple's Xcode and Instruments, extending to various third-party tools like Crashlytics or Sentry. Some of these tools serve unique purposes, while others, like LLDB, are everyday utilities used to investigate nearly any issue in a running app. New utilities and services emerge daily, many proving to be exceptionally useful. However, there are instances when we encounter a problem for which we don't have a tool, or at least, we aren't aware of one. In such situations, it's beneficial not to limit our search to modern, sophisticated utilities and services but also to revisit some of the classic tools. Today, we will dust off one such tool from our arsenal.

Use case

To illustrate a use case for the tool we plan to explore, consider the following scenario. You're working on a feature within a large, well-established app with an extensive codebase. As part of your development process, you repeatedly run the app, following a specific path to test the new functionality you've implemented. This involves navigating through various views and adding data to finally reach the feature you're working on. Suddenly, the app halts at an assert statement. Notably, you haven't even reached your code segment yet. This interruption is caused by something else, something that previously functioned flawlessly but now has inexplicably failed. While examining the debugger paused at the assert, nothing particularly interesting or unusual stands out. The issue seems to stem from a function argument being 'nil', contradicting the assert's stipulation that it should not be. This leaves you puzzled, wondering why it's 'nil' now when it wasn't in the countless previous instances. A quick 'git blame' reveals the colleague responsible for this assert, who could potentially shed light on the issue. As you prepare to contact this colleague via Slack, you ponder the best way to convey the problem. Would a screenshot of the assert suffice? Or perhaps printing all the registers at that point? You then realize that the root of the issue might lie deep within the app's memory at that moment. Ideally, you'd want to hand over your laptop as it is, or at least transfer the app in its current loaded state with the debugger frozen at the assert, allowing your colleague to conduct a thorough examination independently.

Core dump: history lesson

You're not alone in your frustration with the assert. The mechanism to 'snapshot' a program at the point where something goes awry was conceived long ago, during the era of UNIX running on large machines in the early 1970s. Back then, PDP computers running UNIX utilized 'magnetic core memory' - a type of RAM unit named for its composition of small ferromagnetic rings called 'cores'.

This is the origin of the term 'core dump'. It refers to the process of dumping a process's address space and registers onto the disk. The files generated by this dump are known as core dumps or core files. Such files were created in exceptional situations, like dereferencing an illegal memory address, and also when a user sent a 'quit' signal to the process. Today, on the modern operating systems used in our phones and laptops, core dumps are fundamentally the same as they were around 50 years ago. Naturally, they are now dramatically larger, and the tools available to work with them have become much more convenient and user-friendly.

Case study

Following our brief historical introduction, let's now dive into the practical side of core dumps. For this, I will utilize a simple sample project, essentially an Xcode template for an iOS application. The test app features a function in the AppDelegate designed to access an array element beyond its bounds. Let's name this function 'fail()' to simplify locating it later. This function will be invoked from the 'application:didFinishLaunchingWithOptions:' method, ensuring the app crashes immediately upon startup. To generate a core dump file at the point of this crash, we must tell LLDB, the integrated debugger in Xcode, we want it:

(lldb) process save-core /tmp/core.dump

After running this command, pay close attention to the message displayed by LLDB:

Saving 16384 bytes of data for memory region at 0x1003a8000Modified-memory or stack-memory only corefile created. This corefile may not show library/framework/app binaries on a different system, or when those binaries have been updated/modified. Copies are not includedin this corefile. Use --style full to include all process memory.

The message you see from LLDB after executing the core dump command relates to a feature designed to reduce the size of core dump files significantly. Without delving into the complexities of memory layout in this context, it's important to understand that the default action of the 'process save-core' command in LLDB creates what is known as a 'skinny core file'. This type of core file contains only a portion of your process's memory, leading to a smaller file size. For comparison, let's proceed to generate a full core dump. This will involve additional steps or commands in LLDB to ensure that the entire memory space of the process is captured in the core

dump(lldb) process save-core /tmp/full.core.dump --style full

To understand the difference between the 'skinny' and full core dumps, let's compare their sizes. This comparison will illustrate the impact of including the entire memory space in the core dump versus only a partial memory capture:

/tmp $ ls -lh ./*dump-rw------- 1 user wheel 184K Dec 24 20:01 
./core.dump-rw------- 1 user wheel 3.2G Dec 24 20:04 ./full.core.dump

The size comparison between the 'skinny' and full core dumps reveals a striking difference: the 'skinny' core dump is almost 20 times smaller. However, this optimization comes with a limitation: a 'skinny' core dump can only be debugged on the same system on which it was created. This is due to the partial memory capture, which may rely on specific system states or configurations.Now, to proceed with debugging using our core file, we need to instruct LLDB appropriately:

lldb -c /tmp.core.dump

When you execute the command to attach LLDB to the core file, you'll observe the debugger loading the core file into memory. After a short while, an empty prompt will appear. This indicates that LLDB is ready for your input, and you can now use it in the same way you typically do from Xcode. This includes examining thread call stacks, registers, and other aspects of the application state at the time of the crash.However, there's an intriguing point to note here. If you attempt to retrieve the current thread's call stack, you might encounter the following:

(lldb) bt* thread #1, stop reason = ESR_EC_BRK_AARCH64 (fault address: 0x1001a484c) * frame #0: 0x0000000192c959f8 
libswiftCore.dylib`_swift_runtime_on_report frame #1: 0x0000000192d369c8 
libswiftCore.dylib`_swift_stdlib_reportFatalErrorInFile + 204

The observation of only two frames in the call stack might be surprising. You might wonder, "Where is the rest of the stack?" This outcome ties back to our earlier discussion about 'skinny' versus full core dumps. In the 'skinny' mode, to minimize file size, a significant amount of information, including parts of the call stack, is stripped from the core file. This reduction can limit the depth of debugging information available from the core dump.Now, for a more comprehensive view, let's try attaching LLDB to a full core file. This will allow us to compare and understand the difference in the amount of information available between the two types of dumps:

/tmp $ lldb -c /tmp/full.core.dump(lldb) target create --core "/tmp/full.core.dump"Core file '/tmp/full.core.dump' (arm64) was loaded.(lldb) btwarning: could not execute support code to read Objective-C class data in the process. This may reduce the quality of type information available.
* thread #1, stop reason = ESR_EC_BRK_AARCH64 (fault address: 
0x1001a484c) * frame #0: 0x0000000192c959f8 
libswiftCore.dylib`_swift_runtime_on_report frame #1: 
0x0000000192d369c8 
libswiftCore.dylib`_swift_stdlib_reportFatalErrorInFile + 204 frame 
#2: 0x00000001929f43a0 libswiftCore.dylib`closure #1 
(Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in closure #1 
(Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in closure #1 
(Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in 
Swift._assertionFailure(_: Swift.StaticString, _: 
Swift.StaticString, file: Swift.StaticString, line: Swift.UInt, 
flags: Swift.UInt32) -> Swift.Never + 224 frame #3: 
0x00000001929f427c libswiftCore.dylib`closure #1 
(Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in closure #1 
(Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in 
Swift._assertionFailure(_: Swift.StaticString, _: Swift.StaticString, file: Swift.StaticString, line: Swift.UInt, flags: Swift.UInt32) -> Swift.Never + 316 frame #4: 0x00000001929f3ce8 libswiftCore.dylib`Swift._assertionFailure(_: Swift.StaticString, _: Swift.StaticString, file: Swift.StaticString, line: Swift.UInt, flags: Swift.UInt32) -> Swift.Never + 168 frame #5: 0x00000001929d990c libswiftCore.dylib`Swift.Array._checkSubscript(_: Swift.Int, wasNativeTypeChecked: Swift.Bool) -> Swift._DependenceToken + 152 frame #6: 0x00000001929da068 libswiftCore.dylib`Swift.Array.subscript.getter : (Swift.Int) -> τ_0_0 + 84 frame #7: 0x00000001001a463c CoreDumper`AppDelegate.fail(self=0x0000600000227aa0) at AppDelegate.swift:39:18 frame #8: 0x00000001001a4230 CoreDumper`AppDelegate.application(application=0x0000000101b06450, launchOptions=nil, self=0x0000600000227aa0) at AppDelegate.swift:17:9 frame #9: 0x00000001001a4308 CoreDumper`@objc AppDelegate.application(_:didFinishLaunchingWithOptions:) at <compiler-generated>:0 frame #10: 0x00000001852eccfc UIKitCore`-[UIApplication _handleDelegateCallbacksWithOptions:isSuspended:restoreState:] + 312 frame #11: 0x00000001852ee15c UIKitCore`-[UIApplication _callInitializationDelegatesWithActions:forCanvas:payload:fromOriginatingProcess:] + 2788 frame #12: 0x00000001852f2c7c UIKitCore`-[UIApplication _runWithMainScene:transitionContext:completion:] + 856 frame #13: 0x00000001849a0820 UIKitCore`-[_UISceneLifecycleMultiplexer completeApplicationLaunchWithFBSScene:transitionContext:] + 148 frame #14: 0x00000001852eff70 UIKitCore`-[UIApplication _compellApplicationLaunchToCompleteUnconditionally] + 44 frame #15: 0x00000001852f02c0 UIKitCore`-[UIApplication _run] + 832 frame #16: 0x00000001852f3f5c UIKitCore`UIApplicationMain + 124 frame #17: 0x00000001847a33a8 UIKitCore`UIKit.UIApplicationMain(Swift.Int32, Swift.Optional<Swift.UnsafeMutablePointer<Swift.UnsafeMutablePointer<Swift.Int8>>>, Swift.Optional<Swift.String>, Swift.Optional<Swift.String>) -> Swift.Int32 + 100 frame #18: 0x00000001001a47ec CoreDumper`static UIApplicationDelegate.main() at <compiler-generated>:0 frame #19: 0x00000001001a4764 CoreDumper`static AppDelegate.$main(self=@thick CoreDumper.AppDelegate.Type) at <compiler-generated>:0 frame #20: 0x00000001001a4868 CoreDumper`main at AppDelegate.swift:11:7 frame #21: 0x0000000100319544 dyld_sim`start_sim + 20 frame #22: 0x00000001003b2058 dyld`start + 2224

The experience with a full core file is significantly improved. While you can't step through to the next instruction (as you're working with a core file, not a running process), this approach is far more informative than simply sending a screenshot. With a full core file, you have access to a wealth of information. You can examine registers and thread stacks, print variables, and perform most of the tasks you're accustomed to in Xcode.An interesting and often overlooked feature in this context is the 'gui' command in LLDB. This command launches a ncurses-based interface that displays the source code, a list of threads, and other useful information.

However, it's worth noting that the 'gui' feature in LLDB doesn't have extensive documentation and lacks some functionalities compared to the LLDB console itself. It was developed by Greg Clayton, one of the LLDB developers, over a few weeks and seems to have been left largely untouched since then. Despite its limitations, the ncurses interface can be more user-friendly than the command line for many debugging tasks. Plus, using this terminal-based interface can certainly make you look like a pro!

Conclusion

In an era of advanced and fancy tools, it's crucial to acknowledge the enduring value of traditional methodologies like core dump analysis in software development. Core dumps, honed over decades, remains a testament to the ingenuity of earlier software engineers. As we embrace new technologies, integrating core dump analysis into our toolkit not only enriches our problem-solving capabilities but also pays homage to the foundational aspects of software engineering. Remembering and utilizing these classic methods ensures a well-rounded approach to development, blending the best of both worlds.

Resources

For the most curios lets name a few resources to dive even deeper:

Apple tech note on core dumps. Briefly introduces using core dumps on macOS: https://developer.apple.com/library/archive/technotes/tn2124/_index.html#//apple_ref/doc/uid/DTS10003391-CH1-SECCOREDUMPS

Excellent training transcript on working with core dumps on macOS: https://www.amazon.com/Accelerated-Core-Dump-Analysis-Second/dp/1908043717

LLDB mailing list thread about 'gui' feature: https://lists.llvm.org/pipermail/lldb-dev/2019-August/015421.html

LLDB docs is a good starting point if you don't feel comfortable with debugger: https://lldb.llvm.org/use/tutorial.html

Awesome book from Big Nerd Ranch on macOS programming. Core dumps are covered at the end of chapter 8: https://www.amazon.com/Advanced-Mac-OS-Programming-Guides-ebook/dp/B005GWG0L0