A Developer’s Adventures in Rooting

Update: November 2011

As a comparison study, I spent some time going through the process of converting a "commercial" Nexus S 4G device (originally released via Sprint) to a userdebug build from the AOSP code. Comparatively, the process was about the same amount of effort, just spread out over a different set of tasks. The primary advantage here is that I was walking through a fairly well documented set of steps instead of researching each step one after the other...but the process still took a full day’s work to get initialized.

The primary piece of heavy lifting required to go through this process was to set up the build environment for the AOSP. If I can provide any advice here it is to build your environment inside of Ubuntu 10.04...​and nothing else. I started initializing the environment inside of an Ubuntu 8.04 instance, and ran into multiple issues in setting up the build chain that ultimately pointed to required packages I could not get without an upgrade. I imagine trying to set up the environment in something even more different would result in even more headaches...​hardly worth it if your doing this for a simplistic purpose such as this.

Once the environment is initialized, the non-AOSP proprietary drivers and binaries that each device requires must be unpacked and loaded into the code base manually before the image is built. Again, this process is well documented and there are scripts to extract each driver into the proper location…...but it still must be done individually for each device you plan to build for. Beyond that, the process is simple and requires little interaction, even though there is a lot of waiting (syncing the source and building the target code). With about 6 CLI commands you have built and flashed the image onto the device.

If I had to recommend a method, I would choose this avenue just because it caused less frustration. However, you will be limited to the 3-4 devices at any one time that has current support in a branch of the AOSP source tree.

When I started developing applications for Android in early 2009, I never expected that I would ever own a rooted device. I felt that, as a developer, I needed to be testing my applications in an environment that best emulated my users, and rooting would compromise that environment. However, recently I have been forced to take a second look at that opinion in light of some unfortunate barriers Android developers face today. Let me also go on record to say that these are made unfortunate because I believe that they should not exist at all, and would not exist with Google and device OEM support. This is the story of my journey to solve a major problem, and how rooting was the only viable solution.

The Problem

When I set out on this journey, I did so because there were a number of vital debugging tasks that I simply could not do on any physical device; and I was tired of resorting to the emulator just to perform them. As a goal, the primary tasks I wanted on my devices were:

• Using DDMS File Explorer to view my application’s internal storage (the data directory where files and databases created by an application are stored)
• Using the HierarchyViewer tool to debug issues with layouts
• Running ADB in TCP/IP mode (I will explain the importance of this one more later)

I didn’t feel like these demands were unreasonable, they are all a necessary part of developing applications. When I look back, it’s a bit of a shock to me that this is even a problem. My first test phone was the ADP1; the first developer phone that could be purchased directly from Google by registered Android Market developers. This phone set a great precedent because the device was completely unlocked and was running a userdebug build of Android; meaning that all the things I listed above were possible on the device without even a second thought.

Nowadays, the "developer phone" is a thing of the past, and you can no longer select and purchase phones already set up for userdebug in the Android Market portal. Today, if you want to develop applications on a device, you must purchase a commercial device from a retail location. This means that you can not perform any of these useful debugging functions directly on a device...​where they really count.

Google instead now provides support via instructions for building userdebug images from source for their latest devices (the Nexus One and Nexus S at the time of this writing). This seems helpful on the surface, except most application developers do not have the full AOSP build environment set up on their systems. In the days of the ADP1, these images were at least available on the HTC dev site built and ready to be flashed! But I digress...

The Journey Begins...

Just to establish context, I currently have the following devices that I use for developing applications:

• HTC ADP1
• Original Motorola Droid
• HTC EVO 4G
• Samsung Nexus S 4G
• Motorola Droid 2

None of these devices, except the ADP1, are set up to allow proper access to use the debugging tools I wanted. I decided that, since these issues were primarily due to lack of permissions on the device (like the inability for ADB to run in root mode), perhaps these problems could be solved simply by rooting one of my test devices.

Step 1: Root

I chose my OG Droid as the device to use. It had been around the longest, so there should be plenty of information out in the community about rooting it. Plus, if something happened to it in the process I wouldn’t be super sad. I used a program for the PC named SuperOneClick to root the device and install SuperUser, an Android application that helps manage root access to installed applications. I didn’t realize it at the time, but SuperOneClick also installed BusyBox on the system. I’ll explain a bit later why this was actually a bad thing.

With my device rooted, I went to test the success of my efforts by attempting to run ADB over WiFi on my device…​success! I could now run the adb tcpip <portnum> command and subsequently adb connect <ipaddress:portnum> from my development machine to debug applications without being tethered to the actual device. In fact, with root access I could now even install one of the many applications available on Android Market that would enable/disable this service for me so I didn’t need to drop into the shell every time.

I was so excited I couldn’t wait to try out HierarchyViewer and File Explorer...​but for some reason, these tools still did not work on the device. I was confused as to what other piece of the puzzle could be required to get things up and running.

ADB WiFi

Besides the simple convenience of the matter, the primary reason that running ADB over WiFi was so important to me is because oftentimes I’m developing with Android in a context where the USB connector is not accessible. The two main examples of that are when I’m developing with an accessory device connected (like the ADK) or when the Android device is encased or embedded in some larger system where the mechanical enclosure obstructs the USB connector. In these cases, the prospect of debugging wirelessly makes the development process infinitely easier.

Step 2: Userdebug

It didn't take much research to find out that the reason these tools no longer work is because the necessary features required to run them are explicitly disabled in production—or user—builds of the Android OS (this statement is made directly by Googlers from the Android Team all over the web). Instead, an engineering (eng) or userdebug build is required to be installed on the device. One of the primary functions needed was the ability to run the ADB daemon as root (try typing adb root with your device plugged in and you'll see what I mean).

This meant I needed to install a custom system ROM image. Luckily, my device had already been rooted, which is the first step needed to install said custom image. However, finding the image to load would prove a bit more difficult. For most production phones, a stock userdebug image doesn't seem to exist, although if you own one of the Nexus phones it seems you can undertake the task of building one yourself from source (see the link at the beginning of this post). I found the answer in the modding community via Cyanogenmod. While I don't believe most of the popular custom ROMs out there (Cyanogenmod, MIUI, etc.) are technically built as userdebug (the ro.build.type property is still set to "user" on devices I've seen), they are built to allow the processes like ADB the full system access I needed it to have. I used the very helpful ROM Manager application (available in Android Market) to handle the necessary steps of flashing a proper recovery image and installing Cyanogenmod.

The initial attempt to load the Cyanogenmod binary resulted in a phone that would not boot past the initial animation, but a second attempt (manually going into the recovery menu and applying the update again) yielded successful results. So how was I faring with my laundry list now? With Cyanogenmod installed, I was finally successful in attaching to HierarchyViewer on the device!  However, I was still unable to read anything other than the SD Card via File Explorer.

I was 2 for 3, but completely puzzled about what else I could even customize on the device to allow me proper access inside of DDMS. I could drop into the shell on the command line and view all the files in all the directories on the device... so why couldn't DDMS?

ViewServer

I wanted to make a quick note about the solution that came out of the Android team recently with regards to using HierarchyViewer on devices. The ViewServer project, hosted by Romain Guy, is a version of the internal Android ViewServer that can be dropped into individual applications in order to enable the use of HierarchyViewer on devices. I greatly appreciate the effort put forth to provide this to the community, but I still believe it is the wrong solution. Developers should be able to configure their devices to freely use these tools globally throughout the system.

Step 3: Replacing System Binaries

After doing even more research, I came across an obscure set of posts on a few forums indicating that the system Toolbox was the culprit. I mentioned earlier that SuperOneClick installed BusyBox on the device, which replaced the default system Toolbox for all primary commands. Cyanogenmod also relies on BusyBox, so even if the rooting hadn’t done it, I have a feeling I would have ended up with BusyBox by this point anyway.

What Are All These Boxes?

If you’re unfamiliar with what all these “boxes” are, let me explain. Linux systems typically have a collection of individual utility programs in the system directories that are invoked from the command line. Commands like "ls", "cat", "ssh", etc. are actually binaries that get executed to perform those specific tasks. When Linux found its way into mobile and embedded systems, this collection of tools took up far too much space.

Enter BusyBox and Toolbox. BusyBox and Toolbox are examples of single binary applications that implement a host of these common commands in a very tiny package, making it suitable for embedded systems. In short, one of these two binaries is where you get your command line capability from in the Android shell.

The hidden problem that this created is that the BusyBox version of "ls" (which is what DDMS File Explorer uses) does not support all the functions DDMS expects to find (I believe the primary missing feature is file sorting) in the Toolbox version of "ls". So in order to solve this problem, I had to manually copy a good version of Android’s Toolbox back onto the device and re-link "ls" to point to it instead of BusyBox. I obtained the correct copy of Toolbox from an emulator instance running the same base OS version and transferred the file to the system/bin directory of the device. I then had to enter the shell on the device in order to remap the symbolic link for "ls" in the system/xbin directory to point to system/bin/toolbox.

Finally, I now could browse the device’s file system properly from within the DDMS File Explorer! I now had a device worthy of being used for debugging!

The Plea

Needless to say, this process was an absolute pain; especially considering my position that the whole thing should not have been necessary. Android developers should not have to be linux gurus or developing at the firmware level of the AOSP in order to have access to the full set of SDK debugging tools on their devices. The key missing element is that developers need easy access to flash-ready system images that enable the userdebug features of the OS and devices that easily allow these images to be flashed. Devices like the Nexus One, Nexus S, and some of the newer HTC devices that allow easy unlocking of the bootloader is a step in the right direction, but we need Google and the OEMs to step up and help provide the tools to finish the job.

Your community is counting on you!