Much has been said lately about Android's external storage in the tech press. I'll be the first to admit that what we have now seems like a bit of a mess at first. With all the interest focused on the subject, I felt it an opportune time to take a moment and explain in detail exactly how external storage permissions work, both in the past framework versions and today in KitKat.
TL;DR
Versions of Android prior to KitKat exposed a single volume of shared external
storage to applications. This volume may have been located on a removable SD card,
or simply in a location on the internal device flash. Full read/write access to
any location on this volume was protected by a single permission titled
WRITE_EXTERNAL_STORAGE. Read access did not require any special permissions.
There are two main changes here in KitKat:
- Read access now requires the
READ_EXTERNAL_STORAGEpermission (for apps that didn’t already have the pre-existing write permission granted) - Data stored in the managed application directories on external storage
(i.e.
/Android/data/<PACKAGE_NAME>) require no permissions at all if you are the application that owns those files.
In KitKat, the external storage API was split out to include multiple volumes;
one "primary" and one or more "secondary". The primary volume is, for all intents
and purposes, exactly the same as the previous single volume. All APIs that existed
prior to KitKat reference the primary external storage. The secondary volume(s)
modify write permissions a bit; they are globally readable under the same permission
described above. Directories outside the application’s own managed area
(i.e. /Android/data/<PACKAGE_NAME>) are not writable at all by that application.
There is nothing in the requirements Google places on device manufacturers that forced them to put these additional permissions on SD cards; however, there are new restrictions being enforced that require device manufacturers to place permissions on the volume they choose to make secondary storage.
That’s hopefully the same thing you’ve already read elsewhere (if it’s not, please question your source and refer them here). Also, if your understanding is that Android has no sandboxed storage options, please start with this primer on Android storage, specifically what the terms Internal vs. External really mean in this context.
But for me, that isn’t enough detail to understand what’s happening. How does all this really work?
External Storage Permissions Management
Modern versions of Android manage external storage using FUSE. A custom Android daemon participates interactively with the FUSE kernel driver in order to derive the necessary permissions for files at creation, and also to assist in dynamically accepting/rejecting individual requests based on the user/group the request came from.
This is partially how Android is able to apply Linux-style permissions onto removable volumes that are typically formatted with the FAT file system and don’t support such permissions directly. It also allows for multiple levels of permissions control beyond the basic owner/group/everyone enforcement available in the kernel.
Let’s Start By Taking a Step Back...
On earlier devices, external storage is a single volume (also called shared storage in later documents) that can be exposed to a host PC (either by removing it or via USB). It may physically live anywhere (a portion of internal flash memory, an external SD card, etc.); the only rule (enforced by CTS) is that the storage must be mounted by default whenever it isn’t otherwise in use (i.e. currently shared with your host PC). Write access to all the files on this volume is governed by a single, global permission.
Starting in FroYo (2.2 or API Level 8), the concept of application-specific directories
emerged on external storage (found at /Android/data/<PACKAGE_NAME> on the storage volume).
The permissions of these locations were no different from before (any app with the
global write permission could read/write in any other app’s directory), but the
directories are managed by the system. When an application is uninstalled, the
package manager cleans up the contents of this directory. Files outside these
directories remain unless explicitly removed by the user or another application.
The Technical Details
Below is a directory listing of external storage on a typical device running Android 4.3 (the path may vary, but the permissions will be the same):
root@generic:/storage/sdcard # ll
d---rwxr-x system sdcard_rw 2014-03-14 01:20 Alarms
d---rwxr-x system sdcard_rw 2014-03-14 01:21 Android
d---rwxr-x system sdcard_rw 2014-03-14 01:20 DCIM
d---rwxr-x system sdcard_rw 2014-03-14 01:20 Download
d---rwxr-x system sdcard_rw 2014-03-14 01:18 LOST.DIR
d---rwxr-x system sdcard_rw 2014-03-14 01:20 Movies
d---rwxr-x system sdcard_rw 2014-03-14 01:20 Music
d---rwxr-x system sdcard_rw 2014-03-14 01:20 Notifications
d---rwxr-x system sdcard_rw 2014-03-14 01:20 Pictures
d---rwxr-x system sdcard_rw 2014-03-14 01:20 Podcasts
d---rwxr-x system sdcard_rw 2014-03-14 01:20 Ringtones
The subdirectory, Android/data/, (where application-specific data could be stored)
has the same permissions structure as the above listing of the root:
root@generic_x86:/storage/sdcard # ll Android/data/
drwxrwx--- system sdcard_rw 2014-03-14 01:21 com.google.android.apps.maps
The android.permission.WRITE_EXTERNAL_STORAGE permission grants membership in
the sdcard_rw kernel group. Any member of sdcard_rw has R/W access to all files
on this volume. Also, every user id has read access regardless of their membership
in the group (i.e. there was no permission necessary to read all the files on this volume).
The android.permission.READ_EXTERNAL_STORAGE permission grants membership in the
sdcard_r kernel group. This permission was added in 4.1 (API Level 16), but,
up to and including 4.3, it was not enforced by the system.
Behind the Scenes
Underneath the framework API, support for multiple storage volumes has been available
to device manufacturers for some time (certainly in its current form since ICS,
and probably back to Honeycomb without much change). Device manufacturers could
create a single "primary" and multiple "secondary" volumes that the StorageManager
and MountService system services could manage. In these cases, access to the
"primary" volume surfaced as the single external storage volume we are all familiar
with.
Many devices at this time had SD cards but did not use them as their external storage volume (from the perspective of the Android APIs); this was a "secondary" volume on those devices. The Samsung Galaxy devices, for example, fall into this category. From a permissions perspective, the SD card was governed the same way as the exposed external storage volume. However, no framework API existed to access the secondary volume.
Below is a snippet from the AOSP device storage configuration example:
on init
mkdir /mnt/shell/emulated 0700 shell shell
mkdir /storage/emulated 0555 root root
mkdir /mnt/media_rw/sdcard1 0700 media_rw media_rw
mkdir /storage/sdcard1 0700 root root
export EXTERNAL_STORAGE /storage/emulated/legacy
export EMULATED_STORAGE_SOURCE /mnt/shell/emulated
export EMULATED_STORAGE_TARGET /storage/emulated
export SECONDARY_STORAGE /storage/sdcard1
Any part of secondary storage was usable by internal system applications or
third-party apps willing to extract and manually reference the SECONDARY_STORAGE
environment variable where this path was typically exported (although even this
method was not guaranteed on any given device, it was just the convention).
This is the part that is about to change...
Now We Can Move Forward...
Starting in KitKat, the concept of "primary" and "secondary" external storage finally emerges in the framework API:
Context.getExternalCacheDirs()Context.getExternalFilesDirs()Context.getObbDirs()Environment.getStorageState()
Each of these methods previously had a corresponding API that returned a single path on primary external storage to the application’s specific data directory. The new versions return a usable path on all available storage volumes. In these cases, "primary" external storage and the single external storage volume we knew from before are the same thing, both in actual mount location and in the permissions that govern read/write. Secondary external storage emerges as a new area with a different set of (slightly more complicated) permissions rules.
Tip: The first item returned from these new APIs is always the primary volume. Even though the docs don’t say this explicitly, CTS enforces it. Any additional items are considered secondary.
The existing methods on Environment still assume primary external storage as well:
getExternalStorageDirectory()getExternalStoragePublicDirectory()getExternalStorageState()
In KitKat, there are still no public APIs to obtain a path reference to any
top-level directories (i.e. anything outside those application-specific directories)
on secondary storage volumes. In other words, locations like /DCIM or /Pictures
that may already exist on secondary cannot be referenced through an official public framework API.
More Technical Details
Below is a directory listing of external storage on a typical device running Android 4.4 (the path may vary, but the permissions will be the same):
root@generic_x86:/storage/sdcard # ll
drwxrwx--- root sdcard_r 2013-11-27 23:35 Alarms
drwxrwx--x root sdcard_r 2013-11-27 23:36 Android
drwxrwx--- root sdcard_r 2014-03-14 01:33 DCIM
drwxrwx--- root sdcard_r 2013-11-27 23:35 Download
drwxrwx--- root sdcard_r 2013-11-28 04:33 LOST.DIR
drwxrwx--- root sdcard_r 2013-11-27 23:35 Movies
drwxrwx--- root sdcard_r 2013-11-27 23:35 Music
drwxrwx--- root sdcard_r 2013-11-27 23:35 Notifications
drwxrwx--- root sdcard_r 2013-11-27 23:35 Pictures
drwxrwx--- root sdcard_r 2013-11-27 23:35 Podcasts
drwxrwx--- root sdcard_r 2013-11-27 23:35 Ringtones
The android.permission.READ_EXTERNAL_STORAGE permission still grants membership
in the sdcard_r kernel group. Any member of sdcard_r has read access to all
the files on this volume. Note that the ability for all non-owner/non-members to
read files has been removed. This permission is now enforced in a fashion similar
to sdcard_rw in previous versions.
If we look into the application-specific directories, things are slightly different:
root@generic_x86:/storage/sdcard # ll Android/data/
drwxrwx--- u0_a33 sdcard_r 2013-11-27 23:36 com.google.android.apps.maps
root@generic_x86:/storage/sdcard # ll Android/data/com.google.android.apps.maps/
drwxrwx--- u0_a33 sdcard_r 2013-11-27 23:36 cache
drwxrwx--- u0_a33 sdcard_r 2013-11-27 23:36 testdata
In KitKat, full ownership of the app-specific data directories is given to the app’s unique user ID. This means that, going forward, no permission is necessary for an app to read/write to its specific directories on external storage.
Note: In both of the above listings, the sdcard_r group has full +rwx permissions, which obviously is not true in practice. This does not present the full picture, because the FUSE daemon is an active participant in modifying the permissions applied to applications at runtime. However, these bits do have a very important consequence on how the File APIs canRead(), canWrite(), and canExecute() behave. These APIs return the values noted in the kernel’s file system alone, so they will all return true for files here even if an attempt to POSIX open the file will fail.
The android.permission.WRITE_EXTERNAL_STORAGE permission now grants membership
in sdcard_r AND sdcard_rw. Verifying write permissions requires some more
dynamic checking in KitKat, so the FUSE daemon is used to supplement the file system
permissions. The FUSE daemon enforces that applications who own a specific directory
are granted access outright, per the file system permissions described above;
any other process is validated if they are members of sdcard_rw (i.e. they have permission).
The group for which the FUSE daemon enforces write-protected access on non-owners
defaults to sdcard_rw but is configurable using the -w flag.
Looking again at another snippet from the
storage config sample:
service sdcard /system/bin/sdcard -u 1023 -g 1023 -l /data/media /mnt/shell/emulated
class late_start
service fuse_sdcard1 /system/bin/sdcard -u 1023 -g 1023 -w 1023 -d /mnt/media_rw/sdcard1 /storage/sdcard1
class late_start
disabled
We see that the daemon instance controlling the SD card enforces GID 1023
(a.k.a. media_rw, a GID attainable only by system applications) membership in
order to write to directories that a given application does not own.
What Does This Mean at a Higher Level?
Let’s distill this down into what we can and can’t do from an application running on a KitKat device that has multiple external storage volumes. The table below indicates what an application developer might try to do and how KitKat will respond:
| Action | Primary | Secondary |
|---|---|---|
| Read Top-Level Directories | R1 | R1 |
| Write Top-Level Directories | W1 | N1 |
| Read My Package’s Android Data Directory | Y1 | Y1 |
| Write My Package’s Android Data Directory | Y1 | Y1 |
| Read Another Package’s Android Data Directory | R1 | R1 |
| Write Another Package’s Android Data Directory | W1 | N1 |
Some key notes about this chart:
- No permissions are necessary to use any external storage volume for application-specific files.
- With the exception of the above, the permissions of primary external storage have not changed; they behave the same in 4.4 as they did before.
- Secondary volumes are fully readable with the global read permission granted. If a user were to dump a bunch of files in random locations on an SD card and insert it into the device, any application could still read them. Third-party applications just can’t add more files of their own in random locations.
Why Now?
The answer, in an acronym, is CTS. If you aren’t familiar with CTS or how it relates to Android, a good starting point is the official AOSP page.
The latest version of the CDD does not include any new language for external storage. No specific provisions exist for enforcing that removable media cannot be the primary external storage volume. In fact, Google still provides AOSP configuration examples for device manufacturers to include external storage using only a single, physically removable volume. Additionally, rules stating that secondary storage volumes should not be writable by applications have also been in the document since 4.2 (when multi-user support was introduced). At a glance, it looks like nothing has changed here.
However, new tests were added in CTS for 4.4 that validate whether or not secondary storage has the proper read-only permissions in non app-specific directories, presumably because of the new APIs to finally expose those paths to application developers. As soon as CTS includes these rules, OEMs have to support them to keep shipping devices with GMS (Google Play, etc.) on-board.
What About Sharing Files?
It’s a valid question. What if an application needs to share the files it has created on a secondary external storage volume? Google’s answer seems to be that those applications who actively decide to go beyond primary external storage to write content should also expose a secure way of sharing it, either using a content provider or the new Storage Access Framework.
Samsung: A Case Study
Being the largest Android manufacturer, as well as one who regularly includes SD card slots on their devices, Samsung has taken the spotlight in this discussion. In Android 4.3, Samsung emulated the primary external storage volume on the device’s internal flash (NOT the removable SD card). The SD card had always internally been marked as the secondary external storage medium. It was used by system apps (like Camera) that may require expanded storage options. In Android 4.4, the primary external storage volume is STILL on internal flash. They didn’t secretly move the location of primary external storage from the SD card to internal flash during the version change, causing files to get abandoned. Secondary volumes simply gained new API exposure and a set of new permissions. Apps that were using the framework APIs all along wouldn’t even notice this change.
Samsung made the choice to include an SD card slot but not mark that as the primary external storage medium, a choice they are rightfully allowed to make as an Android OEM, and one they made long before KitKat. Was that the right choice or the wrong choice? I don’t know. However, there is no external evidence to suggest that Google (more specifically, CTS) coerced them into that paradigm.
Perhaps as a product of that initial choice when KitKat came along, Samsung may have been faced with the decision to adhere to the new permissions on the SD card OR move the SD card back to being the primary volume. In that context, it would seem they made a choice that would affect the least number of apps and users, without undergoing massive file transferring during the OS update. This is only speculation on my part, though.
Final Thoughts
The most important thing to remember here is that primary external storage as we know it didn’t disappear in KitKat, and the permissions model for working with files on that volume also did not change. Applications using the public framework API should not have seen any ill-effects from the permissions changes that took effect in KitKat. It seems clear that Google’s intention for secondary storage volumes is to provide application-specific containers on those volumes. This does not correlate to someone saying that this is Google’s intention for SD cards; the choice of physical media used for both primary and secondary storage is still 100% up to the manufacturer (including the decision to include any secondary storage at all).
The truth is, applications that saw negative effects from the permissions changes on secondary volumes were accessing the SD card storage in a roundabout way without direct API support. I’m not declaring that makes them inherently bad or wrong; the ability to add new volumes without all the APIs in place to fully interact with them also seems a bit off to me. However, the API is the contract between the framework and the developer. Unless a behavior changes that explicitly breaks that contract, it’s difficult for me to get behind anyone pointing the finger at the platform.
The story here also isn’t over. Take a glance over some of the AOSP code I’ve
linked in this article, and you’ll see that the future for these secondary volumes
is in flux; this looks more like a staged rollout than the final chapter.
If I were to put my two cents in regarding the future here, I’d expect that
secondary volumes become the default destination for larger media, governed by
custom permissions specifically for media access (see AID_SDCARD_PICS and AID_SDCARD_AV).
As a final side note, thanks a million to members of the Android Framework team who took the time to answer questions on the developer forums. Posts like this are possible because of your willingness to share information.
-
R = With Read Permission, W = With Write Permission, Y = Always, N = Never ↩︎