Android anonymous memory analysis
Why do I still need anonymous memory to implement IPC with the binder mechanism? I think the big reason is that binder transmission has size limitations, not to mention the application layer limitations. In the driver, the transmission size of the binder is limited to 4M, and sharing a picture may exceed this limit. The main solution to anonymous memory is to transfer file descriptors through binder, so that both processes can access the same address to achieve sharing.
MemoryFile usage
In normal development android provides MemoryFile to implement anonymous memory. Take a look at the simplest implementation.
Service side
const val GET_ASH_MEMORY = 1000
class MyService : Service() {
val ashData = "AshDemo".toByteArray()
override fun onBind(intent: Intent): IBinder {
return object : Binder() {
override fun onTransact(code: Int, data: Parcel, reply: Parcel?, flags: Int): Boolean {
when(code){
GET_ASH_MEMORY->{//It will be annoying when receiving client request val descriptor = createMemoryFile()
reply?.writeParcelable(descriptor, 0)
reply?.writeInt()
return true
}
else->{
return (code, data, reply, flags)
}
}
}
}
}
private fun createMemoryFile(): ParcelFileDescriptor? {
val file = MemoryFile("AshFile", 1024)//Create MemoryFile val descriptorMethod = ("getFileDescriptor")
val fd=(file)//Reflection gets fd (ashData, 0, 0,)//Write string return (fd as FileDescriptor?)//Return a packaged fd }
}
The server's function is very simple to create a MemoryFile when receiving a GET_ASH_MEMORY request, write a string byte array, and then write fd and character length into reply to return to the client.
Client
class MainActivity : AppCompatActivity() {
val connect = object :ServiceConnection{
override fun onServiceConnected(name: ComponentName?, service: IBinder?) {
val reply = ()
val sendData = ()
service?.transact(GET_ASH_MEMORY, sendData, reply, 0)//Transmission signal GET_ASH_MEMORY val pfd = <ParcelFileDescriptor>()
val descriptor = pfd?.fileDescriptor//Get fd val size = ()//Get the length val input = FileInputStream(descriptor)
val bytes = ()
val message = String(bytes, 0, size, Charsets.UTF_8)//Generate string (this@MainActivity,message,Toast.LENGTH_SHORT).show()
}
override fun onServiceDisconnected(name: ComponentName?) {
}
}
override fun onCreate(savedInstanceState: Bundle?) {
(savedInstanceState)
setContentView(.activity_main)
findViewById<TextView>().setOnClickListener {
//Start the service bindService(Intent(this,MyService::),connect, Context.BIND_AUTO_CREATE)
}
}
}
The client is also very simple. Start the service, send a request to obtain the MemoryFile, and then get the fd and length through reply, use FileInputStream to read the content in the fd, and finally verify that the message has been obtained through toast.
AshMemory creation principle
public MemoryFile(String name, int length) throws IOException {
try {
mSharedMemory = (name, length);
mMapping = ();
} catch (ErrnoException ex) {
();
}
}
MemoryFile is a layer of packaging for SharedMemory, and the specific labor capacity is implemented by SharedMemory. Look at the implementation of SharedMemory.
public static @NonNull SharedMemory create(@Nullable String name, int size)
throws ErrnoException {
if (size <= 0) {
throw new IllegalArgumentException("Size must be greater than zero");
}
return new SharedMemory(nCreate(name, size));
}
private static native FileDescriptor nCreate(String name, int size) throws ErrnoException;
Obtaining fd through a JNI, it can be inferred from here that the java layer is just an encapsulation, and what you get is already created fd.
//frameworks/base/core/jni/android_os_SharedMemory.cpp
jobject SharedMemory_nCreate(JNIEnv* env, jobject, jstring jname, jint size) {
const char* name = jname ? env->GetStringUTFChars(jname, nullptr) : nullptr;
int fd = ashmem_create_region(name, size);//Create anonymous memory block int err = fd < 0 ? errno : 0;
if (name) {
env->ReleaseStringUTFChars(jname, name);
}
if (fd < 0) {
jniThrowErrnoException(env, "SharedMemory_create", err);
return nullptr;
}
jobject jifd = jniCreateFileDescriptor(env, fd);//Create java fd return if (jifd == nullptr) {
close(fd);
}
return jifd;
}
Creation through the ashmem_create_region function in cutils
//system/core/libcutils/
int ashmem_create_region(const char *name, size_t size)
{
int ret, save_errno;
if (has_memfd_support()) {//Compatible with old version return memfd_create_region(name ? name : "none", size);
}
int fd = __ashmem_open();//Open the Ashmem driver if (fd < 0) {
return fd;
}
if (name) {
char buf[ASHMEM_NAME_LEN] = {0};
strlcpy(buf, name, sizeof(buf));
ret = TEMP_FAILURE_RETRY(ioctl(fd, ASHMEM_SET_NAME, buf));//Set name through ioctl if (ret < 0) {
goto error;
}
}
ret = TEMP_FAILURE_RETRY(ioctl(fd, ASHMEM_SET_SIZE, size));//Set the size through ioctl if (ret < 0) {
goto error;
}
return fd;
error:
save_errno = errno;
close(fd);
errno = save_errno;
return ret;
}
Standard driver interaction
Turn on the driver
2. Interact with drivers through ioctl
Let's see the process of open
static int __ashmem_open()
{
int fd;
pthread_mutex_lock(&__ashmem_lock);
fd = __ashmem_open_locked();
pthread_mutex_unlock(&__ashmem_lock);
return fd;
}
/* logistics of getting file descriptor for ashmem */
static int __ashmem_open_locked()
{
static const std::string ashmem_device_path = get_ashmem_device_path();//Get the Ashmem driver path if (ashmem_device_path.empty()) {
return -1;
}
int fd = TEMP_FAILURE_RETRY(open(ashmem_device_path.c_str(), O_RDWR | O_CLOEXEC));
return fd;
}
Go back to the MemoryFile constructor, get the driver's fd and call mapReadWrite
public @NonNull ByteBuffer mapReadWrite() throws ErrnoException {
return map(OsConstants.PROT_READ | OsConstants.PROT_WRITE, 0, mSize);
}
public @NonNull ByteBuffer map(int prot, int offset, int length) throws ErrnoException {
checkOpen();
validateProt(prot);
if (offset < 0) {
throw new IllegalArgumentException("Offset must be >= 0");
}
if (length <= 0) {
throw new IllegalArgumentException("Length must be > 0");
}
if (offset + length > mSize) {
throw new IllegalArgumentException("offset + length must not exceed getSize()");
}
long address = (0, length, prot, OsConstants.MAP_SHARED, mFileDescriptor, offset);//The system's mmap has been called boolean readOnly = (prot & OsConstants.PROT_WRITE) == 0;
Runnable unmapper = new Unmapper(address, length, ());
return new DirectByteBuffer(length, address, mFileDescriptor, unmapper, readOnly);
}
At this point, there is a question: Linux has shared memory. Why do Android do it by itself? I can only look at the implementation of the Ashmemory driver.
The first step of driver is to look at init and file_operations
static int __init ashmem_init(void)
{
int ret = -ENOMEM;
ashmem_area_cachep = kmem_cache_create("ashmem_area_cache",
sizeof(struct ashmem_area),
0, 0, NULL);//create if (!ashmem_area_cachep) {
pr_err("failed to create slab cache\n");
goto out;
}
ashmem_range_cachep = kmem_cache_create("ashmem_range_cache",
sizeof(struct ashmem_range),
0, SLAB_RECLAIM_ACCOUNT, NULL);//create if (!ashmem_range_cachep) {
pr_err("failed to create slab cache\n");
goto out_free1;
}
ret = misc_register(&ashmem_misc);//Register for a misc device ........
return ret;
}
Two memory allocators ashmem_area_cachep and ashmem_range_cachep were created to allocate ashmem_area and ashmem_range
//common/drivers/staging/android/
static const struct file_operations ashmem_fops = {
.owner = THIS_MODULE,
.open = ashmem_open,
.release = ashmem_release,
.read_iter = ashmem_read_iter,
.llseek = ashmem_llseek,
.mmap = ashmem_mmap,
.unlocked_ioctl = ashmem_ioctl,
#ifdef CONFIG_COMPAT
.compat_ioctl = compat_ashmem_ioctl,
#endif
#ifdef CONFIG_PROC_FS
.show_fdinfo = ashmem_show_fdinfo,
#endif
};
Open calls ashmem_open
static int ashmem_open(struct inode *inode, struct file *file)
{
struct ashmem_area *asma;
int ret;
ret = generic_file_open(inode, file);
if (ret)
return ret;
asma = kmem_cache_zalloc(ashmem_area_cachep, GFP_KERNEL);//Assign an ashmem_area if (!asma)
return -ENOMEM;
INIT_LIST_HEAD(&asma->unpinned_list);//Initialize unpinned_list memcpy(asma->name, ASHMEM_NAME_PREFIX, ASHMEM_NAME_PREFIX_LEN);//Initialize a name asma->prot_mask = PROT_MASK;
file->private_data = asma;
return 0;
}
The name and length of ioctl call ashmem_ioctl
static long ashmem_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
{
struct ashmem_area *asma = file->private_data;
long ret = -ENOTTY;
switch (cmd) {
case ASHMEM_SET_NAME:
ret = set_name(asma, (void __user *)arg);
break;
case ASHMEM_SET_SIZE:
ret = -EINVAL;
mutex_lock(&ashmem_mutex);
if (!asma->file) {
ret = 0;
asma->size = (size_t)arg;
}
mutex_unlock(&ashmem_mutex);
break;
}
........
}
The implementation is also very simple, just change the value in asma. Next is the focus of mmap, how to allocate memory.
static int ashmem_mmap(struct file *file, struct vm_area_struct *vma)
{
static struct file_operations vmfile_fops;
struct ashmem_area *asma = file->private_data;
int ret = 0;
mutex_lock(&ashmem_mutex);
/* user needs to SET_SIZE before mapping */
if (!asma->size) {//Judge size is set ret = -EINVAL;
goto out;
}
/* requested mapping size larger than object size */
if (vma->vm_end - vma->vm_start > PAGE_ALIGN(asma->size)) {//Judge whether the size exceeds the virtual memory ret = -EINVAL;
goto out;
}
/* requested protection bits must match our allowed protection mask */
if ((vma->vm_flags & ~calc_vm_prot_bits(asma->prot_mask, 0)) &
calc_vm_prot_bits(PROT_MASK, 0)) {//Permission judgment ret = -EPERM;
goto out;
}
vma->vm_flags &= ~calc_vm_may_flags(~asma->prot_mask);
if (!asma->file) {//Have you created a temporary file, and you have not created it to enter char *name = ASHMEM_NAME_DEF;
struct file *vmfile;
struct inode *inode;
if (asma->name[ASHMEM_NAME_PREFIX_LEN] != '\0')
name = asma->name;
/* ... and allocate the backing shmem file */
vmfile = shmem_file_setup(name, asma->size, vma->vm_flags);//Calling linux function to create temporary files in tmpfs if (IS_ERR(vmfile)) {
ret = PTR_ERR(vmfile);
goto out;
}
vmfile->f_mode |= FMODE_LSEEK;
inode = file_inode(vmfile);
lockdep_set_class(&inode->i_rwsem, &backing_shmem_inode_class);
asma->file = vmfile;
/*
* override mmap operation of the vmfile so that it can't be
* remapped which would lead to creation of a new vma with no
* asma permission checks. Have to override get_unmapped_area
* as well to prevent VM_BUG_ON check for f_ops modification.
*/
if (!vmfile_fops.mmap) {//Set the temporary file operation to prevent the temporary file from being used as mmap. vmfile_fops = *vmfile->f_op;
vmfile_fops.mmap = ashmem_vmfile_mmap;
vmfile_fops.get_unmapped_area =
ashmem_vmfile_get_unmapped_area;
}
vmfile->f_op = &vmfile_fops;
}
get_file(asma->file);
/*
* XXX - Reworked to use shmem_zero_setup() instead of
* shmem_set_file while we're in staging. -jstultz
*/
if (vma->vm_flags & VM_SHARED) {//Does this memory need to cross process ret = shmem_zero_setup(vma);//Set the file if (ret) {
fput(asma->file);
goto out;
}
} else {
/**
The implementation is to set vm_ops to NULL
static inline void vma_set_anonymous(struct vm_area_struct *vma)
{
vma->vm_ops = NULL;
}
*/
vma_set_anonymous(vma);
}
vma_set_file(vma, asma->file);
/* XXX: merge this with the get_file() above if possible */
fput(asma->file);
out:
mutex_unlock(&ashmem_mutex);
return ret;
}
The function is long, but the idea is still clear. Create temporary files and set file operations. The system functions of Linux are called. Look at the shmem_zero_setup function that is actually set
int shmem_zero_setup(struct vm_area_struct *vma)
{
struct file *file;
loff_t size = vma->vm_end - vma->vm_start;
/*
* Cloning a new file under mmap_lock leads to a lock ordering conflict
* between XFS directory reading and selinux: since this file is only
* accessible to the user through its mapping, use S_PRIVATE flag to
* bypass file security, in the same way as shmem_kernel_file_setup().
*/
file = shmem_kernel_file_setup("dev/zero", size, vma->vm_flags);
if (IS_ERR(file))
return PTR_ERR(file);
if (vma->vm_file)
fput(vma->vm_file);
vma->vm_file = file;
vma->vm_ops = &shmem_vm_ops;//A very important operation sets the vm_ops of this virtual memory to shmem_vm_ops
if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
((vma->vm_start + ~HPAGE_PMD_MASK) & HPAGE_PMD_MASK) <
(vma->vm_end & HPAGE_PMD_MASK)) {
khugepaged_enter(vma, vma->vm_flags);
}
return 0;
}
static const struct vm_operations_struct shmem_vm_ops = {
.fault = shmem_fault,//The basics of shared memory implementation of Linux .map_pages = filemap_map_pages,
#ifdef CONFIG_NUMA
.set_policy = shmem_set_policy,
.get_policy = shmem_get_policy,
#endif
};
Here the initialization of shared memory is over.
AshMemory Read and Write
//frameworks/base/core/java/android/os/
public void writeBytes(byte[] buffer, int srcOffset, int destOffset, int count)
throws IOException {
beginAccess();
try {
(destOffset);
(buffer, srcOffset, count);
} finally {
endAccess();
}
}
private void beginAccess() throws IOException {
checkActive();
if (mAllowPurging) {
if (native_pin((), true)) {
throw new IOException("MemoryFile has been purged");
}
}
}
private void endAccess() throws IOException {
if (mAllowPurging) {
native_pin((), false);
}
}
where beginAccess and endAccess are corresponding. All native_pin is a native function, one parameter is true and the other is false. The purpose of pin is to lock this piece of memory and not be recycled by the system, and unlock it when it is not used.
static jboolean android_os_MemoryFile_pin(JNIEnv* env, jobject clazz, jobject fileDescriptor,
jboolean pin) {
int fd = jniGetFDFromFileDescriptor(env, fileDescriptor);
int result = (pin ? ashmem_pin_region(fd, 0, 0) : ashmem_unpin_region(fd, 0, 0));
if (result < 0) {
jniThrowException(env, "java/io/IOException", NULL);
}
return result == ASHMEM_WAS_PURGED;
}
The ashmem_pin_region and ashmem_unpin_region are called to achieve unlocking and unlocking. Implementation or yet
//system/core/libcutils/
int ashmem_pin_region(int fd, size_t offset, size_t len)
{
.......
ashmem_pin pin = { static_cast<uint32_t>(offset), static_cast<uint32_t>(len) };
return __ashmem_check_failure(fd, TEMP_FAILURE_RETRY(ioctl(fd, ASHMEM_PIN, &pin)));
}
The driver of IOCLT notification is also passed. The details of the locking will not be expanded. The specific writing is the sharing achieved using Linux's shared memory mechanism.
Introduction to Linux Sharing Mechanism
The simple way to share is to implement it through the same file as mmap. However, the reading and writing speed of real files is too slow, so using the virtual file system tmpfs, a virtual file is created to read and write. At the same time, this virtual memory exists above, and it also rewrites vm_ops. When a process operates this virtual memory, a page-failure error will be triggered, and then the Page cache will be searched. Since it is the first time, there is no cache, and the physical memory is read, and the Page cache will be added. When the second process comes in, the page cache will be found, and then the Page cache will be found. Then they operate the same piece of physical memory.
Summarize
After reading it, I found that AshMemory is implemented based on shared memory of Linux. A few modifications have been made
- First, turn a whole piece of memory into regions, so that it can be unlocked when not in use to allow the system to recycle.
- Mark the integers of Linux shared memory to share memory, while AshMemory uses fd, so that it can use the binder mechanism to transfer fd.
- The read and write settings are locked, reducing the difficulty of users.
The above is the detailed content of the in-depth analysis of Android anonymous memory. For more information about Android anonymous memory, please follow my other related articles!