make mmap fail on requests larger than PTRDIFF_MAX

Allocations larger than PTRDIFF_MAX can be successfully created on
32-bit with a 3:1 split, or in 32-bit processes running on 64-bit.

Allowing these allocations to succeed is dangerous, as it introduces
overflows for `end - start` and isn't compatible with APIs (mis)using
ssize_t to report either the size or an error. POSIX is guilty of this,
as are many other Android APIs. LLVM even considers the `ptr + size`
case to be undefined, as all pointer arithmetic compiles down to signed
operations and overflow is treated as undefined for standard C pointer
arithmetic (GNU C `void *` arithmetic works differently).

This also prevents dlmalloc from allocating > PTRDIFF_MAX as it doesn't
merge mappings like jemalloc. A similar check will need to be added in
jemalloc's code path for huge allocations.

The musl libc implementation also performs this sanity check.

Change-Id: I5f849543f94a39719f5d27b00cef3079bb5933e9

libc/bionic/mmap.cpp

@@ -30,6 +30,7 @@
 #include <sys/mman.h>
 #include <unistd.h>
 
+#include "private/bionic_macros.h"
 #include "private/ErrnoRestorer.h"
 
 // mmap2(2) is like mmap(2), but the offset is in 4096-byte blocks, not bytes.
@@ -45,6 +46,13 @@ void* mmap64(void* addr, size_t size, int prot, int flags, int fd, off64_t offse
     return MAP_FAILED;
   }
 
+  // prevent allocations large enough for `end - start` to overflow
+  size_t rounded = BIONIC_ALIGN(size, PAGE_SIZE);
+  if (rounded < size || size > PTRDIFF_MAX) {
+    errno = ENOMEM;
+    return MAP_FAILED;
+  }
+
   bool is_private_anonymous = (flags & (MAP_PRIVATE | MAP_ANONYMOUS)) != 0;
   void* result = __mmap2(addr, size, prot, flags, fd, offset >> MMAP2_SHIFT);