Due to https://r.android.com/1310496, sparse_file_write() splits all
blocks larger than 64 MiB. However, the code that splits file-backed
blocks copies the pointer to the filename without duplicating the
underlying memory, causing a double free in backed_block_destroy()
later. Fix this by using strdup(). Also, as long as that is being
fixed, also check for failure.
Test: SANITIZE_HOST=address mmm external/e2fsprogs
mkdir mnt
mkfs.ext4 img 1G
sudo mount img mnt
sudo cp /dev/urandom mnt/file
sudo umount mnt
ext2simg img simg
Before this fix it gave:
==2216498==ERROR: AddressSanitizer: attempting double-free on 0x602000000090 in thread T0:
#0 0x55a52454c9a2 in free out/stage2/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/out/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:52:3
#1 0x7ffa247c82ee in backed_block_destroy(backed_block*) system/core/libsparse/backed_block.cpp:106:5
#2 0x7ffa247c82ee in backed_block_list_destroy(backed_block_list*) system/core/libsparse/backed_block.cpp:124:7
#3 0x7ffa247cd055 in sparse_file_destroy system/core/libsparse/sparse.cpp:49:3
#4 0x55a524587b75 in main external/e2fsprogs/contrib/android/ext2simg.c:239:2
Change-Id: I4607ef5adcf6512645342beaf91aff6033414e54
libsparse uses mapped files for length computation checks and writing
output data. The platform-tools package for Windows is 32-bit, and if
an embedded file in the stream is large enough, mapping will fail. In
theory, this failure mode could happen on 64-bit systems as well.
As a workaround, map files in chunks of 256MB instead. This is
implemented by adding a new "fd_chunk" callback to the sparse ops
struct.
Bug: 273933042
Bug: 268872725
Test: fastboot update on Windows
Change-Id: Ic40696b34a1d0951787c899db701fc2fa204eb18
MappedFile creation can fail due to out of memory, but this condition is
not handled properly when resparsing. The error is silently ignored and
the result is a corrupt file.
Bug: 273933042
Bug: 268872725
Test: fastboot update on Windows
Change-Id: I4d0f24d6ba390e2328de8f0e3637d17663743df5
It was only added in 2012, so it's been dead for 10 times longer than it
was ever built for at this point... YAGNI.
Test: treehugger
Change-Id: I6f31a3d1354f4d3304da463322cd1d9eff9e0597
In python3, division of two integers produces a float,
which cannot be used to multiply a bytes sequence.
Using // to produce an integer quotient.
Bug: 245257511
Test: ./system/core/libsparse/simg_dump.py -sv path/to/image
Change-Id: Ibe95056409ac3676a68f5e7693331ca4a66e9fb7
This function is intended for use on sparse or non-sparse files. But
when used on non-sparse files, it unintentionally reports errors when
trying to detect if the file is sparse or not.
Since the behavior is the same whether or not the detection fails, this
patch disables verbose errors during the import. Verbose errors are
turned on if a resparse is needed.
Bug: 210846593
Test: fastboot flashall
Change-Id: I8404ca7a67adc7fba8a3ac7b4ffa18abfbcafa9d
This adds support for converting files with holes to "don't care"
chunks. This can result in a substantial reduction in the time it takes
to program an image if it has many holes.
Generally, constants compared to argc have been reduced by one, since we
no longer have the program name as the first argument.
Change-Id: I00750edc07d6415dcc07ae0351e9397b0222b7ba
This adds support for filesystem-level sparse files. These files have
holes which are not stored in the filesystem and when read are full of
zeros. While these zeros may be significant in some types of files,
other types of files may not care about the contents of holes. For
example, most filesystem creation tools write to all the blocks they
care about. Those blocks not written to will remain holes, and can be
safely represented by "don't care" chunks. Using "don't care" chunks
instead of fill chunks can result in a substantial reduction of the time
it takes to program a sparse image.
To accomplish this, we extend the existing "sparse" boolean parameter to
be an enum of mode types. This enum represents the strategy we take when
reading in a file. For the most part the implementation is
straightforward. We use lseek to determine where the holes in the file
are, and then use do_sparse_file_read_normal to create chunks for the
data section. Note that every file has an implicit hole at its end.
Change-Id: I0cfbf08886fca9a91cb753ec8734c84fcbe52c9f
Sparse file can come from an untrusted source.
Need more checking to ensure that it is not a malformed
file and would not cause any OOB read access.
Update fuzz test for decoding also.
Test: adb reboot fastboot
fuzzy_fastboot --gtest_filter=Fuzz.Sparse*
fuzzy_fastboot --gtest_filter=Conformance.Sparse*
sparse_fuzzer
Bug: 212705418
Change-Id: I7622df307bb00e59faaba8bb2c67cb474cffed8e
This carves out the core of sparse_file_read_normal and splits it off so
it can be reused in the next patch. No functional change intended.
Change-Id: Id00491fd7e5bb6fa28c517a0bb32b8b506539d4d
assemble_cvd directly or indirectly depends on these modules. To add
assemble_cvd to the com.anroid.virt APEX, these modules are marked as
being available to the APEX.
Bug: 174639526
Test: m com.android.virt
Change-Id: Id3b2989a9f038a1cdc769a2021a116cf09ab1b15
We are porting cuttlefish to Android, it currently uses libsparse to
desparse Android filesystems to be used directly as disk images by
crosvm. Making this library available to 'vendor' enables us to support
desparsing downloaded images on-device as well.
Bug: 167675429
Change-Id: I8ae4a916cfb0b2002dd9e462f32e8cb0e533d821
Merging sparse chunk can make sparse map block bigger than 4GiB,
that can't be covered by unsigned integer type. Fix this by
changing unsigned int to uint64_t type.
Test: sparse build
Bug: 162808120
Change-Id: Id4d3f88f9d531c25c3937c99b2c81efb915605ee
Signed-off-by: Hyeongseok Kim <hyeongseok@gmail.com>
Cc: hyeongseok.kim <hyeongseok.kim@lge.com>
Limit the block size to 64 MB before fastboot executable binary for
windows 64-bit is released.
Bug: 156057250
Change-Id: Ic4edb963a3d99f718d7630aba3f351729a84e994
Fix memory leak in output_file_close():
Arguable whose resposibility to free structs allocated inside a file
struct (the caller or the library function), but the following protocol
(i.e. this fix) would better prevent memory leaks:
1) output_file_close() function will free those structs inside a file struct,
right before closing the file.
* Note: those structs are originally allocated by other libsparse function.
2) If the caller wants to clean up those struct inside a file struct,
it may. Just free those structs, and set the pointer to NULL.
Afterward, when file_close()'s are called, they won't be free'ed twice.
Bug: 142483439
Test: verified by script MemLeak_LastCmd.sh (attached in Bugnizer)
Change-Id: I8bb9f7c9f7d19268663e2830d1a90d27bd5f99bd
Signed-off-by: Robin Hsu <robinhsu@google.com>
this makes libsparse compile with MacOS clang++
Test: "libsparse" compiles and works fine under both Linux and MacOS
Change-Id: Ifcf018b856d2fe5f1dac7eeddd72de2810be66c5
This reverts commit db69f0d47f.
Reason for revert: Broke the build:
In file included from system/core/libsparse/sparse.cpp:26:
system/core/libsparse/output_file.h:34:72: error: unknown type name 'off64_t'; did you mean 'off_t'?
int (*skip_write)(void*, off64_t), void* priv,
^~~~~~~
off_t
/Applications/Xcode6.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/include/sys/_types/_off_t.h:30:25: note: 'off_t' declared here
typedef __darwin_off_t off_t;
^
1 error generated.
Bug: 78793464
Change-Id: I0f8bc4e9aa2f74612bfd8721d00d961e3f7e695f
Currently, sparse_file_callback uses libsparse's
own logic for reading a file into a buffer. However,
a caller may want to use their own logic for doing
this in order to customize the buffer size and parallelize
the reads/writes. Also, a caller may want to implement
fill blocks with their own logic as well. To do this
add sparse_file_typed_callback which calls a different
callback function depending on the type of the sparse
chunk being written.
Test: Use typed callback for fd writes
Bug: 78793464
Change-Id: I75955a464fc05991f806339830fdfa05fda354b9
Refactor elements of sparse file parsing that depend on
an fd into SparseFileSource class, then create implementations
using both fd and buffer. Add sparse_file_read_buf which
reads the given buffer into a sparse file cookie without
copying.
Test: flash system with sparse images
Bug: 78793464
Change-Id: Ice6c8e1ff075d6867e070f80fcf5aa4f530a1b95
adbd (and its dependencies) are marked as recovery_available:true so
that recovery version of the binary is built separately from the one for
system partition. This allows us to stop copying the system version to
the recovery partition and also opens up the way to enable shared
libraries in the recovery partition. Then we can also build adbd as a
dynamic executable.
Bug: 79146551
Test: m -j adbd.recovery
Change-Id: Ib95614c7435f9d0afc02a0c7d5ae1a94e439e32a
This CL updates the callback function signature in
sparse_file_callback() and sparse_file_foreach_chunk().
Before:
int sparse_file_callback(
struct sparse_file *s, bool sparse, bool crc,
int (*write)(void *priv, const void *data, int len), void *priv);
int sparse_file_foreach_chunk(
struct sparse_file *s, bool sparse, bool crc,
int (*write)(
void *priv, const void *data, int len, unsigned int block,
unsigned int nr_blocks),
void *priv);
After:
int sparse_file_callback(
struct sparse_file *s, bool sparse, bool crc,
int (*write)(void *priv, const void *data, size_t len), void *priv);
int sparse_file_foreach_chunk(
struct sparse_file *s, bool sparse, bool crc,
int (*write)(
void *priv, const void *data, size_t len, unsigned int block,
unsigned int nr_blocks),
void *priv);
The length (i.e. 'len') comes from the size of a chunk, which could be
legitimately larger than INT_MAX. Prior to this CL, callers (e.g.
write_sparse_data_chunk()) were already passing unsigned int to the
callbacks. When a chunk size exceeds INT_MAX, the callback would see a
negative value, which could lead to undesired behavior. For example,
out_counter_write(), as one of the internal callbacks in libsparse,
gives a wrong sum of chunk sizes, which in turn fails the fastboot
flashing when given a huge sparse image.
It also defines SPARSE_CALLBACK_USES_SIZE_T that allows clients to keep
their codes compatibile with both versions.
Bug: 78432315
Test: `m dist` (with matching changes to all the clients)
Test: Build fastboot and successfully flash a previously failing (huge)
sparse image.
Change-Id: Iac4bcf7b57039d08af3c57f4be00d75f6b693d93
