Currently, EXT4 image building is done in a 2 pass fashion:
First pass: build ext4 image with small margin/headroom, check the
number of block used in the output image
Second pass: Build ext4 image again with an estimated block count from
first pass
The two pass implementation is done to keep image size at a
minimum. Recently some failures in first pass invocation is observed due
to image being undersized. Since we will reduce the image size in 2nd
pass, it's OK to increase the image size estimate during first pass. Add
a headroom 10% of total file size to fix build failures.
Test: cherry pick r.android.com/q/topic:%22aosp_cf_system_x86_64%22 and
m aosp_cf_system_x86_64
Bug: 321003625
Change-Id: I254310c67a08b98b05d2c858d4ab59a48112a07b
This variable is always false or empty now, so clean up remaining
references to the variable.
Bug: 241346584
Test: Presubmit
Change-Id: I5dd3d356729d0bd2639633580bbb85f8b03582cf
F2FS has been expanded to allow for different block sizes. Currently the
block size must be the same as page size, but this may change in the
future. This gives support for 16K while not restricting the ability to
choose a block size that isn't based on the page size later on.
Bug: 279820706
Test: Boot 16k F2FS device
Change-Id: I4e025d83a5e5ad03d89bc80b02fea390c2cbad69
This change allows us to specify erofs filesystem block size
which will be used to create erofs image.
Either it can be specified at board level using
BOARD_EROFS_BLOCKSIZE. Or per read only partition image
using BOARD_<PARTITION>IMAGE_EROFS_BLOCKSIZE.
Examples:
BOARD_EROFS_BLOCKSIZE := 16384 OR
BOARD_PRODUCTIMAGE_EROFS_BLOCKSIZE := 16384
Bug: 303316443
Test: m and file ${ANDROID_PRODUCT_OUT}/*.img
Change-Id: I9bf3d299ae02165d0bffc62f474a302408d8b9b9
Signed-off-by: Sandeep Dhavale <dhavale@google.com>
build_image.py now accepts a --input-directory-filter-file argument
that contains a list of files to consider from the input directory.
A new, temporary input directory will be created, and only the
considered files will be copied to it.
This removes most of the need for `m installclean`, as old files or
files built manually will no longer end up on the partition.
Bug: 205632228
Test: Presubmits
Change-Id: I07e0bf8e840abf0b51a2fdf67b758905fb3c5b5b
build_image.py now accepts a --input-directory-filter-file argument
that contains a list of files to consider from the input directory.
A new, temporary input directory will be created, and only the
considered files will be copied to it.
This removes most of the need for `m installclean`, as old files or
files built manually will no longer end up on the partition.
Bug: 205632228
Test: Presubmits
Change-Id: I4c82ac7f1ed49a37271755749c8f72afeeb7f295
For the OTA build, target_files.zip requires setting fixed timestamp
for images. However, it caused regression on 'adb sync'.
Restoring image build process for target_files.zip instead of copying
them from the PRODUCT_OUT, only the images in target_files.zip set the
fixed timestamp.
By setting COPY_IMAGES_FOR_TARGET_FILES_ZIP flag, we still may copy
the images for target_files.zip for future use.
Bug: 287534409
Test: m; flashall; adb sync
Test: m dist; flash images from target_files.zip and check the timestamp
Change-Id: I4135eb721c33581f191621847e6595833263d707
When building images via `m` , build_image.py is invoked directly
without going through add_img_to_target_files. To ensure images built in
either way are identical, move uuid/salt computation to build_image.py,
so that the same uuid/salt will be used.
Bug: 281960439
Test: m installclean && m && m target-files-dir , maks sure images in
$OUT and $OUT/obj/PACKING/target_files_intermediates are identical
Change-Id: Icdab29df84f5a0ec7c080f99f9fdbdc3c9b10b90
Cuttlefish needs to rebuild vbmeta at runtime, instead of duplicating
logic in cuttlefish tooling, re-use these logic in release tools.
Test: build_image $OUT misc_info.txt /tmp/vbmeta.img /tmp/otatols
Bug: 149866755
Change-Id: I2050f9e2dd7eed9b81a49a8442ec199147041d20
Should support RW for all other partitions.
Bug: 255454067
Change-Id: I9d87cf39195f8da0f6fe4153c703ba6a372aa2ea
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
We are removing VB related functionalities from release tools. This
change remove the verity related props in build_image.py, and also
remove VB1.0 bootimage builder, which will be unreachable without these
props.
Bug: 241044073
Test: atest under build/make
Change-Id: Ib3f1c100c2c6c9e63a64f49269619b76074a27fd
Currently the only way to do this is to enable deduplication.
Bug: N/A
Test: BOARD_EROFS_COMPRESSOR := none
Change-Id: I7f8b99fbcefe7bbc15dfab7062b321aa3e2cf1ca
Make it possible to provide a prebuilt hints file to specify different compression strategies on a file basis for all partitions using:
BOARD_EROFS_COMPRESS_HINTS := <path>/erofs-hints.txt
or separately for every partition using:
BOARD_SYSTEMIMAGE_EROFS_COMPRESS_HINTS := <path>/system-partition-erofs-hints.txt
BOARD_VENDORIMAGE_EROFS_COMPRESS_HINTS := <path>/vendor-partition-erofs-hints.txt
Hints format: Each line is defined by tokens separated by spaces in the following form:
<pcluster-in-bytes> <match-pattern>
pcluster-in-bytes can be specified as 0 which mean file will not be compressed
Bug: 231934752
Test: used dump.erofs -S system.img to verify the system image isn't compressed after specifying BOARD_SYSTEMIMAGE_EROFS_COMPRESS_HINTS := no-compress.txt, where no-compress.txt contains 0 .*
Test: used dump.erofs --nid=<inode-for-all-apk-files> vendor.img to verify apk files are not compressed (the rest of the partition content is compressed)
Test: used dump.erofs -S <all-images> to verify all images are not compressed after specifying BOARD_EROFS_COMPRESS_HINTS := <path>/no-compress.txt
Change-Id: I64054e26af5ea5781c45d2183b7796a9ad2b7927
Even though build_image uses logger but there was no way to configure it
and even -v (verbose) didn't work.
common.ParseOptions() parses common arguments (e.g. -v -h --logfile).
Bug: n/a
Test: m
Test: build_image -v ... should print logs
Change-Id: I45a61ada9c6a5a616f05d3214e185d08b0f1c152
EROFS images by default will only work on newer kernels. Old versions of
5.4 and 4.19 need an extra flag to mkfs.erofs.
Add BUILD_EROFS_USE_LEGACY_COMPRESSION to force this flag.
Bug: 222373106
Test: build with BUILD_EROFS_USE_LEGACY_COMPRESSION enabled
Change-Id: I3818e14356beaa11073c6b9fea6a6ad999bba4aa
These wrapper scripts are difficult to maintain and there's not really
any value add. The argument strings don't match and every new build flag
has to be connected in an extra place.
Bug: 222715577
Test: make with EROFS enabled
Change-Id: Ie0e51cc30aa08b004d5d8345a2a2d885e193ffa9
build_image.py has been handling fsverity metadata generation in the
packing step, but it can cause issues because the metadata files are
missing in the $OUT directory, and they only exist in result system.img.
This change moves the generation logic into Makefile, and makes the
metadata tracked by ninja graph.
Bug: 206326351
Test: PRODUCT_SYSTEM_FSVERITY_GENERATE_METADATA := true and build
Change-Id: I1f910d8ac6e2cc3c54f35916871733c632f18e44
It's not guaranteed that the requested image size to mkfs is precisely
respected, due to metadata alignment and such. For accurate care maps
use the real image size rather than requested.
Bug: 205541521
Test: smartsync to 7892270, check that care_map.pb has the right block
count
Change-Id: I60fe64f720db13d3c3c4f1d8968341d7293217c9
fsverity digest manifest stores a map from files to fsverity digests.
The manifest is installed as a serialized protobuf file, to a signed apk
system/etc/security/fsverity/BuildManifest.apk.
Bug: 193113311
Test: build with PRODUCT_SYSTEM_FSVERITY_GENERATE_METADATA := true
Change-Id: I55fc10400206b8ce0d5f198faea08fe3930b362c
Using fsverity tool, fsverity metadata for specific artifacts in system
mage can be generated. Users can do that by setting a makefile variable
PRODUCT_SYSTEM_FSVERITY_GENERATE_METADATA to true.
If set to true, the following artifacts will be signed.
- system/framework/*.jar
- system/framework/oat/<arch>/*.{oat,vdex,art}
- system/etc/boot-image.prof
- system/etc/dirty-image-objects
One fsverity metadata container file per one input file will be
generated in system.img, with a suffix ".fsv_meta". e.g. a container
file for "system/framework/foo.jar" will be
"system/framework/foo.jar.fsv_meta".
Bug: 193113311
Test: build with PRODUCT_SYSTEM_FSVERITY_GENERATE_METADATA := true
Change-Id: Ib70d591a72d23286b5debcb05fbad799dfd79b94
This adds BOARD_xIMAGE_EROFS_PCLUSTER_SIZE and
BOARD_EROFS_PCLUSTER_SIZE, which set the "pcluster size" of erofs images
for individual images or all erofs images respectively. The pcluster
size affects the maximum size of a physical compressed block.
This also adds BOARD_EROFS_SHARE_DUP_BLOCKS, which turns on chunk
support in EROFS.
Bug: 201685920
Test: manual test
Change-Id: I27ec0899f89890562796dd9fa567fc74182fbefb
This is another code block where lots of stuff is duplicated, making it
hard to add new partitions or partition features.
Bug: N/A
Test: m otapackage, treehugger
Change-Id: I4c71275303a9246b37c03b24f531925b90d26fc8
This is another code block where lots of stuff is duplicated, making it
hard to add new partitions or partition features.
Bug: N/A
Test: m otapackage, misc_info.txt is identical
Change-Id: I435987605fc66b78bfaf454556f1eaa498e3728f
This adds BOARD_EROFS_COMPRESSOR to change the compression algorithm
globally, and BOARD_{x}IMAGE_EROFS_COMPRESSOR to change it for
individual partitions.
Bug: N/A
Test: manual test
Change-Id: I2ef831558242a4070ee96269140c33b66c689351
After the erofs image is built, it won't be modified again. The reserve
room is only used for the hash footer. There is no need to set an
excessive margin.
Bug: 199248596
Test: erofs image size is smaller than before
Signed-off-by: Huang Jianan <huangjianan@oppo.com>
Change-Id: If2535b9ad2c2d60bea74feeca576a168387ff341
The image size of erofs is only affected by the content of the input
file, so it doesn't care about the input image_size. In addition, the
build time of erofs image is longer than ext4 due to compression. Only
execute mkfs once to reduce unnecessary time consumption.
Bug: 199248596
Test: erofs image build faster than before
Signed-off-by: Huang Jianan <huangjianan@oppo.com>
Change-Id: I23a4e640d5066485de3fea0f808998045e716d1f
When we introduced Dynamic Partitions, we stopped giving readonly
partitions fixed sizes. In addition we introduced deduplication for
ext4. These two factors greatly reduce the impact of sparse images,
since there aren't many fill blocks to optimize.
This patch disables sparsing for images that are rightsized and do not
explicitly specify extra reserved space. This makes the images a little
easier to work with from an engineering perspective. They no longer have
to be unsparsed to interact with any tooling. It also eases a potential
source of bugs, as b/184225422 is not reproducible with sparsing off.
On Pixel, the difference between the sparsed partitions and unsparsed is
12M (out of roughly 4G).
Bug: 198001223
Test: make, treehugger, make target-files-package
dynamic partitions are no longer sparse images
Change-Id: I74459f8abe74a15a24ba5a40cf701e6af2db8179
This option is to reduce system partition size.
Bug: 171942852
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
Change-Id: Idc849cfce33ac0badb2b9b7953bb821c46a24472
build_image adds additional parameters (uuid, hash_seed) if
prop_dict["ext_mkuserimg"] is set to "mkuserimg_mke2fs".
The comparison does not take paths into consideration, so passing a
full path to mkuserimg_mke2fs would cause the parameters to not
be included.
This is currently not an issue for aosp builds, but could cause problems
for customized build systems.
Bug: 187742822
Test: Manual, using vendor build system, also executed 'm droid'
Change-Id: I7a8973dd0c4d8a39aea5aafcfe1aa69750fb1449
If dynamic partitioning is enabled and the partition size is not set,
we will get a KeyError before image_size or partition_size is calculated
when we try to catch exception in BuildImageMkfs.
Bug: 186704243
Test: build_image.py can correctly throw exception
Change-Id: I3d8c143ad5603d07fe94afb8bb911ead244f0bf7
Signed-off-by: Huang Jianan <huangjianan@oppo.com>
The current inode usage estimate applies a factor (1.04x) to account for
differences between the directory and file count and what mkfs.ext4
produces. Bump this to 1.06x to allow more leniency.
This increases bramble partition sizes by 32KB total.
Bug: 182365132
Test: manual test
Change-Id: I9ade82cb89422e31ae8c5df9fff31157e88278c5
Bug:180682930
Test: erofs image can be set to input timestamp and UUID
Change-Id: I71e455350db51096e18ab5a03123500d56ebe252
Signed-off-by: Huang Jianan <huangjianan@oppo.com>
* Prerequisites: external/f2fs-tools: sload compression support
* Must work with corresponding changes in system/extras repository
* If Board config does not change, it falls back to old behavior,
i.e. no compression for the system partition
* Kernel f2fs compression support is a prerequisite if the Board config
enables the compression (see below)
* Necessary board config change
(e.g. device/<vendor>/<product>/BoardConfig-common.mk)
BOARD_SYSTEMIMAGE_FILE_SYSTEM_TYPE := f2fs
BOARD_SYSTEMIMAGE_PARTITION_RESERVED_SIZE := 200000000
BOARD_SYSTEMIMAGE_FILE_SYSTEM_COMPRESS := true
BOARD_SYSTEMIMAGE_F2FS_SLOAD_COMPRESS_FLAGS := <sload compress sub-options>
Setting BOARD_SYSTEMIMAGE_FILE_SYSTEM_COMPRESS to true enables both the
compression support when the initial empty file system be made (mkfs.f2fs)
and the compression flag (-c) when the system image files be side-loaded
by sload.
Sload compress sub-options (i.e. options other than -c) will be provided
by BOARD_SYSTEMIMAGE_F2FS_SLOAD_COMPRESS_FLAGS. If it is not given, or
is empty, the default sub-options will be used
Please refer to the sload.f2fs manual page.
Setting BOARD_SYSTEMIMAGE_FILE_SYSTEM_TYPE to f2fs is trivially necessary.
* File system table (fstab), notably the file 'fstab.hardware', should
also changed:
- The file type must be changed to f2fs
- Perhaps also other f2fs-specific options
Bug: 170918499
Test: Pixel4a userdebug build (from build id 6918751)
Signed-off-by: Robin Hsu <robinhsu@google.com>
Change-Id: Id9d67b5cb35dc806e06ff1320e89114abc996a28
If dynamic partitioning is enabled and the partition size is not set,
du will be used to calculate the partition size.
For compressed file systems, it's better to use the compressed size
to avoid wasting space.
Bug:174816929
Test: erofs image size is smaller than the original file
Change-Id: I1deda85d312c19620680531223fffcfb815e5fd4
Signed-off-by: Huang Jianan <huangjianan@oppo.com>
Modules partition is a dynamic read-write partition.
- AVB is not enabled on the partition
- OTA is file-based; see follow up CL for details
- No build prop files; in particular, no build fingerprint
- No fs_config
- No notice files; notice files are included in individual APEXes
Test: build on CF
Bug: 163543381
Change-Id: Ie397b9ec61dfd1c158450d050196024604854d4d
