Read from the new aconfig storage backing in C++ aconfig codegen, for
READ_ONLY flags. Log if there is a match with the legacy storage, or a
mismatch, or a failure.
Only enabled if instrumentation is enabled, which will be enabled by a
build flag in a follow-up CL.
Test: m && cargo t
Bug: 328444881
Change-Id: I691dfad8860d8f917e93c5d56dac19f8791de943
Restrict the set of exported symbols to those in the aconfig_storage
namespace by way of a version script. This shrinks the shared lib size
by ~75%, from ~800KB to <200KB.
Bug: 336657207
Test: m
Change-Id: I56044fe667a713cf1d94f96c992f379a5725850f
If boot_variable_file is not found locally, look for it inside
input_target_files.
Bug: 335732867
Test: generate OTA with file in target-files.zip and verify metadata
Change-Id: I2e61f50850e82a3795f2e0e1aefcaf2329a8cfb1
Simplify storage write api so that we don't need the storage records pb
file.
Bug: b/312444587
Test: atest -c
Change-Id: I7e336b1d7766983364715dae15786b91b0c0743f
Passing "*" to --products is equivalent to passing the all_named_producs
build variable
Passing "*" to --modules passes the contents of
PRODUCT_OUT/all_modules.txt
The total length of the text of all_modules can easily exceed the
maximum argument size for the OS. The proper solution is likely to call
getconf ARG_MAX, then concatenate the command to be executed to check
against the limit. Instead, the modules are processed in batches of 40k
modules. As long as module names don't get extremely long, this should
keep us under the ARG_MAX limit. In testing, using --modules "*" gets
split into two batches.
Test: build/make/tools/whichgit --modules "*"
Test: build/make/tools/whichgit --modules "*" --unused
Test: build/make/tools/whichgit --modules "*" --products "*"
Existing use-cases should remain unchanged:
TEST: build/make/tools/whichgit --modules framework
Change-Id: Ifa947daea2d439df0145e6def92637b67a8b5d22
The --unused parameter inverts the output of whichgit, reporting which
git projects are not used for a given build target.
Test: build/make/tools/whichgit --unused
Test: build/make/tools/whichgit --unused --modules framework
Existing use-cases should remain unchanged:
Test: build/make/tools/whichgit --modules framework
Change-Id: Ia4e55a5cb0331d522fed76821fe813ef98c25a67
Occasionally sourcing the script would close my overall shell when it
hit an error. By having it just be an executable script, this prevents
it from impacting a user's normal shell environment and can depend on it
always being bash, rather than whatever shell people happen to be using.
Bug: 334870672
Test: tools/check-flagged-apis/check-flagged-apis.sh
Change-Id: Ic46cb4fefdea8d51be018d4f7a92b0d9ca7e57b3
Update the map in soong instead of that in bazel
Bug: 315353489
Test: manual
Merged-In: I3a51be1119109d2a6cf3959bdc8a48192872580f
Change-Id: Ia61a17be4a7d85c00d2e9758ea2409019d451dfd
No need for cache filtering based on container anymore. This is now
performed by build system automatically. The caches feed into storage
generation command are automatically filtered based on owning
container.
Bug: b/312444587
Test: atest aconfig.test
Change-Id: I44f3ac03d50b77a191a82ff7ed4d02766012492d
previously we store three bits per flag: is sticky, is read write, has
override. so when a local override arrives, is sticky bit as well has
override bit are set to true. this becomes problematic when the local
override is removed. we are not sure if we should remove has override
bit as well (server override could still be in place).
therefore restructuring the flag info for each flag into new three bits:
is read write, has server override and has local override. so now server
override and local override bits are independent from each other.
Bug: b/312444587
Test: atest -c
Change-Id: I03b2bb7312480773236c95f2b39f2589fee41924
Create one library for reading protos from all containers, instead of
having numerous libraries perform the same logic. For Java, we will
create a similar library reusing the same
partition_aconfig_flags_paths.txt.
Bug: 324436145
Test: adb shell aflags list # Confirm that various containers appear
Change-Id: I924e281a50f9a609e1c07c03267eebe3dce52752
Add a script to make it easier to check-flagged-apis for the public APIs
and the three flavours of @FlaggedApi.
Bug: 334870672
Test: lunch sdk-next-eng && source check-flagged-apis.sh
Change-Id: I8b5e2642ade84560c5c61ae49d8c0dcdedb841ca
Replace the current unit test runner DeviceJUnit4ClassRunner with JUnit4
and replace the (larger) dependency tradefed with the (smaller)
dependency junit.
This has no impact other than minimizing the unit test static_libs.
Bug: 334870672
Test: atest --host check-flagged-apis-test
Change-Id: I0520ab0feeea5ea2ed15905136ba2647f86162cb
Teach check-flagged-apis to cross-check the data from its three input
sources. This allows the tool to detect
- @FlaggedApi references to non-existent flags
- @FlaggedApi APIs present in the build artifacts even though the flag
is disabled
- @FlaggedApi APIs not present in the build artifacts even though the
flag is enabled
By passing in different sources, the tool can detect these errors for
any of the API surfaces (public, @SystemApi(MODULE_LIBRARIES), etc).
Note: the tool assumes that a disabled flag means that the @FlaggedApi
should not be present in the build output. This is currently true, but
won't be once metalava starts reverting @FlaggedApis to their previous
SDK snapshot.
Bug: 334870672
Test: atest --host check-flagged-apis-test
Test: check-flagged-apis --api-signature out/target/product/mainline_x86/obj/ETC/frameworks-base-api-current.txt_intermediates/frameworks-base-api-current.txt --flag-values out/soong/.intermediates/all_aconfig_declarations.pb --api-versions out/dist/data/api-versions.xml
Change-Id: I790234865f831af7d45895def14d1d6740365622
Teach check-flagged-apis to parse API versions XML; this represents the
APIs after metalava has processed the source and kept APIs as is, or
reverted them to the previous SDK snapshot, according to their
@FlaggedApi flags.
As with the API signature parser, limit support to fields to keep things
simple; support for classes and methods will be added in later CLs.
Note: `m sdk dist` will generate an API versions XML file.
Bug: 334870672
Test: atest --host check-flagged-apis-test
Test: check-flagged-apis --api-signature out/target/product/mainline_x86/obj/ETC/frameworks-base-api-current.txt_intermediates/frameworks-base-api-current.txt --flag-values out/soong/.intermediates/all_aconfig_declarations.pb --api-versions out/dist/data/api-versions.xml
Change-Id: I779a0d0cdb8a50536d3fc8d517fa38ba4b0dcd1c
Teach check-flagged-apis to parse the parsed_flags protobuf generated by
aconfig.
Note: `m all_aconfig_declarations` generates a protobuf file that
contains all info about all flags.
Bug: 334870672
Test: atest --host check-flagged-apis-test
Test: check-flagged-apis --api-signature out/target/product/mainline_x86/obj/ETC/frameworks-base-api-current.txt_intermediates/frameworks-base-api-current.txt --flag-values out/soong/.intermediates/all_aconfig_declarations.pb
Change-Id: I397b32ae2a373b429ef6ce22e0a06a0f15202b91
Teach check-flagged-apis to extract flagged APIs from API signature files.
To keep things simple, only consider fields for now: support for classes
and methods will be added in a later CL.
Note: `m frameworks-base-api-current.txt` will generate an API signature
file that includes both the platform and mainline APIs.
Bug: 334870672
Test: atest --host check-flagged-apis-test
Test: check-flagged-apis --api-signature out/target/product/mainline_x86/obj/ETC/frameworks-base-api-current.txt_intermediates/frameworks-base-api-current.txt
Change-Id: Ic244b896672569f44af793796189b34c1f9d0c36
1, add a new field in storage metadata proto, for each container, there
is a local override file.
2, update write api to expose the api to map a writable file given the
specific path. the cannonical api will require a storage metadata proto,
and then find the right file path and map it.
3, minor update to make the proto lib cc_library instead of
cc_library_static
Bug: b/312444587
Test: atest -c
Change-Id: Iaf0aff44c1ca3ad4bffc5e06bb99bb276b9069c5
Add a value class to represent Flag names. We could use plain Strings
but having a dedicated class (with no overhead compared to String) makes
the intent of the code much clearer.
Bug: 334870672
Test: m check-flagged-apis && check-flagged-apis
Change-Id: Icdd4fb97d3fd49e507b7559504ea173a3dc52dea
check-flagged-apis will read contents from various sources, which use
different formats to represent the same piece information (e.g.
"class#field" or "<class><field>").
Introduce a Symbol value class to represent any API (i.e. a class, field
or method) in a unified format.
Bug: 334870672
Test: m check-flagged-apis && check-flagged-apis
Change-Id: Id9404294a87b23a9d43e5e13ce39ea5a92608e33
ro.product.first_api_level looks like the value we actually want to read
to since it inherits from $PRODUCT_SHIPPING_API_LEVEL. Other variables
we attempted to look at were giving other values (e.g 202504 for pixel
watch)
Bug: 331202590
Test: th
Change-Id: Ied3da247cba45611132db6c9e3cdefb447891f96
Use clikt as the command line options parser library.
Bug: 334870672
Test: m check-flagged-apis && check-flagged-apis
Change-Id: I7c406456b00e29293294dcdbef411d2543a1e8d5
Add a new CLI to verify that the build artifacts contain the right set
of @FlaggedApi APIs, based on the value of the corresponding aconfig
flag.
This CLI will act as an end-to-end test of Soong and metalava.
This CL only adds the project scaffolding; later CLs will add the
implementation.
Bug: 334870672
Test: m check-flagged-apis && check-flagged-apis
Change-Id: Ib00653f2a549217da2b0058867c711f35efd5760
The order of items in `partitions` map is non-deterministic. To make
sure that builds are hermetic, sort the map before use.
Test: th
Bug: 332550989
Change-Id: I76c7037ef9f03904899339813819fcf985a7c893
Currently flag info query api is called get_boolean_flag_attribute, in
this change, we switched it over to a flag value type generic
implementation get_flag_attribute. So in the future we want to add more flag value types, this api can stay the same.
Bug: b/312444587
Test: atest -c
Change-Id: I2b272f3fa3cb1d0edc8b77a44bf37752ffe95925
Add keys to the dict `partitions` in serial to have a deterministic
insertion order.
This guarantees the generated vbmeta.img is always the same on the same
input, fixing image consistency issues.
Reference:
https://android-review.googlesource.com/c/platform/build/+/2585397
Bug: 332550989
Test: Generate vbmeta several times, confirm identical result.
Change-Id: I39da5844045f497fdc89e6477a7e32ddacb70764
* changes:
aconfig: update storage write api
aconfig: update storage read api
aconfig: update storage file creation to allow storage files to store flag index instead of byte offset
aconfig: update storage file to store flag index instead of file byte offset
supporting negative compression levels in ota_from_target_files in the
format of zstd,-$compression_level
Test: ota_from_target_files
Change-Id: If2ca592829f10e7a0226cb41680cb47223d17c71
Now we are storing flag index instead of flag byte offset in storage
files, need to update storage write api accordingly.
Bug: b/321077378
Test: atest -c
Change-Id: I67e55287a242df0af1925ede886f8c087d343c1f
With the update to store flag index instead of flag byte offset in
storage file. We need to update flag read api accordingly.
Bug: b/321077378
Test: atest -c
Change-Id: Iccbf939fc5b7fe0ce0f2c33f7e3aa403c268120a
Previously, in package.map, we store the byte offset of the first flag
in a flag package in a flag value file. In flag.map, we store the within
package flag byte offset. Once we know the total offset, we can locate
the flag in the flag value file. However, this offset may not work for
flag info file. Currently we only have boolean flags. Each boolean flag
value and info takes up 1 byte in flag value and flag info file. So the
byte offsets are the same. But in the future, when we add a new flag
value type, or add more contents to flag info file. Then the byte offset
for flag value file and flag info file may not be same.
Instead, we should store index rather than offset. The package map
should store the index of the first flag in this package. The flag map
stores the within package index. Then flag value read api and flag info
read api can then infer the byte offset independently.
Bug: b/321077378
Test: atest -c
Change-Id: Iad89776d5369715ba2d8faaea18261ac3271f13b
rust_ffi_static right now will export excessive symbols, this lead to a
duplication of symbols when aconfig links against both
libaconfig_storage_file_cc and libaconfig_storage_read_api_cc. This is
because aconfig_storage_read_api crate depends on aconfig_storage_file
crate. To solve this problem, move create flag info file api to
aconfig_storage_write_api crate which is parallel to
aconfig_storage_read_api crate.
Bug: b/321077378
Test: atest aconfig_storage_file.test; atest
aconfig_storage_read_api.test; atest aconfig_storage_write_api.test
Change-Id: Ibbb50193a2da82d52ccbb4087c8e3fb9f320805f
exported mode
The exported mode library will be dropped into google3, and there isn't
the library for this annotation.
Test: atest aconfig.test
Bug: n/a
Change-Id: I318d1c0faf232b22f42587b8ea2f976173c8554a
Add new folks working on finalization, remove folks no longer working on
it.
Also, sort the entires alphabetically.
Bug: 323940469
Test: N/A
Change-Id: Ib8f6c3f485a06e3d1269e4142317174ae7ac276b
Previously, three seperate crates create their own test baseline which
are nearly identical. This is inconvinent when updating the baselines.
All three locations needs to be touched. This cl merges functions to
create test storage file base struct into aconfig_storage_file crate
Bug: b/321077378
Test: atest aconfig.test
Change-Id: Ic97437f856763347a11de8b449592292014bcce3
Check that device is shipped supporting v3 in order to enable it.
Otherwise fallback.
Bug: 331202590
Test: th
Change-Id: I218fbaa6a044420af39be70b055da5ffbfced158
Flag type is encoded as a u16 in flag table file. Previously we are
assinging value of 1 for flag type to indicate that they are all boolean
flags. In this change, we formalize the flag type designation with an
enum.
ReadWriteBoolean -> 0
ReadOnlyBoolean -> 1
FixedReadOnlyBoolean -> 2
Bug: b/321077378
Test: atest aconfg.test; atest aconfig_storage_file.test
Change-Id: I114818435b44f1e82a611f0509787993a5a8e70e
These two TODO tags were improperly formatted, which means automation
will not alert anyone if the ticket is closed.
Bug: 327420679
Test: git diff
Change-Id: I99c8215fe3e24cdf3fa7be517291964cc5eb1ecf
Signed-off-by: Lucas Henneman <henneman@google.com>
Add a mode that lets the user use the new aconfig storage.
Bug: 324436145
Test: adb shell aflags list --use-new-storage
Change-Id: I97d7a229a64c6ef1aea844281298ce5449b02570
Flag info file is another memory mapped file to store the file
information such as if current flag has been staged or not, is it sticky
and etc. This file will be created by aconfig storage daemon during
container initialization.
Bug: b/321077378
Test: atest aconfig_storage_file.test
Change-Id: I8f70e04c74b5770f89dbe981d9f03c787ea5eeed
We need the aconfig APIs to be available on every mainline module, so
set the min_sdk_version as low as possible.
Bug: 328444881
Test: m
Change-Id: I02e569d510a30f4809d4a82f7630326b0b30f263
This reverts commit 54101665ee.
Reason for revert: Fixed build breakage
Bug: 322090587
Change-Id: I065a187bf5863cbb381b4f3308aae7b7200ffa6d
Test: compare build.prop before and after
Test: build multiple times and see build.prop isn't rebuilt
Logging the apk_name will improve debugging process if
`GetMinSdkVersion` returns an exception.
Test: None
Change-Id: I3ddf602d92701dcaaa4821fe42a76a870a721a11
Test: Sign target files package where
SYSTEM/product/media/bootanimation-dark.zip is a symlink to
bootanimation.zip.
Change-Id: I4648c3c39c094cb090cbe337c566c3e9ad894691
In the case that a vendor is on s build (which doesn't have snapuserd),
and is takinga v3 -> v3 upgrade, we want to fallback to v2 vabc ota.
Test: th
Change-Id: Iba259b139b9b423ac77d82e14a113cf35fbfea64
Allow override for vabc_cow version if specified from the command line.
If OPTIONS.vabc_cow_version is already defined, then we don't want to
modify this value since it was defined by --vabc_compression_param=
Test: th
Change-Id: I57b8129faf5caf9de1dc8ad96e7570214a9f3313
To build system.img in Soong, we need all artifacts including
build.prop. This fully migrates buildinfo.prop file into Soong as a
first step to build build.prop on Soong.
Bug: 322090587
Test: compare build.prop before and after
Test: build multiple times and see build.prop isn't rebuilt
Change-Id: I0d5aab21a825cfe22f97a6834209f9df196625d9
Previously we are ensuring that the public rust api to get mapped file
is safe as we are ensuring that the file being mapped has a permission
of 444. However, a file permission of 444 is not possible due to build
system needs to make file 644 while creating img files. Thus need to make the rust api to get mapped file unsafe.
In reality, this should have no impact. Because, even though the storage
files have a file permission of 644, they are on a RO partition like
system. So the files are not writable anyway. This is true for all the
containers (platform partitions and mainline modules) we know so far.
Bug: b/321077378
Test: atest aconfig_storage_read_api.test; atest
aconfig_storage_read_api.test.rust; atest
aconfig_storage_read_api.test.cpp
Change-Id: I643fe191e697a524e2303a32750f04c268f408fd
We have moved the storage records metadata pb file to the boot dir.
Bug: b/321077378
Test: atest aconfig_storage_read_api.test
Change-Id: Ic5bd89052e311ec5b1993ba1779d49be8451caa2
If devkeys are present, we don't need to specify a serialno. In this
case we should conditionally add serial no to our metadata
Test: create_brick_ota
Change-Id: I9c93aa0f9ec5752693a9ddde5b76be4c12d6fc3b
Language services builds are usually executed in a non-building state of the codebase as the user is actively editing the code.
This change ensures we can build as many dependencies as possible for language services by ninja.
Note that this doesn't change the error-limit for soong itself, i.e. we can still fail-early if we encounter errors while generating ninja files.
Change-Id: I594e2ae4591d2e75db082486f28f705833ceb4a8
Now the read api lib no longer owns the memory mapped file. The rust lib
aconfig_storage_read_api api to get memory mapped file just returns an
Mmap rust struct. However cxx does not support Mmap data type, so we
cannot export the api to get memory mapped file from rust to c++. This
means we have to implement c++ apis to get memory mapped files.
This change is basically implementing c++ api to get memory mapped files,
mirroring the rust implementation. In addition, the integration test is
also updated to lock down the new c++ api implementation.
Bug: b/321077378
Test: atest aconfig_storage_read_api.test.cpp
Change-Id: I5e93a220cdd05cc0e5b0eb441284ac3e9dc34da7
Right now, aconfig_storage_read_api rust lib would own mapped files. So
individual flag lib does not need to. The original intent is that all
these flag lib can dynamically link against this lib. So say we have
different variants of flag lib for one flag package, there is just one
unique memory mapped file. In addition, the exposed rust apis hides the
memory mapped files. So when exporting these apis to c++/java, we don't
have to implement memory mapping in c++/java flag read api lib.
However, this design has its own downside. The biggest being that these
memory mapped files are stored in a mutable static hash map. For rust,
it must be protected by a mutex lock. The real effect on the lock is
mostly minimal as it immediately unlocks when the clone of memory
mapping is returned. But still we could get more performance out of it
in exchange of having more c++/java side implementation of mapping
files. (Note, apis that interpret mapped files are still exported by
Rust, so no c++/java implementation needed).
In this change, aconfig_storage_read_api give away its ownership of
memory mapped file in its api to get memory mapped file. The memory
mapped file is supposed to be owned by generated flag lib. By doing so,
no more mutex lock is needed. Also, since each flag lib has one flag
package, which maps to a single owning container. The flag lib can have
the memory mapped files as non mutable static variables. So no mutex
lock penalty on flag lib side.
Bug: b/321077378
Test: atest aconfig_storage_read_api.test; atest
aconfig_storage_read_api.test.rust
Change-Id: I29060bae90db6f7d296252ed73f7e312143f4b7e
Lines now have a new column, and look like this:
com.android.example.flag1 enabled - server read-write system
com.android.example.flag2 enabled (->disabled) server read-write system
The dash represents no change on boot, the (->value) represents a change
to value on the next boot.
Test: adb shell device_config put staged multitasking*com.android.wm.shell.enable_taskbar_navbar_unification true && adb shell aflags list | grep navbar_unification
Bug: 324436145
Change-Id: I022460bc69fcb2ccd9c6db8f060fcbd0337d1ea6
1, add storage file type into storage files serialization
2, update aconfig_storage_read_api integration tests to not rely on
soong to produce read only storage files. Instead, have the test process
create temp read only storage file copies.
Bug: b/312444587
Test: atest aconfig.test; atest.aconfig_storage_file.test
Change-Id: I1607fb000b48e8acffc966b16a5136578911ab3e
aconfig_storage_write_api crate is the lib to be used by aconfig storage
daemon to update flag value at a given offset.
Note that mmap api is unsafe from memmap2 crate. This is due to the
possibility of other code write to the file after mmaping the file into
memory. Therefore the api to write to storage value file is also marked
as unsafe. In reality, the persistent storage value file is protected by
SELinux policy to allow write access only to aconfig storage daemon. In
addition, aconfig storage daemon is single threaded. So at any time,
only one thread is writing to a storage file, and only thru the mmapped
file in memory (not thru direct file write). So it would safe for
storage daemon to call this api.
Bug: b/312444587
Test: m libaconfig_storage_write_api; atest
aconfig_storage_write_api.test
Change-Id: I93cffea0d94e4c40e711d809418c0b18b6d9bfe1