Commit graph

8151 commits

Author SHA1 Message Date
Treehugger Robot
ecf635f507 Merge "Adding option to disable fastboot_info" 2023-06-03 01:56:41 +00:00
Dennis Shen
3b0ca61d36 Merge "aconfig: update rust code gen to use libflags_rust" 2023-06-02 15:13:40 +00:00
Daniel Zheng
ee10d07057 Adding option to disable fastboot_info
Fastboot_info can be disabled if use_fastboot_info is set to false.
Adding this flag as fastboot-info.txt is currently broken

Test: m updatepackage -> inspect contents
Bug: 284263071
Change-Id: I3e0ca13968ba9747cc39284ea6798981d22ad5e5
2023-06-01 14:21:58 -07:00
Kelvin Zhang
89b87f6aa9 Open ab_partitions.txt with just read access
We don't actually need write permission, so going with least privilege
principle. We have observed some mysterious permission denied errors on server environments. Without detailed logs or access to the server it's hard to pinpoint what the root cause is. This is an attempt/hypothesis to fix the permission denied error.

Test: th
Bug: 283033491
Change-Id: I52dc360d593aab57c749109994bf3e1e3625d0ce
2023-06-01 17:28:25 +00:00
Dennis Shen
6461673140 aconfig: update rust code gen to use libflags_rust
Bug: b/284096062, b/279483360
Test: atest aconfig.test
Change-Id: I4c96f35807de5cb2a745ba1c653513d9b22a5013
2023-05-31 14:35:59 +00:00
Mårten Kongstad
00cf045c85 aconfig: restrict valid namespace and flag names
The namespace and flag names will be used as identifiers in the
auto-generated code. Place restrictions on what constitutes a valid
name.

Valid identifiers are those that match /[a-z][a-z0-9_]/. aconfig
explicitly does not implement any automatic translation to make names
valid identifiers: this sidesteps potential conflicts such as "foo.bar"
and "foo_bar" mapping to the same name if dots were translated to
underscores.

Bug: b/284252015
Test: atest aconfig.test
Change-Id: I38d005a74311e5829e540063404d1565071e6e96
2023-05-26 17:09:10 +02:00
Mårten Kongstad
aa977fb083 Merge changes from topic "aconfig-create-rust-lib"
* changes:
  aconfig: first iteration of Rust codegen
  aconfig: improve dump --format=debug output
2023-05-26 06:00:47 +00:00
Mårten Kongstad
f73b963283 aconfig: first iteration of Rust codegen
Add a new `create-rust-lib` command to generate Rust code. The output is
a src/lib.rs file; the build system is assumed to set the generated
crate's name.

For READ_ONLY flags, the generated code returns a hard-coded true or false.

For READ_WRITE flags, the generated code reaches out to DeviceConfig via
the cc_library server_configurable_flags via the
libprofcollect_libflags_rust Rust bindings. The build system is assumed
to add this to the generated crate's dependencies.

Note: libprofcollect_libflags_rust seems generic enough that it should
be moved to an official Rust wrapper for server_configurable_flags. This
is tracked in b/284096062.

Summary of module the built system is assumed to wrap the auto-generated
code in:

  rust_library {
      name: "lib<namespace>_rs",
      crate_name: "<namespace>_rs",
      edition: "2021",
      clippy_lints: "none",
      no_stdlibs: true,
      lints: "none",
      srcs: ["src/lib.rs"],
      rustlibs: [
          "libprofcollect_libflags_rust",
      ],
  }

Also add a set of test input to be used in the unit tests for a more
coherent test strategy. A follow-up CL will migrate the code in
commands.rs, codegen_java.rs and codegen_cpp.rs.

Bug: 279483360
Bug: 283907905
Test: atest aconfig.test
Test: manual: create cache from files in testdata, create rust lib, add to module template above, verify the module builds
Change-Id: I02606aa3686eda921116e33f7e2df8fd1156a7aa
2023-05-25 16:18:59 +02:00
Treehugger Robot
b3adc24301 Merge "Fix merge_target_files in GRF case" 2023-05-25 10:46:32 +00:00
Mårten Kongstad
993111f91c aconfig: improve dump --format=debug output
Use Rust's {:#?} formatter for more readable output.

Bug: 279485059
Test: atest aconfig.test
Change-Id: I127f413e3d7aebfba96cad1dd58d9e261dd613a4
2023-05-24 14:56:45 +02:00
Treehugger Robot
b4406d06f0 Merge "Unsparse images before generating OTA" 2023-05-23 20:51:10 +00:00
Treehugger Robot
8485145bd7 Merge "Add 16K artifacts to PREBUILT_IMAGES/ dir of target_files" 2023-05-23 02:36:17 +00:00
Kelvin Zhang
2268091521 Unsparse images before generating OTA
Test: th
Bug: 283172692
Change-Id: Ie6d3dc704fd9a8c107e2888222e4c2bf804dad3e
2023-05-22 10:22:59 -07:00
Mårten Kongstad
2f95444281 aconfig: sort items in cache by name
Introduce a builder pattern for constructing a cache from flag
declarations and flag values. Teach the builder to sort the flags by
name as the last step. This will ensure consistent dump output
regardless of the order flags are specified in the input files.

Bug: 279485059
Test: atest aconfig.test
Change-Id: Icdd62f51fa3761a469663f17581a83d9909e9ffe
2023-05-22 10:01:05 +02:00
Mårten Kongstad
af677038b3 aconfig: dump: support multiple caches
Teach `aconfig dump` to read multiple caches at the same time.

A cache represents a single namespace, and the Android build creates
multiple caches for different namespaces. By passing in all those cache
files to `aconfig dump`, aconfig will create a complete overview of all
flags in the Android tree.

Caches are traversed in order of namespace to produce the same output
regardless of the order the cache files are given on the command line.
If two caches use the same namespace, their order with respect to each
other is undefined.

Bug: 279485059
Test: atest aconfig.test
Change-Id: I54c3950bbb7b2be7d96c8928e78ae83aa626c2e2
2023-05-22 09:57:48 +02:00
Kelvin Zhang
decee4adb7 Add 16K artifacts to PREBUILT_IMAGES/ dir of target_files
Test: th
Bug: 283506035
Change-Id: I9be3982b5a4324ec839937d10c33266c7d27c525
2023-05-19 14:31:00 -07:00
Treehugger Robot
4266df817d Merge "aconfig: Add first iteration of cpp codegen to aconfig" 2023-05-17 19:37:28 +00:00
Wei Li
eb035dc89f Merge "Some changes to support SBOM generation for b build unbundled APEXs." 2023-05-17 19:19:47 +00:00
Dennis Shen
1dc9ad4662 aconfig: Add first iteration of cpp codegen to aconfig
The general idea to reuse java codegen's very neat tiny template idea.
For generated cpp code, it is in the form of a collection of classes
inside a namespace. The reason we choose a collection of classes rather
than a collection of static functions is because gmock test technology
only supports mocking virtual method.

Bug: b/279483801
Test: atest aconfig.test
Change-Id: I9ba00667437ff7c3e147ff2828171fc95528bebf
2023-05-17 14:57:58 +00:00
Wei Li
16e7aa3c2e Add upstream package of a prebuilt fork package, which will have the package information from the METADATA file.
Test: CIs
Test: lunch barbet-user && m sbom
Change-Id: Ic8eb42c369de8c94c7977b9631ff4b9084dfef01
2023-05-17 01:38:05 +00:00
Joe Onorato
7833547153 Merge "Use the namepace as the java package name in aconfig and call the class "Flags"" 2023-05-16 23:13:52 +00:00
Treehugger Robot
fce28c1b97 Merge "Allow source build to be an extracted target_files" 2023-05-16 22:56:29 +00:00
Treehugger Robot
bb404f2b27 Merge "Support custom payload signer in merge_ota" 2023-05-16 21:39:58 +00:00
Joe Onorato
0c4ef0f88c Use the namepace as the java package name in aconfig and call the class "Flags"
Test: aconfig.test
Change-Id: I15032262aad2c80d1ec1bd8fa81664a846b307ec
2023-05-16 14:27:25 -07:00
Mårten Kongstad
e17ba5f423 aconfig: cache: reject empty namespace and name fields
Add invariant to struct Cache: all flag namespace and name fields added
to the cache are required to be non-empty strings.

Bug: 279485059
Test: atest aconfig.test
Change-Id: I5ff34ec8feccc19e52241d4221fc87699518f3ff
2023-05-16 12:52:43 +02:00
Mårten Kongstad
6b9e382eed aconfig: remove calls to unwrap (outside tests)
Do not call unwrap outside tests: replace existing uses with Result
return values or infallible alternatives.

Bug: 279485059
Test: atest aconfig.test
Change-Id: Ie5919b704b23a0f96bbef84ffbe9270d667cecd8
2023-05-16 11:19:58 +02:00
Mårten Kongstad
ba94e6a6b2 aconfig: rename enum Format -> enum DumpFormat
Rename enum Format to enum DumpFormat to make it more apparent what it
refers to.

Bug: 279485059
Test: atest aconfig.test
Change-Id: I869be020b69618b036fa05247f155d9e35ff85e2
2023-05-16 11:00:16 +02:00
Satoshi Futenma
1f93ce20c5 Support custom payload signer in merge_ota
Add options to handle custom payload signer as it is required in
merge_ota.py as well if the original OTA packages are signed by
the signer.

If input is only one OTA, clone apex_info.pb to the target.

Use common.ZipWriteStr instead of zipfile.writestr, this ensures
that the same permission for the entries as done by
ota_from_target_files.

Bug: 282189563
Test: manual. pass single OTA to merge_ota, with same signing
    parameters as originally used. Confirm that output zip is
    binary identical to input.
Change-Id: I3b926b8cd69bc74dff6ccf8b5ccc72bedffcac6f
2023-05-16 10:21:05 +02:00
Kelvin Zhang
b97416e09e Allow source build to be an extracted target_files
common.LoadInfoDict() already supports loading from extracted
directories, just use it.

Test: generate an incremental OTA where both inputs are directories
Bug: 227848550
Bug: 277028723
Change-Id: Iedba831bb4d65d971df6b2ac95279e3234a02e2f
2023-05-15 11:13:26 -07:00
Mårten Kongstad
e66b89f635 aconfig: use bail! and ensure! where applicable
The Android Rust style guide recommends anyhow's bail! and ensure!
macros where possible. Update the code accordingly.

Bug: 279485059
Test: atest aconfig.test
Change-Id: I7db7cc2d8ec15e1d3450657ad000e879adb19e08
2023-05-15 11:10:12 +02:00
Mårten Kongstad
71f9dabe47 Merge changes from topic "aconfig-part-5"
* changes:
  aconfig: follow Java conventions for Java file paths
  aconfig: separate flag declarations and flag values
2023-05-12 12:21:02 +00:00
Treehugger Robot
8d5d2257d9 Merge "parallelize add_partition_calls only if output_zip is None" 2023-05-12 10:56:23 +00:00
Mårten Kongstad
d42eeeba3d aconfig: follow Java conventions for Java file paths
Update codegen_java to write the generated Java file(s) to
"java/package/File.java" instead of just "File.java".

Also generalize codegen_java::GeneratedFile to commands::OutputFile in
preparation for the upcoming C++ and Rust codegen.

Also change Java package name to 'com.android.internal.aconfig'.

Bug: 279485059
Test: atest aconfig.test
Change-Id: I13978697e35010fe6be8637aa495d4b852dbed7e
2023-05-12 10:01:00 +02:00
Mårten Kongstad
fa23d2993b aconfig: separate flag declarations and flag values
Simplify how aconfig configurations work: remove the ability to set flag
values based on build-id.

The aconfig files now some in two flavours:

  - flag declaration files: introduce new flags; aconfig will assign the
    flags a hard-coded default value (disabled, read-write)

  - flag value files: assign flags new values

`aconfig create-cache` expects flags to be declared exactly once, and
for their values to be reassigned zero or more times.

The flag value files are identical what used to be called override
files.

Also, remove the now obsolete build-id parameter: this was used to
calculate default values before applying overrides, and is no longer
needed.

Also rename a few more structs and functions to be closer to the .proto
names. This will make it easier to use the generated proto structs
directly, and get rid of the hand-crafter wrappers.

Bug: 279485059
Test: atest aconfig.test
Change-Id: I7bf881338b0567f932099ce419cac457abbe8df8
2023-05-12 08:50:49 +02:00
Jeongik Cha
fa9090946e parallelize add_partition_calls only if output_zip is None
if output_zip isn't None, writing into zip file in parallel is not
thread-safe.

Bug: 281960217
Test: m dist
Change-Id: I10d68a4bb779cee244f40410ec95d38ca6040306
2023-05-12 15:14:58 +09:00
Treehugger Robot
96b51b54b6 Merge "parallelize add_partition_calls" 2023-05-12 04:54:11 +00:00
Daniel Zheng
717591dd4e Merge "Add fastboot-info to updatepackage" 2023-05-12 04:31:55 +00:00
Jeongik Cha
03d014a03f parallelize add_partition_calls
it made total time equal to the longest image build

1m10s->30s in local build

Bug: 281960217
Test: m dist
Change-Id: I13d4f45d9b46b39292a014e3b4e1913365d89b7a
2023-05-12 08:43:44 +09:00
Wei Li
fd7e6517d3 Some changes to support SBOM generation for b build unbundled APEXs.
1) Use output file path of installed files in build system since there is no PRODUCT_OUT in Bazel
2) Use CONTAINS to describe the relationship between a APEX and files it contains
3) Generate SBOM of APEXs, which is similar to SBOM of products

Bug: 275472038
Test: CIs
Change-Id: I41622366e5e6ed9dc78cca7bc7bb69a1f8f9bd9f
2023-05-11 13:58:37 -07:00
Cole Faust
99d89154f1 Merge "Allow importing starlark code in makefiles" 2023-05-11 17:04:46 +00:00
Zhi Dou
eb74489b3a aconfig: Add codegen for java
Add codegen for java skeleton

Bug: 279485059
Test: atest aconfig.test
Change-Id: Ia0481cec9c2e137e88e9a77d1b82412529b64adc
2023-05-11 03:08:53 +00:00
Treehugger Robot
ef0a2c21dd Merge changes from topic "aconfig-part-4"
* changes:
  aconfig: introduce namespace, rename proto messages
  aconfig: add dump protobuf format
2023-05-10 19:22:00 +00:00
Treehugger Robot
b1a7ba8735 Merge "Improve finalization perfomance by building only current version." 2023-05-10 15:19:12 +00:00
Mårten Kongstad
3095078981 aconfig: introduce namespace, rename proto messages
Flags belong to a namespace. Update the proto files to reflect this.

Config files can only refer to a single namespace. Override files can
refer to multiple namespaces; an override directive for a flag in a
different namespace than the one represented by the cache will be
silently ignored.

Rename the proto messages to make it more clear what they are. Propagate
this change through the wrappers in aconfig.rs and the rest of the code.

Also, settle on Tracepoint instead of TracePoint.

Bug: 279485059
Test: atest aconfig.test
Change-Id: I16e69dd14687bc498b2ba89d6a35879459903801
2023-05-10 17:17:20 +02:00
Kelvin Zhang
ea70695d92 Merge "Fix non-AB ota generation for directories" 2023-05-10 04:07:09 +00:00
Kelvin Zhang
b036007592 Fix non-AB ota generation for directories
Test: ota_from_target_files -v --block yukawa-target_files-10097950.zip
test.zip

Change-Id: I4291010d66cc28bdb6b6b334cdbed06daf3de837
2023-05-09 20:30:53 -07:00
Alex Buynytskyy
aac2a187d4 Improve finalization perfomance by building only current version.
Bug: 275409981
Test: run locally
Change-Id: If531178a17792166726ff7ed230592c392eeb899
2023-05-10 03:10:32 +00:00
Cole Faust
c63ce1ab58 Allow importing starlark code in makefiles
Adds a new `$(call run-starlark,my/starlark/file.bzl)` function that
will run the starlark file and set all the variables in the
variables_to_export_to_make dictionary as make variables.

Fixes: 280685526
Test: m nothing repeatedly causes no ninja regeneration, but touching all_versions.bzl does. go test, ./out/rbcrun -mode=rbc ./build/make/tests/run.rbc
Change-Id: Ic72e18dd28dba8233ba2dfb658b5d03ccece1bfd
2023-05-09 15:12:58 -07:00
Daniel Zheng
380cb8065a Add fastboot-info to updatepackage
Modifying img_from_target_files to also add fastboot-info to
updatepackage

Test: m updatepackage
Bug: 194686221
Change-Id: I2e08c4269f0d83417b9d7079633bc28796d1cdd6
2023-05-09 13:44:22 -07:00
Mårten Kongstad
a102909e09 aconfig: add dump protobuf format
Introduce a new protobuf format to represent the all flags parsed by
aconfig. This data in this new format is similar to that of the internal
cache object, but the protobuf is a public API for other tools to
consume and any changes to the proto spec must be backwards compatible.

When aconfig has matured more, Cache can potentially be rewritten to
work with proto structs directly, removing some of the hand-written
wrapper structs and the logic to convert to and from the proto structs.
At this point, the intermediate json format can be replaced by the
protobuf dump.

Also, teach `aconfig dump` to accept an --out <file> argument (default:
stdout).

Also, teach `aconfig dump` to read more than once cache file.

Note: the new protobuf fields refer to existing fields. It would make
sense to split the .proto file in one for input and one for output
formats, and import the common messages, but the Android build system
and cargo will need different import paths. Keep the definitions in the
same file to circumvent this problem.

Bug: 279485059
Test: atest aconfig.test
Change-Id: I55ee4a52c0fb3369d91d61406867ae03a15805c3
2023-05-09 15:27:42 +02:00