Commit graph

25 commits

Author SHA1 Message Date
Colin Cross
c85750bfe3 Use generics for DepSets
Use Go's generics for DepSets so they don't require a type-specific
wrapper and reflection.

Test: depsets_test.go
Change-Id: I22ba0b7d680d37d2cd05230b0f560d166c4dd20b
2023-06-22 08:18:33 -07:00
Peter Collingbourne
ff56c0124f cuttlefish: Stop building the CVD host package tarball by default.
This change causes us to stop building the CVD host package during
the default `m` build, as this is one of the most time consuming steps
of incremental builds, taking around a minute on my machine. Instead,
we build a directory containing the host package contents. A companion
change to acloud teaches it to upload the directory if it exists.

The host package tarball will continue to be built with `m dist` and `m
hosttar`. When building the host package, just create it from the package
contents directory instead of going through a zip file as we were before.

Bug: 273862156
Change-Id: Id7ab47cd5a2e96d073fcc21ed82d3719119df6b9
2023-03-30 16:44:28 -07:00
Cole Faust
18994c73f1 Replace SortedStringKeys with SortedKeys
Now that we have generics.

Bug: 193460475
Test: presubmits
Change-Id: I1594fd8feb505175d5c09c03ef397e5ffd5b09cb
2023-02-28 16:51:32 -08:00
Cole Faust
909d237c84 Revert "Dynamically assemble the list of python launcher shared libs for precompiling"
This reverts commit aa5b377ee9.

Reason for revert: this broke musl builds by removing the install dependency from python binaries to the launcher shared libraries: https://android-build.googleplex.com/builds/submitted/9600042/unit_tests_x86_64_musl/latest/view/logs/build_error.log

Change-Id: I4e13f333f287831ba13724f13ab2be789a92b476
2023-02-13 23:17:40 +00:00
Cole Faust
aa5b377ee9 Dynamically assemble the list of python launcher shared libs for precompiling
Still fall back to the hardcoded list for what to put in LOCAL_SHARED_LIBRARIES,
because I couldn't figure out a good way to get those modules at the moment.

Bug: 259718110
Test: Presubmits
Change-Id: Ia992ffea8819d8c67280b08bc45505f77af09de0
2023-02-02 13:47:56 -08:00
Jooyung Han
a883428293 Move GatherPackagingSpecs out of CopyDepsToZip
This gives a PackageModule a chance to filter/customize the contents of
resulting package.

Bug: 225121718
Test: m (no changes)
Change-Id: I45505e8234dff42201dc40d4f038e7b08eea89f0
2022-03-28 14:29:14 +09:00
Jooyung Han
99c5fe6a38 Keep 'partition' in PackagingSpec
Add a property 'partition' to PackagingSpec and fill it when
PackagingSpec is created.

This will enable PackageModule to filter PackgingSpec based on
partition. (Will be used in a follow-up change)

Bug: 225121718
Test: m (nothing changes)
Change-Id: I7657edd380e215df66bae57d8b07dd4181841fdf
2022-03-28 14:29:14 +09:00
Dan Willemsen
9fe1410213 Add android_sdk_repo_host to build platform-tools&build-tools
The Android SDK has been built with Make up until now, monolithically,
then split up into several sdk-repo zip files for different purposes.
The Mac and Windows SDKs really only need to contain the platform-tools
and build-tools pieces, but due to this monolithic sdk zip, we had to
build the whole SDK first.

This adds an `android_sdk_repo_host` module that can build these
platform-tools and build-tools zips.

Bug: 187222815
Change-Id: I55809e1d7447dd65e22461f921b2b8abb6d5f822
2021-10-13 01:25:02 -07:00
Jooyung Han
df09d173ca Generate linker config with packaged items.
Previously, we gather all deps without checking if a dep is actually one
of packaged items (CopyDepsToZip contents)

For example, following items shouldn't be listed.

provideLibs: "platform-mainline-sdk_libartpalette-system@current.so"
provideLibs: "platform-mainline-sdk_liblog@current.so"
provideLibs: "runtime-module-sdk_libc@current.so"
provideLibs: "runtime-module-sdk_libdl@current.so"
provideLibs: "runtime-module-sdk_libdl_android@current.so"
provideLibs: "runtime-module-sdk_libm@current.so"

Now, we gather only those modules contributing CopyDepsToZip to generate
linker config for android_system_image.

Test: m nothing
Test: m microdroid
Test: conv_linker_config print -s
    .../microdroid/android_common/root/system/etc/linker.config.pb
    shouldn't list sdk members.
Change-Id: Ife5d2193ab59970367d10e196aaaa922c6582862
2021-05-11 11:56:54 +09:00
Jooyung Han
092ef811a0 Fix PackagingBase.CopyDepsToZip
CopyDepsToZip() zips direct dependencies with tags implementing
PackagingItem interface.

Previously, it relied on InstallNeededDependencyTag which has a
different meaning.
- InstallNeededDependencyTag tells whether a dependency is required to
  be installed together.
- PackagingItem tells whether a dependency (of PackagingBase) is
  required to be packaged.

With the separation of InstallNeededDependencyTag and PackagingItem,
PackagingBase module can distinguish cases which were not available
before.(I = InstallNeededDependencyTag, P = PackagingItem)

   a (PackagingBase module)
   |
   |`--(I)--> b
   |
   |`--(P)--> c --(I)--> d
   |
    `--(I/P)--> e

a's CopyDepsToZip(): [c, d, e]

Test: m nothing (packaging_test)
Change-Id: I71fce29b19b0f00dc394981bcf4240e9c1041c7a
2021-03-17 11:54:53 +09:00
Paul Duffin
7e2fad3f89 Merge changes I0f746a11,I08e61a24
* changes:
  Make OutputPath implement objPathProvider
  Allow CopyDepsToZip() to work on WritablePath
2021-02-03 10:01:39 +00:00
Paul Duffin
4076a75fd7 Allow CopyDepsToZip() to work on WritablePath
Makes the method more generally usable as it can work with any
WritablePath not just OutputPath. Also, makes it independent of whether
PathForModuleOut() returns ModuleOutPath or OutputPath.

Bug: 179124768
Test: m nothing
Change-Id: I08e61a24722198aa92fa2f8a2d096b1a8bf9c5c2
2021-02-02 13:32:12 +00:00
Jiyong Park
6446b62e9e add prebuilt_kernel_modules module
prebuilt_kernel_modules is a build rule for processing and installing
kernel module files. The module files are processed by depmod to build
modules.{load|alias|deps|softdeps} files. Debug symbols are stripped
from the module files and then all the files are installed to
./lib/modules/<kernel_version> directory.

This is the Soong-equivalent of the 'build-image-kernel-modules-dir`
macro and is expected to replace it in the future.

Bug: 178980227
Test: m

Change-Id: I772ac816824921d275869e35b93a083d239a2450
2021-02-02 00:41:57 +09:00
Jiyong Park
2136d15326 arch.<arch>.deps now works in android_filesystem
android_filesystem is a multi targets module type inhering from
PackagingBase. Since it's a multi-targets module, it is mutated only for
the common arch and therefore arch-specific properties are NOT squashed
into the matching top level property (i.e. setArchProperties is no op).

As a result, the values set in arch.<arch>.deps property were ignored.

This fixes the issue by explicitly adding the properties in
arch-specific structs and reading the properties in the matching arch.

Also added a set of tests for non-multi-target modules.

Bug: N/A
Test: m
Change-Id: Ibd4b509e73fa1760cc38b3661a08f83a6f639705
2021-02-02 00:41:28 +09:00
Colin Cross
ffe6b9d9ba Add TransitivePackagingSpecs
Add TransitivePackagingSpecs to return the PackagingSpecs for a
module and any of its transitive dependencies that have dependency
tags for which IsInstallDepNeeded returns true.

Bug: 124313442
Test: m checkbuild
Change-Id: I1d6750db830d1601d696349674f0b7071372ca11
2020-12-07 12:27:50 -08:00
Colin Cross
897a5ad507 Merge "Pass pctx and ctx to NewRuleBuilder" 2020-12-03 20:07:30 +00:00
Jiyong Park
f10995b7b6 Merge "Remove the arch-based filtering hack in packaging.go" 2020-12-02 01:24:17 +00:00
Kiyoung Kim
e76ec1cbeb Merge "Generate system stub library list at build time" 2020-12-02 00:28:19 +00:00
Colin Cross
f1a035e6be Pass pctx and ctx to NewRuleBuilder
Enable the RuleBuilder and RuleBuilderCommand methods to access
the BuilderContext by passing it to NewRuleBuilder instead of
RuleBuilder.Build.

Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I63e6597e19167393876dc2259d6f521363b7dabc
2020-12-01 16:22:16 -08:00
Jiyong Park
d630bdd4dd Remove the arch-based filtering hack in packaging.go
aosp/1501613 has landed. The filtering hack is no longer needed.

Also, fixed a bug that the dependency to the dynamic linker is not
treated as install dep.

Bug: N/A
Test: `m hosttar` and compare the built cvd-host_package.tar.gz with and
without this change. There is no removed file.

Change-Id: I19ef64e10d34bc8be2592c5258505cd632c91af9
2020-12-01 19:25:39 +09:00
Jiyong Park
cc1157cd49 Reformat comments in packaging.go
set textwidth=100

Bug: N/A
Test: N/A

Change-Id: I2fb315c9488010d189b7757a4d93be181672b4a6
2020-12-01 05:27:48 +00:00
Kiyoung Kim
24dfc1fbb2 Generate system stub library list at build time
List of stub libraries from system (and bionic APEX) is currently
managed in static file. This change generates the list at build time and
adds to the system config so linkerconfig can use it.

Bug: 172889962
Test: Build cuttlefish and confirmed list is generated in
/system/etc/linker.config.pb

Change-Id: Ie0400e9d1098b0de19c6a7c25f261384aadde0a3
2020-12-01 14:09:03 +09:00
Jiyong Park
65b6224471 Clients of PackagingBase can customize dependency tag to use
Previously, the dep tag used by PackagingBase was fixed, which prevented
some of its clients (e.g. cvd-host-package) from opting in to
android.InstallAlwaysNeededDependencyTag. Now, PackagingBase.AddDeps
accepts the dependency tag to use.

Also, dependencies toward rust dylib, rlib, and proc_macro are
configured to return true on InstallDepNeeded. This is required to
install shared_lib dependencies of the rust modules when they are
depended on by a rust binary.

Exempt-From-Owner-Approval: a trivial change after +2 from the owner.
This has to land ASAP as many users are affected by acloud being
unavailable.

Bug: N/A
Test: m
Test: acloud create --local-instance --local-image
Change-Id: If22aee7c6f314fcb03b9d4fe6901a2557f1e899c
2020-12-01 03:43:05 +00:00
Jiyong Park
dda8f69e43 Introduce PackagingBase
PackagingBase is the base struct that implements the basic packaging
functionalities. Here, packaging means grouping dependencies into an
output file where the built artifacts from the dependencies are placed
in a directory-like structure in the output file. The exact format of
the output file is irrelevant; it could be a filesystem image, tar.gz,
zip, or whatever.

PackagingBase is responsible for traversing the dependencies and copying
their outputs under the package root directory, which is expected to be
provided by the module type that includes PackagingBase. Then the
concrete module type is expected to do final step of converting the
package root directory into the output file of specific format.

Bug: 159685774
Bug: 172414391
Test: m dist out/dist/cvd-host_package.tar.gz
Change-Id: I5446eee4834ce3b6f0f5843d93bb330a26d42fe3
2020-11-19 08:58:06 +09:00
Jiyong Park
073ea55fad add PackagingSpec
Currently, installation of a module is defined as an action of copying
the built artifact of the module to an install path like out/soong/host
(for host modules) and out/target/product/<device>/<partition> (for
device modules). After the modules are installed, the installed files
are further processed to create packages like system.img, vendor.img,
cvd-host-package.tar.gz, etc.

This notion of installation seems to have originated from the old time
when system.img is the primary product of the entire build process
(modulo a few more like root.img). Packaging the installed files as the
filesystem image was considered as a post-build step then.

However, this model doesn't seem to fit well to the current and future
environment where we have a lot more filesystem images (system, vendor,
system_ext, product, ...). The filesystem images themselves are even
grouped together to form a higher-level filesystem image like super.img.
Furthermore, things like cvd-host-package.tar.gz requires us to be able
to group some of the host tools in a format that isn't filesystem image.
Lastly, we are expected to have more filesystem images that are subsets
of system.img (and their friends) for the Android-like mini OS that will
be running on on-device virtual machines. These all imply that the
packaging (which we call installation today) is not a global post-build
step, but a part of the build rules for creating the package-like
modules.

A model better fits to the new sitatuation might be this; a module
specifies its built artifact and the path where it should be placed. The
latter path is not rooted at out/. It's a relative path to the root
directory which will be determined by another module that implements the
packaging. For example, cc_library will have ./lib (or ./lib64), not
out/target/product/<device>/<partition>/lib as the path. Then packages
like system.img, cvd-host-package.tar.gz, etc. are explicitly modeled as
modules and they have deps to other modules. Then the modules are placed
at the relative path under the package root, and the entire root
directory finally is packaged as the output file (be it img, tar.gz, or
whatever).

PackagingSpec is the first step to implement the new model. It abstracts
a request to place a built artifact at a certain path in a package. It
has extra information about whether the path should be a symlink or not,
and whether the path is for an executable. It currently is created when
InstallFiles (and its friends) are called, and can be retrieved via
the new method PackagingSpecs().

In this CL, no one is using PackagingSpec. The installation is still
done by the existing rules created in InstallFiles, etc. and the
structs are not used for the filesystem images like system.img.

Bug: 159685774
Bug: 172414391
Test: m

Change-Id: Ie1dec72d1ac14382fc3b74e5c850472e9320d6a3
2020-11-18 11:30:19 +09:00