Commit graph

775 commits

Author SHA1 Message Date
LaMont Jones
ff387eaacb release_config: various cleanup
- Allow scl and textproto data to coexist for now
- Print warnings to stderr instead of stdout.
- Improve formatting of output
- Set displays the new value for all configs, and what file changed.
- Use prettier error messages for better UX
- put build-flag on the path.

Bug: 328495189
Test: manual
Change-Id: I4c38860c2fb24db5111e0cecf790660a4ff2b8b2
2024-04-30 16:12:50 -07:00
LaMont Jones
dc86819307 Support release configs with only aconfig flags
Release configs with `aconfig_flags_only: true` can inherit build flag
values, but cannot set them.

Bug: 328495189
Test: manual
Change-Id: I2bdc88761d61fb20f10dc734f2ba87114f51b859
2024-04-30 15:38:11 -07:00
LaMont Jones
e41ea1e33e release_config: better default map paths
- Automatically determine the top of the workspace.
- build-flag defaults to using `get_build_var` to get product specific
  release config maps.
- release-config defaults to using PRODUCT_RELEASE_CONFIG_MAPS but does
  not use `get_build_var` unless the argument is given.

Bug: 328495189
Test: manual
Change-Id: I4ba3c5dfab43c4ebc3eeda13318f42e886dada4e
2024-04-29 14:43:09 -07:00
LaMont Jones
db600990a5 release_config: container is a repeated string field
Container is a string, and a flag can be in more than one container.

Bug: 328495189
Test: manual
Change-Id: I5a2a9855532027584d2b67f63f1b9584fce3d8d9
2024-04-26 14:56:58 -07:00
LaMont Jones
f4cc08e114 Write per-partition build_flags.json
Create build_flags_{partition}-{TARGET_PRODUCT}-{TARGET_RELEASE}.json
in {OUT_DIR}/soong/release-config.

Bug: 328495189
Test: manual
Change-Id: I708c37f9b1216b4283886c98cacaf66bfcd28143
2024-04-26 09:24:47 -07:00
LaMont Jones
15788829b2 release_config: various cleanup
- Parse release_config_map.textproto files only once
- Fix inheritance
- Sort flag artifacts by flag name
- Add --all_make option for testing
- Fix value() parsing in crunch_flags

Bug: 328495189
Test: manual
Change-Id: I577e7fb07171bea9a53d61eaf77ec728b60d7a26
2024-04-26 08:28:39 -07:00
LaMont Jones
ccb2207cfa add flag_value.redacted
When set, this removes all traces of the flag from the release config.

Bug: 328495189
Test: manual
Change-Id: Iab39d06e134bfc020ba4875c1b8e89c5cdf18924
2024-04-24 13:06:14 -07:00
LaMont Jones
11209e1255 crunch_flags: keep-sorted lines are not descriptions
Also fix a typo in parsing declare-release-config.

Bug: 328495189
Test: manual, TH
Change-Id: Ibfed4a217275df0dd6bd9c2fb124e0d1f791be08
2024-04-24 13:05:55 -07:00
LaMont Jones
b9014c74f4 Add crunch-flags and build-flag binaries
- crunch-flags automates converting build flags from starlark to protobuf.
- build-flag is used to set, get and trace flag values.

Bug: 328495189
Test: manual
Change-Id: I941a4420a8bdfa2df73d94e52b3f34a6d1ea3278
2024-04-18 11:21:03 -07:00
LaMont Jones
14e2ac68e3 release_config: cleanup how we emit results
- Allow each of the formats (json, pb, textproto) to be individually
   controlled.
 - Include product and release in the name of the aritfact.

Bug: 328495189
Test: manual
Change-Id: Ia08e7224200a48994bea882a61b8199d53576d38
2024-04-18 11:21:03 -07:00
LaMont Jones
7ac07deb9b Print a warning on redundant overrides
Bug: 328495189
Test: manual
Change-Id: I35bff9d6751e9aa304e4c2d7e24b9a44a3994264
2024-04-18 10:25:25 -07:00
LaMont Jones
f018d726ca Add description to release_config_map message
Bug: 328495189
Test: manual
Change-Id: Ifff2dc9a2f33595bde9ef57e407bbb4b529f6e08
2024-04-18 10:25:25 -07:00
LaMont Jones
ac79679f77 release_config: make the code more sharable
This will allow the `build-flag` command to use release_config_lib code.

Bug: 328495189
Test: manual
Change-Id: If068597b1d68c52c941788931997b7f77c73d2b3
2024-04-15 17:41:44 -07:00
LaMont Jones
e6f5468ef2 release_config: add namespace to proto
flag_declaration needs to include the namespace, and origin is not
needed.

Bug: 328495189
Test: manual
Change-Id: Icbadbe6d9888323874c07d84ccd6f50c284e1a1c
2024-04-15 14:05:15 -07:00
LaMont Jones
1eccbf24c5 release_config: Initial implementation
Bug: 328495189
Test: manual
Change-Id: I25a2a197f0227bfa7c63e0dfb7cab09e7d756d90
2024-04-10 09:34:26 -07:00
Treehugger Robot
ac45b4bc63 Merge "Move ELF build-id reader into a separate library." into main 2024-03-20 00:13:10 +00:00
Peter Collingbourne
941ff1dbc7 Move ELF build-id reader into a separate library.
Bug: 328702178
Change-Id: I188a8d20d22e67e4f0c7e3441e3781fff369c828
2024-03-15 17:07:36 -07:00
Cole Faust
256cfbee24 Remove starlark_import
This is no longer used since the roboleaf cancellation.

Bug: 315353489
Test: m nothing --no-skip-soong-tests
Change-Id: Ie6ee093c2810498306ea4a2288050eed17a35357
2024-03-07 10:53:41 -08:00
Cole Faust
331b27f511 Remove --multitree-build
Multitree has been discontinued.

Test: m nothing --no-skip-soong-tests
Change-Id: Ie4d96a11279bf0f20a5e1a49837d9df4b24b1662
2024-02-28 15:35:49 -08:00
Anas Sulaiman
437e9470c0 Fix non-deterministic output for merge_zips
A map iteration was causing a non-deterministic order
in output entries. The default behaviour is to preserve
the same order of the input zips, which is necessary
to ensure a deterministic output for deterministic inputs.

Bug: b/322788229
Test: Ran a couple of builds and confirmed no cache misses from
the output of merge_zips.

Change-Id: I3217e6887ab108d213a290b59da5b33d51b8241f
2024-02-12 22:00:55 +00:00
Joe Onorato
e5ed34746e Add new soongdbg command and a big json file full of soong debugging info.
In order to use soongdbg, you must run analysis with GENERATE_SOONG_DEBUG=true set
in the environment.

Test: GENERATE_SOONG_DEBUG=true m nothing ; soongdbg ...
Change-Id: If43676fe2784f05cd87c0ecb4a46ab676b91023f
2024-02-03 14:44:11 -08:00
Sorin Basca
78885cae81 Use Java 1.8 when generating build files
Bug: 320294595
Test: m
Change-Id: I5d4405001e02f2baa56650203be23caceb1094c0
2024-01-26 16:47:40 +00:00
Elliott Hughes
10363161e7 "master" was renamed "main".
Test: N/A
Change-Id: I86d5578eaac260e55a9583db7ab49812b4ba1f5d
2024-01-09 22:09:07 +00:00
Colin Cross
138d37dd6c Merge "Support generating module_info.json in Soong" into main 2024-01-03 17:55:47 +00:00
Cole Faust
8c0b11ef79 Add direct deps on glob result files
n2 (a reimplemntation of ninja) adds a check that depfiles only add
dependencies on source files, not generated files, because generated
files are not guaranteed to exist before an action runs if there isn't
an explicit dependency between the two.

This is the case for our glob result files. However, we can rework the
build so that they're directly depended on instead of using a depfile.

Bug: 318434287
Test: rm -rf out && m nothing && m nothing
Change-Id: I513fa5536136e6bf19c347710f0722d696199612
2024-01-02 17:02:52 -08:00
Colin Cross
d6fd013394 Support generating module_info.json in Soong
Generate module_info.json for some Soong modules in Soong in order to
pass fewer properties to Kati, which can prevent Kati reanalysis when
some Android.bp changes are made.

Soong modules can export a ModuleInfoJSONProvider containing the
data that should be included in module-info.json.  During the androidmk
singleton the providers are collected and written to a single JSON
file.  Make then merges the Soong modules into its own modules.

For now, to keep the result as similar as possible to the
module-info.json currently being generated by Make, only modules that
are exported to Make are written to the Soong module-info.json.

Bug: 309006256
Test: Compare module-info.json
Change-Id: I996520eb48e04743d43ac11c9aba0f3ada7745de
2024-01-02 16:03:43 -08:00
Cole Faust
b0a21dc871 Merge "Reapply "Handle symlinks in sbox"" into main 2023-12-15 02:00:40 +00:00
Cole Faust
ca355af1fc Reapply "Handle symlinks in sbox"
This reverts commit 6a855191bd.

Reason for revert: Last issue fixed with aosp/2875001

Change-Id: I961ea6d97c936df8a6475995b768a7eda861e6ee
2023-12-14 23:38:38 +00:00
Treehugger Robot
a53330a525 Merge "Revert "Handle symlinks in sbox"" into main 2023-12-14 01:58:17 +00:00
Cole Faust
6a855191bd Revert "Handle symlinks in sbox"
This reverts commit 14a2362183.

Reason for revert: Breaks art tests on errorprone build

Change-Id: I2e21867fba6c8c1660ffa564273b6200c63221f3
2023-12-13 23:48:11 +00:00
Cole Faust
a1eb34a709 Merge "Reapply "Handle symlinks in sbox"" into main 2023-12-13 17:47:07 +00:00
Colin Cross
b63d7b3af7 Remove infrastructure to run bp2build
Bug: 315353489
Test: m blueprint_tests
Change-Id: Idcf6377d389b94c39e4e6ff4b8efa8a9f9e78b17
2023-12-08 13:51:07 -08:00
Colin Cross
e51c6e4109 Merge "Remove bp2build and bazel from soong_ui" into main 2023-12-08 21:43:26 +00:00
Cole Faust
14a2362183 Reapply "Handle symlinks in sbox"
This reverts commit 69b5a5f0bf.

Bug: 307824623
Test: Presubmits
Change-Id: Id7dd1518ae6b04232c48c0aa138cad2c59b228c3
2023-12-08 12:43:00 -08:00
Colin Cross
8d411ff8f8 Remove bp2build and bazel from soong_ui
Bug: 315353489
Test: m blueprint_tests
Change-Id: I9df31b18caaae24e3cf2994e56bb90b50523f11e
2023-12-07 17:14:27 -08:00
Cole Faust
69b5a5f0bf Revert "Handle symlinks in sbox"
This reverts commit dd9529050d.

Reason for revert: There are at least b/315337725 and b/315036698 issues, reverting this cl while I fix them.

Change-Id: I644bbbe594b9dd53a99a3b6513fbcba68afe0b4a
2023-12-07 21:40:08 +00:00
Cole Faust
dd9529050d Handle symlinks in sbox
Previously, sbox would follow the symlink and copy the symlink's target
into the sbox, instead of the symlink itself.

camera-its is a genrule that zips up a folder that includes a symlink,
so we want the symlink to still be there instead of a copy of the file
it points to.

Bug: 307824623
Test: ./build/soong/tests/genrule_sandbox_test.py camera-its
Change-Id: Ib81f6fd6745ec2c58580082c73e33760d5051f5e
2023-11-28 17:34:38 -08:00
Jihoon Kang
8c4387e478 Merge "Revert^2 "Default from-text stub generation in build"" into main 2023-11-10 04:10:24 +00:00
Colin Cross
4ade19fe94 Merge changes I4d0b27ba,Ic48e9cf2 into main
* changes:
  Fix soong_ui file rlimits for go 1.21
  Fix TestReverseSlice for go 1.21
2023-11-09 18:47:40 +00:00
Colin Cross
611a2fbdd7 Fix soong_ui file rlimits for go 1.21
Go 1.21 modifies the file limit but restores the original when
execing subprocesses if it hasn't be overridden.  Call Setrlimit
even if it doesn't appear to be necessary so that the syscall package
considers it set.

Bug: 309895579
Test: m nothing
Change-Id: I4d0b27bac90a2a88bfc68a8491d54dc106e1ec13
2023-11-08 22:11:28 -08:00
Jihoon Kang
2a929ad250 Revert^2 "Default from-text stub generation in build"
This change defaults Java stubs to be generated from API text files
during build. Using the `--build-from-source-stubs` flag, users can
toggle between the feature.

This change relands aosp/2617457.

Test: m nothing && verify ninja path exists between android_stubs_current and android_stubs_current.from-text, and does not exist between android_stubs_current.from-source, m nothing --build-from-source-stub && verify the opposite
Bug: 301522358
Change-Id: Ic129dafc3231e71470af982374739930947b69cb
2023-11-08 04:25:52 +00:00
LaMont Jones
9a9128610b Readd support for PRODUCT_RELEASE_CONFIG_MAPS
This allows product config to determine what build flags are present in
a product's build, as well as their value.

As product config moves to a declarative specification, this can be
handled with less unnecessary work.

Bug: b/302593603, b/308849337, b/309477343
Test: manual
Change-Id: Ic0551990864bba6d03b6bf49fcfd4ae2523d9c6d
2023-11-07 18:39:40 +00:00
Colin Cross
6a4fe68cc5 Revert "Add support for PRODUCT_RELEASE_CONFIG_MAPS"
Revert submission 2815850-product-flags

Reason for revert: b/309477343

Reverted changes: /q/submissionid:2815850-product-flags

Bug: 302593603
Bug: 309477343
Change-Id: I5b5a5ade1d85195c43377ed7639deba6eef86e23
2023-11-06 21:32:03 +00:00
LaMont Jones
127127b608 Add support for PRODUCT_RELEASE_CONFIG_MAPS
This allows product config to determine what build flags are present in
a product's build, as well as their value.

As product config moves to a declarative specification, this can be
handled with less unnecessary work.

Bug: b/302593603
Bug: b/308849337
Test: manual
Change-Id: If57622059bb7d1c33df7498321db621c0c2d30c0
2023-11-03 19:19:59 +00:00
Colin Cross
fa24df6b23 Always reset timestamps in merge_zips
Jacoco uses ziptime and then merge_zips.  merge_zips is already rewriting
all the zip headers so its trivial to move the ziptime functionality
into merge_zips.

Bug: 308765940
Test: TestMergeZips
Change-Id: I8158c0b99878492233ac01fde3820e1951c9088e
2023-11-01 14:41:40 -07:00
Colin Cross
66d6409e06 Merge "Add performance counter metrics to build.trace.gz" into main 2023-10-30 22:21:14 +00:00
Colin Cross
46b0c75204 Add performance counter metrics to build.trace.gz
Start a background goroutine at the beginning of soong_build that
captures the CPU usage, heap size, and total system memory every
second.  Propagate the values through soong_build_metrics.pb back
to soong_ui, and then into build.trace.gz.

Test: m nothing, examine build.trace.gz
Change-Id: Iad99f8f1f088f4f7f7d5f76566a38c0c4f4d0daa
2023-10-30 12:39:07 -07:00
MarkDacek
39825ead17 Remove Bazel metrics/finalize-bazel-metrics.
Test: m nothing
Change-Id: Idd4610aca77f0a5deeadc3b895869879ded6c16e
2023-10-26 20:14:36 +00:00
Sebastian Pickl
1c4188c9da Revert "Default from-text stub generation in build"
This reverts commit 182b56b870.

Reason for revert: prime suspect for breaking boot tests b/307495247, b/307411752

Bug:307495247
Change-Id: Iea05703b767d2699ca3cf69377eb44b1d21697ad
2023-10-24 11:20:06 +00:00
Jihoon Kang
182b56b870 Default from-text stub generation in build
This change defaults Java stubs to be generated from API text files
during build. Using the `--build-from-source-stubs` flag, users can
toggle between the feature.

Test: m nothing && verify ninja path exists between android_stubs_current and android_stubs_current.from-text, and does not exist between android_stubs_current.from-source, m nothing --build-from-source-stub && verify the opposite
Bug: 274805756
Change-Id: I28834f92c1b1311e3fe0a71a6ea9e8ec2e278d7e
2023-10-20 23:11:19 +00:00