For rules which use RuleBuilder with Ninja variables (using the
BuildWithUnescapedNinjaVars function) but are not sandboxed, the command
output by RuleBuilder must not be escaped, or else Ninja variables will
not be evaluated until they are run on the command line.
Test: m libnum_traits liboid_registry crosvm droid
Change-Id: I81611368ab1975220f7f517df2f57646deac2af8
The RuleBuilder API would not expand Ninja variables because the
variables would be written verbatim to the sandbox manifest file. This
commit allows a rule to specify that the manifest file should be written
in an un-escaped format so that Ninja variables are expanded before
writing the manifest file.
Bug: 286077158
Test: rust sandboxing topic + go test
Change-Id: I1915431f6e24d04d343dacc213c9079674ec8251
The use case for this is for building rules_go's root builder which runs
into issues when built in a directory that contains a symlink to
prebuilts/go
The implementation will involve two changes of working dir
- `sbox` to change the working directory to
__SBOX_SANDBOX_DIR__
- the generated manifest will change the working
directory to mixed build execution root relative to that
Implemenation details
1. Create a unique intermediate path by hashing the outputs of a buildAction.
"out/bazel/output/execroot/__main__/" was deliberately not chosen as
the outpuDir for the sandbox because ruleBuilder would wipe it.
`sbox` will generate the files in __SBOX_SANDBOX_DIR__ and then place
the files in this intermediate directory.
2. After the files have been generated in (1), copy them to
out/bazel/output/execroot/__main__/...
3. For bazel depsets that are inputs of an action, copy the direct
artifacts into the sandbox instead of the phony target
4. Make sandboxing an opt-in. Currently we will only use it for
`GoToolchainBinaryBuild`
In the current implementation, (3) will increase the size of the ninja
file. With sboxing turned on for only GoToolchainBinaryBuild, this will
increase the size of the ninja file by around 1.3% on aosp's cf
Test: m com.android.neuralnetworks (will build soong_zip from source
using rules_go)
Test: OUT_DIR=out.other m com.android.neuralnetworks
Bug: 289102849
Change-Id: I7addda9af583ba0ff306e50c1dfa16ed16c29799
Add support to sbox for only writing output files if they are changed
to support enabling restat. Move the clearing of the output directory
into sbox, and when write if changed is enabled delete obsolete output
files without deleting output files that will be replaced.
Enable restat for local metalava rules. Restat for metalava rules
run remotely is still blocked on b/183630617.
Bug: 222095735
Test: TestRuleBuilder_Build
Test: m SystemUI
Change-Id: If8fc47e711f4a6f43947ab69f17cccc8ba6ae61a
Assume calls to RuleBuilderCommand.BuiltTool may refer to a tool
that was built against musl libc, and add it as a dependency
so that it is copied into the sandbox. This emulates the
behavior of compiling against glibc, which is available from the
host sysroot when running in the sandbox.
Bug: 190084016
Test: m USE_HOST_MUSL=true sdk-repo-build-tools
Change-Id: Ieafdcceb818f9c31595487aab3ffbafba1412b3a
Move cc module installation rules into Soong by overriding
InstallBypassMake. Update the locations that find host tools
to look in the Make installation directory instead of the Soong
installation directory, which will no longer be used.
The methods that find host tools are also used on go binaries,
so update the config methods that tell Blueprint where to install
go binaries to the Make installation directory too.
Bug: 204136549
Test: m checkbuild
Change-Id: Id172592c195e506102982a4af0084f6d9c68a896
Use pathForInstall instead of PathForOutput for HostToolPath,
HostJNIToolPath and sboxPathForToolRel so that they internally produce
an InstallPath that can later support being converted to Make install
path.
Bug: 204136549
Test: m checkbuild
Change-Id: Ie16a62641d113873daeec4d1dd4261251bc0d0eb
Previously, the RuleBuilder would add all supplied Paths directly to
one of its lists of Paths without checking to make sure it was not nil
only to panic somewhere in its Build() method when it attempted to
convert it to a string. Deferring the failure made it harder to debug.
This change fails fast by checking every Path when it is passed to the
RuleBuilder.
Test: m nothing
Change-Id: I06b35da02101f6dfab80d2daaf3d8e44ded8b6be
Major operations:
* Fix the go_package entries to be consistent, as the new tool
complains when it doesn't have a '/'.
* Regenerate with the new protoc-gen-go tool.
* github.com/golang/protobuf -> google.golang.org/protobuf
* proto.[Un]MarshalText -> prototext.[Un]Marshal
Change-Id: Ie1147bd2457fafb66ba555461b3bf14f0561a25d
Add RuleBuilderCommand.Validation and RuleBuilderCommand.Validations
that propagate to BuildParams.Validations.
Test: TestRuleBuilder_build, TestRuleBuilder
Change-Id: I9a08bbc073a432fd0b53b87d8948ac0f6afcf637
Order only dependencies that were added to a RuleBuilderCommand were
only propagated to the rule that was created for a missing dependencies
error. Set them on the main rule instead.
Test: TestRuleBuilder_build
Change-Id: I049d1bb921778d18d678439d236f69081927dc70
Use an interface instead of a string to retrieve the value for
$(location) or $(locations) expansions in genrules to allow
delaying the evaluation until the RuleBuilderCommand is available.
This allows using helpers like RuleBuilderCommand.PathForInputs
to properly rewrite the values for sandboxing.
Also remove the standalone SboxPathFor* methods that don't operate
on a specific RuleBuilderCommand that are now unnecessary.
Test: genrule_test.go
Change-Id: I8bb2647332ef118204a216cead23d062517e2b8c
The lint rule is manually creating a second rsp file because Ninja
only supports on per rule. Move the support into RuleBuilder so
that it can apply the same rewrites that it does to the primary
one.
Test: TestRuleBuilder_Build
Change-Id: Iec250a2d60e74ccf1b4ad085a960fec6867ea559
The current implementation causes inputs listed in an rsp file used with
sbox to be duplicated 3 times in the build.ninja file; once as a
dependency of the rule, once in the rspfile_content field of the rule
with the paths rewritten to be relative to the sandbox, and once in the
rule to write the sbox manifest. When RBE is enabled it also gets a
fourth copy in the list of files to be treated as inputs by rewrapper.
Reduce this to a single copy by using "$in" for the rspfile_content so
that the files only have to be listed in the input dependencies of the
rule, and then add support to sbox to rewrite the rsp file while copying
it into the sandbox, and pass it to rewrapper as well.
Test: m lint-check
Change-Id: I3f46f61119508d39a8bb231c99fc130153fb6f04
rewrapper supports a comma separate list of rsp files, replace
REParams.RSPFile with REParmas.RSPFiles.
Test: remoteexec_test.go
Change-Id: I7850c071c23d368d6fad4480dd527d146c13c6d3
Allow passing a remoteexec.REParams to RuleBuilder to configure it to
run the rule remotely through RBE. Requires the rule to use
SandboxInputs, which ensures that RuleBuilder is aware of all of the
inputs and outputs of the rule.
Running sbox in RBE initially seems unnecessary, as RBE is already a
good sandbox, but reproxy can execute RBE actions locally when configured
for local execution, local fallback or racing. Using sbox in RBE ensures
that these local actions are also sandboxed, giving consistent results
between directly executed actions, local RBE actions, and remote RBE
actions.
Bug: 182612695
Test: manual
Change-Id: Icf2f24dde8dee833eb680ba22566a8e1c0143b15
When RuleBuilder.SandboxInputs() is called configure sbox to copy
all the input files into the sandbox directory and then change the
working directory there when running the command.
Copying input files into the sandbox directory gets tricky when
the input file is the output file from another rule, and could
be at an arbitrary, possibly absolute path based on the value
of OUT_DIR. They will need to be copied to a directory in the
sandbox using the path relative to OUT_DIR.
RSP files need special handling, they need to both be copied into
the sandbox as an input, rewritten to contain paths as seen in the
sandbox, and references to them on the command line need to use
sandbox paths.
Bug: 182612695
Test: rule_builder_test.go
Change-Id: Ic0db961961b186e4ed9b76246881e3f04971825c
Output files in sandboxed RuleBuilder rules must be under the output
directory, but output paths that were marked as temporaries were
not error checked.
Bug: 182612695
Test: rule_builder_test.go
Change-Id: I09616402ef1637c2c455ec7d345a296711582e8f
WriteFileRule shouldn't force the caller to ninja escape the input,
and should shell escape spaces.
Bug: 182612695
Test: manual
Change-Id: Ide2f1ed92783eef7883279238de209d992d8f735
Now that rsp files use an explicit path instead of $out.rsp the
unescaped spans and NinjaEscapedCommands() support can be removed
from RuleBuilder.
Bug: 182612695
Test: rule_builder_test.go
Change-Id: I6705151ff0528c59aa1af56ee52c294da15a7a66
Using $out.rsp as the rsp file adds extra complexity around keeping
the $ unescaped. Make callers to FlagWithRspFileInputList provide
an explicit path for the rsp file instead.
Bug: 182612695
Test: rule_builder_test.go
Change-Id: I3f531d80c1efa8a9d09aac0a63790c5b11a9f0c6
The CommandDeps and Rspfile properties can contain arbitrary paths
that may need ninja escaping.
Bug: 182612695
Test: genrule_test.go
Change-Id: I1426e9dfeab3bde9f65b61d0c47840f97c1c7526
This relands I38393900677c5dbe8e72fe06a7dd7d71f3c09f82 after
I8038ed5c6f05c989ac21ec06c4552fb3136b9a7a, which makes the ASAN
variants of libraries have PackagingSpecs so they can be copied
into sandboxed genrules.
Copy tools used by genrules into the sandbox directory. This
ensures correct dependencies on all files used by tools, and
is one step closer to enabling genrules inside unselected
namespaces.
Bug: 124313442
Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I57c0d5fc8bba216fac4deb972d0d2098593e8963
Copy tools used by genrules into the sandbox directory. This
ensures correct dependencies on all files used by tools, and
is one step closer to enabling genrules inside unselected
namespaces.
Bug: 124313442
Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I38393900677c5dbe8e72fe06a7dd7d71f3c09f82
Enable the RuleBuilder and RuleBuilderCommand methods to access
the BuilderContext by passing it to NewRuleBuilder instead of
RuleBuilder.Build.
Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I63e6597e19167393876dc2259d6f521363b7dabc
This relands I3b918a6643cea77199fd39577ef71e34cdeacdb1 with a fix
to create the directory for the output depfile if doesn't exist.
In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.
Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I8af00c8c0b25a92f55a5032fcb525715ae8297c2
In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.
Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I3b918a6643cea77199fd39577ef71e34cdeacdb1
In preparation for more complicated sandboxing support in sbox, use
a single implementation of the sbox sandboxing by moving genrule to
use RuleBuilder's sbox support instead of creating an sbox rule
directly.
Also move genrule's input list hash support into RuleBuilder.
Test: genrule_test.go
Test: rule_builder_test.go
Change-Id: I292184d02743c7e6887ebbcd232ba565db2ab0cc
This CL adds symlink_outputs to various locations in Soong that creates
actions that creates symlink outputs, and explicitly mark them as such.
Test: m
Bug: 160568334
Change-Id: I322751bada52a9f49011c74731d84761586e03e7
The sandbox property indicates whether metalava should only read
inputs explicitly specified on the command line. This CL adds the
property and sets the appropriate configuration for RBE depending
on whether the sandbox is set or not.
Test: built aosp_crosshatch-userdebug with/without RBE_METALAVA.
Change-Id: I7256d29f18e0af18dbe65d1c7dbbf62fd3d65f4c
This CL fixes a bug with RuleBuilder's handling of remoteable actions.
It adds a new type of pool to identify remoteable rules by the android
module context. The pool is then set to nil to actually run actions at
NINJA_REMOTE_NUM_JOBS parallelism.
Test: built aosp crosshatch userdebug
Change-Id: I29452f6fc7a161b94189731e3e3cc1f34907b80c
This CL adds RBE support to javac, r8, and d8 rules which is only
enabled if respective environment variables are set.
Test: an aosp_crosshatch build with and without the new variables.
Change-Id: Ic82f3627944f6a5ee7b9f3228170c2709b1bfcb8
Create a highmem pool based on the total RAM and the number of CPUs,
with an override via the NINJA_HIGHMEM_NUM_JOBS variable. Put
metalava into the highmem pool.
Ninja does not support nested pools, and when goma or RBE is enabled
the maximum ninja parallelism is set very high with local jobs in a
local pool. When both the local pool and highmem pool are enabled,
the total number of local jobs will be as high as the sum of the sizes
of the two pools. Keep the highmem pool limited to 1/16th of the
local pool when remote builds are enabled to try to minimize the
effect while still limiting highmem jobs.
Fixes: 142644983
Test: m nothing, examine pools
Test: m USE_GOMA=true nothing, examine pools
Change-Id: Id79f11f44948992960ac34ecf831dacbe21bd332
sbox will generate a random directory for the output root, and most
tools will encode that directory name in the output target of the
depfile.
So embed the library from dep_fixer into sbox so that it can rewrite the
output filename to a static (reproducible) value. Ninja doesn't care
what that value is, so it's just "outputfile".
Also fix up rule_builder to actually tell sbox about the depfile.
Test: mmma system/iorap; check the contents of:
out/soong/.intermediates/system/iorap/libiorap-binder/android_arm_armv7-a-neon_core_static/gen/aidl/system/iorap/binder/com/google/android/startop/iorap/IIorap.cpp.d
Change-Id: I3640a2e8b0c034f143a35e398a8418a6d621b265
Allow RuleBuilderCommands to use an rspfile with FlagWithRspFileInputList.
This requires being more careful with ninja escaping, as the rspfile
will be $out.rsp, and the value for $out may not be known yet so it
must be inserted as a ninja variable that must not be escaped.
Test: rule_builder_test.go
Change-Id: Ifa91e24a0bb8f0ceeb5c9bfa5689be2a4ff3b9cd
Replace the common pattern of:
cmd.Tool(ctx.Config().HostToolPath(ctx, "tool"))
with:
cmd.BuiltTool("tool")
And similarly for PrebuiltBuildTool.
Test: m checkbuild
Change-Id: I7d63188505362c7df6a3b3e7330b4a2cca5a2409
