tequilaOS/platform_build_blueprint
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
platform_build_blueprint/blueprint.bash

67 lines
2.2 KiB
Bash
Raw Normal View History

Add build wrapper to do stage selection before ninja This wrapper script can be used instead of ninja to ensure the build won't get stuck building the primary builder. An example of when this would happen: 1. Do a successful build 2. Sync/make a change in the primary builder (soong, etc) that depends on a blueprint change. 3. The next build would notice that change, and rewind to the primary stage to rebuild the builder. That build would fail. 4. Sync/fix the blueprint code. 5. The next build would still fail. The bootstrap stage would need to be run in order to fix the primary stage, but we're still stuck in the primary stage. The only way to switch stages is to successfully complete everything required to choose the next stage. This generally isn't a problem in the main stage, since there is no code being built in the dependency chain leading up to stage selection. Any existing wrappers (like soong) can execute this wrapper (optionally skipping ninja execution) if they're worried about the above situation. This isn't strictly required -- running ninja directly will still work in most cases, you'll just need to re-run bootstrap.bash if you get into a bad state. Change-Id: I5901d7240a1daa388a786ceb1c8259502fc14058
2015-12-10 03:03:13 +01:00
#!/bin/bash
# This script is intented to wrap the execution of ninja so that we
# can do some checks before each ninja run.
#
# It can either be run with a standalone Blueprint checkout to generate
# the minibp binary, or can be used by another script as part of a custom
# Blueprint-based build system. When used by another script, the following
# environment variables can be set to configure this script, which are
# documented below:
#
# BUILDDIR
# SKIP_NINJA
#
# When run in a standalone Blueprint checkout, bootstrap.bash will install
# this script into the $BUILDDIR, where it may be executed.
#
# For embedding into a custom build system, the current directory when this
# script executes should be the same directory that $BOOTSTRAP should be
# called from.
set -e
# BUILDDIR should be set to the path to store build results. By default,
# this is the directory containing this script, but can be set explicitly
# if the custom build system only wants to install their own wrapper.
[ -z "$BUILDDIR" ] && BUILDDIR=`dirname "${BASH_SOURCE[0]}"`
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
# NINJA should be set to the path of the ninja executable. By default, this
# is just "ninja", and will be looked up in $PATH.
[ -z "$NINJA" ] && NINJA=ninja
if [ ! -f "${BUILDDIR}/.blueprint.bootstrap" ]; then
echo "Please run bootstrap.bash (.blueprint.bootstrap missing)" >&2
exit 1
fi
Add build wrapper to do stage selection before ninja This wrapper script can be used instead of ninja to ensure the build won't get stuck building the primary builder. An example of when this would happen: 1. Do a successful build 2. Sync/make a change in the primary builder (soong, etc) that depends on a blueprint change. 3. The next build would notice that change, and rewind to the primary stage to rebuild the builder. That build would fail. 4. Sync/fix the blueprint code. 5. The next build would still fail. The bootstrap stage would need to be run in order to fix the primary stage, but we're still stuck in the primary stage. The only way to switch stages is to successfully complete everything required to choose the next stage. This generally isn't a problem in the main stage, since there is no code being built in the dependency chain leading up to stage selection. Any existing wrappers (like soong) can execute this wrapper (optionally skipping ninja execution) if they're worried about the above situation. This isn't strictly required -- running ninja directly will still work in most cases, you'll just need to re-run bootstrap.bash if you get into a bad state. Change-Id: I5901d7240a1daa388a786ceb1c8259502fc14058
2015-12-10 03:03:13 +01:00
# .blueprint.bootstrap provides saved values from the bootstrap.bash script:
#
# BOOTSTRAP
# BOOTSTRAP_MANIFEST
#
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
source "${BUILDDIR}/.blueprint.bootstrap"
GEN_BOOTSTRAP_MANIFEST="${BUILDDIR}/.minibootstrap/build.ninja.in"
if [ -f "${GEN_BOOTSTRAP_MANIFEST}" ]; then
if [ "${BOOTSTRAP_MANIFEST}" -nt "${GEN_BOOTSTRAP_MANIFEST}" ]; then
"${BOOTSTRAP}" -i "${BOOTSTRAP_MANIFEST}"
Add build wrapper to do stage selection before ninja This wrapper script can be used instead of ninja to ensure the build won't get stuck building the primary builder. An example of when this would happen: 1. Do a successful build 2. Sync/make a change in the primary builder (soong, etc) that depends on a blueprint change. 3. The next build would notice that change, and rewind to the primary stage to rebuild the builder. That build would fail. 4. Sync/fix the blueprint code. 5. The next build would still fail. The bootstrap stage would need to be run in order to fix the primary stage, but we're still stuck in the primary stage. The only way to switch stages is to successfully complete everything required to choose the next stage. This generally isn't a problem in the main stage, since there is no code being built in the dependency chain leading up to stage selection. Any existing wrappers (like soong) can execute this wrapper (optionally skipping ninja execution) if they're worried about the above situation. This isn't strictly required -- running ninja directly will still work in most cases, you'll just need to re-run bootstrap.bash if you get into a bad state. Change-Id: I5901d7240a1daa388a786ceb1c8259502fc14058
2015-12-10 03:03:13 +01:00
fi
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
else
"${BOOTSTRAP}" -i "${BOOTSTRAP_MANIFEST}"
fi
Add build wrapper to do stage selection before ninja This wrapper script can be used instead of ninja to ensure the build won't get stuck building the primary builder. An example of when this would happen: 1. Do a successful build 2. Sync/make a change in the primary builder (soong, etc) that depends on a blueprint change. 3. The next build would notice that change, and rewind to the primary stage to rebuild the builder. That build would fail. 4. Sync/fix the blueprint code. 5. The next build would still fail. The bootstrap stage would need to be run in order to fix the primary stage, but we're still stuck in the primary stage. The only way to switch stages is to successfully complete everything required to choose the next stage. This generally isn't a problem in the main stage, since there is no code being built in the dependency chain leading up to stage selection. Any existing wrappers (like soong) can execute this wrapper (optionally skipping ninja execution) if they're worried about the above situation. This isn't strictly required -- running ninja directly will still work in most cases, you'll just need to re-run bootstrap.bash if you get into a bad state. Change-Id: I5901d7240a1daa388a786ceb1c8259502fc14058
2015-12-10 03:03:13 +01:00
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
# Build minibp and the primary build.ninja
Clean up unnecessary dependencies Other than cleaning up the code, the only change in behavior is to run the docs build in parallel with the primary builder, saving about a second on a Soong bootstrap (10s -> 9s). Change-Id: Iaff51d6d1a37af842f294f12db11d5774d48d440
2016-11-02 06:44:25 +01:00
"${NINJA}" -w dupbuild=err -f "${BUILDDIR}/.minibootstrap/build.ninja"
Add build wrapper to do stage selection before ninja This wrapper script can be used instead of ninja to ensure the build won't get stuck building the primary builder. An example of when this would happen: 1. Do a successful build 2. Sync/make a change in the primary builder (soong, etc) that depends on a blueprint change. 3. The next build would notice that change, and rewind to the primary stage to rebuild the builder. That build would fail. 4. Sync/fix the blueprint code. 5. The next build would still fail. The bootstrap stage would need to be run in order to fix the primary stage, but we're still stuck in the primary stage. The only way to switch stages is to successfully complete everything required to choose the next stage. This generally isn't a problem in the main stage, since there is no code being built in the dependency chain leading up to stage selection. Any existing wrappers (like soong) can execute this wrapper (optionally skipping ninja execution) if they're worried about the above situation. This isn't strictly required -- running ninja directly will still work in most cases, you'll just need to re-run bootstrap.bash if you get into a bad state. Change-Id: I5901d7240a1daa388a786ceb1c8259502fc14058
2015-12-10 03:03:13 +01:00
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
# Build the primary builder and the main build.ninja
Clean up unnecessary dependencies Other than cleaning up the code, the only change in behavior is to run the docs build in parallel with the primary builder, saving about a second on a Soong bootstrap (10s -> 9s). Change-Id: Iaff51d6d1a37af842f294f12db11d5774d48d440
2016-11-02 06:44:25 +01:00
"${NINJA}" -w dupbuild=err -f "${BUILDDIR}/.bootstrap/build.ninja"
Add build wrapper to do stage selection before ninja This wrapper script can be used instead of ninja to ensure the build won't get stuck building the primary builder. An example of when this would happen: 1. Do a successful build 2. Sync/make a change in the primary builder (soong, etc) that depends on a blueprint change. 3. The next build would notice that change, and rewind to the primary stage to rebuild the builder. That build would fail. 4. Sync/fix the blueprint code. 5. The next build would still fail. The bootstrap stage would need to be run in order to fix the primary stage, but we're still stuck in the primary stage. The only way to switch stages is to successfully complete everything required to choose the next stage. This generally isn't a problem in the main stage, since there is no code being built in the dependency chain leading up to stage selection. Any existing wrappers (like soong) can execute this wrapper (optionally skipping ninja execution) if they're worried about the above situation. This isn't strictly required -- running ninja directly will still work in most cases, you'll just need to re-run bootstrap.bash if you get into a bad state. Change-Id: I5901d7240a1daa388a786ceb1c8259502fc14058
2015-12-10 03:03:13 +01:00
# SKIP_NINJA can be used by wrappers that wish to run ninja themselves.
if [ -z "$SKIP_NINJA" ]; then
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
"${NINJA}" -w dupbuild=err -f "${BUILDDIR}/build.ninja" "$@"
Add build wrapper to do stage selection before ninja This wrapper script can be used instead of ninja to ensure the build won't get stuck building the primary builder. An example of when this would happen: 1. Do a successful build 2. Sync/make a change in the primary builder (soong, etc) that depends on a blueprint change. 3. The next build would notice that change, and rewind to the primary stage to rebuild the builder. That build would fail. 4. Sync/fix the blueprint code. 5. The next build would still fail. The bootstrap stage would need to be run in order to fix the primary stage, but we're still stuck in the primary stage. The only way to switch stages is to successfully complete everything required to choose the next stage. This generally isn't a problem in the main stage, since there is no code being built in the dependency chain leading up to stage selection. Any existing wrappers (like soong) can execute this wrapper (optionally skipping ninja execution) if they're worried about the above situation. This isn't strictly required -- running ninja directly will still work in most cases, you'll just need to re-run bootstrap.bash if you get into a bad state. Change-Id: I5901d7240a1daa388a786ceb1c8259502fc14058
2015-12-10 03:03:13 +01:00
else
exit 0
fi
Reference in a new issue Copy permalink