Commit graph

101 commits

Author SHA1 Message Date
Treehugger Robot
0ddc5724b0 Merge "Add type hints to mk2rbc" 2022-03-16 00:40:11 +00:00
Cole Faust
f92c9f2809 Add type hints to mk2rbc
Type hints have the format #RBC# type_hint MY_VAR list
and must be specified at the top of the Makefile. Setting
one will cause that variable to have that type for the
remainder of the Makefile. This can be used where mk2rbc's
type inference detects the wrong type and it must be
manually changed.

Bug: 224601891
Test: go test
Change-Id: I6db2c50056d0298227e1d2801a522adf8bbd1df8
2022-03-15 12:52:20 -07:00
Cole Faust
e2a37988ff Simplify and correct variable assignments
- Remove asgnMaybeAppend, it was only used to indicate
  that the asignment needs a setDefault. But really, all
  types of variable assignments need setDefault if they're
  self-referential.
- Remove local_append/local_set_default because there was
  no implementation for them written in product_config.rbc
  anyways.
- Correct productConfigVariable.emitDefined using the global
  variables instead of the product config variables.
- Emit setDefaults for all types of assignments if they're
  self referential.

Bug: 222737841
Test: go test
Change-Id: I06a0f90f16d5900049d473281e6d5ef5e93e67da
2022-03-10 15:00:57 -08:00
Cole Faust
7d49b1e70a Merge "Call rblf.setDefault() when appending to a variable without +=" 2022-03-08 22:58:58 +00:00
Cole Faust
816e080c4d Call rblf.setDefault() when appending to a variable without +=
Bug: 222737841
Test: go test
Change-Id: I10e9e994fb1979e2e06ad30bbe66a21657d1e3db
2022-03-08 22:58:42 +00:00
Cole Faust
3c4fc99593 Support variables with dashes in their names
Bug: 221946551
Test: go test
Change-Id: I085fc35159c4f3afe53868fbc731fcaeac3a69a8
2022-03-07 11:24:45 -08:00
Cole Faust
93f8d3987a Fix include paths that match one file being included unconditionally
In aosp/1962181 an "optimization" was added where if an include
path with a variable only matches one file, that file would be
called directly instead of going through the whole _entry
variable.

This doesn't work when there is an inherit-product-if-exists call
that matches one file, but that file isn't the one that is specified
at runtime. In reality, the inherit-product-if-exists call should
see that the file doesn't exist and not include anything.

Fixes: 222341691
Test: go test
Change-Id: If8789de0013726137683078355161eecdec009b1
2022-03-02 13:31:30 -08:00
Cole Faust
fc74246c98 Merge "Convert add-product-dex-preopt-module-config to Starlark" 2022-03-02 18:40:36 +00:00
Treehugger Robot
d5f7ad3fb4 Merge "Fix issue with referencing the loop variable in a foreach" 2022-03-02 03:32:54 +00:00
Treehugger Robot
6028b556ec Merge "Inline board configuration call into printvars call" 2022-03-02 03:31:25 +00:00
Cole Faust
b67aa082aa Fix issue with referencing the loop variable in a foreach
Fixes: 222160672
Test: go test
Change-Id: I3f9238d4f1684cf4a2d24c4f7f49c832c3f72b97
2022-03-01 16:11:50 -08:00
Cole Faust
a060466be6 Inline board configuration call into printvars call
This is so that:
 - It's consistent with product configuration
 - The return type of board/product config's main functions
   and the arguments to printvars can be changed without
   also making a change in the soong repository. (what I'm about
   to do)
 - Removes obsolete TODO

Bug: 221312856
Test: Presubmits
Change-Id: I593f180881eab9521012df202af8f0233bc0b6ae
2022-02-28 11:56:06 -08:00
Cole Faust
1cc088523f Convert add-product-dex-preopt-module-config to Starlark
Bug: 221877397
Test: go test
Change-Id: I79f5209bb42f2c689af96ded376c08522cbec24e
2022-02-28 11:12:08 -08:00
Cole Faust
1add0bc5e2 Remove mention of BOARD_CONFIG_VENDOR_PATH
It has been removed from all product configurations.

Fixes: 220169197
Test: Presubmits
Change-Id: Ib9330ff71078697a5c93757426215bd0f3efdba4
2022-02-22 11:50:34 -08:00
Treehugger Robot
190e536e64 Merge "Allow filter calls with a list as a pattern" 2022-02-11 00:46:58 +00:00
Cole Faust
9932f75151 Allow filter calls with a list as a pattern
This commit doesn't attempt to replace the filter
calls with anything more idomatic for now.

It also removes the case for filter-out, because
the actual function name is filter_out and wasn't
being used anyways. Even if I were to change it
to filter_out, that would produce buggy results:

ifneq (,$(filter $(TARGET_BUILD_VARIANT), userdebug eng))
endif
ifneq (,$(filter-out $(TARGET_BUILD_VARIANT), userdebug eng))
endif

Both of these would produce:

if g["TARGET_BUILD_VARIANT"] in ["userdebug", "eng"]:
  pass

Fixes: 218702402
Test: go test
Change-Id: I566079e5d3a364c42db14045aa1bab9d99eba05f
2022-02-09 14:15:47 -08:00
Cole Faust
f4e72cf642 Update warning message
As requested on aosp/1975486

Fixes: 217248902
Test: Presubmits
Change-Id: I14e65722d8ab96125827d34e8501b4a518a27b57
2022-02-08 13:10:26 -08:00
Treehugger Robot
cb1703d8b1 Merge "Link to documentation about variables in include statements" 2022-02-08 20:04:57 +00:00
Cole Faust
7dbdfac3cc Link to documentation about variables in include statements
As requested in aosp/1962181

Fixes: 217248902
Test: Presubmits
Change-Id: Ib0f99a4b86999782bfcf3ccf89375a929cf5127f
2022-02-07 12:31:20 -08:00
Cole Faust
9b6111aaed Remove --root, require the cwd to be the root
This allows us to use relative paths everywhere.
It also produces more accurate emulation of TOPDIR,
which should be an empty string.

Bug: 213508006
Test: go test
Change-Id: Ie4e357687486e84e9f4aad0f6776d8feb2b9fc63
2022-02-07 11:45:01 -08:00
Cole Faust
8d47c48082 Use soong's finder to find included makefiles in mk2rbc
Having soong generate a list of makefiles for mk2rbc
to look through is much faster than having mk2rbc search
itself. Profiling the readLinesFromeFile() function that
reads the list of makefiles shows it takes on the order
of 200 microseconds, much faster than the ~5 seconds
it takes for mk2rbc to search the tree itself.

This CL also allows include statements that are prefixed
with a variable. The concern with this was that there
would be a lot of load statemnts emitted for generic
include statements, causing the generated code to look
ugly, and converting and loading all those files could
cause performance issues. On the performance issues
front, there's already a check that it doesn't result
in over 150 potentially included files. We can lower
that number if necessary, but it's probably good for now.
On the generated code front, while it's true that it's
ugly, it's better to have working but ugly generated
code than refusing to generate anything working at all.

To ensure the soong finder step isn't slowed down due
to having to find a bunch of new makefiles, I profiled
the combination of newSourceFinder and FindSources in main.go:

Baseline incremental:
338.011634ms
340.853335ms
348.541762ms
333.229644ms
349.124824ms

Baseline clean:
1.003836419s
1.006203912s
996.193648ms
1.031005604s
1.03691152s

Modified incremental:
349.029285ms
349.264496ms
351.774948ms
337.63187ms
359.425306ms

Modified clean:
1.028238704s
1.053103506s
1.032757506s
1.016631201s
1.04288108s

So we can see the times are barely affected by this change.

Fixes: 213508006
Test: go test
Change-Id: Iab18bfb127ba3b7e63f2c01f69064805a8398764
2022-02-07 11:41:28 -08:00
Cole Faust
dd569aea07 Return starlarkNodes from the functions that parse them
Currently, mk2rbc is structured around having a global
"receiver" object that accepts all starlarkNodes. As soon
as they are parsed they are added to the receiver.

Returning the parsed nodes to the calling function is more
flexible, as it allows the calling function to restructure
them as necessary. This is the first step to supporting
complicated statements involving $(eval), such as
`$(foreach v,$(MY_LIST),$(eval MY_LIST_2 += $(v)))`

Test: go test
Change-Id: Ia194123cf090d2b9559a25b975ccbc5749357cc0
2022-02-02 14:59:59 -08:00
Cole Faust
7940c6a5b5 Remove RBC hints from generated starlark
The hints no longer need to be there in the generated code.

Fixes: 217269465
Test: go test
Change-Id: If2049780a874b42eb9df9351dda4f29c85482470
2022-02-02 11:53:19 -08:00
Cole Faust
f8a4bb6d7f Merge "Allow variable-prefixed include statements in mk2rbc" 2022-02-01 18:44:28 +00:00
Treehugger Robot
6a21d959e5 Merge "Remove variableDefinedExpr" 2022-02-01 04:16:02 +00:00
Cole Faust
069aba64f6 Allow variable-prefixed include statements in mk2rbc
mk2rbc was already searching the whole android tree
for Makefiles, so allowing variable-prefixed include
statements doesn't affect performance on the file searching
front. On the generated code front, there's already a cap
of 150 potentially-included makefiles that prevents the
performance from getting too bad, and it can be lowered
if necessary.

Bug: 213508006
Test: go test
Change-Id: I3a4e81acb3d97bee08ac3dbe63052a274acf5793
2022-01-31 15:35:28 -08:00
Sasha Smundak
845cb29c2b Fix the names of the dynamically inherited modules.
The module name passed to rbld.inherit should be its path without the suffix.

Bug: 215182113
Test: internal
Change-Id: Ic65a5b73037be84f31f8db29f71f793b6c6034bb
2022-01-28 17:41:15 -08:00
Cole Faust
71514c07d2 Remove variableDefinedExpr
VariableDefinedExpr was under-developed, and would not
take into account if a variable was from the globals
or product config dictionary.

It also always emitted `g.get("VARIABLE") != None`, which
is not correct behavior. In this example makefile:

```
MY_VAR :=

ifdef MY_VAR
$(info MY_VAR is defined)
else
$(info MY_VAR is not defined)
endif

ifdef MY_UNDEFINED_VAR
$(info MY_UNDEFINED_VAR is defined)
else
$(info MY_UNDEFINED_VAR is not defined)
endif

MY_VAR ?= true
MY_UNDEFINED_VAR ?= true


$(info MY_VAR after ?= is $(MY_VAR))
$(info MY_UNDEFINED_VAR after ?= is $(MY_UNDEFINED_VAR))


.PHONY: all
all:
	@:
```

We get the output:

MY_VAR is not defined
MY_UNDEFINED_VAR is not defined
MY_VAR after ?= is
MY_UNDEFINED_VAR after ?= is true

So we can see that even if a variable was set, it's considered
not defined if it was set to an empty value. However, ?= works
differently, and does require the != None, so that was left
as is.

Just use a variableRefExpr and rely on the fact
that variables will be truthy if they're defined.

Fixes: 216700361
Test: go test
Change-Id: If8944da2579e8658e3fc4f666b1f3b2815f8c8b1
2022-01-28 19:47:34 +00:00
Alexander Smundak
3f0088944b Merge "Check missing uncoditionally loaded missing modules at runtime" 2022-01-14 17:56:27 +00:00
Sasha Smundak
6bc132aff9 Check missing uncoditionally loaded missing modules at runtime
A potentially inherited (via dynamically calculated path) module may in turn
unconditionally load a module that does no exist in a source tree -- it is
not an error if this potentially inherited module is actually never inherited
because its dynamically calculated path will never reference it. Instead of
emitting an uncoditional `load` for a non-existent file (which is going to fail
in the Starlark parser), emit conditional load and a runtime check.

Fixes: 213922819
Test: internal
Change-Id: I92177878e199a1f00e5f0c4045c0c0daeddd6bdb
2022-01-10 19:58:14 -08:00
Treehugger Robot
35f94eba51 Merge "Consolidate conversions of is-board-platform(2)" 2022-01-11 03:51:21 +00:00
Treehugger Robot
4cfd37e617 Merge "Convert math functions" 2022-01-11 03:28:01 +00:00
Cole Faust
b2e0b60126 Consolidate conversions of is-board-platform(2)
mk2rbc implements conversions for both
is-board-platform(-in-list) and
is-board-platform(-in-list)2. The conversions
are not the same despite the functions being the
same. Make them convert to the same thing.

Bug: 201477826
Test: go test
Change-Id: I688a2f6114e1688c713e42f0f9229cbc5ee6a917
2022-01-07 15:46:58 -08:00
Cole Faust
6c934f6341 Make include_top comments only apply to the next statement
Previously adding an include_top comment would make it apply
for the rest of the file. This was confusing behavior, and
could also cause extra roots to be searched in a situation
like the following:

\#RBC# include_top foo
include $(VAR)/file.mk
\#RBC# include_top bar
include $(VAR)/file.mk

Here the second include would have file.mk from both the foo
and bar directories in its _entry dictionary.

Bug: 213508006
Test: go test
Change-Id: If3430594759bee1390255400fe29b43d77f7b6a6
2022-01-07 15:09:16 -08:00
Cole Faust
b1103e2578 Convert math functions
Converts the following functions from build/make/common/math.mk:
 - math_max
 - math_min
 - math_gt_or_eq
 - math_gt
 - math_lt

Fixes: 213497955
Test: go test
Change-Id: I8c4e8fee321f03b813428fa10038fa8f06188cef
2022-01-07 14:59:55 -08:00
Cole Faust
9ebf6e44df Refactor knownFunctions
By having knownFunctions contain parsing interfaces
instead of just metadata about the function, we can
remove some hardcoded special cases.

Bug: 201700692
Test: go test
Change-Id: I1723c7f50b72d42b03380e999aa9601b455a969e
2022-01-04 14:12:52 -08:00
Cole Faust
69e8731101 Merge "Fix "unknown binary op: string + list" errors" 2021-12-23 18:41:34 +00:00
Cole Faust
0484c2378f Fix "unknown binary op: string + list" errors
Convert lists to strings when adding them to a string.

Bug: 201700692
Test: go test
Change-Id: Iefb68f48191136e7115a6d6bfa0608c73d5afdac
2021-12-22 14:08:08 -08:00
Cole Faust
7321b0985e Improve error message when a required makefile can't be found
Bug: 193566316
Test: go test
Change-Id: I41a28808c95df76f212980ef1e59047312fe587b
2021-12-21 16:11:16 -08:00
Cole Faust
f7ed5343de Prevent duplicate entries in ctx.include_tops
The include_tops hints are global for the whole parse
context (which is probably also something to fix), which
means that if an include_top hint is duplicated there will
be duplicated keys generated in the _entry starlark dictionary.

Bug: 193566316
Test: go test
Change-Id: I01a0546ac9be91ef46c5248e87e1a40e0f211193
2021-12-21 14:34:51 -08:00
Cole Faust
fc43868a5f Remove starlarkExpr.Eval()
It was only used to substitute variable references to
predefined variables with the predefined value, which
is an easy condition to directly parse into instead
of having a separate evalutation pass.

Bug: 201700692
Test: go test
Change-Id: I543d20a1d6435bfabd9faa90ffb09af3084ed28c
2021-12-16 13:20:22 -08:00
Cole Faust
a6628d24c4 Merge "Handle foreach expressions in mk2rbc" 2021-12-16 21:16:49 +00:00
Cole Faust
b0d32ab960 Handle foreach expressions in mk2rbc
Bug: 201700692
Test: go test
Change-Id: Ia23494a63567a1fe2d4bb797a2d4dd5925b3431d
2021-12-15 13:06:47 -08:00
Cole Faust
0e9418ced0 Allow generic $(findstring) calls
Previously they were only allowed in if statements.
Also update the if statement condition to allow comparisons
to the search term.

Bug: 201700692
Test: go test
Change-Id: I8bee8f716819d383423c0de233ab970e49890543
2021-12-13 16:33:25 -08:00
Cole Faust
c36c962670 Handle substitution references in mk2rbc
Bug: 201700692
Test: go test
Change-Id: I5ba5e51848e795e39f1f65dc153e4c1530066860
2021-12-08 17:32:57 -08:00
Cole Faust
4eadba7438 Support if expressions in mk2rbc
Bug: 201700692
Test: go test
Change-Id: Icdf30c4d625d81974db946bd91660a29a0373ac7
2021-12-08 10:00:14 -08:00
Cole Faust
eec0d8137e Fix filter-out being negated in rbc product configuration
Bug: 201700692
Test: go test
Change-Id: I3245e33dc88dd2e456a24780c8cc20ed605d67a6
2021-12-06 16:23:51 -08:00
Cole Faust
864028a71b Support passing input variables to the product configuration
Since rblf_env / rblf_cli are not typed properly, accept
input variables via a file so that they can be converted
with the correct types.

Bug: 201700692
Test: go test
Change-Id: I9b56067cfe396d1bcd8d62c353ff222dd61a6c9f
2021-12-02 16:19:19 -08:00
Cole Faust
b3b28012b9 Merge "Print product vars in board config launcher" 2021-11-23 20:46:45 +00:00
Cole Faust
3c1868bbcc Print product vars in board config launcher
Bug: 201700692
Test: ./build/bazel/ci/rbc_regression_test.sh -b aosp_cf_x86_64_phone-userdebug
Change-Id: I1d3f6f13fc662807db2a86d1daffcc81433d82fc
2021-11-22 16:34:11 -08:00