So that it's easier to see what actually has changed in VTS tests for
version 1.3
Bug: 139120468
Test: m
Change-Id: I09797f5f3898501a008186a22dd411b00e9e2c67
Merged-In: I09797f5f3898501a008186a22dd411b00e9e2c67
(cherry picked from commit 3b13b55ac1)
The frameworks/ml/nn CL "Improve performance of Burst executions"
changed the signature of a utility function used by NNAPI's VTS tests.
This CL modifies the test accordingly.
Bug: 132073143
Test: mma
Test: VtsHalNeuralnetworksV1_2TargetTest (with sample-all)
Test: VtsHalNeuralnetworksV1_3TargetTest (with sample-all)
Change-Id: I56c3b04e6fec11d526e2a1ff5b896f31cb12eb5f
After this update, types.hal v1.3 no longer refers to API level 30.
Test: none needed
Bug: 140132458
Change-Id: If707dfbcade6be1a0885fc21f6ddf47f3c27b244
So that it's easier to see what actually has changed in VTS tests for
version 1.3
Bug: 139120468
Test: m
Change-Id: Ief294d21349ca6531595612a16fa3ae3382f83ac
Merged-In: Ief294d21349ca6531595612a16fa3ae3382f83ac
(cherry picked from commit 3b13b55ac1)
There are certain pieces of NeuralNetworks.h and of our various *.hal
files that ought to be kept in sync -- most notably the operand type
and operation type definitions and descriptions in our
NeuralNetworks.h and types.hal files. To avoid having to do this
manually, a tool can be employed.
The old solution to this problem is sync_enums_to_hal.py, which parses
the existing NeuralNetworks.h and types.hal files, and then rewrites
the types.hal files: It identifies the operand type and operation type
portions of the files, and in the types.hal files replaces them with
transformed versions of the portions from NeuralNetworks.h. This
approach is brittle -- it is very sensitive to the structure of the
files, as was recognized when this tool was created. Changes to those
files since the script was last updated essentially broke the script,
as noted in http://b/140132458.
The new solution employs a new tool generate_api.py to combine a
single "specification file" with one "template file" per API file
(NeuralNetworks.h or types.hal) to produce that API file. The script
generate_api.sh invokes generate_api.py once per API file, passing
appropriate arguments. See frameworks/ml/nn/tools/api/README.md for
more details.
In the process of combining information from NeuralNetworks.h and
*/types.hal, some formatting, syntactic, and semantic changes have
been made to those files. For example:
- types.hal files no longer refer to API levels or (similarly)
to HAL versions other than earlier HAL versions
Bug: 130169064
Bug: 140132458
Test: cd neuralnetworks ; mma
Change-Id: Iceb38effc2c7e6c32344e49225b85256144c0945
Merged-In: Iceb38effc2c7e6c32344e49225b85256144c0945
(cherry picked from commit 2cae5c8b01)
* changes:
Modify NNAPI VTS tests to run on version 1.3
Copy VTS tests from v1.2 to v1.3
Create NNAPI HAL v1.3 and add TENSOR_QUANT8_ASYMM_SIGNED OperandType
There are certain pieces of NeuralNetworks.h and of our various *.hal
files that ought to be kept in sync -- most notably the operand type
and operation type definitions and descriptions in our
NeuralNetworks.h and types.hal files. To avoid having to do this
manually, a tool can be employed.
The old solution to this problem is sync_enums_to_hal.py, which parses
the existing NeuralNetworks.h and types.hal files, and then rewrites
the types.hal files: It identifies the operand type and operation type
portions of the files, and in the types.hal files replaces them with
transformed versions of the portions from NeuralNetworks.h. This
approach is brittle -- it is very sensitive to the structure of the
files, as was recognized when this tool was created. Changes to those
files since the script was last updated essentially broke the script,
as noted in http://b/140132458.
The new solution employs a new tool generate_api.py to combine a
single "specification file" with one "template file" per API file
(NeuralNetworks.h or types.hal) to produce that API file. The script
generate_api.sh invokes generate_api.py once per API file, passing
appropriate arguments. See frameworks/ml/nn/tools/api/README.md for
more details.
In the process of combining information from NeuralNetworks.h and
*/types.hal, some formatting, syntactic, and semantic changes have
been made to those files. For example:
- types.hal files no longer refer to API levels or (similarly)
to HAL versions other than earlier HAL versions
Bug: 130169064
Bug: 140132458
Test: cd neuralnetworks ; mma
Change-Id: Iceb38effc2c7e6c32344e49225b85256144c0945
So that it's easier to see what actually has changed in VTS tests for
version 1.3
Bug: 139120468
Test: m
Change-Id: Ief294d21349ca6531595612a16fa3ae3382f83ac
This CL removes a dependency on the VTS test runner by dynamically
discovering all NN HAL service instances in the gtest binary itself,
and runs through all service instances with parameterized tests.
This CL converts TEST_F cases to TEST_P cases, where the test parameter
is the name of the service instance. For existing TEST_P cases (such as
the generated test cases), the service instance name is made to be the
first test parameter.
This CL enables the NN VTS tests to be more portable, e.g., they can
run directly as a presubmit test.
Fixes: 124540002
Test: mma
Test: VtsHalNeuralnetworksV1_*TargetTest (with sample-all)
Test: cd $ANDROID_BUILD_TOP/hardware/interfaces/neuralnetworks && atest
Change-Id: I1e301d7c9f9342bb8f35a267bef180f510944b19
Merged-In: I1e301d7c9f9342bb8f35a267bef180f510944b19
(cherry picked from commit 7076f629b7)
This CL additionally adds missing documentation for timing
information when returning results on the resultChannel.
Bug: 133773876
Test: mma
Change-Id: I1eb1affbb4a912d5fdeab012e2be7e7005deb04d
Merged-In: I1eb1affbb4a912d5fdeab012e2be7e7005deb04d
(cherry picked from commit f99bffd08a)
This CL removes a dependency on the VTS test runner by dynamically
discovering all NN HAL service instances in the gtest binary itself,
and runs through all service instances with parameterized tests.
This CL converts TEST_F cases to TEST_P cases, where the test parameter
is the name of the service instance. For existing TEST_P cases (such as
the generated test cases), the service instance name is made to be the
first test parameter.
This CL enables the NN VTS tests to be more portable, e.g., they can
run directly as a presubmit test.
Fixes: 124540002
Test: mma
Test: VtsHalNeuralnetworksV1_*TargetTest (with sample-all)
Test: cd $ANDROID_BUILD_TOP/hardware/interfaces/neuralnetworks && atest
Change-Id: I1e301d7c9f9342bb8f35a267bef180f510944b19
This CL additionally adds missing documentation for timing
information when returning results on the resultChannel.
Bug: 133773876
Test: mma
Change-Id: I1eb1affbb4a912d5fdeab012e2be7e7005deb04d
Under certain circumstances, we guarantee that a prepared model can be
executed successfully. In describing those circumstances, we
neglected to specify that operation input operands must have legal
values for the guarantee to hold. For example, the guarantee doesn't
hold if an ADD operation has an activation input that is not one of
the defined values; or if a RESHAPE operation has a shape input in
which two or more components are -1.
This change modifies the guarantee to apply only when operation input
operands have legal values. It also documents this guarantee for
burst execution.
Note that if an operation has an input operand that can be proven to
have an illegal value at preparation time (e.g., a constant value that
is illegal), model preparation might (but is not required to) fail for
that reason.
Bug: 135933040
Test: $ cd neuralnetworks ; mma
Change-Id: I8b421550dd89e4bbbdae899e7cb5e9e88a46d2fb
(cherry picked from commit 48544cc38a)
This CL does additional NNAPI VTS test cleanup, including consolidating
duplicate functionality. Specifically, this CL:
* consolidates the createPreparedModel function, removing the duplicate
* consolidates the std::out ErrorStatus and DeviceStatus code into Utils
* changes non-null constant pointers to constant references
* removes redudant leading namespace specifiers (V1_0::, ::testing, etc.)
* makes the Valdiation tests free functions
* renames device to kDevice and mTestModel to kTestModel
Bug: N/A
Test: mma
Test: VtsHalNeuralnetworksV1_*TargetTest (with sample-all)
Change-Id: Ic401bb1f1760cc10384ac0d30c0c93409b63a9c7
Merged-In: Ic401bb1f1760cc10384ac0d30c0c93409b63a9c7
(cherry picked from commit e16af0a44b)
Under certain circumstances, we guarantee that a prepared model can be
executed successfully. In describing those circumstances, we
neglected to specify that operation input operands must have legal
values for the guarantee to hold. For example, the guarantee doesn't
hold if an ADD operation has an activation input that is not one of
the defined values; or if a RESHAPE operation has a shape input in
which two or more components are -1.
This change modifies the guarantee to apply only when operation input
operands have legal values. It also documents this guarantee for
burst execution.
Note that if an operation has an input operand that can be proven to
have an illegal value at preparation time (e.g., a constant value that
is illegal), model preparation might (but is not required to) fail for
that reason.
Bug: 135933040
Test: $ cd neuralnetworks ; mma
Change-Id: I8b421550dd89e4bbbdae899e7cb5e9e88a46d2fb
This CL does additional NNAPI VTS test cleanup, including consolidating
duplicate functionality. Specifically, this CL:
* consolidates the createPreparedModel function, removing the duplicate
* consolidates the std::out ErrorStatus and DeviceStatus code into Utils
* changes non-null constant pointers to constant references
* removes redudant leading namespace specifiers (V1_0::, ::testing, etc.)
* makes the Valdiation tests free functions
* renames device to kDevice and mTestModel to kTestModel
Bug: N/A
Test: mma
Test: VtsHalNeuralnetworksV1_*TargetTest (with sample-all)
Change-Id: Ic401bb1f1760cc10384ac0d30c0c93409b63a9c7
Fixes: 123092187
Test: mma all VTS
Test: Attempt to modify test enum value,
confirm that there is build error with proper error message.
Change-Id: I0129378c87637b33bb145ec93307634273a9d2fd
Merged-In: I0129378c87637b33bb145ec93307634273a9d2fd
(cherry picked from commit cbfa6b998e)
This CL is to modify validateBurstSanitized implementation in accordance
with the new test generator and test harness change in the same topic.
Bug: 123092187
Test: 1.2 VTS
Change-Id: Iacfe0e3bba8b4b44b7aee023d1c8129cbd089d4a
Merged-In: Iacfe0e3bba8b4b44b7aee023d1c8129cbd089d4a
(cherry picked from commit 8fc4622bf1)
Comparing with v1.1, the converter for 1.2 HIDL model has additional support
for extraParam, dynamic output shape, and zero-sized output.
Modify CompilationCachingTests to use the new test struct.
Bug: 123092187
Bug: 138718240
Test: All VTS
Change-Id: I54ac97f62898e47a338b51cc6d895a0309ab001f
Merged-In: I54ac97f62898e47a338b51cc6d895a0309ab001f
(cherry picked from commit 491b0a8913)
This CL is very similar to the 1.0 VTS CL. The only difference is that
1.1 Models have an additional field for relaxed model computation.
Bug: 123092187
Bug: 138718240
Test: All VTS
Change-Id: I9264e5b2468b9c6db47d86683d24f8c2c5ec46aa
Merged-In: I9264e5b2468b9c6db47d86683d24f8c2c5ec46aa
(cherry picked from commit 6aad040278)
Comparing with v1.1, the converter for 1.2 HIDL model has additional support
for extraParam, dynamic output shape, and zero-sized output.
Modify CompilationCachingTests to use the new test struct.
Bug: 123092187
Bug: 138718240
Test: All VTS
Change-Id: I54ac97f62898e47a338b51cc6d895a0309ab001f
Merged-In: I54ac97f62898e47a338b51cc6d895a0309ab001f
(cherry picked from commit 491b0a8913)
This CL is very similar to the 1.0 VTS CL. The only difference is that
1.1 Models have an additional field for relaxed model computation.
Bug: 123092187
Bug: 138718240
Test: All VTS
Change-Id: I9264e5b2468b9c6db47d86683d24f8c2c5ec46aa
Merged-In: I9264e5b2468b9c6db47d86683d24f8c2c5ec46aa
(cherry picked from commit 6aad040278)
Fixes: 123092187
Test: mma all VTS
Test: Attempt to modify test enum value,
confirm that there is build error with proper error message.
Change-Id: I0129378c87637b33bb145ec93307634273a9d2fd
This CL is to modify validateBurstSanitized implementation in accordance
with the new test generator and test harness change in the same topic.
Bug: 123092187
Test: 1.2 VTS
Change-Id: Iacfe0e3bba8b4b44b7aee023d1c8129cbd089d4a
Comparing with v1.1, the converter for 1.2 HIDL model has additional support
for extraParam, dynamic output shape, and zero-sized output.
Modify CompilationCachingTests to use the new test struct.
Bug: 123092187
Bug: 138718240
Test: All VTS
Change-Id: I54ac97f62898e47a338b51cc6d895a0309ab001f
This CL is very similar to the 1.0 VTS CL. The only difference is that
1.1 Models have an additional field for relaxed model computation.
Bug: 123092187
Bug: 138718240
Test: All VTS
Change-Id: I9264e5b2468b9c6db47d86683d24f8c2c5ec46aa
Implement converter utilities constructing HIDL model and request from
TestModel.
Bug: 123092187
Bug: 138718240
Test: All VTS
Change-Id: I0b26b7f41d31d5e63ed083ab5f6f269a3620f034
See change I2c0366fb87c96851fa6e0f8fe9ceac012d8e3513
Bug: 136097638
Test: m VtsHalNeuralnetworksV1_0TargetTest
Test: m VtsHalNeuralnetworksV1_1TargetTest
Test: m VtsHalNeuralnetworksV1_2TargetTest
Test: m VtsHalNeuralnetworksV1_2CompatV1_0TargetTest
Change-Id: I6fdede028422145d313d46532b5d2154ef0d40bc
Merged-In: I6fdede028422145d313d46532b5d2154ef0d40bc
(cherry picked from commit 2bcfdc82a0)
Prior to this CL, HalInterfaces.h polluted the global namespace through
the "using" declarations. To fix b/72880287, those names were put in a
new ::android::nn::hal namespace. This CL fixes the compilation errors
that occur due to this cleanup.
Bug: 72880287
Test: mma
Change-Id: Idb228e37615af7c0343a09f9a51de378e63bdb5a
Merged-In: Idb228e37615af7c0343a09f9a51de378e63bdb5a
(cherry picked from commit 4745a3d96e)
See change I2c0366fb87c96851fa6e0f8fe9ceac012d8e3513
Bug: 136097638
Test: m VtsHalNeuralnetworksV1_0TargetTest
Test: m VtsHalNeuralnetworksV1_1TargetTest
Test: m VtsHalNeuralnetworksV1_2TargetTest
Test: m VtsHalNeuralnetworksV1_2CompatV1_0TargetTest
Change-Id: I6fdede028422145d313d46532b5d2154ef0d40bc
The VTS Callback files are a subset of the Callback files in
frameworks/ml/nn/runtime/Callbacks.*. This CL syncs the implementations,
removing the functionality that is not needed in VTS.
Fixes: 132322149
Test: mma
Test: VtsHalNeuralnetworksV1_0TargetTest
Test: VtsHalNeuralnetworksV1_1TargetTest
Test: VtsHalNeuralnetworksV1_2TargetTest
Change-Id: I114ce7f3b6c3d58de0196e9508209614d0a73e11
Merged-In: I114ce7f3b6c3d58de0196e9508209614d0a73e11
(cherry picked from commit 23d0e562e0)
Having separate VTS models and tests for difference HAL version. This
compiler flag is no longer needed.
Bug: 122740334
Test: All VTS 1.x and 1.xCompat1.y
Change-Id: Idbd96a954da9bd5a0e0e66afd9120d84a1efc784
Merged-In: Idbd96a954da9bd5a0e0e66afd9120d84a1efc784
(cherry picked from commit 7763d53545)
To make it easier to create the next version of NNAPI, this change
removes the following nonsensical dependence:
- NNAPI 1.0 VTS depends on NNAPI 1.1 and 1.2
- NNAPI 1.1 VTS depends on NNAPI 1.2
In particular, I made the following changes:
- split GeneratedTestHarness.cpp into three separate implementations,
- created a restricted version of Callbacks.h for 1.0 and 1.1,
- removed the dependency on frameworks/ml/nn/HalInterfaces.h,
- refactored Android.bp files for more autonomy between 1.0, 1.1, and 1.2,
- consolidated some common code into Utils.h,
- created structure for sharing code between VTS versions (VtsHalNeuralNetworksV1_0_utils).
Bug: 74827824
Bug: 124462414
Test: VtsHalNeuralnetworksV1_0TargetTest
Test: VtsHalNeuralnetworksV1_1TargetTest
Test: VtsHalNeuralnetworksV1_1CompatV1_0TargetTest
Test: VtsHalNeuralnetworksV1_2TargetTest
Test: VtsHalNeuralnetworksV1_2CompatV1_0TargetTest
Test: VtsHalNeuralnetworksV1_2CompatV1_1TargetTest
Change-Id: I4243d0b5e574255cef1070850f4d0a284f65f54e
Merged-In: I4243d0b5e574255cef1070850f4d0a284f65f54e
(cherry picked from commit 1d6b465997)
Prior to this CL, HalInterfaces.h polluted the global namespace through
the "using" declarations. To fix b/72880287, those names were put in a
new ::android::nn::hal namespace. This CL fixes the compilation errors
that occur due to this cleanup.
Bug: 72880287
Test: mma
Change-Id: Idb228e37615af7c0343a09f9a51de378e63bdb5a
The VTS Callback files are a subset of the Callback files in
frameworks/ml/nn/runtime/Callbacks.*. This CL syncs the implementations,
removing the functionality that is not needed in VTS.
Fixes: 132322149
Test: mma
Test: VtsHalNeuralnetworksV1_0TargetTest
Test: VtsHalNeuralnetworksV1_1TargetTest
Test: VtsHalNeuralnetworksV1_2TargetTest
Change-Id: I114ce7f3b6c3d58de0196e9508209614d0a73e11
Having separate VTS models and tests for difference HAL version. This
compiler flag is no longer needed.
Bug: 122740334
Test: All VTS 1.x and 1.xCompat1.y
Change-Id: Idbd96a954da9bd5a0e0e66afd9120d84a1efc784
To make it easier to create the next version of NNAPI, this change
removes the following nonsensical dependence:
- NNAPI 1.0 VTS depends on NNAPI 1.1 and 1.2
- NNAPI 1.1 VTS depends on NNAPI 1.2
In particular, I made the following changes:
- split GeneratedTestHarness.cpp into three separate implementations,
- created a restricted version of Callbacks.h for 1.0 and 1.1,
- removed the dependency on frameworks/ml/nn/HalInterfaces.h,
- refactored Android.bp files for more autonomy between 1.0, 1.1, and 1.2,
- consolidated some common code into Utils.h,
- created structure for sharing code between VTS versions (VtsHalNeuralNetworksV1_0_utils).
Bug: 74827824
Bug: 124462414
Test: VtsHalNeuralnetworksV1_0TargetTest
Test: VtsHalNeuralnetworksV1_1TargetTest
Test: VtsHalNeuralnetworksV1_1CompatV1_0TargetTest
Test: VtsHalNeuralnetworksV1_2TargetTest
Test: VtsHalNeuralnetworksV1_2CompatV1_0TargetTest
Test: VtsHalNeuralnetworksV1_2CompatV1_1TargetTest
Change-Id: I4243d0b5e574255cef1070850f4d0a284f65f54e
Bug: 121347610 document that NNAPI Execution inputs/outputs and HAL Request inputs/outputs must not be modified
Test: cd hardware/interfaces/neuralnetworks/1.0/vts/functional ; mma
Test: cd hardware/interfaces/neuralnetworks/1.2/vts/functional ; mma
Change-Id: Iac71d6d5ad92a90afd1b6babb7cfa128d7484c64
We don't have a framework for handcrafting tests resulting in execution
failure. To produce this test, I created
strided_slice_invalid_output_dims.mod.py and used our VTS test generator
to create the corresponding example.cpp and model.cpp files. I put those
under the new directory neuralnetworks/1.2/vts/functional/generated to
distinguish them from the streamlined generated tests.
Bug: 79856511
Bug: 132155416
Test: VtsHalNeuralnetworksV1_2TargetTest
Change-Id: Ib331778605c2e3828195e1e7d1651091873985a0
Merged-In: Ib331778605c2e3828195e1e7d1651091873985a0
(cherry picked from commit 1a18c08052)
