This CL adds validation tests for all of the existing generated models.
The strategy of this CL is this: given a valid model or request, make a
single change to invalidate the model or request, then verify that the
vendor service driver catches the inconsistency and returns
INVALID_ARGUMENT.
Bug: 67828197
Test: mma
Test: VtsHalNeuralnetworksV1_0TargetTest
Test: VtsHalNeuralnetworksV1_1TargetTest
Change-Id: I8efcdbdccc77aaf78992e52c1eac5c940fc81a03
When calling update-makefiles.sh there are
some unrelated changes that were missed in
previous commits.
Bug: 8675309
Test: compilation
Change-Id: I5bf67fbcc809de36bde1869ada7b835566a5198b
Bug: http://b/74200014
Building this module with asan triggers an internal error in gold.
Test: mma SANITIZE_TARGET=address in
hardware/interfaces/neuralnetworks/1.1/vts/functional
Change-Id: I3ce5352a6c86c8dca257d35af06989e57f871749
There are some NN VTS tests that assume a service is able to generate a
model consisting only of a floating point add operation. However, some
drivers do not support floating point operations. This CL relaxes the
test requirements to allow a test to be skipped if the service does not
support floating point add.
Bug: 72764145
Test: mma
Test: VtsHalNeuralnetworksV1_0TargetTest
Change-Id: I6b0644432680fc2f8098b5187795dc2953df03f9
We have determined that for Android P it is sufficient to have a
mechanism for a developer to specify on a per-model basis that it is
acceptable for FLOAT32 operations to be carried out as if they were
FLOAT16 operations. This CL extends the model to include information on
these precision requirements.
Bug: 63911257
Test: mm
Test: NeuralNetworksTest
Test: VtsHalNeuralnetworksV1_0TargetTest
Change-Id: I5331e69c05c20c588035897322a6c7dceca374a0
- Create VtsHalNeuralnetworksTest_utils module to be shared between
v1.x tests.
- Split the existing tests into two categories: basic, and generated.
- Created v1.1 VTS tests ensuring no regression in existing ML models.
Bug: 63911257
Test: mm
Test: NNAPI VTS tests pass on v1.0 and v1.1 sample drivers
Change-Id: Ic77c90a3a5bbd96b0ce2acd03764dde4b3034cc9
Removing whenever I see these in code reviews.
Test: none
Merged-In: I4322f533a837d55618ec2ed2125e8966ace9d61d
Change-Id: I4322f533a837d55618ec2ed2125e8966ace9d61d
Address comments in ag/2926388
Test: VtsHalNeuralnetworksV1_0TargetTest on Angler with sample driver
Bug: 67597726
Change-Id: Ie3118dcbfb11c25dbc84d2becdefbbae868ff28f
RequestArgument for optional inputs should have its .hasNoValue set
and other fields left empty. The test harness was incorrectly ignoring
the hasNoValue field and initializing the offset field even for missing
arguments. That inconsistency was caught by the sample driver and caused
an invalid argument error. With this CL the test harness now populates
requests correctly for tests with intentionally missing inputs, and that
fixes lstm* and lsh_projection.
Test: VtsHalNeuralnetworksV1_0TargetTest on Angler
Bug: 63905942
Change-Id: I8729124b720f0d4163f4b9efd452f82b825b0885
Some service drivers won't be able to support all operations. In those
cases, try to prepare the model anyway. If it works, proceed with the
test; if it fails, terminate the test early.
For example, the hvx nn service does not support svdf, so it should
attempt to prepare the model, but the test should not fail if the hvx
driver reports an error.
Bug: 63905942
Test: mm, expected behavior when running on hvx service
Change-Id: I51aa1911145f0d17590906c21314d2bdc24cffea
Some service drivers won't be able to support all operations. In those
cases, run the test as far as it can go, but ignore the output data.
For example, the hvx nn service does not support svdf, so it should
attempt to prepare and execute the model, but the test should not fail
if the hvx driver reports an error. However, hvx indicates it is able to
run MAX_POOL_2D, so the test should fail if that output is incorrect.
Bug: 63905942
Test: mm
Test: run vts binary on hvx service
Change-Id: I3ebc05836c8e070b66daa2e38b107f96637ca812
Allow optional operands to be specified as having no value in the model.
Make sure that both these and model arguements that have no value can
be passed to through the HAL. Change the validation to ensure that all
required ops are there.
Bug: 63905942
Test: Compiled and ran runtime/test and VTS tests.
Change-Id: Ia20954b9712042f5c0571822967521aedb66150b
IEvent was a synchronization primitive which caused some confusion
in the interface. Originally the event object was paired with an
asynchronous task, and the asynchronous task would signal this event
when the corresponding output was ready to be used.
In the case of IDevice::prepareModel, the function call would return an
IPreparedModel object that was not guaranteed to be prepared until the
runtime had returned from waiting on the corresponding event object.
The event object has been changed to two explicit callbacks--
IPreparedModelCallback and IExecutionCallback. Now,
IDevice::prepareModel no longer returns an unfinished IPreparedModel;
instead, it will pass the IPreparedModel object to the runtime through
IPreparedModelCallback::notify. When the runtime retreives the
IPreparedModel object, the asynchronous task has already finished
preparing the model.
The two callbacks are used for different purposes. Each has its own
version of notify to pass the data back to the runtime:
* IPreparedModelCallback::notify(ErrorStatus, IPreparedModel)
* IExecutionCallback::notify(ErrorStatus)
Bug: 63905942
Test: mm, vts, ml/nn/runtime/tests
Change-Id: I0c88cd262ba762e0af15e9da31ebe813a5d150b2
Removed operationTuple from Model in the HAL, as the data type was
redundant information.
Removed supportedOperationTuples from Capabilities, as real drivers need
more complex restrictions than this provided. For the OC-MR1, we'll just
rely on getSupportedNodes.
Also removed the unused cachesCompilation.
Bug: 63905942
Test: Compiled and ran tests.
Change-Id: I15f33d14634f2e1c8d726b1bd01d5b9e123b47ea
NDK libc++ does not yet support full C++14. This CL replaced use of
std::get with type as well as auto lambda arguments with equivalent
C++11 constructs in the VTS test harness for NNAPI.
Test: VtsHalNeuralnetworksV1_0TargetTest
Bug: 63905942
Change-Id: If75e7c088e9221a70bcc47bc647e0dd7b045bfa1
This CL adds negative tests to:
* IDevice::getSupportedOperations
* IDevice::prepareModel
* IPreparedModel::execute
It does not make sense to have negative tests for the remaining HAL
entries:
* IDevice::getStatus (all returned values are valid)
* IDevice::getCapabilities (no input or state)
* IEvent::notify (synchronization callback primitive)
Bug: 63905942
Test: mm, running on sample driver has expected output
Change-Id: Ia3cf5ce05dd07ff3f688395b8d0e581851909028
Filtered out outputs that are not needed for correctness verification
during result validation.
Bug: 63905942
Test: VtsHalNeuralnetworksV1_0TargetTest on Angler
Change-Id: Ic967a821c2a60194c9d9f386601a865f5c409a4f
Tests like RNN needs slightly more relaxed tolerance than 4ULP.
Use 1e-5 from corresponding TFLite tests.
Bug: 63905942
Test: VtsHalNeuralnetworksV1_0TargetTest on Angler
Change-Id: I721711e13575f56e956d87e3ad537467f85e93f4
Created initial test harness for test models and examples from
NNAPI test generator in VtsHalNeuralnetworksV1_0TargetTest. As
an example, also added a test generated from test spec at
frameworks/ml/nn/tools/test_generator/tests/P_vts_full/.
Generated model setup code and examples are from:
frameworks/ml/nn/runtime/test/generated/examples and
frameworks/ml/nn/runtime/test/generated/vts_models respectively.
Bug: 63905942
Bug: 63525563
Test: VtsHalNeuralnetworksV1_0TargetTest with sample driver enabled
by cherry-pick
Change-Id: Ief029eed9718c8724ef0b64fc6a7f6b9a7bc7b7b
Use an explicit description of the lifetime of an operand
rather than relying on poolIndex to do the job.
Bug: 63905942
Test: Ran unit tests
Change-Id: I53c4c10b40fd3312232b0689b43d0f72a2818490
- Add "_2D" to CONV, DEPTHWISE_CONV, AVG_POOL, MAX_POOL, and L2_POOL,
since they are only supposed to do 2D computation.
- Remove CAST, GATHER, SPLIT.
- Also update the tests to reflect the changes.
Bug: 63905942
Test: mm
Test: NeuralNetworksTests pass for both CPU path and sample hidl driver.
Change-Id: I5bcf219c4c6cd4f866c1023d1f35cdc83ca69a85
The NNAPI requires requests on a model to be asynchronously
processed. This CL implements a basic Event that can later
be used to block the runtime thread until the asynchronous
request has completed.
Bug: 63905942
Test: VtsHalNeuralnetworksV1_0TargetTest (32-bit, 64-bit) with sample driver enabled by cherry-pick
frameworks/ml/nn/runtime/test with and without sample driver enabled
Change-Id: Ie27a574aaaac312e7cbb731750f9c06278357a1c
* For registerTestService/getServiceName, no need to pass
any hard coded string of HAL service FQName.
* Affect test: VtsHalCameraProviderV2_4TargetTest,
VtsHalGraphicsComposerV2_1TargetTest,
VtsHalNeuralnetworksV1_0TargetTest
Bug: 62946472
Bug: 64203181
Test: make vts
vts-tradefed run vts -m VtsHalCameraProviderV2_4Target
Merged-In: If365ab2ed9a91eb4013d71769804b9d4bf089d66
Change-Id: Id0bddbc2949337147557f45cc60dbfaa114ce25e
(cherry picked from commit d71b654d6d)
