Commit graph

671 commits

Author SHA1 Message Date
TreeHugger Robot
36e9e40436 Merge "Clarify NNAPI control flow shape constraints" into rvc-dev 2020-06-26 12:07:06 +00:00
Slava Shklyaev
709842ab98 Clarify NNAPI control flow shape constraints
Bug: 156918813
Bug: 158557728
Test: m
Change-Id: I4c2b83e11b27b791b3aa624be68474011b92a269
2020-06-23 13:08:05 +01:00
Xusong Wang
12f483fdfd Merge "Revert "Allow compilations in TOCTOU tests to fail with GENERAL_FAILURE."" into rvc-dev 2020-06-23 00:53:05 +00:00
Xusong Wang
81fa7605cf Revert "Allow compilations in TOCTOU tests to fail with GENERAL_FAILURE."
This reverts commit 362dfd64d5.

Bug: 157489048
Test: 1.2/1.3 VTS TOCTOU tests
Change-Id: Ifdf4a8fab129cd90a3be019c3c598d980b785aaa
2020-06-22 18:36:07 +00:00
Slava Shklyaev
a6d65ad073 Merge "Log NNAPI VTS test case to logcat" into rvc-dev 2020-06-17 12:08:45 +00:00
Slava Shklyaev
8c8efd68e8 Log NNAPI VTS test case to logcat
Sample output:

    I VtsHalNeuralnetworksV1_2TargetTest: [Test Case] TestGenerated/GeneratedTest.Test/nnapi_sample_all_add BEGIN
    ...
    I VtsHalNeuralnetworksV1_2TargetTest: [Test Case] TestGenerated/GeneratedTest.Test/nnapi_sample_all_add END

Bug: 159060565
Test: VtsHalNeuralnetworksV1_0TargetTest (inspected logcat output)
Test: VtsHalNeuralnetworksV1_1TargetTest (inspected logcat output)
Test: VtsHalNeuralnetworksV1_2TargetTest (inspected logcat output)
Test: VtsHalNeuralnetworksV1_3TargetTest (inspected logcat output)
Change-Id: Ic1a0c90e0853ca84dd925d0f7df65e159bb284aa
2020-06-17 12:10:00 +01:00
TreeHugger Robot
c55a62bc79 Merge "Remove temporary test filters" into rvc-dev 2020-06-16 23:15:47 +00:00
Miao Wang
eeb0068015 Remove temporary test filters
Bug: 155577050
Test: mm
Change-Id: Ibe8a3ed79f640249ba54f94bdde9649f156383ea
2020-06-16 15:31:24 +00:00
Xusong Wang
362dfd64d5 Allow compilations in TOCTOU tests to fail with GENERAL_FAILURE.
A compilation failure is not related to the security aspect of the
TOCTOU test, but it will skip one iteration of security testing. This CL
allows the compilation to fail with GENERAL_FAILURE in TOCTOU tests, and
issues a retry once it happens to ensure enough test coverage.

Bug: 157489048
Test: 1.2/1.3 VTS
Change-Id: Idc88e0365c5d2799187093b6fd7b4abf8f8b463d
2020-06-15 13:22:27 -07:00
Xusong Wang
d454751e00 Add checks of output shapes when OUTPUT_INSUFFICIENT_SIZE.
Add checks that all returned output dimensions must be at
least as fully specified as the union of the information about the
corresponding operand in the model and in the request.

Bug: 154054474
Test: VTS
Change-Id: I934d084c7665160a98da9828604ce8297fef73b8
2020-05-26 15:08:12 -07:00
Lev Proleev
614ba3ef2e Merge "NNAPI: Clarify behaviour of ops that reduce dimensions" into rvc-dev 2020-05-18 15:31:13 +00:00
Miao Wang
f118c86ba2 Merge "Add AndroidTest.xml for NNAPI 1.3 VTS and temporarily blacklist failing tests" into rvc-dev 2020-05-15 22:33:33 +00:00
Miao Wang
338acf5c15 Add AndroidTest.xml for NNAPI 1.3 VTS and temporarily blacklist failing tests
- Also adds TEST_MAPPING filters here to make sure sample driver tests
  are still being filtered out.
  - There may still be failing tests, but should not cause OOM in test harness.

Bug: 156691406
Test: mm
Test: atest VtsHalNeuralnetworksV1_3TargetTest
Change-Id: Ie05350d71dede04967d94dad4147bd3ed3435f7d
2020-05-15 13:46:13 -07:00
Lev Proleev
74b04e9649 Merge "NNAPI: Clarify behaviour of REDUCE_* ops" into rvc-dev 2020-05-15 10:28:39 +00:00
Lev Proleev
d4bba83f6a NNAPI: Clarify behaviour of ops that reduce dimensions
Clarify behaviour of the following ops in the case when inputs would
cause an output to have an empty shape:
* MEAN
* ARGMIN/ARGMAX
* STRIDED_SLICE

Bug: 155508675
Bug: 155660285
Bug: 155508675
Bug: 155238914
Test: VtsHalNeuralnetworksV1_3TargetTest
Change-Id: I92d1c3866a462a99b4eed7782b72f62bb6a076c6
2020-05-13 21:57:58 +01:00
Lev Proleev
eec7d86b44 NNAPI: Clarify behaviour of REDUCE_* ops
Add a clarification on the behaviour of REDUCE_* ops when all dimensions
of an input tensor are reduced.

Bug: 155508675
Test: VtsHalNeuralnetworksV1_3TargetTest
Change-Id: I3a67225c3af80af40a11362ca5d0f55350cbfa05
2020-05-13 15:00:32 +01:00
Miao Wang
4429d856cd Filter out failing NNAPI validation tests
Bug: 155577050
Test: atest VtsHalNeuralnetworksV1_*TargetTest
Change-Id: Id8d497dddd5521670689ff2ea08610c3aaeaf748
2020-05-10 19:38:17 -07:00
Miao Wang
8ebe2a04c5 Add AndroidTest.xml for NNAPI 1.0 VTS and blacklist failing tests
- Also adds TEST_MAPPING filters here to make sure sample driver tests
  are still being filtered out.

Bug: 155577050
Test: mm
Test: atest VtsHalNeuralnetworksV1_0TargetTest
Change-Id: Ic8f556a957e6af340d500e2c4696b41b2a6e8516
2020-05-05 15:28:57 -07:00
Miao Wang
f7c55d5f9b Add AndroidTest.xml for NNAPI 1.2 VTS and blacklist failing tests
- Also adds TEST_MAPPING filters here to make sure sample driver tests
  are still being filtered out.

Bug: 155674368
Test: mm
Test: atest VtsHalNeuralnetworksV1_2TargetTest
Change-Id: I5f5d272742d4cdd6a8cc87ec035b5e2508eaa98f
2020-05-04 20:10:48 -07:00
David Gross
12ab6c57cc More tests for graph validation.
- detect cycle (CycleTest)
- detect bad execution order (mutateExecutionOrderTest)
- detect lifetime inconsistent with whether operand is written (mutateOperandLifeTimeTest)
- detect lifetime inconsistent with Model inputIndexes/outputIndexes (mutateOperandInputOutputTest)
- detect incorrect number of consumers (mutateOperandNumberOfConsumersTest)
- detect operand written multiple times (mutateOperandAddWriterTest)
- detect operand never written (mutateOperationRemoveWriteTest)

Bug: 66478689
Test: VtsHalNeuralnetworksV1_*TargetTest

Change-Id: Id4ba19660bbd31a16f8a675f7b6437f4d779e8da
Merged-In: Id4ba19660bbd31a16f8a675f7b6437f4d779e8da
(cherry picked from commit af51663e99)
2020-05-02 02:34:30 +00:00
Slava Shklyaev
baa4cf02f2 Exclude unused operands from removeOperandTest
Bug: 148208229
Test: TestGenerated/ValidationTest.Test/nnapi_sample*_while*_unused_output*
Change-Id: I39bebefd0e40d370882d614d4beb97ab36c9da8f
2020-04-16 15:03:26 +01:00
Xusong Wang
d923a3f06f Merge "Remove BLOB AHWB tests from 1.2 VTS." into rvc-dev 2020-04-10 17:44:43 +00:00
Xusong Wang
b642700c7f Remove BLOB AHWB tests from 1.2 VTS.
Bug: 153282844
Test: 1.2 VTS
Change-Id: I866fe1388a64a36372f90c60fef340d2fe4c8682
2020-04-09 14:26:54 -07:00
TreeHugger Robot
cb141698c1 Merge "Update loopTimeoutDuration documentation in NNAPI" into rvc-dev 2020-04-09 20:30:43 +00:00
Dan Shi
aad51fa000 Rename vts-core to vts
Bug: 151896491
Test: local build
Exempt-From-Owner-Approval: This CL update suite name vts-core to vts as
the suite name is updated. This CL won't change test logic or behavior.

Change-Id: I562b4dc50765e953800a814a8fd84a01c1b9352b
Merged-In: I562b4dc50765e953800a814a8fd84a01c1b9352b
2020-04-07 16:37:40 -07:00
Slava Shklyaev
6a222a3eec Update loopTimeoutDuration documentation in NNAPI
Bug: 149686804
Test: m
Change-Id: I0096a2fc2e579d39a2d59a605a4217d53a381691
2020-04-07 13:24:15 +01:00
Steven Moreland
6ca8276e8b Add VtsHalNeuralnetworksV1_3TargetTest to vts.
Bug: 139438327
Test: N/A
Change-Id: I48342f6e242735e3ee693a7f2bdbe3334a072f81
2020-04-02 14:43:49 -07:00
Miao Wang
077c0339d4 Update the specification for the following operations
- RANDOM_MULTINOMIAL
 - L2_NORMALIZATION

Bug: 136279892
Bug: 140177375
Test: mm
Change-Id: Iab38ef29ebf6d1f5c0a408436b3d564e45e537a0
2020-03-25 16:46:13 -07:00
Michael Butler
3c8593c0fe Merge "Remove extra tests from NNAPI VTS validation tests" into rvc-dev 2020-03-23 22:07:32 +00:00
Miao Wang
1be73e85a1 Merge "Fix the timing initialization error for failed executeFenced case" into rvc-dev 2020-03-23 20:47:22 +00:00
Miao Wang
ad7678c8cb Fix the timing initialization error for failed executeFenced case
Fixes: 152075771
Test: mm
Test: VtsHalNeuralnetworksV1_3TargetTest
Change-Id: I5829397346354ee3fc4a58b0a418197a1eee47cc
2020-03-23 20:46:58 +00:00
Xusong Wang
e71fa3a11a Merge "Add BLOB AHWB tests in VTS." into rvc-dev 2020-03-21 06:38:36 +00:00
Xusong Wang
9c415917e0 Add fenced compute path to memory domain validation test.
Bug: 147777318
Test: 1.3 VTS
Change-Id: I0b731d10384ef2024241af1d908acf3ba760d73f
2020-03-20 14:49:54 -07:00
Xusong Wang
fed2f5213f Add memory domain VTS validation tests.
Bug: 147777318
Test: 1.3 VTS with sample driver
Change-Id: Ia2097345924726d8fb627845fd7438cc3eb35eb6
2020-03-20 13:56:40 -07:00
Michael Butler
678a10600c Remove extra tests from NNAPI VTS validation tests
Do not run validation on "inputs_as_internal" and
"all_tensors_as_inputs" variants.

Bug: 138149072
Bug: 149840439
Test: mma
Test: VtsHalNeuralnetworksV1_*TargetTest
Change-Id: I0699ed6703e48b6c4bc0e7a392b79c12770f04c9
2020-03-19 18:37:23 -07:00
Xusong Wang
962fdb26ca Exclude invalid test models from QuantizationCouplingTest.
Bug: 151674996
Test: 1.3 VTS
Change-Id: I381eb6d39926054279f1791bd71d28a963acc0b9
2020-03-16 17:13:41 -07:00
Xusong Wang
41adc5bc11 Add BLOB AHWB tests in VTS.
Bug: 149847930
Test: 1.3 VTS
Change-Id: I9c795dcb7696c843afd12551927463c5529a4b60
2020-03-13 20:04:38 -07:00
TreeHugger Robot
140597c6a0 Merge "Fix VTS of fenced execution with zero-sized output." into rvc-dev 2020-03-14 02:16:31 +00:00
Xusong Wang
5882d6d911 Fix VTS of fenced execution with zero-sized output.
Fixes: 151441390
Test: 1.3 VTS
Change-Id: I100329e18c34c377d217af155c2abc8e67078778
Merged-In: I100329e18c34c377d217af155c2abc8e67078778
(cherry picked from commit 9fcccf8b85)
2020-03-13 15:15:39 -07:00
Michael Butler
b715967a15 Merge "NNAPI VTS: Add validation for Priority" into rvc-dev 2020-03-13 22:13:12 +00:00
Michael Butler
f0d0616d7c Merge "NNAPI VTS: decouple 1.2 tests from 1.3 types" into rvc-dev 2020-03-13 20:04:25 +00:00
Michael Butler
4876af1ba1 NNAPI VTS: decouple 1.2 tests from 1.3 types
Bug: 141718368
Test: mma
Test: VtsHalNeuralnetworksV1_*TargetTest
Change-Id: Ied7b95e146e4d2d95622642bfbb959cf134b5a7a
2020-03-13 19:59:43 +00:00
Michael Butler
da1a692880 NNAPI VTS: Add validation for Priority
This CL also cleans up the validation framework code.

Bug: 67828197
Test: mma
Test: VtsHalNeuralnetworksV1_*TargetTest
Change-Id: I84661fb2b8204148788d10425ca0ac986158b15f
2020-03-12 11:29:08 -07:00
Lev Proleev
f161cc74a1 Clarify supported ranks for new ops in R
Fix: 151108533
Test: mma
Change-Id: I85b66ab5940d09b2786ecb14bff5bc1c1dee894a
2020-03-12 13:34:18 +00:00
Miao Wang
fa161c7a0a Add tests to make sure executeFenced validate unspecified output shapes
Bug: 148979873
Test: mm
Test: VtsHalNeuralnetworksV1_3TargetTest
Change-Id: Ib4960c71ca46ca034777cc7b02d7d2885a98691d
2020-03-04 14:56:46 -08:00
Miao Wang
34dfa2f74c Add more tests exercising IPreparedModel::executeFenced API
- executeFenced with device memory
  - executeFenced with depending sync fences

Bug: 148979873
Test: mm
Test: VtsHalNeuralnetworksV1_3TargetTest
Change-Id: I772c5c85f75cee56e2af06470c2de5b810f8078d
2020-02-26 13:17:22 -08:00
Lev Proleev
3ab17d6b02 Add align_corners and half_pixel_centers parameters to resize ops
Fix: 135147454
Test: NNTest_static
Change-Id: I4115120c8b6261c5518d561c043cda913d47dd45
2020-02-20 11:01:07 +00:00
Michael Butler
c7327c57f9 Fix NNAPI QoS Deadline test
The Deadline test was directly loading into the field of a std::optional
instead of emplacing, which is undefined behavior. This has been
corrected to emplace.

Additionally, previously the deadline test always checked the tensor
results of the execution. This is changed to only check the results
when the driver reports that the execution completed successfully.

Finally, this CL merges similar switch case statements.

Bug: 149766387
Test: mma
Test: VtsHalNeuralnetworksV1_3TargetTest
Change-Id: I1068fe70e74a12006dea3cdf4ae9da50004ad24b
2020-02-19 18:33:16 +00:00
Michael Butler
9b9a804fe4 Relax NNAPI QoS deadline parameter
Prior to this CL, the NNAPI QoS deadline is a strict deadline: a task
with a deadline must either complete or abort by the deadline. To
avoid as much overhead as possible, it is requested for Android R to
have a relaxed deadline, where the task *may* be aborted after the
deadline has passed or when the task is estimated to take too long.

Because the deadline is now relaxed, the querying method
IDevice::supportsDeadlines can be removed.

Bug: 149766387
Test: mma
Test: VtsHalNeuralnetworksV1_3TargetTest
Change-Id: Ib9937b5bb95646c2fd82e1aa17f2b0b639e2c420
2020-02-19 10:07:56 -08:00
Slava Shklyaev
f283a54fcc Merge "Add NNAPI loop timeout VTS test" 2020-02-19 11:19:58 +00:00