Commit graph

403 commits

Author SHA1 Message Date
Slava Shklyaev
a6d65ad073 Merge "Log NNAPI VTS test case to logcat" into rvc-dev 2020-06-17 12:08:45 +00:00
Slava Shklyaev
8c8efd68e8 Log NNAPI VTS test case to logcat
Sample output:

    I VtsHalNeuralnetworksV1_2TargetTest: [Test Case] TestGenerated/GeneratedTest.Test/nnapi_sample_all_add BEGIN
    ...
    I VtsHalNeuralnetworksV1_2TargetTest: [Test Case] TestGenerated/GeneratedTest.Test/nnapi_sample_all_add END

Bug: 159060565
Test: VtsHalNeuralnetworksV1_0TargetTest (inspected logcat output)
Test: VtsHalNeuralnetworksV1_1TargetTest (inspected logcat output)
Test: VtsHalNeuralnetworksV1_2TargetTest (inspected logcat output)
Test: VtsHalNeuralnetworksV1_3TargetTest (inspected logcat output)
Change-Id: Ic1a0c90e0853ca84dd925d0f7df65e159bb284aa
2020-06-17 12:10:00 +01:00
TreeHugger Robot
c55a62bc79 Merge "Remove temporary test filters" into rvc-dev 2020-06-16 23:15:47 +00:00
Miao Wang
eeb0068015 Remove temporary test filters
Bug: 155577050
Test: mm
Change-Id: Ibe8a3ed79f640249ba54f94bdde9649f156383ea
2020-06-16 15:31:24 +00:00
Xusong Wang
362dfd64d5 Allow compilations in TOCTOU tests to fail with GENERAL_FAILURE.
A compilation failure is not related to the security aspect of the
TOCTOU test, but it will skip one iteration of security testing. This CL
allows the compilation to fail with GENERAL_FAILURE in TOCTOU tests, and
issues a retry once it happens to ensure enough test coverage.

Bug: 157489048
Test: 1.2/1.3 VTS
Change-Id: Idc88e0365c5d2799187093b6fd7b4abf8f8b463d
2020-06-15 13:22:27 -07:00
Lev Proleev
614ba3ef2e Merge "NNAPI: Clarify behaviour of ops that reduce dimensions" into rvc-dev 2020-05-18 15:31:13 +00:00
Lev Proleev
74b04e9649 Merge "NNAPI: Clarify behaviour of REDUCE_* ops" into rvc-dev 2020-05-15 10:28:39 +00:00
Lev Proleev
d4bba83f6a NNAPI: Clarify behaviour of ops that reduce dimensions
Clarify behaviour of the following ops in the case when inputs would
cause an output to have an empty shape:
* MEAN
* ARGMIN/ARGMAX
* STRIDED_SLICE

Bug: 155508675
Bug: 155660285
Bug: 155508675
Bug: 155238914
Test: VtsHalNeuralnetworksV1_3TargetTest
Change-Id: I92d1c3866a462a99b4eed7782b72f62bb6a076c6
2020-05-13 21:57:58 +01:00
Lev Proleev
eec7d86b44 NNAPI: Clarify behaviour of REDUCE_* ops
Add a clarification on the behaviour of REDUCE_* ops when all dimensions
of an input tensor are reduced.

Bug: 155508675
Test: VtsHalNeuralnetworksV1_3TargetTest
Change-Id: I3a67225c3af80af40a11362ca5d0f55350cbfa05
2020-05-13 15:00:32 +01:00
Miao Wang
4429d856cd Filter out failing NNAPI validation tests
Bug: 155577050
Test: atest VtsHalNeuralnetworksV1_*TargetTest
Change-Id: Id8d497dddd5521670689ff2ea08610c3aaeaf748
2020-05-10 19:38:17 -07:00
Miao Wang
f7c55d5f9b Add AndroidTest.xml for NNAPI 1.2 VTS and blacklist failing tests
- Also adds TEST_MAPPING filters here to make sure sample driver tests
  are still being filtered out.

Bug: 155674368
Test: mm
Test: atest VtsHalNeuralnetworksV1_2TargetTest
Change-Id: I5f5d272742d4cdd6a8cc87ec035b5e2508eaa98f
2020-05-04 20:10:48 -07:00
David Gross
12ab6c57cc More tests for graph validation.
- detect cycle (CycleTest)
- detect bad execution order (mutateExecutionOrderTest)
- detect lifetime inconsistent with whether operand is written (mutateOperandLifeTimeTest)
- detect lifetime inconsistent with Model inputIndexes/outputIndexes (mutateOperandInputOutputTest)
- detect incorrect number of consumers (mutateOperandNumberOfConsumersTest)
- detect operand written multiple times (mutateOperandAddWriterTest)
- detect operand never written (mutateOperationRemoveWriteTest)

Bug: 66478689
Test: VtsHalNeuralnetworksV1_*TargetTest

Change-Id: Id4ba19660bbd31a16f8a675f7b6437f4d779e8da
Merged-In: Id4ba19660bbd31a16f8a675f7b6437f4d779e8da
(cherry picked from commit af51663e99)
2020-05-02 02:34:30 +00:00
Xusong Wang
d923a3f06f Merge "Remove BLOB AHWB tests from 1.2 VTS." into rvc-dev 2020-04-10 17:44:43 +00:00
Xusong Wang
b642700c7f Remove BLOB AHWB tests from 1.2 VTS.
Bug: 153282844
Test: 1.2 VTS
Change-Id: I866fe1388a64a36372f90c60fef340d2fe4c8682
2020-04-09 14:26:54 -07:00
Dan Shi
aad51fa000 Rename vts-core to vts
Bug: 151896491
Test: local build
Exempt-From-Owner-Approval: This CL update suite name vts-core to vts as
the suite name is updated. This CL won't change test logic or behavior.

Change-Id: I562b4dc50765e953800a814a8fd84a01c1b9352b
Merged-In: I562b4dc50765e953800a814a8fd84a01c1b9352b
2020-04-07 16:37:40 -07:00
Miao Wang
077c0339d4 Update the specification for the following operations
- RANDOM_MULTINOMIAL
 - L2_NORMALIZATION

Bug: 136279892
Bug: 140177375
Test: mm
Change-Id: Iab38ef29ebf6d1f5c0a408436b3d564e45e537a0
2020-03-25 16:46:13 -07:00
Michael Butler
3c8593c0fe Merge "Remove extra tests from NNAPI VTS validation tests" into rvc-dev 2020-03-23 22:07:32 +00:00
Michael Butler
678a10600c Remove extra tests from NNAPI VTS validation tests
Do not run validation on "inputs_as_internal" and
"all_tensors_as_inputs" variants.

Bug: 138149072
Bug: 149840439
Test: mma
Test: VtsHalNeuralnetworksV1_*TargetTest
Change-Id: I0699ed6703e48b6c4bc0e7a392b79c12770f04c9
2020-03-19 18:37:23 -07:00
Xusong Wang
41adc5bc11 Add BLOB AHWB tests in VTS.
Bug: 149847930
Test: 1.3 VTS
Change-Id: I9c795dcb7696c843afd12551927463c5529a4b60
2020-03-13 20:04:38 -07:00
Michael Butler
b715967a15 Merge "NNAPI VTS: Add validation for Priority" into rvc-dev 2020-03-13 22:13:12 +00:00
Michael Butler
4876af1ba1 NNAPI VTS: decouple 1.2 tests from 1.3 types
Bug: 141718368
Test: mma
Test: VtsHalNeuralnetworksV1_*TargetTest
Change-Id: Ied7b95e146e4d2d95622642bfbb959cf134b5a7a
2020-03-13 19:59:43 +00:00
Michael Butler
da1a692880 NNAPI VTS: Add validation for Priority
This CL also cleans up the validation framework code.

Bug: 67828197
Test: mma
Test: VtsHalNeuralnetworksV1_*TargetTest
Change-Id: I84661fb2b8204148788d10425ca0ac986158b15f
2020-03-12 11:29:08 -07:00
Lev Proleev
13c43c0059 Add parallel linking support for BIDIRECTIONAL_SEQUENCE_{LSTM|RNN}
Change-Id: I51396f82de8dd7282db85a26b0a1be84b236b8c1
Fix: 138653129
Test: NNTest_static and VtsHalNeuralnetworksV1_3TargetTest
2020-02-18 21:00:44 +00:00
Slava Shklyaev
1f98e2e929 Add control flow support to NNAPI VTS tests
See change I98a3edd1.

Bug: 148077633
Bug: 148601177
Bug: 136735929
Test: VtsHalNeuralnetworksV1_0TargetTest
Test: VtsHalNeuralnetworksV1_1TargetTest
Test: VtsHalNeuralnetworksV1_2TargetTest
Test: VtsHalNeuralnetworksV1_3TargetTest
Change-Id: I1e436cdba404b68026a45797ac4fb3a34f8be76a
2020-02-12 14:29:27 +00:00
Automerger Merge Worker
c231f0be30 Merge "Revert "Remove VtsHalNeuralnetworksV1_2TargetTest from vts-core suite"" am: 20a5d89241 am: 3251b30fb2 am: 38d5f26305
Change-Id: I0db62643a33ad66d76da15455afc7a4e35efd895
2020-02-11 19:27:18 +00:00
Treehugger Robot
20a5d89241 Merge "Revert "Remove VtsHalNeuralnetworksV1_2TargetTest from vts-core suite"" 2020-02-11 18:48:38 +00:00
Dan Shi
6e48bde237 Revert "Remove VtsHalNeuralnetworksV1_2TargetTest from vts-core suite"
This reverts commit 07f4235b53.

Reason for revert: b/144527956

Change-Id: I70c3e99689901556e279200751a90e77fb4fd840
2020-02-11 01:04:07 +00:00
Michael Butler
aba3fc08f2 Merge "NN 1.3 HAL: ANAPIC review follow up" 2020-02-07 02:16:13 +00:00
Miao Wang
e3966e907f Merge "Add a VTS test to validate NNAPI driver name string." 2020-02-06 23:09:55 +00:00
Miao Wang
2729d825cb Add a VTS test to validate NNAPI driver name string.
Bug: 134755158
Test: VtsHalNeuralnetworksV1_2TargetTest
Change-Id: Iead93f0a251fdbbf94df6dfdf27919d3f6fd71a8
2020-02-05 17:24:02 -08:00
David Gross
0db1b54e9a redefine _Float16 as __fp16 for x86[_64] to allow removing workaround from inhouse clang
At present, the tests might not actually use _Float16 (the generated
tests that are incorporated into the functional tests are built in
frameworks/ml rather than in hardware/interfaces); but the makefiles
are changed anyway, in case _Float16 is added in the future.

Bug: 138709788

Test: for (walleye, aosp_x86 [on emulator], aosp_x86_64 [on emulator])
        Use clang where _Float16 is disabled for x86[_64]
        $ cd $ANDROID_BUILD_TOP/neuralnetworks/1.0/vts/functional
        inspect build commands for tests as to whether or not
          expected options related to redefinition are present
        $ cd $ANDROID_BUILD_TOP/neuralnetworks/1.1/vts/functional
        inspect build commands for tests as to whether or not
          expected options related to redefinition are present

Change-Id: I9225b228947696d249771b3a35b1f82de9c26b6e
(cherry picked from commit 2452ee5a80)
2020-02-05 11:09:22 -08:00
Michael Butler
6a4172cd3d NN 1.3 HAL: ANAPIC review follow up
This CL makes the following three interface changes:
* Removes @1.3::Operand.ExtraParams, because it was the same as
  @1.2::Operand.ExtraParams
* Changes int32_t token to uint32_t for IDevice::allocate and for
  @1.3::Request.MemoryPool
* Renames OptionalTimePoint::nanoseconds to
  OptionalTimePoint::nanosecondsSinceEpoch

This CL also makes test changes in response to the interface changes,
and fixes a minor comment issue in @1.2::types.hal and @1.3::types.hal.

Bug: 148617339
Test: mma
Test: VtsHalNeuralnetworksV1_3TargetTest
Change-Id: I95b49ab34b0b79f587dda035ede700b73dc44a38
2020-02-05 03:09:13 +00:00
David Gross
2452ee5a80 redefine _Float16 as __fp16 for x86[_64] to allow removing workaround from inhouse clang
At present, the tests might not actually use _Float16 (the generated
tests that are incorporated into the functional tests are built in
frameworks/ml rather than in hardware/interfaces); but the makefiles
are changed anyway, in case _Float16 is added in the future.

Bug: 138709788

Test: for (walleye, aosp_x86 [on emulator], aosp_x86_64 [on emulator])
        Use clang where _Float16 is disabled for x86[_64]
        $ cd $ANDROID_BUILD_TOP/neuralnetworks/1.0/vts/functional
        inspect build commands for tests as to whether or not
          expected options related to redefinition are present
        $ cd $ANDROID_BUILD_TOP/neuralnetworks/1.1/vts/functional
        inspect build commands for tests as to whether or not
          expected options related to redefinition are present

Change-Id: I9225b228947696d249771b3a35b1f82de9c26b6e
2020-02-03 11:55:38 -08:00
Michael Butler
79a41d77c0 Update NNAPI 1.3 VTS tests with new types
Bug: 136739795
Bug: 142902514
Bug: 145300530
Test: mma
Test: atest VtsHalNeuralnetworksV1_3TargetTest
Change-Id: Ie76da9dc9d6993a56bf644cfe20c5f5b421672c9
Merged-In: Ie76da9dc9d6993a56bf644cfe20c5f5b421672c9
(cherry picked from commit 9449a28b2f)
2020-01-27 12:51:34 -08:00
Michael Butler
9449a28b2f Update NNAPI 1.3 VTS tests with new types
Bug: 136739795
Bug: 142902514
Bug: 145300530
Test: mma
Test: atest VtsHalNeuralnetworksV1_3TargetTest
Change-Id: Ie76da9dc9d6993a56bf644cfe20c5f5b421672c9
2020-01-21 16:02:05 -08:00
Lev Proleev
c9d7f1cdc4 NNAPI: Update docs for ops with signed quantization
types.hal files are generated from current types.spec.

Bug: 136735770
Test: mma
Change-Id: Ide24606f8b1d5e755a71d46f230409942d8df7e9
Merged-In: Ide24606f8b1d5e755a71d46f230409942d8df7e9
(cherry picked from commit eb7f175613)
2020-01-07 13:36:03 +00:00
Lev Proleev
eb7f175613 NNAPI: Update docs for ops with signed quantization
types.hal files are generated from current types.spec.

Bug: 136735770
Test: mma
Change-Id: Ide24606f8b1d5e755a71d46f230409942d8df7e9
2020-01-06 12:52:06 +00:00
Lev Proleev
af5fc2374d Fix use-after-free error in NNAPI VTS tests am: 56cda83f8c am: 0854f5f759
am: 5f688eb948

Change-Id: Ibea07442f2d230caef39f6a3f73e3acf40f5cf00
2019-12-09 12:08:38 -08:00
Lev Proleev
56cda83f8c Fix use-after-free error in NNAPI VTS tests
Replace std::initializer_list with std::vector.

Fix: 145705114
Test: VtsHalNeuralnetworksV1_[23]TargetTest
Change-Id: I92cbc56b2b8fbb36b3f3225e7cd0d4b05c84041e
2019-12-09 14:56:09 +00:00
Lev Proleev
5595da3741 Add TENSOR_QUANT8_ASYMM_SIGNED support for more ops
Updated:
  CONCATENATION
  REDUCE_MIN
  REDUCE_MAX
  STRIDED_SLICE
  RESHAPE
  SQUEEZE
  MEAN

Bug: 143934554
Bug: 143935314
Bug: 143934726
Bug: 143934771
Bug: 143934557
Bug: 143934306
Bug: 143935140
Test: quantization coupling tests in CTS and VTS
Change-Id: Ie73d070fd6f0a27185c1f26561710635b6e03b26
Merged-In: Ie73d070fd6f0a27185c1f26561710635b6e03b26
(cherry picked from commit 91134e7fb9)
2019-12-02 11:40:55 +00:00
Przemyslaw Szczepaniak
90fc2cc193 Add TENSOR_QUANT8_ASYMM_SIGNED support for more ops
Updated:
 BATCH_TO_SPACE_ND
 CHANNEL_SHUFFLE
 DEPTH_TO_SPACE
 GROUPED_CONV_2D
 PAD
 PAD_V2
 QUANTIZE
 RESIZE_BILINEAR
 RESIZE_NEAREST_NEIGHBOR
 SPACE_TO_BATCH_ND
 SPACE_TO_DEPTH
 SPLIT
 TILE
 TOPK_V2
 TRANSPOSE

Bug: 143934582
Bug: 143934585
Bug: 143934628
Bug: 143934630
Bug: 143934721
Bug: 143935039
Bug: 143935052
Bug: 143935113
Bug: 143935115
Bug: 143935141
Bug: 143935353
Bug: 143935355
Bug: 143935392
Bug: 143935394
Bug: 143935413
Test: quantization coupling tests in CTS and VTS
Change-Id: I7e1b65507ea0f7dcdfdb5fd98e7871d84f569ed7
Merged-In: I7e1b65507ea0f7dcdfdb5fd98e7871d84f569ed7
(cherry picked from commit 6d3cdc322b)
2019-12-02 11:40:40 +00:00
Lev Proleev
91134e7fb9 Add TENSOR_QUANT8_ASYMM_SIGNED support for more ops
Updated:
  CONCATENATION
  REDUCE_MIN
  REDUCE_MAX
  STRIDED_SLICE
  RESHAPE
  SQUEEZE
  MEAN

Bug: 143934554
Bug: 143935314
Bug: 143934726
Bug: 143934771
Bug: 143934557
Bug: 143934306
Bug: 143935140
Test: quantization coupling tests in CTS and VTS
Change-Id: Ie73d070fd6f0a27185c1f26561710635b6e03b26
2019-11-28 11:54:29 +00:00
Przemyslaw Szczepaniak
6d3cdc322b Add TENSOR_QUANT8_ASYMM_SIGNED support for more ops
Updated:
 BATCH_TO_SPACE_ND
 CHANNEL_SHUFFLE
 DEPTH_TO_SPACE
 GROUPED_CONV_2D
 PAD
 PAD_V2
 QUANTIZE
 RESIZE_BILINEAR
 RESIZE_NEAREST_NEIGHBOR
 SPACE_TO_BATCH_ND
 SPACE_TO_DEPTH
 SPLIT
 TILE
 TOPK_V2
 TRANSPOSE

Bug: 143934582
Bug: 143934585
Bug: 143934628
Bug: 143934630
Bug: 143934721
Bug: 143935039
Bug: 143935052
Bug: 143935113
Bug: 143935115
Bug: 143935141
Bug: 143935353
Bug: 143935355
Bug: 143935392
Bug: 143935394
Bug: 143935413
Test: quantization coupling tests in CTS and VTS
Change-Id: I7e1b65507ea0f7dcdfdb5fd98e7871d84f569ed7
2019-11-27 15:09:09 +00:00
Lev Proleev
3683c785f2 Add QUANT8_ASYMM_SIGNED support to SELECT op
Also fix repo hook complaining about comment formatting.

Bug: 143935354
Test: NNTest_static and VTS_1_3  with --gtest_filter="*QuantizationCouplingTest*select*"
Change-Id: I1b0d1e987ebccc6700dd172b4222f9996105513d
Merged-In: I1b0d1e987ebccc6700dd172b4222f9996105513d
(cherry picked from commit 04bf939c7a)
2019-11-21 12:06:36 +00:00
Lev Proleev
0d4ba3f33a Combine test parameters into TestConfig structure
Test: VtsHalNeuralnetworksV1_2TargetTest --gtest_filter="GeneratedTests*"
Change-Id: I928aaa42e4745b4a8e0e461046e9632b052d0135
Merged-In: I928aaa42e4745b4a8e0e461046e9632b052d0135
(cherry picked from commit 66f598e10d)
2019-11-21 12:06:31 +00:00
Lev Proleev
5c305e52ef Merge changes from topic "quant_coupling"
* changes:
  Add QUANT8_ASYMM_SIGNED support to SELECT op
  Add quantization coupling test
2019-11-20 10:56:14 +00:00
Dan Shi
4c3a94d48d Merge "Remove VtsHalNeuralnetworksV1_2TargetTest from vts-core suite" am: 86c7aabcc0 am: 572970bcc6
am: 2327f621e7

Change-Id: Ibf66f98dd9d5d82d620f8907629eb56dae65ca81
2019-11-18 13:05:14 -08:00
Lev Proleev
04bf939c7a Add QUANT8_ASYMM_SIGNED support to SELECT op
Also fix repo hook complaining about comment formatting.

Bug: 143935354
Test: NNTest_static and VTS_1_3  with --gtest_filter="*QuantizationCouplingTest*select*"
Change-Id: I1b0d1e987ebccc6700dd172b4222f9996105513d
2019-11-18 17:20:55 +00:00
Dan Shi
07f4235b53 Remove VtsHalNeuralnetworksV1_2TargetTest from vts-core suite
The test takes over 12 hours to run. Remove it from vts-core suite.

Bug: 144527956
Test: None
Change-Id: If37a7b04e6d7b044d1594dcb4df07432236a1cf3
2019-11-18 08:40:19 -08:00
Lev Proleev
66f598e10d Combine test parameters into TestConfig structure
Test: VtsHalNeuralnetworksV1_2TargetTest --gtest_filter="GeneratedTests*"
Change-Id: I928aaa42e4745b4a8e0e461046e9632b052d0135
2019-11-12 15:38:36 +00:00