Lee Campbell 220ca84223 init: Add C++ tokenizer. ... Adds a C++ tokenizer along with unit tests. This tokenizer will replace the current C implementation which does a poor job of keeping track of pointers. This CL is a prerequisite for up coming changes to the parser. This CL does not wire up this tokenizer and changes no exsiting code. All that builds is the unit tests. Change-Id: Iec3740bce7153640adc5e5bbdc57e644cedf0038 TEST: Unit tests all pass. No leaks under valgrind BUG: 22843198		2015-07-30 18:45:17 +00:00
..
tokenizer.cpp	init: Add C++ tokenizer.	2015-07-30 18:45:17 +00:00
tokenizer.h	init: Add C++ tokenizer.	2015-07-30 18:45:17 +00:00
tokenizer_test.cpp	init: Add C++ tokenizer.	2015-07-30 18:45:17 +00:00