The attached version of TokenStream.java is essentially where we got to in our previous work on it in class, where it skips whitespace (but not comments) and recognizes LITERALs, but classifies all other characers as OTHER. The main procedure prints out all the tokens up to (but not including) the EOF. Now, using Java Generics and in particular the Collection classes, we can play around with modifications such as:
Printing out the tokens in reverse order (last one first).
Printing out all the distinct tokens, without repetitions if the same token occurs more than once.
Printing out a table showing for each distinct token how many times it occurs