The Tokenizer (aka Lexical Analysis) is a phase within a compiler that converts an input code into an array of tokens.
Create a simple tokenizer program that will accept a single line assignment statement written in Java.
Examples:
1) int x = 1;
2) String str = "Hello World",
3) double number;
The Tokenizer program should then identify the lexemes in your statement and categorize them into their respective tokens. Refer to the symbol table below.
Lexemes: int, double, char, String
Tokens:
<datatype> - <assignment operator> : <delimiter> 0 ... 9, string values in "double quotes", char values <value> in 'single quotes', numbers with decimal place, Strings without double quote, <identifier>
Sample input/output:
Enter Source Language: int num = 2;
Output is: <datatype> <identifier> <assignment operator> <value> <delimiter>