Strings
StringTokenizer
Deconstruct the legacy StringTokenizer utility, comparing it with String.split and regular expression parsers.
Interview: Tests character-matching tokenization efficiency, legacy API deprecation status, and compatibility limits.
The StringTokenizer class is a legacy utility in the java.util package. It splits a string into tokens based on specified delimiter characters. While not deprecated, it is considered a legacy class whose use is discouraged in modern Java code in favor of String.split() or regex patterns.
Core Idea
A legacy character-based tokenizer that parses strings without regex, returning tokens sequentially via an Enumeration interface.
Why It Matters
Faster than String.split() for simple single-character delimiters because it avoids compiling regular expressions.
Interview Lens
Compares performance with String.split(), checks empty token behaviors, and evaluates compatibility.
Comparison: StringTokenizer vs String.split()
When choosing between tokenizing options, understand these key differences:
| Feature | StringTokenizer | String.split() |
|---|---|---|
| Matching Engine | Plain character matching (no regex) | Regular Expression matching |
| Performance | Fast (low overhead) | Slower (regex compilation overhead) |
| Empty Tokens | Adjacent delimiters are skipped by default | Retains empty strings in the output array |
| Memory | Low (lazy generation via nextToken()) | Higher (pre-allocates the entire array) |
Common Pitfalls
- Skipping Empty Data Fields: Using StringTokenizer on CSV files where empty fields (e.g.
value1,,value3) are present. It skips the empty field and parses the next token, scrambling the column order. - Delimiters treated as independent characters: Setting a delimiter parameter like
"OR"splits the string on either'O'or'R', not on the word"OR". - Integration issues: StringTokenizer does not implement
Iteratoror support collections directly.
Best Practices
- Use
String.split()or regex patterns by default in modern applications. - Use StringTokenizer only in legacy systems or when parsing huge, simple character-separated text streams where regex compilation is a bottleneck.
- Avoid StringTokenizer when empty data slots are meaningful.
Interview-Relevant Information
Q1: How does StringTokenizer handle consecutive delimiters?
Answer: By default, StringTokenizer treats adjacent delimiters as a single delimiter, skipping empty tokens. For example, tokenizing "a,,b" with "," yields "a" and "b", skipping the empty value between them.
Q2: Why is String.split() preferred over StringTokenizer in modern code?
Answer: String.split() supports complex regular expressions, returns a clean array, integrates with collections, and handles empty fields consistently. This makes it more robust for standard applications.
Quick Checklist
Can you identify why StringTokenizer outperforms split() for simple char matches, predict empty token behaviors, explain delimiter parsing rules, and manage legacy integrations? If yes, you understand StringTokenizer.
Use Cases
Parsing simple, legacy log files containing space-separated parameters.
Deconstructing path strings rapidly in low-level file path parsing code.
Common Mistakes
Using StringTokenizer to parse CSV columns where empty entries must be preserved.
Supplying multi-character words as delimiters and expecting them to be matched as a unit instead of separate characters.