WikiNow lets you discover the news you care about, follow the topics that matter to you and share your favourite stories with your friends.

© WikiNow

In computer science, lexical analysis is the process of converting a sequence of characters into a sequence of tokens . A program that performs lexical analysis may be called a lexer, tokenizer, or scanner . Such a lexer is generally combined with a parser, which together analyze the syntax of programming languages, web pages, and so forth.

In computer science, lexical analysis is the process of converting a sequence of characters (such as in a computer program or web page) into a sequence of tokens (strings with an identified "meaning"). A program that performs lexical analysis may be called a lexer, tokenizer, or scanner (though "scanner" is also used to refer to the first stage of a lexer). Such a lexer is generally combined with a parser, which together analyze the syntax of programming languages, web pages, and so forth.