Open
Description
I have extracted test cases from the SQLite test suite for the lexer. I originally wrote them using a basic helper method while I was building the lever. Now, they need to be converted into the MiniTest suite. Add a new describe block to the test/feather/test_lexer.rb file
and file the code style already present. Here are the test cases:
# adapted from sqlite/test/misc5.test
assert_tokenized_as(
"SELECT 123abc",
[[:TK_SELECT, "SELECT"], [:TK_SPACE, " "], [:TK_ILLEGAL, "123abc"]]
)
assert_tokenized_as(
"SELECT 1*123.4e5ghi",
[[:TK_SELECT, "SELECT"], [:TK_SPACE, " "], [:TK_INTEGER, "1"], [:TK_STAR, "*"], [:TK_ILLEGAL, "123.4e5ghi"]]
)
assert_tokenized_as(' / ', [[:TK_SPACE, " "], [:TK_SLASH, "/"], [:TK_SPACE, " "]])
assert_tokenized_as('!*', [[:TK_ILLEGAL, "!"], [:TK_STAR, "*"]])