tokens

Whitespace and quoted-strings tokenizer

Overview

WORK IN PROGRESS

Example

use tokens::tokens;

let input = r#"
  fn main() {
    println!("Hello, world!");
  }
"#;

for token in tokens(input) {
println ! ("{}", token);
} 

Output:

fn
main()
{
println!(
Hello, world!
);
}

License

Licensed under either of

• MIT license (see LICENSE-MIT) or
• Apache License, Version 2.0 (see LICENSE and NOTICE)

at your option.

Contribution

Any contributions to tokens are greatly appreciated. All contributions intentionally submitted for inclusion in the work by you, shall be dual licensed as above, without any additional terms or conditions.