Formatted output is lexically different from the input (inside comment).

[daniel-s-larsen]

Test case

//   ____  ____
//  /   /\/   /
// /___/  \  /
// \   \   \/
//  \   \
//  /   /
// /___/   /\
// \   \  /  \
//  \___\/\___\

(standard Xilinx header for IP they're sharing)

Include any options or configuration used.
--wrap_spaces 2 was used, but is not relevant here.

Actual output

-: Formatted output is lexically different from the input.    Please file a bug.  Details:
Mismatch in token sequence lengths: 10 vs. 14
First mismatched token [4]: ("// end of line comment") (#714: "//  \\   \\") vs. ("// end of line comment") (#714: "//  \\   ")
; problematic formatter output is
//   ____  ____
//  /   /\/   /
// /___/  \  /
// \   \   \/
//  \   \
//  /   /
// /___/   /\
// \   \  /  \
//  \___\/\___\
<<EOF>>

Include any possible diagnostic messages from the formatter.

Expected or suggested output
Comment line should not have been adjusted at all, as this is just line art.

Citations to published style guides would help.

[hzeller]

Interesting, they use backslash-escapes at the end.

I can't reproduce on unix, I suspect this might be happening on Windows ?
Converting the input with unix2dos, I can also see that.

Is this a corner-case that we should check @IEncinas10 (context #2371) ?

[hzeller]

I suspect the lexer reads the backslash CRLF as escaped CR + linefeed, and then things go wrong.

Interestingly with an input of CRLF and an output of CRLF it works again.

@daniel-s-larsen can you try to format the output as DOS line-endings by adding the --line_terminator=CRLF option ? That way the file stays with CRLF.

[daniel-s-larsen]

Wow you're fast, and accurate :O . This was on Windows, and adding --line_terminator=CRLF fixes the issue. I should take a closer look at the rest of the possible arguments and make sure I'm not missing anything else potentially important.

This does resolve my issue today, but it might be worth patching somehow to make the error message clearer to someone new like myself.

Should I leave the issue as 'open', or close it since I'm not blocked, and it was a matter of configuration?

Thank you!

[IEncinas10]

Thanks for looking at this @hzeller.

I have #2370 (comment) in my TODO list, but I'm not too confident in changing the lexer without proper testing and I don't have a Windows machine to play with...

Maybe we could add a --ignore-lexical-differences flag to work around issues regarding newlines and strange corner cases and suggest it in the error message?

Something like:

-: Formatted output is lexically different from the input.    Please file a bug.  Details:
Mismatch in token sequence lengths: 10 vs. 14
First mismatched token [4]: ("// end of line comment") (#714: "//  \\   \\") vs. ("// end of line comment") (#714: "//  \\   ")
; problematic formatter output is
//   ____  ____
//  /   /\/   /
// /___/  \  /
// \   \   \/
//  \   \
//  /   /
// /___/   /\
// \   \  /  \
//  \___\/\___\
<<EOF>>

NOTE: Some lexical differences are due to issues in the lexer concerning newlines that might not affect the end result. You can use --ignore-lexical-differences to disable this check.

[hzeller]

yes, maybe we need to re-visit the lexical difference comparison. We get the issue both ways: a LF file converted to CRLF or a CRLF file converted to LF, so we have a chance to test on non-windows machines, @IEncinas10

In the meantime, I made #2426 which adds an 'auto' option to the --line_terminator option, default to 'auto'. So in this particular case it would 'just work' as the CRLF stays CRLF. Of course not a solution, but it would've not surfaced in this case.

[hzeller]

So with the latest release that introduced the auto line terminator (i.e. use the output line terminator as observed on the input), this case will be handled automatically without providing any flag.

I leave this issue open until we have gotten to the bottom of the lexical difference report at hand.