JetBrains Language Support Plugin

Question

I am trying to implement language support plugin for a basic language. I am following jetbrains's tutorial for simple language support (.properties file support basically) and on the side I have rust plugin for reference. However the complexity gap between them is huge so some questions are hard to find answers to from both sources.

Here is my question: What is the best way to allow spaces between tokens which DO NOT require spaces, however force spacing between tokens which do?

i.e. class Foo{ <- here first space is mandatory, but the second one (before '{' symbol) can be ommited.

Answer 1

That is a task for the Parser, not the Lexer. In fact the Lexer lexes the whitespaces but does not pass them to the Parser. To enforce or not enforce the whitespaces in your example you need to implement a .bnf file with the Grammar-Kit extension https://github.com/JetBrains/Grammar-Kit.

Answer 2

In general, parser doesn't handle whitespaces. In com.intellij.lang.ParserDefinition, you can find:

@NotNull
default TokenSet getWhitespaceTokens() {
    return TokenSet.WHITE_SPACE;
}

Returns the set of token types which are treated as whitespace by the PSI builder. Tokens of those types are automatically skipped by PsiBuilder.

Your Lexer finds all white characters and marks them as TokenType.WHITE_SPACE. If there is no whitespace character in your example, it will simply treat the whole classFoo as a single unknown token.

In your implementation of com.intellij.lang.ParserDefinition, you can override method:

default @NotNull SpaceRequirements spaceExistenceTypeBetweenTokens(ASTNode left, ASTNode right)

to return MAY, MUST, MUST_NOT or MUST_LINE_BREAK depending on the type of nodes.

You can also specify more advanced spacing for auto-formatting in your FormattingModelBuilder described in the tutorial here: https://plugins.jetbrains.com/docs/intellij/formatter.html