My VS Code Extension includes a Semantic Tokens Provider that runs into issues with large files. When opening files with one million lines, the Extension Host crashes.
The problem lies in how the provider parses the text of the active editor into tokens. Every time there is a change in the text, it loops through each line individually. This process makes the extension faster for smaller amounts of text, but handling one million lines becomes too much for it, leading to the termination of the Extension Host:
https://i.sstatic.net/DGB1l.png
To address this issue, I thought about monitoring the visible range of the editor as the user scrolls and updating the document's semantic tokens accordingly. By parsing different ranges of lines on each visible range change, we can improve efficiency.
I know one approach involves obtaining the top and bottom lines of the visible range in the active editor:
const window = vscode.window
const textEditor = window.activeTextEditor;
if(textEditor) {
window.onDidChangeTextEditorVisibleRanges(() => {
const visibleRange = textEditor.visibleRanges[0];
const topLineIndex = visibleRange.start.line;
const bottomLineIndex = visibleRange.end.line;
console.log(topLineIndex, bottomLineIndex);
});
}
However, I am unsure how to dynamically update tokens as the visible range changes since returning a SemanticTokens instance on an event is not possible.
Do you know of a way to update the semantic tokens based on an event? If not, do you have a more efficient solution than looping through each line for tokenizing the text?