How can I change Github Copilot settings in VSCode to increase the token limit to 4098 or 8196 as promised in the Copilot product announcements and documentation? OpenAI is supposed to provide a 32K token limit.
What I Tried: (to find how to change the VSCode Github Copilot configuration):
- I asked Copilot in VSCode: "How can I change Github Copilot settings to increase the token limit to 4098 as promised in the documentation?"
It responded: "The token limit is a built-in feature of the AI model and cannot be changed by users. If you have questions about GitHub Copilot's settings or features, I recommend checking the official GitHub Copilot documentation or reaching out to GitHub support."
I checked the documentation and Github support forums and could find nothing.
I checked all Github Copilot settings in my Github account, and there is no way to change the token limit.
I asked OpenAI ChatGPT 4: "What is the token limit for Python code generation for ChatGPT 4?"
ChatGPT 4 responded: "The token limit for Python code generation in ChatGPT-4, including the code, comments, and any outputs, typically aligns with the overall token limit of a ChatGPT-4 response. This limit is approximately 4096 tokens in total. However, it's important to note that the specific limit can vary slightly based on the model configuration and the context of the conversation.
For Python code generation, the tokens are used for both the code itself and any textual output it generates. This means that longer code snippets or those producing extensive outputs might reach the token limit more quickly."
I asked OpenAI ChatGPT 4: "What is the token limit for Python code generation for ChatGPT 3.5 Turbo?"
ChatGPT 4 responded: "In ChatGPT 3.5 Turbo, the token limit for Python code generation is approximately 4096 tokens, which includes both the code itself and any associated comments or outputs. This token limit is consistent with the overall response token limit for this model."
Copilot is clearly throttling the OpenAI model or it is not up-to-date with ChatGPT 4 as Microsoft announced.
###################################################
This causes problems because we must limit and "chunk down" what we ask Copilot to assist us with, reducing developer bandwidth and throughput per day.
QUESTIONS:
Where is the Copilot API documented, so that we could change VSCode settings or hook into the module that communicates with the LLM, like we can do with LangChain and OpenAI API applications ??
How can I "unthrottle" Copilot to increase the token limit when generating Python code?
The Copilot API is not publicly documented, but if you want to tinker with its internals, you are able to modify its source code in its extension directory. For standard copilot (using the codex model for code completion), this is located here:
%USERPROFILE%\.vscode\extensions\github.copilot-x.xxx.x\~/.vscode/extensions/github.copilot-x.xxx.x/~/.vscode/extensions/github.copilot-x.xxx.x/And for the chat functionality, this is located here:
%USERPROFILE%\.vscode\extensions\github.copilot-chat-x.xxx.x\~/.vscode/extensions/github.copilot-chat-x.xxx.x/~/.vscode/extensions/github.copilot-chat-x.xxx.x/Specifically, the files that do the interfacing with the API are in
dist/extension.js. I'd recommend using a JavaScript de-minifier such as https://www.unminify2.com/ to help read the code, although it still takes some time to reverse engineer.After a brief search, it appears that the area around line 331 (of the minified code) might be responsible for limiting the token length to 4096. Playing around with doing full document find-replacement of this value might yield some interesting results.
Be aware that any modifications you make will be overwritten if you have automatic updates enabled for extensions.
As a side note, if you want to call GPT-4-turbo or codex directly in the same manner that Copilot does, you can use https://github.com/B00TK1D/copilot-api and https://github.com/B00TK1D/freegpt.