Language models like ChatGPT have transformed the field of natural language processing, but they still struggle with basic tasks like arithmetic and fact-checking.
Last Thursday, researchers from Meta announced Toolformer, an AI language model that can teach itself to use external tools like search engines, calculators, and calendars without sacrificing its core language modeling abilities.
The possibilities of this line of research are endless, with potential applications such as natural language processing and translation across multiple languages.
Meta’s Toolformer Allows AI Language Models To Use External Tools
The real breakthrough of Toolformer is that it can use APIs (application programming interfaces) to communicate with other applications in a seamless and automated way.
During training, researchers showed Toolformer a small set of human-written examples demonstrating how to use each API, then allowed it to annotate a large language modeling dataset with potential API calls. In a “self-supervised” way, it learned to predict each text-based API call as if it were any other form of text, and can insert the calls as needed when generating text in response to human input.
This API-calling ability allows Toolformer to use external software tools like search engines, calculators, language translators, and factual references. For example, Toolformer can use a calculator program to handle arithmetic, or use an API link to a calendar app to add a date to a user’s calendar.
Toolformer is based on a pre-trained GPT-J model with 6.7 billion parameters. Experiments conducted by the researchers on various tool-using tasks demonstrated that Toolformer outperformed the much larger GPT-3 model, which has 175 billion parameters.
While researchers have previously attempted to make up for limitations in language models, most approaches have relied on human annotations or have been limited to specific task-specific settings. Toolformer, on the other hand, can learn to use a range of tools in a generalized way that does not require specialized training for specific tasks.
This development could lead to a future where language models augmented with the ability to use external apps become more versatile and reliable assistants. However, the ability to perform API calls also raises concerns about data security and privacy, as well as the potential for unintentional harm caused by the model while using external tools.
In summary, Toolformer represents a significant step forward in the development of language models, offering the potential for more reliable and versatile AI assistants. While the possibilities are exciting, it’s important to consider the potential risks and challenges that come with this new technology.