Being able to use \u00a0ChatGPT\u00a0 in the cloud is fantastic. It is always available, remembers our previous chats, and responds quickly and efficiently. However, relying solely on that service can have disadvantages, such as \u00a0cost and privacy concerns\u00a0. This is where an exciting possibility arises: executing local AI models. For example, you can \u00a0assemble a local ChatGPT\u00a0.<\/p>\n
This capability was recently showcased on \u00a0Xataka\u00a0, particularly the trial of the \u00a0GPT-Oss-20B model\u00a0, which reportedly runs smoothly with a minimum of \u00a016 GB of memory\u00a0.<\/p>\n
Sam Altman declared that the upper model (120B parameters) could be run on a high-end laptop, while the smaller variant can even execute on a mobile device. However, our experiences confirm that these statements may require further investigation.<\/p>\n
After testing the model for a couple of hours, I found Altman’s assertions somewhat exaggerated. I utilized a \u00a0Mac Mini M4\u00a0 with \u00a016 GB of unified memory\u00a0 and had been experimenting with AI models for several months through \u00a0Ollama\u00a0, an application designed for easy model downloads and execution.<\/p>\n
The process for attempting the OpenAI “small” model was straightforward:<\/p>\n
Initiating this command starts the \u00a0download of the model\u00a0, which is about \u00a013 GB\u00a0 in size, and then the system begins executing it. The loading time to use the model takes a bit, as it requires transferring those \u00a013 GB\u00a0 from the disk to the unified memory of the Mac. After a couple of minutes, I received the green light to interact with the \u00a0GPT-Oss-20B model\u00a0.<\/p>\n
Excitedly, I took the opportunity to ask some basic questions, such as counting the number of “r’s” in the phrase: “San Roque’s dog has no tail because Ram\u00f3n Ram\u00edrez has cut it.”<\/p>\n
Interestingly, GPT-Oss-20B began processing my question and showed its \u00a0Chain of Thought\u00a0 in a different color. To my satisfaction, it correctly identified the number of “r’s” present in the phrase. However, the response was \u00a0very slow\u00a0.<\/p>\n
At the time of testing, I had multiple \u00a0Firefox\u00a0 instances open, each with several tabs, in addition to a \u00a0Slack\u00a0 session running. This overload proved problematic, as the \u00a0GPT-Oss-20B\u00a0 model alone requires at least \u00a013 GB of RAM\u00a0. The additional resource usage caused my system to freeze, necessitating a hard restart.<\/p>\n
After rebooting, I tried executing the model again. Despite the slowness of its responses, I was somewhat able to explore its functionalities. However, it quickly became evident that \u00a0this model is a reasoning model\u00a0, which means it attempts to provide smarter responses but consequently consumes more resources. This characteristic posed challenges in an already constrained environment.<\/p>\n
After sharing my experiences on \u00a0X\u00a0, several users encouraged me to try again, this time with \u00a0LM Studio\u00a0. This tool offers a user-friendly interface that\u2019s more aligned with what \u00a0ChatGPT\u00a0 provides in a browser.<\/p>\n
Post-installation, I once again attempted to use the model, but I encountered a runtime error, stating that there weren\u2019t enough resources. The issue was tied to the \u00a0graphics memory allocation\u00a0, which was found to be insufficient. By diving into the application\u2019s configuration settings, I realized the unified graphics memory had been allocated sub-optimally, with \u00a010.67 GB\u00a0 set aside for graphics.<\/p>\n
It was crucial to “lighten” the model\u2019s execution process. This involved adjusting the \u00a0GPU offload\u00a0, which dictates how many model layers are loaded into the GPU. Reducing it to \u00a010 layers\u00a0 proved to be a beneficial compromise. Other parameters, such as deactivating \u00a0“Offload KV Cache to GPU Memory”\u00a0, and decreasing the \u00a0“evaluation batch size”\u00a0, also contributed to better performance.<\/p>\n
After these tweaks, I successfully loaded the model, which operated much more efficiently and quickly. To my surprise, I encountered a local AI that performed at a level comparable to ChatGPT, responding effectively to questions.<\/p>\n
However, when I tasked the model with creating a table of the countries with the most FIFA football championships, it faltered, fabricating years and modifying runner-up numbers, despite being asked to verify the information.<\/p>\n
I then explored generating some code in \u00a0Python\u00a0 for data visualization. While the model doesn\u2019t directly create images, it can generate code that does. After executing the code it provided, I was pleasantly surprised with the accuracy of the graphical representation.<\/p>\n
The impression left by the local AI model was surprisingly positive. While it may still make errors, its performance was comparable to more advanced cloud models, making it a significant step in local AI capabilities.<\/p>\n
The initial hurdles in testing reflect a deeper reality\u2014the statement by Sam Altman and OpenAI has a hidden caveat. While \u00a0GPT-Oss-120B\u00a0 and \u00a0GPT-Oss-20B\u00a0 are free to use, they come with hardware prerequisites. To effectively execute these models:<\/p>\n
Importantly, this memory should be \u00a0graphics memory\u00a0. Unlike PCs that have separate RAM and graphics memory, Apple\u2019s unified memory allows interchangeability between the two, enhancing performance when working with local AI models.<\/p>\n
Performance metrics show that Apple\u2019s unified memory is significantly faster and more efficient than conventional RAM found in PCs. Influencing the overall ability to run local AI models smoothly are two main factors: \u00a01) the amount of graphic memory available and 2) the memory bandwidth\u00a0.<\/p>\n
In conclusion, the landscape for implementing local AI models is evolving. There\u2019s potential for accessible, efficient model execution without the connection to cloud servers. By leveraging the right system configurations and infrastructure, the \u00a0future of local AI\u00a0 looks promising, with the possibility of deploying sophisticated AI on personal devices, enhancing privacy and minimizing costs.<\/p>\n