AI & ML


From the editor's desk: Groq – the future of AI processing?

28 March 2025 AI & ML


Peter Howells, Editor

For the past few years, the world has been hit by a storm of AI-generated information, mostly using generative pre-trained transformer (GPT) models to perform the AI inferencing. These models are excellent at performing large language model (LLM) requests, but they do have one drawback. The response time or lag is noticeable. This is largely due to the hardware that these models are being processed on, namely GPUs.

Many of the GPUs running AI models in professional data centres are Nvidia’s A100 (or similar) series which contain thousands of CUDA cores, many more than the handful of processing cores in a standard CPU. These CUDA cores work together to answer the language requests directed at them as they are designed for parallel processing and are optimised for tasks like scientific simulations. But are they really optimised?

Groq seems to be the new kid in all this AI hoopla, but they have been around since 2016 when the company was founded by a group of former Google employees led by Jonathan Ross, one of the designers of the Tensor Processing Unit (TPU), and Douglas Wightman, an engineer at Google X. The TPU is an AI accelerator Application-Specific Integrated Circuit (ASIC), a custom-designed chip tailored for a specific task. These ASICs offer optimised performance and efficiency compared to general-purpose processors.

And this is where the story gets exciting. The Groq AI model runs on ASICs as opposed to GPU architecture to deliver similar responses to the current slew of GPT models in use. Groq’s architecture is developed to expedite machine learning workloads, providing unparalleled speed and efficiency. This is a big deal – Groq needs much less energy to answer these same requests, and more importantly, does it with seemingly no lag. This last property is down to the speed at which the ASICs perform these ‘application-specific’ tasks.

Real-world testing by myself bears this out. When asking exactly the same technical question to both chatGPT 3.5 and 4.0 models and also to Groq, and then comparing the response times, I can without a doubt say that Groq certainly has minimal lag compared to the GPT models. The information produced in the responses is presented in a different format, but compares favourably with each other. Groq’s response is almost immediate whereas the other models take a few seconds before beginning to display an answer.

The introduction of Groq’s ASIC-based approach to AI inferencing marks a significant shift in the landscape of LLMs. By prioritising speed and efficiency, Groq is challenging the current dominance of GPU-driven AI, offering near-instantaneous responses while consuming less power. As AI applications continue to expand, this technology could redefine the way we interact with AI systems, setting a new benchmark responsiveness.

Whether this signals a broader industry shift remains to be seen, but one thing is clear – Groq has introduced a compelling alternative that demands attention.


Credit(s)



Share this article:
Share via emailShare via LinkedInPrint this page

Further reading:

From the Editor's desk: Growth through inclusivity
Technews Publishing News
As the engineering fields in South Africa continue to make progress toward gender equality, we are finally starting to see the presence and contribution of women in engineering and industrial roles.

Read more...
Photonic chip to slash AI energy consumption
AI & ML
Arago, a Paris and Silicon Valley-based deeptech startup pioneering a new class of energy-efficient AI chips powered by light, has raised $26 million in seed funding to accelerate the commercialisation of its photonic processor.

Read more...
Exploring AI
AI & ML
Powered by an ESP32-S3 MCU with 512?kB SRAM and 16?MB Flash, the Unihiker K10 features a vibrant 2,8-inch colour screen, built-in Wi-Fi and Bluetooth?5.0, and a 2-megapixel camera.

Read more...
Empowering innovation with ST’s AI processors
Altron Arrow AI & ML
Artificial intelligence is no longer just a futuristic concept – it is here, and it is transforming industries at an unprecedented pace.

Read more...
From the editor's desk: Fostering a love for engineering through DIY projects
Technews Publishing Editor's Choice
Many students are turning away from these perceived ‘hard’ STEM subjects, moving instead toward soft sciences and fields that seem less intimidating or more immediately rewarding.

Read more...
From the editor's desk: AI – a double-edged sword
Technews Publishing News
As with any powerful tool, AI presents challenges, some of which, if not carefully managed, threaten to undo the potential that it can offer.

Read more...
Vision-language models
AI & ML
Vision-Language Models are changing the AI game again, blending image and text understanding to power everything from autonomous vehicles to robotics to AI-driven assistants.

Read more...
Expert perspectives on robotics and AI
AI & ML
Mouser Electronics has announced a new eBook in collaboration with Analog Devices and Samtec exploring how robotics, artificial intelligence and machine learning are transforming the landscape of manufacturing, logistics, and beyond.

Read more...
From the editor's desk: Is the current AI really what we want?
Technews Publishing Editor's Choice
The companies that develop LLMs need to change direction and concentrate on freeing up our time, not so that we can have more time to do the tasks we don’t want to do in the first place, but rather to allow us more time to do what we love.

Read more...
From the editor's desk: Are we really being ripped off?
Technews Publishing News
To the surprise of many customers, installing solar panels does not always eliminate their utility bill – and in some cases, the power utility may impose additional charges on solar-powered homes.

Read more...









While every effort has been made to ensure the accuracy of the information contained herein, the publisher and its agents cannot be held responsible for any errors contained, or any loss incurred as a result. Articles published do not necessarily reflect the views of the publishers. The editor reserves the right to alter or cut copy. Articles submitted are deemed to have been cleared for publication. Advertisements and company contact details are published as provided by the advertiser. Technews Publishing (Pty) Ltd cannot be held responsible for the accuracy or veracity of supplied material.




© Technews Publishing (Pty) Ltd | All Rights Reserved