Ubuntu 24.10 is available now. This is an interim release and it will be supported till July 2025. If you are on 24.04 LTS, you may stick to it unless you want the shiny new GNOME 47 features.
PikaPods allows you to quickly deploy your favorite open source software. All future updates are handled automatically by PikaPods while you enjoy using the software. I use it to self-host Umami analytics. Did I tell you that they also share revenue with the original developers of the software?
Oh! You also get a $5 free credit to try it out and see if you can rely on PikaPods.
If you use Gmail, to ensure that Google shows you the informational FOSS Weekly newsletter, move our emails to inbox and star them. Else, it will keep on going to the Promotional tab even when this is not a marketing email.
In recent versions of GNOME, you can use the shortcut CTRL+ALT+TAB to select various parts of the shell UI. This is not fully functional, but works in many instances. Expect to encounter small issues.
🤣 Meme of the Week
Also suitable for desktop Linux vs server Linux ;)
🗓️ Tech Trivia
Fortran, one of the first high-level programming languages, was born in October 1956 with the release of its 60-page reference manual. However, the language’s true power wasn't fully realized until its compiler was completed six months later, marking a revolutionary step in programming.
When it comes to electronics, cooling is an essential part of its working lifecycle. Of course, if the components or hardware does not cool, and gets extremely hot, there are chances that it will melt away and get permanently damaged.
And, we would not want that to happen, right? So, we need some sort of cooling in our devices, components, and anything that runs on electricity.
But, not everything needs to be cooled the same way. So, it is important to understand the two different types of cooling processes — active and passive cooling.
In this article, I tell you all the essentials about these cooling methods.
Active Cooling
Passive Cooling
Involves one or more fans
Does not involve a fan
Fast and powerful
Slow and less effective
Requires a little extra power for cooling components
No extra power required for cooling components
Generates noise
Quite
Helps prevent thermal throttling effectively
Not effective enough to prevent thermal throttling
Generally found on machines that draw high power
Generally found on devices that uses low power
Examples include a computer, bike, and inverter
Examples include smartphones and mini PCs
Active Cooling: The Most Powerful Method
Active cooling is the most effective form of cooling, as the heat generated is directly pushed away from the device or components using a cooling device, whether it is a fan or a combination of liquid cooler and fan.
It is a straightforward mechanism that allows devices to utilize maximum power and still work well enough.
You can commonly find active cooling in a PC (except all-in-one desktops). Here's what it looks like:
0:00
/0:02
On a PC, you can see multiple fans working to actively cool the components by pulling fresh air and pushing out hot air, ensuring a flow of air to keep things cooled.
In some cases, you will see just a single fan as an exhaust, which just pushes out air, which also counts as active cooling.
For other real-life examples, you can think of an air-cooled engine for a bike and the cooling technique used in inverters.
It can also be seen in electric induction cooktops and similar kitchen items. I am sure there are many more examples I might be missing here.
Basically, if you hear a fan, it has an active cooling mechanism.
Passive Cooling: It Works, But It's Not The Best
Passive cooling also works as intended, dissipating the heat using conduction techniques and transferring the heat away from the core components without a fan.
You can find passive cooling in smartphones, slim laptops (with both active+passive cooling), and all-in-one desktops.
If it is compact, and fanless, it solely relies on passive cooling. Smartphones depict passive cooling the best, and manufacturers implement it in various ways to make the most out of it.
Here's an old promotional video for Xiaomi phones showing you what an inside of a smartphone looks like with passive cooling (no fans involved), and liquid cooling in action:
Some SBCs also have a passive cooling casing structure. Take the ZimaBoard for example.
In such cases, passive cooling is incredibly important for the user to comfortably use the machine without encountering any fan noise.
And, in some cases, laptops/mini PCs make use of both passive and active cooling as shown below:
Lenovo ThinkCentre's internal showing a combination of active (fan) and passive cooling
Of course, those laptops/mini PCs may choose to draw less power to make sure they remain cooler, especially, for Apple MacBook.
However, without enough active cooling, such devices are easily subject to thermal throttling if they are used in a hot environment.
Active Cooling vs. Passive Cooling: What's Better?
As I mentioned before, active cooling gets the edge as it is the most effective form of heat dissipation.
Sure, it needs a dedicated hardware component to take away heat from the core components of a system/device, but it ensures maximum performance output of a machine without thermal throttling.
It is great for systems drawing more power to function, as it generates more heat in the process.
That being said, active cooling is noisy if not done right. Even if you equip your machine with the best cooling equipment, you will find some noise when the “active” cooling takes place.
In that case, passive cooling comes to the rescue, just like sleek laptops that do not make any fan noise.
If you are to pick one: it would depend on the use-case, and form factor of a device. For instance, you cannot have active cooling on a smartphone. It is not a matter of choice, but what works where.
If you are dealing with a high-power drawing device with a decent size, active cooling will help if it is not already there.
Though, there are some rare exceptions coming up like this where passive cooling handles as well as active cooling:
If it is not possible to use active cooling, passive cooling is the only way to go. You can take additional steps like making sure to keep the device in an air-conditioned room, giving it time to cool down if it heats up, and things like that to help with cooling.
I hope this clears up any confusion between active cooling and passive cooling. It is important to learn the differences so you can make decisions better for your devices or future purchases.
Just to give you a quick refresher, the Pi 5 is a tiny computer with a 4-core Cortex-A76 CPU, up to 8GB of RAM, and a VideoCore VI GPU. It's basically a pocket-sized computer.
Now, the real fun begins. Our contenders for this experiment include a diverse range of LLMs, each with its own strengths and limitations. We'll be testing Phi-3.5B, Gemma2-2B, Qwen2.5-3B, Mistral-7B, and Llama 2-7B.
Let's see which of these language models can rise to the challenge of running on a Raspberry Pi 5.
Testing Criteria
To ensure a fair and objective evaluation of the LLMs, I used a standardized approach with every model.
I tested all the models directly in Ollama within the terminal, without a GUI, to remove any overhead in performance and provide a bare-metal approach to see how these models will perform.
Task: Each LLM had a task to generate a Docker Compose file for a WordPress installation with a MySQL database.
Metrics:
Inference time: The time elapsed from the prompt being issued to the completion of the Docker Compose file generation. A shorter inference time indicates better performance.
Accuracy: The correctness and completeness of the generated Docker Compose file. We will assess whether the file accurately defines the necessary services, networks, and volumes for a functional WordPress installation.
Efficiency: The resource utilization of the LLM during the task. We will monitor CPU usage, memory consumption, and disk I/O to identify any performance bottlenecks.
Google’s Gemma 2 model is offered in three sizes 2B, 9B, and 27B each with a new architecture that aims to deliver impressive performance and efficiency.
As you can see in the video above, the performance of Google’s Gemma2 model on the Raspberry Pi 5 was impressive.
The inference time was fast, and the response quality was excellent while utilizing only 3 GB of RAM out of the available 8 GB, leaving plenty of headroom for other tasks.
Given these results, I would rate this setup a solid 5 out of 5 stars.
Qwen2.5 is the newest generation in the Qwen series of large language models. It includes various base models and instruction-tuned versions, available in sizes from 0.5 to 72 billion parameters. Qwen2.5 brings several enhancements compared to its predecessor, Qwen2.
It is my first time testing this model and I was highly impressed by it. The inference time was remarkably fast, and the responses were accurate and relevant.
It utilized 5.4 GB of RAM out of the available 8 GB, leaving some headroom for other tasks.
This means you can easily use Qwen2.5 while juggling other personal activities without any noticeable slowdowns.
Phi-3.5-mini is a compact, cutting-edge open model derived from the Phi-3 family.
It is trained on the same datasets, which include synthetic data and curated public websites, emphasizing high-quality, reasoning-rich information.
With a context length of 128K tokens, this model has been refined through a comprehensive process that combines supervised fine-tuning, proximal policy optimization, and direct preference optimization to enhance its ability to follow instructions accurately and maintain strong safety protocols.
In my test of Microsoft’s Phi 3.5 model, the performance was somewhat okayish.
While the inference time wasn’t too shabby and the responses initially seemed good, the model started to hallucinate and produce inaccurate outputs.
I had to forcefully quit it after about 11 minutes, as it showed no signs of stopping and would likely have continued indefinitely.
The model utilized around 5 GB of RAM, which left some capacity for other tasks, but the hallucinations ultimately detracted from the overall experience.
Mistral is a 7-billion-parameter model released under the Apache license, offered in both instruction-following and text completion variants.
According to the Mistral AI team, Mistral 7B surpasses Llama2- 13B across all benchmarks and even outperforms Llama 1 34B in several areas.
It also delivers performance close to CodeLlama 7B for coding tasks, while still excelling in general English language tasks.
I was skeptical about this model since it was a 7b parameter model but during my testing on Pi 5, it did manage to complete the given tasks, although the inference time wasn’t super speedy around 6 minutes.
It utilized only 5 GB of RAM, which is impressive given its size, and the responses were correct and aligned with my expectations.
While I wouldn't rely on this model for daily use on the Pi, it's definitely nice to have as an option for more complex tasks when needed.
Llama 2, developed by Meta Platforms, Inc., is trained on a dataset of 2 trillion tokens and natively supports a context length of 4,096 tokens.
The Llama 2 Chat models are specifically optimized for conversational use, fine-tuned with more than 1 million human annotations to enhance their chat capabilities.
Well well well, as you can see above in my attempt to run the Llama 2 model, I found that it simply didn’t work due to its higher RAM requirements.
Code Llama, based on Llama 2, is a model created to assist with code generation and discussion.
It aims to streamline development workflows and simplify the coding learning process.
Capable of producing both code and explanatory natural language, Code Llama supports a wide range of popular programming languages, such as Python, C++, Java, PHP, Typescript (Javascript), C#, Bash, and others.
Similar to llama2 model, due to its higher RAM requirements, it didn't run at all on my Raspberry Pi 5.
Nemotron-Mini-4B-Instruct is designed to generate responses for roleplaying, retrieval-augmented generation (RAG), and function calling.
It’s a small language model (SLM) that has been refined for speed and on-device deployment using distillation, pruning, and quantization techniques.
Optimized specifically for roleplay, RAG-based QA, and function calling in English, this instruct model supports a context length of 4,096 tokens and is ready for commercial applications.
During my testing of Nemotron-Mini-4B-Instruct, I found the model to be quite efficient.
It managed to deliver responses quickly, with an inference time of under 2 minutes, while using just 4 GB of RAM.
This level of performance makes it a viable option for your personal co-pilot on Pi.
Orca Mini is a series of models based on Llama and Llama 2, trained using the Orca Style datasets as outlined in the paper "Orca: Progressive Learning from Complex Explanation Traces of GPT-4."
There are two versions: the original Orca Mini, which is built on Llama and comes in 3, 7, and 13 billion parameter sizes, and version 3, based on Llama 2, available in 7, 13, and 70 billion parameter sizes.
Orca Mini utilized 4.5 GB of RAM out of the available 8 GB, and the inference time was good.
While I’m not entirely sure about the accuracy of the responses, which will need to be verified by testing the output file, I would still recommend this model for its efficiency and performance.
CodeGemma is a versatile set of lightweight models capable of handling a range of coding tasks, including fill-in-the-middle code completion, code generation, natural language understanding, mathematical reasoning, and following instructions.
My experience with CodeGemma was quite interesting. Instead of responding to any of my queries, the model amusingly began asking me questions, almost as if it had a personality of its own.
I believe this behavior might be due to its focus on code completion, so I plan to test it in an IDE to see how it performs in that context.
Despite the unexpected interactions, it successfully loaded up in Ollama and used only 2.5 GB of RAM, which is impressive for such a lightweight model.
My Ratings
Please note that all the ratings provided are subjective and based on my personal experience testing these models.
They reflect how each model performed for me on the Raspberry Pi 5, but results may vary depending on different setups and use cases.
I encourage you to take these ratings with a grain of salt and experiment for yourself to see what works best for your needs.
LLM
Ratings
Gemma 2 (2b)
⭐⭐⭐⭐
Qwen 2.5 (3b)
⭐⭐⭐⭐⭐
Phi 3.5 (3.8b)
⭐⭐
Mistral (7b)
⭐⭐⭐
Llama 2 (7b)
-
Codellama (7b)
-
Nemotron-mini (4b)
⭐⭐⭐⭐
Orca-mini (3b)
⭐⭐⭐
Codegemma (2b)
⭐
Final Thoughts
Testing a wide range of LLMs on the Raspberry Pi 5 has provided valuable insights into the kinds of models that can realistically run on this compact device.
In general, models under 7 billion parameters are well-suited for the Pi, offering a good balance between performance and resource usage.
However, there are exceptions like Mistral 7B, which, despite being a larger model, ran fine albeit a bit slow.
Models in the 2B, 3B, and 4B range, on the other hand, performed exceptionally well, demonstrating the Pi’s capability to handle sophisticated AI tasks.
As we continue to advance in the field of AI, I believe we’ll see more models being optimized for smaller devices like the Raspberry Pi.
What do you think? Are there any models you’re trying out on your Pi? Do let us know!
Yes, we are all about Linux and open-source. So, if you are thinking that we might be biased, that is a possibility 😉 (just kidding!)
Jokes apart, every operating system has its strong points for which they stand out. And, when it comes to Linux, the list is arguably bigger than its competitors. We are mostly talking about Windows to compare it to here.
So, I shall be highlighting the things you can do on Linux but on Windows:
1. Use Advanced Windows Tiling
Whether you are a Windows or macOS user, looking at a Linux user manage their window screens is an absolute delight.
While Windows 11 has tried a step-up to give users a couple of layouts to organize their Windows, it is still nowhere near what Linux can achieve.
0:00
/0:18
COSMIC Desktop on Pop!_OS 24.04 LTS Alpha
You can use GNOME's tiling extension or Pop!_OS for a starting experience on Linux. In some cases, the windows will be automatically tiled, you get a highlight color border to differentiate between the active and inactive windows.
Ubuntu 22.04 LTS with GNOME's Tiling Extension
And, to go all in, you can try Arch Linux combined with i3, sway, or one among the best window managers out there:
You can navigate your entire screen (and multiple workspaces) using the keyboard shortcuts. And, once you are accustomed to those shortcuts, there is no going back.
2. Use the Desktop Environment of your choice
GNOME desktop on Vanilla OS 2
You do not have a choice with Windows. You will have to go with what Microsoft thinks is the best layout and look/feel for your desktop.
For instance, I liked how things were with Windows 10, but Windows 11 tries to modernize them (which I don't like). But, I don't have the option to keep using Windows 10s look with Windows 11.
However, when it comes to Linux, I can choose between different desktop experiences in the form of desktop environments. In other words, you get to pick the UI style or GUI components when installing a Linux distribution.
If I want a Windows-like layout, I can select Cinnamon or KDE Plasma desktop. If I want something unique, I can go with GNOME or the upcoming COSMIC desktop.
KDE Plasma Desktop
Furthermore, I get more options geared towards performance, like LXQt or XFCE desktop, tailored to be lightweight (yet capable) desktop environments.
3. Customize the look and feel yourself
0:00
/0:06
Archcraft distribution with Openbox window manager and polybar panel
While you already get a great freedom to choose your look/feel of the desktop, it does not stop there.
If you like, you can customize the experience to your heart's extent by configuring things yourself. Not just by using the available options but by diving in to your system configuration files.
With Windows, if there's no option to do it, you have no easy way out to change things without breaking the system. Windows is not built with customization in mind.
Maybe with Windows XP, and a couple of versions, we were able to change the entire theme/look of the taskbar. But, Windows user never had total control of their look.
Yes, you might need certain technical expertise to get what you want. But, you can, if you choose to.
To give you a head start, when using KDE Plasma as your desktop, you can follow our guide to customize it better:
Do you want widgets on your desktop screen? Just like you get on Android/iOS?
Linux is your friend for the matter.
If you want the ability to easily add widgets to improve the functionality and look of your desktop, KDE Plasma desktop environment is a good choice. You will find several pre-install widgets, and you can download more from the KDE Store.
Fret not, if you do not like KDE Plasma, you can resort to options like eww. You need to put a little effort into it to integrate it to the window manager of your choice, but it is a good one.
5. Built-in Support for Programming Languages
To get started with one of the most popular programming languages, Python, for example, you will have to set a few things on Windows.
Sure, it is easier than before to get started programming on Windows.
But, your Linux distribution already comes baked in with the latest available Python package (versions can be different, but it's always pre-installed). And, you can run Python programs with just a command like this:
Even if you do not have a code editor installed, you can just get started with the built-in terminal and nano/vim terminal-based editors and call it a day:
Not just limited to that, I find Linux as a more focused platform for programming.
We have WSL for Windows if you want access to the bash shell, and learn shell scripting, and try automating things. However, it is not applicable for Windows. You are doing it in a separate/isolated environment.
If you are Linux, you get direct access to the bash shell by default on most distributions. Furthermore, you can choose to install another shell like ZSH for different feature-set, and work with it.
I use Archcraft distro and I have ZSH shell by default. So, it is nice to have.
Implementing automation scripts, and making them work, is a bliss on Linux. You do not have that luxury on Windows.
7. Choose your favorite filesystem
With Windows, you are stuck with the NTFS filesystem. It has been around for a longer time, and it is reliable for its use-case.
However, if you are looking for certain features to manage your files (or have more reliable backups/encryptions), you have other file systems existing like BTRFS, and ZFS.
Of course, it is not advisable for users who are new to Linux, and do not understand the differences of such a file system. So, if you have done your research, you can go along with your favorite, and break away from the NTFS file system.
8. Docker and containerization
It is a no-brainer that the situation with Docker and containerization is similar to you being able to run Python programs out of the box.
Yes, you do not need virtualization to run a Docker container on Linux. It directly runs on the Linux kernel, making it efficient, and performance-focused compared to Windows. Not to mention the enormous size of Docker images on Windows.
In addition, the Docker ecosystem on Linux is more useful, and includes images that are less in size (saving storage space).
9. Use the system while it's updating
0:00
/0:09
The most discussed and loved ability of Linux is to be able to update your system without requiring a restart. As you can see above, I am updating my Arch Linux system while I write this article on my PC, without any interruptions.
Sure, at times, you need to reboot for the newer packages to take charge. However, it is a more seamless experience for the majority of the time without needing to reboot.
Moreover, unless it is an upgrade, the reboot after a system update does not take any extra time. But, with Windows, even with the slightest update, you might end up seeing an updating screen and must wait for a while before they are applied.
10. Availability of package managers
By default, Linux distributions handle packages using various package managers.
However, with Windows, package management is a dull process by default. You can utilize WinGet on the command line, or third-party solutions like Chocolatey.
The options available might be able to mimic how Linux does it, but it's not going to be in the same level.
💬What is your favorite ability with Linux onboard? Do you have more things to share? Let me know in the comments below.
Sourav Rudra
Sourav Rudra
Sourav Rudra
Ankush Das
Abhishek Prakash
Abhishek Kumar
Sourav Rudra
Sourav Rudra
Ankush Das
ernie
