The new Exo tool allows for distributed deployment of AI models across various devices using peer-to-peer networks. This software solution aggregates computational resources from computers, smartphones, and single-board computers like Raspberry Pi, dynamically distributing AI model layers based on available memory and computing power.
Features and Compatibility
Exo supports prominent models like LLaMA, Mistral, LlaVA, Qwen, and DeepSeek. The tool is designed for Linux, macOS, Android, and iOS, with a Windows version yet to be released. To function, it requires Python 3.12, and additional components may be necessary for Linux systems using Nvidia GPUs.
- Executable on systems with Linux, macOS, Android, and iOS.
- Supports AI models: LLaMA, Mistral, LlaVA, Qwen, DeepSeek.
- Requires Python 3.12; extra components may be needed for Linux with Nvidia.
Potential and Practical Implications
By creating a distributed network, Exo presents itself as a promising alternative to conventional cloud resources. It's capable of running a model requiring 16 GB of RAM on two laptops each with 8 GB. Theoretically, it can execute a DeepSeek R1 model needing 1.3 TB on a cluster of 170 Raspberry Pi 5 units, each with 8 GB.
While promising, there are significant considerations: network speed and delays may affect performance, and weaker devices can slow processing, presenting security risks when running concurrent tasks.
