Mesh LLM pools your spare GPUs into one inference API
Most teams running LLMs in production rent someone else’s GPUs. They pay a cloud provider per token, send prompts to a black box, and hope the pricing and privacy terms stay put. Mesh LLM v1.0, released yesterday by the iroh team, offers the opposite trade. It takes the GPUs you already own and makes them work as a single inference cluster.
The pitch is direct. Install an 18 MB node binary on each machine. Point any OpenAI-compatible client at localhost:9337/v1. The mesh handles the rest. When a request arrives, one of three things happens: the local GPU runs it, the request routes to a peer that already has the model loaded, or the model splits across several machines as a pipeline. The client never sees the difference.
The interesting part is that last mode. Mesh LLM calls it Skippy. A model too large for any single node gets partitioned by layer ranges. Layers 0 to 15 run on one machine, 16 to 31 on the next, and activations flow down the pipe. The team reports Qwen 235B running at 16 tokens per second across two nodes. Not blistering, but functional for a model that neither machine could hold alone.
Under the hood, every node boots an iroh endpoint. Iroh is a P2P networking layer built on QUIC. Nodes authenticate by public key, punch through NAT, and open direct connections with relay fallback. There is no central server. Two iroh relays in different regions serve as backup paths for nodes that cannot reach each other directly.
The protocol itself is compact. Three QUIC ALPN protocols handle the mesh: one for gossip and routing, one for the control plane, and one for latency-sensitive activation transport between split-model stages. Inside the main connection, stream types are identified by a single leading byte. Gossip announcements, inference proxies, route queries, peer lifecycle events, and plugin RPC all share one multiplexed channel.
The catalog ships with 40-plus models. Small models that fit on a laptop, mid-range models for workstations, and MoE giants up to 235B parameters. A heuristic router classifies incoming requests by task type and complexity, then picks the cheapest model that can handle it. If the top pick is busy, a fallback ladder walks to the next option.
Several features are already implemented beyond basic inference. Mixture of Agents fans a request across multiple worker models and synthesizes the result. MoE expert sharding splits experts across nodes, though the team admits results there are underwhelming. Multimodal input is working for vision and audio, with image generation on the roadmap. A mobile chat app that joins a mesh by QR code scan is in progress.
The project launched on GitHub as Mesh-LLM/mesh-llm and hit 1,400 stars within 24 hours. It sat at number one on Hacker News with 279 points and 65 comments, where the discussion centered on practical token speeds and whether pooled consumer GPUs can actually compete with a dedicated inference provider on latency.
The honest answer is they cannot, not for single-request latency. What Mesh LLM offers is a different economic model. You stop renting compute by the token. You use hardware you already paid for. You control which models run, when they update, and where your data goes. For teams with idle GPUs in offices and home labs, that is a trade worth examining.
The roadmap lists speculative decoding, demand-based rebalancing, image generation peers, and a Swift SDK for mobile. The team is also working on an ACP agent standard integration and desktop applications that can host the mesh as a background process.