Still, developers say that bringing code from Nvidia’s CUDA to ROCm isn’t a smooth process, which means they typically focus on building for just one chip vendor.
“ROCm is amazing, it’s open source, but it runs on one vendor’s hardware,” Lattner told the crowd at AMD’s Advancing AI event in June. Then he made his pitch for why Modular’s software is more portable and makes GPUs that much faster.
Lattner’s talk at AMD is representative of the kind of dance that Lattner and Davis need to do as they spread the Modular gospel. Today, Nvidia and AMD are both crucial partners for the firm. In a future universe, they’re also direct competitors. Part of Modular’s value proposition is that it can ship software for optimizing GPUs even faster than Nvidia, as there might be a months-long gap between when Nvidia ships a new GPU and when it releases an “attention kernel”—a critical part of the GPU software.
“Right now Modular is complimentary to AMD and Nvidia, but over time you could see both of those companies feeling threatened by ROCm or CUDA not being the best software that sits on top of their chips,” says Munichiello. He also worries that potential cloud customers may balk at having to pay for an additional software layer like Modular’s.
Writing software for GPUs is also something of a “dark art,” says Waleed Atallah, the cofounder and CEO of Mako, a GPU kernel optimization company. “Mapping an algorithm to a GPU is an insanely difficult thing to do. There are a hundred million software devs, 10,000 who write GPU kernels, and maybe a hundred who can do it well.”
Mako is building AI agents to optimize coding for GPUs. Some developers think that’s the future for the industry, rather than building a universal compiler or a new programming language like Modular. Mako just raised $8.5 million in seed funding from Flybridge Capital and the startup accelerator Neo.
“We’re trying to take an iterative approach to coding and automate it with AI,” Atallah says. “By making it easier to write the code, you exponentially grow the number of people who can do that. Making another compiler is more of a fixed solution.”
Lattner notes that Modular also uses AI coding tools. But the company is intent on addressing the whole coding stack, not just kernels.
There are roughly 250 million reasons why investors think this approach is viable. Lattner is something of a luminary in the coding world, having previously built the open source compiler infrastructure project LLVM, as well as Apple’s Swift programming language. He and Davis are both convinced that this is a software problem that must be solved outside of a Big Tech environment, where most companies focus on building software for their own technology stack.
“When I left Google I was a little bit depressed, because I really wanted to solve this,” Lattner says. “What we realized is that it’s not about smart people, it’s not about money, it’s not about capability. It’s a structural problem.”
Munichiello shared a mantra common in the tech investing world: He says he’s betting on the founders themselves as much as their product. “He’s highly opinionated and impatient, and also right a lot of the time,” Munichiello said of Lattner. “Steve Jobs was also like that—he didn’t make decisions based on consensus, but he was often right.”
