Issue 677  September 6th 2024

I found myself needing to upgrade to macOS Sequoia this week, so I finally got a chance to try Xcode’s new AI-powered “Predictive Code Completion”. 🤖

First things first. How’s the quality and does it “hallucinate”? I’d say the quality is good, and of course it hallucinates. 😂 I believe that eliminating hallucinations in LLM is, at best, extremely challenging and, at worst, impossible. Did it produce generally useful, modern, Swift code, though? Absolutely.

I’ve some experience with using GitHub Copilot, both inline in VS Code and via its chat interface and the experience of using Xcode’s predictive code completion felt very like Copilot’s inline code completion. Pause typing for a second, and it’ll show some dimmed code. Press tab, and it’ll accept the suggestion. Just like Copilot.

I find Copilot’s single-line completion suggestions to be much more useful than when it suggests a function implementation from a function name or comment, which feels like a gimmick. It’d be impossible for a human to write code from a function name for anything but the most trivial function, let alone an AI. But if you think of it as an advanced code completion rather than “write my code for me”, it delivers. That’s how Apple is pitching it, too, so that’s good.

One thing I prefer about the Xcode implementation is how it handles multi-line predictions. If Copilot wants to insert a fully formed function or a multi-line block, the entire block is visible but dimmed. In contrast, Xcode shows { … } where it wants to insert a block of code, whether that’s a function definition or a block after a guard or if statement. I think I prefer this because this is closer to the single-line completion I just mentioned.

I will admit that I expected it to be more responsive than Copilot given it’s an on-device model. Copilot needs to do a full round-trip to the Microsoft/GitHub servers and calculate the results, but it turns out that an on-device calculation with a consumer-grade CPU (I run an M1 Max) is about the same speed as a network connection + huge Azure servers. From some very non-scientific tests, performance is about the same or slightly worse than what I see with Copilot.

There are some obvious enhancements, which you’d expect from a first release. Having it explain compiler errors and runtime crashes would be a fantastic enhancement, and should be within reach. I’d also love to see something like Copilot chat where you can have a back and forth conversation about your code. I know that the possibility of going off-topic would be at the top of Apple’s mind when implementing something like this, but Copilot chat is very good at not letting the conversation wander away from code. If you have access to it, just try to lead it down a path it doesn’t want to go down. I completely failed.

I also wish Apple would give more information about where they sourced their training data, but I have banged that drum quite a lot now and it’s clear that the industry standard is to keep quiet about sourcing data in the vast majority of cases. I expected better from Apple on this point, though. I don’t want citations with every output, but a broad description of where the data was sourced from would be great.

Overall, I think it’s a win, and it’ll only get better over time!

Dave Verwer