There have been some posts recently on Swift compiler performance problems related to type inference. The issue was discovered by Matt Nedrich and followed up on by Nick O'Neill last week. That specific bug has been fixed but it's a good reminder that it's still early days for Swift and you should occasionally expect problems like this.
In the article I'm linking to here, Robert Gummesson has found some other performance hiccups (and workarounds) and also has a useful Xcode plugin for easily measuring compiler performance. If you're experiencing slow build times on your project it might be a good idea to run this, then go file some bugs!