Linux Vs Macos
Rapheal Charlton
One of the issues I am having with Swift is that it currently seems to be non-trivial to figure out whether a given code snippet will work on Linux or not. As I understand the open source Foundation isn't yet complete. This wouldn't be too big of a problem but the documentation isn't very helpful (when looking at a class in Foundation it's not obvious whether this has yet been implemented in the open source version).
It seems that while SPM and the compiler has ways to detect whether something will work with a targeted OS X version but there's no way to target Linux. Is this correct? This just seems to me like Swift server is a second class citizen... Is there plans to improve on this?
It seems in the near future Swift might be as convenient as C++ in that regard (as long as you only use the standard library and no OS specific stuff you'll probably be fine), but I was hoping Swift could do better in this area since it seems most of the building blocks necessary are already there?
What I am trying to do is this: build a server application but use my mac as a dev machine. So while I want it to compile and run on macOS my primary target is Linux. What is the best practice of doing this?
I obviously could wait for CI to tell me, but it doesn't seem to be super convenient. I would really like to be in a place where if my stuff compiles and works on macOS there's a very high chance it will also compile and work on Linux.
No, that's not correct. The compiler only sees availability. The issue is that Apple's Foundation doesn't include Linux availability, so the only way to see whether your code will compile is to actually build on Linux. This is fairly easy if you can do it locally or using a CI system like GitHub Actions, which is free for public repos. The only way to know if something will work is to run your test suite on Linux to exercise Foundation for Linux.
Others probably have better solutions, but one way to build for Linux locally is to set up a local Docker instance using the Swift images and share your code with that image. That should allow you to do local Mac and Linux builds at roughly the same time. You can also just virtualize a whole Linux instance with something like Parallels.
If you use platform specific features you can just use docker to build and run swift on linux locally on your mac. After you have downloaded and installed docker you can just run the following command:
Cross Compilation from macOS to Linux will be possible with this pitch: Pitch: Cross-Compilation Destination Bundles. You will then no longer need docker to build your code for linux. If you want to run it you will still need a Linux VM after all.
The tricky bit with using the macOS Swift compiler as a cross compiler for Linux is that (at least at the moment) the macOS compiler is built with the Swift driver enabled, while Swift for Linux is still built with the C++ driver. This causes the compiler to complain if you try to link cross-compiled modules with the Swift standard library for Linux. A way to work around this is to build a Swift toolchain (on the Mac) that is built with the same options as the native Linux one (i.e. building with the --skip-early-swift-driver turned on) and install that (including the corresponding destination.json) under /Library/Developer/Toolchains. Then you can use, e.g.:
To summarize: it seems the biggest problem is that Foundation isn't yet fully ported. This will probably be a non-issue in the near future and so I just have to live with the current state until then. This shouldn't be a big deal. Cross-compilation will improve the situation even more and it makes me happy that this is in the works!
So I'd like to get my product into a state where somebody can just clone the repository, change some code, and then run swift test. Afterwards there should be a high probability that the code changes didn't introduce new problems (obviously this mostly requires good tests and only a small part of the problem is the tooling).
I would expect this to work on macOS but this won't compile on Linux because Foo won't exist. But it feels like the compiler should be able to tell me something like "this won't compile on all the platforms you're targeting". So basically what I'd like to do is to annotate Foo like this:
If your project is a SwiftPM project you can use Visual Studio Code dev containers to verify a Linux build? These allow you to edit, build and debug projects inside a Docker container. You can find out more here vscode-swift/remote-dev.md at main swift-server/vscode-swift GitHub
I've used exactly this technique in a couple of projects. I mean, yeah - it would be really nice if there was a better option, but at least being able to invoke and work the code inside a docker container through VSCode is an incredibly useful workaround.
The tricky bit with using the macOS Swift compiler as a cross compiler for Linux is that (at least at the moment) the macOS compiler is built with the Swift driver enabled, while Swift for Linux is still built with the C++ driver. This causes the compiler to complain if you try to link cross-compiled modules with the Swift standard library for Linux. A way to work around this is to build a Swift toolchain (on the Mac) that is built with the same options as the native Linux one (i.e. building with the --skip-early-swift-driver turned on)
Whatever compiler errors you're getting when cross-compiling are unrelated to which compiler driver is used, as both mac/linux toolchains should be using the new driver written in Swift. --skip-early-swift-driver is only used when building the linux toolchain because the linux build doesn't require a prebuilt Swift toolchain, so it uses the legacy C++ Driver to build everything up till the new driver written in Swift, then builds, installs, and uses that new Swift driver.
However, it seems Foundation on Linux and macOS are incompatible. Is it possible to use the same version of Foundation on macOS and Linux? I understand the one on Linux is a re-implementation in Swift and C -- so it should be possible to also use on my mac?
For what it's worth for server side projects we often simply default to not using Foundation "as a whole" but e.g. specifically importing just Date, or JSONEncoder etc. Which may make this adventure also easier for you. I'm more than happy to help with specific trouble you hit and we sometimes have foundation-free solutions to things, feel free to ping me about any needs you hit.
Thank you! I think I got everything working for now. I am looking forward to the time when I can the same implementation of Foundation on macOS and Linux, but for now I can make it work. One thing that really helps is that Swift compilation times are great (at least for someone who is used to C++ ).
For reference (or if someone else in the future is interested in this), I used docker-compose to test locally and with GH actions. This works quite well, though for GH actions I couldn't figure out how to get SPM caching to work (I copy&pasted the official example but it doesn't work). But for now I don't worry as the whole job takes like 5 minutes and I have more free minutes than I can spend anyways (also debugging CI issues is like the least fun thing I can think of right now).
One pitfall I had to figure out is that the test needs to wait for PostgreSQL to come up. I just retry every 2 seconds for 60 seconds which until now has worked every time. It will read the connection information from the environment variables.
I would've preferred to start PostgreSQL from within the test and I spent 2 days trying to figure this out, but PostgreSQL people really don't want you to do that it seems (I could go on a long rant here how PostgreSQL seems to hate best practices and common sense and I would like to point out that it took me 5 minutes looking at their internal testing scripts to find a race condition that would make your tests flaky if you reuse them, but that's not relevant in this context).
I like to work on macOS but I write code that will run on Ubuntu. For this reason (as far as I know) I have to compile on Ubuntu to get the build errors that are specific to that operating system e.g. use FoundationNetworking for URLRequests on Linux.
So right now I develop my command line tools in Xcode on macOS and use Multipass to emulate a shell instance of Ubuntu and build on that. This works great but I wonder if there is better way to do this.
I imagine that I could create a build system maybe use CI to do all this but I'm looking for simple ways that I can set up for any new project in seconds and I want to keep the feedback loop as small as possible.
Seems you're doing it the right way. Xcode doesn't cross-compile for non-Apple platforms, nor does it run within a Linux environment. Most everyone uses a Linux virtual machine or Docker image to compile and run code on Linux using a MacOS host computer. Multipass is one such virtual machine environment. Not really an emulation since the basic instruction set is the same.
Since Multipass is a command line tool under Darwin, conceivably you could build an Xcode run script that runs a series of commands to Multipass, or add an external Makefile target to build the Ubuntu image using Multipass-based commands from within Xcode, but, you just using Xcode as a way to run the Makefile. But, it could work, build and debug on Darwin, build the Ubuntu Makefile target, and, if you want, add another Makefile target to run a test script using the Ubuntu image
Another Xcode trick you want to incorporate into your workflow are Xcode behaviors. Look under Xcode>Behaviors menu item. I've defined two behaviors to open a Terminal and a Finder window in my project's directory structure when I want to get to the file system or a terminal based on the project root from within Xcode. You could add a behavior that runs a Multipass shell and you should be able to inherit the Xcode environment variables so you can get to the DerivedData, Build folders, etc., where your Ubuntu images may be located so you run a debugger under Ubuntu, etc..
d3342ee215