Debian packaging support

@buttaface suggested that I start a new issue here to discuss problems that arise while I am packaging Swift for Debian. I have found the Fedora packaging that @tachoknight has done to be very helpful in this effort. A related work that I just learned of from @etcwilde is work on packaging for Ubuntu in the swift-installer-scripts repo.
The challenges are mainly getting Swift to build into a .deb package using the debuild tool and getting the packing into compliance with Debian policy. It has a very powerful linting tool that checks for many common errors and issues and most Debian Developers won't sign off on a package that isn't lint clean or reasonably close to it.
Here is the biggest issue that I know of right now. The file ConvertUTF.cpp uses a non-free license and is located in three places:

indexstore-db/lib/LLVMSupport/Support/ConvertUTF.cpp
llbuild/lib/llvm/Support/ConvertUTF.cpp
llvm-project/llvm/lib/Support/ConvertUTF.cpp

Debian suggests replacing these files with libicu, which is already a part of Swift. But there are other alternatives out there as well such as utfcpp.
I am not a software engineer by trade so it is unlikely that I could make this replacement on my own without making a mess of things.
Another linting error that I can probably resolve post-bulid on my own is that many libraries have executable permissions. For example:
E: swiftlang: shared-library-is-executable 0755 [usr/libexec/swift/lib/clang/10.0.0/lib/linux/libclang_rt.dyndd-x86_64.so]
And another example of how challenging this packaging can be, I can't strip the binaries or REPL won't work and when I disable the binary stripping the lint tool then complains that the binaries have not been stripped. sigh

The file ConvertUTF.cpp uses a non-free license and is located in three places

That file is under the same University of Illinois Open Source License as a lot of LLVM code. In addition, it requires that disclaimer from Unicode, Inc. in the source, not sure why a simple disclaimer is such a problem.
Since that file is compiled as part of the LLVM support library libLLVMSupport.a, which is presumably already in some Debian LLVM package, you can probably just copy what they do for that existing Debian package here.

I can't strip the binaries or REPL won't work and when I disable the binary stripping the lint tool then complains that the binaries have not been stripped

Surely Debian allows exceptions for developer tools like this that need those symbols to work?

The file ConvertUTF.cpp uses a non-free license and is located in three places

That file is under the same University of Illinois Open Source License as a lot of LLVM code. In addition, it requires that disclaimer from Unicode, Inc. in the source, not sure why a simple disclaimer is such a problem.
Since that file is compiled as part of the LLVM support library libLLVMSupport.a, which is presumably already in some Debian LLVM package, you can probably just copy what they do for that existing Debian package here.

That was a great idea. The llvm-toolchain-14 package has 840 lint errors including this one:
llvm-toolchain-14 | 1:14.0.6-2 |   | error | license-problem-convert-utf-code | [llvm/lib/Support/ConvertUTF.cpp]
I am able to see what they put in their copyright file and will do the same thing. The lint error will just have to be overridden. I see a lot of familiar lint errors in that llvm package so maybe this won't be as big of an issue as I had thought.

I can't strip the binaries or REPL won't work and when I disable the binary stripping the lint tool then complains that the binaries have not been stripped

Surely Debian allows exceptions for developer tools like this that need those symbols to work?
I'm sure they do. I was just complaining how this turns into a three step process of disabling the stripping, disabling the lint errors that result, then probably having to explain to a Debian Developer why stripping was disabled.

Once I have the low hanging lint errors cleaned up I'll start working on getting the remaining errors reviewed by a Debian Developer to see if they have any concerns that I need to address.

Here is something else that would help me. Anything that lintian flags as an error is considered to be either a bug in the lint tool or a Debian policy violation. That said, it seems like there is some understanding that it isn't always that simple and overrides of rules are allowed.
For the following lintian errors it would be very helpful to know if the error is something I should be able to fix, or is expected. And if expected, what is the reason why. Any explanation that I can add to the comments of the lintian overrides will help to get approval to add the package to Debian. Most errors have multiple failures, but only the first is shown for brevity.
E: swift source: binary-with-bad-dynamic-table [llvm-project/llvm/test/tools/llvm-xray/X86/Inputs/elf64-objcopied-instrmap.bin]
N:
N:   This appears to be an ELF file. According to readelf, the program headers
N:   suggests it should have a dynamic section, but readelf cannot find it.
N:
N:   If it is meant to be external debugging symbols for another file, it
N:   should be installed under /usr/lib/debug. Otherwise, this could be a
N:   corrupt ELF file.

E: swiftlang: custom-library-search-path RUNPATH swift_5.6.2/build/buildbot_linux/llvm-linux-x86_64/lib [usr/libexec/swift/bin/lldb-argdumper]
N:
N:   The binary or shared library sets RPATH or RUNPATH. This overrides the
N:   normal library search path, possibly interfering with local policy and
N:   causing problems for multilib, among other issues.
N:
N:   The only time a binary or shared library in a Debian package should set
N:   RPATH or RUNPATH is if it is linked to private shared libraries in the
N:   same package. In that case, place those private shared libraries in
N:   /usr/lib/*package*. Libraries used by binaries in other packages should be
N:   placed in /lib or /usr/lib as appropriate, with a proper SONAME, in which
N:   case RPATH/RUNPATH is unnecessary.
N:
N:   To fix this problem, look for link lines like: gcc test.o -o test
N:   -Wl,--rpath,/usr/local/lib or gcc test.o -o test -R/usr/local/lib and
N:   remove the -Wl,--rpath or -R argument. You can also use the chrpath
N:   utility to remove the RPATH.
N:
N:   Please refer to https://wiki.debian.org/RpathIssue for details.

E: swift source: missing-notice-file-for-apache-license [swift-crypto/NOTICE.txt]
N:
N:   The package appears to be licensed under the Apache 2.0 license and a
N:   NOTICE file (or similar) exists in the source tree. However, no files
N:   called NOTICE or NOTICE.txt are installed in any of the binary packages.
N:
N:   The Apache 2.0 license requires distributing of such files:
N:
N:    (d) If the Work includes a "NOTICE" text file as part of its
N:        distribution, then any Derivative Works that You distribute must
N:        include a readable copy of the attribution notices contained
N:        within such NOTICE file [..]
N:
N:   Please include the file in your package, for example by adding
N:   path/to/NOTICE to a debian/package.docs file.
N:
N:   Please refer to usr/share/common-licenses/Apache-2.0 for details.

E: swiftlang: no-code-sections [usr/libexec/swift/lib/swift_static/linux/libicudataswift.a]
N:
N:
N:   The named members of the static library have no usable code sections.
N:
N:   It happens when shared objects are built with -flto=auto but without
N:   -ffat-lto-objects. dh_strip strips the LTO sections but may leave the
N:   static library without any usable code.
N:
N:   Please refer to Bug#977596 for details.

These don't need to be executable do they? I'm not even sure just how they are getting +x.
E: swiftlang: shared-library-is-executable 0755 [usr/libexec/swift/lib/clang/13.0.0/lib/linux/libclang_rt.asan-x86_64.so]
N:
N:   Shared libraries should be mode 0644.
N:
N:   Please refer to Run-time shared libraries (Section 8.1) in the Debian
N:   Policy Manual for details.

There are many of this kind. I'm not sure what the issue is as they don't go into the package. Maybe there is a way to clean them up just prior to running lintian?
E: swift source: source-is-missing [cmake/Tests/CMakeTests/ELF/elf32lsb.bin]
N:
N:   The source of the following file is missing. Lintian checked a few
N:   possible paths to find the source, and did not find it.
N:
N:   Please repack your package to include the source or add it to
N:   "debian/missing-sources" directory.
N:
N:   Please note, that very-long-line-length-in-source-file tagged files are
N:   likely tagged source-is-missing. It is a feature not a bug.

This is just a warning, but it makes me wonder if I built Swift incorrectly? Maybe it doesn't matter as this is pertaining to the debug symbol package (swiftlang-dbgsym_5.6.2-1_amd64.deb) and I don't think that would be getting added.
W: swiftlang-dbgsym: debug-file-with-no-debug-symbols [usr/lib/debug/.build-id/00/bb2a059d939872ab355e8fa581567eac042f7d.debug]
N:
N:   The binary is installed as a detached "debug symbols" ELF file, but it
N:   does not appear to have debug information associated with it.
N:
N:   A common cause is not passing -g to GCC when compiling.
N:
N:   Implementation detail: Lintian checks for the ".debug_line" and the
N:   ".debug_str" sections. If either of these are present, the binary is
N:   assumed to contain debug information.
N:
N:   Please refer to Bug#668437 for details.

If anyone can help make sense of any of these lintian issues that would be most helpful.

The test files, elf64-objcopied-instrmap.bin and elf32lsb.bin, I'm not sure why your linter is even looking at those: you're not running the tests and those files are not installed at the end, are they? The first one is from LLVM, so I'd look at how the existing package handles that.
The RUNPATH issue is a definite bug on our end, specifically with lldb to have an absolute path like that. Please look into where that's getting added, and I bet the LLVM devs would accept a patch to remove that.
The missing license file may be an oversight on our end, unsure as pretty much all Swift repos use that same Apache license, not sure why your linter is pointing just that one out.
The ICU issue needs to be investigated, could be legit. You could also try removing the Swift build of libicu and just use the Debian system package, which should work too but will require you to bump the Swift package anytime the system libicu is updated.
Those libcompiler-rt shared libraries don't need executable permissions, probably a mistake on our end.
I have no idea about that debug file with no symbols, haven't used that.

The test files, elf64-objcopied-instrmap.bin and elf32lsb.bin, I'm not sure why your linter is even looking at those: you're not running the tests and those files are not installed at the end, are they? The first one is from LLVM, so I'd look at how the existing package handles that.

I looked up Debian's llvm-toolkit-14 package and wouldn't you know it, they have these same lintian errors and many of the others that I am seeing. If they can have them in llvm then swift should be allowed them as well. The lint tool looks at both the completed package as well as the source, so it seems to pick up all sorts of files that shouldn't matter.

* The `RUNPATH` issue is a definite bug on our end, specifically with lldb to have an absolute path like that. Please look into where that's getting added, and I bet the LLVM devs would accept a patch to remove that.

I'm going to look into this some more, but maybe it isn't an issue. Based on the Debian wiki this forces the use of the custom Swift libraries, which I presume is better than picking up the system's libraries for llvm and others. Again, the Debian llvm package has these errors as well so I will contact the maintainer to find out what I can.

* The missing license file may be an oversight on our end, unsure as pretty much all Swift repos use that same Apache license, not sure why your linter is pointing just that one out.

For now I'm going to let the Debian build scripts copy over the file and see if that resolves things.

* The ICU issue needs to be investigated, could be legit. You could also try removing the Swift build of libicu and just use the Debian system package, which should work too but ~will~may require you to bump the Swift package anytime the system libicu is updated.

Well, wouldn't you know it. Debian's libicu-dev package has the exact same error on libicudata.a. So I guess nothing to worry about there.

* Those libcompiler-rt shared libraries don't need executable permissions, probably a mistake on our end.here

I was able to get those fixed by manually running chmod at the appropriate place in the Debian packaging flow. There is a tool that is supposed to fix this automatically for me so maybe it has a bug. I will have to look into that. At no point in the build process do I see the swift tools setting these files to anything but 0644 and they don't become 0755 until they are copied out into the final directory structure so I'm not sure yet who is doing it.

* I have no idea about that debug file with no symbols, haven't used that.

Yah, probably not an issue.
I've just realized that the person that originally wanted to package Swift for Debian back in 2015 was and is a Debian Developer with the LLVM packaging team. If I can pique his interest to sponsor/mentor this package then he already has half of the answers and might even jump at the opportunity to get involved with it again. Crossing my fingers!

I'm at the point of trying to get a Debian Developer to sponsor my package and clean up any policy violations. The violation that was discovered first is that there is an existing Debian packaged named "swift" and they supply a binary named "swift". Since their binary does something completely different I am not allowed to have a binary named "swift" no matter how unlikely it is for someone to want to use the swift that is designed for server object storage and also compile Swift programs on that same server. I could use the "conflict" mechaism to prevent my package and theirs from being installed at the same time, but policy does not allow that unless the binaries provide the same functionality.
The recommended action is to contact the debian-devel mailing list to start a discussion of who has to change their name, or if both have to change their name. At the same time I could ask for an exception to the "conflict" mechanism rule.
I guess my main question here is that since "swift" and "swiftc" are both just symbolic links to "swift-frontend" could I drop "swift" from my package and thus require Debian users to always call swiftc instead? Would that break anything other than reams of documentation?

since "swift" and "swiftc" are both just symbolic links to "swift-frontend" could I drop "swift" from my package and thus require Debian users to always call swiftc instead?

swift has two purposes: it invokes the REPL when run alone and it interprets a Swift file when given one. As such, you could always drop it as long as you don't care about those two uses, or look into renaming it instead.

Would that break anything other than reams of documentation?

No, I doubt there's even that much doc on it.

I don't know yet if this will be allowed by the Debian policy, but what I have done is create a set of scripts such that during the installation process if /usr/bin/swift does not exist the user is presented with a prompt asking if they wish to add it or not. If an actual binary file is located at /usr/bin/swift instead of a symbolic link then a message is printed out warning that "swift" already exists and it cannot be used to call the Swift REPL or compiler/interpreter.

I've been copying /usr/bin/ls to /usr/bin/swift to test the Debian package install in the case where the "swift" package is installed. If I forget to remove this copy before building Swift then it tries to build newswiftdriver with /usr/bin/swift and fails with a less than helpful message. I'm not sure what all I can do on my end to address this (and may not need to do anything), but I'm wondering if Swift can do a sanity check on /usr/bin/swift before trying to build with it. At a minimum it could verify that it is indeed Swift and is past some minimum version since we know Swift 5.5.1 won't work.

I'm wondering if Swift can do a sanity check on /usr/bin/swift before trying to build with it. At a minimum it could verify that it is indeed Swift and is past some minimum version since we know Swift 5.5.1 won't work.

Are you sure the compiler build even uses swift and not swiftc? If so, feel free to submit a pull to do that for the swift toolchain build.

Are you sure the compiler build even uses swift and not swiftc? If so, feel free to submit a pull to do that sanity check for the swift toolchain build.

I tried to reproduce this error on a Swift 5.5.1 build, but was unable to. Here is the log file from trying to build Swift 5.6.2 when Swift is not installed but /usr/bin/swift is some other binary than Swift (copied from /usr/bin/ls in this case):
swift/utils/build-script --preset=buildbot_linux,no_test install_destdir=/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2 installable_package=/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swiftlang_5.6.2-1-debian.tar.gz
[swift/utils/build-script] NOTE: using preset "buildbot_linux,no_test", which expands to

swift/utils/build-script --assertions --swift-enable-ast-verifier=0 --no-swift-stdlib-assertions '--swift-install-components=autolink-driver;compiler;clang-resource-dir-symlink;stdlib;swift-remote-mirror;sdk-overlay;parser-lib;toolchain-tools;license;sourcekit-inproc' '--llvm-install-components=llvm-cov;llvm-profdata;IndexStore;clang;clang-resource-headers;compiler-rt;clangd;lld;LTO' --llbuild --swiftpm --swift-driver --xctest --libicu --libcxx --swiftdocc --build-ninja --install-llvm --install-swift --install-lldb --install-llbuild --install-swiftpm --install-swift-driver --install-xctest --install-libicu --install-prefix=/usr --install-libcxx --install-sourcekit-lsp --install-swiftdocc --build-swift-static-stdlib --build-swift-static-sdk-overlay --build-swift-stdlib-unittest-extra --test-installable-package --toolchain-benchmarks --install-destdir=/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2 --installable-package=/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swiftlang_5.6.2-1-debian.tar.gz --relocate-xdg-cache-home-under-build-subdir --build-subdir=buildbot_linux --lldb --release --test --validation-test --long-test --stress-test --test-optimized --foundation --libdispatch --indexstore-db --sourcekit-lsp --swiftdocc '--lit-args=-v --time-tests' --lldb-test-swift-only --install-foundation --install-libdispatch --reconfigure --skip-test-cmark --skip-test-lldb --skip-test-swift --skip-test-llbuild --skip-test-swiftpm --skip-test-swift-driver --skip-test-xctest --skip-test-foundation --skip-test-libdispatch --skip-test-playgroundsupport --skip-test-libicu --skip-test-indexstore-db --skip-test-sourcekit-lsp --skip-test-swiftdocc

[swift/utils/build-script] NOTE: Using toolchain default
+ cmake --version
+ mkdir -p /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/build/buildbot_linux
+ rm -rf /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/build/buildbot_linux/ninja-build
+ cp -r /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/ninja /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/build/buildbot_linux/ninja-build
+ pushd /home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/build/buildbot_linux/ninja-build
+ env CXX=/usr/bin/clang++ /usr/bin/python3 configure.py --bootstrap
[1/29][  3%][0.045s] INLINE build/browse_py.h
[2/29][  6%][0.057s] CXX build/debug_flags.o
[3/29][ 10%][0.635s] CXX build/browse.o
[4/29][ 13%][0.787s] CXX build/depfile_parser.o
[5/29][ 17%][0.971s] CXX build/disk_interface.o
[6/29][ 20%][1.180s] CXX build/clparser.o
[7/29][ 24%][1.208s] CXX build/edit_distance.o
[8/29][ 27%][1.449s] CXX build/dyndep.o
[9/29][ 31%][1.652s] CXX build/deps_log.o
[10/29][ 34%][2.033s] CXX build/clean.o
[11/29][ 37%][2.073s] CXX build/line_printer.o
[12/29][ 41%][2.137s] CXX build/dyndep_parser.o
[13/29][ 44%][2.163s] CXX build/build_log.o
[14/29][ 48%][2.252s] CXX build/lexer.o
[15/29][ 51%][2.430s] CXX build/graphviz.o
[16/29][ 55%][2.691s] CXX build/metrics.o
[17/29][ 58%][2.967s] CXX build/version.o
[18/29][ 62%][3.019s] CXX build/eval_env.o
[19/29][ 65%][3.108s] CXX build/string_piece_util.o
[20/29][ 68%][3.210s] CXX build/parser.o
[21/29][ 72%][3.246s] CXX build/util.o
[22/29][ 75%][3.353s] CXX build/graph.o
[23/29][ 79%][3.606s] CXX build/subprocess-posix.o
[24/29][ 82%][3.873s] CXX build/manifest_parser.o
[25/29][ 86%][3.964s] CXX build/build.o
[26/29][ 89%][4.120s] CXX build/state.o
[27/29][ 93%][4.192s] AR build/libninja.a
[28/29][ 96%][4.692s] CXX build/ninja.o
[29/29][100%][4.813s] LINK ninja
bootstrapping ninja...
warning: A compatible version of re2c (>= 0.11.3) was not found; changes to src/*.in.cc will not affect your build.
wrote build.ninja.
bootstrap complete.  rebuilding...
+ popd
--- Building earlyswiftdriver ---
Traceback (most recent call last):
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/build-script", line 722, in <module>
    sys.exit(main())
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/build-script", line 717, in main
    return main_normal()
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/build-script", line 673, in main_normal
    invocation.execute()
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/build_script_invocation.py", line 669, in execute
    self._execute(pipeline, all_host_names)
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/build_script_invocation.py", line 727, in _execute
    self.execute_product_build_steps(product_class, host_target)
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/build_script_invocation.py", line 783, in execute_product_build_steps
    product.build(host_target)
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/products/earlyswiftdriver.py", line 69, in build
    run_build_script_helper('build', host_target, self, self.args)
  File "/home/swm/workspace/debian/build/swiftlang/swiftlang_5.6.2/swift/utils/swift_build_support/swift_build_support/products/earlyswiftdriver.py", line 107, in run_build_script_helper
    swiftc_path = os.path.abspath(product.toolchain.swiftc)
  File "/usr/lib/python3.10/posixpath.py", line 378, in abspath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType
ERROR: command terminated with a non-zero exit status 1, aborting

I'm not likely to be doing any debug or pull requests on this any time soon as I'm focused on packaging.

I had a similar issue with Swift being installed while packaging, but that was a mistake on my part (testing one version while building another) and the Fedora builders[1] won't have Swift installed by default.
[1] Dunno how Debian works, but all packages in the Fedora/RHEL environment are built from source on Fedora-controlled hardware, essentially in a special spun-up container; if I don't include it in the %BuildRequires section, it won't be in the container (which was how I discovered building Swift requires rsync 🙃).

Yes, it is the same with Debian. I should be using chroot and a clean build environment each time, but I'm not there yet. There is pretty much no chance that the official Debian package would ever get built with that other "swift" package installed. I have rsync in my build dependencies as well, not because I discovered that on my own, but from benefiting from your work. :-)
I've still got some things to sort out, but I'm going to ask if what I have now is good enough to upload to Debian as a starting point.

out of curiosity, and because I don't have a Debian box handy, what is the other swift program? In Fedora's case there is another swift, but it's part of the OpenStack Object Storage API Python library, so the chance of a conflict with Swift® is effectively nil.

On Debian there is another package "swift" but it doesn't actually provide the binary /usr/bin/swift, that is provided by the related python-swiftclient and they are indeed part of the OpenStack Object Storage API Python library.
$ sudo nala install python3-swiftclient
$ which swift
/usr/bin/swift
$ swift --version
python-swiftclient 3.13.1

Not sure if I got the reviewer on a good day, but I basically said that there is practically zero chance of conflict, given the usage domains were so wildly different; a system using OpenStack Storage is unlikely to also be a developer's machine wanting to use a cool programming language.

Here is the log file from trying to build Swift 5.6.2 when Swift is not installed but /usr/bin/swift is some other binary than Swift (copied from /usr/bin/ls in this case)

It looks like it's reporting correctly that there's no swiftc, though the error is a bit weird. It has nothing to do with the fake swift binary you added.

It looks like it's reporting correctly that there's no swiftc, though the error is a bit weird. It has nothing to do with the fake swift binary you added.

And yet I can repeatedly turn this error on and off by copying a binary to /usr/bin/swift or deleting it.

Not sure if I got the reviewer on a good day, but I basically said that there is practically zero chance of conflict, given the usage domains were so wildly different; a system using OpenStack Storage is unlikely to also be a developer's machine wanting to use a cool programming language.

Good to know. I've asked my package sponsor how best to handle this and mentioned this as being what Fedora has done. I think the only issue would be not having separate runtime libraries and needing to install swiftlang to get them. Have you done any work on exploring separating out the runtime libraries into their own package?

And yet I can repeatedly turn this error on and off by copying a binary to /usr/bin/swift or deleting it.

Ah, you're right, found it. The current build checks for both swift and swiftc and tries to build the early swift driver if it finds just the former, as it assumes both will be installed together. Obviously, that assumption will break if your other OpenStack package is installed instead. I wouldn't worry about this, as very few will likely build the Swift toolchain, and almost nobody will do it with that other Swift package installed.

@tachoknight my Debian sponsor has asked me to move from the Swift provided llvm, cmake, and icu packages to those that come with Debian. I know that lldb is customized by Swift, but what of these others? Do you know and/or do you have a blog post about that sort of issues that I can point him to? He is one of the Debian LLVM package maintainers so he is very experienced in this area.
Of course anyone else that knows is welcome to chime in too.

This was my experience as well. The Swift-provided version of LLVM/LLDB is custom for Swift; the standard LLVM/LLDB you might install on any Debian or Fedora box knows nothing about Swift and it simply won't work. The solution I use in Fedora is to hide everything in /usr/libexec/swift and just symlink swift and swiftc to /usr/bin; the custom lldb executable for Swift is not on the path so there's no collision with an existing LLVM/LLDB installation.
Two things I can remember too: When I was originally getting it working with Fedora, I discovered that a lot of of the code assumes executables are in the same directory, or if dynamically loading a library, it will explicitly say something like ../lib/libFoo.so or whatever. The Swift toolchain is a complete package unto itself and thus it really needs to stay together; many early bugs with Swift on Fedora was simply due to constantly missing a spot where I had tried to redirect the loader to a more general location (e.g. Fedora wants all its 64-bit libs in /usr/lib64) and it was really frustrating. Thus when I was able to just drop everything into /usr/libexec/swift, it made life so much easier, and so many patches (and related bugs) just magically went away.
The other thing that has come up, and I want to say it's somewhere on the Swift forums, that there was some discussion about moving Swift upstream to be part of the LLVM project. I have no idea what, if any, plans there are to do that and there's no point in thinking about it (this is to head off the possibility of someone asking "Why don't they just move everything to the LLVM project?").

Thanks @tachoknight, I have forwarded your reply to my sponsor in the hopes that we will agree to keeping Swift together until upstream makes it possible for us to more easily split it apart.

there was some discussion about moving Swift upstream to be part of the LLVM project.

Not Swift, I believe they just want to slowly upstream all Swift changes from their forked LLVM/clang/lldb to the upstream LLVM repo and get rid of those LLVM forks one day, but I'm guessing that's probably years away.

Ah, that's right, I was remembering it incorrectly.

Thank you all for the information and help. My sponsor would like me to move to as many Debian packages as possible so here is our plan:

Switch to Debian cmake and ninja (Debian ftpmasters will reject this package otherwise)
Release swiftlang to Debian Experimental
Switch to Debian icu, llvm, and clang
Release to Debian Testing (from here it migrates to unstable after 5 days then to stable releases and derivative distributions)

Out of curiosity, what does it mean to switch to the Debian llvm and clang?

Out of curiosity, what does it mean to switch to the Debian llvm and clang?

I'm hoping that will be open for negotiation. The desire is to not be packaging yet another fork of llvm/lldb/clang and having to deal with its bugs, but how do I make use of a tool chain that doesn't support my language? Start submitting all of Swift's changes as pull requests to Debian's LLVM packages? At that point it seems like you're still having to maintain a fork, but now those bugs impact far more Debian users than just the swiflang users. In any case, I must at least give it the old college try.

1. These are build dependencies so not sure why anyone cares which you use, but should be no problem using the Debian packages, provided you specify the right minimum versions to use.
3. That's only for the final package, you still have to build the Swift-forked LLVM and clang from source for the Swift compiler to work. Also, if you don't build the Swift-forked lldb, not only won't the debugger recognize Swift but you won't be able to use the REPL.

The way I see it, the fact that there's an lldb executable in the Swift toolchain's bin directory is just part of the Swift installation; if it had been renamed to xyzzy this wouldn't be anything to discuss. You're not maintaining a fork of LLVM, Apple is, as part of the Swift project. Nothing related to bugs or issues that crop up in the Swift-specific version of LLVM/LLDB applies to the standard LLVM/LLDB project; nobody interacts with the underlying binaries in the Swift toolchain other than swift and swiftc; if I want to write C++ code, I use the system-provided clang/llvm.
Basically my argument is: Swift is a complete package, a veritable iceberg of tools and libraries that should mean nothing to the user of Swift other than what is exposed via /usr/bin; whether the package includes clang, lldb, icu, whatever, it's not going to collide with the existing system-provided equivalents.

1. These are build dependencies so not sure why anyone cares which you use, but should be no problem using the Debian packages, provided you specify the right minimum versions to use.
Right, Fedora uses the system-provided Clang/LLVM to build the Swift equivalent.
3. That's only for the final package, you still have to build the Swift-forked LLVM and clang from source for the Swift compiler to work. Also, if you don't build the Swift-forked lldb, not only won't the debugger recognize Swift but you won't be able to use the REPL.
And I tried to make this work and it simply does not and it makes total sense, the default LLDB has no knowledge of Swift so why should it work?

Ultimately what I think the reviewers are confused is why should there be a whole separate LLVM/LLDB/etc. just for Swift, and the answer is to not think of it as the standard LLVM/LLDB; they're special, part of the Swift toolchain, and maybe it would have been easier to name the binaries something else (e.g. swift-lldb) so it was clear this was something not for general use.

It might be an idea to break the swift installation into smaller parts. Something like a swift-lldb package that "conflicts" with the lldb package might be a way to move forward since they provide effectively the same base functionality.
Does Foundation need to be in the same package as the compiler? A separate swift-Foundation package would mean that you could tackle the ICU details later, while still getting the compiler in start writing programs that don't need Foundation.
Clang might be a tricky spot. We do pretty regularly make special changes to the clang packaged in Apple/llvm-project that may or may not eventually get upstreamed into the actual clang. I think we're currently around feature-parity from my understanding, (though with the experimental C++ interop stuff, that might be changing day-by-day). At the very least, you will need to ensure that whatever C++ compiler you use to compile the Swift runtimes, it supports the Swift calling convention and the Swift async calling conventions. That's guaranteed to work from the right branch of the apple/llvm-project. It might work from another clang. It won't work from GCC.

It might be an idea to break the swift installation into smaller parts. Something like a swift-lldb package that "conflicts" with the lldb package might be a way to move forward since they provide effectively the same base functionality.

Tried it. The confusion is that the versions of LLVM/LLDB/clang between Swift and the standard LLVM installation are mismatched and any bugs that might be encountered when working with C++ code, for example, would go to the wrong place; the user may think it's a LLVM bug and file an issue with the LLVM maintainer, who would be left puzzled as to what they were dealing with. This happened to me on Ubuntu where I had Swift installed in my home directory, with its bin in the path, and had all kinds of weird issues when I was switching back and forth between wanting to work with C++ code or Swift code. The symlink of just swift and swiftc made all the difference and I could easily upgrade either Swift or system LLVM with no issues.
Ultimately, any attempt at changing the way Swift expects to work results in sadness. Even worse, what's valid today may not work tomorrow, and keeping Swift in its original tries-to-live-with-existing-LLVM configuration on Fedora was, frankly, overwhelming. Let the entire toolchain live somewhere (e.g. /usr/libexec/swift) and expose only what's necessary and everything just works. The thank-you emails, and the lack of reported bugs, attest to this. :)

Thank you all, your input and suggestions are very helpful. I have passed along some of them to my sponsor so that maybe he will relax the requirement that I try to make use of Debian llvm/clang/lldb. I guess I'll get started by trying to use Debian's cmake and ninja and go from here.
Debian policy prohibits shipping binaries with the same name as a binary from another package unless they have identical functionality. They also don't want you shipping duplicate libraries for security reasons. Policies are not carved into stone, but they are very difficult to get an exception to.

Does Foundation need to be in the same package as the compiler?

No, it doesn't. If he ever splits off the runtime libraries as discussed above, he'll likely bundle everything in /usr/lib/swift/, including the stdlib, together in a separate package from the compiler, and just have the compiler package list that runtime package as a dependency. You could further split the corelibs up individually as you suggest too: I added libdispatch as a separate package on Android, though that doesn't include the Swift wrapper.

They also don't want you shipping duplicate libraries for security reasons.

The Swift versions of LLVM and lldb are not duplicates, they are proper forks, as @tachoknight emphasized. Clang might not be that different anymore, as @etcwilde noted, but there have been significant changes in the past and still might be.
Only libicu is a stock version, but as noted, that is only done to make sure a particular tested version is always available for Swift alone and may not matter as much anymore now that the Swift stdlib stopped using it, #40340, with only the Swift Foundation library using it now. I use the system packages of clang/lld, libdispatch, and libicu as runtime dependencies my Android package of Swift and have not had a problem (I've not tried building the Swift-forked lldb for Android, so no debugger support or REPL yet, need to get on that), though I have no choice but to compile the Swift-forked LLVM when building the Swift compiler.

The Swift versions of LLVM and lldb are not duplicates, they are proper forks

They are proper forks that are kept roughly in sync with the upstream version (bi-annual-ish rebranch). They should be a proper superset of the functionality of the llvm/llvm-project at the point where the branch was cut, with the additional bits on top of that providing better Swift support. If things are broken on the C++ side, that sounds like a bug. We should be setting the BUG_REPORT_URL appropriately so that folks report bugs in the right places though.

I have deleted the Swift provided/expected local copies of cmake and ninja and turned them build dependencies with a minimum version of what was expected by Swift. I modified build-presets.ini to disable build-ninja for the build that I was using. Is that an appropriate way to stop Swift from trying to build the now missing ninja?
No errors yet relating to the missing cmake so it must have sorted itself out. I'll know for sure in about 4 hours.
I am really looking forward to my new CPU showing up soon as it should cut my build times to about half of what they are now. I have a fanless HDPLEX H3 case so I can't use those high end high TDP CPUs.

I have deleted the Swift provided/expected local copies of cmake and ninja and turned them build dependencies with a minimum version of what was expected by Swift. I modified build-presets.ini to disable build-ninja for the build that I was using. Is that an appropriate way to stop Swift from trying to build the now missing ninja?

Yes that's what I used to do; I decided that since CMake and Ninja were built just to build Swift, not part of the final package, and essentially thrown away, to just let them be built, but I get why you may not want to do that (yet).

No errors yet relating to the missing cmake so it must have sorted itself out. I'll know for sure in about 4 hours.

On a RPI 4, it takes around 24 hours. 🙃

I am really looking forward to my new CPU showing up soon as it should cut my build times to about half of what they are now. I have a fanless HDPLEX H3 case so I can't use those high end high TDP CPUs.

Fastest build time I've seen so far is my M1 MBP; 1.5 hours to build in a Docker container.

My build just generated a .deb package that seems to install and work just fine so I am going to declare the removal of cmake and ninja a success. I've still got an hour or so of linting to wait on for before the build is officially completed.
I've got to get me one of those new MBPs. About 1.5 years ago I replaced my 2012 15" MBPr with a System76 laptop as I didn't like the Apple options at the time. I hate to think how long a Swift build would take on old faithful. My new CPU has arrived so I'll be making the upgrade this evening.

@jblache has added DEB control files here https://github.com/apple/swift-installer-scripts/tree/main/platforms/Linux/DEB for Swift. @melizasw have you seen these?

@jblache has added DEB control files here https://github.com/apple/swift-installer-scripts/tree/main/platforms/Linux/DEB for Swift. @melizasw have you seen these?

Yes, @etcwilde had pointed those out to me. The issue isn't how to make a .deb file, but how to make one that complies with Debian Policies and builds on Debian testing. I had a working .deb file back on Swift 5.5.1, but it was miles away from being a completed Debian package that could become an official package. We are nearly there now, but my debian directory contains 32 files to accomplish that. In the eyes of Debian this is the official installer script location.
One example of the difference is that Debian takes copyrights very seriously so you will see that our version of the copyright file is much more accurate and detailed than the one that is provided in swift-installer-scripts. That doesn't make one wit of difference to the .deb file someone builds for their own use, but will get a package kicked out of the official Debian repository.

Happily it was an easy update from 5.6.2 to 5.6.3 and for fun I tested the Debian built .deb on Pop_OS.

I've been working on switching ICU over to the Debian provided packages and just want to check to see if I've done it correctly.
To disable ICU in the Swift build I removed "libicu" and "install-libicu" from build-presets.ini, made my build depend on libicu-dev, and my package have a dependency on the runtime library libicu71. Swift builds just fine, but is this a good way to make the switch in ICU and how can I test to make sure that ICU is indeed working in my Swift package?

is this a good way to make the switch in ICU and how can I test to make sure that ICU is indeed working in my Swift package?

Yep, the build will pick up the system libicu automatically. To test, check if Foundation is linked against libicu readelf -d /usr/lib/swift/linux/libFoundation.so | ag NEEDED, and maybe check if some small programs built against Foundation work.

I'm not sure what "ag" is so I substituted in "grep".
$ readelf -d /usr/libexec/swift/lib/swift/linux/libFoundation.so | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libswiftDispatch.so]
 0x0000000000000001 (NEEDED)             Shared library: [libicuuc.so.71]
 0x0000000000000001 (NEEDED)             Shared library: [libicui18n.so.71]
 0x0000000000000001 (NEEDED)             Shared library: [libicudata.so.71]
 0x0000000000000001 (NEEDED)             Shared library: [libdispatch.so]
 0x0000000000000001 (NEEDED)             Shared library: [libBlocksRuntime.so]
 0x0000000000000001 (NEEDED)             Shared library: [libswiftGlibc.so]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libswift_Concurrency.so]
 0x0000000000000001 (NEEDED)             Shared library: [libswiftCore.so]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

So it did get linked, which I can also find in the build log:
Found the following ICU libraries:
   uc (required): /usr/lib/x86_64-linux-gnu/libicuuc.so
   i18n (required): /usr/lib/x86_64-linux-gnu/libicui18n.so
   data (optional): /usr/lib/x86_64-linux-gnu/libicudata.so
Found ICU: /usr/include (found version "71.1")

Also, this message implies that symbols from libicuuc.so and libicui18n.so are getting used in libFoundation.so:
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/swiftlang/usr/libexec/swift/lib/swift/linux/libFoundation.      so was not linked against libicudata.so.71 (it uses none of the library's symbols)
And if I'm not mistaken, structs come from the Foundation framework:

And if I'm not mistaken, structs come from the Foundation framework

No, they come from the compiler and stdlib. Try running plutil --help, that's an executable provided by Foundation.

Not this one...
$ sudo apt-file search /usr/bin/plutil
gnustep-base-runtime: /usr/bin/plutil

Are you saying that simply running plutil proves that libicu is linking correctly?
$ /usr/libexec/swift/bin/plutil -help
plutil: [command_option] [other_options] file...
The file '-' means stdin
Command options are (-lint is the default):
 -help                         show this message and exit
 -lint                         check the property list files for syntax errors
 -convert fmt                  rewrite property list files in format
                               fmt is one of: xml1 binary1 json
 -p                            print property list in a human-readable fashion
                               (not for machine parsing! this 'format' is not stable)
There are some additional optional arguments that apply to -convert
 -s                            be silent on success
 -o path                       specify alternate file path name for result;
                               the -o option is used with -convert, and is only
                               useful with one file argument (last file overwrites);
                               the path '-' means stdout
 -e extension                  specify alternate extension for converted files
 -r                            if writing JSON, output in human-readable form
 --                            specifies that all further arguments are file names

Are you saying that simply running plutil proves that libicu is linking correctly?

Yes, the dynamic linker generally will not run the executable otherwise. Of course, that's no guarantee that it actually invoked any functions from libicu, but it's a simple sanity check to try. If you really want to check it, you'd need to track down which functions in the C CoreFoundation library use libicu functions, then invoke those in some sample code.

My package sponsor has been trying to build swiftlang using a tool called cowbuilder but is experiencing failed builds. Cowbuilder uses a clean environment in /var/cache/pbuilder/ using just the packages listed as build dependencies. I've been able to reproduce the error and it happens at the very end of the build when running some tests.
With cowbuilder:
Failed Tests (2):
  swift-package-tests :: swift-package-init-exec.md
  swift-package-tests :: swift-package-init-lib.md

Testing Time: 5.11s
  Unsupported: 10
  Passed     : 19
  Failed     :  2

With my usual debuild:
Testing Time: 4.19s
   Unsupported: 10
   Passed     : 21

I've attached the cowbuilder log file from where it started the tests until it the end. To me this seems to be the pertinent error:
Input was:
<<<<<<
        1: warning: Failed creating default security location, Error Domain=NSCocoaErrorDomain Code=513 "You don’t have permission."
not:42     !~~~~~~                                                                                error: no match expected
        2: warning: Failed creating default configuration location, Error Domain=NSCocoaErrorDomain Code=513 "You don’t have permission."
        3: warning: Failed creating default cache location, Error Domain=NSCocoaErrorDomain Code=513 "You don’t have permission."
        4: Building for debugging...
        5: [1/4] Emitting module Project
        6: [2/4] Compiling Project main.swift
        7: [5/6] Wrapping AST for Project for debugging
        8: [6/6] Linking Project
        9: Build complete! (1.77s)
>>>>>>

Apparently these two tests are trying to create directories but are failing to do so. I can't tell what directory they are trying to create so that I can figure out why permissions are inadequate. Any help would be appreciated
swiftlang_cowbuild_errors.txt

SPM tries to create some directories in the home directory, is that disallowed by cowbuilder?

SPM tries to create some directories in the home directory, is that disallowed by cowbuilder?

I checked the area where the build took place and there is a /root directory with .bashrc in it and an empty /home directory. Since cowbuilder was ran as sudo I think ~/.swiftpm must have resolved to the /home directory, but since that is empty I would have expected a failure to create directory instead of a permissions error. I'm not familiar enough with cowbuilder to know if there is something I can tweak. Maybe I'll su - before running cowbuilder and see if that works.

Maybe I'll su - before running cowbuilder and see if that works.

Nope, no change in the error messages. I'm not sure what to make of it.

Can you run cowbuilder as a normal user that has a home directory, or won't it work that way?

I'll have to see if that is possible, it is a good idea. I'm not keen on having to use sudo to do a package build and when I partitioned my drive I wasn't planning to put multiple ~20GB builds into /var. I can only do one build at a time without filling /var, and even then I can't guarantee that I didn't fill up /var as cowbuilder does an rm -rf on the build directory when it detects a failure.

I did get cowbuilder to complete successfully by adding BUILD_HOME = /build to  /etc/pbuilderrc . It didn't work to put it into ~/.pbuilderrc, if that is what it meant for me to do. Hopefully this isn't something that Debian runs on packages in a way that doesn't allow passing an argument like that to it.

I ended up just deleting the parts of those two tests that parsed the logs for warning and errors. That has got cowbuilder running happily on the default settings so there shouldn't be any issues from that when running builds on Debian servers.
I also got a build made without the Swift provided lldb. This included removing "lldb" and "install-lldb" from build-presets.ini as well as some edits to tests so they wouldn't fail from the missing repl_swift and to point it to the Debian provided lldb. As expected, Swift REPL did not work, but what I did not expect is that normal lldb on a Swift built binary didn't work in that I couldn't set any break points.
(lldb) b 2
error: No selected frame to use to find the default file.
error: No file supplied and no default file available.

I'm sure that is related to removing lldb from the packaging process as the Debian provided lldb used to work with Swift built binaries. Unexpected, but not surprising. I am not planning to debug this and at present have no open issues related to packaging Debian for Swift.

Hi,

@jblache has added DEB control files here https://github.com/apple/swift-installer-scripts/tree/main/platforms/Linux/DEB for Swift. @melizasw have you seen these?

Yes, @etcwilde had pointed those out to me. The issue isn't how to make a .deb file, but how to make one that complies with Debian Policies and builds on Debian testing. I had a working .deb file back on Swift 5.5.1, but it was miles away from being a completed Debian package that could become an official package. We are nearly there now, but my debian directory contains 32 files to accomplish that. In the eyes of Debian this is the official installer script location.

I'll direct you to previous discussions on the subject for the rationale and the current approach; as we've said all along, help is welcome tackling the packaging, and you seem to have done some good work in that direction.
As far as criteria for inclusion in Debian, having all the files in the right place is necessary but not sufficient; as has been noted in the thread, source duplication and embedded shared libraries are two big issues we're facing here.
In the case of LLVM sources, the likely solution is to have the LLVM package in Debian follow in the footsteps of the GCC package and provide an llvm-X-source package. Swift can then be shipped as a patch against the LLVM sources.
Shared libraries de-embedding is an equally complicated affair, I believe I covered that previously.

I'm working on splitting up the Swift packing into four. A doc package, a dev package containing headers, a lib package containing runtime libraries, and the main package containing the binaries and everything else that doesn't fit well into the other packages.
So in regards to the doc package, is docc something that should be considered documentation that goes into the doc package or is it better considered a tool that belongs in the main package?
And for the runtime libraries, does this list look correct?
/usr/libexec/swift/lib/swift/linux/libBlocksRuntime.so
/usr/libexec/swift/lib/swift/linux/libFoundation.so
/usr/libexec/swift/lib/swift/linux/libFoundationNetworking.so
/usr/libexec/swift/lib/swift/linux/libFoundationXML.so
/usr/libexec/swift/lib/swift/linux/libXCTest.so
/usr/libexec/swift/lib/swift/linux/lib_InternalSwiftScan.so
/usr/libexec/swift/lib/swift/linux/lib_InternalSwiftSyntaxParser.so
/usr/libexec/swift/lib/swift/linux/libdispatch.so
/usr/libexec/swift/lib/swift/linux/libswiftCore.so
/usr/libexec/swift/lib/swift/linux/libswiftDispatch.so
/usr/libexec/swift/lib/swift/linux/libswiftGlibc.so
/usr/libexec/swift/lib/swift/linux/libswiftRemoteMirror.so
/usr/libexec/swift/lib/swift/linux/libswiftSwiftOnoneSupport.so
/usr/libexec/swift/lib/swift/linux/libswift_Concurrency.so
/usr/libexec/swift/lib/swift/linux/libswift_Differentiation.so
/usr/libexec/swift/lib/swift/linux/libswift_MatchingEngine.so
/usr/libexec/swift/lib/swift/linux/libswift_StringProcessing.so

Thanks

docc is definitely a tool that belongs in the main package alongside swiftc.

lib_InternalSwiftScan.so and lib_InternalSwiftSyntaxParser.so are host libraries for the tools, as tipped off by the _Internal prefix, so I don't think you should include those with the runtime package. Others look good.

Thanks for the help. I didn't have much time today to work on this, but I did get the four packages build and with nothing but the runtime package installed I was able to run a binary that I had previously built. One point I'm not sure about is if the runtime libraries should be needed for REPL and swiftc as both complained if libswiftCore.so was not present, I'll dig into that a bit more as maybe there are a few more libraries that belong in the main package instead of runtime.
Unfortunately the REPL and swiftc of these new packages are both broken, it seems to be an issue in finding libraries. My guess is that I've missed a few files. When making a single package you just tell it to take everything in directory A and install it to path B, a simple one line command for the whole package. But slicing and dicing up the files between multiple packages means having to use multiple commands to copy the needed files and that introduces lots of places for errors. Hopefully I'll get that sorted out soon.

One point I'm not sure about is if the runtime libraries should be needed for REPL and swiftc

The runtime package should be a dependency of the main toolchain package, both because the compiler itself and several other tools link against the runtime libraries and you can't build new Swift executables without linking against the runtime.

It turns out that my broken packages was due to some missing module.map files from locations such as /usr/lib/swift/CoreFoundation. Forgive my ignorance, should these module.map files go into the dev package with all of the *.h files or into the main package with the binaries?

module.modulemap files generally go in the header file with the .h files.

CoreFoundation should have it's modulemap files though.
Running find . -iname 'module.modulemap' in CoreFoundation
./URL.subproj/module.modulemap
./Parsing.subproj/module.modulemap
./Base.subproj/module.modulemap

Check that those are getting installed with their headers.

My sponsor tried to upload the Swift package and the Debian servers rejected it due to an embedded libyaml library. The idea is to use Debian's libyaml-dev, but I suspect this won't work as it seems that yams is providing libyaml and yams != libyaml. I'm doing a test build now, but just wanted to confirm my suspicion that I can't replace yams with libyaml and that we will need to override this lint error.
swiftlang: lintian output: 'embedded-library usr/libexec/swift/bin/sourcekit-lsp: libyaml', automatically rejected package.
swiftlang: If you have a good reason, you may override this lintian tag.
swiftlang: lintian output: 'embedded-library usr/libexec/swift/bin/swift-build-sdk-interfaces: libyaml', automatically rejected package.
swiftlang: If you have a good reason, you may override this lintian tag.
swiftlang: lintian output: 'embedded-library usr/libexec/swift/bin/swift-driver: libyaml', automatically rejected package.
swiftlang: If you have a good reason, you may override this lintian tag.
swiftlang: lintian output: 'embedded-library usr/libexec/swift/bin/swift-package: libyaml', automatically rejected package.
swiftlang: If you have a good reason, you may override this lintian tag.

The CYaml target from Yams is identically libyaml, so it's true that every executable target depending on Yams will embed a copy of libyaml. I have tried to build Yams against libyaml-dev (with SwiftPM) and it just works.
I read through the Debian policies and, although it's technically possible to override the tag, it seems this is still likely being rejected because libyaml is already provided by Debian.
Since Yams is owned and maintained by @jpsim, I think it's eventually up to him if CYaml can be replaced by system-wide libyaml. Not too many technical blockers here, but it adds to build-system complexity.

@stevapple Thanks for explaining it to me. It sounds like I need to modify Yams to pull in the system libyaml instead of building its own CYaml library which is a duplication of libyaml. I haven't got it working yet, but that is what I am trying to do now.

It took me a while to get both the Swift code and Cmake files changed to pull in the system's libyaml, but it is now working. The swiftlang package is in the NEW queue for Debian Experimental. From here it is a matter of waiting days or weeks until it gets reviewed and either accepted or rejected. And then after that it still needs to get into Unstable before it goes anywhere useful. This is a major milestone and my thanks to everyone that has helped.

Congrats @melizasw! Hope it all goes well, will be very cool to apt-get install swiftlang! Out of curiosity, what architectures are you building it for, x864_64 as well as aarch64?

Some questions around libswiftlang:

Is it required to extract it to a separate package? I believe so far the community still don’t agree on which part of Swift is runtime, toolchain or SDK except for the established pattern on macOS — and the macOS case is so special that it may not work on other platforms. I wish to delay separate packaging until the issue is fully discussed.
Is the name libswiftlang the only option in respect of Debian conventions? If feels a little weird to me though.
I see ./usr/libexec/swift/lib/swift/linux/libXCTest.so in libswiftlang and I would say, on both macOS and Windows we’re not treatingXCTest as a runtime library. It is part of the development suite which shouldn’t leak into the system.
If we want to make libswiftlang a usable thing, is ./usr/lib a more suitable place than ./usr/libexec/swift/lib?

Congrats @melizasw! Hope it all goes well, will be very cool to apt-get install swiftlang! Out of curiosity, what architectures are you building it for, x864_64 as well as aarch64?

Uhhh. Yes? I admit I'm not very familiar with the packaging process yet. But in the past new applications would get built for all architectures for which all dependencies could be met. The swiftlang package is marked as having an architecture of "any" so I think it will be the same way.

Since Yams is owned and maintained by @jpsim, I think it's eventually up to him if CYaml can be replaced by system-wide libyaml. Not too many technical blockers here, but it adds to build-system complexity.

@melizasw @stevapple I'm open to this, just file a PR and we can discuss the specifics.

No, it could have been swift_lang as it is in Fedora or swiftc as was done with rust. The package name swift is already taken. So I opted to use swiftlang which is the name that was used in the ITP bug that was created in 2015.

Ah, I was explicitly asking for libswiftlang, but now I get the naming at some point.

That would be preferred and I will give it a try, but I'm not sure how well the rest of Swift will handle having its libraries moved out of adjacent directories. I've found libswiftlang usable as-is by building a simple binary on Debian and running it on Pop_OS where only libswiftlang was installed.

My concern is that placing the libraries in /usr/libexec/swift/lib diverges from the current installation (and also the slim Docker image). And since SE-0342 will be a thing (hopefully soon), there should be less need for a runtime package.

Debian policy prohibits me from supplying /usr/bin/swift as there already exists one as provided by python3-swiftclient. Maybe we can talk them into picking a new name and freeing up /usr/bin/swift, but until then I cannot provide it. What I have done instead is provide a prompt during the installation of swiftlang to create the link to /usr/bin/swift if it does not already exist and the user selects to create it.

I see. Asking them to rename seems not so realistic, and your workaround makes sense👍 I wonder if this is a valid use case for update-alternative...

Only one package is necessary, which is the approach taken by @tachoknight for Fedora. My Debian sponsor requested a -dev and -doc package so he gets them

swiftlang-dev arouses new questions to me now:

Naming. I think libswiftlang-dev should be the correct name, to correctly indicate that the package is used for developing apps that use libswiftlang, not swiftlang (eg. some imaginary Swift IDE).
Contents. The C++ headers are one of the undecided pieces (on Windows we package it along with SDK, while on macOS both toolchain and SDK hold it). Also, Swift modules should inevitably be a part of development files IMO.

Some more thoughts:

It would be nice have new packages like libswiftlang-dev-arm64-cross to easily enable cross compilation. Actually adopting this feature will reflect some short-sighted designs in toolchain and layout.
I wonder if it's necessary to extract a swiftlang-common package, and keep swiftlang as simplified as what the user can directly see (and use).
It's interesting to see what could be in swiftlang-dev — the package that contains development files for the Swift compiler itself.
Are we going to extract sourcekit-lsp and docc (and even SwiftPM) into separate packages? Depending on use cases, not every Swift user is using these tools. And all these components are pluggable, so it should be fairly easy.🤔

My concern is that placing the libraries in /usr/libexec/swift/lib diverges from the current installation (and also the slim Docker image). And since SE-0342 will be a thing (hopefully soon), there should be less need for a runtime package.

That was an interesting read. One of the premises of SE-0342 is that Linux does not ship with Swift runtime libraries. This will no longer be true of Debian and its derivatives at some point. It is somewhat circular to say that libswiftlang is not needed due to SE-0342 and SE-0342 is needed (in part) because there are no runtime libraries on Linux. Obviously there is much more to consider with static vs dynamic linking, but giving the option of a runtime library package for now seems like a good idea to me. I'll look into changing that lib path though.

I see. Asking them to rename seems not so realistic, and your workaround makes sense👍 I wonder if this is a valid use case for update-alternative...

I don't think it would be allowed since the requirement for update-alternative is that they provide the same kind of functionality. I initially tried to flag swiftlang and being a conflict with package python3-swiftclient, but that isn't allowed unless they are identically functional but having different implementations. I'm not even sure asking the user to create a dynamic link during install will be permitted, but it makes sense to me and we decided to give it a try.
As for the rest of your comments, I appreciate you taking the time to leave them and I will spend more time later looking into them and thinking about what changes could be made to make the packaging better. But for now, lunch break is over.

Another significant milestone was reached today. My Swift 5.6.3 package was approved by the Debian FTP team and is now in experimental. That is still not accessible for most people to install, but it means that the FTP team approved of the various packaging methodologies and workarounds that were developed with your help. From here on out there is no manual review by the FTP team so when approved by my sponsor, it can move out of experimental and start making its way out into the world.
You can see some info about the package here: https://tracker.debian.org/pkg/swiftlang
On the left side under "binaries" you can click to see more info about each of the four packages and which files that they contain.
For now it is only building on "amd64" and failing or not installable due to missing build dependencies on all other architectures. It will take some time for me to work through the various build logs and see what is wrong. https://buildd.debian.org/status/package.php?p=swiftlang&suite=experimental
Next steps of course include updating the package to 5.9.1. I've not been following along so I don't know yet if Swift 5.9.x requires swift to build or not. Hopefully I can get this out in time for it to get pulled into Ubuntu 24.04 LTS.

My Swift 5.6.3 package was approved by the Debian FTP team and is now in experimental.

🎉🎉🎉

I've not been following along so I don't know yet if Swift 5.9.x requires swift to build or not.

It does if you’re going to enable macro/regex literal/… and other related features.
Since these are becoming mandatory dating back to Swift 5.7, things are actually going to be tricky if the Debian team don’t allow some “temporary/modified version” to be used for building the release package. Because Swift in Swift supports only the last minor release, you’ll at least have to:

Use the existing Swift 5.6.3 toolchain to build Swift 5.7.3;
Use the Swift 5.7.3 toolchain to build Swift 5.8.1;
Use the Swift 5.8.1 toolchain to build Swift 5.9.1.

Because Swift in Swift supports only the last minor release, you’ll at least have to:
1. Use the existing Swift 5.6.3 toolchain to build Swift 5.7.3;

2. Use the Swift 5.7.3 toolchain to build Swift 5.8.1;

3. Use the Swift 5.8.1 toolchain to build Swift 5.9.1.

Thanks for the roadmap. Once out of experimental and into unstable I can start working my way up the versions. My sponsor says that we can leave experimental when ever I feel the package is ready, and I think that will be after I fix the non-amd64 builds that should be working, especially arm64.

You can always ask for assistance if you wish:)

Ok. I've been working on the failures to build on other architectures. Some fail because Swift does not support them:
File "/<<PKGBUILDDIR>>/swift/utils/swift_build_support/swift_build_support/targets.py", line 348, in host_target
    raise NotImplementedError('System "%s" with architecture "%s" is not '
NotImplementedError: System "Linux" with architecture "armv8l" is not supported

And the others fail because -fstack-clash-protection is being called on an architecture in which Clang 13.0 does not support that option:
-- Build files have been written to: /<<PKGBUILDDIR>>/build/buildbot_linux/libdispatch-linux-aarch64
+ popd
/<<PKGBUILDDIR>>
<string>:1: DeprecationWarning: 'pipes' is deprecated and slated for removal in Python 3.13
+ /usr/bin/cmake --build /<<PKGBUILDDIR>>/build/buildbot_linux/libdispatch-linux-aarch64 -- -j4 all
[1/82][  1%][0.084s] Building C object src/CMakeFiles/dispatch.dir/apply.c.o
FAILED: src/CMakeFiles/dispatch.dir/apply.c.o
/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-aarch64/bin/clang -DDISPATCH_USE_DTRACE=0 -DHAVE_CONFIG_H -D_GNU_SOURCE=1 -Ddispatch_EXPORTS -I/<<PKGBUILDDIR>>/build/buildbot_linux/libdispatch-linux-aarch64 -I/<<PKGBUILDDIR>>/swift-corelibs-libdispatch -I/<<PKGBUILDDIR>>/swift-corelibs-libdispatch/src -I/<<PKGBUILDDIR>>/build/buildbot_linux/libdispatch-linux-aarch64/src -I/<<PKGBUILDDIR>>/swift-corelibs-libdispatch/private -I/<<PKGBUILDDIR>>/swift-corelibs-libdispatch/src/BlocksRuntime -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -mbranch-protection=standard -O3 -DNDEBUG -std=gnu11 -fPIC -fvisibility=hidden -Werror -Wall -Wextra -Warray-bounds-pointer-arithmetic -Wassign-enum -Watomic-properties -Wcomma -Wconditional-uninitialized -Wconversion -Wcovered-switch-default -Wdate-time -Wdeprecated -Wdocumentation -Wdouble-promotion -Wduplicate-enum -Wexpansion-to-defined -Wfloat-equal -Widiomatic-parentheses -Winfinite-recursion -Wmissing-prototypes -Wnewline-eof -Wnullable-to-nonnull-conversion -Wobjc-interface-ivars -Wover-aligned -Wpacked -Wpointer-arith -Wselector -Wshadow -Wshorten-64-to-32 -Wsign-conversion -Wstatic-in-inline -Wsuper-class-method-mismatch -Wswitch -Wunguarded-availability -Wunreachable-code -Wunused -Wno-unknown-warning-option -Wno-trigraphs -Wno-four-char-constants -Wno-disabled-macro-expansion -Wno-pedantic -Wno-bad-function-cast -Wno-c++-compat -Wno-c++98-compat -Wno-c++98-compat-pedantic -Wno-cast-align -Wno-cast-qual -Wno-documentation-unknown-command -Wno-format-nonliteral -Wno-missing-variable-declarations -Wno-old-style-cast -Wno-padded -Wno-reserved-id-macro -Wno-shift-sign-overflow -Wno-undef -Wno-unreachable-code-aggressive -Wno-unused-macros -Wno-used-but-marked-unused -Wno-void-pointer-to-int-cast -Wno-vla -Wno-error=assign-enum -fno-exceptions -fblocks -MD -MT src/CMakeFiles/dispatch.dir/apply.c.o -MF src/CMakeFiles/dispatch.dir/apply.c.o.d -o src/CMakeFiles/dispatch.dir/apply.c.o -c /<<PKGBUILDDIR>>/swift-corelibs-libdispatch/src/apply.c
clang-13: error: argument unused during compilation: '-fstack-clash-protection' [-Werror,-Wunused-command-line-argument]

The building and linking all go fine, but during the installation steps this compile is being called with -fstack-clash-protection for some reason. I am still working to figure out where this is coming from so that I can address it. Any help on this would be appreciated.

Ok. I've been working on the failures to build on other architectures. Some fail because Swift does not support them:
File "/<<PKGBUILDDIR>>/swift/utils/swift_build_support/swift_build_support/targets.py", line 348, in host_target
    raise NotImplementedError('System "%s" with architecture "%s" is not '
NotImplementedError: System "Linux" with architecture "armv8l" is not supported

For this specific case, ARMv8L is not likely to be supported, because Swift doesn't support 32-bit host. I believe it's supported as a target (i.e. for libswiftlang).

Swift 5.6.3 list the following valid hosts for Linux:
    Linux = Platform("linux", archs=[
        "x86_64",
        "i686",
        "armv6",
        "armv7",
        "aarch64",
        "powerpc64",
        "powerpc64le",
        "s390x"])

This would seem to mean that i686, armv6, and armv7 are valid 32-bit hosts. However, the only hosts from this list that Debian supports are x86_64, i686, aarch64, powerpc64, and s390x (powerpc and riscv64 as well). So I think I can just restrict my package to building on these 5 architectures.
x86_64 builds
i686 fails because it needs libatomic
aarch64 fails because its build is using the unsupported -fstack-clash protection
powerpc64 is also missing libatomic as well as whatever provides __gnu_f2h_ieee, __gnu_h2f_ieee, and __truncdfhf2
s390x is failing because llvm-project/lld/ELF/Target.cpp doesn't see it as a valid architecture
I'm working to fix these, but any help would be appreciated.
Also, I'm curious how to add support for powerpc and riscv64 since they were added in 5.7.x or newer and thus there is no swift compiler on those platforms to use to compile.

Swift 5.6.3 list the following valid hosts for Linux:
    Linux = Platform("linux", archs=[
        "x86_64",
        "i686",
        "armv6",
        "armv7",
        "aarch64",
        "powerpc64",
        "powerpc64le",
        "s390x"])

This would seem to mean that i686, armv6, and armv7 are valid 32-bit hosts.

IIRC 32-bit host support was dropped long before (or hadn’t ever existed). Latest PRs that add new architectures show they’re adding support for targets that are meant to be cross-compiled against. 32-bit ARMv8 host triples should be handled similarly, while 64-bit ones can be mapped into aarch64.

s390x is failing because llvm-project/lld/ELF/Target.cpp doesn't see it as a valid architecture

Maybe the port is not maintained any more and should be removed?

Also, I'm curious how to add support for powerpc and riscv64 since they were added in 5.7.x or newer and thus there is no swift compiler on those platforms to use to compile.

powerpc is target-only IIRC. riscv64 and other newly added architectures should be cross-compiled first. Not sure if the Debian packaging team will allow though. BTW this port doesn’t seem to be complete, so you can skip for now.

Since these are becoming mandatory dating back to Swift 5.7, things are actually going to be tricky if the Debian team don’t allow some “temporary/modified version” to be used for building the release package.

@melizasw In fact, you can tweak the process to first build a toolchain without Swift in Swift support, and then build another one with it. This still work as for Swift 5.9, but will be removed at some time.

Ok, I think the problem is that I was approaching the Swift build process with the notion that it supported a wider range of architectures than it does. I see now that tachoknight's packaged for Fedora only support x86_64 and aarch64. For an optional development tool it seems reasonable to have a smaller set of supported architectures to start with.
Debian does indeed support the addition of new architectures and I was able to find some documentation on bootstrapping and cross-building which may be needed for riscv64 support. But for now I think I'm going to just focus on x86_64 and aarch64 and maybe s390x (Debian's LLVM toolchain builds for s390x).
Thanks for the help.

Ok, I think the problem is that I was approaching the Swift build process with the notion that it supported a wider range of architectures than it does. I see now that tachoknight's packaged for Fedora only supports x86_64 and aarch64. For an optional development tool it seems reasonable to have a smaller set of supported architectures to start with.

Fedora supports multiple architectures as well and for a very brief time I had everything working on the x390 platform, but then a newer version of clang would crash with a stacktrace when building Swift. I filed a bug with the LLVM team that has gotten no traction at all, so I abandoned it.
FYI I would like to know your experience building 5.9 on Debian/aarch64 because currently Fedora doesn't build successfully on that platform (builds and works fine on x86_64).

Fedora supports multiple architectures as well and for a very brief time I had everything working on the x390 platform, but then a newer version of clang would crash with a stacktrace when building Swift. I filed a bug with the LLVM team that has gotten no traction at all, so I abandoned it.

Yep, I ran into the same problem and added to your bug report from 3+ years ago. There was some activity on the bug 2 months ago, but no fix yet. I guess no s390x for now.

FYI I would like to know your experience building 5.9 on Debian/aarch64 because currently Fedora doesn't build successfully on that platform (builds and works fine on x86_64).

It'll take a while before I get there, but I will let you know.

On a RPI 4, it takes around 24 hours. 🙃

That beats the 2.5 days it took to build through QEMU! I ordered an RPI 5 that will be here in a few days to help out with that problem. I've also got a LicheePi 4A arriving later today to help me with building for riscv64 which I have been making some progress on by porting patches from @futurejones to 5.6.3.

Great news - Swift is now on Debian unstable (sid) and in a few days should go out to testing (trixie) on amd64 and arm64.
https://tracker.debian.org/pkg/swiftlang
Unfortunately when Ubuntu pulled it in and attempted to build on Noble it failed.
https://launchpad.net/ubuntu/+source/swiftlang
The issue appears to me to be related to the gold linker
[4402/4423][ 99%][4529.955s] Building CXX object utils/unittest/CMakeFiles/gtest.dir/googletest/src/gtest-all.cc.o
[4403/4423][ 99%][4529.991s] Linking CXX static library lib/libgtest.a
[4404/4423][ 99%][4530.020s] Linking CXX static library lib/libLLVMTestingSupport.a
[4405/4423][ 99%][4530.061s] Creating directories for 'compiler-rt'
[4406/4423][ 99%][4530.071s] No download step for 'compiler-rt'
[4407/4423][ 99%][4530.082s] No update step for 'compiler-rt'
[4408/4423][ 99%][4530.084s] Building CXX object utils/benchmark/src/CMakeFiles/benchmark.dir/benchmark_register.cc.o
[4409/4423][ 99%][4530.094s] No patch step for 'compiler-rt'
[4409/4423][ 99%][4530.094s] Performing configure step for 'compiler-rt'
-- The C compiler identification is Clang 13.0.0
-- The CXX compiler identification is Clang 13.0.0
-- The ASM compiler identification is Clang with GNU-like command-line
-- Found assembler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang
-- Check for working C compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang - broken
CMake Error at /usr/share/cmake-3.27/Modules/CMakeTestCCompiler.cmake:67 (message):
  The C compiler

    "/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: '/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/CMakeScratch/TryCompile-kQ8HgL'

    Run Build Command(s): /usr/bin/ninja -v cmTC_d2d03
    [1/2][ 50%][0.017s] /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang   -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/<<PKGBUILDDIR>>=/usr/src/swiftlang-5.6.3-2 -MD -MT CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o -c /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/CMakeScratch/TryCompile-kQ8HgL/testCCompiler.c
    clang-13: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument]
    [2/2][100%][0.030s] : && /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/<<PKGBUILDDIR>>=/usr/src/swiftlang-5.6.3-2 -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o -o cmTC_d2d03   && :
    FAILED: cmTC_d2d03
    : && /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/<<PKGBUILDDIR>>=/usr/src/swiftlang-5.6.3-2 -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -Wl,-z,relro CMakeFiles/cmTC_d2d03.dir/testCCompiler.c.o -o cmTC_d2d03   && :
    /usr/bin/ld: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/bin/../lib/LLVMgold.so: error loading plugin: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/bin/../lib/LLVMgold.so: cannot open shared object file: No such file or directory
    clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
    ninja: build stopped: subcommand failed.

  CMake will not be able to correctly generate this project.

-- Warning: {} Host toolchain could not locate a compiler to build swift-driver. (Try `--skip-early-swift-driver`)
--- Building cmark ---
Building the standard library for: swift-stdlib-linux-x86_64
Running Swift tests for: check-swift-only_early_swiftdriver-linux-x86_64 check-swift-all-linux-x86_64 check-swift-all-optimize-linux-x86_64
ERROR: command terminated with a non-zero exit status 1, aborting

The same build on Debian produces the following build log:
[4405/4423][ 99%][2053.044s] Building CXX object utils/unittest/CMakeFiles/gtest.dir/googletest/src/gtest-all.cc.o
[4406/4423][ 99%][2053.094s] Linking CXX static library lib/libgtest.a
[4407/4423][ 99%][2053.137s] Linking CXX static library lib/libgtest_main.a
[4408/4423][ 99%][2053.138s] Linking CXX static library lib/libLLVMTestingSupport.a
[4409/4423][ 99%][2053.157s] Creating directories for 'compiler-rt'
[4410/4423][ 99%][2053.175s] No download step for 'compiler-rt'
[4411/4423][ 99%][2053.193s] No update step for 'compiler-rt'
[4412/4423][ 99%][2053.212s] No patch step for 'compiler-rt'
[4412/4423][ 99%][2053.212s] Performing configure step for 'compiler-rt'
-- The C compiler identification is Clang 13.0.0
-- The CXX compiler identification is Clang 13.0.0
-- The ASM compiler identification is Clang with GNU-like command-line
-- Found assembler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-x86_64/./bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for unwind.h
-- Looking for unwind.h - found
-- Looking for rpc/xdr.h
-- Looking for rpc/xdr.h - not found
CMake Warning at cmake/Modules/CompilerRTUtils.cmake:352 (message):
  llvm-config finding testingsupport failed with status 1
Call Stack (most recent call first):
  CMakeLists.txt:81 (load_llvm_config)

I'm not sure why Debian is skipping the check but Ubuntu is not and then failing it. My Debian build doesn't have LLVMgold.so so no surprise that Ubuntu doesn't have it either.
I'm going to try the following to see if they help:

Add binutils and binutils-dev to the build dependencies
patch llvm-project-orig/llvm/cmake/config-ix.cmake as @tachoknight did for his Fedora 5.6.3 build to set LLVM_BINUTILS_INCDIR to /usr/include  (Debian LLVM 16 also sets this path for its build).

Any help or advice would be appreciated as at the moment I am not set up to run test builds on Ubuntu and cannot directly test these without first releasing them to Debian then waiting for Ubuntu to pull it in and build.

More good news. Swift has now migrated to Debian testing and the two fixes mentioned above did indeed get the Ubuntu Noble build past that hang-up. I drug an old system out of retirement and installed Ubuntu on it so that I could work through the build errors on Noble.
Unfortunately now I'm getting another error, something about trying to link a static library and a dynamic library and needed to compile with -fPIC.
[770/863][ 89%][27.919s] Linking CXX shared library /home/swm/workspace/debian/build/swiftlang/swiftlang-5.6.3/build/buildbot_linux/llvm-linux-x86_64/lib/clang/13.0.0/lib/linux/libclang_rt.scudo_standalone-x86_64.so
FAILED: /home/swm/workspace/debian/build/swiftlang/swiftlang-5.6.3/build/buildbot_linux/llvm-linux-x86_64/lib/clang/13.0.0/lib/linux/libclang_rt.scudo_standalone-x86_64.so

/usr/bin/ld: /tmp/lto-llvm-bde8d8.o: warning: relocation against `__gwp_asan_default_options' in read-only section `.text._ZN8gwp_asan7options11initOptionsEPKcPFvS2_zE'
/usr/bin/ld: /tmp/lto-llvm-bde8d8.o: relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)

That doesn't seem too hard, I just need to find the right place to do so when I get more time.

@melizasw I have installed and tried to test your debian packages but they don't work.
I am just trying to run a simple "hello world" using the swift package manager.
The errors are very easy to replicate.
The test environment is a Debian Trixie docker container.
docker run --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -it debian:trixie /bin/bash

apt update

apt install swiftlang

mkdir hello && cd hello && swift package init --type executable && swift run

root@d5e164273bb5:/# mkdir hello && cd hello && swift package init --type executable && swift run
Creating executable package: hello
Creating Package.swift
Creating README.md
Creating .gitignore
Creating Sources/
Creating Sources/hello/main.swift
Creating Tests/
Creating Tests/helloTests/
Creating Tests/helloTests/helloTests.swift
'hello': error: invalidManifestFormat("<module-includes>:1:10: note: in file included from <module-includes>:1:\n#include \"CoreFoundation.h\"\n
 ^\n/usr/libexec/swift/lib/swift/CoreFoundation/CoreFoundation.h:25:10: error: \'sys/types.h\' file not found\n#include <sys/types.h>\n
^\n/usr/libexec/swift/lib/swift/CoreFoundation/CFStream.h:20:10: note: while building module \'CDispatch\' imported from /usr/libexec/swift/lib/swift/CoreFoundation/CFStream.h:20:\n#include <dispatch/dispatch.h>\n
^\n<module-includes>:1:10: note: in file included from <module-includes>:1:\n#include \"dispatch.h\"\n         ^\n/usr/libexec/swift/lib/swift/dispatch/dispatch.h:32:10: note: in file included from
/usr/libexec/swift/lib/swift/dispatch/dispatch.h:32:\n#include <os/generic_unix_base.h>\n
^\n/usr/libexec/swift/lib/swift/os/generic_unix_base.h:24:10:
error: \'sys/param.h\' file not found\n#include <sys/param.h>\n
^\n<unknown>:0: error: could not build C module \'CoreFoundation\'", diagnosticFile: nil)
root@d5e164273bb5:/hello#

This error also occurs when using Debian Unstable.
When trying to build using swiftc we get an another error.
root@d5e164273bb5:/hello# swiftc Sources/hello/main.swift
error: link command failed with exit code 1 (use -v to see invocation)
clang-13: error: invalid linker name in argument '-fuse-ld=gold'

This indicates that binutils is missing from the required dependancies.
But after installing binutils we get another error.
root@d5e164273bb5:/hello# swiftc Sources/hello/main.swift
error: link command failed with exit code 1 (use -v to see invocation)
/usr/bin/ld.gold: error: cannot open Scrt1.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crti.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crtbeginS.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crtendS.o: No such file or directory
/usr/bin/ld.gold: error: cannot open crtn.o: No such file or directory
/usr/bin/ld.gold: error: cannot find -lgcc
/usr/bin/ld.gold: error: cannot find -lgcc_s
/usr/bin/ld.gold: error: cannot find -lc
/usr/bin/ld.gold: error: cannot find -lgcc
/usr/bin/ld.gold: error: cannot find -lgcc_s
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)

I have filed a bug with Debian to track the issue.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1057117

@futurejones this is due to missing dependencies on binutils and gcc. If you install these two packages it will get you up and running for the next few days until I can get a new release out.

@melizasw there appears to be a problem with the relationship between the different swiftlang packages.
The swiftlang package is listing the swiftlang-dev package as a dependency, whereas the swiftlang-dev package only lists swiftlang as recommends`.
The swiftlang-dev package should list both swiftlang and libswiftlang as dependencies.
https://packages.debian.org/trixie/swiftlang-dev
The swiftlang package should not list swiftlang-dev as a dependency.
https://packages.debian.org/trixie/swiftlang

@futurejones I can certainly do that. Can you help me understand though what use swiftlang is without swiftlang-dev? I was under the impression that swiftlang wasn't of much use without both libswiflang and swiftland-dev.

So first, I'm really happy to see this work making progress. I think this will really help with the Swift-on-Linux story.
For some ideas on the separation of the libraries and tools, it might be worth looking at the Windows installer. I don't remember the exact split but it does a good job of separating concerns for developers, CI, and end-consumers of Swift programs. You shouldn't need to download the compiler if you just want to run a pre-compiled program that happens to be written in Swift, and you shouldn't need to download the headers for CoreFoundation if you just want to build a hello-world, for instance. It looks like that separation is mostly covered with the separation of swiftlang and libswiftlang, though perhaps could be split further for the stdlib and corelibs.
I may have missed a few runtime libraries in the listing here, so worth double-checking, but I think it's reasonable breakdown:
Runtimes:

swift runtimes (both with and without swiftmodules/header files (dev and non-dev versions))

swiftCore
swiftGlibc
swiftRemoteMirror
swiftSwiftOnoneSupport
swift_Concurrency
swift_Differentiation
swift_MatchingEngine
swift_StringProcessing
libdispatch*

corelibs

blocksRuntime
Foundation
FoundationNetworking
FoundationXML

* The challenge with libdispatch is that concurrency depends on it. The C bits of corelibs-libdispatch actually get built twice in the toolchain build, so I wonder if it would make sense to configure the build to emit a static archive so that we don't expose that. Then libdispatch.so could be installed as part of the corelibs package.
We should keep the runtime libraries together installed as one. I don't have a strong opinion on the corelibs libraries being together or apart, though mildly in favor of keeping them together as a swiftlang-corelibs package. Then we would want a dev variant of the swiftlang runtimes and swiftlang corelibs packages containing swiftmodules and headers. I think that keeping the corelibs separate from the runtimes is important, especially as work on swift-Foundation progresses, I'm not entirely sure what the future of corelibs-Foundation looks like.
Looking at swiftlang-dev, I'm concerned about the libcxx headers getting installed? If folks are doing C++ interop, which I don't think Swift 5.6 supports anyway, those headers will need to match the C++ runtime they're building for, which natively would be libstdc++ on the system.
Build Tools: (what I assume is the umbrella swiftlang package)

swiftc
swiftpm
clang

The swift-clang will be an interesting tricky bit. We'll want to provide the special swift-clang that has support for swift and swift-async calling conventions so that folks can use the C and C++ interop features. That of course conflicts with the existing clang package. I think the provides mechanism might work here though? A swift-clang package provides clang perhaps?
My concern with naming clang swift-clang is that various build-scripts may try to use clang while consuming a header emitted by Swift. Those headers may contain functions with swift or swiftasync calling conventions, so if they use the wrong C/C++ compiler, they will get a warning about the unknown attribute, but the program will compile with a misaligned ABI. On a good day, that will crash their program.  I'm open to push-back on that.
swiftlang can probably drop the headers and swiftmodules, and instead install those in the runtime-dev package. It's true that the compiler isn't terribly useful without them, but it's more conventional to install the headers associated with the library in the <library>-dev package. See the gcc vs libc6 vs libc6-dev filelists, for example. gcc doesn't come with headers, but libc6-dev comes with the C headers.
Dev Tools:

lldb

Same sort of situation with the swift-lldb as with clang. The lldb from apple/llvm-project has extra bits to make debugging Swift work, but that will conflict with the existing lldb package. Probably the same resolution since it provides lldb + swift extras.
Anyway, my initial thoughts. Thank you for working on this.

Despite what was said about about building Swift 5.7.3 using Swift 5.6.3, I tried without Swift and was able to build 5.7.3 just fine and am in the process of getting that pushed out into Debian. In the mean time I decided to build Swift 5.8.1 without using Swift 5.7.3, which I think should be possible as @tachoknight did so for Fedora. However, I got this bootstrapping error quite similar to what @stevapple nearly 2 years ago - https://github.com/swiftlang/swift/issues/61410
Here is my version of the error, or at least the start of about 5,000 lines of them.

[1215/1425][ 85%][1092.863s] Building swift module Basic
FAILED: bootstrapping1/SwiftCompilerSources/Basic.o /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/bootstrapping1/SwiftCompilerSources/Basic.o
cd /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/SwiftCompilerSources && /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/bootstrapping0/bin/swiftc -c -o /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/bootstrapping1/SwiftCompilerSources/Basic.o -I /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/bootstrapping0/bin/../lib -I /usr/lib -target x86_64-pc-linux-gnu -module-name Basic -emit-module -emit-module-path /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/bootstrapping1/SwiftCompilerSources/Basic.swiftmodule -parse-as-library /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/SwiftCompilerSources/Sources/Basic/SourceLoc.swift /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/SwiftCompilerSources/Sources/Basic/Utils.swift -wmo -Xfrontend -validate-tbd-against-ir=none -Xfrontend -enable-experimental-cxx-interop -Xcc -UIBOutlet -Xcc -UIBAction -Xcc -UIBInspectable -Xfrontend -disable-implicit-string-processing-module-import -O -cross-module-optimization -Xcc -I -Xcc /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include -Xcc -I -Xcc /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/llvm-linux-x86_64/include -Xcc -I -Xcc /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/clang/include -Xcc -I -Xcc /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/llvm-linux-x86_64/tools/clang/include -Xcc -I -Xcc /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/include -Xcc -I -Xcc /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/SwiftCompilerSources/../include -I /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-x86_64/bootstrapping1/SwiftCompilerSources
:0: warning: unable to perform implicit import of "_Concurrency" module: no such module found
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/include/swift/AST/ClangModuleLoader.h:19:10: note: while building module 'Clang_AST' imported from /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/include/swift/AST/ClangModuleLoader.h:19:
#include "clang/AST/DeclTemplate.h"
^
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/clang/include/clang/AST/APValue.h:16:10: note: while building module 'Clang_Basic' imported from /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/clang/include/clang/AST/APValue.h:16:
#include "clang/Basic/LLVM.h"
^
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/clang/include/clang/Basic/Sanitizers.h:20:10: note: while building module 'LLVM_Transforms' imported from /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/clang/include/clang/Basic/Sanitizers.h:20:
#include "llvm/Transforms/Instrumentation/AddressSanitizerOptions.h"
^
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include/llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h:20:10: note: while building module 'LLVM_intrinsic_gen' imported from /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include/llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h:20:
#include "llvm/IR/PassManager.h"
^
:1:10: note: in file included from :1:
#include "IR/Argument.h"
^
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include/llvm/IR/Argument.h:17:10: note: in file included from /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include/llvm/IR/Argument.h:17:
#include "llvm/IR/Attributes.h"
^
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include/llvm/IR/Attributes.h:30:10: note: in file included from /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/llvm-project/llvm/include/llvm/IR/Attributes.h:30:
#include
^
/usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/set:62:10: note: in file included from /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/set:62:
#include <bits/stl_tree.h>
^
/usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_tree.h:2005:5: error: missing '#include "gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_pair.h"'; 'pair' must be declared before it is used
pair<typename _Rb_tree<_Key, _Val, _KeyOfValue,
^
/usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/stl_pair.h:284:12: note: declaration here is not visible
struct pair
^

I can't figure out what is causing this error. Is there a way around it or do I need to give up and build 5.8.1 with 5.7.3?

Problem resolved, though I certainly took the scenic route to get there. The issue is covered in good detail in https://github.com/swiftlang/swift/issues/65028 so no need to repeat it. Swift 5.8.x apparently introduced some cmake file changes that are not compatible with cmake 3.25 and newer. Once I uninstalled Debian's cmake the build reverted to the specified version of cmake (3.19.6) and this build error went away. Looks like the patch will be in 5.9.x.

I was doing well building 5.8.1 when Debian switched to Clang 19 which introduced -Wcast-function-type-mismatch and then enabled it by default. This has caused an error building swift-corelibs-libdispatch which I have reported on github in case they wish to fix it. For now I'd just like to add -Wno-cast-function-type-mismatch to the flags for swift-corelibs-libdispatch to turn off the warning/error. Can someone point me in the right direction to inject this flag into the swift build scripts?

@melizasw  Many thanks for your work on Debian packaging.
I noticed you have uploaded 5.8.1 to Debian also, that is great. I am a Debian riscv porter, so I am looking into how to build it on riscv64 now on Debian.
Now the issue is below:
...
-- The C compiler identification is Clang 13.0.0
-- The CXX compiler identification is Clang 13.0.0
-- The ASM compiler identification is Clang with GNU-like command-line
-- Found assembler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-riscv64/bin/clang
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-riscv64/bin/clang
-- Check for working C compiler: /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-riscv64/bin/clang - broken
CMake Error at /usr/share/cmake-3.30/Modules/CMakeTestCCompiler.cmake:67 (message):
  The C compiler

    "/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-riscv64/bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: '/<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-riscv64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/CMakeScratch/Tr
yCompile-BalgBc'

    Run Build Command(s): /usr/bin/ninja -v cmTC_587eb
   [1/2][ 50%][0.460s] /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-riscv64/bin/clang   -g -O2 -Werror=implicit-function-declaration
 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -MD -MT CMakeFiles/cmTC_587eb.dir/testCCo
mpiler.c.o -MF CMakeFiles/cmTC_587eb.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_587eb.dir/testCCompiler.c.o -c /<<PKGBUILDDIR>>/build/bui
ldbot_linux/llvm-linux-riscv64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/CMakeScratch/TryCompile-BalgBc/testCCompiler.c
    [2/2][100%][0.888s] : && /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-riscv64/bin/clang -g -O2 -Werror=implicit-function-declarat
ion -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro CMakeFiles/cmTC_587eb.dir
/testCCompiler.c.o -o cmTC_587eb   && :
    FAILED: cmTC_587eb
    : && /<<PKGBUILDDIR>>/build/buildbot_linux/llvm-linux-riscv64/bin/clang -g -O2 -Werror=implicit-function-declaration -ffile-prefix-ma
p=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro CMakeFiles/cmTC_587eb.dir/testCCompiler.c.o -
o cmTC_587eb   && :
    /usr/bin/ld: cannot find Scrt1.o: No such file or directory
    clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
    ninja: build stopped: subcommand failed.
...

although the log is suggestive enough, but I am not convinced a little. Could you have an idea about this? Certainly, I will continue to try it on my side. TIA.

@yuzibo Yes, I got 5.8.1 into Debian by using the clang-18 package as my workaround for now. However, the arm64 build passes with Clang 16 and fails with Clang 18 so I have my work cut out for me there.
Yes, I am familiar with the ld cannot find Scrt1.o error on RISCV. I've been slowly chipping away at it, but the patches in Debian are close, but not complete yet. My build is still running so I don't know yet if these are enough, but try adding them to your patches:
--- swiftlang-5.8.1.orig/swift/utils/build-presets.ini
+++ swiftlang-5.8.1/swift/utils/build-presets.ini
@@ -800,7 +800,9 @@ no-swift-stdlib-assertions
 [preset: mixin_linux_install_components_with_clang]

 swift-install-components=autolink-driver;compiler;clang-resource-dir-symlink;stdlib;swift-remote-mirror;sdk-overlay;static-mirror-lib;toolchain-tools;license;sourcekit-inproc
-llvm-install-components=llvm-ar;llvm-cov;llvm-profdata;IndexStore;clang;clang-resource-headers;compiler-rt;clangd;lld;LTO;clang-features-file
+llvm-install-components=llvm-ar;llvm-cov;llvm-profdata;IndexStore;clang;clang-resource-headers;clangd;lld;LTO;clang-features-file
+
+skip-build-compiler-rt

 [preset: mixin_linux_installation]
 mixin-preset=

@@ -1475,7 +1475,7 @@ mixin-preset=
 [preset: LLDB_Nested]
 skip-build-benchmarks
 install-destdir=%(swift_install_destdir)s
-llvm-targets-to-build=X86;ARM;AArch64;PowerPC;SystemZ;Mips
+llvm-targets-to-build=X86;ARM;AArch64;PowerPC;SystemZ;Mips;RISCV

 [preset: LLDB_Swift_DebugAssert]
 mixin-preset=

If these work then I'll need to figure out a clean way to only enable them for RISCV and not the other architectures.
I've gathered my RISCV patches from various sources, but these two changes are thanks to @futurejones and his patches for 5.8 - https://github.com/swift-riscv/swift-riscv64/tree/bcb3fad816f5dd89fcc38804a5d5a1b0a58a1bc3/patches/release-5.8-branch  I've not incorporated all of his patches as I try to only enable them as needed to fix a specific error so that I can understand the need for each one and make sure I don't grab any that are not needed for Debian.
I've already got 5.9.2 building on x86_64 with a build depends on 5.8.1 so RISCV really needs to get onboard at 5.8.1 or it will get left behind and require some cross building or bootstrapping later on.

@melizasw  Many thanks for your sharing about this, I will try it on my side.
BTW, do you need one real riscv64 hardware to debug something on here? if so, I can offer you with access remote with ssh.

Yes, I am getting the same error now. It can't seem to locate /usr/include/riscv64-linux-gnu/bits/wordsize.h  even though it has no problem doing so on x86_64 and arm64. I will try to dig into this tonight after work. This is the same error that I got on 5.7.3 that stopped me and I did not get past. My best guess is an issue configuring include paths in the CMake files.

I was doing well building 5.8.1 when Debian switched to Clang 19 which introduced -Wcast-function-type-mismatch and then enabled it by default. This has caused an error building swift-corelibs-libdispatch which I have reported on github in case they wish to fix it. For now I'd just like to add -Wno-cast-function-type-mismatch to the flags for swift-corelibs-libdispatch to turn off the warning/error. Can someone point me in the right direction to inject this flag into the swift build scripts?

I added this to my build script and it works now:
export CFLAGS="-Wno-error=cast-function-type echo $CFLAGS"

@melizasw It turned out to be a little bit more difficult than what I initially wrote. Here's the patch file I addd to make it work reliably. Hopefully that helps. :)

@melizasw It turned out to be a little bit more difficult than what I initially wrote. Here's the patch file I addd to make it work reliably. Hopefully that helps. :)

Yah, I ended up using this patch:
--- swiftlang-5.8.1.orig/swift-corelibs-libdispatch/cmake/modules/DispatchCompilerWarnings.cmake
+++ swiftlang-5.8.1/swift-corelibs-libdispatch/cmake/modules/DispatchCompilerWarnings.cmake
@@ -65,6 +65,7 @@ else()
   add_compile_options($<$<OR:$<COMPILE_LANGUAGE:C>,$<COMPILE_LANGUAGE:CXX>>:-Wno-used-but-marked-unused>)
   add_compile_options($<$<OR:$<COMPILE_LANGUAGE:C>,$<COMPILE_LANGUAGE:CXX>>:-Wno-void-pointer-to-int-cast>)
   add_compile_options($<$<OR:$<COMPILE_LANGUAGE:C>,$<COMPILE_LANGUAGE:CXX>>:-Wno-vla>)
+  add_compile_options($<$<OR:$<COMPILE_LANGUAGE:C>,$<COMPILE_LANGUAGE:CXX>>:-Wno-error=cast-function-type-mismatch>)

   if(CMAKE_SYSTEM_NAME STREQUAL Android)
     add_compile_options($<$<OR:$<COMPILE_LANGUAGE:C>,$<COMPILE_LANGUAGE:CXX>>:-Wno-incompatible-function-pointer-types>)

The issue now is that 5.8.1 builds fine on amd64, but fail with a stack dump on arm64 with Clang 18 and 19. I'm a little stumped, but haven't spent a lot of time on it yet.
1.	Swift version 5.8.1 (swift-5.8.1-RELEASE)
2.	Compiling with the current language version
3.	While evaluating request IRGenRequest(IR Generation for module StdlibUnittest)
4.	While emitting prespecialized metadata for type 'Optional<Int>'
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x6570028) [0x5555f8b10028]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x656e06c) [0x5555f8b0e06c]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x6570450) [0x5555f8b10450]
linux-vdso.so.1(__kernel_rt_sigreturn+0x0) [0x7fff002107b0]
/lib/aarch64-linux-gnu/libc.so.6(+0x87240) [0x7ffeff627240]
/lib/aarch64-linux-gnu/libc.so.6(gsignal+0x20) [0x7ffeff5d67a0]
/lib/aarch64-linux-gnu/libc.so.6(abort+0xf8) [0x7ffeff5c1a48]
/lib/aarch64-linux-gnu/libc.so.6(+0x2f860) [0x7ffeff5cf860]
/lib/aarch64-linux-gnu/libc.so.6(+0x2f8d4) [0x7ffeff5cf8d4]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x1190f38) [0x5555f3730f38]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x1190dc8) [0x5555f3730dc8]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x1190d14) [0x5555f3730d14]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x11888dc) [0x5555f37288dc]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x126a590) [0x5555f380a590]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x1269d68) [0x5555f3809d68]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x12699ac) [0x5555f38099ac]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x11f9628) [0x5555f3799628]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x1178f64) [0x5555f3718f64]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x1063738) [0x5555f3603738]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x10af384) [0x5555f364f384]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x107309c) [0x5555f361309c]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0x106463c) [0x5555f360463c]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0xd6ac54) [0x5555f330ac54]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0xd67028) [0x5555f3307028]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0xd66190) [0x5555f3306190]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0xd76f70) [0x5555f3316f70]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0xd687e0) [0x5555f33087e0]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0xd679f0) [0x5555f33079f0]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0xbdad64) [0x5555f317ad64]
/lib/aarch64-linux-gnu/libc.so.6(+0x2229c) [0x7ffeff5c229c]
/lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x9c) [0x7ffeff5c237c]
/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend(+0xbd9df0) [0x5555f3179df0]
<unknown>:0: error: unable to execute command: Aborted
<unknown>:0: error: compile command failed due to signal 6 (use -v to see invocation)
[1419/1463][ 96%][5365.217s] Compiling /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/stdlib/public/StringProcessing//LINUX/aarch64/_StringProcessing.o
FAILED: stdlib/public/StringProcessing/LINUX/aarch64/_StringProcessing.o /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/stdlib/public/StringProcessing/LINUX/aarch64/_StringProcessing.o
cd /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/stdlib/public/StringProcessing && /usr/bin/cmake -E env LD_LIBRARY_PATH=/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/bootstrapping1/lib/swift/linux /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/binforpython/python /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/swift/utils/line-directive @/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/stdlib/public/StringProcessing/b7c966f3242f458771abca1842040366b98e82db.txt -- /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/./bin/swiftc -c -sdk / -target aarch64-unknown-linux-gnu -resource-dir /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/./lib/swift -O -D SWIFT_ENABLE_EXPERIMENTAL_CONCURRENCY -D SWIFT_ENABLE_EXPERIMENTAL_DISTRIBUTED -D SWIFT_ENABLE_EXPERIMENTAL_DIFFERENTIABLE_PROGRAMMING -D SWIFT_ENABLE_EXPERIMENTAL_STRING_PROCESSING -D SWIFT_RUNTIME_OS_VERSIONING -D SWIFT_STDLIB_ENABLE_UNICODE_DATA -D SWIFT_STDLIB_ENABLE_VECTOR_TYPES -D SWIFT_STDLIB_HAS_COMMANDLINE -D SWIFT_STDLIB_HAS_STDIN -D SWIFT_STDLIB_HAS_ENVIRON -Xcc -DSWIFT_STDLIB_HAS_ENVIRON -D SWIFT_THREADING_LINUX -tools-directory /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/llvm-linux-aarch64/./bin -module-cache-path /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/./module-cache -no-link-objc-runtime -enable-library-evolution -library-level api -Xfrontend -require-explicit-availability=ignore -Xfrontend -enforce-exclusivity=unchecked -D SWIFT_ENABLE_REFLECTION -module-name _StringProcessing -swift-version 5 -runtime-compatibility-version none -disable-autolinking-runtime-compatibility-dynamic-replacements -Xfrontend -disable-autolinking-runtime-compatibility-concurrency -Xfrontend -disable-objc-interop -Xfrontend -public-autolink-library -Xfrontend swift_RegexParser -warn-implicit-overrides -Xfrontend -enable-ossa-modules -Xfrontend -enable-lexical-lifetimes=false -Xfrontend -disable-implicit-concurrency-module-import -Xfrontend -disable-implicit-string-processing-module-import -Xfrontend -disable-implicit-string-processing-module-import -Xfrontend -prespecialize-generic-metadata -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 9999:macOS\ 9999,\ iOS\ 9999,\ watchOS\ 9999,\ tvOS\ 9999 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.0:macOS\ 10.14.4,\ iOS\ 12.2,\ watchOS\ 5.2,\ tvOS\ 12.2 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.1:macOS\ 10.15,\ iOS\ 13.0,\ watchOS\ 6.0,\ tvOS\ 13.0 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.2:macOS\ 10.15.4,\ iOS\ 13.4,\ watchOS\ 6.2,\ tvOS\ 13.4 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.3:macOS\ 11.0,\ iOS\ 14.0,\ watchOS\ 7.0,\ tvOS\ 14.0 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.4:macOS\ 11.3,\ iOS\ 14.5,\ watchOS\ 7.4,\ tvOS\ 14.5 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.5:macOS\ 12.0,\ iOS\ 15.0,\ watchOS\ 8.0,\ tvOS\ 15.0 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.6:macOS\ 12.3,\ iOS\ 15.4,\ watchOS\ 8.5,\ tvOS\ 15.4 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.7:macOS\ 13.0,\ iOS\ 16.0,\ watchOS\ 9.0,\ tvOS\ 16.0 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.8:macOS\ 13.3,\ iOS\ 16.4,\ watchOS\ 9.4,\ tvOS\ 16.4 -Xfrontend -define-availability -Xfrontend SwiftStdlib\ 5.9:macOS\ 9999,\ iOS\ 9999,\ watchOS\ 9999,\ tvOS\ 9999 -Xfrontend -target-min-inlining-version -Xfrontend min -module-link-name swift_StringProcessing -whole-module-optimization -parse-as-library -resource-dir /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/./lib/swift -I /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/./lib/swift/linux -o /home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/stdlib/public/StringProcessing//LINUX/aarch64/_StringProcessing.o @/home/swm/workspace/debian/build/swiftlang/swiftlang-5.8.1/build/buildbot_linux/swift-linux-aarch64/stdlib/public/StringProcessing/b7c966f3242f458771abca1842040366b98e82db.txt

Confirmed that 5.8.1 build failures on Debian Sid with default Clang package (Clang 19) on Arm64 are fixed with the patch provided with bug report #70345
Assertion `!FoundVal && "Key already in new map?"' failed.

I have 5.9.2 building fine on x86_64, but on aarch64:
clang: error: no such file or directory: '/usr/libexec/swift/lib/swift/linux/x86_64/swiftrt.o'
Sigh

On to the next issue with packaging 5.9.2. It builds on x86_64 but not aarch64. I've traced it down to an error parsing json by swiftpm/Utilities/bootstrapping because switfc outputs some non-json on aarch64 that trips up the json parser.
x86_64:
usr/bin/swiftc -print-target-info
{
  "compilerVersion": "Swift version 5.9.2 (swift-5.9.2-RELEASE)",
  "target": {
    "triple": "x86_64-pc-linux-gnu",
    "unversionedTriple": "x86_64-pc-linux-gnu",
    "moduleTriple": "x86_64-pc-linux-gnu",
    "compatibilityLibraries": [ ],
    "librariesRequireRPath": false
  },
  "paths": {
    "runtimeLibraryPaths": [
      "/home/swm/workspace/debian/build/swiftlang/swiftlang-5.9.2/usr/lib/swift/linux"
    ],
    "runtimeLibraryImportPaths": [
      "/home/swm/workspace/debian/build/swiftlang/swiftlang-5.9.2/usr/lib/swift/linux",
      "/home/swm/workspace/debian/build/swiftlang/swiftlang-5.9.2/usr/lib/swift/linux/x86_64"
    ],
    "runtimeResourcePath": "/home/swm/workspace/debian/build/swiftlang/swiftlang-5.9.2/usr/lib/swift"

aarch64:
usr/bin/swiftc -print-target-info
swift runtime: unable to protect path to swift-backtrace at 0x7fff4c7e5000: 22; disabling backtracing.
swift runtime: unable to protect environment for swift-backtrace at 0x7fff4c7dd000: 22; disabling backtracing.
swift runtime: unable to protect path to swift-backtrace at 0x7fff56635000: 22; disabling backtracing.
swift runtime: unable to protect environment for swift-backtrace at 0x7fff5662d000: 22; disabling backtracing.
{
  "compilerVersion": "Swift version 5.9.2 (swift-5.9.2-RELEASE)",
  "target": {
    "triple": "aarch64-unknown-linux-gnu",
    "unversionedTriple": "aarch64-unknown-linux-gnu",
    "moduleTriple": "aarch64-unknown-linux-gnu",
    "compatibilityLibraries": [ ],
    "librariesRequireRPath": false
  },
  "paths": {
    "runtimeLibraryPaths": [
      "/home/swm/workspace/debian/build/swiftlang/swiftlang-5.9.2/usr/lib/swift/linux"
    ],
    "runtimeLibraryImportPaths": [
      "/home/swm/workspace/debian/build/swiftlang/swiftlang-5.9.2/usr/lib/swift/linux",
      "/home/swm/workspace/debian/build/swiftlang/swiftlang-5.9.2/usr/lib/swift/linux/aarch64"
    ],
    "runtimeResourcePath": "/home/swm/workspace/debian/build/swiftlang/swiftlang-5.9.2/usr/lib/swift"
  }
}

Does anyone have an idea on this one?

5.9.2 is now building on Arm after modifying swift/include/swift/Runtime/Config.h to match Debian's page sizes.
 /// What is the system page size?
-#if defined(__APPLE__) && defined(__arm64__)
+#if defined(__aarch64__)
   // Apple Silicon systems use a 16KB page size
   #define SWIFT_PAGE_SIZE 16384
 #else

Just a happy little update.
With version 5.8.1 I was finally able to figure out why my builds always failed on Ubuntu. It turns out to have been a "known issue" for the Ubuntu folks where they include some GCC LTO flags by default, knowing that it will break a Clang build. So now Swift 5.8.1 is in Ubuntu Plucky and should be available to all Ubuntu derivatives.
Swift 5.9.2 is ready now and should be on its way into the repositories soon.