macOS toolchain issues

[Dan Zheng]

Hi folks,

New macOS toolchains (since 2020-05-03) have a known regression regarding Swift immediate mode and the `TensorFlow` module. TF-1273 tracks it:

// a.swift
import TensorFlow

$ swift a.swift
Assertion failed: (findOption(Name) == Values.size() && "Option already exists!"), function addLiteralOption, file /Users/swiftninjas/s4tf/llvm-project/llvm/include/llvm/Support/CommandLine.h, line 853.
Stack dump:
...

You can work around this by using `swiftc`. On macOS Catalina, you'll also have to set the `DYLD_LIBRARY_PATH` environment variable to work around TF-1260:

$ swiftc tf.swift

$ DYLD_LIBRARY_PATH=/Library/Developer/Toolchains/swift-tensorflow-DEVELOPMENT-2020-05-03-a.xctoolchain/usr/lib/swift/macosx # add this to your shell rc file

# ./tf # no crash

We're looking into fixes for both these issues, I believe the causes of both are known.

A robust fix for TF-1260 will involve removing core stdlib changes from S4TF toolchains (TF-1268). So eventually we'll have to import modules like `TensorFlow` and `_Differentiation` to access differentiation APIs and protocols like `KeyPathIterable` and `VectorProtocol`. On the bright side, this makes S4TF toolchains consistent with swift.org/download toolchains, so code should be more portable across toolchains!

Cheers,

Dan

[Leo Zhao]

When I added this dependency 

.product(name: "TextModels", package: "TensorFlowModels")

to an empty hello world app, it reported this error:

$ swift run exampleApp

[5/5] Linking exampleApp

dyld: Symbol not found: _$s11AllKeyPathss0B12PathIterablePTl

  Referenced from: /Library/Developer/Toolchains/swift-tensorflow-DEVELOPMENT-2020-05-03-a.xctoolchain/usr/lib/swift/macosx/libswiftTensorFlow.dylib

  Expected in: /usr/lib/swift/libswiftCore.dylib

 in /Library/Developer/Toolchains/swift-tensorflow-DEVELOPMENT-2020-05-03-a.xctoolchain/usr/lib/swift/macosx/libswiftTensorFlow.dylib

Abort trap: 6

Is this related to this issue or an issue of its own? I tried to set DYLD_LIBRARY_PATH before running the command, same result.

Thanks,

[Dan Zheng]

Hi Leo,

Could you please share a GitHub repository with the exact SwiftPM project encountering the error?

Thanks,

Dan

--
To unsubscribe from this group and stop receiving emails from it, send an email to swift+un...@tensorflow.org.

[Leo Zhao]

Hi, Dan

This is the demo app that I had issue with. I narrowed it down to line 9 in Package.swift:

        .macOS(.v10_13),

works, but 

        .macOS(.v10_15),

will report the above issue.

https://github.com/leoxzhao/S4TFApp.git

`swift run S4TFApp`

Thanks,

To unsubscribe from this group and stop receiving emails from it, send an email to sw...@tensorflow.org.

[Dan Zheng]

Hi Leo,

Aha, `.macOS(.v10_15)` is known to cause issues - TF-1252 tracks a related issue. The only known workaround is to use a lower deployment target (e.g. `.macOS(.v10_13)`) for now.

I believe the Swift compiler uses Swift system libraries (`/usr/lib/swift`) starting with macOS 10.15: https://swift.org/blog/abi-stability-and-apple.

A robust fix for this issue similarly involves removing core stdlib changes from S4TF toolchains (TF-1268).

Cheers,

Dan

To unsubscribe from this group and stop receiving emails from it, send an email to swift+un...@tensorflow.org.

[Leo Zhao]

Got it. Thanks, Dan.

To unsubscribe from this group and stop receiving emails from it, send an email to sw...@tensorflow.org.

[Dan Zheng]

Hi folks,

TF-1260 (macOS 10.15 `swiftc` binary dynamic linker issues) should be fixed in the latest macOS toolchain (2020-05-08).

This was a longstanding issue that affected even S4TF 0.8 on macOS. There's no need to set `DYLD_LIBRARY_PATH` as a workaround anymore.

// tf.swift

import TensorFlow

print(Tensor(1))

$ swift --version
Swift version 5.3-dev (LLVM 2fd15ba0e5, Swift a68fa1120e)
Target: x86_64-apple-darwin19.4.0

$ swiftc tf.swift; ./tf

1.0 

You shouldn't see errors like the following anymore (but if you do, please report it):

dyld: Symbol not found: _$s11AllKeyPathss0B12PathIterablePTl

  Referenced from: /Library/Developer/Toolchains/swift-tensorflow-DEVELOPMENT-2020-04-23-a.xctoolchain/usr/lib/swift/macosx/libswiftTensorFlow.dylib
  Expected in: /usr/lib/swift/libswiftCore.dylib
 in /Library/Developer/Toolchains/swift-tensorflow-DEVELOPMENT-2020-04-23-a.xctoolchain/usr/lib/swift/macosx/libswiftTensorFlow.dylib
[1]    67267 abort      ./tf

Thank you +Ewa Matejska for the fix! Removing core stdlib changes from S4TF toolchains (TF-1268) is probably still the long-term robust fix.

Cheers,

Dan

[Brennan Saeta]

Thank you Ewa and Dan! Great work. -Brennan

--
To unsubscribe from this group and stop receiving emails from it, send an email to swift+un...@tensorflow.org.

[HP]

Thanks for the ongoing great work!

I have another question regarding building a toolchain on MacOS:

previous self-compiled toolchains worked mostly fine, but now I keep getting stuck in the build during building indexstoredb.

Example output:

/Users/user/swift-source/swift/swift-nightly-install/Library/Developer/Toolchains/swift-tensorflow-LOCAL-2020-05-11-a.xctoolchain/usr/libexec/swift/pm/swiftpm-xctest-helper /Users/user/swift-source/build/buildbot_osx/indexstoredb-macosx-x86_64/x86_64-apple-macosx/release/IndexStoreDBPackageTests.xctest /var/folders/rp/c0_pxspj11g3dzxkc_qfxvdr0000gn/T/TemporaryFile.RYzFiy

and this (and sometimes other tests) hang. Has anyone else experienced this, and/or knows a workaround?

This is on swift branch "tensorflow" with XCode11.4 for ./swift/utils/build-toolchain-tensorflow -p but cherry-picking a compilation fix from 

https://github.com/apple/swift/commit/3c3425d098b495ba9f27426356270fa772993471#diff-b094e70be36918f31a320f9319246fe4

that is not (yet) included in the tensorflow branch.

Many thanks!

[HP]

additional hang point:

/Users/user/swift-source/swift/swift-nightly-install/Library/Developer/Toolchains/swift-tensorflow-LOCAL-2020-05-11-a.xctoolchain/usr/libexec/swift/pm/swiftpm-xctest-helper /Users/user/swift-source/build/buildbot_osx/indexstoredb-macosx-x86_64/x86_64-apple-macosx/release/IndexStoreDBPackageTests.xctest /var/folders/rp/c0_pxspj11g3dzxkc_qfxvdr0000gn/T/TemporaryFile.bMkRjA

[5/35] Testing ISDBTibsTests.TibsCompilationDatabaseTests/testCompilationDatabaseMixedLangTarget

[Dan Zheng]

Hi HP,

It seems like you're building macOS toolchains locally via `utils/build-toolchain-tensorflow`?

There's a known rpath issue regarding `swiftpm-xctest-helper` (SR-12599, SR-12600). You can fix it by cherry-picking this PR: https://github.com/apple/swift-package-manager/pull/2692. We've been cherry-picking it for every macOS toolchain build. Apple folks understandably seem a bit hesitant to accept the patch as is, I just asked for an update here.

Let us know if the cherry-pick fixes the issue for you!

Dan

[HP]

Many thanks for the help!

Unfortunately that pick didn't seem to fix the stalls in the Test runs of indexstoredb.

There seems to be indeed an rpath issue for swiftpm-xctest-helper (see otool -L and otool -l below), and the patch fixes one of the missing search paths, 

but it still cannot find the rpath for XCTest.framework (which I think is only available at 

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks ?

If I add that path as rpath manually, it does not complain about the missing framework anymore, but how would I patch the bootscript to add this as an extra rpath?

I am of course not sure that would solve the hang problem ...

Cheers,

Heiko

otool -L output:

swiftpm-xctest-helper:

/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)

/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1675.129.0)

/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1675.129.0)

@rpath/XCTest.framework/Versions/A/XCTest (compatibility version 1.0.0, current version 16091.0.0)

@rpath/libXCTestSwiftSupport.dylib (compatibility version 1.0.0, current version 1.0.0, weak)

@rpath/libswiftAppKit.dylib (compatibility version 1.0.0, current version 0.0.0, weak)

@rpath/libswiftCore.dylib (compatibility version 1.0.0, current version 0.0.0)

@rpath/libswiftCoreData.dylib (compatibility version 1.0.0, current version 0.0.0, weak)

@rpath/libswiftCoreFoundation.dylib (compatibility version 1.0.0, current version 0.0.0, weak)

@rpath/libswiftCoreGraphics.dylib (compatibility version 1.0.0, current version 0.0.0, weak)

@rpath/libswiftCoreImage.dylib (compatibility version 1.0.0, current version 0.0.0, weak)

@rpath/libswiftDarwin.dylib (compatibility version 1.0.0, current version 0.0.0, weak)

@rpath/libswiftDispatch.dylib (compatibility version 1.0.0, current version 0.0.0, weak)

@rpath/libswiftFoundation.dylib (compatibility version 1.0.0, current version 0.0.0)

@rpath/libswiftIOKit.dylib (compatibility version 1.0.0, current version 0.0.0, weak)

@rpath/libswiftMetal.dylib (compatibility version 1.0.0, current version 0.0.0, weak)

@rpath/libswiftObjectiveC.dylib (compatibility version 1.0.0, current version 0.0.0, weak)

@rpath/libswiftQuartzCore.dylib (compatibility version 1.0.0, current version 0.0.0, weak)

@rpath/libswiftXPC.dylib (compatibility version 1.0.0, current version 0.0.0, weak)

And these are the encoded rpaths after the patch:

Load command 33

          cmd LC_RPATH

      cmdsize 192

         path /Users/user/swift-source/swift/swift-nightly-install/Library/Developer/Toolchains/swift-tensorflow-LOCAL-2020-05-11-a.xctoolchain/usr/lib/swift/macosx (offset 12)

Load command 34

          cmd LC_RPATH

      cmdsize 128

         path /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/lib/swift (offset 12)

Load command 35

          cmd LC_RPATH

      cmdsize 32

         path @loader_path (offset 12)

Load command 36

          cmd LC_RPATH

      cmdsize 192

         path /Users/user/swift-source/swift/swift-nightly-install/Library/Developer/Toolchains/swift-tensorflow-LOCAL-2020-05-11-a.xctoolchain/usr/lib/swift/macosx (offset 12)

Load command 37

          cmd LC_RPATH

      cmdsize 56

         path @executable_path/../lib/swift/macosx (offset 12)

Load command 38

          cmd LC_RPATH

      cmdsize 56

         path @executable_path/../../../lib/swift/macosx (offset 12)

Note that this does not contain 

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks

Is it supposed to pick up the XCTest.swiftmodule instead?

[Dan Zheng]

Hi HP,

Some questions and things to try:

• Do you have Xcode 11.4 (11E146)? Do you have it selected via `sudo --xcode-select /Applications/Xcode.app`?
• Do you have `/Library/Developer/CommandLineTools` installed? If so, you could try uninstalling it as it's currently causing issues.
• I wonder if the XCTest issue is related to this.
• Did you run `swift/utils/update-checkout --clone --skip-repo swift --scheme tensorflow` from `tensorflow` branch?
• What commit of apple/swift are you building from?
• Are you running `swiftl/utils/build-toolchain-tensorflow --pkg`?
• Did you try clean builds?

Dan

[HP]

Many thanks for all these tips!

Yes, these were all fulfilled.

It turns out that the problem of hangs in IndexStoreDB Tests was actually caused by a specific problem in the indexstoredb Test Harness.

Applying the following patch solved the problem:

/* indexstore-db */

diff --git a/Sources/ISDBTibs/Process.swift b/Sources/ISDBTibs/Process.swift

index 59beaee..7c809f4 100644

--- a/Sources/ISDBTibs/Process.swift

+++ b/Sources/ISDBTibs/Process.swift

@@ -44,6 +44,7 @@ extension Process {

   } else {

      p.launch()

   }

+    let data = out.fileHandleForReading.readDataToEndOfFile()

     p.waitUntilExit()

@@ -51,7 +52,6 @@ extension Process {

      throw TibsProcessError.nonZeroExit(p.terminationReason, p.terminationStatus)

    }

-    let data = out.fileHandleForReading.readDataToEndOfFile()

     guard let str = String(data: data, encoding: .utf8) else {

      throw TibsProcessError.invalidUTF8Output(data)

    }

i.e., the Pipe has to be drained before p.waitUntilExit()

is called.

An explanation is given in 

https://stackoverflow.com/questions/33423993/hanging-nstask-using-waituntilexit

Strangely, this hang manifested itself on a machine-dependent basis: a 16-core host hung, a 4-core system didn't ..

To unsubscribe from this group and stop receiving emails from it, send an email to sw...@tensorflow.org.

[Brennan Saeta]

Nice debugging! -Brennan

To unsubscribe from this group and stop receiving emails from it, send an email to swift+un...@tensorflow.org.