Workaround for Pub on Windows XP

[Steve Walker]

I have the following workaround to get pub working on Windows XP.

The fist problem is finding a way to create NTFS junction points on XP. Windows Vista and onwards provide the 'mklink' command but XP does not. To work around this download the sysinternals tool 'junction' from Microsoft and update pub to call it instead of 'mklink'.

1. Download junction from Microsoft: http://technet.microsoft.com/en-gb/sysinternals/bb896768
2. Extract junction.exe and put it somewhere on the path (e.g. C:\WINDOWS)
3. Open a command prompt and type 'junction' to accept the license.
4. In the Dart SDK locate the pub source (in $DART_SDK\util\pub) and edit io.dart:

At about line 310 (in createSymlink())

Change:

    command = 'mklink';
    args = ['/j', to, from];
   
To:    
    command = 'junction';
    args = [to, from];
   
At this point, local SDK packages can be installed in projects by running pub install from the command line.

At this stage I had a problem with online packages which would fail to install because of a 7za error concerning absolute paths. To overcome this download the tools from the MSYS project to provide Windows versions of gzip and tar. Get this from

http://sourceforge.net/projects/mingw/files/MSYS/Base/msys-core/msys-1.0.11/msysCORE-1.0.11-bin.tar.gz/download

Unpack these files somewhere and put the bin folder on the path. Next, revisit io.dart and comment out the following lines in extractTarGz() (around line 624):

//  if (Platform.operatingSystem == "windows") {
//    return _extractTarGzWindows(stream, destination);
//  }


Pub install should now work with online packages.

This approach seems to work fine from the command-line but DartEditor just refuses to run pub on XP. This doesn't seem to be anything to do with pub itself but a hard-coded assumption in the editor.

[Mike]

That's great! Doesn't sound like it would be impossible then for the Dart team to bake in your fixes and find the Editor issue... : )

[Bob Nystrom]

Nice workaround! If you'd like, please file a bug (including all of this email) to get this integrated into pub properly. I don't know if we'll be able to because of licensing reasons, but if it's doable it would be nice if this just worked.

Cheers!

- bob

--
Consider asking HOWTO questions at Stack Overflow: http://stackoverflow.com/tags/dart
 
 

[Christopher Wright]

It should be possible to point people to the junction tool and use it
if present (and running on WinXP).

[Steve Walker]

Since posting the original workaround I've managed to refine it a little and can now present workaround 2.0!

The Microsoft junction utility works well enough but its license requirements prevent it from being a useful, re-distributable solution (and the GUI license dialog is less than impressive!) Fortunately I've found an alternative that is re-distributable in a 'ln' replacement from http://www.flexhex.com/docs/articles/hard-links.phtml. This works much like the UNIX equivalent and is a much better option as it can be re-distributed.

The original workaround actually contains two fixes: one for the junction problem and another for a 7za issue. I've been unable to get 7za to work for any downloaded package and have used the GNU tar and gzip files to provide an alternative. I've seen mention of the7za issue ('cannot use absolute path with this command') elsewhere in this mailing list but no solution appears to be available. Maybe it's an XP issue. Either way I've adopted the MSYS tools instead as they appear to work well enough.

To make life easy for everyone I've combined the files into a download package located at:

http://sourceforge.net/projects/swatwork/files/dart/

To install, unzip to a location of your choice and put the 'bin' directory in the PATH.

(I'm hosting this file myself for the moment but I'm quite happy for it to be moved elsewhere or incorporated into the Dart project in whatever form works best.)

Next the code changes required to Pub.

All the changes are in $DART_SDK/util/pub/io.dart.

CreateSymLink() (line 297) now looks like this:

Future<File> createSymlink(from, to) {
  from = _getPath(from);
  to = _getPath(to);

  var command = 'ln';
  var args = ['-s', from, to];

  if (Platform.operatingSystem == 'windows') {

    // Crude test for WindowsXP
    if (Platform.environment['PROGRAMDATA'] == null) {
      args = [from, to];
    }
    else {
      // Call mklink on Windows to create an NTFS junction point. Only works on
      // Vista or later. (Junction points are available earlier, but the "mklink"
      // command is not.) I'm using a junction point (/j) here instead of a soft
      // link (/d) because the latter requires some privilege shenanigans that
      // I'm not sure how to specify from the command line.

      command = 'mklink';
      args = ['/j', to, from];
    }    
  }

  return runProcess(command, args).transform((result) {
    // TODO(rnystrom): Check exit code and output?
    return new File(to);
  });
}

and extractTarGz() (line 625) now looks like this:

Future<bool> extractTarGz(InputStream stream, destination) {
  destination = _getPath(destination);
 
  if (Platform.operatingSystem == "windows" &&
        Platform.environment['PROGRAMDATA'] != null) { // Ignore for Windows XP
    return _extractTarGzWindows(stream, destination);
  }

  var completer = new Completer<int>();
  var processFuture = Process.start("tar",
      ["--extract", "--gunzip", "--directory", destination]);
  processFuture.then((process) {
    process.onExit = completer.complete;
    stream.pipe(process.stdin);
    process.stdout.pipe(stdout, close: false);
    process.stderr.pipe(stderr, close: false);
  });
  processFuture.handleException((error) {
    completer.completeException(error);
    return true;
  });

  return completer.future.transform((exitCode) => exitCode == 0);
}

These changes try to leave the existing Windows solution intact and use a crude test to detect Windows XP. There may be a better way of doing this. Also I've not been able to test this on Vista or Window 7.

At present, Dart Editor simply refuses to run Pub on Windows XP so no testing has been possible. I've identified the code that needs to change and have raised an issue (6724) for this.

In addition I've added this post to a new issue (6737) as requested.

[Bob Nystrom]

Thanks for filing the tracking bugs!

I don't know when/if we'll get a chance to look at it, but this will be super helpful when we do.

Cheers!

- bob

[Steve Walker]

OK, workaround 3.0, and this time there's no need to put anything in the PATH!

1. Fix for creating directory junctions on Windows XP

Get ln.exe from here: http://www.flexhex.com/docs/articles/download/ln.zip

Extract ln.exe and move it to $DART_SDK/util/pub/7zip.

Now update $DART_SDK/util/pub/io.dart.

createSymlink (line 297) should look like this:

Future<File> createSymlink(from, to) {
  from = _getPath(from);
  to = _getPath(to);

  var command = 'ln';
  var args = ['-s', from, to];

  if (Platform.operatingSystem == 'windows') {
 
    // Crude test for WindowsXP

    if (Platform.environment['PROGRAMDATA'] == null) {

      var scriptPath = new File(new Options().script).fullPathSync();
      var scriptDir = new Path.fromNative(scriptPath).directoryPath;
      command = scriptDir.append('7zip/ln.exe').canonicalize().toNativePath();      
      args = [from, to];

    }
    else {
      // Call mklink on Windows to create an NTFS junction point. Only works on
      // Vista or later. (Junction points are available earlier, but the "mklink"
      // command is not.) I'm using a junction point (/j) here instead of a soft
      // link (/d) because the latter requires some privilege shenanigans that
      // I'm not sure how to specify from the command line.

      command = 'mklink';
      args = ['/j', to, from];
    }
  }

  return runProcess(command, args).transform((result) {

    // TODO(rnystrom): Check exit code and output?
    return new File(to);
  });
}

2. Fix the extract error with 7za

Locate _extractTarGzWindows (now at line 638) and change line 687 to look like this:

    // Untar the archive into the destination directory.
    return runProcess(command, ['x', tarFile], workingDir: destination);

   
Pub should now work on Windows XP.

I've updated issue 6737 with this new information.

[Bernd Wessels]

Sorry guys,

none of the solutions mentioned here works on Windows Server 2003. Some of the workarounds even create damaged symbolic links which can only be deleted with rj.exe.

Can't you just copy the files and folders on pre-Vista machines? The symbolic link stuff is dark magic on Windows anyway.

I think, as long as this problem exists, it will scare away lots of windows developers. I certainly suspended everything for now, because the broken pub really makes the whole Dart Editor feel broken.

Cheers

Bernd

[Steve Walker]

I suggest you try linkd from the Windows Server 2003 resource kit.

See this thread for details - https://groups.google.com/a/dartlang.org/forum/#!topic/misc/PKznAUvoCKw

[Anton Sheld]

Into: C:\dart\dart-sdk\util\pub\7zip // put downloading files and now Dependencies installed! Thank You.

[Bob Nystrom]

On Tue, Nov 20, 2012 at 7:37 PM, Bernd Wessels <wessel...@gmail.com> wrote:

Sorry guys,

none of the solutions mentioned here works on Windows Server 2003. Some of the workarounds even create damaged symbolic links which can only be deleted with rj.exe.

Can't you just copy the files and folders on pre-Vista machines?

That would work for dependencies (though it's a bit cumbersome), but doesn't work as well for your package's self link. When pub creates a "packages" directory for your package, it also includes in there a link back to the package's own "lib" directory. That way you can use "package:" to import stuff from your own package. This is particularly important for examples, where you want the example code to look like what a normal user would write.

We could potentially copy the contents of your lib directory in there, but then you'd have to run pub install any time you made any change to code in your package.

I would like to have a better story for legacy Windows versions, but I have to balance that against lots of other priorities.

Cheers,

- bob

 

The symbolic link stuff is dark magic on Windows anyway.

I think, as long as this problem exists, it will scare away lots of windows developers. I certainly suspended everything for now, because the broken pub really makes the whole Dart Editor feel broken.

Cheers

Bernd

--

[Steve Walker]

There have been a lot of changes to this workaround so I want to present the current 'best practice' version here to clarify.

Firstly, the 7za issue appears to be fixed so there's no longer any need to use MSYS GZip/Tar tools to extract the packages. 

The remaining issue for pre-Vista versions of Windows is the requirement to create directory links. Three tools have been identified to address this: junction, linkd and flexhex ln. Of these, linkd seems to be the best and so far has been found to work on Windows XP and Windows Server 2003, so this is the recommended solution.

To get it, download the Windows Server 2003 Resource Kit Tools from:

http://www.microsoft.com/en-us/download/details.aspx?id=17657

The simplest approach is to just install the package, this will make linkd.exe available via the path.

Alternatively, if you have 7-zip installed and don't want your machine/path cluttered up with these tools you can just extract lind.exe directly:

1. Right-click rktools.exe and select 7zip->Open Archive.

2. Right-click rktools.msi and select Open Inside.

3. Locate linkd.exe and drag to a folder somewhere on the PATH e.g. C:\WINDOWS.

With linkd installed update Pub to use it instead of the mklink command.

In %DART_SDK/util/pub/io.dart locate createSymlink (around line 297) and change the lines in the Windows section to this:

  if (Platform.operatingSystem == 'windows') {

    // Call mklink on Windows to create an NTFS junction point. Only works on
    // Vista or later. (Junction points are available earlier, but the "mklink"
    // command is not.) I'm using a junction point (/j) here instead of a soft
    // link (/d) because the latter requires some privilege shenanigans that
    // I'm not sure how to specify from the command line.

    command = 'linkd';
    args = [to, from];
  }

Pub should now be able to add and remove dependencies for your projects.

[Bob Nystrom]

Thanks for continuing to hack on this! I don't know when I'll have any time to spend on this, but this will definitely make it much quicker for me to make progress here.

Cheers,

- bob

--

[Arete]

Hi Steve,

Just tried your fix as described below on the Solar 3D sample that comes with Dart and it did not work.

For the Solar 3D sample, Pub Update/Install does not install the vector_math_browser.dart file, resulting in an error:

"Cannot find referenced source: package:vector_math/vector_math_browser.dart" being flagged in solar.dart.

Noticed that solar3d\packages\vector_math folder is empty after I run the pub install from the Tools menu in Dart Editor.

Any insight?

--frank

[Steve Walker]

I've tested this with a fresh install of Dart Editor (15699) and the dependencies installed correctly. This is with linkd.exe in the path

and the following fix to io.dart (line 332):

    command = 'linkd';

    args = [to, from];

When I originally developed this fix I would sometimes get empty directories if the 'to' and 'from' arguments were reversed - each different version of the workaround requires a different command line combination and getting it wrong once can stop it from working even after you have changed the code to use the correct arguments. Can you confirm that you have the correct arguments for linkd and also, try clearing out the Pub cache in C:\Documents and Settings\<user>\Application Data\Pub before trying again.

[Steve Walker]

As of M3 this still appears to work.

in io.dart use:

command = 'linkd';
args = [symlink, target];

In addition you'll have to remove the latest "Windows XP users not welcome" trap - it's in pub.dart around line 144.

Frankly, I wouldn't bother.

[mezoni]

>> In addition you'll have to remove the latest "Windows XP users not welcome" trap - it's in pub.dart around line 144.

The line number may different but signature the following.

"Sorry, but pub is not supported on Windows XP."

And the latest source code.

Future validatePlatform() {

  return defer(() {

    if (Platform.operatingSystem != 'windows') return;

    return runProcess('ver', []).then((result) {

      if (result.stdout.join('\n').contains('XP')) {

        log.error('Sorry, but pub is not supported on Windows XP.');

        exit(exit_codes.USAGE);

      }

    });

  });

}

Disable this lines by commenting.

Thanks, Steve Walker.

[mezoni]

The patcher that written in haste.

import 'dart:io';

main() {  

  var patches = {};

  patches["io.dart"] = [];

  patches["pub.dart"] = [];

  patches["io.dart"].add([

    "    command = 'mklink';\r\n    args = ['/j', symlink, target]",

    "    command = 'linkd';\r\n    args = [symlink, target]"]);

  patches["pub.dart"].add([

    "        log.error('Sorry, but pub is not supported on Windows XP.');\n        exit(exit_codes.USAGE)",                       

    "        //log.error('Sorry, but pub is not supported on Windows XP.');\n        //exit(exit_codes.USAGE)"]);

  

  var dartSdk = Platform.environment['DART_SDK'];

  if(dartSdk == null || dartSdk.isEmpty) {

    print('Dart SDK not found.');

    exit(1);

  } else if(!dartSdk.endsWith('\\')) {

    dartSdk = '$dartSdk\\';

  }

  

  var patcher = new Patcher();

  for(var file in patches.keys) {

    var patch = patches[file];

    var filename = '${dartSdk}util\\pub\\${file}';

    patcher.patch(filename, patch);

  }  

}

class Patcher {  

  void patch(String filename, List<String> patches) {

    var file = new File(filename);

    if(!file.existsSync()) {

      print('File "$filename" not found.');      

      exit(1);

    }

    

    var text = file.readAsStringSync();    

    for(var patch in patches) {

      var src = patch[0];

      var dst = patch[1];

      if(text.contains(src)) {

        text = text.replaceFirst(src, dst);    

      } else {

        print('File "$filename" not patched.');

        print('Pattern not found.');

        var lines = src.split('\r\n');

        for(var line in lines) {

          print(line);

        }        

        return;

      }

    }

    

    file.writeAsStringSync(text);

    print('File patched: $filename');

  }

}

[Bernd Wessels]

Nice. To bad that everybody in my company has to develop on win2003 server. Good that there is always a workaround.

[Robert Steed]

I decided workaround by replacing the mklink command with a batch file to emulate mklink /j using the linkd command.  I placed both linkd.exe and mklink.bat both in a directory on the path so cmd will work.  This has the advantage that no modification is required to the original code that calls mklink.

mklink.bat:

echo off

if /I "%1" == "/j" (

     linkd %2 %3

     exit /b 0)

echo mklink %1 fail - can only do junction link in XP

pause

exit /b 1

[Michael Chean]

Thanks, that saves me some time!

[Michael Chean]

It appears that on the latest editor build this isn't possible anymore because of the snapshotting of pub.  Do you have any ideas short of re-compiling the pub.dart (which it seems you can't do on WinXp?

Mike

On Sunday, March 24, 2013 3:06:02 PM UTC-7, Robert Steed wrote:

[R. Dennis Steed]

I've not installed the latest build.

You might try opening the snapshot in a hex editor.  Look for the signature "Sorry, but pub is not supported on Windows XP" in the snapshot.  If you can find "XP" just before that (where pub is doing the version comparison), try changing it to "XZ" or anything that won't match "XP".

Dennis

[Michael Chean]

Thanks.  By the way I've placed a ticket on this item, but have been told that it will not be fixed.  

Issue: http://code.google.com/p/dart/issues/detail?id=10358

[Michael Chean]

By the way that worked!

[Robert Steed]

I've not tried it yet myself, but I'm glad hear the hex editor hack worked on the snapshot.

Incidentally, the workaround that this whole thread is based on (finding a command line alternative to mklink /j for use in XP) appears unnecessary now because last month the use of mklink to create junctions was replaced by direct API calls in dart:io (see issue 9467).  At least for the time being, the Dart team may have undone the incompatibilities with XP.

How ironic.  Could someone check to see if Pub will run on Windows 2000?

[Blackspawn]

What's the current status on this? Is it possible to develop for Dart on WinXP? Do the workarounds still work? Other possibilities? I'm especially interested in coding in Dart, then compiling to JS. Are there maybe other toolchains one could use?

[Bernd Wessels]

Try Webstorm.
But there might be issues in pub as well which is independent from the iDE.

On 15/12/2013 1:00 AM, "Blackspawn" <blacks...@gmail.com> wrote:

What's the current status on this? Is it possible to develop for Dart on WinXP? Do the workarounds still work? Other possibilities? I'm especially interested in coding in Dart, then compiling to JS. Are there maybe other toolchains one could use?

--
For other discussions, see https://groups.google.com/a/dartlang.org/
 
For HOWTO questions, visit http://stackoverflow.com/tags/dart
 
To file a bug report or feature request, go to http://www.dartbug.com/new

[Bob Nystrom]

On Sat, Dec 14, 2013 at 4:00 AM, Blackspawn <blacks...@gmail.com> wrote:

What's the current status on this? Is it possible to develop for Dart on WinXP? Do the workarounds still work? Other possibilities? I'm especially interested in coding in Dart, then compiling to JS. Are there maybe other toolchains one could use?

This thread is quite old.

Since then, the team has decided to not support Windows XP. If you can get it working on your own, that's great, but we can't really offer any help. Since we have no testing infrastructure for XP, it would be impossible for us to guarantee any level of support since we can't even validate it ourselves.

Sorry. :(

- bob

[罗勇刚(Yonggang Luo)]

Is that possible to wrapping the mklink things as dart api? So we directly using dart code to create junctions/symbol links

--
For other discussions, see https://groups.google.com/a/dartlang.org/
 
For HOWTO questions, visit http://stackoverflow.com/tags/dart
 
To file a bug report or feature request, go to http://www.dartbug.com/new

To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

[Bob Nystrom]

On Mon, Dec 16, 2013 at 8:47 PM, 罗勇刚(Yonggang Luo) <luoyo...@gmail.com> wrote:

Is that possible to wrapping the mklink things as dart api? So we directly using dart code to create junctions/symbol links

Dart does now have an API for creating symlinks, which wasn't the case when pub first implemented this. I think we may have switched over to use it? I'm not sure.

Either way, though, there's no way for Dart itself to implement such an API on Windows XP: the operating system itself doesn't have that functionality.

Cheers,

- bob

[罗勇刚(Yonggang Luo)]

I found xp support for junction API by default, no driver need to be installed, it's happens to be a stub for xp to create symbolic staff in xp.

> Cheers,

[Robert Steed]

On Monday, December 16, 2013 12:10:03 PM UTC-5, Bob Nystrom wrote:

...

Since then, the team has decided to not support Windows XP. If you can get it working on your own, that's great, but we can't really offer any help. Since we have no testing infrastructure for XP, it would be impossible for us to guarantee any level of support since we can't even validate it ourselves.

Sorry. :(

- bob

I haven't looked at Dart under XP for a while, so this may be a bit outdated.  Last time I looked, though, Dart checked the OS version and stopped if it was XP.  Makes it kind of hard to even try at our own risk.  Would the team consider changing the mandatory halt under XP to a warning like, "Unsupported operating system, proceed at your own risk" option?

Dennis 

[罗勇刚(Yonggang Luo)]

Its dart or dart editor?
> Dennis 

[Bob Nystrom]

On Thu, Dec 19, 2013 at 8:55 PM, Robert Steed <rds...@steeds.us> wrote:

Would the team consider changing the mandatory halt under XP to a warning like, "Unsupported operating system, proceed at your own risk" option?

Sure, that's fine by me. Can you file a bug please? I have outsourced my brain's memory to the bug tracker.

Thanks!

- bob