"%~sk" SFN bug in "for %k" loops
Liviu
SFN in certain cases. On a typical drive with reasonably many LFNs
for /r X:\ %k in (*) do @if not exist "%~sk" echo "%~sk"
should find quite a few such examples.
The following is perfectly repeatable here under xp.sp3. This simple
test case appears to work under win7.x64, but other more complicated
ones will fail there, too. It has to do with LFNs, and perhaps SFN vs.
LFN name lengths, but I still don't find much rhyme or reason. Has
anyone seen (any reference to) this particular bug?
Liviu
P.S. Xp.sp3 test case setup:
- create a new directory "C:\test" (name is not important);
- create a subdirectory "123456789.abcd" and add 6 files into it
"12.abcd", "123.abcd", "1234.abcd", "12345.abcd", "123456.abcd",
"1234567.abcd" (empty files are fine);
- run the following (watch for line wraps in the posted text):
C:\test>dir /x 123456789.abcd
C:\test>for %k in (123456789.abcd\*) do @echo.
& echo sfn: "%~sk" & echo lfn: "%~fk"
C:\test>for %k in (123456789.abcd\*) do @if not exist "%~sk"
echo -?-: "%~sk"
My results are copied below. Out of the 6 files, 4 return an invalid SFN
which then passes the "if not exist" test.
________________________________
C:\test>dir /x 123456789.abcd
Directory of C:\test\123456789.abcd
05/29/2010 03:51 PM 0 120F6A~1.ABC 12.abcd
05/29/2010 03:51 PM 0 123~1.ABC 123.abcd
05/29/2010 03:51 PM 0 1234~1.ABC 1234.abcd
05/29/2010 03:51 PM 0 12345~1.ABC 12345.abcd
05/29/2010 03:51 PM 0 123456~1.ABC 123456.abcd
05/29/2010 03:51 PM 0 123456~2.ABC 1234567.abcd
6 File(s) 0 bytes
C:\test>for %k in (123456789.abcd\*) do @echo.
& echo sfn: "%~sk" & echo lfn: "%~fk"
sfn: "C:\test\123456~1.ABC\120F6A~1.ABC"
lfn: "C:\test\123456789.abcd\12.abcd"
sfn: "C:\test\123456~1.ABC\123~1.ABCd"
lfn: "C:\test\123456789.abcd\123.abcd"
sfn: "C:\test\123456~1.ABC\1234~1.ABCd"
lfn: "C:\test\123456789.abcd\1234.abcd"
sfn: "C:\test\123456~1.ABC\12345~1.ABCd"
lfn: "C:\test\123456789.abcd\12345.abcd"
sfn: "C:\test\123456~1.ABC\123456~1.ABCd"
lfn: "C:\test\123456789.abcd\123456.abcd"
sfn: "C:\test\123456~1.ABC\123456~2.ABC"
lfn: "C:\test\123456789.abcd\1234567.abcd"
C:\test>for %k in (123456789.abcd\*) do @if not exist "%~sk"
echo -?-: "%~sk"
-?-: "C:\test\123456~1.ABC\123~1.ABCd"
-?-: "C:\test\123456~1.ABC\1234~1.ABCd"
-?-: "C:\test\123456~1.ABC\12345~1.ABCd"
-?-: "C:\test\123456~1.ABC\123456~1.ABCd"
Todd Vargo
> The built-in cmd "for %k" loop appears to return the wrong "%~sk"
> SFN in certain cases. On a typical drive with reasonably many LFNs
> for /r X:\ %k in (*) do @if not exist "%~sk" echo "%~sk"
> should find quite a few such examples.
>
> The following is perfectly repeatable here under xp.sp3. This simple
> test case appears to work under win7.x64, but other more complicated
> ones will fail there, too. It has to do with LFNs, and perhaps SFN vs.
> LFN name lengths, but I still don't find much rhyme or reason. Has
> anyone seen (any reference to) this particular bug?
Yes, this has been discussed here several times and is well known amoung the
regulars of this group. The only good choice is to not rely on the ~s
option, nor the /d or /r switches.
--
Todd Vargo
(Post questions to group only. Remove "z" to email personal messages)
--- news://freenews.netfront.net/ - complaints: ne...@netfront.net ---
foxidrive
>The built-in cmd "for %k" loop appears to return the wrong "%~sk"
>SFN in certain cases. On a typical drive with reasonably many LFNs
> for /r X:\ %k in (*) do @if not exist "%~sk" echo "%~sk"
>should find quite a few such examples.
>
>The following is perfectly repeatable here under xp.sp3. This simple
>test case appears to work under win7.x64, but other more complicated
>ones will fail there, too. It has to do with LFNs, and perhaps SFN vs.
>LFN name lengths, but I still don't find much rhyme or reason. Has
>anyone seen (any reference to) this particular bug?
Yes, it has been discussed here before.
This batch file illustrates it easily and also a workaround
@echo off
md "ABCDEFGHIJKL.XYZ"
cd "ABCDEFGHIJKL.XYZ"
type nul >"mno.1234"
type nul >"mnop.1234"
type nul >"mnopq.1234"
type nul >"mnopqr.1234"
type nul >"mnopqrs.1234"
dir /x
for %%k in (*.*) do echo sfn: "%%~sk" & echo lfn: "%%~fk"
pause
for %%k in (*.*) do if not exist "%%~sk" echo -?-: "%%~sk"
pause
command /c cmd /c for %%k in (*.*) do @echo sfn: %%~sk
pause
--
Regards,
Mic
Liviu
> Liviu wrote:
>> The built-in cmd "for %k" loop appears to return the wrong "%~sk"
>> SFN in certain cases. On a typical drive with reasonably many LFNs
>> for /r X:\ %k in (*) do @if not exist "%~sk" echo "%~sk"
>> should find quite a few such examples.
>>
>> The following is perfectly repeatable here under xp.sp3. This simple
>> test case appears to work under win7.x64, but other more complicated
>> ones will fail there, too. It has to do with LFNs, and perhaps SFN
>> vs. LFN name lengths, but I still don't find much rhyme or reason.
>> Has anyone seen (any reference to) this particular bug?
>
> amoung the regulars of this group. The only good choice is to not rely
> on the ~s option, nor the /d or /r switches.
Thanks. Would you have some references, or recall more details
about what exactly is affected? Google'ing for batch file syntax with
odd chars like "for /r" or "%~s" doesn't go too far.
It's not that I am doubting your advice, just trying to assess the
scope - or maybe whether the fail conditions can be somehow detected
beforehands. All I was trying to actually do was use those "for" loops
to test an unrelated .cmd of mine.
Liviu
Liviu
> On Sat, 29 May 2010 20:11:20 -0500, "Liviu" <lab...@gmail.c0m> wrote:
>
>>The built-in cmd "for %k" loop appears to return the wrong "%~sk"
>>SFN in certain cases. On a typical drive with reasonably many LFNs
>> for /r X:\ %k in (*) do @if not exist "%~sk" echo "%~sk"
>>should find quite a few such examples.
>
>
> This batch file illustrates it easily and also a workaround
>
> @echo off
> md "ABCDEFGHIJKL.XYZ"
> cd "ABCDEFGHIJKL.XYZ"
> type nul >"mno.1234"
> type nul >"mnop.1234"
> type nul >"mnopq.1234"
> type nul >"mnopqr.1234"
> type nul >"mnopqrs.1234"
>
> dir /x
> for %%k in (*.*) do echo sfn: "%%~sk" & echo lfn: "%%~fk"
> pause
> for %%k in (*.*) do if not exist "%%~sk" echo -?-: "%%~sk"
> pause
> command /c cmd /c for %%k in (*.*) do @echo sfn: %%~sk
> pause
Thanks. Doesn't really help with my problem, since I need both
LFNs and SFNs in the same "session" - but then I know that I have
not stated the full context explicitly. I'll remember the trick, though.
Liviu
Todd Vargo
> Todd Vargo wrote...
Sorry, I didn't keep any references to the discussions. Foxidrive provided a
batch example, but personally, I would use VBScript to capture SFNs.
Timo Salmi
> Sorry, I didn't keep any references to the discussions. Foxidrive
> provided a batch example, but personally, I would use VBScript to
> capture SFNs.
A VBS-aided script for a single file to start with
@echo off & setlocal enableextensions
if "%~1"=="" (
echo Usage: %~0 ["LongFileName"]
goto :EOF)
::
:: Check for the file's existence
set found_=
if exist "%~1" if not exist "%~1\" set found_=true
if not defined found_ (
echo File "%~1" not found or is a folder
goto :EOF)
::
:: Build a Visual Basic Script and run it
set vbs_=%temp%\tmp$$$.vbs
set skip=
findstr "'%skip%VBS" "%~f0" > %vbs_%
for /f %%a in ('cscript //nologo "%vbs_%" "%~1"') do set sfn_=%%a
::
:: Display the results
echo %~nx1
echo %sfn_%
::
:: Clean up
for %%f in ("%vbs_%") do if exist %%f del %%f
endlocal & goto :EOF
'
'The Visual Basic Script
WScript.Echo GetShortName(WScript.Arguments.Item(0)) 'VBS
'
Function GetShortName(filespec) 'VBS
Dim fso, f, s 'VBS
Set fso = CreateObject("Scripting.FileSystemObject") 'VBS
Set f = fso.GetFile(filespec) 'VBS
GetShortName = f.ShortName 'VBS
End Function 'VBS
All the best, Timo
--
Prof. Timo Salmi mailto:t...@uwasa.fi ftp & http://garbo.uwasa.fi/
Hpage: http://www.uwasa.fi/laskentatoimi/english/personnel/salmitimo/
Department of Accounting and Finance, University of Vaasa, Finland
Useful CMD script tricks http://www.netikka.net/tsneti/info/tscmd.php
Timo Salmi
> A VBS-aided script for a single file to start with
For more details, examples and extensions see the updated FAQ item
77} How do I list the files in \All Users\Desktop in the short format?
http://www.netikka.net/tsneti/info/tscmd077.htm
foxidrive
>On 30.05.2010 21:12 Timo Salmi wrote:
>> A VBS-aided script for a single file to start with
>(snip)
>
>For more details, examples and extensions see the updated FAQ item
>
> 77} How do I list the files in \All Users\Desktop in the short format?
> http://www.netikka.net/tsneti/info/tscmd077.htm
>
Timo, it would be useful if you added a section that discusses the
random issue where the pure CMD solution can return incorrect short
filenames, and that it is wiser to rely on a VBS aided solution.
--
Regards,
Mic
foxidrive
>Thanks. Would you have some references, or recall more details
>about what exactly is affected? Google'ing for batch file syntax with
>odd chars like "for /r" or "%~s" doesn't go too far.
I don't know how useful the link is - I didn't reread the thread.
I googled on "short filename bug" in groups.
--
Regards,
Mic
Timo Salmi
I included a link to Todd's posting to the effect, and thus to the
thread. Otherwise than that, I have not tested the big myself. Thus the
omission.
foxidrive
>On 31.05.2010 12:33 foxidrive wrote:
>> On Mon, 31 May 2010 10:52:51 +0300, Timo Salmi <t...@uwasa.fi> wrote:
>>> http://www.netikka.net/tsneti/info/tscmd077.htm
>
>> Timo, it would be useful if you added a section that discusses the
>> random issue where the pure CMD solution can return incorrect short
>> filenames, and that it is wiser to rely on a VBS aided solution.
>
>I included a link to Todd's posting to the effect, and thus to the
>thread. Otherwise than that, I have not tested the big myself. Thus the
>omission.
You can run the batch file I posted in this thread to see the issue Timo
- and what I meant was just a terse opening warning like so: Finding
short filenames with CMD can give erroneous results. Use a VBS aided
solution for reliability.
--
Regards,
Mic
foxidrive
wrote:
>On Mon, 31 May 2010 18:24:43 +0300, Timo Salmi <t...@uwasa.fi> wrote:
>
>>On 31.05.2010 12:33 foxidrive wrote:
>>> On Mon, 31 May 2010 10:52:51 +0300, Timo Salmi <t...@uwasa.fi> wrote:
>>>> http://www.netikka.net/tsneti/info/tscmd077.htm
>>
>>> Timo, it would be useful if you added a section that discusses the
>>> random issue where the pure CMD solution can return incorrect short
>>> filenames, and that it is wiser to rely on a VBS aided solution.
>>
>>I included a link to Todd's posting to the effect, and thus to the
>>thread. Otherwise than that, I have not tested the big myself. Thus the
>>omission.
>
>You can run the batch file I posted in this thread to see the issue Timo
>- and what I meant was just a terse opening warning like so: Finding
>short filenames with CMD can give erroneous results. Use a VBS aided
>solution for reliability.
A bit more info on the topic:
I've just tested it in Win7RC1 and Vista that the issue doesn't exist in
those versions. It fails in XP Pro SP3... and I don't have W2K to test.
--
Regards,
Mic
Timo Salmi
>>> On Mon, 31 May 2010 10:52:51 +0300, Timo Salmi <t...@uwasa.fi> wrote:
>>>> http://www.netikka.net/tsneti/info/tscmd077.htm
> You can run the batch file I posted in this thread to see the issue Timo
Link to the said posting added. Thanks.
Todd Vargo
The bug was discovered before XP so. The version of Win95cmd which I used
was ported from W2K had the bug (and several others) also.
Liviu
below involves Unicode strings, it is easier and safer this way.
"foxidrive" <got...@woohoo.invalid> wrote...
>
> I've just tested it in Win7RC1 and Vista that the issue
> doesn't exist in those versions.
Don't know about Vista, but the issue is still not (fully) fixed
in Win7 (pro x64 final, if it matters). As I said in the original
post, my particular test case appears to work in Win7, and
so does the one you posted. But other cases fail in a similar
way there, too.
I haven't found a simple and all-ASCII example, yet, but see
attached for a test case which doesn't work in Win7 either.
You'll need to UnRar it to C:\test preserving subdirectories
and the Unicode names. Then open the for7bug.txt file and
copy paste either command line to a cmd prompt in C:\test.
One (and only one) of the three files in the non-ASCII directory
will be listed with an invalid "%~sk" and fail the "if exist" test.
Liviu
foxidrive
>Don't know about Vista, but the issue is still not (fully) fixed
>in Win7 (pro x64 final, if it matters). As I said in the original
>post, my particular test case appears to work in Win7, and
>so does the one you posted. But other cases fail in a similar
>way there, too.
>
>I haven't found a simple and all-ASCII example, yet, but see
>attached for a test case which doesn't work in Win7 either.
>
>You'll need to UnRar it to C:\test preserving subdirectories
>and the Unicode names. Then open the for7bug.txt file and
>copy paste either command line to a cmd prompt in C:\test.
>One (and only one) of the three files in the non-ASCII directory
>will be listed with an invalid "%~sk" and fail the "if exist" test.
I tested this in Win7RC1 32 bit and found that there are further issues.
With this code:
@echo off
for /f "delims=" %%k in ('dir /b /s') do echo "%%~sk" & echo "%%~k"
a dir /x gave me this, and you can see the short directory name
http://i50.tinypic.com/2dgnsqp.png
but when executed in the batch file it fails to use the short directory
name. I guess I am just confirming that it is fubar.
http://i46.tinypic.com/2a5g9cn.png
--
Regards,
Mic
Liviu
>
> I tested this in Win7RC1 32 bit and found that there are further
> issues.
>
> With this code:
>
> @echo off
> for /f "delims=" %%k in ('dir /b /s') do echo "%%~sk" & echo "%%~k"
>
> a dir /x gave me this, and you can see the short directory name
> http://i50.tinypic.com/2dgnsqp.png
>
> but when executed in the batch file it fails to use the short
> directory name. I guess I am just confirming that it is fubar.
>
> http://i46.tinypic.com/2a5g9cn.png
You are right, but that's arguably a different fubar ;-) The "for /f"
loop has a problem with immediate commands like "in ('dir /b')" when
the returned fileset contains characters outside the active codepage.
This is one reason I try to avoid those constructs when possible.
In contrast, the native "for" loops work fine with all valid Unicode
pathnames, as long as only the LFNs are used. Yet, oddly enough,
they fail to return the correct SFNs in some cases, as noted.
Liviu
foxidrive
>You are right, but that's arguably a different fubar ;-) The "for /f"
>loop has a problem with immediate commands like "in ('dir /b')" when
>the returned fileset contains characters outside the active codepage.
>This is one reason I try to avoid those constructs when possible.
Did you start cmd with the /u switch? Does that change the outcome?
>In contrast, the native "for" loops work fine with all valid Unicode
>pathnames, as long as only the LFNs are used. Yet, oddly enough,
>they fail to return the correct SFNs in some cases, as noted.
In XP the /r and /d versions of the for command fail in different ways
also. I don't recall the circumstances, only that the /f version was
reliable where the others failed.
If those issues continue in Vista or Win7 I don't know...
--
Regards,
Mic
Liviu
> "Liviu" <lab...@gmail.c0m> wrote:
>
>>You are right, but that's arguably a different fubar ;-) The "for /f"
>>loop has a problem with immediate commands like "in ('dir /b')" when
>>the returned fileset contains characters outside the active codepage.
>>This is one reason I try to avoid those constructs when possible.
>
> Did you start cmd with the /u switch? Does that change the outcome?
There may well be a secret incantation to make it work with "cmd /u"
but I haven't found it, yet, after trying quite a few variations.
>>In contrast, the native "for" loops work fine with all valid Unicode
>>pathnames, as long as only the LFNs are used. Yet, oddly enough,
>>they fail to return the correct SFNs in some cases, as noted.
>
> In XP the /r and /d versions of the for command fail in different
> ways also. I don't recall the circumstances, only that the /f version
> was reliable where the others failed.
I'd be curious to hear more about any /r and /d issues (except for
the SFN part). FWIW below is copied a boilerplate .cmd of mine
which passed (so far) test cases with various combinations of
Unicode pathnames and "special" batch characters (%^&!). It has
one "for /d" loop for directories and another "for" one for files. When
successful, a run of e.g. "echodirs.cmd C:\test >nul" should output
nothing at all. If you find a workable alternative using "for /f" loops,
instead, then I'd be interested in that.
Liviu
________________________________
:: echodirs.cmd - recursive directory and file enumerator a la 'dir /s'
:: supports special %^&! characters and unicode pathnames
::
:: usage: echodirs [/call] [/quot] [target_dir] [>nul]
::
:: /call to be used when echodirs is invoked with a 'call'
:: /quot quotes the emitted output
:: target_dir root directory for the recursion
:: >nul trims the output to just errors (echo 1>&2)
::
:: default: echodirs "."
:: ____
::
:: 10-05-31 tested under xp.sp3, no errors reported from either of
:: for /d /r X:\root %%k in (*) do @echodirs "%%~k" >nul
:: for /d /r X:\root %%k in (*) do @call echodirs /call "%%~k" >nul
:: where X:\root was the root directory of the tested pathnames
::
:: n.b. the "for" loops above may fail with "%%~sk" (instead of "%%~k")
:: in certain sfn/lfn cases, due to a cmd bug still present in win7.x64
@echo off
@rem lest 'cmd /v:on' and %1 pathname contains ! marks
setlocal disabledelayedexpansion
if not "%~1"=="/call" ( set "call=" ) else ( set "call=1" & shift )
if not "%~1"=="/quot" ( set "quot=" ) else ( set "quot=1" & shift )
if not "%~1"=="" ( set "tree1=%~1" ) else ( set "tree1=." )
if "%call%"=="1" ( set "tree1=%tree1:^^=^%" & set "call=" )
@rem root-level sanity check
if not exist "%tree1%" (
echo -.- "%tree1%" 1>&2
) else (
@rem 'call' rather than 'goto' as :tree expects quoted ^ carets to be
doubled
@rem double quoted % percents since 'call' will halve (or lose) them
setlocal enabledelayedexpansion
call :tree "!tree1:%%=%%%%!"
endlocal
)
endlocal
goto :eof
:: calls :dir for current root, then recurses into :tree for
subdirectories
::
:: %1 current relative path from original root down
:tree
@rem 'delayed expansion' must be off for ! marks in %~1 to be passed
literally
setlocal disabledelayedexpansion
@rem quote the entire 'set' argument for & ampersands to be passed
literally
set "tree1=%~1"
@rem add a trailing '\' to drive roots
if "%tree1:~-1%"==":" set "tree1=%tree1%\"
@rem halve quoted ^ carets since 'call' did double them
set "tree1=%tree1:^^=^%"
@rem double quoted % percents since 'call' will halve (or lose) them
setlocal enabledelayedexpansion
call :dir "!tree1:%%=%%%%!"
endlocal
@rem remove any trailing '\' from directory
if "%tree1:~-1%"=="\" set "tree1=%tree1:~0,-1%"
@rem 'delayed expansion' must be off for ! marks in %%~d to be passed
literally
for /d %%d in ("%tree1%\*") do (
@rem quote the entire 'set' argument for & ampersands to be passed
literally
set "for1=%%~d"
@rem double quoted % percents since 'call' will halve (or lose) them
setlocal enabledelayedexpansion
call :tree "!for1:%%=%%%%!"
endlocal
)
endlocal
goto :eof
:: directory-level stub, currently echo's the path
:: calls :file for each file in the current directory
::
:: %1 current relative path from original root down
:dir
@rem 'delayed expansion' must be off for ! marks in %~1 to be passed
literally
setlocal disabledelayedexpansion
@rem quote the entire 'set' argument for & ampersands to be passed
literally
set "dir1=%~1"
@rem halve quoted ^ carets since 'call' did double them
set "dir1=%dir1:^^=^%"
@rem dir-level sanity check
if not exist "%dir1%" echo -d- "%dir1%" 1>&2
if "%quot%"=="1" (
@rem quotes required for & ampersands to echo
echo. & echo "%dir1%"
) else (
setlocal enabledelayedexpansion
echo. & echo !dir1!
endlocal
)
@rem 'delayed expansion' must be off for ! marks in %%~f to be passed
literally
for %%f in ("%dir1%\*") do (
@rem quote the entire 'set' argument for & ampersands to be passed
literally
set "for1=%%~f"
@rem double quoted % percents since 'call' will halve (or lose) them
setlocal enabledelayedexpansion
call :file "!for1:%%=%%%%!"
endlocal
)
endlocal
goto :eof
:: file-level stub, currently echo's the filename
::
:: %1 current relative path from original root down
:file
setlocal disabledelayedexpansion
set "file1=%~nx1"
set "file1=%file1:^^=^%"
@rem file-level sanity check
set "path1=%~1"
set "path1=%path1:^^=^%"
if not exist "%path1%" echo -f- "%path1%" 1>&2
if "%quot%"=="1" (
@rem quotes required for & ampersands to echo
echo "%file1%"
) else (
setlocal enabledelayedexpansion
echo !file1!
endlocal
)
endlocal
goto :eof
________________________________