remove duplicates

[Gabor Grothendieck]

I have a string that I wish to prepend to my path, say:

set x=C:\A;C:\A;C:\B;C:\A;C:\B;C:\B;C:\A

Typically x has about 6 components but only 2 unique ones. I do not know ahead of time what the components are. I would like to remove all but the first occurrence of any component. We can assume that duplicate components appear in the exact same form as the non-duplicates, (e.g. we would never have both C:\X\Y and C:/X/Y). In the above example variable x's value is the input and the result would be:

C:\A;C:\B

Some desirable criteria include:
- not to use any software that does not come with Windows
- run quickly as its done on the fly each time I run an associated program so it could affect the start up time if it were too slow
- I would prefer not to write any files in doing this

I guess I could just ignore this issue and just let there be duplicates on the path (and in fact that is what I am currently doing) but if there is a nice solution to this it would make the resulting PATH a bit cleaner.

[Herbert Kleebauer]

On 20.04.2013 19:01, Gabor Grothendieck wrote:
> I have a string that I wish to prepend to my path, say:
>
> set x=C:\A;C:\A;C:\B;C:\A;C:\B;C:\B;C:\A
>
> Typically x has about 6 components but only 2 unique ones.
> I do not know ahead of time what the components are. I would
> like to remove all but the first occurrence of any component.

If there are no spaces within the path:


@echo off

set x=C:\A;C:\A;C:\B;C:\A;C:\B;C:\B;C:\A

set y=

for %%i in (%x%) do call :sub %%i

echo x=%x%
echo y=%y%
goto :eof

:sub
for %%j in (%y%) do if %%j==%1 goto :eof
if not [%y%] == [] set y=%y%;
set y=%y%%1

[Gabor Grothendieck]

Thanks. I think that could work if it first checked if x had any spaces and only performed the rest if not. Any suggestions on how to add that?

[billious]

On 21/04/2013 12:15, Gabor Grothendieck wrote:
>
> Thanks. I think that could work if it first checked if x had any spaces and only performed the rest if not. Any suggestions on how to add that?
>

@ECHO OFF
SETLOCAL
set x=C:\A;C:\A;C:\B;C:\A;C:\B;C:\B;C:\A;c:\something or
other;c:\HELLO;c:\A;c:\cc:\something or other;c:\hello;c:\A;
SET "y="
:loop
FOR /f "tokens=1*delims=;" %%i IN ("%x%") DO (
IF DEFINED y (
ECHO %y%|FINDSTR /i ";%%i;" >NUL
IF ERRORLEVEL 1 SET "y=%y%%%i;"
) ELSE (
SET "y=;%%i;"
)
SET "x=%%j"
IF DEFINED x GOTO loop
)
SET y=%y:~1,-1%
SET y

[foxidrive]

Two problems remain.

1) The directory may contain an & (or a % but that's asking for trouble)
2) The directory entry may be double quoted





--
foxi

[billious]

other;c:\HELLO;c:\A;c:\c;c:\something or other;c:\hello;c:\A;
set x=%x%;c:"something";c:\a;c:\helmet;c:\a
per%%cent%%age\orange;"C:\yellow"
SET x=%x:"=%

SET "y="
:loop
FOR /f "tokens=1*delims=;" %%i IN ("%x%") DO (
IF DEFINED y (

ECHO %y%|FINDSTR /i /c:";%%i;" >NUL

IF ERRORLEVEL 1 SET "y=%y%%%i;"
) ELSE (
SET "y=;%%i;"
)
SET "x=%%j"
IF DEFINED x GOTO loop
)
SET y=%y:~1,-1%
SET y

OK, then. Still a problem with "&" but "%" seems happy. Quotes don't
appear to be a problem, but the object is to put this into a ;-delimited
path anyway, so can safely drop them. Need the /c: switch in the FINDSTR
though.

[foxidrive]

On 21/04/2013 11:04 PM, billious wrote:
> OK, then. Still a problem with "&" but "%" seems happy. Quotes don't
> appear to be a problem, but the object is to put this into a ;-delimited
> path anyway, so can safely drop them. Need the /c: switch in the FINDSTR
> though.

This fixes the & issue.

@ECHO OFF
SETLOCAL
set x=C:\A;C:\A;C:\B;C:\A;C:\B;C:\B;C:\A;c:\something or other;c:\HELLO;c:\A;c:\c;c:\something or
other;c:\hello;c:\A;
set x=%x%;c:"something";c:\a;c:\helmet;c:\a per%%cent%%age\orange;"C:\yellow"

set "x=%x%;c:\a & b"
SET "x=%x:&=^^^&%"

SET x=%x:"=%
SET "y="
:loop
FOR /f "tokens=1*delims=;" %%i IN ("%x%") DO (
IF DEFINED y (
ECHO %y%|FINDSTR /i /c:";%%i;" >NUL
IF ERRORLEVEL 1 SET "y=%y%%%i;"
) ELSE (
SET "y=;%%i;"
)
SET "x=%%j"
IF DEFINED x GOTO loop
)
SET y=%y:~1,-1%
SET y

--
foxi

[Gabor Grothendieck]

Thanks, billious & foxidrive.

One further item that I noticed while testing these solutions. Sometimes my set statements inadvertently had a space at the end and when concatenating several variables that gave me an input such as this:

set "x=c:\y ;c:\z;c:\y ;c:\z"

which after de-duping gets transformed to

y=c:\y ;c:\z

Note the trailing space after c:\y is preserved. Will this cause problems if I try to use it as a Windows path?

[Frank Westlake]

2013-04-20 10:01, Gabor Grothendieck:

> I have a string that I wish to prepend to my path, say:
>
> set x=C:\A;C:\A;C:\B;C:\A;C:\B;C:\B;C:\A
>
> Typically x has about 6 components but only 2 unique ones.
> I do not know ahead of time what the components are. I would
> like to remove all but the first occurrence of any component.
> We can assume that duplicate components appear in the exact
> same form as the non-duplicates, (e.g. we would never have
> both C:\X\Y and C:/X/Y). In the above example variable x's
> value is the input and the result would be:
>
> C:\A;C:\B

Here's a short subroutine which does that; it looks larger because of
all the comments. It is contained in a demonstration script. In it is
set to prefix as you specified but it can be changed to append by moving
REM to the other line where it reads "If DEFINED #".

It can be used to set the path variable so that redundancies do not
occur, or it can be called after all the additions to remove
redundancies.


:: BEGIN SCRIPT :::::::::::::::::::::::::::::::::::::::::::::::::::::
:: From the desk of Frank P. Westlake, 2013-04-21
:: Written on Windows 8.
@Echo OFF
SetLocal EnableExtensions EnableDelayedExpansion

:: EXAMPLE: Trim, don't add.
Set "X=C:\A;C:\B;C:\A;C:\B;C:\A;C:\B"
CALL :trimPath:x
Echo X=!X!

:: EXAMPLE: Trim and add bunchos more.
Set "X=C:\A;C:\B;C:\A;C:\B;C:\A;C:\B"
CALL :trimPath:x "C:\A&B" "C:\C&D" C:\E\F\G
Echo X=!X!

:: EXAMPLE: Trim and add bunchos more with forward slants.
Set "X=C:/A;C:/B;C:/A;C:/B;C:/A;C:/B"
CALL :trimPath:x "C:/A&B" "C:/C&D" C:/E/F/G
Echo X=!X!

Goto :EOF

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:trimPath:<variable to trim> [segment to add]
:: Eliminates redundant path segments from the variable and
:: optionally add new segmants.
:: Example: CALL :trimPath:PATH
:: Example: CALL :trimPath:PATH "C:\A&B"
::
:: Note that only a colon separates the subroutine name and
:: the variable name to edit.
SetLocal EnableExtensions EnableDelayedExpansion
Set "$="
For /F "tokens=2 delims=:" %%a in ("%0") Do Set "old=!%%a!"
For %%a in (!old! %*) Do (
Set "#=%%~a"
For %%b in (!new!) Do (
If /I "!#!" EQU "%%~b" (Set "#=")
)
REM Prefix:
If DEFINED # (Set "new=!#!;!new!")
REM Append:
REM If DEFINED # (Set "new=!new!;!#!")
)
EndLocal & For /F "tokens=2 delims=:" %%a in ("%0") Do Set "%%a=%new%"
Goto :EOF
:: END SCRIPT ::::::::::::::::::::::::::::::::::::::::::::::::::::

Frank

[Frank Westlake]

That version changes the order of previously existing path segments;
this should leave them the same and prefix new segments.

For %%a in (%* !old!) Do (

Set "#=%%~a"
For %%b in (!new!) Do (
If /I "!#!" EQU "%%~b" (Set "#=")
)

[Gabor Grothendieck]

On Sunday, April 21, 2013 10:45:05 AM UTC-4, Frank Westlake wrote:
> That version changes the order of previously existing path segments;
>
> this should leave them the same and prefix new segments.
>
> :: BEGIN SCRIPT :::::::::::::::::::::::::::::::::::::::::::::::::::::
>
> :: From the desk of Frank P. Westlake, 2013-04-21
>
> :: Written on Windows 8.
>

> ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
>
> :trimPath:<variable to trim> [segment to add]
> :: Eliminates redundant path segments from the variable and
> :: optionally add new segmants.
> :: Example: CALL :trimPath:PATH

This is really nice. It even removes trailing space while preserving mid string spaces. I noticed it does preface the result with ; in this case:

Set "X=C:/A;C:/B ;C:/A;C:/B ;C:/A;C:/B ;C:/C;"
set X

CALL :trimPath:x
Echo X=!X!

Result:

X=C:/A;C:/B ;C:/A;C:/B ;C:/A;C:/B ;C:/C;
X=;C:/A;C:/B;C:/C <-- note beginning semicolon

[Frank Westlake]

2013-04-21 09:02, Gabor Grothendieck:

> I noticed it does preface the result with ; in this case:

> Result:

> X=;C:/A;C:/B;C:/C <-- note beginning semicolon

Yup. I didn't look close after I made that last fix/break. Here's the
fix/fix but I only tested it with the previous demo.

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:trimPath:<variable to trim> [segment to add]
:: Eliminates redundant path segments from the variable and
:: optionally add new segmants.
:: Example: CALL :trimPath:PATH

:: Example: CALL :trimPath:PATH "C:\A&B"
::
:: Note that only a colon separates the subroutine name and
:: the variable name to edit.
SetLocal EnableExtensions EnableDelayedExpansion
Set "$="
For /F "tokens=2 delims=:" %%a in ("%0") Do Set "old=!%%a!"
For %%a in (%* !old!) Do (
Set "#=%%~a"
For %%b in (!new!) Do (
If /I "!#!" EQU "%%~b" (Set "#=")
)
If DEFINED # (

If DEFINED new (Set "new=!new!;!#!") Else ( Set "new=!#!")

)
)
EndLocal & For /F "tokens=2 delims=:" %%a in ("%0") Do Set "%%a=%new%"
Goto :EOF

Frank

[billious]

And inserting

SET "y=%y: ;=;%"

before

SET y=%y:~1,-1%

gets rid of the trailing-spaces-before-; problem.

[frank.w...@gmail.com]

This is the same thing but cleaned up.

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:trimPath:<variable to trim> [segment to add]
:: Eliminates redundant path segments from the variable and

:: optionally adds new segmants.
:: Example: CALL :trimPath:PATH
:: Example: CALL :trimPath:PATH "C:\A & B" C:\a\b\c

::
:: Note that only a colon separates the subroutine name and

:: the name of the variable to be edited.
SetLocal EnableExtensions EnableDelayedExpansion
For /F "tokens=2 delims=:" %%a in ("%0") Do (
For %%a in (%* !%%a!) Do (
Set "#=%%~a"
For %%b in (!new!) Do If /I "!#!" EQU "%%~b" Set "#="

[FileGod]

You mean you want to remove duplicate lines of text right? here is that DelDupeText.cmd I wrote a few years ago:

::DelDupeText.cmd
@echo off
for /f "tokens=* delims= " %%a in (Old.txt) do (
find /i "%%a" New.txt
if errorlevel 1 echo %%a>>New.txt
)

[mokomoji]

2013년 4월 21일 일요일 오전 2시 1분 52초 UTC+9, Gabor Grothendieck 님의 말:

@echo off
setlocal

set x=C:\A;C:\A;C:\B;C:\A;C:\B;C:\B;C:\A

for %%i in (%x%) do (call echo %%x%%|find /i /c "%%i" 2>nul>nul&&call set x=%%x:%%i=%%&&call set z_v=%%z_v%%%%i)
echo %z_v%
:end
pause
endlocal

one line................

[mokomoji]

2013년 4월 21일 일요일 오전 2시 1분 52초 UTC+9, Gabor Grothendieck 님의 말:

path

@echo off
setlocal
set "x=%path%"
:main
for /f "delims=;" %%f in ("%x%") do (set "v=%%~f")
echo %x%|find /i /c "%v%" 2>nul>nul&&call set x=%%x:%v%;=%%&&call set "z_f=%%z_f%%%%v%%;"
if "%v%" equ "%x%" goto :pr
goto :main
:pr
echo "%z_f%"
:end
pause
endlocal

[Herbert Kleebauer]

On 21.01.2020 17:26, mokomoji wrote:
> 2013년 4월 21일 일요일 오전 2시 1분 52초 UTC+9, Gabor Grothendieck 님의 말:

I don't think anybody is interested in a solution for a 10 year old
problem, but just for fun:

>> I have a string that I wish to prepend to my path, say:
>>
>> set x=C:\A;C:\A;C:\B;C:\A;C:\B;C:\B;C:\A

>> In the above example variable x's value is the input and the result would be:
>>
>> C:\A;C:\B

> @echo off
> setlocal
> set "x=%path%"
> :main
> for /f "delims=;" %%f in ("%x%") do (set "v=%%~f")
> echo %x%|find /i /c "%v%" 2>nul>nul&&call set x=%%x:%v%;=%%&&call set "z_f=%%z_f%%%%v%%;"
> if "%v%" equ "%x%" goto :pr
> goto :main
> :pr
> echo "%z_f%"
> :end
> pause
> endlocal

A simpler solution without the external find program:

set x=C:\A;C:\A;C:\B;C:\A;C:\B;C:\B;C:\A

set y=;
set z=
for %%i in (%x%) do call set y=%%i;%%y:%%i=%%
for %%i in (%y%) do call set z=%%i;%%z%%
echo %y%
echo %z%

[mokomoji]

2020년 1월 22일 수요일 오전 2시 30분 28초 UTC+9, Herbert Kleebauer 님의 말:

oh~! oh~! good...

@echo off

set x=C:\A;C:\A;C:\B;C:\A;C:\B;C:\B;C:\A
set y=;
set z=
for %%i in (%x%) do call set y=%%i;%%y:%%i=%%

for %%i in (%y%) do call set z=%%z%%%%i;
echo %z%
pause

oh god..~!!