Getting all text differences between all files in 2 directories
knud van eeden
Scott Wagner
knud van eeden
> #man diff
Y.T. Lim
Try software automation with autoit3, which can automate both GUI and command line applications.
/Why Tea
Date: Sat, 5 Dec 2009 09:07:43 -0800
From: knud_va...@yahoo.com
Subject: [TSEPro] Getting all text differences between all files in 2 directories
To: sem...@googlegroups.com
CC: tse...@freelists.org
Find out how Use Messenger in your Hotmail inbox
knud van eeden
> Try software automation with autoit3, which can automate both GUI and command line applications.
> /Why Tea
It is my #1 used tool already for many years, and I am continuously using it.
Why Tea
I'm *definitely* no expert in anything. You have used many different
languages, that shows your wide range of experience in software.
It's fine to use Tse as your "central commander" if you can get it to
do everything you want. I believe have managed to do just that. This
reminds me of emacs - someone once said, "It does everything except
making you a cup of coffee." I think emacs can even browse Web pages
in text mode with a macro. I believe Tse can also do the same (feed
the macro a URL, use cURL or wget to retrieve the pages, do a bit of
intelligent filtering then display the contents). But... there must be
a very good reason to have all those standalone browsers around.
I must admit I used to think the same way in the Dos/Windows world,
i.e. Tse is the center of the universe. Many years ago, I did quite a
lot of work with serial (RS232) communications; programs like telix
was widely popular then. But none of the comm programs came with a
good text editor for its log files. Then I had a light-bulb moment,
"Why don't I put a serial comm into Qedit (or Tse Jr)?" I got some
useful help from Sammy, and managed to get it to work to some extend.
But it was never even close to Telix, so I eventually gave up the
idea. Along the same line, since I enjoyed John Socha's Norton
Commander so much I wrote a Qedit/Tse macro to implement such a file
manage. Again, it worked to some extend, but I gave up in the end.
Admittingly, I don't know the Tse macro language as well as many of
you and that could be my downfall. But I slowly accepted the fact that
Tse couldn't be everything and couldn't do everything well. Since then
I have been looking around for the best tools in each category and
kept them in my "toolbox". I'm sure you'll immediately see a problem
here, how do you remember where everything is? This is where Total
Commander comes in. With its program launching features, especially
the toolbar buttons, putting a shortcut there is a cinch. With release
7.50 and later, a button can even contain a drop-down list of buttons
- perfect for arranging program menus. Since then, I've never looked
back. Many of the applications these days also allow you to select
your own editor if there is a need for one.
I'm not trying to say it's wrong to use Tse as your commander. As far
as software is concerned, it's good as long as it does the job for
you. Most people would stick to what works for them and what they feel
comfortable with. We are spoiled with so many programs around, the
choice could become quite hard and confusing. I tend to use the
analogy of chopping a log; an axe, a saw and a chisel can all do the
job, but which tool would you choose?
From your description, the Tse macros are used as a launcher and glue
between helper applications. I have done similar things in the past in
terms of text searching. This is where I think a Tse macro is most
appropriate; you read a text file and you wonder if a text is referred
to somewhere else, you'd put the cursor there and hit a key to
starting searching (think compressedView). The grep, ctag/cscope and
Google desktop all fall into this category. Besides that, it also
makes sense to run other apps to make up for Tse shortcomings. For
example, launching another editor compliment Tse's shortcoming in
printing files. Without supporting of the Windows GUI components, it's
hard to imagine Tse being an effective program launcher.
An editor is a specialized program to do text manipulation. Tse is
among the best in what it does. The powerful macro feature makes it
almost indefinitely extensible as compared to plugins in other
editors. However, a fully-fletched scripting language can do a lot
more than Tse's macro, barring text manipulation. For example,
vbscript and autoit. The power of such tool as autoit is that it both
provides software automation features. You asked if TotalCmd had a
scripting language; no, it hasn't. But you can use autoit or
autohotkey for such tasks.
After rambling so much, I haven't actually answered the question why a
Tse macro isn't the best option to do what you want. Perhaps I could
propose an alternative and you can be the judge :)
1) Language
- vbscript or autoit (preferred)
2) Design sketch
- Popup data entry box to accept Directory1/2
- Once run, the script calls up the respective helper apps to
do real work. Store outputs in a log file, clean it up a
little, format it and then show it in an inbuilt viewer/editor
with a button option to load it into Tse
You would ask, why? You could do everything in a Tse macro. Agree. But
if you compare both development processes, you will find one to be
more flexible and better suited for the task. In addition, the script
can be published for everyone to use. No Tse? No problem, you can set
the button to load the log file into an editor of user's choice. In
addition, if you work on other platforms, the scripting approach will
work well on any platform although you will need to change the
scripting language.
That's only for the Windows world and we barely touched on Unix. What
would be the best "commander" there? A plain shell or something
else...? Or just emacs with all the Lisp macros??
/Why Tea
knud van eeden
> I'm *definitely* no expert in anything.
You give me the impression that you are ;-)
> It's fine to use Tse as your "central commander" if you can get it to
> do everything you want. I believe have managed to do just that.
Absolutely. For example the content of the below URL has been created automatically from a text record in a text page in one of my text file databases which I edit using TSE.
Using TSE macros (and Total Commander to create the ftp directories and upload).
A TSE macro converts the title of the record to an ftp directory, then Total Commander creates it. TSE save the record to a file, converts using regular expressions its content to HTML. Then that HTML is uploaded by Total Commander to that ftp site. Then TSE creates a URL to update my database there and create a tiny URL. All automatic.
> This reminds me of emacs - someone once said, "It does everything except
> making you a cup of coffee." I think emacs can even browse Web pages
> in text mode with a macro.
I downloaded and installed again latest Emacs, Vim, e-Text editor, Notepad++, SlickEdit, PSPad. UltraEdit is on my system. Still hard for me to imagine that they are going to be used much, TSE is pretty much the winner for me. Load your 500 megabytes file, and enjoy the coffee. Of course that is also a matter of habit.
I believe Tse can also do the same (feed
> the macro a URL, use cURL or wget to retrieve the pages, do a bit of
> intelligent filtering then display the contents). But... there must be
> a very good reason to have all those standalone browsers around.
You know TSE has the ability to include Microsoft Windows APIs.
So almost everything you can do in Windows you can control from TSE (except possibly COM stuff and the like).
So you have about 6000 procedures (e.g. downloading a file from the Internet) which you also
can use in your programs. I sometimes do.
Note: I will probably some day publish a list of this Microsoft Windows APIs (translated in TSE source code), as snippets and for quick insertion. I know where to find an XML list of all this APIs and its parameter, so after writing the appropriate XSL stylesheet it can be converted to the corresponding TSE code.
> I must admit I used to think the same way in the Dos/Windows world,
> i.e. Tse is the center of the universe.
Microsoft Windows is my environment of choice.
I do have to sometimes work with Linux, Solaris, HP-UX and seldom with IBM AIX, and did work with Apple Mac. That is also a matter of habit (and possibly TSE that keeps the things rolling in the Microsoft Windows environment world_.
> But none of the comm programs came with a
> good text editor for its log files.
-TSE macros are used by me to get the 'tail' of large log files to monitor that log files live
-TSE macros convert log files to XML (and via XSL to something else like HTML to show in browser).
-I plan to write a Java thread dump log analyzer in it, and have already control it all other existing Java thread dump log analyzers I could find
-TSE gives syntax highlight to log files
-Similar to Perl, Ruby, you can use the regular expressions in TSE to parse and analyze your log files.
> Admittingly, I don't know the Tse macro language as well as many of
> you and that could be my downfall.
So that is still a little bit a difference at this moment. I have put an enormous amount of time in creating somewhere between 7000 and 10000 TSE macros, and applied with my other knowledge to solve a lot of different situations.
> But I slowly accepted the fact that Tse couldn't be everything and couldn't do everything well.
Using StartPgm() and Dos() you can thus start almost any other program you want, and sometimes 'capture' the output from that program or similar. It can not do everything well, but in combination with other specialized programs you can find the right combination most of the time.
> Since then have been looking around for the best tools in each category and
> kept them in my "toolbox".
Right, big fun to look for these tools. And even better if you find the goldmine. That is a program that is really useful for what you want to do.
TSE, Paperport, 4nt, Acrobat, Filelocator Pro, BeyondCompare are a few of them.
And to control them TSE SAL, Borland Delphi, BBCBASIC for Windows, Perl, Ruby, Java e.g.
> I'm sure you'll immediately see a problem here, how do you remember where everything is?
Using a hierarchical tree you can zoom very quickly through a large amount of possibly items of choice. In TSE you can use for that the MENUs.
You have main MENUs, then divide it further in subMENUs, then subsubMENUs and so on.
If you have a good classification notation for your menus you can thus quickly get to where you want to start or do.
So as I said earlier I have about 1000 MENUs (10 MENUBARS, with 1000 MENU and sub...subMENUS which call between 7000 and 10000 TSE macros). Thus a hierarchical structure to divide it.
> This is where Total Commander comes in. With its program launching features, especially
> the toolbar buttons, putting a shortcut there is a cinch. With release
> 7.50 and later, a button can even contain a drop-down list of buttons
> - perfect for arranging program menus. Since then, I've never looked
> back.
I find in general the menu of Total Commander not following the Microsoft Windows rules.
E.g. typically the 'help' button is completely on the far right, while the other buttons are grouped together on the right.
It has such a wood of options, that you can easy ignore and not find what you want.
Of course all nice to have. But I am a typical 80/20 rule user. I use what I need, and ignore the rest. That is certainly the case with Total Commander.
> Many of the applications these days also allow you to select
> your own editor if there is a need for one.
Yes, that is a nice thing of e.g. Total Commander where you can choose TSE as your editor.
E.g. also 'View Source' in Microsoft Internet Explorer opens TSE here (tip from Carlo).
> I'm not trying to say it's wrong to use Tse as your commander. As far
> as software is concerned, it's good as long as it does the job for
> you. Most people would stick to what works for them and what they feel
> comfortable with.
Right.
> We are spoiled with so many programs around, the
> choice could become quite hard and confusing. I tend to use the
> analogy of chopping a log; an axe, a saw and a chisel can all do the
> job, but which tool would you choose?
I have created earlier a control center in Delphi, using thousands of buttons divided by TABs and subTABs. But not using that anymore at the moment. I am working the whole day with a text editor. And if I need a new menu entry to automate a new task, I can create and compile it (also almost automatically) from my text editor without leaving it or starting another program.
TSE is also a 'simple' system, no frills, and I like minimalistic design. As simple as possible, but not simpler than that.
> From your description, the Tse macros are used as a launcher and glue
> between helper applications.
TSE macros have like almost any programming languages that option, but of course many other types of applications (macros) can be created. There are about 200 to 300 commands in TSE SAL. Most of them thus for text editing purposes, but e.g. using Microsoft Windows APIs you have multiple more possibilities.
> I have done similar things in the past in
> terms of text searching. This is where I think a Tse macro is most
> appropriate; you read a text file and you wonder if a text is referred
> to somewhere else, you'd put the cursor there and hit a key to
>?starting searching (think compressedView). The grep, ctag/cscope and
> Google desktop all fall into this category.
I input 1 query string (e.g. searching for the word 'Proxy').
That first searches TSE text, then copies the query string to the clipboard or passes it as parameter or URL to other search applications (e.g. Google desktop, Google, Bing, ..., about 70 different search engines and knowledge bases (of course you are not going toe check them all every time). Using Everything it searches the filenames on your harddisk, and Grep to find the text. Pretty much sources thus are queried thus at once or after each other automatically with only one query string to input by you.
> Besides that, it also
> makes sense to run other apps to make up for Tse shortcomings. For
> example, launching another editor compliment Tse's shortcoming in
> printing files.
Using FinePrint can be of help maybe here. That is what I successfully use as printer manager.
E.g. you send one document to Fineprint and can then choose from a menu to print it to which printer (e.g. PDF, or (one of the) real printers)
> Without supporting of the Windows GUI components, it's
> hard to imagine Tse being an effective program launcher.
What do you mean by that? TSE can thus as far as I know start almost any other application. You should not have need for GUI possibilities as far as I know.
> An editor is a specialized program to do text manipulation. Tse is
> among the best in what it does. The powerful macro feature makes it
> almost indefinitely extensible as compared to plugins in other
> editors. However, a fully-fletched scripting language can do a lot
> more than Tse's macro, barring text manipulation. For example,
> vbscript and autoit. The power of such tool as autoit is that it both
> provides software automation features. You asked if TotalCmd had a
> scripting language; no, it hasn't. But you can use autoit or
> autohotkey for such tasks.
You can combine TSE with e.g. Perl, Ruby, Python, Cygwin and capture the output results.
See e.g. the 'capture.s' macro. As usual you take some input from TSE, write it as file to disk,
possibly compile it by passing command line parameters, then call your external program (using the filename as parameter), do someting, and via pipe or print to STDOUT, then to file, you get the output back into TSE, A general principle which you can use to run almost any programming language (also WSH, VBScript, AutoHotkey, AutoIt, batch, ..., you name it).
A text editor like e-Text editor has applied this principle more out of the box. I bought it, but it lacks a programming language like TSE also has. SlickEdit has also a programming language, and is actually written itself in it. But it is a very complex environment, can run on 7 platforms and handle something like 50 languages. But in the Microsoft Windows world, looking for being extremely fast, configurable, programmable, stable, then not much comes even close of TSE.
> After rambling so much, I haven't actually answered the question why a
> Tse macro isn't the best option to do what you want. Perhaps I could
> propose an alternative and you can be the judge :)
1) Language
- vbscript or autoit (preferred)
I think with my current knowledge that this automators like AutoIt are not really a solution here, Because what they do is store a series of steps and movements.
But something like searching an undetermined amount of subdirectories for a file, is probably not easy to program using storing of movements, mouse clicks and clicking GUI components.
You might need a program for that. And TSE did provide that.
> 2) Design sketch
> - Popup data entry box to accept Directory1/2
> - Once run, the script calls up the respective helper apps to
> do real work. Store outputs in a log file, clean it up a
> little, format it and then show it in an inbuilt viewer/editor
> with a button option to load it into Tse
That is basically what I have programmed using a TSE recursive directory search macro, combined with WinDiff, WinMerge, KDiff3, BeyondCompare, XML Lint.
That works.
> You would ask, why? You could do everything in a Tse macro. Agree. But
> if you compare both development processes, you will find one to be
> more flexible and better suited for the task. In addition, the script
> can be published for everyone to use. No Tse? No problem, you can set
> the button to load the log file into an editor of user's choice. In
> addition, if you work on other platforms, the scripting approach will
> work well on any platform although you will need to change the
> scripting language.
You must sometimes draw a line because of e.g. time management. Than it becomes a solution in your editor of choice only. Usually you stick to one solution.
> That's only for the Windows world and we barely touched on Unix. What
> would be the best "commander" there? A plain shell or something
> else...? Or just emacs with all the Lisp macros??
I do not go over too much anymore to the world of e.g. Linux, Unix, ... Been there, but matter of habit, amount of programs running going back to Windows as it does for me what it has to do. If I do actions there it goes via the command line. And that is basically just a repetition of the actions I can do via TSE, 4nt with its aliases in Windows. (4nt aliases by the way are also powerful. I use only aliases instead of real filenames and real commands (and manage and create them automatically via TSE and store it in a text file). Then I call via TSE the 4nt program with the alias file as the parameter, together with the alias command. So you only have to change the (configuration) information in one place. That is that where the alias stands for) and your TSE macros work unchanged, because you change the ALIAS variables outside TSE. E.g. job versus home versus some other place, same TSE macro, same aliases. different parameters. DRY.
Why Tea
how much time you have spent on each before declaring it to be not a
worthy contender to Tse.
1) Do you throw it away if it doesn't have a macro/script/plugin
capability?
2) If it does have a macro/script/plugin capability, have you tried to
port
some of your can't-live-without Tse macros across?
3) Do you consider the user interface too?
4) Have you really been *forced* to learn another editor well on a
different
platform? (Sure, you can write a macro to ftp/sftp the file over
and edit
it with Tse. In fact, Pspad and R J TextEd both have ftp client
built-in.)
I hope more people will answer the questions above. It'll benefit
everyone if you can tabulate your personal experience with all the
editors you listed.
There are plenty of good (free) program launchers around. I'm quite
sure you'll find one that suits your needs rather than rolling your
own.
I saw that you didn't have Tcl/Tk/Expect in your scripting language
repertoire. Tcl is unique in many ways (I know, many have complained
about its choice of an unusual language syntax) and is clever in
design. It encompasses CLI (command line interface), GUI and
automation. It's also cross-platform although Expect doesn't work too
well under Windows because Windows never implements stdio/stderr well.
From a purist's point of view, I feel uneasy just to hear Tse macro
being compared to a fully fletched scripting language. Although they
don't have to be mutually exclusive, they were designed for different
purposes. Most of the (Unix) scripting languages have their roots in
awk/sed when it comes to text manipulation. Being a memory based text
editor, the entire file must be loaded before a macro can work on it.
Piping into Tse is OK, but is there a way to pipe Tse output to other
apps? Perhaps through an intermediate file? Sorry, I'm just being a
devil's advocate here :)
/Why Tea
>
> read more »
knud van eeden
>You have given an impressed list of text editors. I'm curious to know
> how much time you have spent on each before declaring it to be not a
> worthy contender to Tse.
Maybe 1/1000th of the time for each or less. So not really any expert in that regard.
You check and are catched or not.
Going through the same experience time as TSE is simply not possible.
I having been using TSE on DOS, OS/2, Microsoft Windows versions 95,98,ME,2000,XP,Vista,... over the years.
And as long as I can remember I had it always with me, first on 1 floppy disk, and now on a USB.
That small footprint and portability of TSE and easy use without having really to install it on a foreign system
is also one of its great qualities. You could just run q.exe, e.exe, and later g32.exe, that executable is minimally
all you need and off you went.
1) Do you throw it away if it doesn't have a macro/script/plugin capability?
Basically yes, I am not spending time in it. E.g. e-Text editor is a nice concept. The ability to automatically download all
latest Cygwin programs at startup. TextMate on Apple Mac is its big example. But it is much much more complex to accomplish
e.g. running of (e.g. Perl, Ruby, Python, ...) scripts from there, not so intuitive. TSE is just a breeze, and the simplest possible.
Nice and intuitive.
I might check out SlickEdit as stated before. Chris Antos who made great macros in TSE is very quiet now, so probably has moved
away from using TSE as his standard editor, and works probably with SlickEdit, you can also program (Rexx like macros, because the origin is at IBM). You can e.g. integrate it with Visual Studio and it has possibly all kind of template expansion stuff built in similar to Visual Studio. But also considerable more complexity and much larger footprint as a result.
2) If it does have a macro/script/plugin capability, have you tried to port some of your can't-live-without Tse macros across?
No, not really.
3) Do you consider the user interface too?
Not really, being configurable and programmable would come first. E.g. Microsoft Word which should have a more developed GUI interface I have on most of my systems, but almost never use it, only if you have to.
4) Have you really been *forced* to learn another editor well on a different platform? (Sure, you can write a macro to ftp/sftp the file over
and edit it with Tse. In fact, Pspad and R J TextEd both have ftp client built-in.)
E.g. PSPad and UltraEdit are standard on the systems I have to work with. But I almost never use them. Always TSE.
On my own system most of the important editors are present. I can start them all via 4nt aliases from a TSE menu in an easy way.
But usage of that programs is low or not existing.
> There are plenty of good (free) program launchers around. I'm quite sure you'll find one that suits your needs rather than rolling your own.
If you can do it yourself you have usually more control in the design.
> I saw that you didn't have Tcl/Tk/Expect in your scripting language repertoire. Tcl is unique in many ways (I know, many have complained
> about its choice of an unusual language syntax) and is clever in design. It encompasses CLI (command line interface), GUI and
> automation. It's also cross-platform although Expect doesn't work too well under Windows because Windows never implements stdio/stderr > well.
Perl has as far as I know incorporated much of the possibilities of Tcl/Tk. And having to learn or use a 'sub'language was not
seen as too relevant until now. Matter of choice and following the 'mainstream' thus maybe.
> From a purist's point of view, I feel uneasy just to hear Tse macro
> being compared to a fully fletched scripting language. Although they
> don't have to be mutually exclusive, they were designed for different
> purposes. Most of the (Unix) scripting languages have their roots in
> awk/sed when it comes to text manipulation. Being a memory based text
> editor, the entire file must be loaded before a macro can work on it.
> Piping into Tse is OK, but is there a way to pipe Tse output to other
> apps? Perhaps through an intermediate file? Sorry, I'm just being a
> devil's advocate here :)
Yes, most script languages need a file to work with.
You pass e.g. that filename as a command line parameter.
E.g. create 'yourfilename', then call e.g.
Perl.exe yourfilename
Ruby.exe yourfilename
Python.exe yourfilename
...
that will all work if you have that program installed (download and unzip or run the installer).
It is as easy as that.
So if you can create such a file off you go.
And of course your TSE macro can create such a file.
E.g. using the TSE macro commands
SaveFile() to save a whole file in your TSE editor
SaveBlock() to save a highlighted piece of text in your TSE editor
Putting '>', '>>' and '|' after the command line you can get the output back in a filename.
E.g.
Perl.exe yourfilename >youroutputfilename
Ruby.exe yourfilename >youroutputfilename
Python.exe yourfilename >youroutputfilename
will put any output from that script languages in that 'youroutputfilename'.
Then TSE loads that filename
(e.g. using TSE macro commands like
EditFile() to load a file
InsertFile() to load a file at cursor position.
Then you have the output back in TSE and can manipulate and edit it further (manually or with other macros).
Why Tea
have used emacs or SlickEdit first and become very familiar with its
macro/scripting language, what are the chances you will get on to Tse?
I think this behavior is common in the software world - you stick with
what your are familiar with. The longer you stick with it, the more
you know about the product. The more you know about the product, the
faster you can perform your tasks. Then... there is no incentive to
spend much time on other similar products and discover new ideas. Any
comparison like that is hard to be impartial. I have seen the same
behavior in many of my colleagues whom I've tried to "sell" Tse but
without any success.
Porting your essential Tse macros across instead of "Hello World!" is
a good gauge of what the other editors can do. Different platforms
tend to have different ways of working. There are cases when certain
operations under Windows become awkward in Unux. You might find that
you don't need some of the Tse macros on a different platform.
Thanks for the quick tutorial on piping and output redirections. They
are less well known in the Windows world, but are fundamental in the
Unix CLI world.
/Why Tea
knud van eeden
Good idea. Thank you for that. I will implement some hello world algorithms in Emacs and SlickEdit to get a feeling
about it.
Thanks also very much for mentioning again Tcl/Tk. I will add it to my list of interesting script languages, similar to Perl, Python, Ruby besides Haskell.
with friendly greetings,
Knud van Eeden
Buying TSE has been one of the important things. Similar to the Apple iPhone it gives you a lot of fun and is near perfect.
Why Tea
On Dec 10, 1:15 am, knud van eeden <knud_van_ee...@yahoo.com> wrote:
> > Hello world
>
> Good idea. Thank you for that. I will implement some hello world algorithms in Emacs and SlickEdit to get a feeling
> about it.
>
English is our 2nd or 3rd language.
> Thanks also very much for mentioning again Tcl/Tk. I will add it to my list of interesting script languages, similar to Perl, Python, Ruby besides Haskell.
>
nice software automation features. Works well on Unix, but not so well
under Windows. To do hello world in Tcl is simply: puts "Hello
World!". Not too exciting.