A tacl macro to copy just a particular number of files from a location

[Shiva]

Like it is mentioned in the subject, I'd like to write a tacl macro which will copy a particular set of files from a location (a common sub volume, yes) - where all the file names are listed in a master file, and paste these particular set of files in another location (again, a common sub volume).

I tried to imitate Keith's style from a different TACL macro and have written this (I know nothing about TACL programming though, as of now) and I'm sure it is pretty wrong, but still - whether any changes to this will work, or do we need an altogether different one? Could anyone provide any assistance on this?

?tacl macro
#frame
[#push file files]
[MASTERFILE %1% files]
[#loop |while| not [#emptyv files]]
|do|
[#set file [#extract files]]
[fup dup \NODE.$VOL.SUBVOL.[file], \NOD2.$VO2.SUBVO2.[file], purge]
]

[Keith Dick]

Shiva wrote:
> Like it is mentioned in the subject, I'd like to write a tacl macro which will copy a particular set of files from a location (a common sub volume, yes) - where all the file names are listed in a master file, and paste these particular set of files in another location (again, a common sub volume).
>
> I tried to imitate Keith's style from a different TACL macro and have written this (I know nothing about TACL programming though, as of now) and I'm sure it is pretty wrong, but still - whether any changes to this will work, or do we need an altogether different one? Could anyone provide any assistance on this?
>
> ?tacl macro
> #frame
> [#push file files]

> [MASTERFILE %1% files] <------------------ I don't know what this line is intended to do

> [#loop |while| not [#emptyv files]]
> |do|
> [#set file [#extract files]]
> [fup dup \NODE.$VOL.SUBVOL.[file], \NOD2.$VO2.SUBVO2.[file], purge]
> ]

Except for the line I point out, this TACL has the right structure. The point that it does not do is get the list of files to be copied into the files variable. If the word MASTERFILE were FILETOVAR and this macro were invoked with its argument being the name of the file containing the list of files to be copied, I think it would do what you want.

[Jesús Ávila]

dear Shiva,

i would like to understand you, do you need this macro for your daily jobs? or do you want to build it to learn?

in the first case i don't think necesary to build a macro, you can use the DUPLICATE command from FUP

FUP DUP[LICATE] from-fileset-list , to-fileset

regards

[Keith Dick]

Using FUP DUP only works if the set of files you want to DUP can easily be expressed as a list of filesets. If you have a few hundred, or even a few dozen, files to duplicate, and their names don't follow some simple patterns, listing the names in a file to be read by the TACL code, as Shiva asked, probably is more practical than listing the names in a DUP command.

Of course, the approach could be to create a file that contains a set of DUP commands and give that file as input to FUP. That would take a little more typing to prepare the file, but would avoid the need for any TACL code.

[Robert Hutchings]

There is also wild-carding and SQL-like components to FUP these days. For example, one could construct a statement...

FUP DUP \NODE1.$VOL.SUBVOL.*, \NODE2.$VOL.SUBVOL, WHERE FILECODE = 101

The August 2012 FUP Manual details these newer functionalities.

[Shiva]

@Robert: Wow! That's a cool addition that "where FILECODE = 101"!

@Jesus: Keith answered your question better. I want to build to learn, it was not a necessity. I'm trying to create situations (imaginary) where such tools would be required. Thanks!

[Shiva]

@Keith: It did work. I didn't know that FILETOVAR was a keyword.And I did change the code as you suggested, and it worked. Thanks again, keith! :)

[Jesús Ávila]

I agree with Robert Hutchings, FUP has Qualified File Sets, very useful when we need to apply specific rules, so if you combine the capabilities from TACL and FUP you'll get powerfull macros.

But if Shiva wants to learn, it will be a good exercise.

[Keith Dick]

Shiva wrote:
> @Robert: Wow! That's a cool addition that "where FILECODE = 101"!

You can select on quite a lot more than just filecode, and can have quite complex expressions. qualified fileset list is the term for it.

[Tone]

You can type FUP HELP QUALEXPR to see what options there are.

[Tone]

Rather than running FUP multiple times which is inefficient you could
modify this to use inline processing.

?tacl macro
#frame
inlprefix ++
[#push file files]
[FILETOVAR %1% files]

fup/inline/

[#loop |while| not [#emptyv files] |do|
[#set file [#extract files]]

++ dup \NODE.$VOL.SUBVOL.[file], \NOD2.$VOL2.SUBVO2.[file],purge
]

++ exit

#unframe

If it encounters an error during the DUP then FUP and the macro will
fail. You can ask FUP to continue by placing the following line before
the loop.

++ allow 1000 severe errors

This way if it encountered security errors on some files it would carry
on (upto 1000 errors).

[Shiva]

@Tone: Two questions. How different is this? And How is this efficient? :) Thanks.

@everyone else: Thanks, I'll keep them ideas in mind.

[Keith Dick]

Shiva wrote:
> @Tone: Two questions. How different is this? And How is this efficient? :) Thanks.
>
> @everyone else: Thanks, I'll keep them ideas in mind.

Tone may want to add more, but the main point is that starting a process on a NonStop system is a resource-intensive operation and slow, when compared to a PC or Unix system starting a process. So when you have a choice between starting FUP once for each command, which the original approach took and starting FUP once and giving that one instance of FUP a series of commands, which is what his suggestion does, the latter usually will run faster. For a small number of commands, the difference probably doesn't matter. The more commands you are giving to the same program, the more difference it makes.

Using inline when you start a program in TACL causes TACL to give the program being started TACL's own process name as the IN file, so your TACL code can compose lines that will be sent to the program when it prompts for input. The "inlprefix ++" command tells TACL that it should send any line that begins with ++ to the current program that is using inline. You'll see the DUP command and the EXIT command are prefixed with ++.

One way it differs is that when sending each command to a separate invocation of FUP, the commands are completely independent of each other, but when using inline, the commands can interact. That's the reason Tone suggested including the allow errors command when using inline -- so FUP would not stop if one of the commands got an error.

[Tone]

I can't add anything to Keith's explanation.

[Robert Hutchings]

Yes, this is how I was "taught" to use FUP or SPOOLCOM or PATHCOM, etc. Start the program once and then feed commands to that program. Much better performance. And, you can start a FUP process in multiple processors for parallel processing:

FUP/inline, cpu 0, inv f1_in, outv f1_out, nowait, name $fup0/
FUP/inline, cpu 1, inv f2_in, outv f2_out, nowait, name $fup1/

#appendv f1_in INFO $DATA1.*.*
#appendv f2_in INFO $DATA2.*.*

sink [#wait f1_in]
#outputv f1_out
sink [#wait f2_in]
#outputv f2_out

[Keith Dick]

I'm pretty sure you don't want the "inline" in your example. I'm pretty sure that you cannot use both inline and inv in the same command. inv does something similar to inline, in that TACL will pass TACL's own process name to the process begin started. With inline, the lines that go to the process when it reads are those that start with whatever inlprefix was set to. For inv, the lines the go to the process are whatever you append to the variable specified with inv. You can have only one process using inline at any given time, but you can have many processes using inv with different TACL variables at the same time.

[Robert Hutchings]

My bad. It should be more like this:

FUP/cpu 0, inv f0_in, outv f0_out, nowait, name $fup0/
FUP/cpu 1, inv f1_in, outv f1_out, nowait, name $fup1/

#appendv f0_in INFO $DATA1.*.*
#appendv f1_in INFO $DATA2.*.*

sink [#wait f0_in]
#outputv f0_out

[Shiva]

Well explained, thank you again Keith!

Robert, you mean to say that inv produces much better performance than inline? That line of code, is different. Looks efficient, the way you explain it. I'll test it and get back to you.

[Robert Hutchings]

Shiva,

Try this. In an edit file, add this "code"...I'm not 100% sure of the syntax here, but it gives you and idea...

FUP/cpu 0, inv f0_in, outv f0_out, nowait, name $fup0/
FUP/cpu 1, inv f1_in, outv f1_out, nowait, name $fup1/

#push x
#set x 0

#appendv f0_in INFO $DATA1.*.*
#appendv f1_in INFO $DATA2.*.*

[#loop |do|
[#case [#wait f0_in F1_in]]
| f0_in | #outputv f0_out
| f1_in | #outputv f1_out
] == end case
[#compute x = x + 1]
|until|
[#compute x > 1]
] == endloop

[Robert Hutchings]

I don't know if "inv" is better or worse that "inline" as far as performance. Might be an interesting test...

[Keith Dick]

Shiva wrote:
> Well explained, thank you again Keith!
>
> Robert, you mean to say that inv produces much better performance than inline? That line of code, is different. Looks efficient, the way you explain it. I'll test it and get back to you.

It isn't what I'm guessing you are thinking.

If your TACL code is controlling a single instance of a program, inv and inline are essentially identical in what they do. You access what they do slightly differently, but the effect is basically the same -- the process you are running sees the TACL process as its IN file and the TACL process feeds it lines supplied by your TACL code. But inline works only with one program at a time. If you want your TACL code to control two or more programs at the same time, having them run in parallel, you can't use inline for that, but you can use inv for that, because each process gets its own TACL variable to use to send lines to it.

So the better performance comes from doing two things in parallel. The work gets done quicker, but you are using more resources during the time the work is being done, so it is not more efficient, just faster.

[Robert Hutchings]

Yes, precisely correct Keith. Again, the actual TACL syntax is my example(s) may not be correct, but I think they convey the ideas.

[Shiva]

Cool. I'll try them tomorrow! :)