Intermittent problems with DOS MOVE command

[Bob Dubery]

I am currently developing some code to assist testers. This is happening on MVOn, but will need to be ported to UV once I'm done with it.

In a loop, the program calls various other subroutines, which eventually call executable binaries EG

 TheCMd = "DOS /c 'c:\OurApps\Pcl2Pdf32 ":FilePath:BatchID:' ':FilePath:PdfID:"'"

The sequence of events is (or should be)

1) Generate PCL in &HOLD&

2) Convert to PDF, with output to another folder (IE the &HOLD& entry is retained)

3) Move from one folder to another

For (3) I have a command like

DOS /C MOVE  D:\MyFolder\bob1.pdf D:\MyFolder\sub1\sub2\anotherName.pdf

This mostly works, doesn't always. If I test with the same set of data over and over again, it will always fail on the same files.

If I do this
DosCmd = "DOS /C MOVE  D:\MyFolder\bob1.pdf D:\MyFolder\sub1\sub2\anotherName.pdf" RETURNING TheStat CAPTURING Results

TheStat is ALWAYS zero, whether the file was moved or not.

In the case when the file was NOT moved, results will read
 0 file(s) moved.

The process cannot access the file because it is being used by another process.

There are no READU locks set. If I trap the command output, write it out and look for non-printable characters, there aren't any.

So it seems that some process outside of MVon (in this case) or UniVerse has the file open and thus renders it unmoveable.

Now there might be ways to handle this (I have tried a crude SLEEP 10, before the MOVE, but that didn't help) if I could see what was locking the file, or could clear what I presume is some sort of lock (I don't have supervisor rights). Just looping around, waiting for the MOVE to eventually work doesn't achieve anything.

Say there are 14 PDFs to be generated for the test, it will always be the same files that produce a problem. EG I run a test and the 2nd and 7th files (as we loop) give this error. I go and do something else for half an hour, repeat the test, and 2 and 7 fail again.

I tried changing the names of the files that I am working with, thinking that maybe some other process at some other time had locked a file with the same name, but this has made no difference. So I presume - but can't prove - that it is the program that converts the PCL to PDF. I don't have the source code for that program.

Does anybody have any ideas as to how I can determine what is preventing the file being moved or even clear the lock? Please remember that although I'm working in MVOn right now, I have to port the code to UniVerse (on Windows).

[Brian Speirs]

Well, here is a thread dealing with this issue:  https://stackoverflow.com/questions/26741191/ioexception-the-process-cannot-access-the-file-file-path-because-it-is-being

As that thread is dealing with C#, it doesn't help directly but may give you some ideas.

You could also investigate using the handle utility from the Sysinternals suite:  https://docs.microsoft.com/en-us/sysinternals/downloads/handle

That should at least reveal which process has the file open.

I also wonder whether you should try restructuring your way of running this process. It sounds like you are EXECUTE'ing a sequence of commands. You could try writing your DOS commands into a batch file, and then executing the batch file.

HTH,

Brian

[stope19]

Do you perhaps have an anti-virus scanner/process running that attempts to scan newly created files as they appear? This may account for why it works with some files and not others, if perhaps tiny files are scanned quickly and so appear ‘closed’ when you attempt the move – while larger files are maybe still ‘open’ and being scanned when the move command runs (and therefore fails)

You could test by disabling such software and repeating your test 'that always fails'...

[Ashley Chapman]

I have experienced the same issue with the OS MOVE command.  Spent a long time trying to trace the root cause of the issue.  It may well be something to do with the fact that MOVE is an internal part of CMD.EXE, and that it uses the OS provided write-caching to update NTFS.  Whatever the cause it's unpredictability was annoying.

In my particular situation, I changed the code to use "COPY /B" instead.  This worked 100% of the time.

Ashley

[Bob Dubery]

Almost certainly we have some such in place. But I have no control over it, and this is something that regular users have to use without having to ask our admin team to turn off the AV.

On Tuesday, 23 August 2022 at 07:41:55 UTC+2 stope19 wrote:

Do you perhaps have an anti-virus scanner/process running that attempts to scan newly created files as they appear? This may account for why it works with some files and not others, if perhaps tiny files are scanned quickly and so appear ‘closed’ when you attempt the move – while larger files are maybe still ‘open’ and being scanned when the move command runs (and therefore fails)

You could test by disabling such software and repeating your test 'that always fails'...

[Wols Lists]

On 23/08/2022 07:35, Bob Dubery wrote:
> Almost certainly we have some such in place. But I have no control over
> it, and this is something that regular users have to use without having
> to ask our admin team to turn off the AV.
>

If this is a database server, it should not be running AV. At least, not
something that sits there all the time.

Back in the day, AV was considered a necessity on workstations, but on
servers it was a major cause of problems and knackered performance...

Cheers,
Wol

[Bob Dubery]

On Tuesday, 23 August 2022 at 07:44:00 UTC+2 Ashley Chapman wrote:

I have experienced the same issue with the OS MOVE command.  Spent a long time trying to trace the root cause of the issue.  It may well be something to do with the fact that MOVE is an internal part of CMD.EXE, and that it uses the OS provided write-caching to update NTFS.  Whatever the cause it's unpredictability was annoying.

In my particular situation, I changed the code to use "COPY /B" instead.  This worked 100% of the time.

This has resolved half of my problem.

A MOVE is a COPY followed by a DELETE. I wanted to to get the file into the destination folder, then delete it so as stop the source folder filling up with unwanted stuff. 

The COPY does the first part every time, and because there is no delete it doesn't have to worry about locks. But now I have to delete to keep things tidy. A regular DELETE in BASIC fails. STATUS() returns a -2 wether the basic delete was succesful or not. The files that couldn't be MOVEd are now the files that can't be deleted.

So this is better because the file always reaches it's destination, but these pesky locks are still causing me issues.

[Bob Dubery]

On Tuesday, 23 August 2022 at 07:32:37 UTC+2 Brian Speirs wrote:

Well, here is a thread dealing with this issue:  https://stackoverflow.com/questions/26741191/ioexception-the-process-cannot-access-the-file-file-path-because-it-is-being

As that thread is dealing with C#, it doesn't help directly but may give you some ideas.

You could also investigate using the handle utility from the Sysinternals suite:  https://docs.microsoft.com/en-us/sysinternals/downloads/handle

That should at least reveal which process has the file open.

That's a non-starter. The guys who are held responsible for the server are not going to let some programmer install stuff because he thinks it's a good idea. I can't even decide what to install on my laptop - all those decisions get made for me. 

This is not a complaint. It's the way things are in a company with lots of confidential data, and who take the security of that data very seriously. 

I also wonder whether you should try restructuring your way of running this process. It sounds like you are EXECUTE'ing a sequence of commands. You could try writing your DOS commands into a batch file, and then executing the batch file.

This occured to me about 2am this morning (LOL). I'm going to give it a try. 

[Rex Gozar]

I've had similar problems trying to ftp a newly created file. Iit would fail complaining that the file was still open.

If this was a linux server, you would run the "sync" command a couple of times so the file was completely written to disk.

Since this is a windows server, I had to simulate "sync". I created a simple subroutine named "FSYNC", giving it the pathname. The subroutine opens the sequential file and reads all the bytes. I read 1024 bytes at a time; I don't keep them - the goal is to read all of them. This seems to have the effect of sync'ing the disk. Subsequent DOS or ftp commands have no problem with the file being open.

rex

--
You received this message because you are subscribed to
the "Pick and MultiValue Databases" group.
To post, email to: mvd...@googlegroups.com
To unsubscribe, email to: mvdbms+un...@googlegroups.com
For more options, visit http://groups.google.com/group/mvdbms
---
You received this message because you are subscribed to the Google Groups "Pick and MultiValue Databases" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mvdbms+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mvdbms/df601470-5e07-4206-a855-3266bb2a7597n%40googlegroups.com.

[George Gallen]

I had an issue with UV when I tried to execute a native NT ftp call and it would always fail,

however, when I ran it manually it always worked.

turned out when you execute it creates a new process which does not inherit the parent

access rights without giving permission for that application to access ports for ftp.

but usually when that happened, a message would show up on the console of the NT server

and that process would hang.

George

[Joe G]

We've had similar issues with sftp served files being picked up before they're ready. We started creating the file under a different extension like ".part" and then renaming the file to the final extension after the writing was completed. Since the transfer process was initiated by the remote machine and it was only looking at files with specific extensions this would guarantee that it wouldn't be picked up until the writing was finished.

I don't know if this would work on Windows but it's possible that rename would work even if your file is in use. Rename on windows file systems just changes the name in the directory folder without moving the actual file to a different part of the disk. Some articles indicate that attempting to rename a file can release any locks on it. You could test it by attempting a rename of the original file after your copy /b. Give it an extension like "del". At the end of your process scan the folder for any leftover "del" files and try deleting them again. Eventually it should work.

It's a messy workaround and there are probably better solutions. If Rex's suggestion works then I'd go with that.

To view this discussion on the web visit https://groups.google.com/d/msgid/mvdbms/CA%2Bzh1UyVQGknG9-A%3D%3DLnSKt10W%2Bvj0KxXd_suiuKFSR8fpuhCQ%40mail.gmail.com.

[Kevin Powick]

On Tuesday, August 23, 2022 at 1:32:37 AM UTC-4 Brian Speirs wrote:

 

....  You could try writing your DOS commands into a batch file, and then executing the batch file.

 

This is a solution that has worked for us for years.  We use DOS Move and Ren (rename) extensively from within dynamically created batch files.

--

Kevin Powick

[Jim Idle]

On Aug 23, 2022 at 8:00:44 PM, Rex Gozar <rkg...@juno.com> wrote:

I've had similar problems trying to ftp a newly created file. Iit would fail complaining that the file was still open.

If this was a linux server, you would run the "sync" command a couple of times so the file was completely written to disk.

No. That has nothing to do with a file being open. It is merely syncing the memory maps to logical disk. Besides, you won’t get this issue on Linux because even deleting a file leaves any process with the file open as it is. The actual file data is not deleted until all references are closed. If you imagine a mv as creating a new pointer to the same disk space, then deleting the existing one, it is the same thing - at least so long as the mv is on the same file system.

Since this is a windows server, I had to simulate "sync". I created a simple subroutine named "FSYNC", giving it the pathname. The subroutine opens the sequential file and reads all the bytes. I read 1024 bytes at a time; I don't keep them - the goal is to read all of them. This seems to have the effect of sync'ing the disk. Subsequent DOS or ftp commands have no problem with the file being open.

I cannot see any reason why doing this would have any effect. Or why it would affect subsequent opens. Other than perhaps you are just delaying subsequent operations. 

To view this discussion on the web visit https://groups.google.com/d/msgid/mvdbms/CA%2Bzh1UyVQGknG9-A%3D%3DLnSKt10W%2Bvj0KxXd_suiuKFSR8fpuhCQ%40mail.gmail.com.

[Jim Idle]

This is the correct way to obviate the bugs in UniVerse EXECUTE and, so it would seem, MVOn. It is to do with process permissions and synchronization. 

--

You received this message because you are subscribed to
the "Pick and MultiValue Databases" group.
To post, email to: mvd...@googlegroups.com
To unsubscribe, email to: mvdbms+un...@googlegroups.com
For more options, visit http://groups.google.com/group/mvdbms
---
You received this message because you are subscribed to the Google Groups "Pick and MultiValue Databases" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mvdbms+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mvdbms/3c1ea1fb-f7eb-4b7c-b64a-ba9cb370c997n%40googlegroups.com.

[Bob Dubery]

On Wednesday, 24 August 2022 at 06:32:13 UTC+2 ji...@temporal-wave.com wrote:

This is the correct way to obviate the bugs in UniVerse EXECUTE and, so it would seem, MVOn. It is to do with process permissions and synchronization. 

How would it address permissions problems? Everything done with one process, so all files have the same owner?

I ask, because I thought it might be a synchronisation problem, but I am now more inclined to think it's a permissions problem (as previously noted, I tried to address the synch problem with liberal use of SLEEP. (Just to try and prove a point, I wouldn't have left them in the code and burdened the user with all that waiting).

Our system has a web browser front end, so all the basic programs are run as phantoms, with all phantoms being run as with the same user name and permissions. It is possible to log in to the system as that user IF you know the password, but the password (rightly) is distributed on a need to know basis - and I don't need to know.

I wrote a test program that can be run from the command line and which does the same operations in the same sequence. If I run via the browser interface, the move always fails on the same record. If I wait 30 minutes and run the job again, it will fail in the same place. If I log in as me using a telnet client and run it from the command line, it always works.

This is the reverse of the usual situation on our system, whereby the phantoms have more access than I do, but my point is that the problem seems to be moving with the user. But even this is odd, if the problem were caused by permissions surely ALL moves would fail or all succeed?

[Jim Idle]

I would guess that your phantoms are running in such a way that they never release the process they use to run your command. Either bugs in your phantom code, or bug in the MV system. Execute a batch file as Kevin says is probably the only way.

--

You received this message because you are subscribed to
the "Pick and MultiValue Databases" group.
To post, email to: mvd...@googlegroups.com
To unsubscribe, email to: mvdbms+un...@googlegroups.com
For more options, visit http://groups.google.com/group/mvdbms
---
You received this message because you are subscribed to the Google Groups "Pick and MultiValue Databases" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mvdbms+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mvdbms/29225c69-cd02-4d42-922e-5dbee858212cn%40googlegroups.com.