Locking file while reading/emptying it

[JiiPee]

Two processes are using the file to transfer data. So it needs to be
locked when one is using it. I need to first read its content and then
delete its content. But how to lock it during these 2 operations? I know
how to read file and then delete but in between there is no locking and
the other process could mess up there things.

Currently:
1. Read files content - is locked
2. After file closed the file is not locked (tiny time) until delete
starts, so the other process could do something.
3. Delete the files content - is locked

So I would need something like this:

LOCK FILE
1. Read files content
2. Delete the files content
UNLOCK FILE

Does anybody know how to do this? (I use Windows).

[Jorgen Grahn]

How does the reader know that data becomes available? It sounds brittle.

IMHO you should consider using a real IPC mechanism, a pipe or
something: https://en.wikipedia.org/wiki/Inter-process_communication

But of course I don't know all your requirements (and I don't know Windows).

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

[JiiPee]

On 27/01/2018 06:16, Jorgen Grahn wrote:
> How does the reader know that data becomes available? It sounds brittle.

When the program opens the file, then other process cannot open it to
write (the opening fails). So it waits until its closed.
So its polling.

[JiiPee]

On 27/01/2018 06:16, Jorgen Grahn wrote:
>

> How does the reader know that data becomes available? It sounds brittle.

This javascript writing and C++ reading.

[Robert Wessel]

While OT, in native Windows you can specify file sharing in the thrid
parameter of CreateFile(), and in .NET, in the fourth parameter of
File.Open().

The sharing modes allow you to control how other programs can open the
file while you have it open. So you might require exclusive access if
your writing a file, but allow other programs read access if you're
only opening it for read. Note that some of those will be defaulted
based on the type of access requested.

If you want to use the native handle with the normal C streams, you
can convert the system handle to a descriptor with _open_ofshandle(),
and convert the descriptor to a FILE * with _fdopen(). I think you
can do the same with a C++ fstream by passing the descriptor to
fstream::attach(), but I've not done it myself.

[JiiPee]

On 27/01/2018 07:03, Robert Wessel wrote:
> The sharing modes allow you to control how other programs can open the
> file while you have it open.

But my situation is that after closed the opened file I would like it
*remain locked* as I need to then also delete the file contend after I
read its content. Because there is a tiny break after I closed the file
and before I open it again to remove to content. So in this gap the
other process could do something which is a problem.

[Robert Wessel]

Consider having the reader delete the file when he's done.

Or consider the byte range locking mechanism to implement a series of
semaphore-like objects. Open the file with shared access, but reserve
a bit of the file for the locks. LockFile(Ex native, Filestream.Lock
for .Net. At least with LockFileEx you might be able to avoid
polling.

Or just open the file shared, and have a byte for both sides, and
update those bytes as status progresses on both side. Each side then
polls the other's status byte.

Put either of those last two in a separate control file if you need it
to persist past opens and closes of the main file.

To much logic like that with shared files has always been a problem,
though, and as has been suggested a proper IPC mechanism might be a
better idea. Or you might punt the data storage to a database.

[JiiPee]

On 27/01/2018 18:37, Robert Wessel wrote:
> Consider having the reader delete the file when he's done.

Yes I read about this and googled, but my understanding is that it is
not possible to read and delete the content at the same time. Is this
true? This would be best solution for now (as I dont have time to do
other solutions). Also, the other process is jacascript, so its not
going to be so easy to do other ways....

[Robert Wessel]

Sorry, not sure where I got .Net from...

In Windows, the reader would have to close the file before issuing the
delete. But if the writer doesn't care about the file after he's
written it, and the reader knows not to process the file while he's
doing the delete process.

There's a problem if you have multiple readers. But since those are
native code, you can just use a global mutex (assuming all the readers
are on one machine) around some of the file management operations. For
example acquire the mutex before scanning the work directory looking
for a new file to read, release it when you've found and opened one
(for exclusive access). Acquire it again just before the close and
release it after the delete. That way all of the "meta" operations
are serialized. This becomes a bit more difficult if the readers are
spread over multiple machines, but then you could use a lock file to
implement the mutex.

[JiiPee]

On 27/01/2018 19:31, Robert Wessel wrote:
> There's a problem if you have multiple readers.

There is one reader and one writer

[JiiPee]

On 27/01/2018 19:31, Robert Wessel wrote:

> work directory looking
> for a new file to read, release it when you've found and opened one
> (for exclusive access).

its only one file they are using common. Also how can the
node.js/javascript check the mutex?
But the writer is not allowed to write when the file is opened or before
the file id deleted. So the release cannot happen after opening, but
should happen after the file has been read+deleted.

> Acquire it again just before the close and

but between this the writer ccould write new data which is not good.

[Robert Wessel]

On Sat, 27 Jan 2018 20:59:24 +0000, JiiPee <n...@notvalid.com> wrote:

>On 27/01/2018 19:31, Robert Wessel wrote:
>> work directory looking
>> for a new file to read, release it when you've found and opened one
>> (for exclusive access).
>
>its only one file they are using common. Also how can the
>node.js/javascript check the mutex?

By itself it can't. As far as I'm aware, there's not a standard way
to call native code from JS, but many implementations have an
extension that allows that. Those may have installation and security
issues that you might not care to deal with, and, of course, they're
all different.

>But the writer is not allowed to write when the file is opened or before
>the file id deleted. So the release cannot happen after opening, but
>should happen after the file has been read+deleted.
>
>> Acquire it again just before the close and
>
>but between this the writer ccould write new data which is not good.

Never write to a file a second time. The writer creates a file,
writes to it, closes it. The reader scans the directory for matching
files, if it finds one, opens it and processes it and then deletes it.
When the writer has more to write, it creates another file (different
name), and the process repeats. If you need the reader to process the
data in the order the writer created it, include a timestamp or
counter in the filename and have the reader select the oldest.

[JiiPee]

On 27/01/2018 21:17, Robert Wessel wrote:
> Never write to a file a second time.

Oh ok, so you are suggesting not use one shared file but rather use
multiple files. Ok I ll think about this, sure did not think about this

[Vir Campestris]

Alternatively, don't delete it, just delete the contents.

Andy

[JiiPee]

Its not possible to delete the content after reading before closing that
file, thats the problem. I googled about it. And if I close the file
then the other program can wrongly add something there before I empty
the file.

[Joost]

So basically you want something like this?
open the file in read/write mode with a lock,
read the contents,
use ftruncate (or SetFilePointer + SetEndOfFile for Win32),
close the file.

[Öö Tiib]

On Saturday, 27 January 2018 08:29:30 UTC+2, JiiPee wrote:
> On 27/01/2018 06:16, Jorgen Grahn wrote:
> >
> > How does the reader know that data becomes available? It sounds brittle.
>
> This javascript writing and C++ reading.

Your design using hard drive files for communication
between processes feels inefficient hack.

Make named pipes on Windows for communication between
programs. AFAIK node.js supports those.

[JiiPee]

Ok, thanks... just read. I ll try this.

[Chris M. Thomasson]

Using memory mapped files for "fast" inter-process communication can be fun.

;^)

[Jorgen Grahn]

I don't know what the smiley implies, but IME memory-mapped files for
IPC is easy to get wrong. It's not my first choice.

[Chris M. Thomasson]

On 1/29/2018 10:36 PM, Jorgen Grahn wrote:
> On Mon, 2018-01-29, Chris M. Thomasson wrote:
>> On 1/29/2018 6:25 AM, Öö Tiib wrote:
>>> On Saturday, 27 January 2018 08:29:30 UTC+2, JiiPee wrote:
>>>> On 27/01/2018 06:16, Jorgen Grahn wrote:
>>>>>
>>>>> How does the reader know that data becomes available? It sounds brittle.
>>>>
>>>> This javascript writing and C++ reading.
>>>
>>> Your design using hard drive files for communication
>>> between processes feels inefficient hack.
>>>
>>> Make named pipes on Windows for communication between
>>> programs. AFAIK node.js supports those.
>>>
>>
>> Using memory mapped files for "fast" inter-process communication can be fun.
>>
>> ;^)
>
> I don't know what the smiley implies

It implies hand crafted atomic data-structures to get some things correct.

> but IME memory-mapped files for
> IPC is easy to get wrong. It's not my first choice.

Hard to disagree with that!

[Chris M. Thomasson]

Fwiw, gotta love WAIT_ABANDONED for windows:

https://msdn.microsoft.com/en-us/library/windows/desktop/ms687032(v=vs.85).aspx

or EOWNERDEAD wrt POSIX.

[Jorgen Grahn]

On Tue, 2018-01-30, Chris M. Thomasson wrote:
...
> or EOWNERDEAD wrt POSIX.

Sounds like the excuse for putting a dog to sleep. I still prefer EIEIO.

[red floyd]

On 1/30/2018 11:37 AM, Jorgen Grahn wrote:
> On Tue, 2018-01-30, Chris M. Thomasson wrote:
> ...
>> or EOWNERDEAD wrt POSIX.
>
> Sounds like the excuse for putting a dog to sleep. I still prefer EIEIO.
>

Isn't that a PIC-32 instruction?

[Chris M. Thomasson]

A PowerPC instruction eieio?

https://en.wikipedia.org/wiki/Enforce_In-order_Execution_of_I/O

[Jorgen Grahn]

On Tue, 2018-01-30, Chris M. Thomasson wrote:
> On 1/30/2018 12:29 PM, red floyd wrote:
>> On 1/30/2018 11:37 AM, Jorgen Grahn wrote:
>>> On Tue, 2018-01-30, Chris M. Thomasson wrote:
>>> ...
>>>> or EOWNERDEAD wrt POSIX.
>>>
>>> Sounds like the excuse for putting a dog to sleep.  I still prefer EIEIO.
>>>
>>
>> Isn't that a PIC-32 instruction?
>
> A PowerPC instruction eieio?
>
> https://en.wikipedia.org/wiki/Enforce_In-order_Execution_of_I/O

Ah, that's right; I confused them.

There used to be a humorous list of fake errno codes floating around;
/that/ was what EOWNERDEAD reminded me of.

I thought I had a copy, but cannot find it right now. Unless it was
this one from USENIX 86 -- but these aren't as funny as I remembered
them.

https://www.gnu.org/fun/jokes/errno.2

[red floyd]

Thanks. It's been 10 years. It *was* PPC, not PIC.