Inheriting and replacing the default VFS

[Nikolaus Rath]

Hello,

Is it possible to define a new VFS that inherits from, but also replaces the default file system?

In other words, if I instantiate VFS(makedefault=True, base=""), will super() correctly defer to the "old" default?

Thanks!
-Nikolaus

[Roger Binns]

Yes that will work fine. The base is looked up at the time of the call and the pointer retained for all future operations. Then becoming the default is done at the end of the call. You will not end up referring to yourself.

Roger

[Nikolaus Rath]

Hi,

I'm struggling to get this to work. I've created the following example which I had expected to print lots of xRead/xWrite messages - but I'm getting nothing:

#!/usr/bin/env python3

import random
import apsw

initsql = (
           'PRAGMA synchronous = OFF',
           'PRAGMA journal_mode = OFF',
           'PRAGMA foreign_keys = OFF',
           'PRAGMA locking_mode = EXCLUSIVE',
           'PRAGMA recursize_triggers = on',
           'PRAGMA page_size = 4096',
           'PRAGMA temp_store = FILE',
           'PRAGMA legacy_file_format = off',
           )

class VFS(apsw.VFS):
    def __init__(self):
        super().__init__(name='WriteTracker', base='', makedefault=True)

    def xOpen(self, name: str, flags: list[int]):
        print(f'xOpen(name)')
        return VFSFile(name, flags)

class VFSFile(apsw.VFSFile):
    def __init__(self, fname: str, flags: list[int]) -> None:
        super().__init__(vfs='', filename=fname, flags=flags)

    def xRead(size: int, off: int):
        print(f"read(len={size}, off={off})")
        return super().xRead(size, off)

    def xWrite(buf: bytes, off: int):
        print(f"write(len={len(buf)}, off={off})")
        return super().xWrite(buf, off)


def populate(cur):
    cur.execute('CREATE TABLE foo (id INT PRIMARY KEY, word VARCHAR);')
    cur.execute('CREATE INDEX foo_idx ON foo(word);')
    with open('/usr/share/dict/words', 'r') as fh:
        for (i, s) in enumerate(fh):
            cur.execute('INSERT INTO foo VALUES(?,?)', (i,s))


def modify(cur):
    for i in range(10):
        idx = random.randint(0, 102773)
        print(f'Modifying row {idx}...')
        cur.execute('UPDATE foo SET word=? WHERE id=?',
                    ('replaced-by-%d' % idx, idx))

file_ = 'test.db'
conn = apsw.Connection(file_)
cur = conn.cursor()
for s in initsql:
    cur.execute(s)

populate(cur)

vfs = VFS() # should call sqlite3_vfs_register()

modify(cur)

What am I doing wrong?

Best,

Nikolaus

[Nikolaus Rath]

FWIW:

> python3 -c 'import apsw; print(apsw.apswversion())'
3.34.0-r1

[Roger Binns]

On Sun, 2 Oct 2022, at 04:51, Nikolaus Rath wrote:
>> I'm struggling to get this to work.

It does work, but one detail is important!

>> conn = apsw.Connection(file_)

The vfs is looked up at the time the Connection is made, so this uses the existing vfs.

>> populate(cur)
>> vfs = VFS() # should call sqlite3_vfs_register()
>> modify(cur)

This still operates on the existing opened connection. Changing VFS registration only affects new connections. Changed to:

populate(cur)
vfs = VFS() # should call sqlite3_vfs_register()

conn = apsw.Connection(file_) # reopen with new vfs
cur = conn.cursor() # and new cursor on new vfs
modify(cur)

Which would work except for this:

class VFSFile(apsw.VFSFile):
def __init__(self, fname: str, flags: list[int]) -> None:
super().__init__(vfs='', filename=fname, flags=flags)

At the point the super().__init__ is called, your VFS is now the default, so the super call calls itself, which calls itself, which calls itself ...

I'd recommend you call apsw.vfsnames() at the very beginning which will tell you the default name and that way you can always reference the one intended.

Roger

[Nikolaus Rath]

On Oct 02 2022, "Roger Binns" <rog...@rogerbinns.com> wrote:
> On Sun, 2 Oct 2022, at 04:51, Nikolaus Rath wrote:
>>> I'm struggling to get this to work.
>
> It does work, but one detail is important!
>
>>> conn = apsw.Connection(file_)
>
> The vfs is looked up at the time the Connection is made, so this uses the existing vfs.
>
>>> populate(cur)
>>> vfs = VFS() # should call sqlite3_vfs_register()
>>> modify(cur)
>
> This still operates on the existing opened connection. Changing VFS registration only affects new connections. Changed to:
>
> populate(cur)
> vfs = VFS() # should call sqlite3_vfs_register()
> conn = apsw.Connection(file_) # reopen with new vfs
> cur = conn.cursor() # and new cursor on new vfs
> modify(cur)
>

Ah, thanks, that makes sense!

> Which would work except for this:
>
> class VFSFile(apsw.VFSFile):
> def __init__(self, fname: str, flags: list[int]) -> None:
> super().__init__(vfs='', filename=fname, flags=flags)
>
> At the point the super().__init__ is called, your VFS is now the
> default, so the super call calls itself, which calls itself, which
> calls itself ...
>
> I'd recommend you call apsw.vfsnames() at the very beginning which
> will tell you the default name and that way you can always reference
> the one intended.

I'm not sure I follow. Do you mean something like this?

default_vfs = apsw.vfsnames()[0]

class VFSFile(apsw.VFSFile):
def __init__(self, fname: str, flags: list[int]) -> None:

super().__init__(vfs=default_vfs, filename=fname, flags=flags)


Best,
-Nikolaus

--
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

[Roger Binns]

On Sun, 2 Oct 2022, at 13:10, Nikolaus Rath wrote:
> I'm not sure I follow. Do you mean something like this?
>
> default_vfs = apsw.vfsnames()[0]
> class VFSFile(apsw.VFSFile):
> def __init__(self, fname: str, flags: list[int]) -> None:
> super().__init__(vfs=default_vfs, filename=fname, flags=flags)

Yes that will work.

Roger