Caching in upspinfs - how is it supposed to work?
36 views
Skip to first unread message
Filip Filmar
Apr 4, 2024, 2:06:19 AMApr 4
to Upspin
Hi folks.
Is the following proper upspinfs behavior?
I’m doing some admittedly unholy things with upspin, but perhaps you can forgive me. I noticed that the contents of a file created on one machine is not visible on another, seemingly indefinitely. But its name is visible.
Is this expected? If yes, how does one handle this situation? Is caching a no-no in this scenario?
To wit:
Machine 1:
╰─>$ cat u/f...@hdlfactory.com/cfg/f+cfg/pcc/db/db.sqlite
W/3_indexAnnotationsByFileAnnotationLocationsCREATE UNIQUE INDEX AnnotationsByFile
ON
AnnotationLocations(
Workspace,
Path,
Line
)-33tableAnnotationLocationsAnnotationLocationsCREATE TABLE AnnotationLocations (
Id INTEGER PRIMARY KEY AUTOINCREMENT,
Workspace TEXT NOT NULL,
Path TEXT NOT NULL,
Line INTEGER,
AnnId INTEGER,
FOREIGN KEY(AnnId) REFERENCES Annotations(Id)
ON DELETE CASCADE
)P++Ytablesqlite_sequencesqlite_sequenceCREATE TABLE sqlite_sequence(name,seq)##UtableAnnotationsAnnotationsCREATE TABLE Annotations (
Id INTEGER PRIMARY KEY AUTOINCREMENT,
FUSE logs say this. All expected, but no call to the store server.
2024/04/04 05:51:44.044821 FUSE <- Open [ID=0x4402 Node=0x6 Uid=89956 Gid=89939 Pid=3851612] dir=true fl=OpenReadOnly+OpenDirectory+OpenNonblock
2024/04/04 05:51:44.044920 dir/remote("upspin.hdlfactory.com:443").Glob("f...@hdlfactory.com/cfg/f+cfg/pcc/db/*")
2024/04/04 05:51:44.045265 rpc/dirserver: "f+...@hdlfactory.com": dir.Glob("f...@hdlfactory.com/cfg/f+cfg/pcc/db/*")
2024/04/04 05:51:44.045301 dir/dircache: Glob "f...@hdlfactory.com/cfg/f+cfg/pcc/db/*"
2024/04/04 05:51:44.045519 FUSE -> [ID=0x4402] Open 0x7 fl=0
2024/04/04 05:51:44.045698 FUSE <- Getattr [ID=0x4404 Node=0x6 Uid=89956 Gid=89939 Pid=3851612] 0x0 fl=0
2024/04/04 05:51:44.045724 Attr f...@hdlfactory.com/cfg/f+cfg/pcc/db 0x6 valid=1m0s ino=6 size=0 mode=drwx------ 2024-04-04 05:46:39 +0000 UTC
2024/04/04 05:51:44.045734 FUSE -> [ID=0x4404] Getattr valid=1m0s ino=6 size=0 mode=drwx------
2024/04/04 05:51:44.045869 FUSE <- Read [ID=0x4406 Node=0x6 Uid=89956 Gid=89939 Pid=3851612] 0x7 4096 @0x0 dir=true fl=0 lock=0 ffl=OpenReadOnly+OpenDirectory+OpenNonblock
2024/04/04 05:51:44.045894 FUSE -> [ID=0x4406] Read 40
2024/04/04 05:51:44.045989 FUSE <- Release [ID=0x4408 Node=0x6 Uid=0 Gid=0 Pid=0] 0x7 fl=OpenReadOnly+OpenDirectory+OpenNonblock rfl=0 owner=0x0
2024/04/04 05:51:44.046000 FUSE -> [ID=0x4408] Release
2024/04/04 05:51:44.063683 FUSE <- Open [ID=0x440a Node=0x6 Uid=89956 Gid=89939 Pid=3851612] dir=true fl=OpenReadOnly+OpenDirectory+OpenNonblock
2024/04/04 05:51:44.063721 dir/remote("upspin.hdlfactory.com:443").Glob("f...@hdlfactory.com/cfg/f+cfg/pcc/db/*")
2024/04/04 05:51:44.063888 rpc/dirserver: "f+...@hdlfactory.com": dir.Glob("f...@hdlfactory.com/cfg/f+cfg/pcc/db/*")
2024/04/04 05:51:44.063902 dir/dircache: Glob "f...@hdlfactory.com/cfg/f+cfg/pcc/db/*"
2024/04/04 05:51:44.064058 FUSE -> [ID=0x440a] Open 0x7 fl=0
2024/04/04 05:51:44.064224 FUSE <- Getattr [ID=0x440c Node=0x6 Uid=89956 Gid=89939 Pid=3851612] 0x0 fl=0
2024/04/04 05:51:44.064243 Attr f...@hdlfactory.com/cfg/f+cfg/pcc/db 0x6 valid=1m0s ino=6 size=0 mode=drwx------ 2024-04-04 05:46:39 +0000 UTC
2024/04/04 05:51:44.064283 FUSE -> [ID=0x440c] Getattr valid=1m0s ino=6 size=0 mode=drwx------
2024/04/04 05:51:44.064394 FUSE <- Read [ID=0x440e Node=0x6 Uid=89956 Gid=89939 Pid=3851612] 0x7 4096 @0x0 dir=true fl=0 lock=0 ffl=OpenReadOnly+OpenDirectory+OpenNonblock
2024/04/04 05:51:44.064412 FUSE -> [ID=0x440e] Read 40
2024/04/04 05:51:44.064508 FUSE <- Release [ID=0x4410 Node=0x6 Uid=0 Gid=0 Pid=0] 0x7 fl=OpenReadOnly+OpenDirectory+OpenNonblock rfl=0 owner=0x0
2024/04/04 05:51:44.064518 FUSE -> [ID=0x4410] Release
2024/04/04 05:51:45.687575 FUSE <- Lookup [ID=0x4412 Node=0x6 Uid=89956 Gid=89939 Pid=3851613] "db.sqlite"
2024/04/04 05:51:45.687656 Attr f...@hdlfactory.com/cfg/f+cfg/pcc/db/db.sqlite 0x29 valid=1m0s ino=41 size=20480 mode=-rwx------ 2024-04-04 05:46:39 +0000 UTC
2024/04/04 05:51:45.687677 FUSE -> [ID=0x4412] Lookup 0x7 gen=3 valid=1m0s attr={valid=1m0s ino=41 size=20480 mode=-rwx------}
2024/04/04 05:51:45.687856 FUSE <- Access [ID=0x4414 Node=0x7 Uid=89956 Gid=89939 Pid=3851613] mask=0x0
2024/04/04 05:51:45.687870 FUSE -> [ID=0x4414] Access
2024/04/04 05:51:45.691758 FUSE <- Open [ID=0x4416 Node=0x7 Uid=89956 Gid=89939 Pid=3851615] dir=false fl=OpenReadOnly
2024/04/04 05:51:45.691794 FUSE -> [ID=0x4416] Open 0x7 fl=0
2024/04/04 05:51:45.691966 FUSE <- Read [ID=0x4418 Node=0x7 Uid=89956 Gid=89939 Pid=3851615] 0x7 4096 @0x0 dir=false fl=0 lock=0 ffl=OpenReadOnly
2024/04/04 05:51:45.692057 FUSE -> [ID=0x4418] Read 4096
2024/04/04 05:51:45.692211 FUSE <- Read [ID=0x441a Node=0x7 Uid=89956 Gid=89939 Pid=3851615] 0x7 4096 @0x1000 dir=false fl=0 lock=0 ffl=OpenReadOnly
2024/04/04 05:51:45.692290 FUSE -> [ID=0x441a] Read 4096
2024/04/04 05:51:45.692454 FUSE <- Read [ID=0x441c Node=0x7 Uid=89956 Gid=89939 Pid=3851615] 0x7 4096 @0x2000 dir=false fl=0 lock=0 ffl=OpenReadOnly
2024/04/04 05:51:45.692526 FUSE -> [ID=0x441c] Read 4096
2024/04/04 05:51:45.692620 FUSE <- Read [ID=0x441e Node=0x7 Uid=89956 Gid=89939 Pid=3851615] 0x7 4096 @0x3000 dir=false fl=0 lock=0 ffl=OpenReadOnly
2024/04/04 05:51:45.692683 FUSE -> [ID=0x441e] Read 4096
2024/04/04 05:51:45.692775 FUSE <- Read [ID=0x4420 Node=0x7 Uid=89956 Gid=89939 Pid=3851615] 0x7 4096 @0x4000 dir=false fl=0 lock=0 ffl=OpenReadOnly
2024/04/04 05:51:45.692848 FUSE -> [ID=0x4420] Read 4096
2024/04/04 05:51:45.693746 FUSE <- Flush [ID=0x4422 Node=0x7 Uid=89956 Gid=89939 Pid=3851615] 0x7 fl=0x0 lk=0xbcfc996faae383cb
2024/04/04 05:51:45.693759 FUSE -> [ID=0x4422] Flush
2024/04/04 05:51:45.693935 FUSE <- Release [ID=0x4424 Node=0x7 Uid=0 Gid=0 Pid=0] 0x7 fl=OpenReadOnly rfl=0 owner=0x0
2024/04/04 05:51:45.693948 FUSE -> [ID=0x4424] Release
Machine 2:
┬─[fmil@fmil4:~]─[10:51:10 PM]
╰─>$ cat u/f...@hdlfactory.com/cfg/f+cfg/pcc/db/db.sqlite
cat: u/f...@hdlfactory.com/cfg/f+cfg/pcc/db/db.sqlite: No such file or directory
And logs confirm. Call to store server fails - blob not found.
2024/04/04 05:54:02.577737 FUSE <- Lookup [ID=0x88a34 Node=0x1 Uid=89956 Gid=89939 Pid=1938092] "f...@hdlfactory.com"
2024/04/04 05:54:02.577836 Attr f...@hdlfactory.com/ 0x2 valid=1m0s ino=2 size=5769 mode=drwx------ 2024-04-04 05:39:51 +0000 UTC
2024/04/04 05:54:02.577857 FUSE -> [ID=0x88a34] Lookup 0x2 gen=0 valid=1m0s attr={valid=1m0s ino=2 size=5769 mode=drwx------}
2024/04/04 05:54:02.577996 FUSE <- Lookup [ID=0x88a36 Node=0x2 Uid=89956 Gid=89939 Pid=1938092] "cfg"
2024/04/04 05:54:02.578019 Attr f...@hdlfactory.com/cfg 0x3 valid=1m0s ino=3 size=1459 mode=drwx------ 2024-04-04 05:40:18 +0000 UTC
2024/04/04 05:54:02.578030 FUSE -> [ID=0x88a36] Lookup 0x3 gen=0 valid=1m0s attr={valid=1m0s ino=3 size=1459 mode=drwx------}
2024/04/04 05:54:02.578179 FUSE <- Lookup [ID=0x88a38 Node=0x3 Uid=89956 Gid=89939 Pid=1938092] "f+cfg"
2024/04/04 05:54:02.578227 Attr f...@hdlfactory.com/cfg/f+cfg 0x4 valid=1m0s ino=4 size=621 mode=drwx------ 2024-04-04 05:21:51 +0000 UTC
2024/04/04 05:54:02.578242 FUSE -> [ID=0x88a38] Lookup 0x4 gen=0 valid=1m0s attr={valid=1m0s ino=4 size=621 mode=drwx------}
2024/04/04 05:54:02.578380 FUSE <- Lookup [ID=0x88a3a Node=0x4 Uid=89956 Gid=89939 Pid=1938092] "pcc"
2024/04/04 05:54:02.578403 Attr f...@hdlfactory.com/cfg/f+cfg/pcc 0x5 valid=1m0s ino=5 size=1120 mode=drwx------ 2024-04-04 05:21:50 +0000 UTC
2024/04/04 05:54:02.578413 FUSE -> [ID=0x88a3a] Lookup 0x5 gen=0 valid=1m0s attr={valid=1m0s ino=5 size=1120 mode=drwx------}
2024/04/04 05:54:02.578560 FUSE <- Lookup [ID=0x88a3c Node=0x5 Uid=89956 Gid=89939 Pid=1938092] "db"
2024/04/04 05:54:02.578586 Attr f...@hdlfactory.com/cfg/f+cfg/pcc/db 0xa valid=1m0s ino=10 size=0 mode=drwx------ 2024-04-04 05:46:39 +0000 UTC
2024/04/04 05:54:02.578596 FUSE -> [ID=0x88a3c] Lookup 0x9 gen=1 valid=1m0s attr={valid=1m0s ino=10 size=0 mode=drwx------}
2024/04/04 05:54:02.578724 FUSE <- Open [ID=0x88a3e Node=0x9 Uid=89956 Gid=89939 Pid=1938092] dir=true fl=OpenReadOnly+OpenDirectory+OpenNonblock
2024/04/04 05:54:02.578766 dir/remote("upspin.hdlfactory.com:443").Glob("f...@hdlfactory.com/cfg/f+cfg/pcc/db/*")
2024/04/04 05:54:02.579284 rpc/dirserver: "f+...@hdlfactory.com": dir.Glob("f...@hdlfactory.com/cfg/f+cfg/pcc/db/*")
2024/04/04 05:54:02.579319 dir/dircache: Glob "f...@hdlfactory.com/cfg/f+cfg/pcc/db/*"
2024/04/04 05:54:02.579566 FUSE -> [ID=0x88a3e] Open 0x6 fl=0
2024/04/04 05:54:02.579752 FUSE <- Read [ID=0x88a40 Node=0x9 Uid=89956 Gid=89939 Pid=1938092] 0x6 4096 @0x0 dir=true fl=0 lock=0 ffl=OpenReadOnly+OpenDirectory+OpenNonblock
2024/04/04 05:54:02.579779 FUSE -> [ID=0x88a40] Read 40
2024/04/04 05:54:02.579885 FUSE <- Release [ID=0x88a42 Node=0x9 Uid=0 Gid=0 Pid=0] 0x6 fl=OpenReadOnly+OpenDirectory+OpenNonblock rfl=0 owner=0x0
2024/04/04 05:54:02.579899 FUSE -> [ID=0x88a42] Release
2024/04/04 05:54:03.957217 FUSE <- Lookup [ID=0x88a44 Node=0x9 Uid=89956 Gid=89939 Pid=1938093] "db.sqlite"
2024/04/04 05:54:03.957310 Attr f...@hdlfactory.com/cfg/f+cfg/pcc/db/db.sqlite 0x18 valid=1m0s ino=24 size=20480 mode=-rwx------ 2024-04-04 05:46:39 +0000 UTC
2024/04/04 05:54:03.957335 FUSE -> [ID=0x88a44] Lookup 0xa gen=3 valid=1m0s attr={valid=1m0s ino=24 size=20480 mode=-rwx------}
2024/04/04 05:54:03.957457 FUSE <- Access [ID=0x88a46 Node=0xa Uid=89956 Gid=89939 Pid=1938093] mask=0x0
2024/04/04 05:54:03.957470 FUSE -> [ID=0x88a46] Access
2024/04/04 05:54:03.961659 FUSE <- Open [ID=0x88a48 Node=0xa Uid=89956 Gid=89939 Pid=1938095] dir=false fl=OpenReadOnly
2024/04/04 05:54:03.961707 dir/remote("upspin.hdlfactory.com:443").Lookup("f...@hdlfactory.com/cfg/f+cfg/pcc/db/db.sqlite")
2024/04/04 05:54:03.961982 rpc/dirserver: "f+...@hdlfactory.com": dir.Lookup("f...@hdlfactory.com/cfg/f+cfg/pcc/db/db.sqlite")
2024/04/04 05:54:03.962007 dir/dircache: Lookup "f...@hdlfactory.com/cfg/f+cfg/pcc/db/db.sqlite"
2024/04/04 05:54:03.962393 FUSE => InvalidateNode 0xa Off:0 Size:0
2024/04/04 05:54:03.962411 FUSE => InvalidateNode 0xa Off:0 Size:-1
2024/04/04 05:54:03.962688 store/remote("upspin.hdlfactory.com:443").Get("0703B49E8C3A816823351BA6E19B52343052A5BD40E3DD165D2315B18400FAE8")
2024/04/04 05:54:03.963104 rpc/storeserver: "f+...@hdlfactory.com": store.Get("0703B49E8C3A816823351BA6E19B52343052A5BD40E3DD165D2315B18400FAE8")
2024/04/04 05:54:03.963126 store/storecache: Get "0703B49E8C3A816823351BA6E19B52343052A5BD40E3DD165D2315B18400FAE8"
2024/04/04 05:54:03.963159 store/remote("upspin.hdlfactory.com:443").Get("0703B49E8C3A816823351BA6E19B52343052A5BD40E3DD165D2315B18400FAE8")
2024/04/04 05:54:04.424009 store/storecache: Get "0703B49E8C3A816823351BA6E19B52343052A5BD40E3DD165D2315B18400FAE8" failed: store/server.Get: item does not exist:
file does not exist
2024/04/04 05:54:04.424052 rpc/storeserver: "f+...@hdlfactory.com": store.Get("0703B49E8C3A816823351BA6E19B52343052A5BD40E3DD165D2315B18400FAE8")
2024/04/04 05:54:04.424418 Open: f...@hdlfactory.com/cfg/f+cfg/pcc/db/db.sqlite: item does not exist:
cache.open:
store/storecache.Get "0703B49E8C3A816823351BA6E19B52343052A5BD40E3DD165D2315B18400FAE8":
store/server.Get:
file does not exist
2024/04/04 05:54:04.424466 FUSE -> [ID=0x88a48] Open error=ENOENT: Open: f...@hdlfactory.com/cfg/f+cfg/pcc/db/db.sqlite: item does not exist:
cache.open:
store/storecache.Get "0703B49E8C3A816823351BA6E19B52343052A5BD40E3DD165D2315B18400FAE8":
store/server.Get:
file does not exist
David Presotto
Apr 4, 2024, 4:31:37 PMApr 4
to Filip Filmar, Upspin
oops, forgot to replyall.
On Thu, Apr 4, 2024 at 1:30 PM David Presotto <pres...@gmail.com> wrote:
Two possibilities I can think of:1) Machine 1 never managed to get the blocks written to the store at some point in the past. What I'm seeing here is it just reading them from its cache. The store cache is different from upasfs's cache by the way so you are dealing with two caches.2) Machine 2 is missing some permissions for reading blocks on the store.I would suggest testing from simplest to most complicated. First factor out upasfs on both sides. Access via the upas cmd and see what you get. Do it both with the existing file and with a new one. They try it without the store cache on one side and the other. Then upasfs without the storecache on one side or the other. It is an unfortunately too complicated system.
--
You received this message because you are subscribed to the Google Groups "Upspin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to upspin+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/upspin/CAKaOXihTv2PL4kbBAurgMLh_P%3DaNH-YVwxjagXwL9O_2-ukvEg%40mail.gmail.com.
Filip Filmar
Apr 4, 2024, 7:14:03 PMApr 4
to David Presotto, Upspin
On Thu, Apr 4, 2024 at 1:31 PM David Presotto <pres...@gmail.com> wrote:
1) Machine 1 never managed to get the blocks written to the store at some point in the past. What I'm seeing here is it just reading them from its cache. The store cache is different from upasfs's cache by the way so you are dealing with two caches.2) Machine 2 is missing some permissions for reading blocks on the store.
Looks like (2) is out and (1) is in.
I tested (2) by using the exact same credentials to read the file from a different machine (say B). This failed, leading me to believe that (2) has been invalidated. Machine (say) A can read the file just fine with cacheserver.
Turning the cacheserver off on machine A, and attempting to read, led to an error. See below.
So it seems that the file never made it to the store to begin with, consistent with (1).
But it is present in the cacheserver's cache of A.
```
2024/04/04 22:09:10.073548 key/remote("").Dial("f+...@hdlfactory.com", "remote,key.upspin.io:443")
2024/04/04 22:09:10.073576 key/remote("key.upspin.io:443").Lookup("f...@hdlfactory.com")
2024/04/04 22:09:10.214204 dir/remote("").Dial("f+...@hdlfactory.com", "remote,upspin.hdlfactory.com:443")
2024/04/04 22:09:10.214251 dir/remote("upspin.hdlfactory.com:443").Lookup("f...@hdlfactory.com/cfg/f+cfg/pcc/db/db.sqlite")
2024/04/04 22:09:10.335345 store/remote("").Dial("f+...@hdlfactory.com", "remote,upspin.hdlfactory.com:443")
2024/04/04 22:09:10.335375 store/remote("upspin.hdlfactory.com:443").Get("E199D05CA4BF3E5158390086B179075084119302EAB4EC69B8B50778E666B0DE")
2024/04/04 22:09:10.335380 store/remote("upspin.hdlfactory.com:443").Get("metadata:HTTP-Base")
upspin: get: client.Get: f...@hdlfactory.com/cfg/f+cfg/pcc/db/db.sqlite: item does not exist:
2024/04/04 22:09:10.073576 key/remote("key.upspin.io:443").Lookup("f...@hdlfactory.com")
2024/04/04 22:09:10.214204 dir/remote("").Dial("f+...@hdlfactory.com", "remote,upspin.hdlfactory.com:443")
2024/04/04 22:09:10.214251 dir/remote("upspin.hdlfactory.com:443").Lookup("f...@hdlfactory.com/cfg/f+cfg/pcc/db/db.sqlite")
2024/04/04 22:09:10.335345 store/remote("").Dial("f+...@hdlfactory.com", "remote,upspin.hdlfactory.com:443")
2024/04/04 22:09:10.335375 store/remote("upspin.hdlfactory.com:443").Get("E199D05CA4BF3E5158390086B179075084119302EAB4EC69B8B50778E666B0DE")
2024/04/04 22:09:10.335380 store/remote("upspin.hdlfactory.com:443").Get("metadata:HTTP-Base")
upspin: get: client.Get: f...@hdlfactory.com/cfg/f+cfg/pcc/db/db.sqlite: item does not exist:
store/server.Get:
file does not exist
2024/04/04 22:09:10.701125 shutdown: status code 1
```
David Presotto
Apr 5, 2024, 3:07:20 PMApr 5
to Filip Filmar, Upspin
OK, If you cat some of them you might find your missing stuff. that's stuff that has to go back to your storage. The question is why it isn't. Time to look at your store cache server logs. It should be trying to write them back over and over if it fails.
On Fri, Apr 5, 2024 at 11:38 AM Filip Filmar <fil...@gmail.com> wrote:
Yes, there's quite a bunch of stuff there, around 50 files.FOn Fri, Apr 5, 2024 at 10:26 AM David Presotto <pres...@gmail.com> wrote:on the machine with the file in its cache, under your upsin caches directory, do you have one for "storewritebackqueue" and if so is there anything in it?On Fri, Apr 5, 2024 at 8:06 AM David Presotto <pres...@gmail.com> wrote:I'll take a look at the conditions under which the cache server gives up trying to put the blocks. I may have screwed up somehow.
David Presotto
Apr 5, 2024, 3:09:12 PMApr 5
to Filip Filmar, Upspin
by the way, you might learn something by starting the cacheserver standalone (no other upspin stuff running) with cacheserver -log=info
Filip Filmar
Apr 5, 2024, 4:41:04 PMApr 5
to David Presotto, Upspin
Thanks, that was the key:
```
2024/04/05 20:32:49.023750 store/remote("upspin.hdlfactory.com:443").Put(b2f594a3207d593ceef2a9c9bd22d912...) (20480 bytes) error: permission denied:
store/perm.Put: user f+...@hdlfactory.com: user not authorized
2024/04/05 20:32:49.023815 store/storecache.scheduler: writeback failed: store/remote("upspin.hdlfactory.com:443").Put: permission denied:
store/perm.Put: user f+...@hdlfactory.com: user not authorized
store/perm.Put: user f+...@hdlfactory.com: user not authorized
2024/04/05 20:32:49.023815 store/storecache.scheduler: writeback failed: store/remote("upspin.hdlfactory.com:443").Put: permission denied:
store/perm.Put: user f+...@hdlfactory.com: user not authorized
```
Perhaps I didn't do `setupwriters` but should have? That said, I don't think I have the contents of the `deploy` dir anymore - what to do?
F
Filip Filmar
Apr 5, 2024, 4:51:55 PMApr 5
to David Presotto, Upspin
Yup, that was it: the `upspin@mydomain/Group/Writers` was not set up correctly.
Thanks for the guidance. I completely forgot about the Writers.
I ended up having to regenerate the key for the above user because I lost the deploy file and keys (but not the babbleprint)
Then rewrote the file with the correct contents.
Now, the remote cacheserver is happy and replaying its logs.
Thanks for the guidance. I completely forgot about the Writers.
F
David Presotto
Apr 5, 2024, 5:04:13 PMApr 5
to Filip Filmar, Upspin
For what it's worth, I forget it too.
Özgür Kesim
Apr 6, 2024, 4:29:12 AMApr 6
to David Presotto, Filip Filmar, Upspin
Is that (content of Writers) related to issue #633?
(I still need to write a test-case for that one)
-oec
Thus spake David Presotto (pres...@gmail.com):
(I still need to write a test-case for that one)
-oec
Thus spake David Presotto (pres...@gmail.com):
> --
> You received this message because you are subscribed to the Google Groups "Upspin" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to upspin+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/upspin/CAC_Z_pRGQx6Y4-FdBj5yYaSvRMSE8xnHbqQX_76wZBQzh9UeJA%40mail.gmail.com.
> You received this message because you are subscribed to the Google Groups "Upspin" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to upspin+un...@googlegroups.com.
David Presotto
Apr 6, 2024, 10:41:46 AMApr 6
to Özgür Kesim, Filip Filmar, Upspin
Related only in being a pain to diagnose.
Reply all
Reply to author
Forward
0 new messages