Stream Write to Upspin

[Albert-Jan de Vries]

Hi,

Because I had some memory issues writing large files to Upspin using the upspin.Client. I created a naive implementation of the upspin.Client with a Write(p upspin.Pathname, r io.Reader) method.

It's available here:

- https://github.com/storageforbusiness/client

-- AJ

[Joe Blue]

Thanks for this. Being able to use a  io.Reader  for writing data to upspin is exactly what i need to build apps that use Upspin for their storage.

[Andrew Gerrand]

This is a convenient API for sure. It's a shame that it's so much code to implement it! Is that because you had to take a bunch of the internals of the client package to do this? Maybe some of that should be exported so that people can reuse it. (Desgning a nice API for it may be a challenge though...)

Another way of framing this would be to have a method that returns a WriteCloser

  func (Client) NewWriter(ctx context.Context, name upspin.PathName) io.WriteCloser

of course, that doesn't give you the resulting DirEntry in that case. I wonder how often that is used? Curious.

To get around that you could have it return a Writer type that looks something like:

  type Writer struct {

    io.WriteCloser

  }

  // DirEntry returns the DirEntry for the written file.

  // It should only be called after the Close method returns a nil error.

  func (Writer) DirEntry() *upspin.DirEntry

I wonder if that's more or less convenient than your Write method (I'd probably name yours PutReader or PutFromReader, something that gives a clue that it's a Put that's happening ultimately).

I have some dim memory of having written something like this before, but I can't find it now. :-/

Andrew

--
You received this message because you are subscribed to the Google Groups "Upspin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to upspin+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/upspin/9c048522-3a38-4454-8168-d76ef7c8595bn%40googlegroups.com.

[Albert-Jan de Vries]

Hi Andrew,

Thanks for your reply. I completely agree with your comments, I developed it in a short amount of time. Maybe more efficient caching and the pack trick I used, should live in the file implementation:

- https://github.com/upspin/upspin/blob/master/client/file/file.go

Maybe that the client interface shouldn't change. 

The use of the internal code was necessary, but can be improved and maybe the packing should have an improved interface and also the logic for Access and Group files should maybe live somewhere else.

I'll give it some thought, and maybe refactor some parts of it. If you are into pairing I'm always available for a pairing session on this subject ;-)

Take care,

-- AJ

Op maandag 4 juli 2022 om 01:51:20 UTC+2 schreef Andrew Gerrand: