Posting With Poster

[Andreas Kollegger]

I've posted a new tutorial on the Shared Records wiki that describes a
simple process for uploading files using Poster, a Firefox extension.

Posting With Poster -
http://www.sharedrecords.org/trac/sharedrecords/wiki/PostingWithPoster

Please take a look and let me know if you have any comments.

Thanks,
Andreas

[Saq Imtiaz]

Andreas,

That is very well written and easy to follow, thank you. I'm curious about one thing though. I've heard frequent mention of the 'proxy' server, especially in reference to the TiddlyWiki integration work with SharedRecords. Then there is the test.sharedrecords.org server, and one mentioned in the tutorial: http://sra.sharedrecords.org:8080/SRCDataStore/RESTServlet/

For my purposes, that is for use with TiddlyWiki, which one should I be looking to use?
Thank you.

Saq

--
Tiddly Learning ( http://lewcid.org ) : TiddlyWiki news, plugins, themes and educational usage

[Andreas Kollegger]

You should use what we should probably start referring to as the Public
(or Community, or ?) Shared Record server located at sra.sharedrecords.org.

The proxy server is a confusing name because all Shared Records server
instances behave as a proxy. They pair a web-based interface with
backing storage (which can be another Shared Records server). The
difference between a being a proxy or not is whether a particular server
has been configured to use local storage. What had been referred to as
the Proxy Server should instead be called the Token Interface (because
it handles the use of QR code "tokens").

We are also not careful about distinguishing between a type of server
and an instance of that type, which adds to the confusion.

So, some clarification of terms:

REST API - The lowest-level http interface to a SharedRecords server,
which emulates Amazon's S3 interface.
Token API - What had been known as the "Proxy Server" is an interface
that handles QR code "tokens", providing automatic encryption/decryption
services.
Annotation API - An interface which focuses on providing a clean
interface to accessing Record metadata.

Public Shared Records Server - The host at sra.sharedrecords.org which
supports the REST and Token interfaces.
Test Server - The host at test.sharedrecords.org which is used by
developers. It is a staging server for new interfaces, feature additions
and bug testing.

Does that make more sense?

-Andreas

Saq Imtiaz wrote:
> Andreas,
>
> That is very well written and easy to follow, thank you. I'm curious
> about one thing though. I've heard frequent mention of the 'proxy'
> server, especially in reference to the TiddlyWiki integration work
> with SharedRecords. Then there is the test.sharedrecords.org

> <http://test.sharedrecords.org> server, and one mentioned in the

> tutorial: http://sra.sharedrecords.org:8080/SRCDataStore/RESTServlet/
> <http://sra.sharedrecords.org:8080/SRCDataStore/RESTServlet/>
>
> For my purposes, that is for use with TiddlyWiki, which one should I
> be looking to use?
> Thank you.
>
> Saq
>
> On Dec 7, 2007 3:42 AM, Andreas Kollegger < and...@kollegger.name

[Saq Imtiaz]

Andreas,

That is perfect and clears up all the confusion. I have a feeling quite a few people will find this useful. Thank you!

Saq

[Saq Imtiaz]

Andreas,

I've got a potentially silly question.

http://www.sharedrecords.org/trac/sharedrecords/wiki/ProxyPut

Is that method for storing metadata for a record only, or can it be used for creating a new record too?

Does the Shared Records api support a method where PUT results in the data being stored and the record id (hash) being returned as the response or as a header? This would be the easiest way to upload files to Shared Records, as it would skip the need to generate the hash on the client side.

Saq

[Martin Budden]

Andreas,

thanks for this, I'll give this a go on Monday.

Martin

[Andreas Kollegger]

The badly named ProxyPut method is used to create a new record on a
Shared Records server through the Token interface. A new symmetric key
is generated and used to encrypt the file. In the body of the result you
receive an image of the QR code that represents the symmetric key and
the record ID. You need that QR code to retrieve the file back from the
server through the Token interface. Accessing the record directly
through the REST interface will give you the encrypted file, with no way
to decrypt it.

We could implement a simple file upload/download API that supports a PUT
with any file data, storing the unencrypted file and returning the
record-id. The advantage of the current REST API is that it fails early
-- if there is a collision (because the hashcode already exists,
probably from a previous upload) then the operation fails immediately
rather than at the end of a possibly long upload.

Any thoughts on the security implications of direct file uploading
without a hashcode? Anyone?

-Andreas

Saq Imtiaz wrote:
> Andreas,
>
> I've got a potentially silly question.
>
> http://www.sharedrecords.org/trac/sharedrecords/wiki/ProxyPut
>
> Is that method for storing metadata for a record only, or can it be
> used for creating a new record too?
>
> Does the Shared Records api support a method where PUT results in the
> data being stored and the record id (hash) being returned as the
> response or as a header? This would be the easiest way to upload files
> to Shared Records, as it would skip the need to generate the hash on
> the client side.
>
> Saq
>
> On Dec 7, 2007 3:42 AM, Andreas Kollegger <and...@kollegger.name

[Ed Schwartz]

> Any thoughts on the security implications of direct file uploading
> without a hashcode? Anyone?

Not as security, but to generally provide the right user experience, I
think it's a really big advantage of Shared Records that the client
computes a hash and that the server rejects deposits if the hash
doesn't match. This way, the client knows that Shared Records server
actually has the right data. AOL's xDrive is an example of a service
that doesn't have any mechanism like checking hashes and I've found
that I can easily have corrupted uploads.

> > http://www.sharedrecords.org/trac/sharedrecords/wiki/ProxyPut
> >
> > Is that method for storing metadata for a record only, or can it be
> > used for creating a new record too?

> The badly named ProxyPut method is used to create a new record on a

> Shared Records server through the Token interface.

I'd like to share how I'd explain things since I originally had many
of these confusions, but now think it all makes good sense.

RecordDeposit = create record

ProxyPut = create record + encrypt + return JPEG image of barcode

AddMetaData = store metadata

If you want to use encryption by the server, then use ProxyPut to
create records and AddMetaData to store metadata. If you don't want
encryption at all or want to it on the client, then use RecordDeposit
to create records and AddMetaData to store metadata.

I like to think of Shared Records as providing two kinds of storage:
Records are immutable (write-once) storage. Metadata is append-only
storage. I divide what my application needs to store into immutable
and append-only first and worry about the names for Shared Record's
API calls second.

-- Ed

[Martin Budden]

Andreas,

thanks for the instructions. I'm managing to POST successfully to the
public server at sra, eg

http://sra.sharedrecords.org:8080/SRCDataStore/RESTServlet/A94A8FE5CCB19BA61C4C0873D391E987982FBBD3.data

but get an error code when posting to the test server:

http://test.sharedrecords.org:80/SRCDataStore/RESTServlet/A94A8FE5CCB19BA61C4C0873D391E987982FBBD3.data
Data Store error, code is: 4096

Is the POST API available on the test server?

Martin

[Andreas Kollegger]

That looks like a problem with either the configuration or permissions
of the server. The "Data Store" is whatever the server is using for
actual storage of files, which is probably the local filesystem on test.
I'll go check it out.

-Andreas

[Andreas Kollegger]

Yes, that was it: the configuration was expecting to find the local data
store at a DOS-styled path, rather than a UNIX path.

Martin, try it now.

Feature Requests:

1. Default data store -- create default, platform-dependent data store
locations that are used in the absence of a specified path in the
configuration. Also, the standard distribution should not override those
internal default values, so that it just works most of the time
2. Admin interface -- provide a through-the-web admin interface for
changing runtime configuration properties
3. Refactor configuration code -- adopt something like Apache commons
configuration for more flexible configuration options. Also, remove
inline references to configuration. Most objects should have no
knowledge about a configuration framework; it isn't their concern and
creates an unnecessary dependency. The application framework should
apply configuration from the outside, rather than objects pulling from
the inside. That orientation allows any object to be configured.
4. Application Framework -- consider placing the servlets within an
application framework like Spring. As we add servlets to the collection,
a good framework will make it possible to deploy a single war with a
mix-and-match collection of modules with shared functionality.

Just making note of my own development requests for future
consideration. Possible for Shared Records 2.0 (is it too early to start
thinking about the rewrite?).

-Andreas

[Cory Zue]

shouldn't the test server now be serving off the url "records" based on the latest recommendations/server code?

e.g:
http://test.sharedrecords.org/records/A94A8FE5CCB19BA61C4C0873D391E987982FBBD3.data

[Andreas Kollegger]

You're right that everyone should start using the "records" endpoint
instead, and test.sharedrecords.org is also serving from that endpoint.
Test looks like it has both records.war and SRCDataStore.war deployed,
with identical deploy dates.

Martin, can you try using that instead?

Should we undeploy SRCDataStore? Or leave it for comparison purposes? I
don't think there is any value to having it around. Perhaps we could
deploy a "sorry this resource has moved to another endpoint" servlet.

Cory Zue wrote:
> shouldn't the test server now be serving off the url "records" based
> on the latest recommendations/server code?
>
> e.g:
> http://test.sharedrecords.org/records/A94A8FE5CCB19BA61C4C0873D391E987982FBBD3.data
>
> On Dec 10, 2007 10:44 AM, Andreas Kollegger <and...@kollegger.name

> <and...@kollegger.name <mailto:and...@kollegger.name>>

> >> wrote:
> >>
> >>> You should use what we should probably start referring to as
> the Public
> >>> (or Community, or ?) Shared Record server located at

> >>> sra.sharedrecords.org <http://sra.sharedrecords.org>.

> sra.sharedrecords.org <http://sra.sharedrecords.org> which

> >>> supports the REST and Token interfaces.
> >>> Test Server - The host at test.sharedrecords.org

> <http://test.sharedrecords.org> which is used by

> >>>> <mailto: and...@kollegger.name

[Martin Budden]

Andreas,

I've got this working now, so you don't need to reply.

Martin