node-mdb Virtual Appliance

[rtweed]

I've just released a pre-built, ready-to-run VMWare Virtual Machine
that will allow you to test, evaluate and run the new Node.js-based
version of M/DB within a few minutes of downloading it.

Full details at http://www.mgateway.com/node-mdb.html

Please download it and try it out!

Rob

[Scrappydog]

I'm getting "Signatures don't match" exceptions trying to run tests
using existing integration test code that runs fine against Amazon...

My code is using the AWSSDK for .NET v 1.3.6.0

On Jun 6, 7:50 am, rtweed <rob.tw...@gmail.com> wrote:
> I've just released a pre-built, ready-to-run VMWare Virtual Machine
> that will allow you to test, evaluate and run the new Node.js-based
> version of M/DB within a few minutes of downloading it.
>

> Full details athttp://www.mgateway.com/node-mdb.html

[rtweed]

Almost certainly it's because the string to sign is differing between
the client and node-mdb because of the different endpoint URL + port.
Trust .Net! :-)

Could you post here the trace from the node-mdb terminal window that
shows the entire set of interactions it went through when processing
an incoming request and I'll see if there's anything obvious

If not then I'll get you to use an instance I have running on an EC2
server and I'll be able to see what's coming in for myself from your
client.

Rob

[Eric Bowen]

Screenshot attached... (not sure if this works in Google Groups...)

> --
> You received this message because you are subscribed to the Google Groups "M/DB Community Forum" group.
> To post to this group, send an email to mdb-commu...@googlegroups.com.
> To unsubscribe from this group, send email to mdb-community-f...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mdb-community-forum?hl=en-GB.
>
>

[rtweed]

I'm sure it's not the problem but can you confirm that you are using
the correct secret key: mdbsecretkey all in lower case.

Here's what I'd expect the string to sign to look like - note that the
one thing that is different is the format of the Timestamp:

StringToSign = POST
192.168.1.134:8081
/
AWSAccessKeyId=administrator&Action=ListDomains&MaxNumberOfDomains=100&SignatureMethod=HmacSHA1&SignatureVersion=2&Timestamp=2011-06-06T22%3A39%3A30%2B00%3A00&Version=2009-04-15
hash = WJqEwdrFl5TdfwjYEn4IMkXcNks=; Signature =
WJqEwdrFl5TdfwjYEn4IMkXcNks=


Specifically you would not expect to see a period at the end of it: it
should be escaped. So it appears that the AWS .Net client is sending
the Timestamp incorrectly escaped and node-mdb is going to have to do
some kind of a workaround. I had a problem with their Java client too
which didn't properly conform to their documented standard and had to
work around that too! Meanwhile all the third-party clients appear to
have no problems - it's just AWS's own ones!

Anyway I've privately sent you details on how to contact me so that we
can do some more detailed debugging against my test server - hopefully
we can figure out the issue and I can implement a workaround.

Rob

>  mdb_trace.png
> 13KViewDownload

[Rob Tweed]

OK I see the problem - nothing to do with the time stamp. The AWS .Net client is not adding the :8081 to the string to sign!  This is bizarre because the AWS Java client (and every other 3rd party client) *does* actually include the port number in the string to sign!

If I remove the :8081 and recalculate the hash, I get the signature that the .Net client is sending.  What I think I'll do is to get Node.js to try to calculate the hash a second time without the port number if it first finds an error.  I was actually wondering if I'd find a client that doesn't include the port number at some point

Anyway I'll post a fix tomorrow

Thanks for your patience and for sending the info!

Rob

--
Rob Tweed
Director, M/Gateway Developments Ltd
http://www.mgateway.com
------------------
EWD Mobile: Build mobile applications faster
http://www.mgateway.com/ewd.html

[Eric Bowen]

I patched the AWSSDK for .NET to add the Port to the "string to sign"
and that resolved the error...

(Now... still none of my integration tests pass... but we are on to
another error for tomorrow... :-)

[rtweed]

OK Eric glad that did the job! One down, more to come. I'm intrigued
to know what you found but just to pre-empt what they might be, please
note that node-mdb does not currently support:

- the Batch* APIs
- MaxNumberOfDomains
- NextToken
(all results are always returned in one response)
- Numeric values are treated as numeric, not strings as in SDB

So...what ya got for me? :-)

Rob

On Jun 7, 3:36 am, Eric Bowen <e...@scrappydog.com> wrote:
> I patched the AWSSDK for .NET to add the Port to the "string to sign"
> and that resolved the error...
>
> (Now... still none of my integration tests pass... but we are on to
> another error for tomorrow... :-)
>
>
>
>
>
>
>
> On Mon, Jun 6, 2011 at 5:17 PM, Rob Tweed <rob.tw...@gmail.com> wrote:
> > OK I see the problem - nothing to do with the time stamp. The AWS .Net
> > client is not adding the :8081 to the string to sign!  This is bizarre
> > because the AWS Java client (and every other 3rd party client) *does*
> > actually include the port number in the string to sign!
> > If I remove the :8081 and recalculate the hash, I get the signature that the
> > .Net client is sending.  What I think I'll do is to get Node.js to try to
> > calculate the hash a second time without the port number if it first finds
> > an error.  I was actually wondering if I'd find a client that doesn't
> > include the port number at some point
> > Anyway I'll post a fix tomorrow
> > Thanks for your patience and for sending the info!
> > Rob
>

[Scrappydog]

That explains it... virtually all of my integration tests use BatchPut/
BatchDelete to setup/teardown test data...

I'll have to look around to see what I can easily test without that...
your other unsupported items are probably all very minor from my
perspective.

Curious: How are you determining if a value is numeric?

[rtweed]

Eric - I'm planning on implementing the Batch APIs sometime soon (fee
earning work just keeps getting in the way! :-)

The numeric aspect is kind of interesting. Everything is stored in
GT.M as strings.

At present the Select API logic is using the original M/DB logic which
is written in the native GT.M scripting language called M. It's the M
language that is determining the numeric value. It's all pretty
straightforward and I'll probably do something similar as and when I
get round to porting the Select API logic to Javascript. M treats
everything as strings, but evaluates them as numeric values if used in
a numeric context. It simply looks at the start of the string and
uses any leading numeric characters. So:

10 + "12abc" = 22
10 + "abc" = 10

Similar behaviour occurs (or can be made to occur) pretty simply in
Javascript too, since it's similarly loosely-typed.

So no need, in M/DB, to pack out numeric values with leading zeros etc
as you often have to do in SDB

Anyway, if you have any suggestions/ideas/thoughts about node-mdb,
please let me know!

Cheers

Rob

[rtweed]

I've posted a new build of mdb.js at https://github.com/robtweed/node-mdb,
which you'll find in the /lib directory

The new build adds a 2nd pass to the signature checking. The first
pass tries to match the signature where the string to sign contains
the port number after the host name. If that fails, it tries a second
time without adding the port (ie just using the host name as line 2 of
the string to sign).

I've not created a new, updated VM (that's a 3-hour upload job!), so
if anyone wants to use the new version, just replace /mdb/mdb.js in
your Virtual Appliance with the new version and restart:

node mdb.js

Rob

[Eric Bowen]

FYI: My interest in node-db is primarily as a local test platform for
code that will run on SimpleDB in production. And so my interest in
alternate numeric data handling is fairly low, but I was curious to
see what you were doing and why...

Thanks for your efforts on this!

[K.S. Bhaskar]

Eric --

If you get any data on performance of SimpleDB vs. node-db or M/DB, please do post them.  Also anything you can say about cost & recoverability.  Thank you very much.

Full disclosure of my interest - I manage GT.M (http://fis-gtm.com & http://sourceforge.net/projects/fis-gtm).

Regards
-- Bhaskar

Windows does to computers what smoking does to humans

[Rob Tweed]

By the way, I'm intending to extend the SDB API to include some additional system management Actions for node-MDB that will allow you to do the following:

- snapshot the M/DB database.  This will copy the entire ^MDB global into another global that will then contain a snapshot of your M/DB database at that time.  This will use a MERGE command in GT.M and so should be very fast.  However it will not be a secure save - if the GT.M database is corrupted, the snapshot will be useless too.

- recover a snapshot. M/DB will be wound back to the state it was in for the selected snapshot

- delete a snapshot

- archive M/DB database to a specified text file, allowing you to secure a backup that you can extract from the M/DB machine and retain offline if required.

- recover M/DB database from a specified text file.  This will allow you to recover your M/DB database to a brand new instance of an M/DB system or appliance, either to create a copy of your database, or to recover from a GT.M database failure/corruption.

Archiving will be a little slower than taking snapshots.  I'll probably gzip the archive file to make it as small as possible.

Only accessKeyIds that are flagged as being administrators will be allowed to use these Actions.  No other users will have access to them.

The idea is to allow the M/DB database to be managed and secured/recovered without any knowledge being required of GT.M, the M language or the GT.M utilities.

Rob

[Rob Tweed]

Oh, and there should be a "list snapshots" action too that will provide the id and date of the snapshot.

Rob

[rtweed]

I've released a new version of the node-mdb appliance:

http://mgateway.s3.amazonaws.com/node-mdb-appliance-v2.7z

Please give it a try and let me have any feedback.

It fixes a number of bugs and adds the BatchDeleteAttributes API.
This means that it now includes the entire range of SimpleDB APIs.

Rob

On Jun 7, 6:02 pm, Rob Tweed <rob.tw...@gmail.com> wrote:
> Oh, and there should be a "list snapshots" action too that will provide the
> id and date of the snapshot.
>
> Rob
>

> > On 7 June 2011 17:34, K.S. Bhaskar <bhas...@bhaskars.com> wrote:
>
> >> Eric --
>
> >> If you get any data on performance of SimpleDB vs. node-db or M/DB, please
> >> do post them.  Also anything you can say about cost & recoverability.  Thank
> >> you very much.
>
> >> Full disclosure of my interest - I manage GT.M (http://fis-gtm.com&
> >>http://sourceforge.net/projects/fis-gtm).
>
> >> Regards
> >> -- Bhaskar
>

> >> On Tue, Jun 7, 2011 at 12:21 PM, Eric Bowen <e...@scrappydog.com> wrote:
>
> >>> FYI: My interest in node-db is primarily as a local test platform for
> >>> code that will run on SimpleDB in production. And so my interest in
> >>> alternate numeric data handling is fairly low, but I was curious to
> >>> see what you were doing and why...
>
> >>> Thanks for your efforts on this!
>

> ...
>
> read more »