Select is not working... and foreign characters..

[Anders Hansson]

Hello!

First I must say that I am totally blown away by this project (M/DB)
and even more so about node-mdb! It is simply amazing!

However, I've run into some trouble when I try to migrate some
existing code over to M/DB from SimpleDB.

The code works fine on SimpleDB but not so much on M/DB. I recently
installed M/DB version 1.11 (more precisely the package
mgwtools-1.11_amd64.deb) according to the instructions for Ubuntu.

I have two problems:

1. SELECT now working

When running "select * from `my_domain`" it returns an empty
SelectResult (status is 200). However if I use get_attributes on the
item using the item name on the exact same domain (my_domain) I will
receive the correct item. So clearly the domain is not empty.

This is also not consistent with how SimpleDB behaves. Could someone
enlighten me? Select seems like such a basic and commonly used
operation so this can not have been gone unnoticed if this was the
case. However the exact same select is working fine towards SimpleDB.

Is there some sort of dependency on system libraries that can mess
stuff up (I'm running on Ubuntu 11.04 / Natty)?

2. FOREIGN characters fails to authenticate

I'm getting the error: "The request signature we calculated does not
match the signature you provided. Check your Secret Access Key and
signing method."

I get the above error when I try to write an attribute containing the
characters "åäö". Note that it is not the attribute name but the
actual value. If I change the string contents to "abc" it works fine.
Also in SimpleDB both cases work without any problem.

I read somewhere that there are some basic data type handing in M/DB?
Could it be that it get's confused somehow by these characters?

Ok, thanks in advance!

It seems like great stuff - but I need to get it working! ;-)

BR,
Anders

[rtweed]

Anders

Glad you like it! :-)

Regarding the problems:

1) Can you try "select * from my_domain" ie without the single
quotes around the domain name and see if that works - it should. My
understanding is that this is the correct syntax. However if SimpleDB
accepts the domain name with single quotes around it, then I'll look
into making a change to allow that too.

2) Foreign characters - yes it kind of doesn't surprise me (we Brits
never think about such things!! ;-) Let me try to find out what's
needed.

Unfortunately, because M/DB and node-mdb is just really a side
project, work on it has to take a back-seat from my normal fee-paying
work, so I can't guarantee when I can get these fixes in place, but
I'll do my best to get something done soon.

...of course if anyone out there wants to provide some funding.. ;-)

Oh, one last question. You say you've built the M/DB system using
Mike's installer, but are you using the older Apache/M version of M/
DB, or did you upgrade it to use the Node.js version?

Cheers

Rob


On Jun 16, 10:45 pm, Anders Hansson <anders.hansso...@gmail.com>
wrote:

[Eric Bowen]

Rod-

I want to make sure you understand the distinction between the backtick (`) - ASC(39) vs the single quote (') - ASC(96).

As far as SimpleDB is concerned.  Domain names and field names MUST be enclosed in BACKTICKS if they contain special characters.  Some libraries enclose them all the time to avoid having to test them...  

Single quotes are INVALID surrounding domain and attribute names.

Single quotes and double quotes can be used interchangeably to enclose attribute VALUES in the where clause.

Example (valid):  

   select * from `my$domain` where `foo$`  = '123' and `bar$` = "456"

Cheers-

-EAB

--
You received this message because you are subscribed to the Google Groups "M/DB Community Forum" group.
To post to this group, send an email to mdb-commu...@googlegroups.com.
To unsubscribe from this group, send email to mdb-community-f...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mdb-community-forum?hl=en-GB.

[Rob Tweed]

Eric

Interesting - I was unaware of these rules.   Not sure where these rules are documented in SimpleDB.  Indeed this conflicts with Amazon's own documentation which states, for Domain Names:

3-255 characters (a-z, A-Z, 0-9, '_', '-', and '.')

However I keep being told about undocumented (or mis-documented) anomalies, and since the aim of M/DB is to work identically to SimpleDB, I suppose I'll have to keep patching! :-)

As it happens it should be relatively simple to fix this, since there's nothing unusual in these strings as far as GT.M is concerned, so I should be able to just remove back-ticks if I find them in the query surrounding a domain name or item name and it should then work as per SimpleDB

Rob

--
Rob Tweed
Director, M/Gateway Developments Ltd
http://www.mgateway.com
------------------
EWD Mobile: Build mobile applications faster
http://www.mgateway.com/ewd.html

[Eric Bowen]

The excellent Query 101 document has a good discussion of these rules: http://aws.amazon.com/articles/1231

[Rob Tweed]

Actually I just found the AWS documentation that covers the rules, but thanks for this link!

Cheers

Rob

[Anders Hansson]

Hi Rob!

Thanks for your answers, and I totally understand your side project priorities, no worries!

Anyway:

1. Yes it DOES work without the backticks. I can live without them for the time being. Thanks!

2. Foreign characters... Does this mean that any special characters inside an attribute value can mess things up. Not just foreign characters. How about quotations, back-ticks, etc.

3. NEW: It does seem like the put_attribute replace option is not working the same as SimpleDB. When I use replace and put multiple attributes of the same name only one of them gets stored. In SimpleDB both is stored. I.e. replace only refers to what is only already in the database _before_ the put_attributes are called. I can work around this but the behavior is different from SimpleDB.

Anyway, good work!

BR,

Anders

[Anders Hansson]

I forgot,

regarding the version I use the "old". I have not yet upgraded to node-mdb. 

Are there any drawbacks by doing this upgrade? Select is not as elaborate I understand?

/Anders

On Fri, Jun 17, 2011 at 10:37, rtweed <rob....@gmail.com> wrote:

[Rob Tweed]

OK that's why you're seeing the replace not working properly.  You should find that it's fixed in the node-mdb version.

The node-mdb version is where all the ongoing development is focused - the old M/DB version will be deprecated.  In fact there will be parts of the old M/DB code that will be used by the node-mdb version, but over time that will disappear.  For example, the SELECT API is actually still handled by the old M/DB code, but pretty much everything else is now coded in Javascript.

So I'd recommend you upgrade to use the node-mdb version.  You'll need to install Node.js and NPM (the package manager for Node) - after that it's pretty simple to get it hooked up and working with your existing setup.  By the way, node-mdb uses the same back-end data structures in the GT.M database, so data you create in the old M/DB version will still be compatible with node-mdb.

As to the foreign characters, I'll need to do some investigation before I can adequately answer your questions

Rob

[Anders Hansson]

Ok, I understand. It makes sense.

I'll upgrade to node-mdb in the near future. You say that it utilises the "old" code for some of the select implementation. Does that mean I have to keep the old installation around. Or should I do some kind of "clean" upgrade and remove the old? I have no data that needs to be preserved as it stands right now.

/Anders

[Rob Tweed]

If you grab node-mdb from the GitHub repository, you'll find the latest GT.M routine file MDB.m - that's the legacy code with all the latest changes needed for node-mdb.  That replaces the current copy in your system - if you used Mike Clayton's installer, that means replacing /usr/local/gtm/ewd/MDB.m and /usr/local/gtm/zmwire.m

You'll need both those .m files for the forseeable future

Rob

[Anders Hansson]

Ok, thanks. I'll try it out!

/Anders

[rtweed]

Anders

I've been working on a new build of M/DB that fixes all the issues
you've highlighted and it seems to be going well. However I have a
question regarding accented (foreign) characters. I've figured out
how to handle these in node-mdb and as far as I can see I'm
implementing it as you would *expect* SimpleDB itself to work, by
converting the accented characters to their 2-byte UTF8 representation
(eg %C3%B9 ) within the HTTP request.

However, if I try sending such a request to SimpleDB itself, I just
get the "Signatures do not match" error. I've Googled the issue and
there seems to be a recognised problem in this area. I've tried a
variety of alternatives but cannot find a way to get the request's
signature matched by SDB. However, your message implied that you've
been able to successfully save accented characters into SimpleDB, so
I'd be interested to know exactly how it's done.

For example, I can't even get Amazon's own SimpleDB scratchpad to
work, eg I get a signature mismatch error when the following is
submitted via the scratchpad:


https://sdb.amazonaws.com?DomainName=test&ItemName=item21&Attribute.1.Name=attr1&Attribute.1.Value=abc%C3%B9ef&Action=PutAttributes&Version=2009-04-15&AWSAccessKeyId=[my
key]&SignatureVersion=2&SignatureMethod=HmacSHA1&Timestamp=2011-07-06T11%3A03%3A57.000Z&Signature=[the
signature]

Note the value: abc%C3%B9ef which represents abcùef

Are you able to confirm what a valid SimpleDB request that includes
accented characters should look like in order to get correctly sign-
matched by SimpleDB?


Thanks

Rob









On Jun 16, 10:45 pm, Anders Hansson <anders.hansso...@gmail.com>
wrote:

[rtweed]

Can folks please try out the latest node-mdb appliance which should
now fix the escaping issues and should properly support foreign
characters (even though I can't get SimpleDB proper to accept them the
way I *believe* it should!)?

I've also posted the updates to Github (https://github.com/robtweed/
node-mdb and https://github.com/robtweed/mdb)

Rob

On Jul 6, 12:15 pm, rtweed <rob.tw...@gmail.com> wrote:
> Anders
>
> I've been working on a new build of M/DB that fixes all the issues
> you've highlighted and it seems to be going well.  However I have a
> question regarding accented (foreign) characters.  I've figured out
> how to handle these in node-mdb and as far as I can see I'm
> implementing it as you would *expect* SimpleDB itself to work, by
> converting the accented characters to their 2-byte UTF8 representation
> (eg %C3%B9 ) within the HTTP request.
>
> However, if I try sending such a request to SimpleDB itself, I just
> get the "Signatures do not match" error.  I've Googled the issue and
> there seems to be a recognised problem in this area.  I've tried a
> variety of alternatives but cannot find a way to get the request's
> signature matched by SDB.  However, your message implied that you've
> been able to successfully save accented characters into SimpleDB, so
> I'd be interested to know exactly how it's done.
>
> For example, I can't even get Amazon's own SimpleDB scratchpad to
> work, eg I get a signature mismatch error when the following is
> submitted via the scratchpad:
>

> https://sdb.amazonaws.com?DomainName=test&ItemName=item21&Attribute.1...[my
> key]&SignatureVersion=2&SignatureMethod=HmacSHA1&Timestamp=2011-07-06T11%3A 03%3A57.000Z&Signature=[the

[rtweed]

Sorry - for details on downloading the latest node-mdb appliance, see
http://www.mgateway.com/node-mdb.html

Rob

On Jul 7, 5:52 pm, rtweed <rob.tw...@gmail.com> wrote:
> Can folks please try out the latest node-mdb appliance which should
> now fix the escaping issues and should properly support foreign
> characters (even though I can't get SimpleDB proper to accept them the
> way I *believe* it should!)?
>
> I've also posted the updates to Github (https://github.com/robtweed/

> node-mdb andhttps://github.com/robtweed/mdb)

[Anders Hansson]

Thanks Rob, great work!

I will try to figure out the URL used for SimpleDB. I've been using AWS SDK for PHP and it is wrapping stuff up so I'll have to intercept the URL before it going out. (For the record, the only foreign characters I tested with was å ä ö).

Get back to you as soon as I've got some time to test.

/Anders