No dot allowed in key name, how to create an email index?

[David Notik]

I'm trying to create an email_index with paths like email_index/f...@bar.com which would have the user associated with that email.

It seems no dots allowed in key names. I confirmed that here:

https://www.firebase.com/docs/web/guide/understanding-data.html

UTF-8 encoded, cannot contain . $ # [ ] / or ASCII control characters 0-31 or 127

(Side note, using the REST API, I get a 400 status code. Took me a second look to realize that there are multiple reasons for a 400 listed at https://www.firebase.com/docs/rest/api/#section-error-conditions.)

Is a good solution to just replace dots and save and retrieve it that way?

[Erik Beeson]

Yes, you have the right idea. I think it's safest to always run keys through an encode function like:

function encodeKey(s) { return encodeURIComponent(s).replace('.', '%2E'); }

Hope it helps,

Erik

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/93899317-c6e8-4a46-86f7-aae6378137ef%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Notik]

Thanks! So your approach is to actually store the escaped character.

Should it handle other unallowed chars besides a period?

[Jacob Wenger]

Hey David,

Yup, you'll need to handle all the disallowed Firebase characters. That being said, several of the disallowed Firebase characters are also not allowed in emails in the first place. The technical reason for this limitation is that since every Firebase node is a URL, they cannot include characters which are used in URLs. Since periods serve an important role in URLs, we disallow them.

Jacob

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/99e32a33-e071-479a-b2c7-0b6bd69172a3%40googlegroups.com.

[Erik Beeson]

Originally I had written code to only encode the reserved characters, but encodeURIComponent catches everything (except .) and there's no harm in encoding all of it.

I have some utility code that does both ways here: https://github.com/ebeeson/firebase-utils/blob/master/firebase-codec/firebase-codec.js

--Erik

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/99e32a33-e071-479a-b2c7-0b6bd69172a3%40googlegroups.com.

[David Notik]

Thanks all! I followed your lead to tackle all encoding/decoding rather than piecemeal address emails.

Look about right?

  String encodeKey(key) => Uri.encodeComponent(key).replaceAll('.', '%2E').toString();
  String decodeKey(key) => Uri.decodeComponent(key).replaceAll('%2E', '.').toString();  

[David Notik]

Hrm. Why would this path be incorrect? %s are allowed AFAIK.

Uncaught Error: Firebase returned an error.

Path: /email_index/catalina%40catalinaayubi%2Ecom.json

Status code: 400

[Erik Beeson]

I expect Uri.decodeComponent will handle replacing '%2E' for you (even though encodeComponent doesn't encode it), so you're probably actually incorrectly double decode '.' by having replaceAll in decodeKey.

--Erik

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/e47ac68e-719e-45dd-924b-6b4c35cf4418%40googlegroups.com.

[Kato Richardson]

David, since you've encoded those characters, they are literal %40 in Firebase. If you then enter %40 into a URL, you're going to get the ascii representation (the period). So to enter that in a URL, you need to show a literal %40 using %2540

Cheers,

Kato

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/77fe7bb8-7a32-444c-8cd9-dfd1938db376%40googlegroups.com.

[David Notik]

Hm! Haven't encountered this before. So I really have to double it up like the following? It works:

  String encodeKey(key) => Uri.encodeComponent(Uri.encodeComponent(key)).replaceAll('.', '%252E').toString();

  String decodeKey(key) => Uri.decodeComponent(Uri.decodeComponent(key)).toString();    

[Erik Beeson]

Perhaps more context about how you're using the result of the encodeKey function would help here. What I suggested was based on sanitizing the keys in order to pass them to a Firebase client library.

If you're manually constructing URLs, then there's also the issue of properly escaping/encoding the URL.

Here's an example:

The email address is encoded once to use as a key, but when constructing a URL it gets escaped again (see the URL at the bottom).

Note that this isn't really specific to Firebase. Anytime you're trying to build a URL, you need to take care to encode each chunk (between the '/'). Usually some sort of library can help with safely building a URL. Concatenating strings without taking proper care can lead to broken URLs.

Hope it helps,

Erik

P.S. I'm often in #firebase on freenode IRC if you want more help.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/9472bd7e-fc59-4219-859a-f66665943161%40googlegroups.com.

[Jason Berryman]

Hi,

I have 2 questions relating to this issue:

1. Why is a dot not allowed?  In our current API, we post email addresses in the URL.  As long it is after the first "/", it works.
   
2. I'm having the issue that you demonstrate in your screenshot, where I'm trying to manually create some encoded entries via the Dashboard.  When I enter 'foo%2Ebar@example%2Ecom' the URL is shown in the address bar as 'foo%252Ebar@example%252Ecom'.  How can I create a key which I can access by browsing to '../users/foo%2Ebar@example%2Ecom.json' ?
   

Many thanks,

Jason

Dolphin ONE Communications

sales: +44 20 3375 4000
support: +44 20 3375 4 247

...or visit http://www.dolphinone.net

 

P please don't print this email unless you really need to.

[Jason Berryman]

Hi,

I've been doing a further bit of research.  Google's own APIs frequently use a dot in them.  The Email Settings API includes a dot in both the version number and the {domain} query parameter.  I appreciate that this may take some time to implement, but it would make our lives so much easier if it was supported in Firebase.

Best regards,

Jason

[Jacob Wenger]

Hey Jason,

Since every piece of data in Firebase is addressable via a URL, we don't allow characters which would break the URL. A period in a URL is a reserved character and thus cannot be used in a key, since you would not be able to navigate to that URL. I do not see this changing anytime soon even if we wanted to.

To work around this this, we use the following regular expression to escape any string so that it is a valid Firebase key:

function encodeAsFirebaseKey(string) {

  return string.replace(/\%/g, '%25')

    .replace(/\./g, '%2E')

    .replace(/\#/g, '%23')

    .replace(/\$/g, '%24')

    .replace(/\//g, '%2F')

    .replace(/\[/g, '%5B')

    .replace(/\]/g, '%5D');

};

I would suggest using that function to handle storing keys. Just make sure you escape the original % signs first!

As for your second question, it is a bit tricky. It looks like you have too many escapes in your original string. For example, If you do the following:

var ref = new Firebase("https://<YOUR-FIREBASE>.firebaseio.com");

ref.child("foo%2Ebar@example%2Ecom").set(true);

And then try to navigate to https://<YOUR-FIREBASE>.firebaseio.com/foo%252Ebar%40example%252Ecom (note that I had to escape the % signs), you will see the data you want.

Hope that helps!

Jacob

--

You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/1e997ecd-a83b-43ad-9065-2cefd38a0164%40googlegroups.com.

[Jason Berryman]

Hi Jacob,

Thanks for your response.

Dots in the URI

I would dispute that they break a URI.  They are used regularly in Google's own APIs and our current API (which I'm trying to replace with Firebase).  Have a look at https://developers.google.com/admin-sdk/email-settings/#email_settings_api_http_format for an example.

Escape characters in the Dashboard

The issue is not wanting to add the extra %252E, it is to remove it.  When I use the Firebase Dashboard to create a key which contains a dot, it is not allowed.  If I replace the dot with %2E, then the Dashboard doesn't simply write this, it then escapes the % and adds %25 to it.  So instead of a URI which reads foo%2Ebar@example%2Ecom I end up with additional escape codes.

Best regards,

Jason

[Jacob Wenger]

Hey Jason,

You are correct in stating that I was wrong about the dots in the URI. They do not break URIs in general, but with our architecture, we use several special-cased dot values, such as .priority. I don't see us changing this in the near future as that would be an exceptionally breaking change.

As for the Firebase Dashboard, things are working as I expect. If I type in foo.bar as a key in the Dashboard, I get a console message saying periods are not allowed. This fails silently, which is bad UX, so I logged a bug for our UX team to fix this. If I then type in foo%2Ebar as a key, it creates a new key with that exact string. The confusing here comes when you click on that key. We get taken to a URL containing foo%252Ebar, but the key we see in the data tree is still foo%2Ebar, which is what we expected and wanted. The reason the URL is different is because Chrome (or whatever browser you are using) escapes the URL for us. If we navigate to the URL with foo%2Ebar in it, we get a Not Found error since Firebase keys cannot contain periods and Chrome has escaped the %2E into a period for us.

Additionally, if we use the Firebase client to do an on value on the foo%2Ebar child, it gets the data we want. So that seems to work as well.

Are you seeing different behavior? Dp you think any of what I described above is not expected behavior?

Jacob

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/18781b10-d7d4-472a-8b30-cceec6a5a6c2%40googlegroups.com.

[David Notik]

FWIW, here are my functions for encoding and decoding Firebase keys: https://gist.github.com/davenotik/11bb51bc7e77448b3541

It seems I really only use the encode one – I simply encode the email-as-key before saving to or retrieving from paths with that key.

For usernames I don't allow a dot, so I really only have to remember this for emails.

I also have an email_index that looks something like this: http://d.pr/i/Qlvt

--
Woven
http://woven.co
206-351-3948

@DaveNotik

--
You received this message because you are subscribed to a topic in the Google Groups "Firebase Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/firebase-talk/vtX8lfxxShk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAGcMwsAy5PsSyhNAzp5%2B5Lf-98bwMG2AVoF5Q%3Duut8Qzmy5i9A%40mail.gmail.com.

[Erik Beeson]

Hi Jason,

Two things:

While I'm not affiliated with Firebase, I assume that dot is reserved because it's used to signify an extension (add .json to the end of any firebase URL, for example). By reserving dot, it can be insured that there won't be conflict between key and extension, and it leaves the option open for different data formats to be supported in the future.

Second, and more importantly, you definitely always want to be escaping any user input before using it as part of a firebase path anyways lest unexpected user input break your app.

Hope it helps,

Erik

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/18781b10-d7d4-472a-8b30-cceec6a5a6c2%40googlegroups.com.