[data-store-api] How to provide the *built-in* data store for messages?

[Gene Lian]

Hi guys,

I'd like to start with this thread to discuss about how Gecko provides
and defines the *built-in* data store for SMS/MMS messages. This will be
a prerequisite for Gaia to start using the Data Store API to manage the
conversation thread, because the messaging app needs to know which data
store it needs to sync with.

One noticeable thing here is this kind of data store is not provided by
any apps (through add() or put() in the Data Store API), it is a kind of
*built-in* data store managed by the internal RIL or SIM, which means we
have to figure out a formal data store name (e.g., "build-in-messages")
that is predefined for the built-in messages data store, so that the
messaging app can know how to claim the accessibility to that specific
data store in its manifest:

{
datastores-access: {
"built-in-messages": {
"access": "readonly",
"description": ...
}
},
}

Contacts have similar issues. We have to predefine "built-in-contacts"
as a data store name to specify the data store used for the built-in
contacts (compared to other kinds of contacts like "facebook-contacts"
where the contacts are provided by an external app for example).

I need more thoughts and feedbacks about how to well define this kind
of built-in data stores. Any inputs are highly welcome. I also opened
bug 999288 to keep track of this.

Cheers,
Gene

[Gene Lian]

I post a proposal at https://bugzilla.mozilla.org/show_bug.cgi?id=999288

We may discuss on the bug. Thanks!

Gene

> _______________________________________________
> dev-webapi mailing list
> dev-w...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-webapi
>

[Ehsan Akhgari]

I'm not sure if I understand why you want the data store to not be owned
by any app, that seems like a non-goal to me. The current data store
implementation expects there to be an owner with a manifest URL, so this
will ripple its effect through the consumers which is a bad thing.

Can we just make these data stores be owned by the system app?

Cheers,
Ehsan

[Kevin Grandon]

The approach we have been doing so far in gaia is working fairly well where we specify a new application as an owner of a datastore. E.g., we will have a "bookmarks" application which is in charge of bookmarks and provides activities to modify/update/remove these bookmarks. I believe something similar is happening with contacts.

It seems to me like we should have a central app which manages the messages datastore. There are pros and cons of making this be the current messages app.

Best,
Kevin

----- Original Message -----
From: "Ehsan Akhgari" <ehsan....@gmail.com>
To: "Gene Lian" <cl...@mozilla.com>, "dev-webapi" <dev-w...@lists.mozilla.org>
Cc: "Julien Wajsberg" <jwaj...@mozilla.com>, "Gary Chen" <gc...@mozilla.com>, "Bevis Tseng" <bts...@mozilla.com>, "Jonas Sicking" <jo...@sicking.cc>, "JOSE MANUEL CANTERA FONSECA" <jm...@tid.es>, "Ehsan Akhgari" <eh...@mozilla.com>, "Steve Chung" <sch...@mozilla.com>

[Ehsan Akhgari]

On 2014-04-22, 12:10 PM, Kevin Grandon wrote:
> The approach we have been doing so far in gaia is working fairly well where we specify a new application as an owner of a datastore. E.g., we will have a "bookmarks" application which is in charge of bookmarks and provides activities to modify/update/remove these bookmarks. I believe something similar is happening with contacts.
>
> It seems to me like we should have a central app which manages the messages datastore. There are pros and cons of making this be the current messages app.

That sounds fine as far as I'm concerned, but I think that's more of a
gaia call. If there are other reasons for inventing unowned data
stores, please let us know.

Thanks!
Ehsan

[Gene Lian]

Just discussed about this in the WebAPI meeting. Please see the in-line.

> I'm not sure if I understand why you want the data store to not be owned
> by any app, that seems like a non-goal to me.

I originally worried about there could be lots of overhead of doing IPCs
between the b2g process and the messaging app's process. If the RIL is
able to directly save the SMS/MMS into the data store, then if doesn't
make sense to pass the messages to the messaging app and let messaging
app save messages through Data Store API, because it needs two extra
IPCs which is bad.

However if the owner can be the system app, then it wouldn't be an issue
because it must be running on the b2g process.

> The current data store
> implementation expects there to be an owner with a manifest URL, so this
> will ripple its effect through the consumers which is a bad thing.
>
> Can we just make these data stores be owned by the system app?

Agreed. To make our security model more consistent, we had better force
the owner of the data store to be an app.

I think the system app can register for some event handlers through the
existing Messaging API, like onreceived or onsent. Whenever the system
app gets notified about the received/sent messages, it can then update
the Data Store through the Data Store API.

Gene

>
> Cheers,
> Ehsan

[Ehsan Akhgari]

On 2014-04-22, 8:19 PM, Gene Lian wrote:
> Just discussed about this in the WebAPI meeting. Please see the in-line.
>
>> I'm not sure if I understand why you want the data store to not be owned
>> by any app, that seems like a non-goal to me.
>
> I originally worried about there could be lots of overhead of doing IPCs
> between the b2g process and the messaging app's process. If the RIL is
> able to directly save the SMS/MMS into the data store, then if doesn't
> make sense to pass the messages to the messaging app and let messaging
> app save messages through Data Store API, because it needs two extra
> IPCs which is bad.
>
> However if the owner can be the system app, then it wouldn't be an issue
> because it must be running on the b2g process.

Yes, I was thinking of the system app being the owner.

>> The current data store
>> implementation expects there to be an owner with a manifest URL, so this
>> will ripple its effect through the consumers which is a bad thing.
>>
>> Can we just make these data stores be owned by the system app?
>
> Agreed. To make our security model more consistent, we had better force
> the owner of the data store to be an app.
>
> I think the system app can register for some event handlers through the
> existing Messaging API, like onreceived or onsent. Whenever the system
> app gets notified about the received/sent messages, it can then update
> the Data Store through the Data Store API.

Sounds good! Please let us know if you hit any roadblocks!

Cheers,
Ehsan

[JOSE MANUEL CANTERA FONSECA]

________________________________________
De: Kevin Grandon [kgra...@mozilla.com]
Enviado: martes, 22 de abril de 2014 18:10
Para: Ehsan Akhgari
CC: Gene Lian; dev-webapi; Julien Wajsberg; Gary Chen; Bevis Tseng; Jonas Sicking; JOSE MANUEL CANTERA FONSECA; Ehsan Akhgari; Steve Chung
Asunto: Re: [data-store-api] How to provide the *built-in* data store for messages?

The approach we have been doing so far in gaia is working fairly well where we specify a new application as an owner of a datastore. E.g., we will have a "bookmarks" application which is in charge of bookmarks and provides activities to modify/update/remove these bookmarks. I believe something similar is happening with contacts.

It seems to me like we should have a central app which manages the messages datastore. There are pros and cons of making this be the current messages app.

>> For contacts the owner of the datastore will be a different app (a hidden one) than the Contacts App. I think we should follow the same approach here as I stated on the corresponding bug

Best,
Kevin

----- Original Message -----
From: "Ehsan Akhgari" <ehsan....@gmail.com>
To: "Gene Lian" <cl...@mozilla.com>, "dev-webapi" <dev-w...@lists.mozilla.org>
Cc: "Julien Wajsberg" <jwaj...@mozilla.com>, "Gary Chen" <gc...@mozilla.com>, "Bevis Tseng" <bts...@mozilla.com>, "Jonas Sicking" <jo...@sicking.cc>, "JOSE MANUEL CANTERA FONSECA" <jm...@tid.es>, "Ehsan Akhgari" <eh...@mozilla.com>, "Steve Chung" <sch...@mozilla.com>
Sent: Tuesday, April 22, 2014 8:56:26 AM
Subject: Re: [data-store-api] How to provide the *built-in* data store for messages?

I'm not sure if I understand why you want the data store to not be owned

by any app, that seems like a non-goal to me. The current data store

implementation expects there to be an owner with a manifest URL, so this
will ripple its effect through the consumers which is a bad thing.

Can we just make these data stores be owned by the system app?

________________________________

Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at:
http://www.tid.es/ES/PAGINAS/disclaimer.aspx

[Julien Wajsberg]

Le 23/04/2014 02:19, Gene Lian a écrit :
> Just discussed about this in the WebAPI meeting. Please see the in-line.
>

>> I'm not sure if I understand why you want the data store to not be owned
>> by any app, that seems like a non-goal to me.

> I originally worried about there could be lots of overhead of doing IPCs
> between the b2g process and the messaging app's process. If the RIL is
> able to directly save the SMS/MMS into the data store, then if doesn't
> make sense to pass the messages to the messaging app and let messaging
> app save messages through Data Store API, because it needs two extra
> IPCs which is bad.

Is the IPC that bad for this? I mean, receiving 1 message is not really
performance sensitive.

[Gene Lian]

> >> I'm not sure if I understand why you want the data store to not be owned
> >> by any app, that seems like a non-goal to me.
> > I originally worried about there could be lots of overhead of doing IPCs
> > between the b2g process and the messaging app's process. If the RIL is
> > able to directly save the SMS/MMS into the data store, then if doesn't
> > make sense to pass the messages to the messaging app and let messaging
> > app save messages through Data Store API, because it needs two extra
> > IPCs which is bad.
>
> Is the IPC that bad for this? I mean, receiving 1 message is not really
> performance sensitive.

Yeap, not really sensitive. Maybe, the certified messaging app can be
just in charge of owning and managing the built-in SMS/MMS data stores.

I'd prefer running a central app (on the parent process) which is
responsible for managing the data store. In this case, system app is
just fine and doesn't make any IPCs overheads.

Messaging app needs at least two IPCs: one is for the onreceived/onsent
event handler and the other is for calling DataStore.put(), which is
kind resource-wasting to me, even if the messages are not received so
frequently. But the pros are, it's more clear for the messaging app
handling the messages-related stuff, instead of the system app.

Anyway, both works for me, though. ;)

Gene

[Vicamo Yang]

於 4/23/14, 12:23 AM, Ehsan Akhgari 提到:

> That sounds fine as far as I'm concerned, but I think that's more of a
> gaia call. If there are other reasons for inventing unowned data
> stores, please let us know.
>
> Thanks!
> Ehsan
>

Gecko MobileMessaging API hides a lot techinical details from Gaia.

[One very fundamental problem: ACK received SMS T-PDU]

When Gecko receives a incoming SMS T-PDU, it should send an ACK with
fail cause to the network. If that message is successfully processed,
stored into device storage, you send a FCS_OK back; if there is no more
space for this message, send a FCS_MEMORY_CAPACITY_EXCEEDED back;
otherwise, FCS_UNSPECIFIED. Moving data store to Gaia and refuse access
from Gecko follows Gaia should take the responsibility of sending ACK as
well. Gecko has to fire an event to Gaia, make sure Gaia has been
notified, and waits for Gaia's instruction to send ACK back. It's
doable, just add an API |sendAck(id, failCause)|. We have already
onreceived, but how do we make sure that Gaia has been notified and
Messages app is not oom-killed during the process remains a question.
Without sending an ACK back to network, network may stop further SMS
deliverying until next registration.

http://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/RadioInterfaceLayer.js#3092

[Problem-2: no received {Sms,Mms}Message::id field]

Since we don't own the data store, it's obvious that we don't have an
unique id to provide either. It's true we can provide uuid or just
don't give any id any more. However, in the case of a received MMS
notification indication, it could be a dup of previously received one
because of the lack of networking for a long period like powered off
over night. So now you'll get duplicated onreceived events, Gaia has to
re-implement dup detection. That's actually fine until you want
auto-retrieving. There is no auto-retrieving anymore because Gaia has
to decide whether that's a duplicated notification first. We have to
pass all necessary parameters in the retrieveMMS() call as well, not
just a message id. "There is no message id", a talented child says.

http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1796

[Problem-3: SMS delivery status report]

When Gecko receives a SMS status report message, it tries to match a
sending transaction, updates corresponding fields of that sent SMS
message record, and fires either a deliverysuccess or a deliveryerror
event along with a SmsMessage object. However, such delivery report may
return soon in a minute, but it may return in a day, too. We just don't
know if the sent message record is still valid (not deleted), so the
best we can do is fire an event with some kind of identification and
delivery status, but _without_ that message object that used to be.

The same thing also happens on MMS Delivery-Report and Read-Report.
Gaia has to matching them itself.

http://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/RadioInterfaceLayer.js#4177
http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1849
http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1916

[Problem-4: Replaced Short Message Type]

Not actually a problem, but a task to be re-implemented in Gaia. When a
MT short message with one of some special PIDs is received, Gecko
compares its sender to other received messages and replaces the message
body with the new one. Without access to the data store, this just
can't be done in Gecko.

http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MobileMessageDB.jsm#1783

[Problem-5: Cancel retrieving transaction if message deleted]

Not actually a problem, but a task to be re-implemented in Gaia.

mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1796†

[Problem-6: message object in retrieveMMS() error event]

Bug 824717 requests carrying message object in DOMRequest error event as
well. However, for a retrieveMMS() call, the passed parameters are not
enough to construct a MmsMessage back.

https://bugzilla.mozilla.org/show_bug.cgi?id=824717

[Problem-7: SmsMessage, MmsMessage interface changes]

Not actually a problem, but a task to be forfilled by Gaia. We don't
have access to data store, so attributes that may change after be
sent/received or may not be accessible at the time firing events have to
be removed. Following is a list with reasons:

1) id: see problem-2,
2) threadId: conversation grouping is now a task of Gaia,
3) deliveryStatus/deliveryTimestamp, deliveryInfo: see problem-3,
4) deliveryTimestamp: same for deliveryStatus,
5) read: we'll never know this,


All these are not related to conversation grouping. Moving to data
store API is _not_ only re-implementing some phone number matching
mechanism, but also a great change in current API design and control
flow when Gecko don't have access to the data store.

Regards,
Vicamo Yang

[Vicamo Yang]

於 4/23/14, 12:23 AM, Ehsan Akhgari 提到:

> That sounds fine as far as I'm concerned, but I think that's more of a
> gaia call. If there are other reasons for inventing unowned data
> stores, please let us know.
>
> Thanks!
> Ehsan
>

[Vicamo Yang]

於 4/23/14, 12:23 AM, Ehsan Akhgari 提到:

> That sounds fine as far as I'm concerned, but I think that's more of a
> gaia call. If there are other reasons for inventing unowned data
> stores, please let us know.
>
> Thanks!
> Ehsan
>

[Ehsan Akhgari]

On 2014-04-29, 7:57 AM, Vicamo Yang wrote:
> 於 4/23/14, 12:23 AM, Ehsan Akhgari 提到:

>> That sounds fine as far as I'm concerned, but I think that's more of a
>> gaia call. If there are other reasons for inventing unowned data
>> stores, please let us know.
>>
>> Thanks!
>> Ehsan
>>
>

> Gecko MobileMessaging API hides a lot techinical details from Gaia.
>
> [One very fundamental problem: ACK received SMS T-PDU]
>
> When Gecko receives a incoming SMS T-PDU, it should send an ACK with
> fail cause to the network. If that message is successfully processed,
> stored into device storage, you send a FCS_OK back; if there is no more
> space for this message, send a FCS_MEMORY_CAPACITY_EXCEEDED back;
> otherwise, FCS_UNSPECIFIED. Moving data store to Gaia and refuse access
> from Gecko follows Gaia should take the responsibility of sending ACK as
> well. Gecko has to fire an event to Gaia, make sure Gaia has been
> notified, and waits for Gaia's instruction to send ACK back. It's
> doable, just add an API |sendAck(id, failCause)|. We have already
> onreceived, but how do we make sure that Gaia has been notified and
> Messages app is not oom-killed during the process remains a question.
> Without sending an ACK back to network, network may stop further SMS
> deliverying until next registration.
>
> http://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/RadioInterfaceLayer.js#3092

Would this still be an issue if the SMS was written to the data store
directly by Gecko?

> [Problem-2: no received {Sms,Mms}Message::id field]
>
> Since we don't own the data store, it's obvious that we don't have an
> unique id to provide either. It's true we can provide uuid or just
> don't give any id any more. However, in the case of a received MMS
> notification indication, it could be a dup of previously received one
> because of the lack of networking for a long period like powered off
> over night. So now you'll get duplicated onreceived events, Gaia has to
> re-implement dup detection. That's actually fine until you want
> auto-retrieving. There is no auto-retrieving anymore because Gaia has
> to decide whether that's a duplicated notification first. We have to
> pass all necessary parameters in the retrieveMMS() call as well, not
> just a message id. "There is no message id", a talented child says.
>
> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1796

Not sure if I completely understand this issue. Do you have any
suggestions?

> [Problem-3: SMS delivery status report]
>
> When Gecko receives a SMS status report message, it tries to match a
> sending transaction, updates corresponding fields of that sent SMS
> message record, and fires either a deliverysuccess or a deliveryerror
> event along with a SmsMessage object. However, such delivery report may
> return soon in a minute, but it may return in a day, too. We just don't
> know if the sent message record is still valid (not deleted), so the
> best we can do is fire an event with some kind of identification and
> delivery status, but _without_ that message object that used to be.
>
> The same thing also happens on MMS Delivery-Report and Read-Report. Gaia
> has to matching them itself.
>
> http://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/RadioInterfaceLayer.js#4177
>
> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1849
>
> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1916

This just means moving the matching logic to gaia, right? (Please also
note that we can still write to the data store from Gecko if needed.)

> [Problem-4: Replaced Short Message Type]
>
> Not actually a problem, but a task to be re-implemented in Gaia. When a
> MT short message with one of some special PIDs is received, Gecko
> compares its sender to other received messages and replaces the message
> body with the new one. Without access to the data store, this just
> can't be done in Gecko.
>
> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MobileMessageDB.jsm#1783
>
>
> [Problem-5: Cancel retrieving transaction if message deleted]
>
> Not actually a problem, but a task to be re-implemented in Gaia.
>
> mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1796†

It seems that basically doing the writes to the data store in Gaia is a
lot of work. :-)yp

> [Problem-6: message object in retrieveMMS() error event]
>
> Bug 824717 requests carrying message object in DOMRequest error event as
> well. However, for a retrieveMMS() call, the passed parameters are not
> enough to construct a MmsMessage back.
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=824717

Could we not send the ID for a field in the data store instead?

> [Problem-7: SmsMessage, MmsMessage interface changes]
>
> Not actually a problem, but a task to be forfilled by Gaia. We don't
> have access to data store, so attributes that may change after be
> sent/received or may not be accessible at the time firing events have to
> be removed. Following is a list with reasons:
>
> 1) id: see problem-2,
> 2) threadId: conversation grouping is now a task of Gaia,
> 3) deliveryStatus/deliveryTimestamp, deliveryInfo: see problem-3,
> 4) deliveryTimestamp: same for deliveryStatus,
> 5) read: we'll never know this,
>
>
> All these are not related to conversation grouping. Moving to data
> store API is _not_ only re-implementing some phone number matching
> mechanism, but also a great change in current API design and control
> flow when Gecko don't have access to the data store.

Thanks for raising all of these issues, these are really helpful. Note
that there is nothing that prevents us from writing to the data store
from Gecko, and I think that would alleviate a lot of these concerns.
Please let me know if you agree!

Cheers,
Ehsan

[Ehsan Akhgari]

On 2014-04-29, 7:57 AM, Vicamo Yang wrote:
> 於 4/23/14, 12:23 AM, Ehsan Akhgari 提到:

>> That sounds fine as far as I'm concerned, but I think that's more of a
>> gaia call. If there are other reasons for inventing unowned data
>> stores, please let us know.
>>
>> Thanks!
>> Ehsan
>>
>

[Vicamo Yang]

於 4/30/14, 2:58 AM, Ehsan Akhgari 提到:

> On 2014-04-29, 7:57 AM, Vicamo Yang wrote:
>> 於 4/23/14, 12:23 AM, Ehsan Akhgari 提到:
>>> That sounds fine as far as I'm concerned, but I think that's more of a
>>> gaia call. If there are other reasons for inventing unowned data
>>> stores, please let us know.
>>>
>>> Thanks!
>>> Ehsan
>>>

Basically, without access to data store in Gecko, all control flows
involving IndexedDB access have to be replaced by a Gecko -> Gaia ->
Gecko round-trip. Gecko will then only process almost state-less
requests that interact with network, fire an event/return the result to
notify Gaia, and Gaia sends yet another request to continue remaining
process. For example, a MT SMS transaction has to be torn down to:

1) Gecko notifies Gaia new SMS arrival, wait for ACK request,

2) Gaia request Gecko to send ACK with FCS. However, Gecko may
starve if such request is never issued.

A MT MMS transaction becomes:

1-a) Gecko notifies Gaia new notification indication arrival,

2-a) Gaia requst Gecko to send the notification response, and
optionally start the download. However, Gecko may starve if such
request is never issued, so we may prefer plan-B:

1-b) Gecko notifies Gaia new notification indication arrival and
always send DEFERRED response back to network,

2-b) Gaia request Gecko to retrieve if this message should be
automatically retrieved. However, this method may conflict with
carriers' certificate process because we're not following the
specification -- to initiate retrieve process right away when retrieval
preference is set to automatic.

MobileMessaging API will no longer be the one you calls send() and wait
for everything gets done, neither the one you wait for onreceived
silently. Gaia has to aggressively interact with Gecko, or there will
be unexpected consequences.

We can still come up with some solutions, but each with its own new
challenge. That's fine if we decide it's better. All these things have
been done in one way, then they can surely be done in another way. But
I'd like to highlight that it's not only about conversation groups, it's
about how Gecko interacts with network and how Gaia interacts with
Gecko. We'd better fully aware of where we're heading before the first
step.

In contrast, with access to data store in Gecko, all the following
technical details can still be hidden in Gecko. But some public
interfaces will be removed/modified for sure. For example probably
don't need onreceive/onsent/onfailed, because datastore API emits
onchange events for us. It's true there is a slight semantic difference
for Cost Control app, maybe we don't have to talk about this now.
Removing these events seems not be a big deal, the greatest challenge
will be mapping data store changes to IndexedDB that allows us to
perform various query requests.

>> Gecko MobileMessaging API hides a lot techinical details from Gaia.
>>
>> [One very fundamental problem: ACK received SMS T-PDU]
>>
>> When Gecko receives a incoming SMS T-PDU, it should send an ACK with
>> fail cause to the network. If that message is successfully processed,
>> stored into device storage, you send a FCS_OK back; if there is no more
>> space for this message, send a FCS_MEMORY_CAPACITY_EXCEEDED back;
>> otherwise, FCS_UNSPECIFIED. Moving data store to Gaia and refuse access
>> from Gecko follows Gaia should take the responsibility of sending ACK as
>> well. Gecko has to fire an event to Gaia, make sure Gaia has been
>> notified, and waits for Gaia's instruction to send ACK back. It's
>> doable, just add an API |sendAck(id, failCause)|. We have already
>> onreceived, but how do we make sure that Gaia has been notified and
>> Messages app is not oom-killed during the process remains a question.
>> Without sending an ACK back to network, network may stop further SMS
>> deliverying until next registration.
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/RadioInterfaceLayer.js#3092
>>
>
> Would this still be an issue if the SMS was written to the data store
> directly by Gecko?

No. We just need to assure the new arrival can be written to disk
successfully.

>> [Problem-2: no received {Sms,Mms}Message::id field]
>>
>> Since we don't own the data store, it's obvious that we don't have an
>> unique id to provide either. It's true we can provide uuid or just
>> don't give any id any more. However, in the case of a received MMS
>> notification indication, it could be a dup of previously received one
>> because of the lack of networking for a long period like powered off
>> over night. So now you'll get duplicated onreceived events, Gaia has to
>> re-implement dup detection. That's actually fine until you want
>> auto-retrieving. There is no auto-retrieving anymore because Gaia has
>> to decide whether that's a duplicated notification first. We have to
>> pass all necessary parameters in the retrieveMMS() call as well, not
>> just a message id. "There is no message id", a talented child says.
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1796
>>
>
> Not sure if I completely understand this issue. Do you have any
> suggestions?

Almost same as above, we need to save the MMS notification indication on
disk before going on to retrieve automatically or sending notification
response. Or, this has to be torn down to some two-step process as
illustrated in 1-a), 2-a), 1-b), 2-b).

>> [Problem-3: SMS delivery status report]
>>
>> When Gecko receives a SMS status report message, it tries to match a
>> sending transaction, updates corresponding fields of that sent SMS
>> message record, and fires either a deliverysuccess or a deliveryerror
>> event along with a SmsMessage object. However, such delivery report may
>> return soon in a minute, but it may return in a day, too. We just don't
>> know if the sent message record is still valid (not deleted), so the
>> best we can do is fire an event with some kind of identification and
>> delivery status, but _without_ that message object that used to be.
>>
>> The same thing also happens on MMS Delivery-Report and Read-Report. Gaia
>> has to matching them itself.
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/RadioInterfaceLayer.js#4177
>>
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1849
>>
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1916
>>
>
> This just means moving the matching logic to gaia, right?

Just? No. It also follows we don't have convenient ondeliverysuccess,
ondeliveryerror events for everyone interested in. These events,
because of the lack of message object and message id, have to be further
processed in _every_ listener.

> (Please also
> note that we can still write to the data store from Gecko if needed.)
>
>> [Problem-4: Replaced Short Message Type]
>>
>> Not actually a problem, but a task to be re-implemented in Gaia. When a
>> MT short message with one of some special PIDs is received, Gecko
>> compares its sender to other received messages and replaces the message
>> body with the new one. Without access to the data store, this just
>> can't be done in Gecko.
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MobileMessageDB.jsm#1783
>>
>>
>>
>> [Problem-5: Cancel retrieving transaction if message deleted]
>>
>> Not actually a problem, but a task to be re-implemented in Gaia.
>>
>> mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1796†
>>
>
> It seems that basically doing the writes to the data store in Gaia is a
> lot of work. :-)yp

Yup.

>> [Problem-6: message object in retrieveMMS() error event]
>>
>> Bug 824717 requests carrying message object in DOMRequest error event as
>> well. However, for a retrieveMMS() call, the passed parameters are not
>> enough to construct a MmsMessage back.
>>
>> https://bugzilla.mozilla.org/show_bug.cgi?id=824717
>
> Could we not send the ID for a field in the data store instead?

Yes, that's probably the only way, although it's not as convenient as a
message object.

>> [Problem-7: SmsMessage, MmsMessage interface changes]
>>
>> Not actually a problem, but a task to be forfilled by Gaia. We don't
>> have access to data store, so attributes that may change after be
>> sent/received or may not be accessible at the time firing events have to
>> be removed. Following is a list with reasons:
>>
>> 1) id: see problem-2,
>> 2) threadId: conversation grouping is now a task of Gaia,
>> 3) deliveryStatus/deliveryTimestamp, deliveryInfo: see problem-3,
>> 4) deliveryTimestamp: same for deliveryStatus,
>> 5) read: we'll never know this,
>>
>>
>> All these are not related to conversation grouping. Moving to data
>> store API is _not_ only re-implementing some phone number matching
>> mechanism, but also a great change in current API design and control
>> flow when Gecko don't have access to the data store.
>
> Thanks for raising all of these issues, these are really helpful. Note
> that there is nothing that prevents us from writing to the data store
> from Gecko, and I think that would alleviate a lot of these concerns.
> Please let me know if you agree!

I do. Even if we decide owning data store in apps only is somehow a
better way, then it is a better way. I also need some time to construct
a full picture what will we need when data store is not owned by Gecko.
One trivial thing is SMS/MMS will no longer work without an
appropriately implemented messaging app. "Gecko depends on Gaia's
functionality" is one thing I worry about most.

> Cheers,
> Ehsan

Regards,
Vicamo Yang

[Vicamo Yang]

於 4/30/14, 2:58 AM, Ehsan Akhgari 提到:

> On 2014-04-29, 7:57 AM, Vicamo Yang wrote:
>> 於 4/23/14, 12:23 AM, Ehsan Akhgari 提到:
>>> That sounds fine as far as I'm concerned, but I think that's more of a
>>> gaia call. If there are other reasons for inventing unowned data
>>> stores, please let us know.
>>>
>>> Thanks!
>>> Ehsan
>>>

>> Gecko MobileMessaging API hides a lot techinical details from Gaia.
>>
>> [One very fundamental problem: ACK received SMS T-PDU]
>>
>> When Gecko receives a incoming SMS T-PDU, it should send an ACK with
>> fail cause to the network. If that message is successfully processed,
>> stored into device storage, you send a FCS_OK back; if there is no more
>> space for this message, send a FCS_MEMORY_CAPACITY_EXCEEDED back;
>> otherwise, FCS_UNSPECIFIED. Moving data store to Gaia and refuse access
>> from Gecko follows Gaia should take the responsibility of sending ACK as
>> well. Gecko has to fire an event to Gaia, make sure Gaia has been
>> notified, and waits for Gaia's instruction to send ACK back. It's
>> doable, just add an API |sendAck(id, failCause)|. We have already
>> onreceived, but how do we make sure that Gaia has been notified and
>> Messages app is not oom-killed during the process remains a question.
>> Without sending an ACK back to network, network may stop further SMS
>> deliverying until next registration.
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/RadioInterfaceLayer.js#3092
>>
>
> Would this still be an issue if the SMS was written to the data store
> directly by Gecko?

No. We just need to assure the new arrival can be written to disk
successfully.

>> [Problem-2: no received {Sms,Mms}Message::id field]
>>
>> Since we don't own the data store, it's obvious that we don't have an
>> unique id to provide either. It's true we can provide uuid or just
>> don't give any id any more. However, in the case of a received MMS
>> notification indication, it could be a dup of previously received one
>> because of the lack of networking for a long period like powered off
>> over night. So now you'll get duplicated onreceived events, Gaia has to
>> re-implement dup detection. That's actually fine until you want
>> auto-retrieving. There is no auto-retrieving anymore because Gaia has
>> to decide whether that's a duplicated notification first. We have to
>> pass all necessary parameters in the retrieveMMS() call as well, not
>> just a message id. "There is no message id", a talented child says.
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1796
>>
>
> Not sure if I completely understand this issue. Do you have any
> suggestions?

Almost same as above, we need to save the MMS notification indication on
disk before going on to retrieve automatically or sending notification
response. Or, this has to be torn down to some two-step process as
illustrated in 1-a), 2-a), 1-b), 2-b).

>> [Problem-3: SMS delivery status report]
>>
>> When Gecko receives a SMS status report message, it tries to match a
>> sending transaction, updates corresponding fields of that sent SMS
>> message record, and fires either a deliverysuccess or a deliveryerror
>> event along with a SmsMessage object. However, such delivery report may
>> return soon in a minute, but it may return in a day, too. We just don't
>> know if the sent message record is still valid (not deleted), so the
>> best we can do is fire an event with some kind of identification and
>> delivery status, but _without_ that message object that used to be.
>>
>> The same thing also happens on MMS Delivery-Report and Read-Report. Gaia
>> has to matching them itself.
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/RadioInterfaceLayer.js#4177
>>
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1849
>>
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1916
>>
>
> This just means moving the matching logic to gaia, right?

Just? No. It also follows we don't have convenient ondeliverysuccess,
ondeliveryerror events for everyone interested in. These events,
because of the lack of message object and message id, have to be further
processed in _every_ listener.

> (Please also
> note that we can still write to the data store from Gecko if needed.)
>
>> [Problem-4: Replaced Short Message Type]
>>
>> Not actually a problem, but a task to be re-implemented in Gaia. When a
>> MT short message with one of some special PIDs is received, Gecko
>> compares its sender to other received messages and replaces the message
>> body with the new one. Without access to the data store, this just
>> can't be done in Gecko.
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MobileMessageDB.jsm#1783
>>
>>
>>
>> [Problem-5: Cancel retrieving transaction if message deleted]
>>
>> Not actually a problem, but a task to be re-implemented in Gaia.
>>
>> mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1796†
>>
>
> It seems that basically doing the writes to the data store in Gaia is a
> lot of work. :-)yp

Yup.

>> [Problem-6: message object in retrieveMMS() error event]
>>
>> Bug 824717 requests carrying message object in DOMRequest error event as
>> well. However, for a retrieveMMS() call, the passed parameters are not
>> enough to construct a MmsMessage back.
>>
>> https://bugzilla.mozilla.org/show_bug.cgi?id=824717
>
> Could we not send the ID for a field in the data store instead?

Yes, that's probably the only way, although it's not as convenient as a
message object.

>> [Problem-7: SmsMessage, MmsMessage interface changes]
>>
>> Not actually a problem, but a task to be forfilled by Gaia. We don't
>> have access to data store, so attributes that may change after be
>> sent/received or may not be accessible at the time firing events have to
>> be removed. Following is a list with reasons:
>>
>> 1) id: see problem-2,
>> 2) threadId: conversation grouping is now a task of Gaia,
>> 3) deliveryStatus/deliveryTimestamp, deliveryInfo: see problem-3,
>> 4) deliveryTimestamp: same for deliveryStatus,
>> 5) read: we'll never know this,
>>
>>
>> All these are not related to conversation grouping. Moving to data
>> store API is _not_ only re-implementing some phone number matching
>> mechanism, but also a great change in current API design and control
>> flow when Gecko don't have access to the data store.
>
> Thanks for raising all of these issues, these are really helpful. Note
> that there is nothing that prevents us from writing to the data store
> from Gecko, and I think that would alleviate a lot of these concerns.
> Please let me know if you agree!

[Vicamo Yang]

於 4/30/14, 2:58 AM, Ehsan Akhgari 提到:

> On 2014-04-29, 7:57 AM, Vicamo Yang wrote:
>> 於 4/23/14, 12:23 AM, Ehsan Akhgari 提到:
>>> That sounds fine as far as I'm concerned, but I think that's more of a
>>> gaia call. If there are other reasons for inventing unowned data
>>> stores, please let us know.
>>>
>>> Thanks!
>>> Ehsan
>>>

>> Gecko MobileMessaging API hides a lot techinical details from Gaia.
>>
>> [One very fundamental problem: ACK received SMS T-PDU]
>>
>> When Gecko receives a incoming SMS T-PDU, it should send an ACK with
>> fail cause to the network. If that message is successfully processed,
>> stored into device storage, you send a FCS_OK back; if there is no more
>> space for this message, send a FCS_MEMORY_CAPACITY_EXCEEDED back;
>> otherwise, FCS_UNSPECIFIED. Moving data store to Gaia and refuse access
>> from Gecko follows Gaia should take the responsibility of sending ACK as
>> well. Gecko has to fire an event to Gaia, make sure Gaia has been
>> notified, and waits for Gaia's instruction to send ACK back. It's
>> doable, just add an API |sendAck(id, failCause)|. We have already
>> onreceived, but how do we make sure that Gaia has been notified and
>> Messages app is not oom-killed during the process remains a question.
>> Without sending an ACK back to network, network may stop further SMS
>> deliverying until next registration.
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/RadioInterfaceLayer.js#3092
>>
>
> Would this still be an issue if the SMS was written to the data store
> directly by Gecko?

No. We just need to assure the new arrival can be written to disk
successfully.

>> [Problem-2: no received {Sms,Mms}Message::id field]
>>
>> Since we don't own the data store, it's obvious that we don't have an
>> unique id to provide either. It's true we can provide uuid or just
>> don't give any id any more. However, in the case of a received MMS
>> notification indication, it could be a dup of previously received one
>> because of the lack of networking for a long period like powered off
>> over night. So now you'll get duplicated onreceived events, Gaia has to
>> re-implement dup detection. That's actually fine until you want
>> auto-retrieving. There is no auto-retrieving anymore because Gaia has
>> to decide whether that's a duplicated notification first. We have to
>> pass all necessary parameters in the retrieveMMS() call as well, not
>> just a message id. "There is no message id", a talented child says.
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1796
>>
>
> Not sure if I completely understand this issue. Do you have any
> suggestions?

Almost same as above, we need to save the MMS notification indication on
disk before going on to retrieve automatically or sending notification
response. Or, this has to be torn down to some two-step process as
illustrated in 1-a), 2-a), 1-b), 2-b).

>> [Problem-3: SMS delivery status report]
>>
>> When Gecko receives a SMS status report message, it tries to match a
>> sending transaction, updates corresponding fields of that sent SMS
>> message record, and fires either a deliverysuccess or a deliveryerror
>> event along with a SmsMessage object. However, such delivery report may
>> return soon in a minute, but it may return in a day, too. We just don't
>> know if the sent message record is still valid (not deleted), so the
>> best we can do is fire an event with some kind of identification and
>> delivery status, but _without_ that message object that used to be.
>>
>> The same thing also happens on MMS Delivery-Report and Read-Report. Gaia
>> has to matching them itself.
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/RadioInterfaceLayer.js#4177
>>
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1849
>>
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1916
>>
>
> This just means moving the matching logic to gaia, right?

Just? No. It also follows we don't have convenient ondeliverysuccess,
ondeliveryerror events for everyone interested in. These events,
because of the lack of message object and message id, have to be further
processed in _every_ listener.

> (Please also
> note that we can still write to the data store from Gecko if needed.)
>
>> [Problem-4: Replaced Short Message Type]
>>
>> Not actually a problem, but a task to be re-implemented in Gaia. When a
>> MT short message with one of some special PIDs is received, Gecko
>> compares its sender to other received messages and replaces the message
>> body with the new one. Without access to the data store, this just
>> can't be done in Gecko.
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MobileMessageDB.jsm#1783
>>
>>
>>
>> [Problem-5: Cancel retrieving transaction if message deleted]
>>
>> Not actually a problem, but a task to be re-implemented in Gaia.
>>
>> mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1796†
>>
>
> It seems that basically doing the writes to the data store in Gaia is a
> lot of work. :-)yp

Yup.

>> [Problem-6: message object in retrieveMMS() error event]
>>
>> Bug 824717 requests carrying message object in DOMRequest error event as
>> well. However, for a retrieveMMS() call, the passed parameters are not
>> enough to construct a MmsMessage back.
>>
>> https://bugzilla.mozilla.org/show_bug.cgi?id=824717
>
> Could we not send the ID for a field in the data store instead?

Yes, that's probably the only way, although it's not as convenient as a
message object.

>> [Problem-7: SmsMessage, MmsMessage interface changes]
>>
>> Not actually a problem, but a task to be forfilled by Gaia. We don't
>> have access to data store, so attributes that may change after be
>> sent/received or may not be accessible at the time firing events have to
>> be removed. Following is a list with reasons:
>>
>> 1) id: see problem-2,
>> 2) threadId: conversation grouping is now a task of Gaia,
>> 3) deliveryStatus/deliveryTimestamp, deliveryInfo: see problem-3,
>> 4) deliveryTimestamp: same for deliveryStatus,
>> 5) read: we'll never know this,
>>
>>
>> All these are not related to conversation grouping. Moving to data
>> store API is _not_ only re-implementing some phone number matching
>> mechanism, but also a great change in current API design and control
>> flow when Gecko don't have access to the data store.
>
> Thanks for raising all of these issues, these are really helpful. Note
> that there is nothing that prevents us from writing to the data store
> from Gecko, and I think that would alleviate a lot of these concerns.
> Please let me know if you agree!

[Vicamo Yang]

於 4/30/14, 2:58 AM, Ehsan Akhgari 提到:

> On 2014-04-29, 7:57 AM, Vicamo Yang wrote:
>> 於 4/23/14, 12:23 AM, Ehsan Akhgari 提到:
>>> That sounds fine as far as I'm concerned, but I think that's more of a
>>> gaia call. If there are other reasons for inventing unowned data
>>> stores, please let us know.
>>>
>>> Thanks!
>>> Ehsan
>>>

>> Gecko MobileMessaging API hides a lot techinical details from Gaia.
>>
>> [One very fundamental problem: ACK received SMS T-PDU]
>>
>> When Gecko receives a incoming SMS T-PDU, it should send an ACK with
>> fail cause to the network. If that message is successfully processed,
>> stored into device storage, you send a FCS_OK back; if there is no more
>> space for this message, send a FCS_MEMORY_CAPACITY_EXCEEDED back;
>> otherwise, FCS_UNSPECIFIED. Moving data store to Gaia and refuse access
>> from Gecko follows Gaia should take the responsibility of sending ACK as
>> well. Gecko has to fire an event to Gaia, make sure Gaia has been
>> notified, and waits for Gaia's instruction to send ACK back. It's
>> doable, just add an API |sendAck(id, failCause)|. We have already
>> onreceived, but how do we make sure that Gaia has been notified and
>> Messages app is not oom-killed during the process remains a question.
>> Without sending an ACK back to network, network may stop further SMS
>> deliverying until next registration.
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/RadioInterfaceLayer.js#3092
>>
>
> Would this still be an issue if the SMS was written to the data store
> directly by Gecko?

No. We just need to assure the new arrival can be written to disk
successfully.

>> [Problem-2: no received {Sms,Mms}Message::id field]
>>
>> Since we don't own the data store, it's obvious that we don't have an
>> unique id to provide either. It's true we can provide uuid or just
>> don't give any id any more. However, in the case of a received MMS
>> notification indication, it could be a dup of previously received one
>> because of the lack of networking for a long period like powered off
>> over night. So now you'll get duplicated onreceived events, Gaia has to
>> re-implement dup detection. That's actually fine until you want
>> auto-retrieving. There is no auto-retrieving anymore because Gaia has
>> to decide whether that's a duplicated notification first. We have to
>> pass all necessary parameters in the retrieveMMS() call as well, not
>> just a message id. "There is no message id", a talented child says.
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1796
>>
>
> Not sure if I completely understand this issue. Do you have any
> suggestions?

Almost same as above, we need to save the MMS notification indication on
disk before going on to retrieve automatically or sending notification
response. Or, this has to be torn down to some two-step process as
illustrated in 1-a), 2-a), 1-b), 2-b).

>> [Problem-3: SMS delivery status report]
>>
>> When Gecko receives a SMS status report message, it tries to match a
>> sending transaction, updates corresponding fields of that sent SMS
>> message record, and fires either a deliverysuccess or a deliveryerror
>> event along with a SmsMessage object. However, such delivery report may
>> return soon in a minute, but it may return in a day, too. We just don't
>> know if the sent message record is still valid (not deleted), so the
>> best we can do is fire an event with some kind of identification and
>> delivery status, but _without_ that message object that used to be.
>>
>> The same thing also happens on MMS Delivery-Report and Read-Report. Gaia
>> has to matching them itself.
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/RadioInterfaceLayer.js#4177
>>
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1849
>>
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1916
>>
>
> This just means moving the matching logic to gaia, right?

Just? No. It also follows we don't have convenient ondeliverysuccess,
ondeliveryerror events for everyone interested in. These events,
because of the lack of message object and message id, have to be further
processed in _every_ listener.

> (Please also
> note that we can still write to the data store from Gecko if needed.)
>
>> [Problem-4: Replaced Short Message Type]
>>
>> Not actually a problem, but a task to be re-implemented in Gaia. When a
>> MT short message with one of some special PIDs is received, Gecko
>> compares its sender to other received messages and replaces the message
>> body with the new one. Without access to the data store, this just
>> can't be done in Gecko.
>>
>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MobileMessageDB.jsm#1783
>>
>>
>>
>> [Problem-5: Cancel retrieving transaction if message deleted]
>>
>> Not actually a problem, but a task to be re-implemented in Gaia.
>>
>> mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1796†
>>
>
> It seems that basically doing the writes to the data store in Gaia is a
> lot of work. :-)yp

Yup.

>> [Problem-6: message object in retrieveMMS() error event]
>>
>> Bug 824717 requests carrying message object in DOMRequest error event as
>> well. However, for a retrieveMMS() call, the passed parameters are not
>> enough to construct a MmsMessage back.
>>
>> https://bugzilla.mozilla.org/show_bug.cgi?id=824717
>
> Could we not send the ID for a field in the data store instead?

Yes, that's probably the only way, although it's not as convenient as a
message object.

>> [Problem-7: SmsMessage, MmsMessage interface changes]
>>
>> Not actually a problem, but a task to be forfilled by Gaia. We don't
>> have access to data store, so attributes that may change after be
>> sent/received or may not be accessible at the time firing events have to
>> be removed. Following is a list with reasons:
>>
>> 1) id: see problem-2,
>> 2) threadId: conversation grouping is now a task of Gaia,
>> 3) deliveryStatus/deliveryTimestamp, deliveryInfo: see problem-3,
>> 4) deliveryTimestamp: same for deliveryStatus,
>> 5) read: we'll never know this,
>>
>>
>> All these are not related to conversation grouping. Moving to data
>> store API is _not_ only re-implementing some phone number matching
>> mechanism, but also a great change in current API design and control
>> flow when Gecko don't have access to the data store.
>
> Thanks for raising all of these issues, these are really helpful. Note
> that there is nothing that prevents us from writing to the data store
> from Gecko, and I think that would alleviate a lot of these concerns.
> Please let me know if you agree!

[Gene Lian]

> It seems that basically doing the writes to the data store in Gaia is a
> lot of work. :-)yp

Thanks for Vicamo brining this up. Indeed, that would be less painful
if the Gecko can directly modify the data store. Otherwise, we have to
put lots of messaging-specific protocol logic in Gaia.

I think system app can still *own* the data store (claimed in manifest)
so the owner of the data store can still be the manifest URL of the
system app, which can ensure our permission model to be consistent.
However, system app won't directly read or write the data store in any
way, Gecko will do it instead.

To do this, we need to expose some internal XPCOM service for Gecko
to read/write the data store.

Gene

[Gene Lian]

[Ehsan Akhgari]

On 2014-04-29, 11:33 PM, Vicamo Yang wrote:
> 於 4/30/14, 2:58 AM, Ehsan Akhgari 提到:

>> On 2014-04-29, 7:57 AM, Vicamo Yang wrote:
>>> 於 4/23/14, 12:23 AM, Ehsan Akhgari 提到:
>>>> That sounds fine as far as I'm concerned, but I think that's more of a
>>>> gaia call. If there are other reasons for inventing unowned data
>>>> stores, please let us know.
>>>>
>>>> Thanks!
>>>> Ehsan
>>>>
>

> Basically, without access to data store in Gecko, all control flows
> involving IndexedDB access have to be replaced by a Gecko -> Gaia ->
> Gecko round-trip. Gecko will then only process almost state-less
> requests that interact with network, fire an event/return the result to
> notify Gaia, and Gaia sends yet another request to continue remaining
> process. For example, a MT SMS transaction has to be torn down to:
>
> 1) Gecko notifies Gaia new SMS arrival, wait for ACK request,
>
> 2) Gaia request Gecko to send ACK with FCS. However, Gecko may
> starve if such request is never issued.
>
> A MT MMS transaction becomes:
>
> 1-a) Gecko notifies Gaia new notification indication arrival,
>
> 2-a) Gaia requst Gecko to send the notification response, and
> optionally start the download. However, Gecko may starve if such
> request is never issued, so we may prefer plan-B:
>
> 1-b) Gecko notifies Gaia new notification indication arrival and
> always send DEFERRED response back to network,
>
> 2-b) Gaia request Gecko to retrieve if this message should be
> automatically retrieved. However, this method may conflict with
> carriers' certificate process because we're not following the
> specification -- to initiate retrieve process right away when retrieval
> preference is set to automatic.

Right, and we don't need to do any of the above since we can directly
write to the DataStore from Gecko. :-)

> MobileMessaging API will no longer be the one you calls send() and wait
> for everything gets done, neither the one you wait for onreceived
> silently. Gaia has to aggressively interact with Gecko, or there will
> be unexpected consequences.
>
> We can still come up with some solutions, but each with its own new
> challenge. That's fine if we decide it's better. All these things have
> been done in one way, then they can surely be done in another way. But
> I'd like to highlight that it's not only about conversation groups, it's
> about how Gecko interacts with network and how Gaia interacts with
> Gecko. We'd better fully aware of where we're heading before the first
> step.

Agreed.

> In contrast, with access to data store in Gecko, all the following
> technical details can still be hidden in Gecko. But some public
> interfaces will be removed/modified for sure. For example probably
> don't need onreceive/onsent/onfailed, because datastore API emits
> onchange events for us. It's true there is a slight semantic difference
> for Cost Control app, maybe we don't have to talk about this now.
> Removing these events seems not be a big deal, the greatest challenge
> will be mapping data store changes to IndexedDB that allows us to
> perform various query requests.

So one thing to watch out for is the querying part. Currently DataStore
doesn't give you very useful querying primitives (for example, no
indices). For those, we've been relying on people to pull the data into
a local IDB. But we in fact had a meeting just today about the
DataStore API in general <https://etherpad.mozilla.org/datastore> and it
seems like we're going to make DataStore more of a shared IDB now, but
there are no good details on this yet. So perhaps it might make sense
to hold off on making any changes here until those bits are well understood.

>>> Gecko MobileMessaging API hides a lot techinical details from Gaia.
>>>
>>> [One very fundamental problem: ACK received SMS T-PDU]
>>>
>>> When Gecko receives a incoming SMS T-PDU, it should send an ACK with
>>> fail cause to the network. If that message is successfully processed,
>>> stored into device storage, you send a FCS_OK back; if there is no more
>>> space for this message, send a FCS_MEMORY_CAPACITY_EXCEEDED back;
>>> otherwise, FCS_UNSPECIFIED. Moving data store to Gaia and refuse access
>>> from Gecko follows Gaia should take the responsibility of sending ACK as
>>> well. Gecko has to fire an event to Gaia, make sure Gaia has been
>>> notified, and waits for Gaia's instruction to send ACK back. It's
>>> doable, just add an API |sendAck(id, failCause)|. We have already
>>> onreceived, but how do we make sure that Gaia has been notified and
>>> Messages app is not oom-killed during the process remains a question.
>>> Without sending an ACK back to network, network may stop further SMS
>>> deliverying until next registration.
>>>
>>> http://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/RadioInterfaceLayer.js#3092
>>>
>>>
>>
>> Would this still be an issue if the SMS was written to the data store
>> directly by Gecko?
>

> No. We just need to assure the new arrival can be written to disk
> successfully.

For that you should be able to use wait for the promise to be resolved.

>>> [Problem-2: no received {Sms,Mms}Message::id field]
>>>
>>> Since we don't own the data store, it's obvious that we don't have an
>>> unique id to provide either. It's true we can provide uuid or just
>>> don't give any id any more. However, in the case of a received MMS
>>> notification indication, it could be a dup of previously received one
>>> because of the lack of networking for a long period like powered off
>>> over night. So now you'll get duplicated onreceived events, Gaia has to
>>> re-implement dup detection. That's actually fine until you want
>>> auto-retrieving. There is no auto-retrieving anymore because Gaia has
>>> to decide whether that's a duplicated notification first. We have to
>>> pass all necessary parameters in the retrieveMMS() call as well, not
>>> just a message id. "There is no message id", a talented child says.
>>>
>>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1796
>>>
>>>
>>
>> Not sure if I completely understand this issue. Do you have any
>> suggestions?
>

> Almost same as above, we need to save the MMS notification indication on
> disk before going on to retrieve automatically or sending notification
> response. Or, this has to be torn down to some two-step process as
> illustrated in 1-a), 2-a), 1-b), 2-b).

OK, so does accessing the DataStore directly from Gecko mostly solve
this issue too then?

>>> [Problem-3: SMS delivery status report]
>>>
>>> When Gecko receives a SMS status report message, it tries to match a
>>> sending transaction, updates corresponding fields of that sent SMS
>>> message record, and fires either a deliverysuccess or a deliveryerror
>>> event along with a SmsMessage object. However, such delivery report may
>>> return soon in a minute, but it may return in a day, too. We just don't
>>> know if the sent message record is still valid (not deleted), so the
>>> best we can do is fire an event with some kind of identification and
>>> delivery status, but _without_ that message object that used to be.
>>>
>>> The same thing also happens on MMS Delivery-Report and Read-Report. Gaia
>>> has to matching them itself.
>>>
>>> http://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/RadioInterfaceLayer.js#4177
>>>
>>>
>>>
>>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1849
>>>
>>>
>>>
>>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1916
>>>
>>>
>>
>> This just means moving the matching logic to gaia, right?
>

> Just? No. It also follows we don't have convenient ondeliverysuccess,
> ondeliveryerror events for everyone interested in. These events,
> because of the lack of message object and message id, have to be further
> processed in _every_ listener.

Hmm, clearly I don't understand all of the details here. :/

>> (Please also
>> note that we can still write to the data store from Gecko if needed.)
>>
>>> [Problem-4: Replaced Short Message Type]
>>>
>>> Not actually a problem, but a task to be re-implemented in Gaia. When a
>>> MT short message with one of some special PIDs is received, Gecko
>>> compares its sender to other received messages and replaces the message
>>> body with the new one. Without access to the data store, this just
>>> can't be done in Gecko.
>>>
>>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MobileMessageDB.jsm#1783
>>>
>>>
>>>
>>>
>>> [Problem-5: Cancel retrieving transaction if message deleted]
>>>
>>> Not actually a problem, but a task to be re-implemented in Gaia.
>>>
>>> mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1796†
>>>
>>>
>>
>> It seems that basically doing the writes to the data store in Gaia is a
>> lot of work. :-)yp
>

> Yup.

>
>>> [Problem-6: message object in retrieveMMS() error event]
>>>
>>> Bug 824717 requests carrying message object in DOMRequest error event as
>>> well. However, for a retrieveMMS() call, the passed parameters are not
>>> enough to construct a MmsMessage back.
>>>
>>> https://bugzilla.mozilla.org/show_bug.cgi?id=824717
>>
>> Could we not send the ID for a field in the data store instead?
>

> Yes, that's probably the only way, although it's not as convenient as a
> message object.
>

>>> [Problem-7: SmsMessage, MmsMessage interface changes]
>>>
>>> Not actually a problem, but a task to be forfilled by Gaia. We don't
>>> have access to data store, so attributes that may change after be
>>> sent/received or may not be accessible at the time firing events have to
>>> be removed. Following is a list with reasons:
>>>
>>> 1) id: see problem-2,
>>> 2) threadId: conversation grouping is now a task of Gaia,
>>> 3) deliveryStatus/deliveryTimestamp, deliveryInfo: see problem-3,
>>> 4) deliveryTimestamp: same for deliveryStatus,
>>> 5) read: we'll never know this,
>>>
>>>
>>> All these are not related to conversation grouping. Moving to data
>>> store API is _not_ only re-implementing some phone number matching
>>> mechanism, but also a great change in current API design and control
>>> flow when Gecko don't have access to the data store.
>>
>> Thanks for raising all of these issues, these are really helpful. Note
>> that there is nothing that prevents us from writing to the data store
>> from Gecko, and I think that would alleviate a lot of these concerns.
>> Please let me know if you agree!
>

> I do. Even if we decide owning data store in apps only is somehow a
> better way, then it is a better way. I also need some time to construct
> a full picture what will we need when data store is not owned by Gecko.
> One trivial thing is SMS/MMS will no longer work without an
> appropriately implemented messaging app. "Gecko depends on Gaia's
> functionality" is one thing I worry about most.

Note that the ownership is purely a matter of who claims to be an owner
of the DataStore in the application manifest. That does affect the
security checks in place for getting access to a DataStore but it
doesn't tie our hands in any way for manipulating the DataStore from
inside Gecko.

Cheers,
Ehsan

[Ehsan Akhgari]

On 2014-04-29, 11:33 PM, Vicamo Yang wrote:
> 於 4/30/14, 2:58 AM, Ehsan Akhgari 提到:

>> On 2014-04-29, 7:57 AM, Vicamo Yang wrote:
>>> 於 4/23/14, 12:23 AM, Ehsan Akhgari 提到:
>>>> That sounds fine as far as I'm concerned, but I think that's more of a
>>>> gaia call. If there are other reasons for inventing unowned data
>>>> stores, please let us know.
>>>>
>>>> Thanks!
>>>> Ehsan
>>>>
>

>>> Gecko MobileMessaging API hides a lot techinical details from Gaia.
>>>
>>> [One very fundamental problem: ACK received SMS T-PDU]
>>>
>>> When Gecko receives a incoming SMS T-PDU, it should send an ACK with
>>> fail cause to the network. If that message is successfully processed,
>>> stored into device storage, you send a FCS_OK back; if there is no more
>>> space for this message, send a FCS_MEMORY_CAPACITY_EXCEEDED back;
>>> otherwise, FCS_UNSPECIFIED. Moving data store to Gaia and refuse access
>>> from Gecko follows Gaia should take the responsibility of sending ACK as
>>> well. Gecko has to fire an event to Gaia, make sure Gaia has been
>>> notified, and waits for Gaia's instruction to send ACK back. It's
>>> doable, just add an API |sendAck(id, failCause)|. We have already
>>> onreceived, but how do we make sure that Gaia has been notified and
>>> Messages app is not oom-killed during the process remains a question.
>>> Without sending an ACK back to network, network may stop further SMS
>>> deliverying until next registration.
>>>
>>> http://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/RadioInterfaceLayer.js#3092
>>>
>>>
>>
>> Would this still be an issue if the SMS was written to the data store
>> directly by Gecko?
>

> No. We just need to assure the new arrival can be written to disk
> successfully.

For that you should be able to use wait for the promise to be resolved.

>>> [Problem-2: no received {Sms,Mms}Message::id field]
>>>
>>> Since we don't own the data store, it's obvious that we don't have an
>>> unique id to provide either. It's true we can provide uuid or just
>>> don't give any id any more. However, in the case of a received MMS
>>> notification indication, it could be a dup of previously received one
>>> because of the lack of networking for a long period like powered off
>>> over night. So now you'll get duplicated onreceived events, Gaia has to
>>> re-implement dup detection. That's actually fine until you want
>>> auto-retrieving. There is no auto-retrieving anymore because Gaia has
>>> to decide whether that's a duplicated notification first. We have to
>>> pass all necessary parameters in the retrieveMMS() call as well, not
>>> just a message id. "There is no message id", a talented child says.
>>>
>>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1796
>>>
>>>
>>
>> Not sure if I completely understand this issue. Do you have any
>> suggestions?
>

> Almost same as above, we need to save the MMS notification indication on
> disk before going on to retrieve automatically or sending notification
> response. Or, this has to be torn down to some two-step process as
> illustrated in 1-a), 2-a), 1-b), 2-b).

OK, so does accessing the DataStore directly from Gecko mostly solve
this issue too then?

>>> [Problem-3: SMS delivery status report]
>>>
>>> When Gecko receives a SMS status report message, it tries to match a
>>> sending transaction, updates corresponding fields of that sent SMS
>>> message record, and fires either a deliverysuccess or a deliveryerror
>>> event along with a SmsMessage object. However, such delivery report may
>>> return soon in a minute, but it may return in a day, too. We just don't
>>> know if the sent message record is still valid (not deleted), so the
>>> best we can do is fire an event with some kind of identification and
>>> delivery status, but _without_ that message object that used to be.
>>>
>>> The same thing also happens on MMS Delivery-Report and Read-Report. Gaia
>>> has to matching them itself.
>>>
>>> http://mxr.mozilla.org/mozilla-central/source/dom/system/gonk/RadioInterfaceLayer.js#4177
>>>
>>>
>>>
>>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1849
>>>
>>>
>>>
>>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1916
>>>
>>>
>>
>> This just means moving the matching logic to gaia, right?
>

> Just? No. It also follows we don't have convenient ondeliverysuccess,
> ondeliveryerror events for everyone interested in. These events,
> because of the lack of message object and message id, have to be further
> processed in _every_ listener.

Hmm, clearly I don't understand all of the details here. :/

>> (Please also
>> note that we can still write to the data store from Gecko if needed.)
>>
>>> [Problem-4: Replaced Short Message Type]
>>>
>>> Not actually a problem, but a task to be re-implemented in Gaia. When a
>>> MT short message with one of some special PIDs is received, Gecko
>>> compares its sender to other received messages and replaces the message
>>> body with the new one. Without access to the data store, this just
>>> can't be done in Gecko.
>>>
>>> http://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MobileMessageDB.jsm#1783
>>>
>>>
>>>
>>>
>>> [Problem-5: Cancel retrieving transaction if message deleted]
>>>
>>> Not actually a problem, but a task to be re-implemented in Gaia.
>>>
>>> mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MmsService.js#1796†
>>>
>>>
>>
>> It seems that basically doing the writes to the data store in Gaia is a
>> lot of work. :-)yp
>

> Yup.

>
>>> [Problem-6: message object in retrieveMMS() error event]
>>>
>>> Bug 824717 requests carrying message object in DOMRequest error event as
>>> well. However, for a retrieveMMS() call, the passed parameters are not
>>> enough to construct a MmsMessage back.
>>>
>>> https://bugzilla.mozilla.org/show_bug.cgi?id=824717
>>
>> Could we not send the ID for a field in the data store instead?
>

> Yes, that's probably the only way, although it's not as convenient as a
> message object.
>

>>> [Problem-7: SmsMessage, MmsMessage interface changes]
>>>
>>> Not actually a problem, but a task to be forfilled by Gaia. We don't
>>> have access to data store, so attributes that may change after be
>>> sent/received or may not be accessible at the time firing events have to
>>> be removed. Following is a list with reasons:
>>>
>>> 1) id: see problem-2,
>>> 2) threadId: conversation grouping is now a task of Gaia,
>>> 3) deliveryStatus/deliveryTimestamp, deliveryInfo: see problem-3,
>>> 4) deliveryTimestamp: same for deliveryStatus,
>>> 5) read: we'll never know this,
>>>
>>>
>>> All these are not related to conversation grouping. Moving to data
>>> store API is _not_ only re-implementing some phone number matching
>>> mechanism, but also a great change in current API design and control
>>> flow when Gecko don't have access to the data store.
>>
>> Thanks for raising all of these issues, these are really helpful. Note
>> that there is nothing that prevents us from writing to the data store
>> from Gecko, and I think that would alleviate a lot of these concerns.
>> Please let me know if you agree!
>

[Jonas Sicking]

Yeah, I definitely think we should make sure that we can write the message
to the DataStore directly from gecko.

Worst case we can still write the code which writes into the DataStore in
Chrome JS.

/ Jonas

On Apr 30, 2014 3:43 AM, "Gene Lian" <cl...@mozilla.com> wrote:

>
> > It seems that basically doing the writes to the data store in Gaia is a
> > lot of work. :-)yp
>

[Jonas Sicking]

[JOSE MANUEL CANTERA FONSECA]

El 30/04/14 05:33, "Vicamo Yang" <vic...@gmail.com> escribió:

>
>In contrast, with access to data store in Gecko, all the following
>technical details can still be hidden in Gecko. But some public
>interfaces will be removed/modified for sure. For example probably
>don't need onreceive/onsent/onfailed, because datastore API emits
>onchange events for us.

I would keep the specific events as they have more semantics than the
'onchange' event which is rather unspecific

> It's true there is a slight semantic difference
>for Cost Control app, maybe we don't have to talk about this now.
>Removing these events seems not be a big deal, the greatest challenge
>will be mapping data store changes to IndexedDB that allows us to
>perform various query requests.
>
>>>

>Regards,
>Vicamo Yang

[JOSE MANUEL CANTERA FONSECA]