We are creating this new list for those interested in helping with the
development of Adeona. We will begin this mailing list with the
following text from an email we sent to adeona-users.
Thank you all for your interest in Adeona. As you know, Adeona is
currently not living up to its expectations, largely because the back-
end OpenDHT system is not able to tolerate the load imposed by Adeona.
We are actively trying to fix this. But, to be honest, one of the
biggest challenges is finding suitable resources to do this within a
university environment. We keep thinking we are "almost there," only
to have setbacks. For too long we have though that the fix would be
right around the corner, but it now seems that taking such a strategy
is not sufficient, and we apologize for not realizing this sooner.
Despite our best efforts, we are realizing that we cannot do this
alone. We need your help. We have put the source code for Adeona
online here: <https://edge.launchpad.net/adeona>, and we have added
some new people to our team as maintainers of this code. We actively
seek your open source contributions. This is the code for the Adeona
client and the retrieval program.
However, in order for Adeona to work correctly, more than the client
and retrieval code need to work. Namely, Adeona needs a remote
"service" to store values. Currently OpenDHT on PlanetLab, which is
another research project from a different team, is not able to handle
the Adeona load. We need something simpler and more streamlined (not
a DHT) to handle Adeona.
We are therefore asking for assistance. There are at least two
possible directions to proceed in. (1) Move Adeona from OpenDHT to a
DHT that is more "production" quality. (2) Create a new back end
service (not a DHT). We think (2) is the right strategy because it
would allow all existing installations of Adeona to once again work.
Fortunately, for (2), OpenDHT exposes a clean RPC interface, so the
new back end service would simply need to implement that interface.
The downside with (2) is that it would require an entity to run the
service; we have not calculated the exact costs for running such a
service, but we would like to be able to support it once the service
is created.
We will also create a new mailing list, adeona-developers, for those
interested in helping with future development. Once it is created, we
will post a link to this new mailing list on the Adeona web page once
it is created. We sincerely look forward to your contributions.
I don't want to speak for the rest of the original Adeona folks, but
I also think this is a great idea. I'd love to move this thread to
the adeona-developers list, now that we've recently created it: <http://groups.google.com/group/adeona-developers>. I'm going to
cross-post this email to both lists.
A few additional thoughts on this. The first is that we would really
like to be able to support existing users who aren't reading adeona- users, who have Adeona installed, and who won't visit our website to
look for updates. For this reason, replacing the back-end OpenDHT
infrastructure with something more reliable would be very
advantageous -- or at least still providing that support while we
explore new options (more below).
There are also a bunch of other potential alternatives, such as
having the Adeona client email the encrypted location information to
a user's web mail account, etc. For those interested in helping with
Adeona development, I would encourage you to read our original
technical paper, which is available online here <http:// www.usenix.org/events/sec08/tech/full_papers/ristenpart/ ristenpart.pdf> (USENIX Security 2008). This paper will help give
you a sense for where we're coming from, discusses some alternatives
(like email), and why we chose not to pursue them in the context of
Adeona. Specifically, our goal with Adeona was to provide a system
that was very privacy-respecting, and we found that some approaches
(like email) did not achieve the level of privacy that we desired.
One strategy, which I think might be attractive, is for the Adeona
client to allow the user to select one or more approaches from a set
of options.
> Let the user select their own SFTP/SSH account as an option for
> storing data instead of OpenDHT.
> FTP accounts come with many broadband subscriptions, and hosting
> accounts like Dreamhost with SFTP and SSH are very cheap. So for me
> SSH or SFTP would make good solution, where I can retrieve my Adeona
> data easily if and when I need it. Any ideas?
On Apr 22, 8:20 am, Tadayoshi Kohno <yo...@cs.washington.edu> wrote:
> Hi all,
> I don't want to speak for the rest of the original Adeona folks, but
> I also think this is a great idea.
+1 from me
> A few additional thoughts on this. The first is that we would really
> like to be able to support existing users who aren't reading adeona-
> users, who have Adeona installed, and who won't visit our website to
> look for updates. For this reason, replacing the back-end OpenDHT
> infrastructure with something more reliable would be very
> advantageous -- or at least still providing that support while we
> explore new options (more below).
I have not looked at the implementation of Adeona. Is this possible
with the current clients that are deployed in the wild? Do those
clients connect to a DNS address that the Adeona team owns and
controls and could point to a new service with a matching API?
Also, are there people from the OpenDHT community that you can invite
to participate here so that the real issues with OpenDHT are
discussed? Is it really an issue of scale, or is it an impementation
problem inherent in OpenDHT that needs to be resolved?
> There are also a bunch of other potential alternatives, such as
> having the Adeona client email the encrypted location information to
> a user's web mail account, etc. For those interested in helping with
> Adeona development, I would encourage you to read our original
> technical paper, which is available online here <http://www.usenix.org/events/sec08/tech/full_papers/ristenpart/ > ristenpart.pdf> (USENIX Security 2008). This paper will help give
> you a sense for where we're coming from, discusses some alternatives
> (like email), and why we chose not to pursue them in the context of
> Adeona. Specifically, our goal with Adeona was to provide a system
> that was very privacy-respecting, and we found that some approaches
> (like email) did not achieve the level of privacy that we desired.
The user should be able to select what level of privacy they are
comfortable with.
> One strategy, which I think might be attractive, is for the Adeona
> client to allow the user to select one or more approaches from a set
> of options.
> I have not looked at the implementation of Adeona. Is this possible > with the current clients that are deployed in the wild? Do those > clients connect to a DNS address that the Adeona team owns and > controls and could point to a new service with a matching API?
This is a great question! The short answer is that we should be able to point existing Adeona users to other back-end systems.
Here's a bit of a longer answer. After one installs Adeona, there should be an options.adeona file with (among other things) these two lines:
Taken together, these options mean that the Adeona client should regularly download the gateways file from <http://adeona.cs.washington.edu/gateways.adeona >. Currently this URL contains a list of OpenDHT nodes. (Technically, the first line in the gateways file is actually a reference to OASIS, which will then point to an OpenDHT node when OASIS is up and running. This web page describes OASIS: <http://oasis.coralcdn.org/ >.)
By changing the contents of this gateways file, we will be able to redirect users to different machines -- as long as they expose the same Sun RPC interface that OpenDHT exposes.
> Also, are there people from the OpenDHT community that you can invite > to participate here so that the real issues with OpenDHT are > discussed? Is it really an issue of scale, or is it an impementation > problem inherent in OpenDHT that needs to be resolved?
My answers here are a bit more shaky, so I would love it if someone else can chime in. This mailing list probably is the best source for information about the current status of OpenDHT: <http://opendht.org/pipermail/opendht-users/ >.
On Apr 22, 11:43 pm, Tadayoshi Kohno <yo...@cs.washington.edu> wrote:
> > Also, are there people from the OpenDHT community that you can invite
> > to participate here so that the real issues with OpenDHT are
> > discussed? Is it really an issue of scale, or is it an impementation
> > problem inherent in OpenDHT that needs to be resolved?
> My answers here are a bit more shaky, so I would love it if someone
> else can chime in. This mailing list probably is the best source for
> information about the current status of OpenDHT: <http://opendht.org/pipermail/opendht-users/ > >.
May I suggest your team reach out to them and see if they are willing
to get involved?
Seems like this project may be even more moribund than yours was a
week ago based on this last thread (from December) from someone who is
I believe a member of the OpenDHT team.
So, just hopping in here with a suggestion. Since Adeona is the only
software (I know of) that provides any sort of hope for laptop
recovery that works w/ Linux, have you thought about reaching out to
one of the big Linux players for support? I would think that
Canonical, Redhat, or Novell might be interested in hosting and/or
helping develop the more robust backend.
I'm new to the group and have quite a catchup to do in the coming
days, Adeona's time has come, I have much reading to do.
I do believe that QH is on to something, an open source cross
platform
equivalent to “Lojack for your laptop or embedded device”
could certainly use a bit of cooperate muscle and infrastructure
behind
it. Red Hat, Novell or Google. Not to mention the global perception
of a cooperate entity investing resources to right a definite wrong
and care about your privacy, investment and data. This could be
one of the most positive actions and rewarding for all involved.
GrumpyLizard
On Apr 30, 4:30 pm, qhartman <qhart...@gmail.com> wrote:
> So, just hopping in here with a suggestion. Since Adeona is the only
> software (I know of) that provides any sort of hope for laptop
> recovery that works w/ Linux, have you thought about reaching out to
> one of the big Linux players for support? I would think that
> Canonical, Redhat, or Novell might be interested in hosting and/or
> helping develop the more robust backend.
For the sake of getting this rocking again, could there not be an
Amazon S3 option? I can't imagine that the data storage or cost of
traffic would be so big that an idividual would not be willing to pay
the small cost per gig transfer and storage. One could probably
implement an S3 registration in the setup, or provide instructions on
how to setup an S3 account.
AJ
On Apr 30, 10:03 pm, GrumpyLizard <awopa...@gmail.com> wrote:
> I'm new to the group and have quite a catchup to do in the coming
> days, Adeona's time has come, I have much reading to do.
> I do believe that QH is on to something, an open source cross
> platform
> equivalent to “Lojack for your laptop or embedded device”
> could certainly use a bit of cooperate muscle and infrastructure
> behind
> it. Red Hat, Novell or Google. Not to mention the global perception
> of a cooperate entity investing resources to right a definite wrong
> and care about your privacy, investment and data. This could be
> one of the most positive actions and rewarding for all involved.
> GrumpyLizard
> On Apr 30, 4:30 pm, qhartman <qhart...@gmail.com> wrote:
> > So, just hopping in here with a suggestion. Since Adeona is the only
> > software (I know of) that provides any sort of hope for laptop
> > recovery that works w/ Linux, have you thought about reaching out to
> > one of the big Linux players for support? I would think that
> > Canonical, Redhat, or Novell might be interested in hosting and/or
> > helping develop the more robust backend.
To add on to this, the host could be something along the lines of
www.slicehost.com. It's a virtual machine with dedicated resources and
even their $20/month plan is pretty good.
I'm not sure how much bandwidth option 2 above would be -- do we have
any way of estimating what we would need? How much bandwidth would a
laptop use if it was on 24/7 for a month? Another option is to charge
a little for the hosting service... but $1/month would easily cover
the costs. Ideally, it would be covered by donations...
If the backend is non-distributed, people who have their own servers
could easily use them to host the backend. This doesn't help much if
your server is taken with your laptop though...
On May 3, 5:36 pm, Phil <wolvenwra...@gmail.com> wrote:
I don't think that ultimately hosting is the issue, but rather the
unreliability of the hash table software if I understand correctly.
I would suggest the first thing we need to understand, before we go
digging down hosting and code change options, is why the current
backend software failing? Does the adeona team (if there is such a
thing) know the answer to this question?
Is anyone from the OpenDHT team participating in this list?
Can the Adeona team make themselves known to the members of this
list? And what are your reasonable expectations of participating
further in the project?
I feel better that there are strong warnings letting people know that
the software is in a non-functional state now. But what I have not
seen yet from 'the adeona team' is any movement towards a desire to
understand and improve the situation. I think that unless we see such
movement, or someone else steps forward with the desire to take over
an abandoned project, then we are all done here...
These are the things we need to know first. Then, if there is a team,
it can move forward to actually solve problems with code.
Glenn
On May 5, 12:02 am, Phil <wolvenwra...@gmail.com> wrote:
> To add on to this, the host could be something along the lines ofwww.slicehost.com. It's a virtual machine with dedicated resources and
> even their $20/month plan is pretty good.
> I'm not sure how much bandwidth option 2 above would be -- do we have
> any way of estimating what we would need? How much bandwidth would a
> laptop use if it was on 24/7 for a month? Another option is to charge
> a little for the hosting service... but $1/month would easily cover
> the costs. Ideally, it would be covered by donations...
> If the backend is non-distributed, people who have their own servers
> could easily use them to host the backend. This doesn't help much if
> your server is taken with your laptop though...
> On May 3, 5:36 pm, Phil <wolvenwra...@gmail.com> wrote:
> > Hmm, I might be interested in contributing to keep this afloat as
> > well.
> > Among the things I could possibly do is 64-bit support, debian/ubuntu
> > packaging, etc.
> > I am really interested in the potential for this program. Have we
> > decided on which service we will be using for this?
Sorry for the silence. Alas, through attrition the team right now
consists largely of Arvind Krishnamurthy and myself, though we are
actively in the process of preparing a grant proposal for additional
funds within UW so that we can hire a part or full time person to
focus on Adeona development and support.
As a brief status report, here is a recent thread on the OpenDHT
mailing list: <http://opendht.org/pipermail/opendht-users/2009-May/ 000430.html>. Note that Sean Rhea has publicly announced that he
will not continue to maintain OpenDHT. We took this as an
opportunity to try to take over the maintenance role. We may face
obstacles, but we are very hopeful. Notice especially the most
recent observation by Bogdan Nicolae, and a patch he created to use
the latest version of Berkeley DB.
At the expense of not supporting existing users, we are also
exploring the use of OpenLookup, which exhibits an XML-RPC interface
like OpenDHT (but Adeona currently uses OpenDHT's Sun RPC
interface). This may not be the long term solution. But our hope is
that it will be a short term solution, coupled with your suggestion
of asking users to enter their email addresses before they download.
> I don't think that ultimately hosting is the issue, but rather the
> unreliability of the hash table software if I understand correctly.
> I would suggest the first thing we need to understand, before we go
> digging down hosting and code change options, is why the current
> backend software failing? Does the adeona team (if there is such a
> thing) know the answer to this question?
> Is anyone from the OpenDHT team participating in this list?
> Can the Adeona team make themselves known to the members of this
> list? And what are your reasonable expectations of participating
> further in the project?
> I feel better that there are strong warnings letting people know that
> the software is in a non-functional state now. But what I have not
> seen yet from 'the adeona team' is any movement towards a desire to
> understand and improve the situation. I think that unless we see such
> movement, or someone else steps forward with the desire to take over
> an abandoned project, then we are all done here...
> These are the things we need to know first. Then, if there is a team,
> it can move forward to actually solve problems with code.
> Glenn
> On May 5, 12:02 am, Phil <wolvenwra...@gmail.com> wrote:
>> To add on to this, the host could be something along the lines
>> ofwww.slicehost.com. It's a virtual machine with dedicated
>> resources and
>> even their $20/month plan is pretty good.
>> I'm not sure how much bandwidth option 2 above would be -- do we have
>> any way of estimating what we would need? How much bandwidth would a
>> laptop use if it was on 24/7 for a month? Another option is to charge
>> a little for the hosting service... but $1/month would easily cover
>> the costs. Ideally, it would be covered by donations...
>> If the backend is non-distributed, people who have their own servers
>> could easily use them to host the backend. This doesn't help much if
>> your server is taken with your laptop though...
>> On May 3, 5:36 pm, Phil <wolvenwra...@gmail.com> wrote:
>>> Hmm, I might be interested in contributing to keep this afloat as
>>> well.
>>> Among the things I could possibly do is 64-bit support, debian/ >>> ubuntu
>>> packaging, etc.
>>> I am really interested in the potential for this program. Have we
>>> decided on which service we will be using for this?
I see another way to make this project work, but it adds some
complexity and would require additional coding. However, it would
remove the need for the service to be maintained as a separate entity,
and could probably be maintained easily in a university environment.
I'm proposing that we shard out the entire database to all of the
users of the service. This could easily be distributed through
bittorrent, or a similar service. We would distribute copies of each
adeona update to many machines (enough so the probability that there
are no copies available when a user requests the update is low).
Using this method, you can ensure that the service scales with demand,
with minimal expenditures at the top level. You could even have a
weighted system, so users who make more resources on their system
available have their updates distributed to more machines.
Since many users of Adeona are laptop users, connectivity might be
sparse. One way to mitigate this issue is to create a passive system,
in which people log into their account and request X number of
updates, and can give a date range. Each time the laptops achieves
connectivity, the adeona daemon could ask the server if there are any
updates requested that it has on its box, if there are updates, it
just uploads them to the server, and then the server can send an email
to the requesting user notifying them that it has been retrieved. If
the user does not retrieve the adeona updates they requested after X
amount of time, they can just be removed. The time requests are
availiable, in addition to the number of updates they request could be
determined by their karma rating.
The karma rating is a composite score built from many statistics about
the user. One would be how often they are connected, the more they
are connected, the higher karma they have. Secondly, karma could be
determined based on the amount of disk space a user allows adeona to
use to store user data. Finally, you could give more karma points
based on how many successful retrievals the user has provided to other
users. This would encourage users to run adeona on multiple systems,
possibly on their desktop (which is less likely to be stolen, and more
likely to be connected and have larger hard drives), which would lead
to much higher retrieval rates, leading to higher karma for the user
who hosts it.
This is in no way a bulletproof system, but I believe it will serve
the same purpose as the current system, but with a much higher level
of reliability, even with the disconnected state of laptops. Please
let me know what you think.
Thanks,
Preston
On Apr 21, 1:32 pm, Adeona Team <ade...@cs.washington.edu> wrote:
> We are creating this new list for those interested in helping with the
> development of Adeona. We will begin this mailing list with the
> following text from an email we sent to adeona-users.
> Thank you all for your interest in Adeona. As you know, Adeona is
> currently not living up to its expectations, largely because the back-
> end OpenDHT system is not able to tolerate the load imposed by Adeona.
> We are actively trying to fix this. But, to be honest, one of the
> biggest challenges is finding suitable resources to do this within a
> university environment. We keep thinking we are "almost there," only
> to have setbacks. For too long we have though that the fix would be
> right around the corner, but it now seems that taking such a strategy
> is not sufficient, and we apologize for not realizing this sooner.
> Despite our best efforts, we are realizing that we cannot do this
> alone. We need your help. We have put the source code for Adeona
> online here: <https://edge.launchpad.net/adeona>, and we have added
> some new people to our team as maintainers of this code. We actively
> seek your open source contributions. This is the code for the Adeona
> client and the retrieval program.
> However, in order for Adeona to work correctly, more than the client
> and retrieval code need to work. Namely, Adeona needs a remote
> "service" to store values. Currently OpenDHT on PlanetLab, which is
> another research project from a different team, is not able to handle
> the Adeona load. We need something simpler and more streamlined (not
> a DHT) to handle Adeona.
> We are therefore asking for assistance. There are at least two
> possible directions to proceed in. (1) Move Adeona from OpenDHT to a
> DHT that is more "production" quality. (2) Create a new back end
> service (not a DHT). We think (2) is the right strategy because it
> would allow all existing installations of Adeona to once again work.
> Fortunately, for (2), OpenDHT exposes a clean RPC interface, so the
> new back end service would simply need to implement that interface.
> The downside with (2) is that it would require an entity to run the
> service; we have not calculated the exact costs for running such a
> service, but we would like to be able to support it once the service
> is created.
> We will also create a new mailing list, adeona-developers, for those
> interested in helping with future development. Once it is created, we
> will post a link to this new mailing list on the Adeona web page once
> it is created. We sincerely look forward to your contributions.
This is a fantastic suggestion. I really like this. I agree with you
about both the potential for a distributed system such as this, and
the challenges to creating it (e.g., determining the right parameters
for such a system in order to scale for a sufficient number of users).
To be honest, I'll mention that we did explore something like this a
little bit. One encouraging fact is that some large scale systems
already leverage a similar construct, and we can learn from them. At
one point in time one person working with us, who has since moved on
to something else, tried to hook Adeona together with libdht <http://libdht.sourceforge.net/ > but wasn't successful. I am going to try to dig into this and find
out more about what the problems were. Of course, if anyone on this
list is enthusiastic about this approach or knows something that we
didn't, please do let us know.
Thanks!
Yoshi
On May 7, 2009, at 10:41 PM, Preston Marshall wrote:
> I see another way to make this project work, but it adds some
> complexity and would require additional coding. However, it would
> remove the need for the service to be maintained as a separate entity,
> and could probably be maintained easily in a university environment.
> I'm proposing that we shard out the entire database to all of the
> users of the service. This could easily be distributed through
> bittorrent, or a similar service. We would distribute copies of each
> adeona update to many machines (enough so the probability that there
> are no copies available when a user requests the update is low).
> Using this method, you can ensure that the service scales with demand,
> with minimal expenditures at the top level. You could even have a
> weighted system, so users who make more resources on their system
> available have their updates distributed to more machines.
> Since many users of Adeona are laptop users, connectivity might be
> sparse. One way to mitigate this issue is to create a passive system,
> in which people log into their account and request X number of
> updates, and can give a date range. Each time the laptops achieves
> connectivity, the adeona daemon could ask the server if there are any
> updates requested that it has on its box, if there are updates, it
> just uploads them to the server, and then the server can send an email
> to the requesting user notifying them that it has been retrieved. If
> the user does not retrieve the adeona updates they requested after X
> amount of time, they can just be removed. The time requests are
> availiable, in addition to the number of updates they request could be
> determined by their karma rating.
> The karma rating is a composite score built from many statistics about
> the user. One would be how often they are connected, the more they
> are connected, the higher karma they have. Secondly, karma could be
> determined based on the amount of disk space a user allows adeona to
> use to store user data. Finally, you could give more karma points
> based on how many successful retrievals the user has provided to other
> users. This would encourage users to run adeona on multiple systems,
> possibly on their desktop (which is less likely to be stolen, and more
> likely to be connected and have larger hard drives), which would lead
> to much higher retrieval rates, leading to higher karma for the user
> who hosts it.
> This is in no way a bulletproof system, but I believe it will serve
> the same purpose as the current system, but with a much higher level
> of reliability, even with the disconnected state of laptops. Please
> let me know what you think.
> Thanks,
> Preston
> On Apr 21, 1:32 pm, Adeona Team <ade...@cs.washington.edu> wrote:
>> Dear Adeona-Developers,
>> We are creating this new list for those interested in helping with
>> the
>> development of Adeona. We will begin this mailing list with the
>> following text from an email we sent to adeona-users.
>> Thank you all for your interest in Adeona. As you know, Adeona is
>> currently not living up to its expectations, largely because the
>> back-
>> end OpenDHT system is not able to tolerate the load imposed by
>> Adeona.
>> We are actively trying to fix this. But, to be honest, one of the
>> biggest challenges is finding suitable resources to do this within a
>> university environment. We keep thinking we are "almost there," only
>> to have setbacks. For too long we have though that the fix would be
>> right around the corner, but it now seems that taking such a strategy
>> is not sufficient, and we apologize for not realizing this sooner.
>> Despite our best efforts, we are realizing that we cannot do this
>> alone. We need your help. We have put the source code for Adeona
>> online here: <https://edge.launchpad.net/adeona>, and we have added
>> some new people to our team as maintainers of this code. We actively
>> seek your open source contributions. This is the code for the Adeona
>> client and the retrieval program.
>> However, in order for Adeona to work correctly, more than the client
>> and retrieval code need to work. Namely, Adeona needs a remote
>> "service" to store values. Currently OpenDHT on PlanetLab, which is
>> another research project from a different team, is not able to handle
>> the Adeona load. We need something simpler and more streamlined (not
>> a DHT) to handle Adeona.
>> We are therefore asking for assistance. There are at least two
>> possible directions to proceed in. (1) Move Adeona from OpenDHT to a
>> DHT that is more "production" quality. (2) Create a new back end
>> service (not a DHT). We think (2) is the right strategy because it
>> would allow all existing installations of Adeona to once again work.
>> Fortunately, for (2), OpenDHT exposes a clean RPC interface, so the
>> new back end service would simply need to implement that interface.
>> The downside with (2) is that it would require an entity to run the
>> service; we have not calculated the exact costs for running such a
>> service, but we would like to be able to support it once the service
>> is created.
>> We will also create a new mailing list, adeona-developers, for those
>> interested in helping with future development. Once it is created,
>> we
>> will post a link to this new mailing list on the Adeona web page once
>> it is created. We sincerely look forward to your contributions.
I like this idea in principle, Preston, and appreciate your tabling
it. I will add, though, that we need to be careful in how this is
implemented. It sounds like the "Supernode" feature in Skype and
universities are known to ban Skype because it creates a drain on
bandwidth. There's a registry hack for the Windows version of Skype
that will disable Supernode functionality but the Mac version doesn't
support this yet. A quick Google search will show that there's a lot
of uproar over this as well as the fact that many university and
businesses block Bittorrent type of activity. I love the idea of
having a distributed model but we need to have a checkbox somewhere in
the configuration that will disable the server service. It can even be
something that needs to be configured post-installation; as an IT guy
at a university, I wouldn't mind that. It would then allow the Adeona
to be installed on more computers and there would be enough out there
that acted as servers to ensure that the database was well shared.
Has anyone contacted Google, GoDaddy, Amazon or anybody else? It seems
like there should be enough large entities already maintaining clouds
that could easily support database hosting. Google has open source
projects and there's a very altruistic arm to the organization. If
they or someone else can host it and let the open source community
maintain it, we could potentially move forward quickly. Google may
even be interested in putting some of their interns on it during
Summer of Code 2009. This type of thing was mentioned earlier but I
haven't seen that anyone has actually made contact.
Yoshi: Is this something you have the authority to do? Now that the
project has expanded to include OpenDHT, having a large company that
can host us in their cloud seems like a reasonable approach. That
would allow us to move forward and support the existing Adeona user
base while looking at moving to a more distributed P2P or other model
and how to migrate existing users as well.
Thanks for all your thoughts on this! Hmm.. I'm not quite sure
where to begin. I love your idea of involving Google Summer of Code
2009. From looking at the Summer of Code webpage, <http:// socghop.appspot.com/document/show/program/google/gsoc2009/timeline>,
I'm concerned that we might have missed the deadlines for
participation this year. Does anyone know if there are other ways of
participating in Summer of Code, besides what's outlined on this
page? (I'll try to dig a bit more myself too.)
As you suggest, having an existing cloud service support and maintain
an OpenDHT (or OpenDHT-compatible) infrastructure for legacy Adeona
clients would be great, especially as we transition the client code
to something better. As background, OpenDHT runs the Bamboo DHT code
on PlanetLab machines. For a while we were actually trying to run
Bamboo under Amazon EC2, but Bamboo kept crashing. This recent post
on the OpenDHT mailing list may provide a partial explanation for
these crashes: <http://opendht.org/pipermail/opendht-users/2009-May/ 000430.html>. I think our first order of business will be to work
with the original OpenDHT maintainer and author and see if we can run
a patched version of Bamboo on the PlanetLab machines in a way that
can scale to the existing number of Adeona clients. If the patched
version of Bamboo can withstand the Adeona load, then a whole world
of possibilities opens up to us re: support for legacy clients and
I'll be happy to explore all the options you suggested.
If there are any experienced P2P developers on this list, please feel
free to let me know either privately or on the list itself.
> I like this idea in principle, Preston, and appreciate your tabling
> it. I will add, though, that we need to be careful in how this is
> implemented. It sounds like the "Supernode" feature in Skype and
> universities are known to ban Skype because it creates a drain on
> bandwidth. There's a registry hack for the Windows version of Skype
> that will disable Supernode functionality but the Mac version doesn't
> support this yet. A quick Google search will show that there's a lot
> of uproar over this as well as the fact that many university and
> businesses block Bittorrent type of activity. I love the idea of
> having a distributed model but we need to have a checkbox somewhere in
> the configuration that will disable the server service. It can even be
> something that needs to be configured post-installation; as an IT guy
> at a university, I wouldn't mind that. It would then allow the Adeona
> to be installed on more computers and there would be enough out there
> that acted as servers to ensure that the database was well shared.
> Has anyone contacted Google, GoDaddy, Amazon or anybody else? It seems
> like there should be enough large entities already maintaining clouds
> that could easily support database hosting. Google has open source
> projects and there's a very altruistic arm to the organization. If
> they or someone else can host it and let the open source community
> maintain it, we could potentially move forward quickly. Google may
> even be interested in putting some of their interns on it during
> Summer of Code 2009. This type of thing was mentioned earlier but I
> haven't seen that anyone has actually made contact.
> Yoshi: Is this something you have the authority to do? Now that the
> project has expanded to include OpenDHT, having a large company that
> can host us in their cloud seems like a reasonable approach. That
> would allow us to move forward and support the existing Adeona user
> base while looking at moving to a more distributed P2P or other model
> and how to migrate existing users as well.
Again, I would state that the issues with adeona are NOT with
hosting. This is not an issue of server capacity. This is not an
issue of not having a partner that will take the load off of the
OpenDHT folks.
This IS an issue of the unreliability of the now unmaintained OpenDHT
projects code that Adeona relies upon. It is not able to survive
under load or for an indefinite period of time without manual
intervention and potential data loss.
While the distributed DB approach is certainly interesting, I don't
think the Adeona team currently has the bandwidth or inclination to
design and code an entirely new distributed database application that
is highly available and secure. This is pie in the sky thinking at
this point. I think it is highly unrealistic, given past indications,
that this project will succeed if you try to take on the design and
development of a new distributed database.
What I would suggest people focus on is the most pragmatic approach to
re-using as much of the current client software while swapping out the
back end for something which is known to be reliable. This could be
for example:
Backward compatible solutions:
- Recommended only as a short term solution : A version of OpenDHT
that implements fixes to the known unreliability issues. Once this is
done, finding hosting for the database processes is a much simpler
affair. This is the ONLY solution I am aware of which retains 100%
backwards compatibility with the estimated 50,000-100,00 clients that
are out there who THINK that they are protected now but are in fact
the emperor has no clothes.
- RECOMMENDED LONG TERM : Write an application which hosts an API that
is identical to that which OpenDHT uses for puts and gets. This app
could can and should provide more than one API interface version. For
example, one API interface that is backwards compatible with older
versions of Adeona that are our there, and an alternative new
interface (e.g. based on Oauth for access control) which works for
various new clients the come up in the future that are provided by
Adeona or anyone who wants to use this service as their back-end while
writing their own clients. You could write a very thin web app which
allows users to retrieve their stolen machine data thus eliminating
the complexity of the client side command line retrieval tools. This
app could store its data in any back-end key value store, some of
which are listed below...
There are many highly scalable key/value store type databases that are
out there now that can handle massive scale. Some are:
- CloudKit (http://getcloudkit.com/) or something like it, fronting
Tokyo Cabinet or the like. No costs other than hosting which I am
sure we can find a provider for (e.g. Joyent, or the University).
- An API in front of CouchDB, which is a distributed key value DB with
excellent replication built in.
- An API that runs on Google's app engine which gives you the benefit
of the massive scalability of Google's BigTable schema-free DB. Free
for up to 5 million page views per month I believe. Small costs based
on usage after that.
- An API which fronts Amazon's Simple DB which is a massively
scalable, and distributed DB. There are relatively small costs
associated with this.
OpenDHT is not the only, or even the recommended answer. My primary
point is, you don't have to write the database, this is an
expoentially harder task. This is an area of DB design that has been
changing dramatically since Adeona was first released. All you need
is a thin API layer which fronts the new key value store. This
approach also gives you the flexibility to swap out the back end store
while keeping the front end API to your liking.
My $0.02
Glenn
On May 8, 8:54 am, Joe <joseph.mcl...@verizon.net> wrote:
> I like this idea in principle, Preston, and appreciate your tabling
> it. I will add, though, that we need to be careful in how this is
> implemented. It sounds like the "Supernode" feature in Skype and
> universities are known to ban Skype because it creates a drain on
> bandwidth. There's a registry hack for the Windows version of Skype
> that will disable Supernode functionality but the Mac version doesn't
> support this yet. A quick Google search will show that there's a lot
> of uproar over this as well as the fact that many university and
> businesses block Bittorrent type of activity. I love the idea of
> having a distributed model but we need to have a checkbox somewhere in
> the configuration that will disable the server service. It can even be
> something that needs to be configured post-installation; as an IT guy
> at a university, I wouldn't mind that. It would then allow the Adeona
> to be installed on more computers and there would be enough out there
> that acted as servers to ensure that the database was well shared.
> Has anyone contacted Google, GoDaddy, Amazon or anybody else? It seems
> like there should be enough large entities already maintaining clouds
> that could easily support database hosting. Google has open source
> projects and there's a very altruistic arm to the organization. If
> they or someone else can host it and let the open source community
> maintain it, we could potentially move forward quickly. Google may
> even be interested in putting some of their interns on it during
> Summer of Code 2009. This type of thing was mentioned earlier but I
> haven't seen that anyone has actually made contact.
> Yoshi: Is this something you have the authority to do? Now that the
> project has expanded to include OpenDHT, having a large company that
> can host us in their cloud seems like a reasonable approach. That
> would allow us to move forward and support the existing Adeona user
> base while looking at moving to a more distributed P2P or other model
> and how to migrate existing users as well.
> While the distributed DB approach is certainly interesting, I don't
> think the Adeona team currently has the bandwidth or inclination to
> design and code an entirely new distributed database application that
> is highly available and secure. This is pie in the sky thinking at
> this point. I think it is highly unrealistic, given past indications,
> that this project will succeed if you try to take on the design and
> development of a new distributed database.
Why redesigning it; the bittorrent network is already here, why don't
just leverage it :) ?
Ummm... because BitTorrent is a a P2P file transfer protocol and not
a distributed key/value store...
But seriously, ,maybe you would like to elaborate on how you would
envision integration with BitTorrent to work?
Questions I would have would be...
How would you limit access to your private location information as
once you upload info to a tracker and seed you lose ALL expectations
of privacy or control?
What happens if there are no seeds available for your laptop location
info when your laptop is stolen? Do you just patiently wait?
Are you going to generate and post a new .torrent file for every
location update? To what tracker?
How would you collect all of the various torrent files into a coherent
location history for your stolen laptop?
OK I am new to Adeona and from what I have read the OpenHDT is a large
database of laptop/computer keys that have the data encrypted. This
Database is distributed over a number of servers and every time a
device with Adeona on it connects to the Net it drops encrypted data
into one of these databases. When you need to recover a laptop or
computer the recovery program accesses the database find the correct
server and returns the encrypted data. You then decode it and you have
your target IP and other data. Now the OpenHDT has problems and it
needs a maintainer or a new approach. Is this on the right track???
Cheers
Zeropoint
Yes -- this is almost exactly correct. The minor distinction that I
would make is that OpenDHT runs on a worldwide collection of PlanetLab
machines, rather than volunteer laptops/computers. You are also
correct that OpenDHT is experiencing significant problems (and the
maintainer has announced his decision to no longer support it), and
those problems are translating to problems for Adeona.
> OK I am new to Adeona and from what I have read the OpenHDT is a large
> database of laptop/computer keys that have the data encrypted. This
> Database is distributed over a number of servers and every time a
> device with Adeona on it connects to the Net it drops encrypted data
> into one of these databases. When you need to recover a laptop or
> computer the recovery program accesses the database find the correct
> server and returns the encrypted data. You then decode it and you have
> your target IP and other data. Now the OpenHDT has problems and it
> needs a maintainer or a new approach. Is this on the right track???
> Cheers
> Zeropoint
> OK I am new to Adeona and from what I have read the OpenHDT is a large
> database of laptop/computer keys that have the data encrypted. This
> Database is distributed over a number of servers and every time a
> device with Adeona on it connects to the Net it drops encrypted data
> into one of these databases. When you need to recover a laptop or
> computer the recovery program accesses the database find the correct
> server and returns the encrypted data. You then decode it and you have
> your target IP and other data. Now the OpenHDT has problems and it
> needs a maintainer or a new approach. Is this on the right track???
> Cheers
> Zeropoint
I am also new to the group but I am coming from a large technology
corporation using Linux on hundreds of laptops (thousands if you count
Windows and Mac also). This project is very interesting as it could
provide to us the ability to track our laptops especially in the
instance where they are stolen. (It does happen once in a while.) I am
curious to know how hard it would be to make this more of a
centralized back end where there are options not only to connect to a
central server that all people connect to but also have the ability to
create a "personal" server that say a large corporation could manage
for its company's laptops only. And while we are at it, why is this a
distributed hash back end as opposed to a centralized database? In my
mind, from a corporate view, most corporations prefer to maintain
their own servers and not risk having their data on someone else's
server where they have no control of uptime. From my "newbie" view of
the project, I would think that a system more like a Linux repository
might be more appropriate where you have servers that are open to all
and also the ability to create servers that are only used by say a
corporation. By default the client would point to the "open to all"
servers with the capability change where they point. Also, switching
to a mysql/postgresql/etc. database for this connection would provide
the scalability you are looking for on a large scale. These databases
also have the capability of being scaled over multiple servers if
needed (especially posgresql). You could also set the client up to
look at a "database list" which would provide a list of available
servers that it could connect to and upload its data and then setup
the servers to update each other on regular intervals. Again, I am
looking at the scalability of the back end and also looking at this
from a standard user vs. corporate user standpoint. Please let me know
if I am totally off my rocker.
Also, two other notes...
1. Would it be possible to get the source into a cvs solution maybe on
sourceforge or something else?
2. Now that the Adeona guys are managing the openDHT project, you
might want to update the openDHT webpage to not indicate that the
project is going away rather is under new direction...
Thanks!
Jonathan
On May 20, 11:43 am, Glenn Rempe <gl...@rempe.us> wrote:
> On May 19, 6:24 am, Zeropoint <michaelbarriesm...@bigpond.com> wrote:
> > OK I am new to Adeona and from what I have read the OpenHDT is a large
> > database of laptop/computer keys that have the data encrypted. This
> > Database is distributed over a number of servers and every time a
> > device with Adeona on it connects to the Net it drops encrypted data
> > into one of these databases. When you need to recover a laptop or
> > computer the recovery program accesses the database find the correct
> > server and returns the encrypted data. You then decode it and you have
> > your target IP and other data. Now the OpenHDT has problems and it
> > needs a maintainer or a new approach. Is this on the right track???
> > Cheers
> > Zeropoint
I read part of the paper linked to in this discussion, specifically
the part about using email as a delivery method for location data. It
seems to me that the privacy concerns with using email can be resolved
by simply encrypting the location data before sending. It could be
done in a user-friendly way, e.g. save it as an attachment which a
retrieval program can easily import and display. The user then just
needs to keep a private key somewhere safe. Would this not provide a
quick and safe fix to get Adeona up-and-running again until a server-
based solution to the problem can be implemented?
HI
If I understand very well the problem is the server for
recording is down, why we don't have new server, hosted in new place,
make sure has enough bandwidth, the company where I am working is
willing to provide the server, can you please let us the amount of the
bandwidth and the capacity of the server needed for that?
On Jun 6, 5:46 pm, mister-g <glub...@googlemail.com> wrote:
> I read part of the paper linked to in this discussion, specifically
> the part about using email as a delivery method for location data. It
> seems to me that the privacy concerns with using email can be resolved
> by simply encrypting the location data before sending. It could be
> done in a user-friendly way, e.g. save it as an attachment which a
> retrieval program can easily import and display. The user then just
> needs to keep a private key somewhere safe. Would this not provide a
> quick and safe fix to get Adeona up-and-running again until a server-
> based solution to the problem can be implemented?
|
