Once again we are unable to push our repos using SSH (we're getting
permission denied; user morais).
I'm tired of sending emails to support, so this time my rant will be
public.
A code hosting service needs to just work. bitbucket doesn't. It seems
every week there's a new, minor-but-blocking-getting-work-done,
problem.
Jesper, what is being done to improve the level of service?
> Once again we are unable to push our repos using SSH (we're getting > permission denied; user morais).
I'm having the same issue (user dhellmann).
> I'm tired of sending emails to support, so this time my rant will be > public. > A code hosting service needs to just work. bitbucket doesn't. It seems > every week there's a new, minor-but-blocking-getting-work-done, > problem.
> Jesper, what is being done to improve the level of service?
> Thanks, > Pedro
> -- > You received this message because you are subscribed to the Google Groups "bitbucket-users" group. > To post to this group, send email to bitbucket-users@googlegroups.com. > To unsubscribe from this group, send email to bitbucket-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/bitbucket-users?hl=en.
On Sep 23, 9:21 pm, Doug Hellmann <doug.hellm...@gmail.com> wrote:
> On Sep 23, 2010, at 6:31 AM, Pedro Morais wrote:
> > Hi,
> > Once again we are unable to push our repos using SSH (we're getting
> > permission denied; user morais).
> I'm having the same issue (user dhellmann).
It should be OK again now.
> > I'm tired of sending emails to support, so this time my rant will be
> > public.
> > A code hosting service needs to just work. bitbucket doesn't. It seems
> > every week there's a new, minor-but-blocking-getting-work-done,
> > problem.
> Indeed, this is getting old.
Yes it is. For us as well.
Long story:
When we migrated the new hardware, some things changed on the backend.
On EC2, we had to scale out the repositories across several instances,
as the I/O throughput on a single instance wasn't enough to serve
everyone at decent speeds. For this reason, every load balancer needed
to have a copy of authorized_keys on it, and we did this by using a
"fanout" exchange in our queue (RabbitMQ.) Every time a key was added
(or removed), every load balancer would be told to update the file
accordingly.
On the new hardware, we have the luxury of actual physical hardware,
and thus, we decided on getting a monster fileserver (with redundancy
of course), and serve up everything to the frontends via NFS. We used
the old code from the load balancers to handle the key files just the
same.
What first happened was that every frontend (3) would simultaneously
receive a job going "hey, update the key file", and by doing so,
something would get jumbled every 25 keys or so, and somehow the file
would get truncated completely and we'd end up with a file on the NFS
mount with 1 or 2 keys in it, and everyone would be in trouble.
We handled this in the simplest way: Disable the job on every box, but
one. This worked fine for a while. Then, however, one of the frontends
died, and had to be rebooted. When it came back up, it faithfully
started the key-job again, and now we had 2 boxes manipulating the
keyfile, and the problem from the first time around came back, and we
had the keyfile wiped again.
After the migration to the new hardware, we focused mainly on
improving service in general, taking advantage of some of the luxuries
we couldn't before on EC2. One of them was to improve the way the
keyfile was handled. We got rid of the "fanout" exchange and switched
it over to "round robin" class job, so one server would pick up the
key job, manipulate the file, and every frontend would see the change.
This is what's supposed to be the final solution to this.
Right now, I'm not sure what wiped the file, but that is indeed what
happened again. I'm tired of having to deal with repercussions of the
migration--after all, we did this to improve general service, not
degrade it.
So what are we doing to improve the level of service? Well, for
starters, we've started to thoroughly document every disruption of
service we experience, and write a PIR (Post Incidence Response.)
These are visible to our new (managed) data center operators, where we
have a respectable SLA. Over the last incident, for example, they were
quick to restore the key file while we were asleep (not having 24h
coverage sucks, but at least they call us.)
As when we were on EC2, a lot of man hours are being spent on keeping
the service running, and while it hasn't been *very* evident yet, at
least to the outside, we *are* spending less time pulling out hair,
and we generally get better sleep. The plan is that this will lead to
more quality time to develop much-needed and much-wanted features for
everyone to peruse, and a lot less time having to keep the boat
afloat, so to speak.
We're very aware of how we look to the outside, and I personally more
than anything, want to make Bitbucket a kick ass product. I can't
emphasize enough how much the community support has helped us since
day 1, and I urge everyone to not despair. Changes are coming, a team
totaling 8 people has been hired (they're all starting mid-October),
and things will get better.
> On Sep 23, 2010, at 6:31 AM, Pedro Morais wrote:
> > Hi,
> > Once again we are unable to push our repos using SSH (we're getting
> > permission denied; user morais).
> I'm having the same issue (user dhellmann).
> > I'm tired of sending emails to support, so this time my rant will be
> > public.
> > A code hosting service needs to just work. bitbucket doesn't. It seems
> > every week there's a new, minor-but-blocking-getting-work-done,
> > problem.
> Indeed, this is getting old.
> Doug
> > Jesper, what is being done to improve the level of service?
> > Thanks,
> > Pedro
> > --
> > You received this message because you are subscribed to the Google Groups "bitbucket-users" group.
> > To post to this group, send email to bitbucket-users@googlegroups.com.
> > To unsubscribe from this group, send email to bitbucket-users+unsubscribe@googlegroups.com.
> > For more options, visit this group athttp://groups.google.com/group/bitbucket-users?hl=en.
>>> I'm tired of sending emails to support, so this time my rant will be >>> public. >>> A code hosting service needs to just work. bitbucket doesn't. It seems >>> every week there's a new, minor-but-blocking-getting-work-done, >>> problem.
>> Indeed, this is getting old.
> Yes it is. For us as well.
> Long story:
> When we migrated the new hardware, some things changed on the backend. > On EC2, we had to scale out the repositories across several instances, > as the I/O throughput on a single instance wasn't enough to serve > everyone at decent speeds. For this reason, every load balancer needed > to have a copy of authorized_keys on it, and we did this by using a > "fanout" exchange in our queue (RabbitMQ.) Every time a key was added > (or removed), every load balancer would be told to update the file > accordingly.
> On the new hardware, we have the luxury of actual physical hardware, > and thus, we decided on getting a monster fileserver (with redundancy > of course), and serve up everything to the frontends via NFS. We used > the old code from the load balancers to handle the key files just the > same.
> What first happened was that every frontend (3) would simultaneously > receive a job going "hey, update the key file", and by doing so, > something would get jumbled every 25 keys or so, and somehow the file > would get truncated completely and we'd end up with a file on the NFS > mount with 1 or 2 keys in it, and everyone would be in trouble.
File-locking on NFS is notoriously unreliable.
> We handled this in the simplest way: Disable the job on every box, but > one. This worked fine for a while. Then, however, one of the frontends > died, and had to be rebooted. When it came back up, it faithfully > started the key-job again, and now we had 2 boxes manipulating the > keyfile, and the problem from the first time around came back, and we > had the keyfile wiped again.
> After the migration to the new hardware, we focused mainly on > improving service in general, taking advantage of some of the luxuries > we couldn't before on EC2. One of them was to improve the way the > keyfile was handled. We got rid of the "fanout" exchange and switched > it over to "round robin" class job, so one server would pick up the > key job, manipulate the file, and every frontend would see the change. > This is what's supposed to be the final solution to this.
That does seem like it would be more reliable.
> Right now, I'm not sure what wiped the file, but that is indeed what > happened again. I'm tired of having to deal with repercussions of the > migration--after all, we did this to improve general service, not > degrade it.
Perhaps 2 key updates came at the same time, and were distributed to 2 different servers to process? It seems like a fail-over system, instead of round-robin, would be a more appropriate architecture here. Pass all requests to the same server, unless it stops responding, then pick another one and give all of the requests to it, etc.
> So what are we doing to improve the level of service? Well, for > starters, we've started to thoroughly document every disruption of > service we experience, and write a PIR (Post Incidence Response.) > These are visible to our new (managed) data center operators, where we > have a respectable SLA. Over the last incident, for example, they were > quick to restore the key file while we were asleep (not having 24h > coverage sucks, but at least they call us.)
> As when we were on EC2, a lot of man hours are being spent on keeping > the service running, and while it hasn't been *very* evident yet, at > least to the outside, we *are* spending less time pulling out hair, > and we generally get better sleep. The plan is that this will lead to > more quality time to develop much-needed and much-wanted features for > everyone to peruse, and a lot less time having to keep the boat > afloat, so to speak.
> We're very aware of how we look to the outside, and I personally more > than anything, want to make Bitbucket a kick ass product. I can't > emphasize enough how much the community support has helped us since > day 1, and I urge everyone to not despair. Changes are coming, a team > totaling 8 people has been hired (they're all starting mid-October), > and things will get better.
> Hang in there with us. You'll see.
I appreciate your openness about the behind-the-scenes processes and issues, Jesper. It's the primary reason I haven't walked away already. Service *has* become much more stable on your new hosting service. It sounds like this ssh key issue is a little tricky, and I hope you can get it worked out soon.
> -- > You received this message because you are subscribed to the Google Groups "bitbucket-users" group. > To post to this group, send email to bitbucket-users@googlegroups.com. > To unsubscribe from this group, send email to bitbucket-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/bitbucket-users?hl=en.
> On Sep 23, 9:21 pm, Doug Hellmann <doug.hellm...@gmail.com> wrote: >> On Sep 23, 2010, at 6:31 AM, Pedro Morais wrote:
>>> Hi,
>>> Once again we are unable to push our repos using SSH (we're getting >>> permission denied; user morais).
>> I'm having the same issue (user dhellmann).
> It should be OK again now.
>>> I'm tired of sending emails to support, so this time my rant will be >>> public. >>> A code hosting service needs to just work. bitbucket doesn't. It seems >>> every week there's a new, minor-but-blocking-getting-work-done, >>> problem.
>> Indeed, this is getting old.
> Yes it is. For us as well.
> Long story:
> When we migrated the new hardware, some things changed on the backend. > On EC2, we had to scale out the repositories across several instances, > as the I/O throughput on a single instance wasn't enough to serve > everyone at decent speeds. For this reason, every load balancer needed > to have a copy of authorized_keys on it, and we did this by using a > "fanout" exchange in our queue (RabbitMQ.) Every time a key was added > (or removed), every load balancer would be told to update the file > accordingly.
> On the new hardware, we have the luxury of actual physical hardware, > and thus, we decided on getting a monster fileserver (with redundancy > of course), and serve up everything to the frontends via NFS. We used > the old code from the load balancers to handle the key files just the > same.
> What first happened was that every frontend (3) would simultaneously > receive a job going "hey, update the key file", and by doing so, > something would get jumbled every 25 keys or so, and somehow the file > would get truncated completely and we'd end up with a file on the NFS > mount with 1 or 2 keys in it, and everyone would be in trouble.
> We handled this in the simplest way: Disable the job on every box, but > one. This worked fine for a while. Then, however, one of the frontends > died, and had to be rebooted. When it came back up, it faithfully > started the key-job again, and now we had 2 boxes manipulating the > keyfile, and the problem from the first time around came back, and we > had the keyfile wiped again.
> After the migration to the new hardware, we focused mainly on > improving service in general, taking advantage of some of the luxuries > we couldn't before on EC2. One of them was to improve the way the > keyfile was handled. We got rid of the "fanout" exchange and switched > it over to "round robin" class job, so one server would pick up the > key job, manipulate the file, and every frontend would see the change. > This is what's supposed to be the final solution to this.
> Right now, I'm not sure what wiped the file, but that is indeed what > happened again. I'm tired of having to deal with repercussions of the > migration--after all, we did this to improve general service, not > degrade it.
> So what are we doing to improve the level of service? Well, for > starters, we've started to thoroughly document every disruption of > service we experience, and write a PIR (Post Incidence Response.) > These are visible to our new (managed) data center operators, where we > have a respectable SLA. Over the last incident, for example, they were > quick to restore the key file while we were asleep (not having 24h > coverage sucks, but at least they call us.)
> As when we were on EC2, a lot of man hours are being spent on keeping > the service running, and while it hasn't been *very* evident yet, at > least to the outside, we *are* spending less time pulling out hair, > and we generally get better sleep. The plan is that this will lead to > more quality time to develop much-needed and much-wanted features for > everyone to peruse, and a lot less time having to keep the boat > afloat, so to speak.
> We're very aware of how we look to the outside, and I personally more > than anything, want to make Bitbucket a kick ass product. I can't > emphasize enough how much the community support has helped us since > day 1, and I urge everyone to not despair. Changes are coming, a team > totaling 8 people has been hired (they're all starting mid-October), > and things will get better.
> Hang in there with us. You'll see.
> Jesper
> -- > You received this message because you are subscribed to the Google Groups "bitbucket-users" group. > To post to this group, send email to bitbucket-users@googlegroups.com. > To unsubscribe from this group, send email to bitbucket-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/bitbucket-users?hl=en.
