nao down?
BanMido
the Wiki is inaccessible too.
It seems that the server is down, is anyone else seeing this? If the server
is down, is it due to scheduled maintainance? I did not see any notice to
that effect on http://alt.org/nethack.
But again, I am not too good at paying attention to stuff like that and
still I play Nethack and attempt to ascend, go figure ;)
--
Regards,
Ban.
dtype
> I am not able to telnet to nethack.alt.org(the connection times out) and
> the Wiki is inaccessible too.
Yep, just put a quick update up on site:
"Had a compromise event on the server, but unlike Sony we won't be
down for a month. New OS is installed and going through and slowly
confirming services to bring back up. Nethack stuff is hopefully back
within a day or so. After confirming some things through audit we're
worst case to to a 2 day old snapshot, but hopefully will be up
without loss if we can confirm that it will be safe."
-drew
Janis Papanagnou
>
> "Had a compromise event on the server, but unlike Sony we won't be
> down for a month. New OS is installed and going through and slowly
> confirming services to bring back up. Nethack stuff is hopefully back
> within a day or so. After confirming some things through audit we're
> worst case to to a 2 day old snapshot,
I wouldn't be sad if my recent stupid death[*] gets erased that way. :-}
> but hopefully will be up
> without loss if we can confirm that it will be safe."
Thanks.
Janis
[*] Sloppy and too fast play and inattentiveness.
BanMido
> On Jun 7, 8:06 am, BanMido <smartrefere...@gmail.com> wrote:
>> I am not able to telnet to nethack.alt.org(the connection times out) and
>> the Wiki is inaccessible too.
[...]
> After confirming some things through audit we're
> worst case to to a 2 day old snapshot, but hopefully will be up
> without loss if we can confirm that it will be safe."
Thanks.
I hope everything is safe and you guys can make a full recovery.
--
Regards,
Ban.
dtype
> On Jun 7, 8:06 am, BanMido <smartrefere...@gmail.com> wrote:
>
> > I am not able to telnet to nethack.alt.org(the connection times out) and
> > the Wiki is inaccessible too.
>
> Yep, just put a quick update up on site:
And now an update so you can throw rocks at me (although I'll plead
for mercy given the circumstances), also at http://alt.org :
EDIT Jun 7 1408 PDT: It looks now like I have a plausible explanation
for the major thing that made me suspect a hack. A bug in the Amazon
EC2 AMI I was using will re-generate the ssh host keys on every reboot
rather than just on the first reboot. So a reboot after 500 days
uptime gave me new host keys and I immediately hit the kill switch.
We're still treating this as a hack though since we've already
started, and going clean audit for the new install. This does mean
that there's high confidence that we'll use the saves/etc from the
moment I hit the panic button this morning.
Just for the curious, the series of events were:
* We saw a couple of fishy events that we still haven't explained, but
aren't enough to suspect something horribly bad on their own without
other factors.
* I made a major booboo and tried to ssh to alt.org while logged in as
a service user on another machine, instead of what I thought was my
local console. This caused an ssh key error (it had a cached very very
old key). Cut me slack on this one it was 6am and paxed has just asked
me if something was fishy with the server so I assumed the worst. :)
* I hit the kill switch on alt.org to halt any hack activity.
* I realized I needed some more info from the server, and put it up
for a minute or two, at which point the stupid bug on the EC2 AMI
we're using actually _did_ reset the ssh host key. Of course from here
as I was verifying the key change, it was actually true. Since I keep
snapshot incremental backups I could tell that the key chaged in the
last 2 days.
* I started from a clean OS image, remounted the remaining images, and
started forensics analysis. Since I have a job I wasn't able to get to
more analysis until later in the afternoon, and a couple hours later I
discovered the above.
So we're still unsure about the initial anomoly, but at this point are
just going forward assuming the worst and going with clean audit and
install, especially as we do have a couple of minor unexplained events
that we're still investigating. The good news is that changes we've
made since previous server issues have proven to be very effective,
and it isn't the world's worst task rebuilding the server from the
ground up in the Amazon AWS/EC2 cloud. So that's what paxed and I are
up to now, hopefully to be done in a day or so...
karadoc
I'm somewhat impressed that you have such rigorous procedures for
dealing with hacks. (kill switch, reinstall etc.)
FizzBinary
Patric Mueller
It's variant appreciation day.
Go play your favorite variant.
If you don't have one, play UnNetHack on (us.)un.nethack.nu! ;-)
Bye
Patric
--
NetHack-De: NetHack auf Deutsch - http://nethack-de.sf.net/
UnNetHack: http://apps.sf.net/trac/unnethack/
karadoc
> It's variant appreciation day.
>
> Go play your favorite variant.
Heh. That's a good way to look at it.
I only ever play my own mod these days anyway — and although I don't
really like self promotion, I'm kind of proud of my mod. I think it's
a significant improvement over vanilla. UnNetHack is probably the best
way to go though, because it has its own server & community, and has
ongoing support. (whereas my mod only has when-karadoc-feels-like-
playing-nethack support.)
That reminds me, Patric, did you ever try my two sokoban levels? I
think you said you would consider including them in UnNetHack; and I
think I mentioned that I was expecting someone to say "wtf" about one
of the levels, but no one has...
(for those who are interested, my mod is available at k-mod.sf.net,
and at github.com/karadoc/k-mod)
Janis Papanagnou
> On Jun 8, 5:13 pm, Patric Mueller <bh...@bigfoot.com> wrote:
>> It's variant appreciation day.
>>
>> Go play your favorite variant.
>
> Heh. That's a good way to look at it.
>
> way to go though, because it has its own server & community, and has
What about Spork?
Janis
Jorgen Grahn
Yes -- although some of us old farts dislike the term "hack" being
used as a synonym for "break-in attempt".
/Jorgen
--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .
Pasi Kallinen
Spork's in hibernation.
--
Pasi Kallinen
pa...@alt.org
http://bilious.homelinux.org/ -- NetHack Patch Database
Janis Papanagnou
> Janis Papanagnou <janis_pa...@hotmail.com> wrote:
>> On 08.06.2011 13:52, karadoc wrote:
>>> On Jun 8, 5:13 pm, Patric Mueller <bh...@bigfoot.com> wrote:
>>>> It's variant appreciation day.
>>>>
>>>> Go play your favorite variant.
>>>
>>> Heh. That's a good way to look at it.
>>>
>>> [...] UnNetHack is probably the best
>>> way to go though, because it has its own server & community, and has
>>> ongoing support. [...]
>>
>> What about Spork?
>
> Spork's in hibernation.
Interesting. Is Spork now in some stable final state? Or is the author,
(who was, as far as I recall, very critical about Vanilla's Devteam
hibernation) now also in hibernation?
Janis
karadoc
author asking if the project was still active. I didn't receive a
response.
So although Sporkhack has a lot of good stuff going for it, I don't
think it has the ongoing support that UnNetHack currently has.
Ray
Sigh. I'm just going to have to chalk up "hack" in its current
usage to language change and get over it. Grumble, grumble.
But, meanwhile, I find it nearly ironic that they care enough
to implement such procedures, while still using telnet (which
transmits passwords in the clear) as opposed to SSH (which does
not).
Without going into details of each protocol, I'd like to remind
everyone that using a telnet service exposes the password -- so
don't use the same password that you do for say, your online
banking or your Fetlife profile.
Also? Telnet isn't installed by default on anything anymore.
Every machine I've worked on in the last few years has ssh, but
none of them have telnet unless people have gone out and downloaded
it and clicked through all the dire warnings that the install
packages display. ISP's who still allow shell access (and yes
there are some) don't serve telnet sessions anymore; you use ssh
or you don't get the shell. Your Mileage May Vary, 'cause I
work mainly with servers and *nixes, but from my perspective
the sun has set on Telnet and it has passed into the great
night of obsolescence.
So I'm afraid that telnet-only connectivity is making NAO less
accessible to new users, besides exposing users who aren't aware
of the password risk to sniffing attacks.
We could also discuss the implementation of the service using
a batch script and the resulting potential attack that starts
by transmitting a well-timed U+03 character, but let's leave
that for a different day.
Bear
dtype
> So I'm afraid that telnet-only connectivity is makingNAOless
> accessible to new users, besides exposing users who aren't aware
> of the password risk to sniffing attacks.
The reason for this has to do with the amount of change we'd have to
make to ssh to secure it. We operate in a chroot with very little
passed in, in an environment where we have very strict control over
all the binaries. ssh has unfortunately proven to be a bit more
difficult for us to make work in this environment, in as much as it
supports all manner of remote execution of code, environment variables
to be passed, etc. While all this is well and good for normal unix
accounts, it is a bit problematic for our own environment.
I'd love to just setup ssh and have it work the same way, but to date
it seems we'd need to modify ssh somewhat moderately and I'm worried
that without a lot of attention I'd miss something and leave us
vulnerable to basic ssh functionality we forgot to strip out. telnet
has just offered few enough options that we can pretty easily work
around the potential issues.
As stated in a couple spots, we definitely recommend against using the
same password on NAO as anywhere else.
So there are technical reasons, and I'd rather enable ssh, but
unfortunately right now the wide feature set has made it harder than
trivial to setup without introducing issues.
>
> We could also discuss the implementation of the service using
> a batch script and the resulting potential attack that starts
> by transmitting a well-timed U+03 character, but let's leave
> that for a different day.
I don't think that this represents any potential attack with our
implementation, but always interested to learn otherwise. We don't
execute any shell scripts at all in the nethack environment on NAO.
Everything is explicitly controlled from the first chroot, privs shed,
and fork of nethack and the ttyrec processes.
-drew
Bruno Kazer
> karadoc wrote:
> > I found this story interesting. Thanks. :)
>
> > I'm somewhat impressed that you have such rigorous procedures for
> > dealing with hacks. (kill switch, reinstall etc.)
>
> Sigh. I'm just going to have to chalk up "hack" in its current
> usage to language change and get over it. Grumble, grumble.
>
> But, meanwhile, I find it nearly ironic that they care enough
> to implement such procedures, while still using telnet (which
> transmits passwords in the clear) as opposed to SSH (which does
> not).
>
> Without going into details of each protocol, I'd like to remind
> everyone that using a telnet service exposes the password -- so
> don't use the same password that you do for say, your online
> banking or your Fetlife profile.
I know a guy who lives aboard a space station. He is biologically
maintained in a room in the center which he never leaves. There is a
keypad lock on the door into his chamber which he used as a nethack
password, and wouldn't you know it, someone sniffed it and opened the
door and he was killed because the biologics were not made to handle
open air. Talk about being security challenged. That guy should have
used a different password than the one that kept him alive.
D
Janis Papanagnou
>
> We could also discuss the implementation of the service using
> a batch script and the resulting potential attack that starts
> by transmitting a well-timed U+03 character, but let's leave
> that for a different day.
What's that "U+03 character" issue; mind to elaborate in a
sentence or two?
Janis
Ray
Um, okay. When someone logs in a shell script pops up and
runs nethack. But in order to run the shell script a shell
must first be loaded. While the game traps control-c once it's
running and taking input, the shell sees it as a command to
abort a shell script until that time.
So if someone logs in and then manages to get a control-c (U+03)
character sent to the host in the half-second or so while the shell
is executing the command (ie, in the load time before nethack comes
up and opens stdin) they can abort the script. There are more
steps to it, but in the worst case, an attacker could get a shell
prompt.
What damage they could do, exactly, depends on what the chroot
jail protects, but a shell prompt is adequate, for example, to
create and delete files - so even if critical files are well
protected, the attacker could, eg, still fill disk space until
the partition is full, whereupon new games (if trying to write
in the same partition) would bomb out with a write error.
Bear
(who doesn't crack systems anymore since 1986 when they started
getting serious about making it illegal)
Janis Papanagnou
> Janis Papanagnou wrote:
>
>> What's that "U+03 character" issue; mind to elaborate in a
>> sentence or two?
>
> runs nethack. But in order to run the shell script a shell
> must first be loaded. While the game traps control-c once it's
> running and taking input, the shell sees it as a command to
> abort a shell script until that time.
Aha! - I always wrote that Ctrl-C (instead of U+03), so I thought
you meant something else by U+03, something related with characters
created and exchanged in an ordinary way. Thanks for clearing that.
>
> So if someone logs in and then manages to get a control-c (U+03)
> character sent to the host in the half-second or so while the shell
> is executing the command (ie, in the load time before nethack comes
> up and opens stdin) they can abort the script. There are more
> steps to it, but in the worst case, an attacker could get a shell
> prompt.
I thought that such login-like tools would not necessarily have
to run a shell, rather just exec the new program to replace the
current process.
Janis
>
> [...]
dtype
wrote:
> I thought that such login-like tools would not necessarily have
> to run a shell, rather just exec the new program to replace the
> current process.
Correct. At NAO there is no shell involved anywhere in our process.
Folks that are interested are more than welcome to peruse the open
source dgamelaunch code that handles things.
I'm not claiming there will never be a hack into dgamelaunch, but this
particular kind of easy attack has been well thought about.
At this point we're on to how well we can lock down the potential
overruns since even though our potential library for breaking chroot
is small and unlikely, that's probably the best avenue of getting
something done that is truly bad. Otherwise you could affect the dgl
environment maybe (and even that should be fairly hard), but that's
backed up constantly and I guess I'm just not to worried about that
particular area in the chroot itself given how easy it is to restore
(and at worst lose some game progress for folks, which I guess could
be disastrous to the right person's conduct game. :)
-drew
Ray
> On 04.07.2011 23:58, Ray wrote:
>> Janis Papanagnou wrote:
>>
>>> What's that "U+03 character" issue; mind to elaborate in a
>>> sentence or two?
>>
>> Um, okay. When someone logs in a shell script pops up and
>> runs nethack. But in order to run the shell script a shell
>> must first be loaded. While the game traps control-c once it's
>> running and taking input, the shell sees it as a command to
>> abort a shell script until that time.
>
> Aha! - I always wrote that Ctrl-C (instead of U+03), so I thought
> you meant something else by U+03, something related with characters
> created and exchanged in an ordinary way. Thanks for clearing that.
Same character. I wasn't initially going to say the input method
for fear of giving instructions to script kiddies. But since NAO
is apparently well-protected from this issue, (comment downthread)
I guess it doesn't matter.
>> So if someone logs in and then manages to get a control-c (U+03)
>> character sent to the host in the half-second or so while the shell
>> is executing the command (ie, in the load time before nethack comes
>> up and opens stdin) they can abort the script. There are more
>> steps to it, but in the worst case, an attacker could get a shell
>> prompt.
> I thought that such login-like tools would not necessarily have
> to run a shell, rather just exec the new program to replace the
> current process.
Um, right.... that would be the "more steps to it" that I mentioned.
And the new process to replace the current process would, to facilitate
almost any attack in the world, be a shell prompt. I mean, it's true
the attacker doesn't have to start a shell. But given the choice, I
expect most attackers to go for one if one is available, and I think
that it is the worst case for security.
Bear
dtype
> Um, right.... that would be the "more steps to it" that I mentioned.
> And the new process to replace the current process would, to facilitate
> almost any attack in the world, be a shell prompt. I mean, it's true
> the attacker doesn't have to start a shell. But given the choice, I
> expect most attackers to go for one if one is available, and I think
> that it is the worst case for security.
And just fwiw, there isn't a shell anywhere in the chroot environment.
The only binaries are dgamelaunch, nethack, and a couple of service
programs. (There are actually shell scripts in there but they aren't
executed from within the chroot, but rather the base OS.)
So again while some overrun may be possible, at least you'd be stuck
at that point executing your own binaries rather than a shell.
(Notwithstanding long discussions of how insecure chroots are with
Linux, but our particular implementation should at least make things
hard.)
-drew
Janis Papanagnou
> Janis Papanagnou wrote:
>
>> On 04.07.2011 23:58, Ray wrote:
>>> Janis Papanagnou wrote:
>>>
>>>
>>> [...] While the game traps control-c once it's
>>> running and taking input, the shell sees it as a command to
>>> abort a shell script until that time.
>>
>> Aha! - I always wrote that Ctrl-C (instead of U+03), so I thought
>> you meant something else by U+03, something related with characters
>> created and exchanged in an ordinary way. Thanks for clearing that.
>
That part was what confused me; speaking about "characters".
U+03 is a typical character notation (Unicode), while Ctrl-C
is a typical keyboard-key command notation, in this case to
generate *signals*. That characters are involved as transport
medium between keyboard-controls and IPC signals to processes
is not the essential part of the issue you pointed out. Anyway,
it's clear now know what you meant. :-)
Janis