My main linux box at home got hit by some sort of system cracker. Not a very good one, I think, or at least whatever he exploited wasn't as dangerous as it could have been.
I do not think he got root...he just found a testing account that had a password that was too easy to crack. He didn't (or couldn't) make much of an attempt to cover his tracks, as the tools I used to find and disable his stuff were top, ps, ls, crontab, kill and killall...usually the more dangerous crackers override those, from what I understand.
My guess on how he got in: I have AT&T U-Verse, including their residential gateway. I set its firewall up to let port 80 and port 22 point to my linux box, so I could ssh home from work, and so my wife could do some html/css experiments on it (she hasn't done that yet though). Sometimes the gateway seems to get a bur under its saddle and says "there is a router behind this router, do you want me to resolve the problem?" If you say yes, it puts one machine into "DMZplus" mode, which as far as I can tell sends ALL ports to that machine. Nice. My linux machine was DMZplus-ified when I went in to shut off ports 80 and 22...no telling how long it had been that way. So that doesn't narrow down what he could have exploited, but maybe points to how the doors all got unlocked.
As near as I can tell the main activity he was doing was portscanning, but what he might have done with the results, I don't know. After killing his account I shut the machine off. We had been having internet trouble across the board since the thunderstorm this weekend, and I'm wondering whether maybe this cracker had something to do with the trouble. It's still not 100% great though and we have a tech coming to our house this evening.
I'm thinking about burning the test account's home dir to a CD for inspection. I haven't deleted any of his files, only his crontab entry that spawned a new job every minute if it got killed. Would anyone on the list want a look at the files?
Beyond that...to get the machine back to a usable state...does anyone have any suggestions? It seems like it would be foolish to assume just killing that account would take care of the problem for good, but would that be sufficient to hobble along for a week or so until I can cp my own home dir elsewhere and reinstall? Or would it be safer to keep the machine off the network until I can get that done?
Back up any data that's irreplaceable and reinstall. It's not worth your
time to try and verify that the rest of the system wasn't compromised.
Don'ts:
1) Don't use a firewall that does stupid stuff like that. Buy a $30
Linksys.
2) Use the system in a meaningful way until it's been reinstalled.
Dos:
1) If you need to remotely SSH into the box, use the AllowUsers feature of
SSH and TCPwrappers.
2) Reinstall any other systems on your network if they aren't current on
patches or share common passwords with the compromised system.
Jeffrey.
On Thu, Jun 18, 2009 at 3:32 PM, Kendric Beachey
<kendric.beac...@gmail.com>wrote:
> My main linux box at home got hit by some sort of system cracker. Not
> a very good one, I think, or at least whatever he exploited wasn't as
> dangerous as it could have been.
> I do not think he got root...he just found a testing account that had
> a password that was too easy to crack. He didn't (or couldn't) make
> much of an attempt to cover his tracks, as the tools I used to find
> and disable his stuff were top, ps, ls, crontab, kill and
> killall...usually the more dangerous crackers override those, from
> what I understand.
> My guess on how he got in: I have AT&T U-Verse, including their
> residential gateway. I set its firewall up to let port 80 and port 22
> point to my linux box, so I could ssh home from work, and so my wife
> could do some html/css experiments on it (she hasn't done that yet
> though). Sometimes the gateway seems to get a bur under its saddle
> and says "there is a router behind this router, do you want me to
> resolve the problem?" If you say yes, it puts one machine into
> "DMZplus" mode, which as far as I can tell sends ALL ports to that
> machine. Nice. My linux machine was DMZplus-ified when I went in to
> shut off ports 80 and 22...no telling how long it had been that way.
> So that doesn't narrow down what he could have exploited, but maybe
> points to how the doors all got unlocked.
> As near as I can tell the main activity he was doing was portscanning,
> but what he might have done with the results, I don't know. After
> killing his account I shut the machine off. We had been having
> internet trouble across the board since the thunderstorm this weekend,
> and I'm wondering whether maybe this cracker had something to do with
> the trouble. It's still not 100% great though and we have a tech
> coming to our house this evening.
> I'm thinking about burning the test account's home dir to a CD for
> inspection. I haven't deleted any of his files, only his crontab
> entry that spawned a new job every minute if it got killed. Would
> anyone on the list want a look at the files?
> Beyond that...to get the machine back to a usable state...does anyone
> have any suggestions? It seems like it would be foolish to assume
> just killing that account would take care of the problem for good, but
> would that be sufficient to hobble along for a week or so until I can
> cp my own home dir elsewhere and reinstall? Or would it be safer to
> keep the machine off the network until I can get that done?
> Kendric Beachey
--
"He that would make his own liberty secure must guard even his enemy from
oppression; for if he violates this duty he establishes a precedent that
will reach to himself." -- Thomas Paine
> My main linux box at home got hit by some sort of system cracker. Not
> a very good one, I think, or at least whatever he exploited wasn't as
> dangerous as it could have been.
> I do not think he got root...he just found a testing account that had
> a password that was too easy to crack. He didn't (or couldn't) make
> much of an attempt to cover his tracks, as the tools I used to find
> and disable his stuff were top, ps, ls, crontab, kill and
> killall...usually the more dangerous crackers override those, from
> what I understand.
> My guess on how he got in: I have AT&T U-Verse, including their
> residential gateway. I set its firewall up to let port 80 and port 22
> point to my linux box, so I could ssh home from work, and so my wife
> could do some html/css experiments on it (she hasn't done that yet
> though). Sometimes the gateway seems to get a bur under its saddle
> and says "there is a router behind this router, do you want me to
> resolve the problem?" If you say yes, it puts one machine into
> "DMZplus" mode, which as far as I can tell sends ALL ports to that
> machine. Nice. My linux machine was DMZplus-ified when I went in to
> shut off ports 80 and 22...no telling how long it had been that way.
> So that doesn't narrow down what he could have exploited, but maybe
> points to how the doors all got unlocked.
> As near as I can tell the main activity he was doing was portscanning,
> but what he might have done with the results, I don't know. After
> killing his account I shut the machine off. We had been having
> internet trouble across the board since the thunderstorm this weekend,
> and I'm wondering whether maybe this cracker had something to do with
> the trouble. It's still not 100% great though and we have a tech
> coming to our house this evening.
> I'm thinking about burning the test account's home dir to a CD for
> inspection. I haven't deleted any of his files, only his crontab
> entry that spawned a new job every minute if it got killed. Would
> anyone on the list want a look at the files?
> Beyond that...to get the machine back to a usable state...does anyone
> have any suggestions? It seems like it would be foolish to assume
> just killing that account would take care of the problem for good, but
> would that be sufficient to hobble along for a week or so until I can
> cp my own home dir elsewhere and reinstall? Or would it be safer to
> keep the machine off the network until I can get that done?
I agree with Jeffrey's assessement, for the most part. You really
shouldn't trust that machine again, until you reinstall.
You really won't have a choice about using that Residential Gateway.
To get service, at all, I think you'll have to leave that machine in
place. But don't trust it as your firewall. Put another firewall
between you and it, and just consider everything outside your firewall
own as untrusted.
If you are going to put services out there where they are reachable on
the internet, you need to plan on keeping them patched. You should
either set it up to update automatically (does anyone here have
experience with that) or set up some way to remind you to do it. You
really can't afford to let yourself get significantly behind on
patches for a service that is internet facing.
On Thu, Jun 18, 2009 at 3:52 PM, Jeffrey Watts<jeffrey.w.wa...@gmail.com> wrote:
> Back up any data that's irreplaceable and reinstall. It's not worth your
> time to try and verify that the rest of the system wasn't compromised.
> Don'ts:
> 1) Don't use a firewall that does stupid stuff like that. Buy a $30
> Linksys.
> 2) Use the system in a meaningful way until it's been reinstalled.
> Dos:
> 1) If you need to remotely SSH into the box, use the AllowUsers feature of
> SSH and TCPwrappers.
> 2) Reinstall any other systems on your network if they aren't current on
> patches or share common passwords with the compromised system.
> Jeffrey.
> On Thu, Jun 18, 2009 at 3:32 PM, Kendric Beachey <kendric.beac...@gmail.com>
> wrote:
>> My main linux box at home got hit by some sort of system cracker. Not
>> a very good one, I think, or at least whatever he exploited wasn't as
>> dangerous as it could have been.
>> I do not think he got root...he just found a testing account that had
>> a password that was too easy to crack. He didn't (or couldn't) make
>> much of an attempt to cover his tracks, as the tools I used to find
>> and disable his stuff were top, ps, ls, crontab, kill and
>> killall...usually the more dangerous crackers override those, from
>> what I understand.
>> My guess on how he got in: I have AT&T U-Verse, including their
>> residential gateway. I set its firewall up to let port 80 and port 22
>> point to my linux box, so I could ssh home from work, and so my wife
>> could do some html/css experiments on it (she hasn't done that yet
>> though). Sometimes the gateway seems to get a bur under its saddle
>> and says "there is a router behind this router, do you want me to
>> resolve the problem?" If you say yes, it puts one machine into
>> "DMZplus" mode, which as far as I can tell sends ALL ports to that
>> machine. Nice. My linux machine was DMZplus-ified when I went in to
>> shut off ports 80 and 22...no telling how long it had been that way.
>> So that doesn't narrow down what he could have exploited, but maybe
>> points to how the doors all got unlocked.
>> As near as I can tell the main activity he was doing was portscanning,
>> but what he might have done with the results, I don't know. After
>> killing his account I shut the machine off. We had been having
>> internet trouble across the board since the thunderstorm this weekend,
>> and I'm wondering whether maybe this cracker had something to do with
>> the trouble. It's still not 100% great though and we have a tech
>> coming to our house this evening.
>> I'm thinking about burning the test account's home dir to a CD for
>> inspection. I haven't deleted any of his files, only his crontab
>> entry that spawned a new job every minute if it got killed. Would
>> anyone on the list want a look at the files?
>> Beyond that...to get the machine back to a usable state...does anyone
>> have any suggestions? It seems like it would be foolish to assume
>> just killing that account would take care of the problem for good, but
>> would that be sufficient to hobble along for a week or so until I can
>> cp my own home dir elsewhere and reinstall? Or would it be safer to
>> keep the machine off the network until I can get that done?
>> Kendric Beachey
> --
> "He that would make his own liberty secure must guard even his enemy from
> oppression; for if he violates this duty he establishes a precedent that
> will reach to himself." -- Thomas Paine
You should not count on being able to find hidden files with any
option on the 'ls' command. There are various kernel modules which
can be included in rootkits and used to hide files and/or processes.
Once these kernel modules are loaded, they can be used to do things
like make an executable file read as though it were a different file
when opened for reading, so that the checksum can appear to match the
distributed version of the executable even though the binary has
change.
Seriously, folks, it is really more trouble than you want to invest to
make sure your system is clean without reinstalling it. Don't try to
clean it up. Just backup the files you really need, and reinstall.
On Thu, Jun 18, 2009 at 3:55 PM, Joseph Kearns<joekearns....@gmail.com> wrote:
> Kendric,
> Be sure to look for hidden folders: `ls -ap` or some other method. Crackers
> tend to hide their utilities.
> You might also find your log files helpful (unless the cracker was clever
> enough to modify them).
> Personally, I would follow Jeffery's advice and do a backup/reinstall.
> --Joe
> On Thu, Jun 18, 2009 at 3:32 PM, Kendric Beachey <kendric.beac...@gmail.com>
> wrote:
>> My main linux box at home got hit by some sort of system cracker. Not
>> a very good one, I think, or at least whatever he exploited wasn't as
>> dangerous as it could have been.
>> I do not think he got root...he just found a testing account that had
>> a password that was too easy to crack. He didn't (or couldn't) make
>> much of an attempt to cover his tracks, as the tools I used to find
>> and disable his stuff were top, ps, ls, crontab, kill and
>> killall...usually the more dangerous crackers override those, from
>> what I understand.
>> My guess on how he got in: I have AT&T U-Verse, including their
>> residential gateway. I set its firewall up to let port 80 and port 22
>> point to my linux box, so I could ssh home from work, and so my wife
>> could do some html/css experiments on it (she hasn't done that yet
>> though). Sometimes the gateway seems to get a bur under its saddle
>> and says "there is a router behind this router, do you want me to
>> resolve the problem?" If you say yes, it puts one machine into
>> "DMZplus" mode, which as far as I can tell sends ALL ports to that
>> machine. Nice. My linux machine was DMZplus-ified when I went in to
>> shut off ports 80 and 22...no telling how long it had been that way.
>> So that doesn't narrow down what he could have exploited, but maybe
>> points to how the doors all got unlocked.
>> As near as I can tell the main activity he was doing was portscanning,
>> but what he might have done with the results, I don't know. After
>> killing his account I shut the machine off. We had been having
>> internet trouble across the board since the thunderstorm this weekend,
>> and I'm wondering whether maybe this cracker had something to do with
>> the trouble. It's still not 100% great though and we have a tech
>> coming to our house this evening.
>> I'm thinking about burning the test account's home dir to a CD for
>> inspection. I haven't deleted any of his files, only his crontab
>> entry that spawned a new job every minute if it got killed. Would
>> anyone on the list want a look at the files?
>> Beyond that...to get the machine back to a usable state...does anyone
>> have any suggestions? It seems like it would be foolish to assume
>> just killing that account would take care of the problem for good, but
>> would that be sufficient to hobble along for a week or so until I can
>> cp my own home dir elsewhere and reinstall? Or would it be safer to
>> keep the machine off the network until I can get that done?
I also use fail2ban for ssh. When someone tries to guess an account's
password, they get three tries, and then they are blocked from all IP
connectivity from their address for one year in iptables.
If you're interested in analysing the attack, boot to some livecd, and
make two copies of the entire drive to image files on other drives
with dd. Then compare both images' hashes and the hash of the drive.
Then restore your last backup after reformatting the current drive, or
on a new drive. Then loopback mount the image, and use gnu find to
look for all files changed during the period of suspected compromise.
They could have changed the times of files they owned though, so the
better way would be to compare its state at time of imagine to its
state at the time of your last trusted backup.
On Thu, Jun 18, 2009 at 15:52, Jeffrey Watts<jeffrey.w.wa...@gmail.com> wrote:
> Back up any data that's irreplaceable and reinstall. It's not worth your
> time to try and verify that the rest of the system wasn't compromised.
> Don'ts:
> 1) Don't use a firewall that does stupid stuff like that. Buy a $30
> Linksys.
> 2) Use the system in a meaningful way until it's been reinstalled.
> Dos:
> 1) If you need to remotely SSH into the box, use the AllowUsers feature of
> SSH and TCPwrappers.
> 2) Reinstall any other systems on your network if they aren't current on
> patches or share common passwords with the compromised system.
> Jeffrey.
> On Thu, Jun 18, 2009 at 3:32 PM, Kendric Beachey <kendric.beac...@gmail.com>
> wrote:
>> My main linux box at home got hit by some sort of system cracker. Not
>> a very good one, I think, or at least whatever he exploited wasn't as
>> dangerous as it could have been.
>> I do not think he got root...he just found a testing account that had
>> a password that was too easy to crack. He didn't (or couldn't) make
>> much of an attempt to cover his tracks, as the tools I used to find
>> and disable his stuff were top, ps, ls, crontab, kill and
>> killall...usually the more dangerous crackers override those, from
>> what I understand.
>> My guess on how he got in: I have AT&T U-Verse, including their
>> residential gateway. I set its firewall up to let port 80 and port 22
>> point to my linux box, so I could ssh home from work, and so my wife
>> could do some html/css experiments on it (she hasn't done that yet
>> though). Sometimes the gateway seems to get a bur under its saddle
>> and says "there is a router behind this router, do you want me to
>> resolve the problem?" If you say yes, it puts one machine into
>> "DMZplus" mode, which as far as I can tell sends ALL ports to that
>> machine. Nice. My linux machine was DMZplus-ified when I went in to
>> shut off ports 80 and 22...no telling how long it had been that way.
>> So that doesn't narrow down what he could have exploited, but maybe
>> points to how the doors all got unlocked.
>> As near as I can tell the main activity he was doing was portscanning,
>> but what he might have done with the results, I don't know. After
>> killing his account I shut the machine off. We had been having
>> internet trouble across the board since the thunderstorm this weekend,
>> and I'm wondering whether maybe this cracker had something to do with
>> the trouble. It's still not 100% great though and we have a tech
>> coming to our house this evening.
>> I'm thinking about burning the test account's home dir to a CD for
>> inspection. I haven't deleted any of his files, only his crontab
>> entry that spawned a new job every minute if it got killed. Would
>> anyone on the list want a look at the files?
>> Beyond that...to get the machine back to a usable state...does anyone
>> have any suggestions? It seems like it would be foolish to assume
>> just killing that account would take care of the problem for good, but
>> would that be sufficient to hobble along for a week or so until I can
>> cp my own home dir elsewhere and reinstall? Or would it be safer to
>> keep the machine off the network until I can get that done?
>> Kendric Beachey
> --
> "He that would make his own liberty secure must guard even his enemy from
> oppression; for if he violates this duty he establishes a precedent that
> will reach to himself." -- Thomas Paine
On Thu, Jun 18, 2009 at 4:04 PM, Adrian Griffis <adrian...@gmail.com> wrote:
> You really won't have a choice about using that Residential Gateway. > To get service, at all, I think you'll have to leave that machine in > place. But don't trust it as your firewall. Put another firewall > between you and it, and just consider everything outside your firewall > own as untrusted.
Thanks for clarifying. I meant for him to put the new firewall in between his AT&T box and the rest of the network. :)
If you are going to put services out there where they are reachable on
> the internet, you need to plan on keeping them patched. You should > either set it up to update automatically (does anyone here have > experience with that) or set up some way to remind you to do it. You > really can't afford to let yourself get significantly behind on > patches for a service that is internet facing.
Most Linux distros have an update script that should run automatically. If it's not, it usually just needs to be chkconfiged on.
J.
--
"He that would make his own liberty secure must guard even his enemy from oppression; for if he violates this duty he establishes a precedent that will reach to himself." -- Thomas Paine
My two cents worth, in addition to the plethora of good suggestions you have already gotten, I always have my external ssh on a non-standard port. No sense in making it an easier for hackers than you have to. ;) Rezty
<tongue location="cheek"> I use random ports for everything. That
makes me more secure.... Surely history has shown that anything
standard is insecure... I also randomize the pocket where I keep my
wallet, and car keys, Occasionally drive on the opposite side of the
road, etc...</tongue>
On Mon, Jun 22, 2009 at 09:37, Rezty Felty<rfe...@kc-felty.net> wrote:
> My two cents worth, in addition to the plethora of good suggestions you have
> already gotten, I always have my external ssh on a non-standard port. No
> sense in making it an easier for hackers than you have to. ;)
> Rezty
On Mon, Jun 22, 2009 at 9:37 AM, Rezty Felty <rfe...@kc-felty.net> wrote: > My two cents worth, in addition to the plethora of good suggestions you > have already gotten, I always have my external ssh on a non-standard port. > No sense in making it an easier for hackers than you have to. ;)
I'm with Chris, and will also add... that tossing around the word "expert"
along with "intrusion" and "postmortem" would inevitably in my circles lead
you to someone who gets _paid_ a fairly significant chunk of money to
perform such services. I note replies from experts in our market are
conspicuously absent. I thought you might like to know why. =)
D.
On Mon, Jun 22, 2009 at 2:38 PM, Christofer C. Bell <
christofer.c.b...@gmail.com> wrote:
> On Mon, Jun 22, 2009 at 9:37 AM, Rezty Felty <rfe...@kc-felty.net> wrote:
>> My two cents worth, in addition to the plethora of good suggestions you
>> have already gotten, I always have my external ssh on a non-standard port.
>> No sense in making it an easier for hackers than you have to. ;)
> nmap?
> --
> Chris
-- "If only there were evil people somewhere insidiously committing evil deeds,
and it were necessary to separate them from the rest of us and destroy them.
But the line dividing good and evil cuts through the heart of every human
being. And who is willing to destroy a piece of his own heart?"
~ Alexander Solzhenitsyn
hahaha! No joke I'm sure. :-) I have no actual need for the
postmortem...just offering up the evidence in case anyone likes
looking at these things as a hobby sort of thing.
Nick Mucci took me up on the offer; I burned the bad dude's home dir
to a CD and passed it to him.
The actual machine has been reinstalled from scratch with jaunty, and
all updates applied (and set to do security updates automatically).
I've been unable to get the nvidia glx drivers working since the
reinstall, so I'm on the old "nv" driver, but as a consolation, that
means my screen actually shows what I'm doing when I'm logging in.
With the "real" nvidia drivers it apparently used some weird mode that
my monitor couldn't show, so I had to login blindly.
On Mon, Jun 22, 2009 at 3:42 PM, Dustin Decker<dustin.dec...@gmail.com> wrote:
> I'm with Chris, and will also add... that tossing around the word "expert"
> along with "intrusion" and "postmortem" would inevitably in my circles lead
> you to someone who gets _paid_ a fairly significant chunk of money to
> perform such services. I note replies from experts in our market are
> conspicuously absent. I thought you might like to know why. =)
> D.
> On Mon, Jun 22, 2009 at 2:38 PM, Christofer C. Bell
> <christofer.c.b...@gmail.com> wrote:
>> On Mon, Jun 22, 2009 at 9:37 AM, Rezty Felty <rfe...@kc-felty.net> wrote:
>>> My two cents worth, in addition to the plethora of good suggestions you
>>> have already gotten, I always have my external ssh on a non-standard port.
>>> No sense in making it an easier for hackers than you have to. ;)
>> nmap?
>> --
>> Chris
> --
> "If only there were evil people somewhere insidiously committing evil deeds,
> and it were necessary to separate them from the rest of us and destroy them.
> But the line dividing good and evil cuts through the heart of every human
> being. And who is willing to destroy a piece of his own heart?"
> ~ Alexander Solzhenitsyn
|
