But twice is simply unacceptable.
> Once is unfortunate, and I even had some sympathy for Gradwell.
>
> But twice is simply unacceptable.
I agree, and not a word from Gradwell - I'd expect to receive an email
at least.
Unfortunately my Dev account was automatically renewed recently but I
very much doubt it will be again.
Stuart
--
cut that out to reply
This is outrageous - to leave the stable door wide open, and the shit
not properly shovelled up from the last time. A ticket a raised about
the permissions problem yesterday morning has not even been looked at.
I shudder to think what will happen over Christmas. For myself, I do
not intend still to be here then. I have been a faithful customer of
Peter's since he had Clicknames; but enough is enough. How often am I
going to have to clean up this shit, desperately hoping that I can do so
before my clients become aware of it??
This is NOT how I planned to spend the last Sunday before Christmas -
sitting in the office, frantically running FTP. I was planning to go
out with my other half for a really nice Sunday lunch, and then help a
dear but non-techie friend get her laptop set up on her home network.
I wonder how Gradwell staff are spending today?
--
Molly Mockford
Pagination Associates
http://www.pagination.co.uk
Seconded. A quick check of one of my sites set off all manner of malware
alerts. At least Google haven't yet flagged them as infected...
Oh dear.
Can someone please tell us when this nasty hole is going to be properly
plugged?
--
David Pearson
Same here. Luckily (but also slightly embarassingly) I was contacted
by my customers the first time as the exploit broke my main page.
After restoring the normal files I wrote a php script to clean it all
and ran it on my site, with still no word from gradwell.
Then two days ago the shop stopped running, I contacted gradwell, they
said they would look into it the following day - I have 3 days before
shutting the shop and I can't have it down for almost a day. That is
when I found out the hack wasn't just me. Turned out I couldn't write
sessions, so I put a fix for that on the site.
The following day gradwell 'status' finally said something was up. I
am really lucky I can patch around these faults myself, as otherwise
it would have been down for days.
I now have put a cron script to check the site and disinfect it if it
happens again.
Not so much annoyed it happened, just really unhappy with the lack of
contact about it. I have now signed up with a new ISP for after
christmas (more issues than just that). I have been a faithful
customer for almost 10 years too, but I just get the feeling they
don't really care they way they used to.
On Twitter:
------------------------------------------------------------
Update #3: Sunday 19th December 2010 @ 12:47 (GMT) (Posted by Dean
Green)
Our script is still running and we will provide a further update later
this afternoon
------------------------------------------------------------
Ah - that's all OK then (not) :(
I've had no email from Gradwell yet - waiting...
--
David Pearson
See also: http://www.gradwellstatus.com/viewpost.php?id=154
Injected code on hosted websites
Sunday 19th December 2010 @ 12:55 (GMT) (Posted by Dean Green)
Affecting Web Hosting
Impact Level: High
Overview
Reported injected code on hosted websites
Customer Impact
Some users are seeing injected code on the end of existing webpages
Gradwell Action
On Friday the 15th some users reported their sites had been injected
with malicious code by a third party, this code was adding an iframe
onto the end of existing webpages.
We executed a cleanup script for this which ran through all users
files looking at files with all usual extensions .html, .htm, .php etc
and
removed the offending piece of code.
A further injection attempt was made this morning around 5:30, but
this attempt was blocked by an on-call sysadmin before the exploit
could fully deliver its payload.
The target for the exploit is our PHP 4.4 cluster, which we announced
the end of life for a while ago. To keep our network secure for
everyone we will be removing these servers from service by the end of
this week (week ending Christmas day) and moving all users across to
our PHP 5.2 platform.
We have a test link for you to check your websites compatibility with
PHP 5.2, and you can use the test link in the following manner - if
your site name is 'www.example.com', you would enter
'www.example.com.v-web-php52.gradwell.net'. This will route you to a
PHP
5.2 server rather than a PHP 4.4 server where you may evaluate your
webpages and your code.
Our sysadmin team are rerunning the clean up script across all users
filespace, one fileserver at a time, to remove the injected code. This
does take quite a while to complete as each file needs to be inspected
and adjusted if necessary but we will continue to work until we have
disinfected all fileservers.
Once this work is finished, we will update further.
If you still have webpages showing injected code after our work has
completed, ie if you are using non-standard file extensions, please
raise a support ticket and we will alter the script and rerun on a
user-by-user basis.
Status
We are currently working to resolve this issue and expect to provide
the next update by Sunday 19th December 2010 @ 18:00 (GMT)
Finally a more helpful and "open" update:
http://www.gradwellstatus.com/viewpost.php?id=154
Sure this is by no means the end of, or the full, story - but it's a
start.
Looks like it's going to be a busy week re-loading everything daily!
--
David Pearson
As it's Christmas, I can safely take down my site rather than risk it
being blacklisted. Never done that before. I presume I just delete all
files via FTP and reload once the mess has been sorted or look for a new
ISP in the New Year.
--
David Lawson
Just ftp rename your webs directory. If the script is anywhere sensible
(OK, doubtful, perhaps), it'll still find the affected files and fix 'em
for you.
--
Tony van der Hoff | mailto:to...@vanderhoff.org
Buckinghamshire England |
As this is in the public domain already, in the above status page:
"The target for the exploit is our PHP 4.4 cluster, which we announced
the end of life for a while ago. "
A while ago!
In a post in the newsgroup dated 09 Oct *2009* a Gradwell email was
quoted:
"If you have any domains remaining on the old cluster after Wed, 19
Aug 2009, they will be migrated to the PHP 4.4 cluster."
I recall such an email myself. Indeed, I contributed to the same
thread [Subject: Re: upgrading websites from PHP 4.4 to PHP 5.2]
detailing problems with the Gradwell migration instruction and how I
resolved those problems.
So, even if the migration schedule was staggered across the customer
base it's difficult to see why the cluster is still opertaional over
12 months later.
Chris
--
Chris Salter
> So, even if the migration schedule was staggered across the customer
> base it's difficult to see why the cluster is still opertaional over
> 12 months later.
Perhaps some customers had major issues with the migration and thinbgs
were delayed to allow their websites to remain operational while they
were re-coded.
Rgds
Denis McMahon
[Senior moment! I didn't spot the "*to* the PHP 4.4 cluster", probably
because the thread subject was "upgrading websites *from* PHP 4.4 to
PHP 5.2".]
>I recall such an email myself. Indeed, I contributed to the same
>thread [Subject: Re: upgrading websites from PHP 4.4 to PHP 5.2]
>detailing problems with the Gradwell migration instruction and how I
>resolved those problems.
>
>So, even if the migration schedule was staggered across the customer
>base it's difficult to see why the cluster is still opertaional over
>12 months later.
Instead of the above, I am now unable to locate any end of life notice
for the PHP 4.4 cluster apart from being offered as an alternative
(PHP 4.4 virtual cluster) to the migration to PHP 5.2 in a Gradwell
notice dated 31st July 2009. I had moved all my sites to the PHP 5.2
cluster in May 2009.
Apologies for the confusion.
Chris
--
Chris Salter
See my correction to my own post. It would appear the 'PHP 4.4 virtual
cluster' was offered as an alternative to migration to PHP 5.2 circa
2nd/3rd quarter 2009. Maybe only customers on the PHP 4.4 virtual
cluster received an subsequent 'end of life notice'. I appreciate your
point, just seems a long time for an interim measure to be in place.
Chris
--
Chris Salter
I've taken our sites off line by changing the .htaccess file to
password protect the web folder. Anyone trying to access the site is
prompted for a password and blocked from viewing any of the pages.
This leaves everything in place, and still allows us to access the
site to check things after the clean up, without exposing our users to
the infected files.
>As this is in the public domain already, in the above status page:
>
>"The target for the exploit is our PHP 4.4 cluster, which we announced
>the end of life for a while ago. "
This is simply not true. My sites on PHP5.2 were hacked.
Linux v-web-php52-5.gradwell.net 2.6.18-92.1.6.el5 #1 SMP Wed Jun 25
13:49:24 EDT 2008 i686
> >"The target for the exploit is our PHP 4.4 cluster, which we announced
> >the end of life for a while ago. "
>
> This is simply not true. My sites on PHP5.2 were hacked.
As were ours. I think what it means is that they gained access via a
weakness in the 4.4 cluster. Once they had done that they could write
to any files. The files in the /home directory are in the same
location whether you are using PHP 4.4 or the 5.2.
They are not saying that just files server by 4.4 were affected.
Yes, better late than never I suppose.
--
Woody
That was my interpretation as well although I am not cognisant of the
nuts and bolts.
Chris
--
Chris Salter
Is it possible that the initial 4.4 cluster exploit let them gain
sufficient privileges inside the Gradwell network to compromise other
machines?
In which case the target of the exploit that blew security away was the
4.4 cluster, and having compromised the 4.4 cluster they then had
sufficient "insider" access from the 4.4 cluster to other systems that
they were able to run scripts on those as well?
Tools, systems, processes etc which exist to make system management
easier across the different clusters may have provided a pathway for
executing commands on "your" cluster from the exploited 4.4 cluster
after the exploit had broken the 4.4 cluster security.
As an example, from the root account on my desktop machine I can access
files on the NFS mounted volumes on the server as root, so if my desktop
is compromised, you also get access to the files on the server.
Rgds
Denis McMahon
Wile that will stop you being done over again, it will screw up your
listing in the Search Engines (what's left of them) if they visit and
get a 404 they will drop you better to put a closed pages up and
redirect your site to it
Barrie,
--
The Association of Blind Piano Tuners | Be Environmentally Friendly
URL: http://www.uk-piano.org | To Your Neighbour.
| HAVE YOUR PIANO TUNED
If I knew how, I might try. In the meantime, I'm checking date stamps
every morning, as they are a sure sign of these hacks. I then overwrite
by uploading my backup, which is only around 500 HTML files, so it only
takes a short time.
--
David Lawson
This is correct. Once they had access to the PHP 4.4 application
servers they had access to all home directories via. a local root
exploit.
--
Kristian Van Der Vliet
Senior Sysadmin
Gradwell dot com
>Once is unfortunate, and I even had some sympathy for Gradwell.
Every web host gets hacked eventually. The measure of quality is how it's
handled, how prepared they are, and how they communicate.
--
Tony Evans
Saving trees and wasting electrons since 1993
blog -> http://perceptionistruth.com/
books -> http://www.bookthing.co.uk/
[ anything below this line wasn't written by me ]
>I shudder to think what will happen over Christmas. For myself, I do
>not intend still to be here then. I have been a faithful customer of
>Peter's since he had Clicknames; but enough is enough.
I hate to say the obvious but I have been recommending this for some
time.
--
Geoff Berrow (Put thecat out to email)
It's only Usenet, no one dies.
My opinions, not the committee's, mine.
Simple RFDs www.4theweb.co.uk/rfdmaker
And in what way is that comment helpful?
--
Tony van der Hoff | mailto:to...@vanderhoff.org
Buckinghamshire, England |
>On 19/12/10 23:24, Geoff Berrow wrote:
>> On Sun, 19 Dec 2010 10:16:18 +0000, Molly Mockford
>> <|enqu...@pagination.co.uk> wrote:
>>
>>> I shudder to think what will happen over Christmas. For myself, I do
>>> not intend still to be here then. I have been a faithful customer of
>>> Peter's since he had Clicknames; but enough is enough.
>>
>>
>> I hate to say the obvious but I have been recommending this for some
>> time.
>
>And in what way is that comment helpful?
I imagine it helps to make Geoff feel superior.
I must say that, having known him and got on well with him for years, I
had not expected him suddenly to spit bile in my direction. But I have
more important things to deal with than petty king-of-the-castle games.
--
Molly Mockford
I suspect it's not directed at the posters, but represents Geoff's
frustration at his treatment from Gradwell, and is more a general 'I've
been telling people to get away from Gradwell for some time'.
Still not that helpful - but to be frank, I've been trying to resist
saying the same thing since this thread started.
The thing that keeps me most quiet is the fact that all web hosts get
hacked eventually.
--
Tony Evans
Saving trees and wasting electrons since 1993
blog -> http://perceptionistruth.com/
books -> http://www.bookthing.co.uk
It was somewhat similar.
cheers
peter
>On 20/12/2010 09:37, Molly Mockford wrote:
>> On Mon, 20 Dec 2010, Tony van der Hoff <to...@nospam.vanderhoff.org> wrote:
>>
>>> On 19/12/10 23:24, Geoff Berrow wrote:
>>>> On Sun, 19 Dec 2010 10:16:18 +0000, Molly Mockford
>>>> <|enqu...@pagination.co.uk> wrote:
>>>>
>>>>> I shudder to think what will happen over Christmas. For myself, I do
>>>>> not intend still to be here then. I have been a faithful customer of
>>>>> Peter's since he had Clicknames; but enough is enough.
>>>>
>>>>
>>>> I hate to say the obvious but I have been recommending this for some
>>>> time.
>>>
>>> And in what way is that comment helpful?
>>
>> I imagine it helps to make Geoff feel superior.
>>
>> I must say that, having known him and got on well with him for years, I
>> had not expected him suddenly to spit bile in my direction. But I have
>> more important things to deal with than petty king-of-the-castle games.
>
>I suspect it's not directed at the posters, but represents Geoff's
>frustration at his treatment from Gradwell, and is more a general 'I've
>been telling people to get away from Gradwell for some time'.
Exactly, and I'm sorry you took it that way Molly. I thought you knew
me better than that.
My comment is meant more for Gradwell. I've been saying for some time
that their focus has been away from hosting. Ok now Peter says he is
still keen to carry on being a hosting company. I just want to let
him know that, from my point of view I feel undervalued and that if he
really does want to remain a hosting company he's got to try a whole
lot harder.
It's tempting to draw a parallel with current events at Heathrow.
--
John Hall
"I look upon it, that he who does not mind his belly,
will hardly mind anything else."
Dr Samuel Johnson (1709-84)
>Exactly, and I'm sorry you took it that way Molly.
Thank you for your apologies, Geoff, both here and in e-mail. 100%
accepted.
>I thought you knew me better than that.
So did I - which is why I was so surprised!
The problem is, Geoff, that whilst what you're saying is correct, in
essence it's no different to what many of people here have been saying
for the last year or two. You therefore came across as intolerably smug.
There is no evidence that <G> are taking these comments on board; it is
therefore pointless making them. In fact I recall Peter recently telling
us that the majority of customers think he's great!
The weight of the last straw is going to be different for everybody. I,
like Molly, have, for a variety of reasons, certainly been reluctant to
move, but I figure, for myself at least, the point of no return has now
been reached.
I now have to admit to struggling with setting up a secure mail server
on my VPS. Ah, well, I'll get there in the end...
Why? this is a hack on a server that they knew had happened to a group
of customers who they have the contact details for.
Heathrow is a group of hundreds of people working for many different
companies in an airport overloaded by snow where people don't know if it
is possible for planes to take off until the last minute,worked out on a
minute by minute basis.
Where is the parallel?
--
Woody
>In uk.net.providers.gradwell, "David" <da...@spam.com> wrote:
>
>>Once is unfortunate, and I even had some sympathy for Gradwell.
>
>Every web host gets hacked eventually. The measure of quality is how it's
>handled, how prepared they are, and how they communicate.
Indeed. Shit happens, how you deal with it is what counts. Once is
understandable, twice in a matter of days suggests deeper systemic
problems. I can think of better ways to spend a Sunday than the
way I had to - cleaning out garbage, trying to get Google to
remove the "attack site" flag and reassuring users and clients
that all will be well.
On the more general question of hosting, as so many others have
said, it now appears to be a very small part of Gradwell's
business and is now way down the list of priorities; gone are the
days when a support request gets sorted in a timely and fuss-free
manner. I will say in Peter's defence that any time in the past
I've had a problem and contacted him personally, things have been
sorted at lightning speed. But the reality seems to be that as
Gradwell has grown, his job has naturally become more complex,
he's had to take his eye off some of the balls, more staff have
been needed to do what he used to do and some of them just aren't
up to the job.
Example. I rarely contact support but the last time I did so I
went into minute detail about what the problem was and what I
needed to know in order to fix it. I got a one-line reply from
some eejit saying "What is it you're trying to do?" I haven't got
time to waste on wading through crap like that so I found another
solution/kludge.
I've stuck with Gradwell because when things work, they work well
enough for what I need and I don't really have the time to inflict
on myself all the hassle of migrating everything somewhere else.
But the hassle/benefit balance has tipped.
--
DG
> I've stuck with Gradwell because when things work, they work well
> enough for what I need and I don't really have the time to inflict
> on myself all the hassle of migrating everything somewhere else.
> But the hassle/benefit balance has tipped.
I think for many people as well, moving to a cPanel (or equivalent)
operation is a jolt at first, compared to the Gradwell tools. Sometimes
you don't have as much control as you're used to, or it doesn't appear
to be as flexible.
The risk is that once you've moved to a cPanel solution somewhere, it's
even easier to move around again later when you're used to the setup
(because they tend to be all the same).
The initial move is hard - but the more failures the Gradwell system
suffers, the more likely people will make that first jump, get over the
learning curve and then be more mobile in future.
--
Tony Evans
Saving trees and wasting electrons since 1993
blog -> http://perceptionistruth.com/
books -> http://www.bookthing.co.uk
cPanel is indeed a bit of a culture shock - but it isn't as bad as I
feared. It bewildered me that I had to set up all but the first site as
subdomains of the original but, when I told them I didn't like that way
of doing things, TSOhost support suggested that I use .htaccess (which
they supplied) to prevent people from accessing
http://domain2.domain1.dom, which means that they have to go the proper
route via http://www.domain2.dom - and they even added a line to
.htaccess to redirect http://domain2.dom to the www version, without
being asked. Quick, efficient, helpful and - above all - thoughtful.
Title: Injected code on hosted websites
Event: The post has been updated.
The following update was provided:
We are now running an adapted version of the clean up script across all
fileservers to scan for other files, ones with non-standard extensions,
that might have also been infected. We will update again when this has
completed.
In other words, they must have finished running the script for all the
standard .html and .php files.
So why are all the Webalizer .html files still infected? I've cleaned
up some of mine - those which I expect the sites' owners to access
regularly - but the others are still untouched.
And various sites are throwing up 403-Forbidden on every access, only
resolved by one or two refreshes. Presumably this is caused by the
permissions cock-up script? I just hope that when that one has finished
running, my PHP/MySQL sites will work again; at present, they simply
don't.
--
Molly Mockford
Yeh, I guess if I'd spent another few days trying to learn the intricacies,
I'd have been happy. But I'm at the point where getting another VPS,
installing Apache / PHP / mySQL, setting up the config and porting my sites
across was really easy. This was moving away from 1and1 not Gradwell this
time - but the reason it was easy is because when I moved away from
Gradwell that was the direction I went (VPS) rather than a cPanel host.
Which re-inforces my point, once you've done the migration to a particular
setup, it's easy to stay with that setup. TSOhost will work to keep you -
because moving away from them to another cPanel host is now quite easy for
you.
Moving away from Gradwell's custom tooling is a big step which requires a
lot of pressure to get over. I just hope they appreciate the amount of
pressure the current situation is generating.
I also found the TSOhost guys quick to respond and very helpful - just
their specific setup didn't suit me.
--
Tony Evans
Saving trees and wasting electrons since 1993
blog -> http://perceptionistruth.com/
books -> http://www.bookthing.co.uk/
> And various sites are throwing up 403-Forbidden on every access, only
> resolved by one or two refreshes. Presumably this is caused by the
> permissions cock-up script? I just hope that when that one has finished
> running, my PHP/MySQL sites will work again; at present, they simply
> don't.
The permissions were changed by the person exploiting the vulnerability, the
script is comparing recent backups' permissions with those now current and chmodding
where there is a discrepancy. I'm not sure if the problem you describe is
permissions related, unless you're only seeing successes if you have them cached,
it could be a similar but possibly related problem.
No, David, The files where the iframe code is duplicated are those which
got missed in the first cleanup. The second attack simply added the
second iframe.
> And various sites are throwing up 403-Forbidden on every access, only
> resolved by one or two refreshes. Presumably this is caused by the
> permissions cock-up script?
Yes, I am getting this now. I removed the exploit myself, as I didn't
want my customers having that problem (both times). the original
permissions problem took my site out as I could no longer use sessions
as I didn't have permissions to write to phptmp, so I made my own
session directory.
This new thing which I assume is a gradwell added problem started around
lunchtime today, so I shut the shop, I was going to close it tomorrow
anyway, it wasn't worth the hassle of customers contacting to ask why
every third click the site didn't work.
--
Woody
I hope not, because nothing has been touched on our site. There are
still 2,500 infected files in our directory - mostly with a .php
extension.
I assume that when the status updates mention running the scripts on
the four servers they are talking about clearwater, dixie, flathead,
and sawtooth. It would be nice to know which is 1,2,3 and 4. Our files
are on flathead, but I don't know it that server is supposed to have
been cleaned yet or not.
> And various sites are throwing up 403-Forbidden on every access, only
> resolved by one or two refreshes. Presumably this is caused by the
> permissions cock-up script? I just hope that when that one has finished
> running, my PHP/MySQL sites will work again; at present, they simply
> don't.
There was a problem at the weekend, in the period when our site was up
between the first infection and the second, when the permissions on
the PHP temp folder had been changed such that sites could no longer
create their session files in that folder. I had change our code to
use our own session folder to get our sites to function properly.
>The permissions were changed by the person exploiting the vulnerability,
That is the first I have heard of that. The Server Status e-mail
suggests that the permissions were changed by Gradwell:
Brief:
Due to a security issue we have had to amend some users file permissions
Impact:
Some websites may not be functioning as expected
Action:
Our server admin team are currently working on a full resolution
That *could* be read:
"...amend some users file permissions back to what they were before the
exploit"
We'll never know, will we? A full and open incident report is unlikely
to be issued, as it might have been before they reinvented themselves.
Just to confirm, the status was raised because we needed to recover the file
permissions as they were from a recent backup.
True. I just wish they would sort out this 403-Forbidden problem. On
one of my sites it's occurring so badly (not just the notice itself, but
images and stylesheets failing to load, presumably also on grounds of
permissions) that you have to refresh the browser four or five times to
get the page as it should be, and I've had to upload a special page
explaining about this and apologising to visitors. This sort of thing
is downright embarrassing.
David, are you in a position to explain why we keep getting these
spurious 403s on file after file? I really need an explanation to give
to some of my clients, who are up in arms about it all.
Well, my permissions were fine after the exploit, once I had fixed the
exploit itself. It was the following day the permissions went, so unless
there was a further attack that didn't put any code in but fiddled
around with the permissions, I would say this one was entirely gradwell.
--
Woody
Ah, OK, I stand corrected.
Hi,
I was able to reproduce this on my own site. It looks like at least one
of the nodes in the PHP 5.2 cluster was initially configured differently
when it was deployed (possibly), and could not handle some of the new enhancements
our sys-admin team have been working on to improve security with home directory
access from the web cluster.
The node has been dropped from the pool and I am no longer able to reproduce the
problem.
Many thanks, David - that's superbly quick service! And I now find that
my PHP logins, which were failing, are working again, so I guess the PHP
tmp permissions have been fixed as well.
Now it's just a matter of fingers crossed that nothing else shows up...
I was getting this morning not just on php but html as well but all
seem to be OK this evening also now can get back into admin areas of
the site
Barrie
--
The Association of Blind Piano Tuners | Be Environmentally Friendly
URL: http://www.uk-piano.org | To Your Neighbour.
Email: ab...@uk-piano.org | HAVE YOUR PIANO TUNED
There was a single PHP 5 server running in the cluster which we were
actively testing with a new version of MPM ITK which turned out not to
work fully with the changes we made to the home file server NFS
mounts.
The server has been taken out of production, so that shouldn't be
happening any more.
--
Kristian Van Der Vliet
Senior Sysadmin
Gradwell dot com
>>Wile that will stop you being done over again, it will screw up your
>>listing in the Search Engines (what's left of them) if they visit and
>>get a 404 they will drop you better to put a closed pages up and
>>redirect your site to it
>
>If I knew how, I might try.
I feel sure there must be a simpler way than the following (which Ive
used in the past) but my knowledge of Regular Expressions is limited.
Put this in a .htaccess file and it will redirect all your pages back to
your index page - where you should put your "site closed" message
Redirectmatch ^/[^i][^n][^d][^e][^x].*html http://[domain]/index.html
Be sure not to match end-of-line (i.e. dont put a $ after the *html) as
you probably want URLs with appended query strings to be redirected too.
--
David Gibson
Spam-cloaked message: The Reply-to: address *is* valid.
pagination now says "It Works! in <H1> size along with a JS script via
http://1.2.3.8/
Looked at yours as I wondered if moderation.org.uk was a one off but
both show the same...
... another exploit?
--
On the 12th Day of Christmas, the spammers sent to me: 12 Sexy websites,
11 Foreign charsets, 10 Body 'largements, 9 launder money, 8 Mortgage
offers, 7 Start a Business, 6 Bods-a-phishing, 5 Rolex, 4 Dating sites,
3 Hot tips, 2 Dodgy offers and a new set of rules for killfiles.
>pagination now says "It Works! in <H1> size along with a JS script via
>http://1.2.3.8/
>
>Looked at yours as I wondered if moderation.org.uk was a one off but
>both show the same...
>
>... another exploit?
Oh god, not again!!! I just had an e-mail from a client whose site was
showing the same. I re-uploaded the home page and all was OK, but I
couldn't see anything whatsoever in the files which led to the "It
works!" - the index.html hadn't been changed, and I saw no js file
anywhere.
(Mind you, I can't see the same exploit in either pagination.co.uk or
moderation.org.uk - I suppose it is skipping around testing its access,
and restoring files once it knows it can get in.)
I have just about had enough of this!!!
A glitch in the move to the new server, I suspect. I was seeing the
same on one of our sites but it seems to have settled down now.
I'm also getting a 500 Internal Server Error on one site, which I've
traced to cache control settings in the .htaccess file that worked on
the old server but not on the new one. Eg:
# 1 Year
<FilesMatch "favicon\.ico$">
Header set Cache-Control "public, max-age=31556926"
</FilesMatch>
But it isn't. The "it works!" is back again. SHIT.
I've spoken to Support and they admit that it was they who caused it.
They assure me it will not happen again. I am getting really seriously
fed up of grovelling to my clients with apologies for disruptions to
their sites which are totally beyond my control.
pagination showing up properly here now after a refresh but moderation
still with "It works!" even with refresh.
View source on pagination at the time (which I still had up in a browser
so could look at before trying refresh) contained as the sole contents
<html><script src="http://1.2.3.8/bmi-int-js/bmi.js"
language="javascript"></script><body><h1>It
works!</h1></body></html><script language="javascript"><!--
bmi_SafeAddOnload(bmi_load,"bmi_orig_img",0);//-->
</script>
--
Pedt
uk.announce ~ moderated group to announce news / events of specific interest to
a wider uk.* readership than the group(s) which their subjects would naturally
place them. See charter at <http://www.usenet.org.uk/uk.announce.html>
>pagination showing up properly here now after a refresh but moderation
>still with "It works!" even with refresh.
moderation works for me. What do you see for
<http://www.helmeandhallett.co.uk>? I hope it looks OK for you - I have
assured the client is it OK when I look at it.
God knows what will happen if Google have happened across any of these
sites while the kiddie-script was in operation!
>View source on pagination at the time (which I still had up in a
>browser so could look at before trying refresh) contained as the sole
>contents
>
><html><script src="http://1.2.3.8/bmi-int-js/bmi.js"
>language="javascript"></script><body><h1>It
>works!</h1></body></html><script language="javascript"><!--
>bmi_SafeAddOnload(bmi_load,"bmi_orig_img",0);//-->
></script>
The site above had simply:
<html><body><h1>It works!</h1></body></html>
I am very unhappy indeed about this - Gradwell messing around with our
sites' home pages without any prior notification.
> ><html><script src="http://1.2.3.8/bmi-int-js/bmi.js"
> >language="javascript"></script><body><h1>It
> >works!</h1></body></html><script language="javascript"><!--
> >bmi_SafeAddOnload(bmi_load,"bmi_orig_img",0);//-->
> ></script>
>
> The site above had simply:
>
> <html><body><h1>It works!</h1></body></html>
The javascript is being added by Pedt's ISP, not Gradwell, so you'd
not see it.
> I am very unhappy indeed about this - Gradwell messing around with our
> sites' home pages without any prior notification.
The "It Works!" message was coming from Apache, because it, well, it
was working! That is, it was working in the sense that it was running,
but it wasn't finding your pages. You're pages hadn't actually been
changed. I know that's somewhat hair-splitting, since anyone visiting
your sites would be served the 'It Works!' page for the index (and a
404 for other pages).
It appeared that debweb-1.web.thdo.gradwell.com was working OK on the
new cluster, but debweb-2.web.thdo.gradwell.com was failing to find
the page files. So sometimes your get the correct page and sometimes
you'd get "It Works". They both seem to be serving pages OK now.
>And I now find that my PHP logins, which were failing, are working
>again, so I guess the PHP tmp permissions have been fixed as well.
>
>Now it's just a matter of fingers crossed that nothing else shows up...
And of course it has, because since the sites were moved to the new
server, the PHP temp permissions are borked again, and my clients cannot
log in to password-protected areas of their sites.
This is really getting beyond a joke.
>The "It Works!" message was coming from Apache, because it, well, it
>was working! That is, it was working in the sense that it was running,
>but it wasn't finding your pages. You're pages hadn't actually been
>changed. I know that's somewhat hair-splitting, since anyone visiting
>your sites would be served the 'It Works!' page for the index (and a
>404 for other pages).
>
>It appeared that debweb-1.web.thdo.gradwell.com was working OK on the
>new cluster, but debweb-2.web.thdo.gradwell.com was failing to find
>the page files. So sometimes your get the correct page and sometimes
>you'd get "It Works". They both seem to be serving pages OK now.
Thanks, Susan.
It would have been nice if someone from Gradwell had bothered to make
that explanation, wouldn't it?
Note - I'm not addressing my response at Molly.
>> <html><script src="http://1.2.3.8/bmi-int-js/bmi.js"
>> language="javascript"></script><body><h1>It
>> works!</h1></body></html><script language="javascript"><!--
>> bmi_SafeAddOnload(bmi_load,"bmi_orig_img",0);//-->
>> </script>
The IP is in an APNIC range which is used to detect routing failures for
recently released ip blocks. I can't access the ip, so I guess that
Demon, or their upstream (but probably demon), need to update their filters.
I did try asking someone in aus to wget the file and email it to me, but
the ip is unreachable for him too.
> The site above had simply:
>
> <html><body><h1>It works!</h1></body></html>
This sounds like it could be the default page for an out of the box
server configuration. Maybe there is or was a misconfigured server in
the cluster that wasn't picking up the right configuration file(s).
Taking a wider view of the recent problems, and with the benefit of
hindsight.
It seems that the webserver clusters all use the same fileserver
backend, and the customer files for all the hosted websites are on this
server. This meant that by attacking a weakness in PHP 4, combined with
presumably a root exploit, the intruder was able to access files across
the whole fileserver, which explains how PHP 5 webserver files were
affected by an exploit in the PHP 4 webserver cluster.
If Gradwell had not continued to support PHP 4 then this particular
attack would probably not have occurred. Supporting PHP 4 was probably a
commercial decision taken in light of customer demand. With hindsight,
perhaps Gradwell could have been more proactive in removing PHP 4
support, but doubtless this would have alienated some customers.
If Gradwell had used separate fileserver clusters to back-end the PHP 4
and PHP 5 webserver clusters, only sites on the PHP 4 webserver clusters
would have been affected. However, using a shared fileserver has
technical benefits, not least of which would be the ease of migrating
sites from PHP 4 to PHP 5 environments. There was probably no reason to
think before the recent attacks that having a shared fileserver might be
such a security weakness.
It has appeared that Gradwell may have been a bit slow on the uptake as
to where and how the intrusion occurred, the fact it was repeated on a
second occasion, albeit intercepted at some point, is a pretty good
indicator that after the first intrusion, insufficient steps were taken
to prevent a recurrence. However, without much more information than I
have available, I can't call this one. If the first intrusion was as
successful as it appears to have been, the only clues apart from the
inserted code might have been gaps in the log files, and whilst gaps in
the log files might tell you when things happened, they don't tell you
what or how.
In the old days some event notifications went to the system console,
which was a hardcopy device, as well as or instead of log files. This
makes it a bit harder for intruders to cover their tracks. If the fact
that the attacker was able to edit or delete logs during the first
intrusion was an issue, perhaps Gradwell need to look at rsyslog. If the
logging system can't be located in the NOC, it should either be KVMd, or
modemed up from the serial port. Network login of any sort is not an
option if you're trying to protect the logs from being affected as part
of an intrusion!
There have clearly been communication issues, and I think that Gradwell
recognise that. I also think that the brevity of some communications has
been such that they may have been misinterpreted at times, and that has
exacerbated the disillusionment and despair of some customers.
A failure to communicate with customers what actions they were taking to
address and recover from the exploit certainly led to some confusion and
possibly even conflicts between the activities of Gradwell's own staff
and the actions that customers were taking to try and fix their own sites.
Gradwell's failings can be perceived by their customers' customers to be
the failings of their customers. This can impact significantly on
Gradwell's customers business. Gradwell needs to work with their
customers, and not in this forum, to try and repair the damage done.
All in all, I don't think this has been an unmitigated disaster for
Gradwell, but I think that there are some lessons from this that they
need to take forwards in several areas, and there's definitely some
fences to mend as well.
Rgds
Denis McMahon
Fine from here. Though at less than 3Kbps it was obviously a bit slow to
load!
(That figure is unfortunately correct, I'm on a mobile dongle and the
local mast fell off the local water tower last week - next nearest mast
is far enough away that connection is poor and rather slower than a
sloth on Mogadon.
I'm operating off a broadband mobile dongle[1] at the moment. A quick
shufti looking at the JS source code it seems to be related to being
able to select the quality of image. I've managed to grab the source
http://fairfieldtowers.net/bmi.txt which seems to confirm it.
[1] T-Mobile. Not the world's best at getting the default content lock
removed and it's rather sensitive - news.bbc.co.uk was blocked FFS.
Could you share the lines in .htaccess, please?
I haven't implemented this yet, but this is what they told me:
/Quote starts/
If you don't want the site to be accessible via the subdomain, just
create a .htaccess file in the document root for the addon domain with
the following content:
RewriteEngine On
RewriteCond %{HTTP_HOST} !^www.youraddon.com$
RewriteRule (.*) http://www.youraddondomain.com/$1 [R=301]
This will give you all the SEO benefit and also handle non-www to www
redirects.
/Quote ends/
--
Molly Mockford
I think the CP very simple to use got confused at first using the set
up wizard to use your browser as an S FTP but when I realised I
could use my own WSFTP I was happy chappy . The database wizard is
ace
The help movies are good as well. The CP may be different on a resell
account if that what your have
I will be keeping my own accounts on Gradwell as they will learn well
form this and should come out a better ISP We were moving the UK piano
page to a VDS as <g> do not do them anyway just got moved foreword.
Have a good one