Message from google-cloud-compliance+noreply@google.com

[Ken Task]

(apologize, in advance for the length of this)

Informing me that the google cloud platform/API I had established (two, one CentOS 7.1 and the other a Ubuntu LTS version) had been reported as being involved in scanning etc. of another network.  The warning message says I should take care of it within 3 days or the project (they provided the project ID) will be disabled.    They didn't mention which instance ... no IP no OS no nothing.

The CentOS instance and I had gone through and installed the lastest updates/fixes etc. for that OS and then setup ssh keys (as it was recommended as an alternative to what I'm guessing is the browser based method).   Only one user.  Had also setup a AMP stack with Moodle via git ... so latest most secure code there.    The Ubuntu instance, I later learned, was the instance (IP address shown in a full report is that servers activities) that had been hacked.

I've installed quite a few standalone servers with CentOS and Ubuntu LTS versions and know one can get the minimums done for security purposes then come back to finish the job at a later date.

Observations on the CentOS ... within 2 minutes of that server online, already pokes and probes on port 22 ... yes, to be expected.   Installed denyhost and logwatch and much to my surprise the majority of those pokes and probes where apparently coming from a competitior ... Amazon! AND one or two from the same IP class C block the server had been assigned to.   It was a little more than 'Noisy Neighbors' - more like "nosy".

Since I wasn't informed as to which instance (IP address) until later, I decided to wipe them both clean ... ie, remove the entire instances and begin again.   Afterwards, I do get a reponse which indicated it was only the Ubuntu instance.   Grrrrrr!

Since then, have begun again ... under the same project ID yet another CentOS 7.1 instance and made sure I locked it down ... even left SeLinux running and went the trouble of configuring that for a Moodle installation via git.   No Ubuntu this time.   And this time, didn't setup ssh keys, etc.. ... use the provided 'button' to access server via ssh.

 Yet, I rec. the same warning message about hourly ... today is the final day so am waiting to see if I've just waisted at least two days time with this project or not.

Has anyone had a like experience?

I, for one, will not recommend to others they do anything mission critical on the platform IF this is how such things are handled.

A few thoughts have come to me ... :
1. there should be a mandatory first run that acquires the latest updates for the OS.
2. we all know that the instances OS's don't launch all services and that OP is responsible for installs and config, but would think that the default firewall rules for ssh should be limited to the server itself first instead of 0.0.0.0/ALL
3. information about the other 10. IP addresses I see in the networking setup ... I assume those controled by Google itself and cannot be used as a 'back door' by other customers.
4. is these are based on usage, does the customer pay for the pokes and probes, etc. usage in defending vs those that would do harm to your systems?   Don't know how that would be 'metered' but could see Amazon or other such providers getting 'involved' on the sly to show folks that their servers/services are much better than Googles.  NOT saying they would do that 'officially' ... could be a devote/Amazon fan boy, etc. doing Amazon a 'favor' ... who knows!  (see, I trust no one these days! ;\)

Sorry of the rant ... hopefully someone (a human) will take the time to tell me where I went wrong and or how to prevent such events in the future (other than RTFM .. . I HAVE and know it quite well).

'spirit of sharing', Ken

[Jesse Scherer (Google Cloud Support)]

Hi Ken, there's quite a bit here but I hope I can get to everything.

On Friday, May 6, 2016 at 8:48:15 AM UTC-7, Ken Task wrote:

(apologize, in advance for the length of this)

Informing me that the google cloud platform/API I had established (two, one CentOS 7.1 and the other a Ubuntu LTS version) had been reported as being involved in scanning etc. of another network.  The warning message says I should take care of it within 3 days or the project (they provided the project ID) will be disabled.    They didn't mention which instance ... no IP no OS no nothing.

Ouch. I've only seen a couple notices like this and while they included IP addresses, they may have been using static IPs. Either way this sounds like a bug. I'll talk to a couple of my colleagues on the abuse team and see if they have ideas.

 

The CentOS instance and I had gone through and installed the lastest updates/fixes etc. for that OS and then setup ssh keys (as it was recommended as an alternative to what I'm guessing is the browser based method).   Only one user.  Had also setup a AMP stack with Moodle via git ... so latest most secure code there.    The Ubuntu instance, I later learned, was the instance (IP address shown in a full report is that servers activities) that had been hacked.

I've installed quite a few standalone servers with CentOS and Ubuntu LTS versions and know one can get the minimums done for security purposes then come back to finish the job at a later date.

Agreed. You mentioned what you'd installed on CentOS (which did not get compromised.) What, if anything, did you install on Ubuntu?

Observations on the CentOS ... within 2 minutes of that server online, already pokes and probes on port 22 ... yes, to be expected.   Installed denyhost and logwatch and much to my surprise the majority of those pokes and probes where apparently coming from a competitior ... Amazon! AND one or two from the same IP class C block the server had been assigned to.   It was a little more than 'Noisy Neighbors' - more like "nosy".

You're right; it's normal for a machine on a publicly routable IP (static or not) to get lots of drive-by traffic. I'd assume much of this traffic is up to no good; that somebody who is up to no good is using GCE or Amazon is not a surprise at all.

 

Since I wasn't informed as to which instance (IP address) until later, I decided to wipe them both clean ... ie, remove the entire instances and begin again.   Afterwards, I do get a reponse which indicated it was only the Ubuntu instance.   Grrrrrr!

Since then, have begun again ... under the same project ID yet another CentOS 7.1 instance and made sure I locked it down ... even left SeLinux running and went the trouble of configuring that for a Moodle installation via git.   No Ubuntu this time.   And this time, didn't setup ssh keys, etc.. ... use the provided 'button' to access server via ssh.

 Yet, I rec. the same warning message about hourly ... today is the final day so am waiting to see if I've just waisted at least two days time with this project or not.

Has anyone had a like experience?

This is not normal at all. Can you reply to me privately with your project ID?

 

I, for one, will not recommend to others they do anything mission critical on the platform IF this is how such things are handled.

A few thoughts have come to me ... :
1. there should be a mandatory first run that acquires the latest updates for the OS.

We generate new OS images on a pretty regular basis. Those images will include updated versions of all default software. I wonder: which images did you create your instances from?

 

2. we all know that the instances OS's don't launch all services and that OP is responsible for installs and config, but would think that the default firewall rules for ssh should be limited to the server itself first instead of 0.0.0.0/ALL

If you're talking about firewall rules burned into the install images, that would have the effect of preventing you from SSHing into new instances. For one-off instances created via the Cloud Console this shouldn't be a problem, but for users who create dozens of instances at once via API (using the gcloud command for example) this default would be a problem. Either way, you could certainly add a firewall rule to all of your instances at the time of creation, though in my case I'd probably still allow my home's IP as a fallback from browser-based SSH.

 

3. information about the other 10. IP addresses I see in the networking setup ... I assume those controled by Google itself and cannot be used as a 'back door' by other customers.

Correct. 10.0.0.0/8 is a private use IP range. On GCE every project will have its own private IP range. But this gives me an idea: are these two instances alone in this project?

 

4. is these are based on usage, does the customer pay for the pokes and probes, etc. usage in defending vs those that would do harm to your systems?   Don't know how that would be 'metered' but could see Amazon or other such providers getting 'involved' on the sly to show folks that their servers/services are much better than Googles.  NOT saying they would do that 'officially' ... could be a devote/Amazon fan boy, etc. doing Amazon a 'favor' ... who knows!  (see, I trust no one these days! ;\)

There is no charge for incoming traffic. I'm not sure what you're suggesting about other providers, but my assumption would be that these probes are from hackers and not part of a marketing campaign by another hosting provider. 

 

Sorry of the rant ... hopefully someone (a human) will take the time to tell me where I went wrong and or how to prevent such events in the future (other than RTFM .. . I HAVE and know it quite well).

No problem. I hope something I've brought up will be helpful (or spawn better ideas by others in the community).

Regards,

Jesse