Results aggregation - ideas

[Tomasz Miklas]

Hi

This list was created possibly mainly to discuss how we can aggregate
our kippo results and have a high level view of what's going on with
our sensors... and sensors of our friends across the world.

1. Centralized approach:
One of the ideas proposed was to use central SQL database for logging
- which is supported by default, but then the question will be who
will hold the data? Remember that with the privilege ;-) come
responsibility - to keep it reliably up and on-line so we can log to
it, search through it and not loose the data if it would go down for
some reason. This is a bit of a burden in my opinion and is not very
'live' approach but anyway, we would need some kind of central DB to
check what's going on, etc.

2. Decentralized approach:
We all run our own databases and send copy of events to other users.
Ideal model for that would be pub-sub kind of things, so people can
subscribe to data feed from the group or maybe even from particular
sensor. That would require fault proof system if possible - to be
honest that's why the bad guys use things like IRC for simple botnets.
We can learn from that :)

We could have a simple script that feeds data over XMPP or even IRC if
needed to be simple... it could the send to several people/bots
(whitelist approach) or to a conference room (like on IRC) or both...
and all of us would have their own copy of all the logs and we could
monitor in real-time what's going on, etc... We could use Google Talk
as transfer medium (you can't beat their uptime hihi) but I'm not sure
if it has conferences as in typical XMPP/jabber server but there is
also IRC as possibility. My vibe is to use free public services and
have sensors/us as clients to it, rather than any of us becoming
service provider for something.

Downsides:
- need to write the script (send our logs, read others and feed to local DB)
- we depend on google talk or other similar service provider (I can
live with that)
- friend or foe - how do we tell them apart?

Upsides:
- less maintenance (infrastructure)
- not exposing central point/database - it doesn't exist
- ability to limit (whitelist/blacklist) who receives our log (by
configuring xmpp destiantions)
- can code something fancy later - say Ajax interface that would feed
off the live chatter stream and produce live dashboard
- development is decentralized, people can write and *share* their own
stuff - corelation/statistics tools that feed data to public, like
@netmenaces by @tliston
- ... I bet I can come up with some more :P

Now... with great power comes great responsibility, so how do we tell
who can connect to the feed?
We want data in real time but don't want the bad guys to be able to
listen to it that easily... otherwise they don't need to fingerprint
honeypot, just listen and alert if they see their own IP - I'm sure
that would get us Pwnie Award nomination in most epic FAIL category
:-> Channel passwords on IRC and sending messages via XMPP to selected
peers makes sense here, although it requires to build trust
relationship - so we trust that the person we send logs to is not the
bad guy.

Thoughts, comments, criticism... I'm all ears.

Tomasz

[Jacob]

When it come to the untrusted watching us we could have "private" site that members have access to, and we can have "public" site that publish reports, like patterns we see or whatever. I know this doesn't solve the whole problem. I mean who do we let see the private site? just my 2 cents.

- Jacob

--
You received this message because you are subscribed to the Google Groups "kippo users" group.
To post to this group, send email to kippo...@googlegroups.com.
To unsubscribe from this group, send email to kippousers+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/kippousers?hl=en.

[Muhammad Najmi Ahmad Zabidi]

On 8/12/10, Jacob <three...@gmail.com> wrote:
> When it come to the untrusted watching us we could have "private" site that
> members have access to, and we can have "public" site that publish reports,
> like patterns we see or whatever. I know this doesn't solve the whole
> problem. I mean who do we let see the private site? just my 2 cents.
>
> - Jacob

> <http://jacobkuehndorf.com>
> <ja...@jacobkuehndorf.com>

hi,

I believe that's how Shadowserver works right, but having two separate
report/views

>> kippousers+...@googlegroups.com<kippousers%2Bunsu...@googlegroups.com>

[Mig]

On Thu, Aug 12, 2010 at 6:39 AM, Tomasz Miklas <miklas...@gmail.com> wrote:

1. Centralized approach:

 This is a bit of a burden in my opinion and is not very
'live' approach but anyway, we would need some kind of central DB to
check what's going on, etc.

I agree - the potential lag /overhead in writing to a remote DB server puts me off that idea :)

2. Decentralized approach:

We could have a simple script that feeds data over XMPP or even IRC if
needed to be simple...

This is of course a more elegant idea. I haven't coded such a thing before but I do know that Dionaea, also a python-coded honeypot, features a method of sharing downloaded payloads via XMPP. The information is, I think, attempted to be anonymised as best as possible. Of course it is a public XMPP chat that anyone can join, but it's a system that seems to work well. If Markus is here on the kippo group, maybe he can share some info on how the XMPP logic works

Some links:

http://carnivore.it/2010/01/26/xmpp_-_basics

http://g3nto.blogspot.com/2010/04/dionaea-xmpp-function.html

http://carnivore.it/2010/05/15/xmpp_-_take_3

 

I believe I've also seen Markus suggest the XMPP idea to desaster in #honeypots IRC maybe a month or more ago.

I don't know if you could really 'privatise' the feature to some sort of moderated opt-in xmpp network successfully without some of the maintenance/overhead you've mentioned - I'd sooner take advantage of the feature and simply assume malevolent users/bots *are* watching the data.. same as I have to presume not all users in #honeypots on IRC are 'on our side' :)

That said, there'd be no harm in trying to keep them out.. maybe something in the kippo.cfg like a unique random string, some sort of handshaking upon joining the XMPP conference.. I don't know enough about the protocol to know whether something like that is possible.

Mig

[Kyle Creyts]

Sounds like a good idea, I actually think I have the capacity to host such a set of sites, and the private database for concentrating the data.

On Aug 11, 2010 7:22 PM, "Muhammad Najmi Ahmad Zabidi" <najmi....@gmail.com> wrote:

On 8/12/10, Jacob <three...@gmail.com> wrote:

> When it come to the untrusted watching us we could...

> <http://jacobkuehndorf.com>
> <ja...@jacobkuehndorf.com>


hi,

I believe that's how Shadowserver works right, but having two separate
report/views

>
> On Wed, Aug 11, 2010 at 3:39 PM, Tomasz Miklas
> <miklas...@gmail.com>wrote:
>
>> Hi
>>

>...

>> kippousers+...@googlegroups.com<kippousers%2Bunsu...@googlegroups.com>

>> .
>> For more options, visit this group at
>> http://groups.google.com/group/kippousers?hl=en.

>>...

--

You received this message because you are subscribed to the Google Groups "kippo users" group.

To po...

[Kyle Creyts]

Chris and I have elected to begin work on a script that will connect to the db and upload/parse logfiles to central point. We're willing to host it, etc.

--
Kyle Creyts

Information Assurance Professional

[Tomasz Miklas]

Kyle,

so will that be over xmpp or central db?

In my idea xmpp is just transport between peers and nothing more. Like
real conversation - you get the data if you listen to it.... xmpp
queues the messages as well :) so if your listener drops off, nothing
happens - messages will be delayed but will reach their destination.

Anyway, that was just _my_ idea ;-)

Tomasz

[Kyle Creyts]

We were talking about throwing it all in a central db, since an xmpp with a group could get pretty chatty pretty fast... also, central db has implications for statistical modeling, and availability of data, and could be leveraged for research

--

You received this message because you are subscribed to the Google Groups "kippo users" group.

To post to this group, send email to kippo...@googlegroups.com.

To unsubscribe from this group, send email to kippousers+...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/kippousers?hl=en.

[Kyle Creyts]

plus it would allow us to aggregate more data (potentially) from more users (potentially) and provide a framework for trending attacks and tracking unique or popular attacks.

[Muhammad Najmi Ahmad Zabidi]

If that's the case it's something like MWcollect Alliance does;

https://alliance.mwcollect.org/

where there's public statistics and special account views. The
privileged users are able to download the binaries using assigned keys
and through certain period.

There's article on Kojoney networks (hence Kojonet?) here in PDF;
http://packetstormsecurity.org/filedesc/HITB-Ezine-Issue-003.pdf.html

But not in detail, but we can see the pattern though

[Kyle Creyts]

a few issues we're wrestling with right now: authentication scheme for data submission, what data to track about (unique?) kippo implementations (since configuration could be seen to have an effect on attacker interaction), organization and planning of the db, what users are okay with having tracked, what data users are most interested in.

Any input we receive would be greatly appreciated, and expect many drastic changes in the first few releases we put out.

[Markus]

Hi,

> > 2. Decentralized approach:
>
> > We could have a simple script that feeds data over XMPP or even IRC if
> > needed to be simple...

IRC is very limiting, line length etc -> IRC is useless for sensors
network as it provides no value.

> This is of course a more elegant idea. I haven't coded such a thing before
> but I do know that Dionaea, also a python-coded honeypot, features a method
> of sharing downloaded payloads via XMPP. The information is, I think,
> attempted to be anonymised as best as possible. Of course it is a public
> XMPP chat that anyone can join, but it's a system that seems to work well.
> If Markus is here on the kippo group, maybe he can share some info on how
> the XMPP logic works
>

> I believe I've also seen Markus suggest the XMPP idea to desaster in
> #honeypots IRC maybe a month or more ago.

Yes, for me it is simply the best way to collect&share data, but there
was no finite answer if he was interested.

> I don't know if you could really 'privatise' the feature to some sort of
> moderated opt-in xmpp network successfully without some of the
> maintenance/overhead you've mentioned - I'd sooner take advantage of the
> feature and simply assume malevolent users/bots *are* watching the data..
> same as I have to presume not all users in #honeypots on IRC are 'on our
> side' :)

XMPP is like irc on xml-steroids.
I'm not a real fan of xml, but xmpp is really convincing.

dionaea uses xmpp to allow a distributed setup *without* a single
backend database.

All sensors are allowed join the network anonymously, they join 2
different Multi-User-Chat channels.
The first channel is anon-events, messages to anon events are
'informational things' like accepted connections, detected shellcode
profiles, download offers.
anon-files is used to send files there, base64 encoded to fit into the
xml scheme.

I modified the xmpp server software, (prosody), not to send messages
from 'visitors' to other visitors.
Default 'rank' on the anon-events channel of a sensor is 'visitor', so
sensors do not see the messages of other sensors, you need an account
with elevated privileges on the channel to get messages from the
sensors on the channel.
If a sensors receives a *new* file, he sends it to anon-files.
For anon-files, I decided to allow sensors to receive the files
gathered by other sensors, some kind of 'kickback' for contributing, I
just made the 'anonymous' user which is used by all sensors a
'participant' on the channel, so all sensors receive all other sensors
files on this channel.
If you want to setup your own xmpp server, you can grab the patch for
prosody here:
http://p.carnivore.it/antiLf

The 'anon' in the channel names indicates .. anonymously, therefore
dionaea replaces all information from the messages sent which would
contain the hosts address with 127.0.0.1, keeping the box anonymous.
This is optional, but enabled by default, if you have a trusted
network, you can run it with non-anonymous addresses.
And, you can connect dionaea to multiple xmpp networks and channels,
and define the messages which are to be sent to which channel on which
network in the configuration.
So you are free to join an anonymous network, for the file-sharing-
kickback, while contributing to one ore more trusted networks. These
trusted networks can even be on the same server.

dionaea ships with a backend for the xmpp service, which joins the
network and parses all messages, can write the messages to a database,
and the files gathered to disk.

The benefits of xmpp:
- decentral approach, no single database, everybody can run his own
database, more people have access to the data, more likely somebody
will make something useful of it - maybe a webinterface, whatever.
- no direct database connection for sensors - you do not want every
sensors to get full access to the db anyway
- single connection to the database (makes a difference)
- ssl encrypted
- authenticated
- realtime
- twisted can do xmpp anyway, so you get it for free

If you are looking for it, you even get the number of connected
clients for every channel.
The prosody xmpp server has no rate-limiting for clients, therefore it
is ideal for this scenario.
And you can define multiple 'muc' domains, I use
dionaea.sensors.carnivore.it for dionaea, if you want you can have
kippo.sensors.carnivore.it for free, and use it - even with the same
anonymous user.

For load, somebody connected a sensor with ... many addresses to the
network, not a problem, even though it was I think ~1000 files within
a single day.


MfG
Markus

I'm still waiting for somebody to ask for a full xmpp account so he
can create a screensaver of the world, with red dots in all locations
where attacks come from - in realtime.

[Nepenthes Development Team]

On Thu, Aug 12, 2010 at 6:27 PM, Markus <nepent...@gmail.com> wrote:
> The benefits of xmpp:
>  - decentral approach, no single database, everybody can run his own
> database, more people have access to the data, more likely somebody
> will make something useful of it - maybe a webinterface, whatever.
>  - no direct database connection for sensors - you do not want every
> sensors to get full access to the db anyway
>  - single connection to the database (makes a difference)
>  - ssl encrypted
>  - authenticated
>  - realtime
>  - twisted can do xmpp anyway, so you get it for free

Forgot to mention one thing: the data transported is xml too.
Events send by dionaea are stuffed into an xml container, so each
messages dionaea sends is xml, therefore it is easy to parse and deal
with, plattform/architecture independent.

Markus