plaso and Elasticsearch

[ese...@gmail.com]

Hi All,

I'm trying to get plaso to work successfully with elasticsearch/kibana.  I have plaso 1.0.2 installed with the ubuntu_12_04_dependency_pack-1.0.2 (running on ubuntu 12.04 LTS 64-bit).  I installed the elasticsearch 1.0.1 package and started the server, but when I list the output modules, Elastic isn't one of them.

psort.py -o list

******************************** Output Modules ********************************

    L2tcsv : The CSV format used by log2timeline, with 17 fixed fields.

  Mysql4n6 : Contains functions for outputing as 4n6time MySQL database.

   Dynamic : Dynamic selection of fields for a separated value output format.

     Rawpy : Prints out a "raw" interpretation of the EventObject.

       Raw : Prints out a "raw" interpretation of the EventObject protobuf.

    Sql4n6 : Saves the data in a SQLite database, used by the tool 4n6Time.

  Pstorage : Dumps event objects to a plaso storage file.

--------------------------------------------------------------------------------

Can anyone provide some advice or pointers to get me on the right track?

Thanks!

Erik

[Kristinn Gudjonsson]

forgot to reply back to the list, so I'll reply to everyone

Elasticsearch got added after the last release so you'll need to install from source ATM to get ES support.

Also you need to install the pyelasticsearch module.

However we need to test the new 1.0.1 installation of ES to see if we are still fully compatible with it (the last version I tested was 0.9.X)

--
You received this message because you are subscribed to the Google Groups "log2timeline-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to log2timeline-dis...@googlegroups.com.
To post to this group, send email to log2timeli...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Kveðja/Regards
Kristinn

[Johan Berggren]

I have tested Plaso with pyelasticsearch 0.6.1 and ElasticSearch 1.1.1 without any issues.

[E Herreid]

Thanks Johan, I just installed the latest packages and everything seems to be working great!

For any SIFT 3.0 users out there, I've included a bash script to update Plaso and install Elasticsearch/Kibana.  Be sure to run the script with sudo as a normal (non-UID0) user.  

Once the installation is complete, just use "psort.py -o elastic <dumpfile>" to add one or more data sets to your Elasticsearch instance, open your browser to the dashboard, and query to your heart's content!

Erik.

#!/bin/bash # # This will install latest Plaso engine with Elasticsearch and Kibana # Based on Kristinn Gudjonsson's blog article "Visualize the Output" # http://blog.kiddaland.net/2013/11/visualize-output.html # # This script removes the existing Plaso instance in SIFT 3.0 and installs # the latest development version along with Elasticsearch and Kibana. Run

# this script from a normal user account with the sudo command. # # To start the Kibana interface once everything is installed, open this page: # http://siftworkstation/index.html#/dashboard/file/plaso.json # # NOTE: THIS CONFIGURATION IS NOT SECURE! Do not use in highly sensitive # or mission critical environments without first securing all web services.

# Remove existing Plaso packages and fix any broken dependencies apt-get remove python-plaso -y apt-get install -f # Clean up packages before installing latest Plaso apt-get autoremove -y # Download and install all Plaso dependencies cd ~/Downloads wget https://googledrive.com/host/0B30H7z4S52FleW5vUHBnblJfcjg/1.1.0/preview1/ubuntu_12_04_dependency_pack-11_preview1.tgz tar -zxvf ubuntu_12_04_dependency_pack-11_preview1.tgz cd plaso_source_install sudo -u $SUDO_USER ./install_missing_packages.sh # Update and upgrade all packages apt-get update apt-get upgrade -y # Install necessary web and java packages apt-get install -y nginx openjdk-7-jre-headless # Download and install Elasticsearch cd ~/Downloads wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.1.deb dpkg -i elasticsearch-1.1.1.deb # Configure Elasticsearch to auto-start, then start the service update-rc.d elasticsearch defaults 95 10 /etc/init.d/elasticsearch start # Download and install python libraries to interact with Elasticsearch cd ~/Downloads git clone https://github.com/rhec/pyelasticsearch.git cd pyelasticsearch python setup.py build python setup.py install # Show that Elasticsearch is now an output option in Plaso psort.py -o list # Download and configure Kibana cd ~/Downloads wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz tar -zxvf kibana-3.1.0.tar.gz mv kibana-3.1.0 /usr/share/kibana3/ # Update Kibana configuration with new host name sed -i "s/\"+window.location.hostname+\"/siftworkstation/g" /usr/share/kibana3/config.js # Configure nginx to use Kibana cd ~/Downloads wget https://raw.github.com/elasticsearch/kibana/master/sample/nginx.conf mv nginx.conf /etc/nginx/sites-available/kibana cd /etc/nginx/sites-enabled rm default ln -s ../sites-available/kibana . # Update nginx configuration with new host name sed -i "s/kibana.myhost.org/siftworkstation/g" /etc/nginx/sites-available/kibana # Install json dashboard for kibana and restart nginx cd ~/Downloads wget https://plaso.googlecode.com/git/extra/plaso_kibana_example.json mv plaso_kibana_example.json /usr/share/kibana3/app/dashboards/plaso.json # Set the dashboard to use dark background (my personal preference) sed -i "s/\"light\"/\"dark\"/g" /usr/share/kibana3/app/dashboards/plaso.json service nginx restart # Reinstall Mantaray package removed due to Plaso dependency apt-get install mantaray -y # Clean up installation packages and directories rm ~/Downloads/ubuntu_12_04_dependency_pack-11_preview1.tgz rm ~/Downloads/elasticsearch-1.1.1.deb rm ~/Downloads/kibana-3.1.0.tar.gz rm -rf ~/Downloads/kibana-3.1.0 rm -rf ~/Downloads/plaso_source_install rm -rf ~/Downloads/pyelasticsearch

[esecrpm]

Thanks Kristinn!

On May 20, 2014, at 2:33 PM, Kristinn Gudjonsson <ki...@kiddaland.net> wrote:

Hi

Just a quick note about this update script of yours.... we are no longer maintaining that ubuntu dependency pack but instead we are distributing everything via the SIFT repo... so if you are using SIFT you just need to change from the "stable" branch over to the "dev" one and then do a:

sudo apt-get update

sudo apt-get upgrade

(probably need to do sudo apt-get install python-plaso)

That should automatically upgrade all the dependencies and plaso should be at trunk or at least very close to it. We will trickle these packages to stable once they reach a release status but the dev branch will keep the trunk or very close to trunk status.

The other part of the script, that is the one that deals with elasticsearch, nginx, etc... still stands. It is just that installing plaso and it's dependencies via apt-get is considerably easier and easier to maintain.

For more options, visit https://groups.google.com/d/optout.

--

Kveðja/Regards
Kristinn

[Kristinn Gudjonsson]

Hi

Just a quick note about this update script of yours.... we are no longer maintaining that ubuntu dependency pack but instead we are distributing everything via the SIFT repo... so if you are using SIFT you just need to change from the "stable" branch over to the "dev" one and then do a:

sudo apt-get update

sudo apt-get upgrade

(probably need to do sudo apt-get install python-plaso)

That should automatically upgrade all the dependencies and plaso should be at trunk or at least very close to it. We will trickle these packages to stable once they reach a release status but the dev branch will keep the trunk or very close to trunk status.

The other part of the script, that is the one that deals with elasticsearch, nginx, etc... still stands. It is just that installing plaso and it's dependencies via apt-get is considerably easier and easier to maintain.

On Tue, May 20, 2014 at 11:21 AM, E Herreid <ese...@gmail.com> wrote:

For more options, visit https://groups.google.com/d/optout.

--

Kveðja/Regards
Kristinn