Thursday, December 01, 2005

Splunk'ed

Via an email from SourceForge, I found out about splunk, a piece of software that indexes and searches log files (actually not only logs, but any "fast-moving IT data", as they put it). I downloaded the free version and installed it on a server I have, then indexed the /var/log/messages file and played with it a bit.

Here is the search results page for "Failed password". A thing to note is that every single word on the results page is clickable, and if you click on it a new search is done on that word. If you want to add the word to the current search words, click Ctrl and the word, or if you want to exclude the work from the search, click Ctrl-Alt and the word.




Pretty impressive. It uses various AJAX techniques to enhance the user experience, and best of all, part of the server software is written in Python! The search interface is based on Twisted:

root 504 1 0 11:26 pts/0 00:00:04 python /opt/splunk/lib/python2.4/site-packages/twisted/scripts/twistd.py --pidfile=/opt/splunk/var/run/splunk/splunkSearch.pid -noy /opt/splunk/lib/python2.4/site-packages/splunk/search/Search.tac

Definitely worth checking it out.

4 comments:

Anonymous said...

when/if you've reached the limits of splunk's scale, go buy SenSage.

Anonymous said...

we'll just have to wait and see

Anonymous said...

haven't hit any limits yet with 2.1b2, and splunk's search is years ahead of anything else.

Anonymous said...

hahah sensage... that's a laugh. splunk is light years beyond sensage (just ask any of the sensage employees that now work for splunk)

Modifying EC2 security groups via AWS Lambda functions

One task that comes up again and again is adding, removing or updating source CIDR blocks in various security groups in an EC2 infrastructur...