gba@20120928 This article is a DRAFT.
This article discusses a method of synchronizing Splunk Search Peers using lsyncd.
As a it's installation scales out, different components of a Splunk can be distributed to separate systems across a network. For example, the Search Head and Indexer components of a Splunk Installation can be configured on two different servers. In this setup the Indexer component is known as a 'Search Peer' of the Search Head.
Out of the box Splunk supports two different methods of synchronizing the configuration between a Search Head and its Search Peers: Bundle Replication and Mounted Bundles. Each of these methods has its advantages, but for Splunk Storm we needed a method that scaled in a slightly different way, so we developed a method that uses lsyncd for synchronizing configuration.
lsyncd is a Lua-based daemon that monitors inotify Linux kernel filesystem changes. Based on the changes it's been configured to watch, lsyncd, via rsync, can synchronize the deltas (changes) of files. We've utilized this to replace the existing behavior of bundle replication between Search Peers.
Assuming you've already configured Distributed Search:
Disable Bundle Replication on the Search Head, where SEARCH_PEER is the IP or Hostname of the Search Peer from your distributed search setup:
# $SPLUNK_HOME/etc/system/local/distsearch.conf
[distributedSearch]
shareBundles = false
servers = SEARCH_PEER:8089
Configure lsyncd to monitor the $SPLUNK_HOME/etc/ directory for changes, where SEARCH_PEER is the IP or Hostname of the Search peer from your distributed search setup:
# /etc/lsyncd.conf.lua
sync{
rsyncssh,
source='/opt/splunk/etc',
host='spsync@SEARCH_PEER',
targetdir='/opt/searchpeer'
}
Enable Mounted Bundles on the Search Peer, where SEARCH_HEAD is the IP or Hostname of the Search Head from your distributed search steup:
# $SPLUNK_HOME/etc/system/local/distsearch.conf
[searchhead:SEARCH_HEAD]
mounted_bundles=true
bundles_location=/opt/searchpeer