Riemann is a great application for dealing with event processing but it doesn’t have a lot of documentation or newbie friendly tutorials. There are some cool pictures that explain the principles of the app but nothing beyond that. At some point I want to try and contribute some better documentation to the official project but in the meantime here’s a few points that I think are useful for getting started.
I’m assuming that you’ve followed these instructions to get a working Riemann installation and you’ve followed the instructions on how to submit events to Riemann via the Ruby Riemann client interface.
At this point you want to start making your own processing rules and it is not clear how to start.
Well the starting point is the idea of streams when an event arrives in Riemann it is passed to each stream that has been registered with what is called the core. Essentially a stream is a function that takes an event and some child streams and these functions are stored in a list in the core atom under the symbol :streams.
Okay let’s look at an example. The first obvious thing you want to do is print out the events that you are sending to Riemann. If you’ve got the standard download open the etc/riemann.config file, set the syntax for the file to be Clojure, as this is read into Clojure environment in the riemann/config namespace and you can use full Clojure syntax in it. In the config file add the following at the end. Now either run the server or if it is running reload the config file with kill -HUP <Riemann PID>.
(streams prn)
prn is a built-in function that will print an event and pass it on to following streams.
In irb let’s issue an event:
r << {host: "rrees.me", service: "posts", metric: 5}
You should see some output in the Riemann log along the following lines.
#riemann.codec.Event{:host "rrees.me", :service "posts", :state nil, :description nil, :metric 5, :tags nil, :time 1366450306, :ttl nil}
I’m going to assume this has worked for you. So now let’s see how events get passed on further down the processing chain. If we change our streams function to the following and reload it.
(streams prn prn)
Now we send the event it should get printed twice! Simples!
Okay now let’s look at how you can have multiple processing streams working off the same event. If we add a second print stream we should get three prints of the event.
(streams prn prn) (streams prn)
Each stream that is registered can effectively process the event in parallel so some streams can process an event and send it to another system while another can write it to the index.
Let’s change one of our prints slightly so we can see this happen.
(streams (with :state "normal" prn) prn) (streams prn)
We should now get three prints of the event and in one we should see that the event has the state of “normal”. Okay great! Let’s break this down a bit.
Every parameter of streams is a stream and a stream can take an event and child streams. So when an event occurs it is passed to each stream, each stream might specify more streams that the transformed event should be passed to. That’s why we pass prn as the final parameter of the with statement. We’re saying add the key-value pair to the event and pass the modified event to the prn stream.
Let’s try implementing this by ourselves, there is a bit of magic left here, call-rescue is an in-built function that will send our event to other streams you can think of it as a variant of map:
(defn change-event [& children]
(fn [event]
(let [transformed-event (assoc event :hello :world)]
(call-rescue transformed-event children))))
(streams (change-event prn))
If this works then we should see an event printed out that has the “hello world” key-value pair in it. change-event is a stream handler that takes a list of “children” streams and returns a function that handles an event. If the function does not pass the event onto the children streams then the event stream stops processing, which is a bit like a filter. The event is really just a map of data like all good Clojure.
At this point you actual have a good handle on how to construct your own streams. Everything else is going to be a variation on this pattern of creating a stream function that returns an event handler. The next thing to do now is go and have a look at the source code for things like with, prn and call-rescue. Peeking behind the curtain will take a certain amount Clojure experience but it really won’t be too painful, I promise, the code is reasonable and magic is minimal. Most of the functions are entirely self-contained with no magic so everything you need to know is in the function code itself.
