While playing with Janus and using its features in creative ways can be quite fun, managing and monitoring a Janus instance often isn’t, and can be problematic. Applications may not be working as expected, PeerConnections may be broken for different reasons, or simply tracking what’s happening in a specific plugin can be very hard to figure out.
Luckily enough, Janus does provide some tools to help with that. Some months ago, for instance, we introduced the Admin API and how it can be used to inspect the internals of sessions and handles, in particular with respect to WebRTC connections from the Janus perspective. That said, while powerful and useful the Admin API is a poll-based protocol: this means that you have to query for information yourself, and if you want to keep up-to-date with what is happening, you have to do that on a regular basis. Things get even more daunting when you have tons of sessions and handles running in your application, as you may not be immediately aware of which session/handle corresponds to what, or which of them belong to the same scenario (e.g., all PeerConnections established in the context of the same VideoRoom).
This is where Event Handlers, a feature we merged recently, come to the rescue. Event Handlers are a new type of plugin in Janus, that take advantage of a new mechanism we implemented within Janus itself. More specifically, Janus and other plugins can now generate real-time events generated to several different aspects that may be happening during its lifetime: this events are then passed to all the available event handler plugins, plugins that can then decide to do with these events whatever they want. They might choose to store these events somehow, aggregate, dump or process them, format and send them to an external application, and so on. This really depends on what these events should be used for. At the time of writing, a single event handler plugin is available, one that we implemented ourselves as a proof of concept. This plugin does nothing more than formatting the event to a JSON object and sending it, via HTTP, to an external backend. While a simple operation, this already opens the doors for a much more complex management of these events, as it allows an external application to just implement a web backend in order to receive these events, to then handle them somehow.
Unsurprisingly enough, this is exactly what we’ll try to do in this blog post here, in what I hope you’ll see as a nice Christmas present from me! We’ll briefly describe what kind of events may be triggered by Janus, and how we can process them. Specifically, we’ll implement a simple node.js application (code available here) that receives events via HTTP from Janus and saves them to a database. We’ll then enable the generation of events in Janus, show how these events will be available after some sessions take place, and how we can just query the database itself for some useful information and ex-post processing. This should provide a decent overview on the potential event handlers provide, and give a good idea of how different logics could be applied to a similar usage of what’s currently available, e.g., to provide real-time monitoring and troubleshooting (for example, a web page that automatically displays new sessions, connections, etc., highlighting current troubles) rather than ex-post evaluations as the very dumb example that follows describes.
First of all, let’s start with the type of events that can be intercepted. We tried to have Janus originate events of several different types, in order to provide subscribers with all the information they might need. Specifically, Janus can generate events related to:
- session related events (e.g., session created/destroyed, etc.);
- handle related events (e.g., handle attached/detached, etc.);
- JSEP related events (e.g., got/sent offer/answer);
- WebRTC related events (e.g., PeerConnection up/down, ICE updates, DTLS updates, etc.);
- media related events (e.g., media started/stopped flowing, stats on packets/bytes, etc.);
- generic events originated by the Janus core (e.g., Janus started/stopped);
- events originated by plugins (content specific plugins themselves);
- events originated by transports (see above).
Each event also has a timestamp associated to it, so that we can know exactly when the event was generated in the first place from the server’s perspective, no matter when we received it. The generic format of events is the following:
{
"type" : <numeric event type identifier>,
"timestamp" : <time of when the event was generated>,
"session_id" : <unique session identifier>,
"handle_id" : <unique handle identifier, if provided/available>,
"event" : {
<event body, custom depending on event type>
}
}
How this translates to different messages for different event types is summarized in the examples provided in the sample event handler code. For instance, a new session being created might look like this:
{
"type": 1,
"timestamp": 1482416633784203,
"session_id": 2004798115,
"event": {
"name": "created"
}
}
while an event generated from a plugin might look something like this instead:
{
"type": 64,
"timestamp": 1482419547257987,
"session_id": 2004798115,
"handle_id": 3708519405,
"event": {
"plugin": "janus.plugin.echotest",
"data": {
"audio_active": "true",
"video_active": "true",
"bitrate": 0
}
}
}
For the sake of brevity we won’t explicitly list here all the kind of events you can intercept. For more insight, you can either refer to the above mentioned summary, or implement a simple listener of events and have a hands-on experience of what you might get to handle. The reference implementation we implemented also prints all incoming events before saving them to database, so you can use that as well in order to study the syntax of them all.
Since we want to save all these events in a database, we’ll need different tables able to host all of them in a structured way that doesn’t result in any loss of information. I have MariaDB installed on my Fedora, so here’s a quick and simple SQL example that shows how we might create a new database and several tables to store the different events:
CREATE DATABASE janusevents; USE janusevents; CREATE TABLE sessions (id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, session BIGINT(30) NOT NULL, event VARCHAR(30) NOT NULL, timestamp datetime NOT NULL); CREATE TABLE handles (id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, session BIGINT(30) NOT NULL, handle BIGINT(30) NOT NULL, event VARCHAR(30) NOT NULL, plugin VARCHAR(100) NOT NULL, timestamp datetime NOT NULL); CREATE TABLE core (id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, name VARCHAR(30) NOT NULL, value VARCHAR(30) NOT NULL, timestamp datetime NOT NULL); CREATE TABLE sdps (id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, session BIGINT(30) NOT NULL, handle BIGINT(30) NOT NULL, remote BOOLEAN NOT NULL, offer BOOLEAN NOT NULL, sdp VARCHAR(3000) NOT NULL, timestamp datetime NOT NULL); CREATE TABLE ice (id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, session BIGINT(30) NOT NULL, handle BIGINT(30) NOT NULL, stream INT NOT NULL, component INT NOT NULL, state VARCHAR(30) NOT NULL, timestamp datetime NOT NULL); CREATE TABLE selectedpairs (id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, session BIGINT(30) NOT NULL, handle BIGINT(30) NOT NULL, stream INT NOT NULL, component INT NOT NULL, selected VARCHAR(200) NOT NULL, timestamp datetime NOT NULL); CREATE TABLE dtls (id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, session BIGINT(30) NOT NULL, handle BIGINT(30) NOT NULL, stream INT NOT NULL, component INT NOT NULL, state VARCHAR(30) NOT NULL, timestamp datetime NOT NULL); CREATE TABLE connections (id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, session BIGINT(30) NOT NULL, handle BIGINT(30) NOT NULL, state VARCHAR(30) NOT NULL, timestamp datetime NOT NULL); CREATE TABLE media (id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, session BIGINT(30) NOT NULL, handle BIGINT(30) NOT NULL, medium VARCHAR(30) NOT NULL, receiving BOOLEAN NOT NULL, timestamp datetime NOT NULL); CREATE TABLE stats (id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, session BIGINT(30) NOT NULL, handle BIGINT(30) NOT NULL, medium VARCHAR(30) NOT NULL, base INT, lsr INT, lostlocal INT, lostremote INT, jitterlocal INT, jitterremote INT, packetssent INT, packetsrecv INT, bytessent BIGINT, bytesrecv BIGINT, nackssent INT, nacksrecv INT, timestamp datetime NOT NULL); CREATE TABLE plugins (id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, session BIGINT(30), handle BIGINT(30), plugin VARCHAR(100) NOT NULL, event VARCHAR(3000) NOT NULL, timestamp datetime NOT NULL); CREATE TABLE transports (id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, session BIGINT(30), handle BIGINT(30), plugin VARCHAR(100) NOT NULL, event VARCHAR(300) NOT NULL, timestamp datetime NOT NULL); GRANT ALL PRIVILEGES ON janusevents.* TO 'janusadmin'@'localhost' IDENTIFIED BY 'overlord';
This ugly example creates a new database called janusevents, a new user 'janusadmin'@'localhost' that can access it (which we’ll need in our application later on), and different tables (sessions, handles, stats, etc.) each devoted to the storage of a specific nature of event. Obviously, most if not all of the tables share some identifiers (typically session and handle IDs), which will allow us to relate them to each other.
Now that we have a working database, we can work on the application that will write to it. A quick and simple way to write such an application is using node.js for the purpose, so that’s what we’ll do as well in this example. Conventiently enough, there’s a mysql npm package we can use to access the database. With these assumptions, connecting to the database is actually very easy:
config.db = {
host: "localhost",
user: "janusadmin",
password: "overlord",
database: "janusevents"
}
connection = mysql.createConnection(config.db);
connection.connect();
We also need this application to implement a web server to receive the events from Janus, and a trivial way to do that is relying on the builting HTTP server for the purpose; in order to add authentication to it, if needed, we can use the basic-auth package for that:
config.http = {
port: 8080,
auth: {
username: "myuser",
password: "mypwd"
}
}
http.createServer(function (req, res) {
// Check credentials, read payload, handle event
[..]
// Write event to DB
var insert = { [..] };
var query = connection.query('INSERT INTO ' + table + ' SET ?', insert);
At this point, we have all the ingredients we need to store the incoming events. In fact, we have a working connection to the database, and when receiving an HTTP request from Janus we can prepare the right SQL command to save it to the table it belongs to. Again, we won’t delve into the details of all events that can be handled and how exactly this happens: anyway, as anticipated before you can find in this archive a complete example of the ugly prototype I implemented for this article, which means you can use that for testing and playing with events the same way I did here. To start the application, just install the dependencies and launch the backend:
npm install node events-db.js
Once we have the setup ready, all we need now is for Janus to start generating some events for us. By default, the generation of events is disabled, which means we’ll need to tweak some of the configuration files to enable it instead. First of all, we’ll need to enable the sample event handler, the one that as anticipated pushes all generated events via HTTP. This is done by editing the janus.eventhandler.sampleevh.cfg configuration file. There are several settings we can manipulate, namely which events should be forwarded, if it should be possible to group more or them into a single request, the backend to send events to, and so on. An example of a working configuration, considering the way we have set up the HTTP server in the node.js backend, is the following:
[general] enabled = yes events = all grouping = yes backend = http://localhost:8080 backend_user = myuser backend_pwd = mypwd
Once done with that, all that is left is enabling the generation of events within Janus itself. You can do this by either editing janus.cfg:
[events] broadcast = yes
or more simply launching Janus with the -e or --event-handlers command line argument:
/path/to/janus -e
which is the preferred approach if you don’t want to always enable event handlers in your setup, but just do that when experimenting with it.
As soon as you’ll start Janus, if everything’s working you’ll see something like this in the console:
Loading event handler plugin 'libjanus_sampleevh.so'... JANUS SampleEventHandler plugin initialized!
and something like this will instead appear in the logs of our node.js application:
Connected to DB: janusevents
Janus events DB backend started
{ type: 256,
timestamp: 1482422062092710,
event: { status: 'started' } }
This means an event of the core type (which is what type: 256 stands for) has already been generated by Janus, basically to tell us it just started. Our application should have written this to the database, so we can check if that did indeed happen by querying it directly:
MariaDB [janusevents]> select * from core; +----+--------+---------+---------------------+ | id | name | value | timestamp | +----+--------+---------+---------------------+ | 1 | status | started | 2016-12-22 16:54:22 | +----+--------+---------+---------------------+
If that happened, then good news! Our setup is working, and our application is indeed storing all these juicy pieces of information to the database for us. Anyway, while knowing when Janus started and stopped is useful, we’re more interested in getting events on a real session, so let’s see what happens when we launch the Echo Test demo.
You’ll immediately notice that a lot of events have started appearing: new session (the session the browser created with Janus), new handle (the connection to the Echo Test plugin), new negotiation (SDP offer and answer), ICE and DTLS state changes, live statistics on the transmission and a few events the Echo Test plugin itself generated. In my case, the session ID was 5994009527647740:
MariaDB [janusevents]> select * from sessions where event='created'; +----+------------------+---------+---------------------+ | id | session | event | timestamp | +----+------------------+---------+---------------------+ | 1 | 5994009527647740 | created | 2016-12-22 17:22:39 | +----+------------------+---------+---------------------+
which means I can check all the handles this session created:
MariaDB [janusevents]> select handle,plugin from handles where session=281912593775997 and event='attached'; +------------------+-----------------------+ | handle | plugin | +------------------+-----------------------+ | 5571389325143311 | janus.plugin.echotest | +------------------+-----------------------+
As expected, it’s only one (5571389325143311), and to the Echo Test plugin. By using the session and handle IDs as “keys”, we can get other details on what happened in this session, for instance what happened during the WebRTC PeerConnection establishment. For the sake of brevity, we’ll omit the SDPs that have been exchanged, as they’ll be quite large, and will focus on some other events instead:
MariaDB [janusevents]> select stream,component,state,timestamp from ice where session=5994009527647740 and handle=5571389325143311; +--------+-----------+------------+---------------------+ | stream | component | state | timestamp | +--------+-----------+------------+---------------------+ | 1 | 1 | connecting | 2016-12-22 17:22:39 | | 1 | 1 | ready | 2016-12-22 17:22:39 | +--------+-----------+------------+---------------------+ MariaDB [janusevents]> select selected,timestamp from selectedpairs where session=5994009527647740 and handle=5571389325143311; +-----------------------------------------------------------------+---------------------+ | selected | timestamp | +-----------------------------------------------------------------+---------------------+ | 192.168.1.70:58734 [host,udp] 192.168.1.69:36284 [host,udp] | 2016-12-22 17:22:39 | +-----------------------------------------------------------------+---------------------+ MariaDB [janusevents]> select state,timestamp from dtls where session=5994009527647740 and handle=5571389325143311; +-----------+---------------------+ | state | timestamp | +-----------+---------------------+ | trying | 2016-12-22 17:22:39 | | connected | 2016-12-22 17:22:39 | +-----------+---------------------+ MariaDB [janusevents]> select state,timestamp from connections where session=5994009527647740 and handle=5571389325143311; +----------+---------------------+ | state | timestamp | +----------+---------------------+ | webrtcup | 2016-12-22 17:22:39 | +----------+---------------------+ MariaDB [janusevents]> select medium,receiving,timestamp from media where session=5994009527647740 and handle=5571389325143311; +--------+-----------+---------------------+ | medium | receiving | timestamp | +--------+-----------+---------------------+ | audio | 1 | 2016-12-22 17:22:39 | | video | 1 | 2016-12-22 17:22:39 | +--------+-----------+---------------------+
all events that give us a good idea of what lead to a working PeerConnection in the first place. Just as we described in the Admin API guidelines, this is where you’d look if there were any troubles in a PeerConnection.
Since the plugin sent some events too, we can inspect them as well:
MariaDB [janusevents]> select event,timestamp from plugins where session=5994009527647740 and handle=5571389325143311;
+------------------------------------------------------------+---------------------+
| event | timestamp |
+------------------------------------------------------------+---------------------+
| {"audio_active":true,"video_active":true,"bitrate":0} | 2016-12-22 17:22:39 |
| {"audio_active":true,"video_active":true,"bitrate":0} | 2016-12-22 17:22:39 |
| {"audio_active":true,"video_active":true,"bitrate":128000} | 2016-12-22 17:22:46 |
+------------------------------------------------------------+---------------------+
just to see at a certain point the user hit the “bandwidth” knob to force 128kbps as a video bitrate.
Of even greater interest are the statistics, as every second or so Janus generates events related to the transmission on a PeerConnection, with respect to both outgoing and incoming data. In my case, there were about 200 events related to statistics (which means about 100 events for audio and video respectively), as I was in the demo for about a couple of minutes:
MariaDB [janusevents]> select count(*) from stats where session=5994009527647740 and handle=5571389325143311; +----------+ | count(*) | +----------+ | 210 | +----------+
Inspecting the statistics here to see what you can evince out of them can be a useful exercise, so I encourage you to do that!
Another useful exercise might be adding the context of a specific plugin or application scenario to the database, and extract more information this way. For instance, it might be useful to have an additional table that stores associations between a handle and a specific VideoRoom room: in fact, the simple tables we created here don’t allow for that (at least not in an easy way), as all plugin events are stored as plain JSON objects, which means we can’t search/sort on their content (in this case, any reference to a specific room) in the database directly. Making the database and the application more plugin aware would open the doors to such a processing as well.
Turning all this into a live application, or in something that notifies you right away when something’s wrong, would be even more fun, but that’s for another time, or another blog post 🙂
That’s all… I hope this brief introduction will make the concept of Event Handlers a bit clearer, and that it will encourage more and more people to start playing with them. Whether it’s to expand the very simple approach described here, using the sample event plugin in a more effective way, or writing an entirely new event plugin, let us know about your efforts on this, and help us improve the mechanism we have in place along the way!
