Broker¶
The Broker is the part of Comet which receives messages from authors and distributes them to subscribers. It can also subscribe to other brokers and act upon events received.
The Comet Execution Model¶
The Comet Broker runs as a single Python process which interfaces with the outside world in three different ways:
- As an “Event Receiver”, it accepts VOEvent submissions from authors;
- As an “Event Broadcaster”, it distributes VOEvents to subscribers;
- As an “Event Subscriber”, it listens to streams of events from upstream brokers.
Multiple instances of each of these major modes may be used simultaneously. That is, a single Comet instance can receive events from authors on multiple distinct interfaces (TCP ports, Unix domain sockets, etc); can accept subscription requests from subscribers on multiple interfaces; and can subscribe to multiple upstream brokers.
In addition, Comet can process the events it receives through a set of “handlers” which can perform arbitrary actions based on the event contents.
It’s worth noting that Comet maintains a single internal database of events, and handles every event received in the same way regardless of its source. That means that every event received, whether by direct submission from an author or by subscribing to an upstream broker, is processed through the same set of event handlers, and is rebroadcast to all subscribers (modulo their individual filtering settings). It is not possible to configure different event handlers or broadcast policies based on their source.
Running Comet¶
Background Information: Twisted Applications¶
The Comet broker is implemented as a Twisted application. This means that it is not invoked directly
on the command line, but rather using the twistd
tool provided as part
of Twisted.
twistd
provides a wide range of generic configuration options related to
daemonizing, logging, profiling, and so on. This fall outside the scope of
this documentation, but the interested reader is encouraged to familiarize
themselves with the contents of twistd(1)
.
It is worth noting that, by default, processes invoked with twistd
are
immediately daemonized. That is, twistd
is invoked, it returns control to
your shell prompt almost immediately, but Comet is now running in the
background. This is generally very convenient, but sometimes it’s useful to
have the application run in the foreground when testing and debugging: to
achieve this, invoke twistd -n
.
Invoking the Broker¶
Specify name of the Twisted application to run after twistd
and its
options. In this case, we run comet
, and pass it the --help
option to
provide a brief usage message:
usage: twistd [options] comet [-h] [--verbose] [--local-ivo LOCAL_IVO]
[--eventdb EVENTDB] [--receive [RECEIVE]]
[--receive-whitelist [RECEIVE_WHITELIST [RECEIVE_WHITELIST ...]]]
[--broadcast [BROADCAST]]
[--broadcast-test-interval BROADCAST_TEST_INTERVAL]
[--broadcast-whitelist [BROADCAST_WHITELIST [BROADCAST_WHITELIST ...]]]
[--subscribe SUBSCRIBE] [--filter FILTERS]
[--cmd HANDLERS] [--print-event] [--save-event]
[--save-event-directory SAVE_EVENT_DIRECTORY]
optional arguments:
-h, --help show this help message and exit
--verbose, -v Increase verbosity (may be specified more than once).
Standard arguments:
Global options affecting broker operation.
--local-ivo LOCAL_IVO
IVOA identifier for this system. Required if using
--receive or --broadcast.
--eventdb EVENTDB Event database root [default=/var/folders/g9/5c5t1myd5
4scz1kz39g718b40000gn/T].
Event Receiver:
Receive events submitted by remote authors.
--receive [RECEIVE] Add an endpoint for receiving events.
--receive-whitelist [RECEIVE_WHITELIST [RECEIVE_WHITELIST ...]]
Networks from which to accept event submissions
[default=accept from everywhere].
Event Broadcaster:
Broadcast events to remote subscribers.
--broadcast [BROADCAST]
Add an endpoint for broadcasting events.
--broadcast-test-interval BROADCAST_TEST_INTERVAL
Interval between test event broadcasts (seconds)
[default=3600].
--broadcast-whitelist [BROADCAST_WHITELIST [BROADCAST_WHITELIST ...]]
Networks from which to accept subscription requests
[default=accept from everywhere].
Event Subscriber:
Subscribe to event streams from remote brokers.
--subscribe SUBSCRIBE
Add a remote broker to which to subscribe.
--filter FILTERS XPath filter to be applied to events received from
remote brokers.
Event Processors:
Define 'event handlers' which are applied to all events processed by this
system.
--cmd HANDLERS External command to spawn when an event is received.
--print-event Enable the print-event plugin.
--save-event Enable the save-event plugin.
--save-event-directory SAVE_EVENT_DIRECTORY
Directory in which to save events
When asked to provide --help
, the application exits immediately after
printing this message: further configuration is required put Comet to work.
Environment Variables¶
-
COMET_PLUGINPATH
¶ By default, Comet will search the
comet/plugins
directory in its own source tree for plugins. This search path may be augmented by settingCOMET_PLUGINPATH
in the environment:COMET_PLUGINPATH=/path/to/plugins:/path/to/more/plugins twistd comet ...
Multiple paths may be specified, as shown above; use
;
to separate them on Windows, or:
on other platforms.
Configuration¶
All configuration of the Comet broker is performed by specifying command line options: Comet does not read a configuration file.
Note
At least one --receive
, --broadcast
, or --subscribe
option is
required to enable Comet’s functionality. If none are supplied, there is no
work to be done and Comet will exit immediately.
Global Options¶
Global options take effect whichever of the various modes (receiver, broadcaster, subscriber) are active.
Verbosity¶
The --verbose
or -v
flag increases the amount of information that
Comet writes to its log. It may be specified more than once; the effects are
cumulative.
Site Identification¶
Comet identifies itself to other systems by means of an International Virtual Observatory Identifier or IVOID: see the IVOA Identifiers Version 2.0 standard for details.
Note
The VOEvent standard uses the older term “IVORN” rather than IVOID; Comet prefers the more modern usage.
You should specify some appropriate IVOID for your site using the
--local-ivo
option. This is required when operating as a receiver or
broadcaster; it’s optional, but will be used if provided, when operating as a
subscriber. IVOIDs take the form ivo://${organization}/${name}
; for
example, ivo://org.transientskp/comet_broker
.
Event Database Location¶
In order to prevent looping on the network (ie, two brokers exchanging the
same event ad infinitum), a database of previously seen event is maintained.
By default, the database is written to a system-dependent default location,
but a specific directory on the filesystem may be specified by the
--eventdb
option. Events which are recorded in the database are not
forwarded by Comet. This is important: looping would degrade the quality of
the VOEvent network for all users! Note that events persist in the database
for 30 days, after which they are expired to save space.
Event Receiver¶
The event receiver is enabled by specifying the --receive
option.
--receive
optionally takes an argument which specifies the endpoint on
which to listen for events. This is specified as a Twisted server endpoint.
For example, possible arguments include:
tcp:8098
, to listen on TCP port 8098;unix:/some/file/name
to listen to the Unix domain socket at path/some/file/name
.
If an integer is provided, it is assumed to correspond to a TCP port on which to listen.
If no argument is provided to --receive
, the receiver will listen on TCP
port 8099 on all network interfaces.
--receive
may be specified multiple times, each corresponding to a
different endpoint; Comet will listen for event submissions on all of them
simultaneously.
When acting as an event receiver, Comet will only accept new events for
publication from hosts which have been specified as “whitelisted”. Hosts (or,
indeed, networks) may be included in the whitelist using the
--receive-whitelist
option. This option accepts either CIDR or
dot-decimal notation including a subnet mask. For example,
--receive-whitelist 127.0.0.1/32
and --receive-whitelist
127.0.0.1/255.255.255.255
both permit the local host to submit events to the
broker. Multiple networks may be specified, separated by spaces. To accept
submissions from any host, specify --receive-whitelist 0.0.0.0/0
; this is
the default if no --receive-whitelist
option is supplied.
Warning
The whitelist applies only to events received over the network; it will be ignored for connections using Unix domain sockets.
Event Broadcaster¶
The event broadcaster is enabled by specifying the --broadcast
option.
--broadcast
optionally takes an argument which specifies the endpoint on
which to listen for subscribers. This is specified as a Twisted server
endpoint. For example, possible arguments include:
tcp:8099
, to listen on TCP port 8099;unix:/some/file/name
to listen to the Unix domain socket at path/some/file/name
.
If an integer is provided, it is assumed to correspond to a TCP port on which to listen.
If no argument is provided to --broadcast
, the broadcaster will listen on
TCP port 8098 on all network interfaces.
--broadcast
may be specified multiple times, each corresponding to a
different endpoint; Comet will listen for event submissions on all of them
simultaneously.
When acting as an event receiver, Comet will only accept subscription requests
from hosts which have been specified as “whitelisted”. Hosts (or, indeed,
networks) may be included in the whitelist using the --broadcast-whitelist
option. This option accepts either CIDR or dot-decimal notation including a
subnet mask. For example, --broadcast-whitelist 127.0.0.1/32
and
--broadcast-whitelist 127.0.0.1/255.255.255.255
both permit the local host
to submit events to the broker. Multiple networks may be specified, separated
by spaces. To accept subscription requests from any host, specify
--broadcast-whitelist 0.0.0.0/0
; this is the default if no
--broadcast-whitelist
option is supplied.
Warning
The whitelist applies only to events received over the network; it will be ignored for connections using Unix domain sockets.
By default, Comet will broadcast a content-free test event to all subscribers
every hour to help with network debugging. The interval between test events
may be configured using the --broadcast-test-interval
option, which
accepts a value in seconds. Set it to 0
to disable the test broadcast
completely.
Event Subscriber¶
The event receiver is enabled by specifying the --subscribe
option.
--subscribe
requires an argument which specifies the remote broker to
which to connect. This is specified as a Twisted client endpoint. For
example, possible arguments include:
tcp:hostname:8099
, to make a subscription request over TCP on port 8099 to the broker with hostnamehostname
;unix:/some/file/name
to make a subscription request over the Unix domain socet at path/some/file/name
.
If the protocol specification is omitted, TCP is assumed; if the port is omitted, 8099 is assumed.
Optionally, the subscriber may request that the remote broker apply filters to
the event stream, limiting the events which it sends to the client. These
filters are specified with --filter
, in the form of XPath 1.0 expressions. The broker will evaluate the
expression against each event it processes, and only forward the event to the
subscriber if it produces a non-empty result. For more details see the section
on filtering.
Event Processors¶
The same set of event processors are applied to all events received by the broker, whether they come through direct submission by an author to the Event Receiver, or by broadcast from an upstream broker to the Event Subscriber.
Plugins¶
Custom code may be run to perform arbitrary local processing on an event when
it is received. For more details, see the section on event handlers. A plugin is enabled by giving its name as a command line option
(--plugin-name
). Plugins may also take arguments from the command line.
These are given in the form --plugin-name-argument=value
.
Comet ships with two plugins which both serve as examples of how to write
event handlers and which may be useful in their own right. The first simply
writes events to Comet’s log as they are received. This is the print-event
plugin: enable it by invoking Comet with the --print-event
option.
The second plugin shipped with Comet is save-event
, which writes events to
file. It is enabled with the --save-event
option. By default, events are
written to the default working directory (normally the directory in which you
invoked Comet): this may be customized using the --save-event-directory=
option. The filename under which an event is saved is based on its IVOID, but
modified to avoid characters which are awkard to work with on standard
filesystems.
Additional, user-defined, plugins may be added by placing them either in the
comet/plugins
directory, or in the location specified by
COMET_PLUGINPATH
.
Spawning External Commands¶
Similarly, received events may be sent to one or more external commands for
processing. These are specified using the --cmd
option. They should accept
the event on standard input and perform whatever processing is required before
exiting. The standard output and error from the external process will be
written to Comet’s log with level DEBUG
. If it returns a value other than
0
, it will be logged as a failure. Note that external commands are run in
a separate thread, so will not block the subscriber from processing new
events; however, the user is nevertheless responsible for ensuring that they
terminate in a timely fashion.