Broker

The Broker is the part of Comet which receives messages from authors and distributes them to subscribers. It can also subscribe to other brokers and act upon events received.

Twisted Applications

The Comet broker is implemented as a Twisted application. This means that it is not invoked directly on the command line, but rather using the twistd tool provided as part of Twisted.

twistd provides a wide range of generic configuration options related to daemonizing, logging, profiling, and so on. This fall outside the scope of this documentation, but the interested reader is encouraged to familiarize themselves with the contents of twistd(1).

It is worth noting that, by default, processes invoked with twistd are immediately daemonized. That is, twistd is invoked, it returns control to your shell prompt almost immediately, but Comet is now running in the background. This is generally very convenient, but sometimes it’s useful to have the application run in the foreground when testing and debugging: to achieve this, invoke twistd -n.

Invoking Comet

Specify name of the Twisted application to run after twistd and its options. In this case, we run comet, and pass it the --help option to provide a brief usage message:

$ twistd comet --help
Usage: twistd [options] comet [options]
Options:
  -r, --receive                   Listen for TCP connections from authors.
  -b, --broadcast                 Re-broadcast VOEvents received.
  -v, --verbose                   Increase verbosity.
  -q, --quiet                     Decrease verbosity.
      --print-event               Enable the print-event plugin.
      --save-event                Enable the save-event plugin.
      --local-ivo=                IVOA identifier for this system (required).
      --eventdb=                  Event database root. [default: /tmp]
      --receive-port=             TCP port for receiving events. [default: 8098]
      --broadcast-port=           TCP port for broadcasting events. [default:
                                  8099]
      --broadcast-test-interval=  Interval between test event brodcasts (in
                                  seconds; 0 to disable). [default: 3600]
      --whitelist=                Network to be included in submission
                                  whitelist. [default: 0.0.0.0/0]
      --remote=                   Remote broadcaster to subscribe to
                                  (host[:port]).
      --filter=                   XPath filter applied to events broadcast by
                                  remote.
      --cmd=                      Spawn external command on event receipt.
      --save-event-directory=     Target directory [default:
                                  current working directory]
      --help                      Display this help and exit.
      --version                   Display Twisted version and exit.

Basic Modes of Operation

If the --receive option is supplied, Comet will fulfil the Broker role in an Author to Broker connection. In other words, it will listen for TCP connections from remote authors and accept events for distribution. The TCP port on which Comet will listen may be specified with the --receive-port option.

If the --broadcast option is supplied, Comet will listen for Subscribers to connect and then it will fulfil the Broker role in a Broker to Subscriber connection with each of the Subscribers. Any VOEvents received (either by direct connection from authors, or by subscribing to remote brokers) are rebroadcast to subscribers. The TCP port on which Comet will allow subscribers to connect may be specified with the --broadcast-port option.

If one or more --remote options are supplied, Comet will subscribe to the remote host specified and fulfil the Subscriber role in the resulting Broker to Subscriber connection. If just given a hostname Comet will attempt to subscribe on port 8099. Optionally, a different port may be specified by appending it to the hostname, separated by a :.

A single Comet daemon will accept any combination of --receiver, --broadcast and one or more --remote options and play all of the specified roles simultaneously. If none of --receiver, --broadcast or --remote are supplied, there is no work to be done and Comet will exit immediately.

Identification

Whatever the mode of operation, Comet identifies itself by means of an International Virtual Observatory Resource Name or IVORN: see the VOEvent standard for details. You should specify some appropriate IVORN for your site using the --local-ivo option. This should be of the format ivo://${organization}/${name}; for example, ivo://org.transientskp/comet_broker.

VOEvent Network Maintenance

In order to prevent looping on the network (ie, two brokers exchanging the same event ad infinitum), a database of previously seen event is maintained. This database is written to the filesystem in the location specified by the --eventdb option. Events which are recorded in the database are not forwarded by Comet. This is important: looping would degrade the quality of the VOEvent network for all users! Note that events persist in the database for 30 days, after which they are expired to save space.

Receiver Options

When acting as a receiving broker (with --receive), Comet will only accept new events for publication from hosts which have been specified as “whitelisted”. Hosts (or, indeed, networks) may be included in the whitelist using the --whitelist option. This option accepts either CIDR or dot-decimal notation including a subnet mask. For example, --whitelist 127.0.0.1/32 and --whitelist 127.0.0.1/255.255.255.255 both permit the local host to submit events to the broker. This option may be specified multiple times and the results are cumulative. To accept submissions from any host, specify --whitelist 0.0.0.0/0; this is the default if no --whitelist option is supplied.

Broadcaster Options

By default, Comet will broadcast a content-free test event to all subscribers every hour. The aim is to help with network debugging. The interval between test events may be configured using the --broadcast-test-interval option, which accepts a value in seconds. Set it to 0 to disable the test broadcast completely.

Subscriber Options

When subscribing to a remote broker (with --remote), one or more filters may be specified which limit the events which will be received. These filters are specified with --filter, in the form of XPath 1.0 expressions. The broker will evaluate the expression against each event it processes, and only forward the event to the subscriber if it produces a non-empty result. For more details see the section on filtering.

Common Options

Plugins

Custom code may be run to perform arbitrary local processing on an event when it is received. For more details, see the section on event handlers. Plugin actions will be taken whether Comet receives an event from an author (--receive) or an upstream broker (--remote). A plugin is enabled by giving its name as a command line option (--plugin-name). Plugins may also take arguments from the command line. These are given in the form --plugin-name-argument=value.

Comet ships with two plugins which both serve as examples of how to write event handlers and which may be useful in their own right. The first simply writes events to Comet’s log as they are received. This is the print-event plugin: enable it by invoking Comet with the --print-event option.

The second plugin shipped with Comet is save-event, which writes events to file. It is enabled with the --save-event option. By default, events are written to the default working directory (normally the directory in which you invoked Comet): this may be customized using the --save-event-directory= option. The filename under which an event is saved is based on its IVORN, but modified to avoid characters which are awkard to work with on standard filesystems.

Spawning External Commands

Similarly, received events may be sent to one or more external commands for processing. These are specified using the --cmd option. They should accept the event on standard input and perform whatever processing is required before exiting. The standard output and error from the external process is ignored. If it returns a value other than 0, it will be logged as a failure. Note that external commands are run in a separate thread, so will not block the subscriber from processing new events; however, the user is nevertheless responsible for ensuring that they terminate in a timely fashion.

Logging

The amount of information Comet writes to its log may be adjusted using the --verbose and --quiet options.