QuakeWatch Information Distribution System (QWIDS)

Table of Contents

  1. Introduction
  2. Operational Overview
  3. Software Requirements
  4. Quick Start Test Setup
  5. API
  6. Example Transient Client
  7. Example Persistent Client
  8. Open Source
  9. Operating Systems
  10. Architecture
  11. Performance and Scalability
  12. Communications Standards
  13. Message Format Neutral
  14. Message Size Limitation
  15. Security
  16. Scalability Discussion
  17. Redundancy and Robustness
  18. Data Provider Description
  19. Data Server Description
  20. User Agreement and Copyright

Introduction

The QuakeWatch Information Distribution System (QWIDS) implementation of the Earthquake Information Distribution System (EIDS) provides end-to-end support for reliable XML-based messaging.  Its features include multi-platform support, a high level of configurability, persistent archiving of messages, and missed-message tracking/recovery.  The QuakeWatch modules were developed as a next-generation earthquake notification system to alert users, in near real-time, of seismic data and vital earthquake-hazards information following significant events.  The modules have been enhanced and expanded to implement the EIDS specification put forth by the USGS.  QWIDS was developed and is maintained by ISTI.

QWIDS is written entirely in Java, allowing it to run on all major operating systems and hardware.  The messaging system uses a leading open-source version of the industry-standard Common Object Request Broker Architecture (CORBA).

This document provides an summary of how the components of QWIDS may be used to provide a reliable messaging system for Earthquake Information Distribution.  See also the "QWIDS Installation" document.

The component roles of QWIDS are described briefly below, but each has its own configuration and further documentation.
  1. Data Provider - QWServer with QWViaFileFeeder or QWCorbaFeeder plug-in - is a component that is placed at a generator of earthquake data products (e.g., a seismic network).
  2. Persistent Data Server - QWServer with QWRelayFeeder plug-in - is a component that provides a dedicated data service to end-users with a pre-defined quality of service. This component resides downstream from the Data Provider and is located at a service provider's site.
  3. Transient Data Server - QWWebServices Server - is a component that provides a non-dedicated data service to clients via Web Services transactions. This component also resides downstream from the Data Provider and is located at a service provider's site.
  4. Persistent Data Recipient - QWClient client - an end-user client that maintains a persistent connection with a Persistent Data Server and reliably receives messages.  See the sample Console Output, Cube Output, File Output and JNI Client Receiver clients for examples.  The CISN Display is also an example of a persistent client.
  5. Transient Data Recipient - QWWebServices client - an end-user client that polls a Transient Data Server using Web Services and reliably receives messages.  See the sample Web Services client for an example.
The following directories are included in the QWIDS distribution:

QWServer:  QuakeWatch Server distribution (doc).
QWWebServices:  QuakeWatch Web Services distribution (doc).
QWConOutClient:  QuakeWatch Console Output Client distribution (doc).
QWGenericClient:  QuakeWatch Generic Client distribution (doc).
QWCubeOutClient:  QuakeWatch Cube Output Client distribution (doc).
QWFileOutClient:  QuakeWatch File Output Client distribution (doc).
QWJNIClient:  QuakeWatch JNI Client Receiver distribution (doc).
QWMsgValidateClient:  QuakeWatch Message Validating Client distribution (doc).
QWLibs:  QuakeWatch Library distribution (doc).
QWWebSvcsManager:  QuakeWatch Web Services Manager distribution (doc).
QWStatusProcessor:  QuakeWatch Status Processor distribution (doc).
QWLogin:  PHP-based client-login provisioning system (doc).
ANSS_EQ_XML:  Files documenting the ANSS-EQ-XML message format (doc).
InstallationSamples:  Example configuration files for test installations.
images:  Image files for documentation.

Each distribution includes a ".zip" archive containing source code.

Note:  As of QWIDS distribution version 1.4, all QuakeWatch client programs include support for connecting to a QuakeWatch Web Services Server.  QuakeWatch client programs may gain web-services server support by having the "QWWebSvcsManager.jar" file copied into their installation directory and setting the 'webServicesServerFlag' configuration-file parameter to 'true'.  This file may be found in the QuakeWatch Web Services Manager ("QWWebSvcsManager") distribution directory.

Return to Table of Contents

Operational Overview

The QWIDS product is comprised of 3 elements: Data Provider (or Data Publisher), Data Server, and receiver client APIs (plus existing clients). Each element consists of various components of the QWIDS product interconnected to perform a specific role.  The QuakeWatch Server (QWServer) is used in a Data Provider role as well as a Data Server role. It provides feeder plug-in support for data input from files (like the existing QDDS), directly from QDDS servers, from CORBA methods, or from other QWServers (via the 'QWRelayFeeder' plug-in). The QWServer can act as both a server and a client simultaneously, depending on which plug-in modules it is configured with.

The QWServer in the Data Provider role is a server of messages to the Data Server element. The QWServer in the Data Server role is a client of the Data Provider QWServer and a server to downstream recipients, which may be end-user connections or other QWServers. In each of these roles the mode of delivery is a reliable push of messages over a persistent connection that the client establishes and maintains. An additional Web Services module may use an encapsulated QWServer program to support transient clients.

Figure 1 Data Provider and Data Server components
Data Provider and Data Server components

Return to Table of Contents

Software Requirements

Data Provider and Data Server

Role
 QWIDS Modules Required
 Additional Items Needed
Data Provider
 QWServer (ViaFile/Corba Feeder)
 Optional: Certificate, Public Key, Private Key
Persistent Data Server
 QWServer (QWRelay Feeder)
 Optional: Login information to Data Provider, Certificate of Authority, Certificate, Public key, Private Key, MySQL server
Transient Data Server QWServer (QWRelay Feeder) QWWebServices Module Optional: Login information to Data Provider, Certificate of Authority, Certificate, Public key, Private Key
Required: Apache Tomcat Application Server, Apache AXIS Web Services Server

Additional software required to run QWIDS:
  1. Java Runtime Environment (JRE) - version 1.4 or higher
  2. Any Certificate of Authority (CoA) Software (OpenSSL.org can be used) - if authentication of messages is to be used with QWIDS.
  3. MySQL 4.X - if authorization of Persistent Data Recipients (via SQL database) is required.
  4. Hosts on which Data Providers and Data Servers are running must be time synchronized using a service like NTP.

Client/Server Programs

The program "jar" files for the client and server programs contain all of the class and library files needed to operate.  To build the programs from source requires the "QuakeWatch Library" distribution and serveral third-party libraries.  See the documentation included with the source code for more information.

Return to Table of Contents

Quick Start Test Setup

The QWIDS distribution is pre-configured to operate in a test setup, with all components running on the same system.  The instructions below show how to start up the various components in the test setup.  See the documentation that comes with each component for details on how to customize the configuration.  ISTI recommends that a deployment plan complete with public/private key infrastructure (PKI) authentication be developed for full configuration of a QWIDS setup.

If running on a Unix-based system, the script files (i.e. "startAll", "stopAll", "run", etc) will need to be marked as "executable" via the "chmod" command.  For example, "chmod 755 startAll".

QWServer Data Provider

The QWServer is pre-configured to run using a 'QWCorbaFeeder' as its data source.  In this test setup, input data will come from the console.  Before running the QuakeWatch server, the Notification Service CORBA-event-channel program (included with the QWServer distribution) must be started.

Start up both programs by executing the "startAll" script (Unix) or "runAll.bat" batch file (Windows) from the "QWServer" directory.  The server will generate log output files to the "log" directory and message-archive files to the "storage" directory (both in the "QWServer" directory).

The QWServer program jar ("QWServer.jar") includes the test program 'CorbaSenderTest' for entering data from the console into a QWServer running a 'QWCorbaFeeder'.  Start up the the test program by executing the "runSender" script (Unix) or "runSender.bat" batch file (Windows) from the "QWServer" directory.  While the "CorbaSenderTest" is running, text characters typed into the console will be entered into the QWServer after the <Enter> key is pressed.  If the message is successfully entered into the QWServer then the text "Feeder response:  true" will be displayed.  Entering the 'Q' character followed by the <Enter> key will terminate the program.  Note that in the default test setup, duplicate messages are not allowed and will be rejected.

Stopping the QWServer:  Under Unix, the "stopAll" script may be used to terminate the QuakeWatch server and Notification Service CORBA-event-channel program.  Under Windows, the programs will each run in a console window.  Selecting the console window and hitting "Ctrl-C" will terminate the program.

See the "Running QWServer" section of the QWServer Manual for more information on starting and stopping the server.

Persistent Clients

The 'QWConOutClient' program is a test client that connects to a QWServer, receives messages, and outputs them to the console.  To start up the program, execute the "run" script (Unix) or "run.bat" batch file (Window) from the "QWConOutClient" directory.  With the above QWServer test setup and the 'QWConOutClient' program running, messages entered into the 'CorbaSenderTest' should be echoed by the 'QWConOutClient'.  Note that the messages shown by the 'QWConOutClient' program will be "wrapped" by "<QWmessage>" and "<DataMessage>" XML elements.  The program will generate log output to the "log" directory in the "QWConOutClient" directory.  Entering the 'Q' character followed by the <Enter> key will terminate the program.  See the "Console Output Client" documentation for more information.

The 'QWFileOutClient' program is a client that connects to a QWServer, receives messages, and outputs them to individual files (one file per message).  To start up the program, execute the "run" script (Unix) or "run.bat" batch file (Window) from the "QWFileOutClient" directory.  With the above QWServer test setup and the 'QWFileOutClient' program running, messages entered into the 'CorbaSenderTest' should be echoed into files in the "fileOutData/msgOutput" directory in the "QWFileOutClient" directory.  The program will generate log output to the "log" directory in the "QWFileOutClient" directory.  Hitting "Ctrl-C" will terminate the program.  See the "File Output Client" documentation for more information.

The 'QWCubeOutClient' program operates like the 'QWFileOutClient' program, except that the recevied messages are converted to CUBE format before being outputted to the files.  The 'QWCubeOutputClient' may be used as a functional equivalent to the QDDS program (operating as a "leaf").  See the "Cube Output Client" documentation for more information.

The 'QWMsgValidateClient' program is a test client that connects to a QWServer, receives messages, and validates them against an XML schema.  See the "Message Validating Client" documentation for more information.

The QuakeWatch JNI Client Receiver ("QWJNIClient") provides a mechanism for 'C'/'C++' client programs to access a QuakeWatch Server.  See the "JNI Client Receiver" documentation for more information.

Note:  A successful connection to the QWServer is indicated by the message "Connection verified to server".  Multiple instances of the client programs may be run at the same time (as log as each instance operates out of a separate directory).  In a larger, real-world setup, the "Data Provider" QWServer would send messages to one or more a "Persistent Data Server" QWServers, and the clients would connect to the "Persistent Data Server" QWServers.

Client-Server Login Authorization

The QWServer may be configured such that connecting clients must present a valid username/password entry when they connect.  For "large scale" messaging setups a database is used to hold the username/password entries--see the "Password Database Configuration" section of the QWServer Manual and the PHP-based client-login provisioning system for more information.  Alternately, a QWServer may use a local password file to hold the username/password entries--see the 'localPasswordFile' parameter description in the QWServer Manual for more information.  The above test setup may be modified to enforce client-server login authorization (using a local password file) as follows:

Terminate any QWIDS components that are running.  The "test/TestLocalPwdFile.txt" file in the "QWServer" directory contains a sample local password file.  Add the following line to the content of the "<Settings>" element in the "QWServer/conf/QWServerConfig.xml" file:
    localPasswordFile = "test/TestLocalPwdFile.txt"

The "test/TestLoginInfoFile.txt" file in the "QWConOutClient" directory contains a sample server login information file.  See the 'loginInfoFileName' parameter description in the QuakeWatch Connection Parameters document for more information on this file.  Add the following line to the content of the "<Settings>" element in the "QWConOutClient/conf/QWCOutClientConfig.xml" file:
    loginInfoFileName = "test/TestLoginInfoFile.txt"

The other client programs may be configured to use a server login information file in a similar fashion.

Starting up the test setup (as per the previous instructions) should result in the client using the username/password entry from the file to successfully connect to the QWServer.  The QWServer will reject a connection from any client that does not present a valid username/password entry.

Persistent Data Server / Other QWServer Feeders

The QWServer may be configured to use various data-input feeders by modifying the QWServer's "conf/QWServerConfig.xml" configuration file.  The 'QWRelayFeeder' may be used to have a QWServer ("Persistent Data Server") receive data from another QWServer ("Data Provider").  The "QWServerConfig_relay.xml" file in the "QWServer/conf" directory provides a sample QWRelayFeeder-based configuration.

An instance of a QWServer may be configured to use more than one 'QWRelayFeeder', allowing for input from multiple "Data Provider" QWServer sources.  Various combinations of other feeders are also possible.

Multiple instances of QWServer may be run on the same machine as long as they operate out of separate directories and use different port numbers (via the 'serverPortNum' parameter in the QWServer configuration).  They may share a single Notification Service event channel, or a separate event channels at an alternate port numbers may be launched--see the 'notifSvcPortNum' parameter description in the QWServer Manual for more information.  Each instance of a QWServer should always have a unique server-ID name setup via the 'serverIdName' parameter in the QWServer configuration.

The 'QWViaFileFeeder' may be used to have a QWServer ("Data Provider") receive messages via input files (similar to how a QDDS "hub" operates).  The "QWServerConfig_viafile.xml" file in the "QWServer/conf" directory provides a sample QWViaFileFeeder-based configuration.

See the "QuakeWatch Feeder Modules" section of the QWServer Manual for more information on QWServer feeders and their settings.

Transient Data Server

The QuakeWatch Web Services server implements a system for delivering messages via the Simple Object Access Protocol (SOAP) over HTTP via a polling mechanism.  The Web Services server uses an encapsulated QWServer to manage the collection and storage of messages.  This QWServer is usually configured to use a 'QWRelayFeeder', receiving messages from a source "Data Provider" QWServer.  The QuakeWatch Web Services must be deployed into a Axis-compatible web server, such as Apache Tomcat.  See the "QuakeWatch Web Services Manual" for more information.

Return to Table of Contents

Application Programming Interfaces:

QWServer API

The QWServer has two CORBA Interface Definition Language (IDL) specifications, one for gaining access to the services and one for the services themselves. They are defined by the "QWAcceptor.idl" and "QWServices.idl" files, which are included in the "QWLibs" source code (in the "QWLibs/QuakeWatch/QWCommon/idl" directory).

Directly accessing the QWServer through the QWAcceptor/QWServices IDLs is not recommended because doing so will bypass the connection and message handling code in the QWCommon library.  The recommended procedure for creating a new "persistent" client with Java is to extend the 'QWClientBase' or 'QWTrackingClient' class found in the QWCommon library.  For 'C/C++' access, the best approach may be to receive the messages into a Java-based "receiver" client and then pass them to the 'C/C++' program via CORBA methods or JNI (Java Native Interface) calls.  (ISTI has created test versions of each.)

QWServer Data Provider API (Via QWCorbaFeeder Plug-In)

Data messages may be fed into a QWServer (configured to use a 'QWCorbaFeeder') via CORBA methods, as defined by the "QWFeeder.idl" file.  See the sample "com.isti.quakewatch.server.test.CorbaSenderTest" class in the QWServer source code for an example.

QWWebServices API

Web Services ("transient") clients may use the methods described in the Web Services documentation.  See the sample Web Services client for an example.

QuakeWatch Clients

QuakeWatch client programs may be created using Java by extending the 'QWClientBase' class in the QWCommon library (part of the "QuakeWatch Library" distribution).  The code in 'QWClientBase' implements the details of server connection and message handling.  The client program implements a call-back to process the incoming messages (via the 'QWDataMsgProcessor' interface).  The QWConOutClient program is an example of this type of client.

The 'QWTrackingClient' class (in the QWCommon library) enhances 'QWClientBase' by using a tracking file to hold information about the last-received message.  As such, client programs created by extending the 'QWTrackingClient' class will "remember" the last-received message before shutdown and request any missed messages after being restarted.  The client program implements a call-back to process the incoming messages (via 'MsgProcessorInterface').  The QWFileOutClient program is an example of this type of client.

Return to Table of Contents

Example Transient Client

See the Web Services client documentation for more information.

Return to Table of Contents

Example Persistent Client

See the Console Output Client, Cube Output Client, File Output Client and JNI Client Receiver documentation for more information.

Return to Table of Contents

Open Source

The QWIDS product does not contain any proprietary components that require licensing if used for non-commercial use and is entirely open source. It uses the free OpenORB Java-based CORBA engine (found at http://openorb.sourceforge.net) for the framework of the server and clients using persistent connection push modes. For the storage of persistent client login information it uses the MySQL database engine, which is free for non-commercial use (see http://www.mysql.com for licensing terms). On the publisher and subscriber side IDLs, the open source, C++-based TAO CORBA engine has been used successfully to interface C-based programs with the QWServer modules. The C++ TAO CORBA engine is free for any usage, commercial or non-commercial.

ISTI provides the source code for all its components for free non-commercial usage with any delivery of the QWIDS product and thus it is an open-source product. ISTI has copyrighted the source code for the QWServer and all of its components that were developed by ISTI with ISTI funding. ISTI has developed several of the plug-in and client modules with its own funding and provides these at no additional cost with source code. In particular, ISTI has developed the QWViaFile and QWRelay feeder plug-in modules for the QWServer and also the QW File-Output Delivery Client on its own.  While these are proprietary components, we will provide these in source-code for free non-commercial usage.

Non-commercial usage in the context of this quote is that the CISN and ANSS can use the system to deliver any messages to any end-users. Any component of the system can be used for this message distribution. However, commercial entities may only use the QWRelayFeeder plug-in for receiving messages through this system and not for their own internal message distribution.

Return to Table of Contents

Operating Systems

The QWIDS components for the server side roles have been tested on the following server operating systems and platforms:

The QWIDS components for persistent client side roles have been tested on the following operating systems:

The Web Services (WS) Transient Clients, using their web protocols, are able to run on any platform.

Return to Table of Contents

Architecture

The QWIDS components are depicted in Figure 1 above. A less detailed architecture showing the scalability of the system is shown in Figure 2 below.

Architecture Diagram

Figure 2 QWIDS architecture diagram showing message flow

An example of message flow (red lines show direction) from Data Provider 1 (DP1) through to the various Server modules (S#) which are downstream.  The flow of data is that DP1 generates a message and pushes the message to both S1 and S2, which in turn each push the message to S4 and S5. S4 also pushes the message to S5 and S6. The client C receives a pushed message from the servers if it is a persistently connected client or polls a service to obtain the message if it is a passive client (more on the client interactions later).

There are two tiers of server modules to provide a way for interconnections between servers and provide reliability of message delivery. The first tier server S1-3 (Tier 1 or T1) provides a connection point for a Data Provider to inject messages into at least 2 servers (note line from each DP to at least two T1 servers). The second tier of servers S4-6 (Tier 2 or T2) is a complete interconnect switch between the Tier 1 servers such that any message injected into Tier 1 makes it to every Tier 2 server.  Note that Tier 2 servers are all interconnected to one another such that they relay messages between themselves.

While it is hard to convey in this schematic, the communication between Data Provider and Data Servers and between Data Servers are bi-directional. In each case, the downstream server initiates the connection to the upstream source, but the data are pushed. Since duplicate messages are tossed out at a server, there may be redundant transmissions of a given message inside the cloud of servers. The redundant transmissions can use up bandwidth – however, that is the cost of redundancy. The client will only receive one copy of the message from DP1 because duplicates never propagate from any server.

Duplicates are removed if the message content is an exact match using a string hashing function. That is, if two messages with exactly the same content are injected into the system one hour apart the second copy will not propagate to any Data Server elements (and the Data Server is configured to hold messages longer than one hour). Messages are retired from the cache after the pre-configured server cache message-hold-time expires.

Return to Table of Contents

Performance and Scalability

The QWIDS product has been stress tested to provide throughput of a single message to 1000 persistent connected clients within 1.4 seconds from a single server host (See Appendix A) and single QWServer process. The system is easily scalable by adding in additional QWServers in Data Server and Data Provider roles. The system is designed to shed load to alternate servers when a configurable number of persistent client connections has been made and has a built-in capability for providing Data Recipients with lists of alternate servers to connect to.

Return to Table of Contents

Communications Standards

The QWIDS product uses CORBA for all communications between QWServers and persistent clients. CORBA uses standard Internet communications protocols (IIOP over TCP/IP) and is endorsed by the Object Management Group (OMG). The OpenORB CORBA engine that QWIDS uses is CORBA compliant at the OMG CORBA version 2.4.2 level. For the Transient Service from the Data Servers, we will be using Web Services (i.e., SOAP and WSDL) for providing polling of messages to transient clients.

Return to Table of Contents

Message Format Neutral

The QWIDS product is message neutral. That is, it accepts any XML formatted message and passes it through the system end to end without modifications. Thus it allows unlimited updates and new messages to be passed through the system.

The QWIDS product currently has a versioning capability built into the QWServer to notify clients of software updates. This versioning is provided not as a message but as a service (via CORBA methods) for persistently connected clients. The clients each have a client-type that can be passed along to the server along with the client’s current version number. The client can then be rejected at login time that their version is out of sync with the latest version of that client-type and/or they can receive a message and URL where they can obtain an updated client version. In the case of the CISN Display QWClient, the system can automatically upgrade itself after a user confirms the operation.

The QWIDS product has system-operations messages that are periodically sent from the server to any persistently connected server or client. These messages are already predefined.

Return to Table of Contents

Message Size Limitation

The QWIDS product can reject messages at the point of publication based on the size of the incoming message. The message-size limit is configurable in the QWServer (via the 'maximumMessageSize' parameter). Any messages that are larger than the size limitation are logged and any error messages are sent to a logfile.

Return to Table of Contents

Security

Connections between Data Provider and Data Server System

Login transactions between the Data Provider and Data Server system are encrypted and all subsequent traffic is done using digital signing of messages. Logins are stored in a password file encrypted using DES or in a MySQL database. If the MySQL database is used, then a MySQL cluster can be configured to serve the password file to all Data Server elements to provide uniform access across a tier of QWServers.

Secure Connections between Data Server System and Persistent Clients

Login transactions between the Persistent Clients and Data Server system are encrypted. Furthermore, Transient and Persistent Clients can use the signature on the message envelope to verify that the message was sent from an authorized Data Provider.

Transient Clients, while not addressed in this requirement, can be secured simply by using the HTTPS protocol instead of HTTP since all traffic is web based.

Authentication of Data Providers and Persistent Clients

The Data Server system may be configured such that all Persistent Clients and Data Providers must provide and use valid username/password combinations. In the case of Data Providers, they provide a username/password combination to the Data Server element for the connection to be established to them. The passwords and logins are stored in a protected file or in a database on the QWServer host and all passwords are DES encrypted.

Digital Signatures for Messages

The QWIDS product supports RSA signing of messages at the Data Provider along with X509 Certificates to provide a complete authentication process for all messages that are sent using the system. The QWClient library has the capability to verify any message was sent from an authorized source by requesting the Certificate of Authority (CoA) and Certificate Revocation List (CRL) as well as the Data Provider’s certificate. The CoA and CRL can be obtained from URL’s provided by the designated certificate authority (i.e., the generator of the certificates for the system). The Data Provider’s certificate is passed downstream to the next QWServer element where the message is confirmed and resigned using the QWServer’s certificate. In addition to providing a means to authenticate the source of the message, the digital signature also allows a client to assure that the message has not been tampered with and has arrived intact. The QWIDS product does not supply the tools with which to create keys or the certificates necessary to perform digital signing but we recommend the OpenSSL open-source toolkit for accomplishing this task.

Return to Table of Contents

Scalability Discussion

The QWIDS system has been tested with a single QWServer able to handle a load of 1000 persistent clients. Since the QWServers can be interconnected indefinitely and are designed to minimize duplication of messages, the system should scale well as more servers are added to support more users. A well-designed topology of Data Server elements will be able to accommodate any increased load. 

An example topology of servers is described in Figure 2 (above) in which there are two tiers of servers. In this example, each Tier 2 server is interconnected to every other Tier 2 server as well as to some Tier 1 servers. Each server is hosted on a different computer. Redundancy is provided by having several interconnects between the Tier 2 and Tier 1 servers to provide alternate routes for messages in case of network failures.

Because the QWIDS Data Server element can scale simply by adding another QWServer as a client of an upstream node, the Tier 2 can be splintered into a Tier 3 at each institution hosting a Data Server system. As an example, the Tier 2 server S4 at the USGS starts to saturate with local clients and alternate servers need to be added.  Three new servers could be added at the USGS location and the load on S4 can be dispersed to S7, S8, and S9 as shown in Figure 4 below. Thus, the S4 server would have 3 new downstream clients, and could shed its load to them when it reaches its client carrying capacity to other Tier 2 servers but also to its new Tier 3 clients (which are QWServers).  In this example, when the USGS installation had just the S4 server it would handle 1000 persistent clients, but with the additional servers added, 3000 more persistent clients could be supported.

Scalability Example
Figure 3 Scaling of Tiers of Data Servers

The scaling of the QWIDS system is built into the persistent client login protocol where a list of alternate servers is provided at startup. In addition, if a persistent client is logging into a QWServer that has reached capacity, it will be redirected to try another QWServer that may have less load.

Return to Table of Contents

Redundancy and Robustness


The redundancy in the QWIDS product depends on how it is deployed as a whole system. Each element in the Data Server system can have a redundant component such that if one element fails, a second one can take over its role. Likewise, by having redundant components there is increased capacity in the system for some elements. Each element has different features built-in to it to provide for fail-over to other components.

Component Redundancy

The Data Provider component can have multiple data sources feed into a single QWServer and the redundancy of message generation will be dependent upon each of those data sources. Likewise, a Data Provider can use two QWServers to feed messages downstream. As explained earlier in section 1.3, duplicate messages are dropped at the QWServer.

The Data Server component is designed to have multiple QWServer processes work together to provide a service to Transient and Persistent clients. If a well-designed network topology of QWServers is built, then the failure of any single QWServer process will not effect the operation of the service. The next section (1.8.2) describes how multiple servers should be used to provide redundancy.

The Data Recipient component is designed to detect failures of the QWServer and failover to an alternate server.

Multiple Servers

The Data Server element is critically dependent upon how the servers are deployed and interconnected (the topology). An example topology with two tiers of QWServers is show in the Figure 4 below. In the example Data Provider 1 (DP1) pushes a Message A (blue arrow) to Servers 1 and 2 (S1 and S2), and S1 pushes the message on to S4 which pushes it to persistent Client 1 (C1). In addition, S2 attempts to push the message to S4 because of the redundant path, but S4 sees that it already received the message from S1 and ignores the push of that message from S2. The red arrows represent redundant pushes of Message A to other Sever elements and are further described in the next subsection (1.8.3) below. The order of the messages from a single Data Provider will be guaranteed to be in the order provided (see section 1.3.7.6 for a detailed description of the failover process).

Redundancy Example

Figure 4 Failover and Recovery example

Server Failure and Recovery

The failure of any single QWserver in a well-designed topology will not effect the distribution of messages within the QWIDS system. The best way to explain the recovery performance of the system is through a failure and recovery example using Figure 4 above. If server hosting QWServer S5 should lose its connection to the Internet or crash before DP1 generates Message A, then Message A pushes from S1, S2, and S4 will not make it to S5 until it recovers. Meanwhile Client 2 (C2) has been receiving messages from S5 until it crashed, so it doesn’t receive Message A pushed immediately, but it knows about S4 and S6 in the server cloud (it learned of their existence at login time from S5). C2 then successfully connects to S6, which received Message A pushed from S4. C2 gets pushed Message A from S6. Upon recovery, S5 receives Message A pushed from S4 (or S1 or S2) and completes its list of messages for delivery to any new clients that should connect to it.

The example outlined in the previous paragraph highlights the importance of a good server topology designed for redundancy. If the interconnect between S4 and S6 did not exist, then the failure of S5 would delay the delivery of Message A to any clients connected to S6 until S5 recovered. An additional interconnect could be made between S2 and S6 to further improve this example topology.

Reliability

As an example of the reliability of the QWServer core we show that the ‘qws2’ server (a RedHat 9 Linux Pentium-class PC) has been running for over 224 days continuously without a restart (as of this writing):

[qws@qws2 ~]$ uptime
 07:27:01  up 241 days, 16:44,  3 users,  load average: 0.00, 0.00, 0.00
[qws@qws2 ~]$ ps -ef |grep java
root     18346     1  1 Feb27 ?        2-17:55:08 java -Xms16m -Xmx256m -jar ISTINotifServer.jar -l=debug -e -t=100 -f=1800 -g=1 -x=10 -p=80
root     18348     1  0 Feb27 ?        10:25:23 java -Xms16m -Xmx256m -jar QWServer.jar
qws      15816 15793  0 07:27 pts/2    00:00:00 grep java
[qws@qws2 ~]$ date
Tue Aug 17 07:27:18 PDT 2004

Failure of an individual Data Server component is not critical to the system if the network topology of servers is well designed and includes redundant links between servers. The QWServer has the capability to shed load by directing a client to any of a list of alternate servers it has in its internal list (configurable). The list of servers is dynamically read by the QWServer and can be distributed around to the Data Serving hosts via the UNIX rdist program.

The tiered approach to the Data Server topology will allow for maximum reliability. That is, the first tier should be dedicated to only providing a downstream pooling of messages and the second tier, which connects to the first via multiple paths, should be dedicated to just serving Data Recipient clients. The second tier will be scaled as necessary to meet client demand.

Upon failure of any given Data Server, clients will redirect themselves to alternate servers. The clients learn of alternate servers when they first login to a QWServer. If they are unable to connect to the server that they first logged into, then they attempt another alternate server on the list until they find one that is up and not at capacity.

Return to Table of Contents

Data Provider Description

The Data Provider element in QWIDS is a persistent-storage server that allows the Data Server elements to connect to it using CORBA to receive a reliable push of messages. The Data Provider server accepts the message from the source, digitally signs it (optional), and stores it persistently on disk for some configured time period. Once a message has been published, any subscribers of the message that are authorized (i.e., that have the capability to login directly to the provider’s server) will receive the message via a CORBA push.  If a Data Server element has been unable to connect to the Data Provider server (because of communications problems), it can retrieve any missed messages that it would have been pushed during the outage. Any subsequent messages will be pushed after the Data Server has caught up to where it left off before the outage.

The QWIDS product’s Data Provider element is a QWServer module with the QWViaFileFeeder plug-in or with the QWCorbaFeeder plug-in (or both in combination). The QWViaFileFeeder plug-in allows a Data Provider to supply XML messages for publishing simply by placing a file in a specific publishing directory (i.e., just like the QDDS system does). Alternatively, the QWCorbaFeeder plug-in allows the publisher of messages to write a C++ or Java based program to do the work. This is represented in Figure 1 as “Data Publisher A”.

Registration with Server

The Data Provider element acts as a server of messages and may be configured to allow logins only from authorized server elements. It is authorized as a provider simply by the act of notifying the authority controlling the Data Servers that they have a login access. It may be further authorized as a legitimate provider of messages by clear signing each message using a private-key and a hash function to create a digital signature and providing a certificate issued by the federation of providers at login time. The QWServer will provide a key for signing and a certificate for presentation to any downstream clients that will receive messages that the server manages. The certificates are provided to clients as part of the login negotiations by the client (i.e., if the client cares to verify the signature inside the data).

While the QWIDS product uses public/private key infrastructure (PKI) technology, it does not provide the capability to generate the necessary keys directly. ISTI recommends the OpenSSL.org project for managing and generating keys for QWIDS. We provide a more detailed discussion on the PKI usage in section 1.4 on QWIDS Security.

Standard Interface

The QWServer QWCorbaFeeder plug-in module provides a way for Data Providers to publish messages using a custom program. The standard interface to this is described using a CORBA Interface Definition Language (IDL) and two example publishers have been written. One example is in C++ and was written by Kalpesh Solanki of Caltech and the other is a Java module written by ISTI and provided with the QWIDS system in source code form.

Persistent Connection with Server

In the QWIDS product, the Data Server system is responsible for maintaining a connection to the Data Provider and not vice versa. This persistent connection is done via CORBA technologies.

Data Source Heartbeats

The QWServer injects regular heartbeats into the system such that any downstream modules can detect a loss of communications with the server. The heartbeat interval is configurable in the server configuration file (via the parameter named 'aliveMsgDelaySec').

Connection Re-establishment

In the QWIDS model, the Data Servers establish and maintain the connection to the Data Provider. The Data Server modules monitor heartbeats from the Data Provider and, should a connection fail, the Data Server receiver will attempt to reconnect to the provider’s server and catch-up where they left off. In the QWIDS Data Provider model, each Data Provider server would be connected to by at least two or more downstream servers for redundancy (see Figure 2 above).

Push of Messages to Server

The standard communication between any QWServer and a client is via CORBA login and then CORBA push of messages. Thus, the messages injected into the Data Provider’s QWServer will immediately be pushed to any Data Server QWServers that are persistently connected.

Guaranteed Order of Delivery

The QWServer’s persistent storage mechanism locally stores messages in the order received and provides them in the same order. Further, each message is uniquely identified by the data source using a sequential message-ID number and the Data Provider’s unique ID.

Message Logging

The QWServer logs and timestamps any messages that it receives. Thus, the QWServer on the Data Provider element will log when the message is injected into the system and the Data Server element’s QWServer will log when the message was received into the server system.

Return to Table of Contents

Data Server System Description

The QWIDS Data Server element consists of a number of QWServer modules interconnected using the QWRelayFeeder plug-in. The Data Servers are connected to the Data Providers using the same feeder plug-in module. Authorization of connections between Data Servers and Data Providers and between Data Servers may be done using a login and password combination. The Data Server element has the responsibility to keep the connection with the upstream Data Provider modules. Once the connection is established, messages are pushed to the Data Server modules. Messages may be signed digitally at the Data Provider’s side and verified at the Data Server module. Any messages that are not signed using a properly authorized key can be rejected and logged. Any messages that are too large (vs. a configurable limit) are also rejected and logged. Duplicate messages received at a Data Server element (e.g, if it came from another QWServer that is relaying a message) are dropped.

All of the hosts on which Data Providers and Data Servers are running must be time synchronized using a service like NTP. This requirement is to ensure that messages received by Data Recipients are synchronized when a Data Recipient queries multiple Data Servers (e.g., during a failover or when a load balancing action is taken). This requirement is only necessary as a fall back mechanism in case the approach described in below is not possible for a given client.

Each Data Server module can talk to every other Data Server module if configured to do so. This configuration allows for an infinite number of interconnects between QWServers and thus provides for scalability of the Data Server element. Figure 2 above shows one example configuration of six servers that can all be interconnected in a grid to provide scalability and redundancy. The first tiers of servers in the figure are for connecting to the Data Providers and providing each data source with at least 2 connection points. The second tier of servers connects to the first for receiving messages and all are interconnected with each other within the tier. This design allows a Data Recipient client to connect to any Tier 2 server and reliably receive a message from any data provider. Tier 2 can scale to add more servers as clients are added to the system and Tier 1 can remain a relatively fixed size since the number of data sources is relatively small.

Configure List of Data Providers

Each QWServer that plays a Data Server role is configured to talk with an authorized Data Provider. Configuring a QWRelayFeeder plug-in to point at a specific Data Provider requires the IP address and port number of the provider and, in some cases, a valid username/password login. One QWRelayFeeder plug-in must be configured per Data Provider. Each QWRelayFeeder plug-in executes in its own program thread that independently maintains the feed from a given Data Provider.

Registration of Authorized Data Providers

Data Provider modules authorize the Data Server element to login to it (not vice versa). In this way, rogue Data Providers are prevented from transmitting messages into the server system. Once a Data Provider authorizes a Data Server element to login, it provides a persistent connection through which digitally signed messages are pushed. The login from Data Server to Data Provider is secure in that the login scheme is encrypted between the two hosts (i.e., no clear text passwords are sent). Messages digitally signed by a Data Provider can be verified at the point of reception at the Data Server. Note that digital signing of messages is an optional feature in the QWServer.

Each Data Provider is provided with a unique ID with which to tag their messages, and each message is tagged with a sequential message number.

Messages Pushed from Authorized Providers

In the QWIDS system, the client side opens all persistent connections and messages are pushed from the server by default. The only case where push will not happen is if a client is trying to catch up from a lost connection and in that case, there is a pull of data. Only authorized Data Providers (those configured on the Data Server side) will be pushing data into the Server system.

Verification of Message Integrity

The QWIDS system uses digital signatures on messages to authenticate the source of a message. The process of verifying that an authorized Data Provider signed the message also insures that the message arrived intact without any changes. That is, a unique hash of the message is created by the Data Provider and encrypted using a private key; the recipient (Data Server) obtains the certificate of the Data Provider and decrypts the signature to obtain the same hash. The recipient then hashes the message using the same algorithm used by the Data Provider and they two hashes should match. A failure to match indicates that the Data Provider is not who they say they are or that the message was corrupted in transmission.

Missed Messages by Data Recipients

When Data Recipients connect to a Data Server element, they can catch up from their last known message that was delivered. Data Server’s issue “alive” heartbeat pushes to all connected clients to provide both a server-alive notification and also an indication of the last message that a server pushed. Upon receiving an “alive” message, a Data Recipient can determine if it has missed any messages since it was last connected to that particular Data Server element. The Data Recipient can then retrieve a list of missed messages from the Data Server. Message numbers and timestamp values are used to insure that all messages are received in the order that they were provided from the Data Providers.

Message Delivery upon Failover

Every data source is assigned a unique ID and maintains its own sequential numbering.  When a message first enters the system, it is "tagged" with the unique ID of its data source and a "data-source" message number (which increments for each message).  Once the data-source ID and message number are attached to a message they are never modified.  Every message in the system will always have a data-source ID and message number.

All QW servers and clients will maintain a tracking table of data-source IDs and message numbers such that the last-received message number for each of the data sources is always known.  This tracking will be separate from the current (server-assigned) message-number/timestamp values, which will still be used to guarantee sequential message delivery while a client is connected to a single server.

Whenever a client establishes a connection to a "new" server, it will provide the new server its list of data-source IDs and the last-received message number for each ID.  In response, the server will locate the oldest message in its cache that corresponds to any specified by the client's list of data-source IDs and message numbers.  The server will then return to the client, for each data source, all messages with message numbers greater than the given message numbers.  (The order of returned messages will be the same order as in the server's cache.)

If, while the server is selecting messages to be returned, a message with an "unexpected" data-source (one not specified by the client) is encountered, the message will be passed along to the client.  This should only happen when a new data source is first introduced to the system.

The client will also provide the timestamp of its last-received message, and the server can use it as a "fall-back" method of selecting messages if none of the message numbers can be matched.  The timestamp will also act as a "reference point" from which to search for messages in the server's cache.

Note that the above references to a QW client also apply to the QWRelayFeeder feeding into a QWServer (acting as a relay), because the QWRelayFeeder runs as a client to another QWServer.

Another advantage of this new scheme is that it should also prevent the occurrence of duplicate messages when a client switches servers because a message-numbering system is always maintained.

Buffer of Received Messages

The QWServer persistently stores any messages received for transmission for a configurable number of days. The combination of a memory cache and disk archive is used to provide fast access and persistent storage.

Transmission of List of Messages

The QWServer provides the Data Recipient with the capability to receive a list of missed messages.  It currently sends the messages that were missed and not a list.

The catch up time (e.g. how late a message can be and still be provided to a recipient) is configurable on the QWServer. This added feature of the QWServer allows clients that have not been connected for a long time to only get a recent subset of messages and not devour all the bandwidth of the server.

Transmission of Individual Messages

The QWServer element can be configured to provide individual messages to clients upon request, but the majority of persistent clients receive messages via the push mechanism and thus do not specifically request individual messages. That is, the QWServer does not support polling directly, but through an intermediate QWWebServices module.

Registration of Data Recipients

The QWServer can be configured to require a login and password from a Data Recipient. Any unauthorized connection attempts to a QWServer that requires a login will be rejected and logged.

Logging of all Registrations

At the time a Data Recipient logs into a QWServer, all of the information about the client is logged to a file. In particular, the IP address, the user name, the version of the client software connecting, and the time of connection.

Configuration Control of Data Provider Element

At the time of login to a QWServer, the server provides information about itself and the connections that will follow. These are configurable in the server startup file. The principal configuration that must happen for a Data Provider element on the Data Server side is the authorization information and location of the Data Provider (IP:port). In addition, the Data Server can discover some of the parameters of the Data Provider (i.e., size of messages, size of buffer), but cannot control them.

Persistent Client Connections

The persistent clients will use CORBA to initiate an outbound TCP/IP connection to a QWServer’s IP address and port. All communications with the server will flow down the client-initiated connection. In addition to communications with the QWServer, a second TCP/IP connection will be established for message push from the Event Channel server (managed by the Notification Service).  The use of a bi-directional connection in CORBA that is initiated by the client aids in connecting through a firewall.

The outbound connections to the QWServer and the Event Channel can be hampered by a firewall that restricts any outbound connection to ports that are not well known. In some cases, firewall restrictions can be overcome if the QWServer and Event Channel are setup at well-known ports for HTTP and HTTPS (e.g., ports 80 and 443 respectively). Administrators will need to allow connection to an outbound port if these well-known ports are not used and the firewall is restrictive. Two caveats to using ports 80 and 443 are: 1 if the firewall device/software does packet introspection, the packets are CORBA IIOP and not HTTP as might be expected, and 2 if an HTTP web proxy is in use, any CORBA connections will not make it through unless the proxy is configured to allow the connections.

Transient Client Connections

The transient clients use SOAP via HTTP to make all connections on the well-known ports 80 or 443 and thus should suffer no firewall restrictions. That is, as long as a site allows outbound connections to these ports; and most do since all web traffic flows to and from them.

Recipient Message Filtering

The QWServer has the built-in ability to filter messages based on type and domain. This additional capability could be extended to existing client subscribers (either transient or persistent). A project at Caltech (CISN Messaging System-CMS) is currently using the filtering capability of the QWServer to allow persistent clients to filter messages by subscribing with the server for only desired types.

Automatic Re-subscription and Message Recovery

Persistent Clients utilize a QWClient library that automatically recovers from connection drop-outs and allows the client to reconnect to the server and catch up any missed messages during the outage. Furthermore, if the client is unable to reconnect to the server, the client can switch-over to an alternate QWServer for messages. A list of alternate QWServers is provided to the clients during the login phase.

Transient Clients will use Web Services methods to re-issue requests to the QWServer and the server will provide results based on the method used. There is no subscription phase, so when the connection with the server is available the transactions can be processed. That is, Transient Clients will simply repeat trying to send a transaction to the server until it succeeds or will simply try another server.

Automatic Subscription

Persistent clients can be required to present a username and password entry at the time of login. All sensitive login information is encrypted. Persistent Clients obtain username and password entries from the Data Server authority. ISTI has built a PHP website for provision and management of QuakeWatch login identifiers and passwords.

Transient Clients, by the nature of using SOAP and HTTP, are automatically subscribed when they issue a method call to the server. There is no login phase for transient clients since WS connections are single events. Because SOAP’s underlying protocol is HTTP, the IP of the client is sent along during a method call.

Cancellation of Subscription at Shutdown

Persistent Clients monitor the QWServer "alive" messages and can detect the shutdown of the server by the lack of these message types appearing in the Event Channel.  When a persistent client shuts down it informs the QWServer that it is disconnecting.

Transient Clients by their nature, will have not have a persistent connection and thus cannot be informed of a server shutdown. Any pending transaction requests will be fulfilled before the server is shutdown if the shutdown happens during a WS method call.

Initiation of all Data Requests

Persistent Client connections and requests to the QWServer are all initiated from the client side.

Transient Clients use Web Services method calls to obtain new messages from the server. By the simple nature of Web Services, the client initiates all transactions.

QWServer Administration

The QWServer is a daemon Java program that is launched from the command line and provides many configuration options that are specified in a configuration file. The configuration file is written in XML and describes the configuration of the QWServer core and the QWServer feeder plug-in modules (Relay, ViaFile, CORBA). The software is remotely configurable if one can use secure shell (SSH) to login to a server system and run an editor or copy files. The UNIX software 'rdist' can be used to remotely distribute alternate server lists and versioning files. The QWServer detects and reacts to changes in its versioning file (and its password file if a MySQL database is not used).

One of the recently added features of the QWServer is the ability for the server to provide versioning information to Data Recipients. The client provides information about the current version of its software and is informed of any updates that are available. Furthermore, the QWServer can be configured to deny a login to a client whose version is too old.

Return to Table of Contents

User Agreement and Copyright

Disclaimer of Earthquake Information
Many data transported by the QWServer Software are preliminary and computer generated, and subject to change. Subsequent human review of earthquake records may result in approval of the event information; significant changes to magnitude, location and time; or deletion of the earthquake information. The data are not intended to, nor do they provide early warning or a prediction of future seismic activity. Many efforts have been made to provide accurate earthquake information, but reliance on, or interpretation of earthquake data from a single source is not advised. Consumers of the earthquake messages are strongly cautioned to carefully consider, and use commonsense in any decisions relating to personal or public safety, and the conduct of business that involves substantial momentary or operational consequences.

Disclaimer of the Software and its Capabilities
INSTRUMENTAL SOFTWARE TECHNOLOGIES INC. (ISTI) SPECIFICALLY DISCLAIM ANY WARRANTIES, INCLUDING THE IMPLIED WARRANTIES OR MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THIS SOFTWARE (THE QUAKEWATCH SERVER and its components) IS PROVIDED ON AN "AS IS" BASIS. ALL REASONABLE ATTEMPTS HAVE BEEN MADE TO RID THE PROGRAM OF SOFTWARE DEFECTS AND BUGS, HOWEVER ISTI HAS NO OBLIGATIONS TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS OR MODIFICATIONS. IN NO EVENT SHALL ISTI BE LIABLE TO ANY PARTY FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES, INCLUDING LOST PROFITS, ARISING FROM THE USE OF THIS SOFTWARE, ITS DOCUMENTATION, OR DATA OBTAINED THROUGH THE USE OF THE SOFTWARE, EVEN IF ISTI HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. BY DOWNLOADING, INSTALLING OR USING THIS PROGRAM, THE USER ACKNOWLEDGES AND UNDERSTANDS THE PURPOSE AND LIMITATIONS OF THE SOFTWARE.

Terms and Conditions of Software Use
Permissions to use this software for non-commercial, and informational (news gathering/reporting) purposes without fee and without a written agreement is hereby granted. Distribution, transfer, or modification of the software without prior written authorization is not permitted. For any other use, including commercial licensing of the QWServer software or any of its components contact Instrumental Software Technologies, Inc. at 77 Van Dam Street, Suite 10, Saratoga Springs, New York 12866 fax: (518) 602-0002, or go to www.isti.com

Ownership of Intellectual Property
Copyright 2010 Instrumental Software Technologies, Inc, ("ISTI").


Return to Table of Contents