NAV Navbar

Binary Market Data

Follow this link to learn what has changed since the previous version.

The deprecated version 1 documentation is available here.

Overview

Trading applications need information about the state of the market in order to effectively execute their strategies on an exchange. Seed CX provides a market data system which delivers Level 3 market data. Seed CX uses an instrument-centric, proprietary binary protocol to communicate with client applications. This document describes the protocol and serves as a comprehensive guide to consuming Seed CX market data.

System Overview

Seed CX Market Data is distributed using a binary protocol known as PitchFork, which is loosely modeled on industry-standard PITCH and ITCH protocols. It is a so-called Level 3 or market-by-order system, where a complete view of the market (down to the level of anonymized individual orders) is made available for every instrument traded at Seed CX. Client applications wishing to consume market data need to be connected to Seed CX.

As stated previously, market data in the Seed CX system is instrument-centric. Each instrument has it's own:

An instrument's market data comes in two flavors: full-depth market snapshots and incremental updates. These are delivered by two distinct services of Seed CX's market data gateway.

Seed CX Market Data System Diagram

The snapshot mechanism is used when joining the data stream and for recovery purposes. Snapshots are obtained by establishing a TCP connection to the Market Snapshot Service and sending an Instrument Snapshot Request. Such requests are meant to be infrequent and are rate-limited. Applications should obtain an initial snapshot and then apply subsequent incremental updates to it.

Incremental updates are sent by the Incremental Update Service on two redundant channels using UDP multicast. These updates are essentially commands (such as add order, delete order, replace order, etc.) to the application telling it how to update its in-memory order book.

Each multicast channel may contain feeds for multiple instruments. Refer to the reference data to obtain the multicast group addresses and ports for desired instruments.

Each UDP packet will only contain messages for a single instrument. Applications can inspect the instrument id contained within the packet header and decide to either ignore or process a given UDP packet. The Consuming Market Data section describes this process in detail.

Reference data

Reference data is crucial for properly interpreting values from the multicast market data feed.

Without reference data it would be impossible to know that instrument id 1 happened to represent COSP:BTC/USD today, while instrument id 7 happened to represent COSP:ETH/USD. It would be impossible to know that COSP:BTC/USD ticks in 5s, while COSP:ETH/USD ticks in 0.1s and its prices have an implied decimal point.

It is recommended that applications fetch a fresh copy of reference data before the start of each trading session, as information can change on daily basis.

The reference data endpoints respond to http GET requests and return a response in JSON format. Refer to the web API section on the /instruments endpoint for more information.

Important fields

Below are fields that are important when decoding binary market data.

id

id is the unique instrument identifier valid for the duration of the trading session. It is broadcast within the packet header.

code

The code is the human-readable name of an instrument. This value is used as symbol for FIX Market Data and FIX Market Access.

contract_size

The contract size is the size of a single contract lot. For example, if instrument's contract_size is 10 and the instrument traded with a quantity of 2, it means 2 * 10 units were traded.

tick_band

Tick band describes how an instrument ticks. It provides the minimum tick increment.

price_decimals

This field specifies the number of implied decimal places within an instrument's prices.

To preserve precision, multicast prices are represented as integers. In order to convert them into floating point numbers (or to format for display purposes), it is necessary to shift their decimal point to the left by the value provided in price_decimals field.

For example:

means the floating point price is obtained by shifting the decimal point 2 positions to the left:

market_data feeds

Every intrument in reference data response contains a section with multicast connectivity information for incremental and snapshot feeds.

Incremental feeds represent redundant UDP multicast channels where an instrument's market data will be broadcast. Each channel has a name (typically A or B), ip address and a port.

"incremental": [
    {
        "name":"A",
        "ip":"239.10.0.1",
        "port":1100
    },
    {
        "name":"B",
        "ip":"239.10.0.2",
        "port":1100
    }
]

Snapshot feeds contain the ip address and port of the TCP Snapshot Server. Only the public server is accessible to clients to use. IP marked as private is for Seed's internal monitoring applications.

"snapshot": [
    {
        "name":"public",
        "ip":"10.50.10.15",
        "port":65000
    }
]

Consuming Market Data

Each instrument has its own market, unique identifier, sequence number, and data feed. While feeds for several instruments may reside on the same multicast channel, every feed may be processed independently using the algorithm outlined below. Each step is expanded in detail later in this section.

  1. Initialize

    a) Retrieve reference data for the desired environment. This is needed for multicast IPs, instrument identifiers, price decimals and ticking.

    b) Open UDP sockets and join multicast groups.

    c) Prepare data structures for desired instruments.

  2. Perform feed recovery

    a) Begin caching incremental updates read from multicast channels.

    b) Request instrument snapshot.

    c) Reconcile snapshot with cached incremental updates.

    d) You're up to date. Go to step 3.

  3. Build book from incremental updates

    a) Read incremental update from multicast channels.

    b) Bad sequence? Go to step 2.

    c) Apply the update to the order book.

    d) Go to step 3.

Detailed Steps

Joining the Multicast Group

A client application should obtain reference data for a given environment and decide which instruments it wants to subscribe to. Each instrument's reference data contains information about the snapshot endpoint, as well as multicast group IP addresses which need to be joined in order to consume incremental updates.

Reading Incremental Updates from Multicast

For increased redundancy, the same incremental market data updates are sent on both A and B multicast channels. Due to the nature of UDP multicast, messages from either feed may arrive out of order. Applications consuming instrument feeds need to inspect the packet header to see if the instrument's sequence number has already been processed.

Sequence Numbers

Each instrument is sequenced separately, which implies that applications need to keep track of the sequence per instrument. The packet header in the Seed CX binary protocol contains the sequence number of the first message in the packet. The sequence numbers of remaining messages are implied. For example:

Packet Header
   instrument id: 1
   sequence number: 10
   message count: 3

Message 0
   implied sequence number: 10 (packet header + 0)

Message 1
   implied sequence number: 11 (packet header + 1)

Message 2
   implied sequence number: 12 (packet header + 2)

In general, applications should keep track of the next expected sequence number to detect potential data gaps.

If a sequence number matches the next expected sequence number, the packet is new and should be processed.

If a sequence number is lower than the next expected sequence number, the packet has already been processed and should be discarded.

If a packet is received with a sequence number higher than expected, it is an indication that data loss has occurred and the instrument's feed needs to be recovered.

Snapshots and Feed Recovery

Whether performing a late join, or recovering from a data gap, applications can request a full market depth snapshot by sending an Instrument Snapshot Request to the Market Snapshot Service. The IP address and port for the service can be found in reference data for a given environment.

The instrument snapshot response returned from the service contains a collection of resting bid and ask orders that are in the market as of sequence number returned in the response.

Note that the orders are transmitted based on how adventageous they are, ie. how far they are from the inside market. Their ordering matches their position in the matching engine's execution queue.

Reconciliation

Before requesting a full-depth snapshot, the client application should first join the multicast groups and start recording any incremental instrument updates into memory.

After the instrument snapshot response is successfully returned by the Market Snapshot Service, it will contain an as of sequence number. Applications should clear their in-memory order book structure for the instrument and bulk insert the orders which arrived in the snapshot. Additionally, consuming applications should traverse the cached incremental updates and discard any update whose sequence number is lesser or equal to the one returned within the instrument snapshot response.

Incremental updates with sequence numbers higher than that of the snapshot's should then be normally applied to the order book. Once completed, the instrument's feed is up to date.

PitchFork Protocol Specification

The following sections describe Seed CX binary market data protocol, known as PitchFork Seed CX reserves the right to change and evolve the PitchFork protocol on an as-needed basis. Rest assured, best efforts will be taken to preserve the existing wire layout.

That said, in an effort to provide transparency to potential future updates, the below are examples subject to our consideration:

  1. Reserved fields may be re-purposed to hold public data;
  2. The header and message lengths may be increased;
  3. New message types may be introduced.

Data Types

The following field types are used within the PitchFork binary market data protocol. They appear in little endian byte order unless specifically stated otherwise.

Byte

Byte is the collection of 8 bits. This data type is primarirly used to represent an opaque sequence of data, such as space reserved for future protocol versions or for padding.

Numeric

Numerics are unsigned integral values expressed in little endian byte order. Typically 1, 2, 4 or 8 bytes long.

Price

Prices are encoded as 64-bit signed integers in little endian byte order. They have an implied decimal point, which varies from instrument to instrument. Use instrument reference data to obtain the number of places after the decimal point.

Quantity

Quantities are encoded as 64-bit unsigned integers in little endian byte order. They are usually used for representing order sizes. Please take note of the contract size from the instrument's reference data.

Instrument Identifier

Instrument Identifiers are encoded as 64-bit unsigned integers in little endian byte order. They may change with each trading session. Use instrument reference data for a given environment to obtain instrument identifiers.

Order and Execution Identifiers

Order Identifiers, as well as Execution Identifiers, are encoded as 128-bit binary objects. They can be printed as integers in little endian byte order, or as base58-encoded alphanumeric strings.

Timestamp

Timestamps are encoded as 64-bit unsigned integers in little endian byte order. They represent the number of nanoseconds since the start of UNIX Epoch (1970-01-01 00:00:00 UTC).

Text

Text is represented as a sequence of ASCII-encoded characters. If the text is fixed length, any excess space must be filled with byte value of 0.

UDP Multicast Messages

Seed CX distributes incremental market data updates using UDP frames on redundant A and B multicast channels.

All UDP frames follow this structure:

UDP wire message structure

Packet Header

Every PitchFork market data UDP packet will begin with this packet header. While multiple messages may be combined into a single packet to save bandwidth, they will always pertain to the same instrument.

Name Type Byte Length Description
Total Length Numeric 2 Total length of all payloads in the packet, including this header
Packet Header Length Numeric 2 The length of the packet header.
Be sure to honor the length fields included in headers, in case the length increases between protocol versions.
Protocol Version Numeric 1 Version of the market data protocol.
Reserved Byte 1 Reserved for future use.
Message Count Numeric 2 Number of messages that are included in this packet, and immediately follow the Packet Header
Instrument ID Instrument Identifier 8 Seed CX Instrument ID, defined in reference data feed.
Sequence Number Numeric 8 Sequence number of message immediately following the packet header.
Sending Time Timestamp 8 Time of packet transmission nanoseconds since UNIX epoch UTC.
Reserved Byte 24 Reserved for future use.

Sequence Number

As described in the reading incremental updates from multicast section, the next expected sequence number can be calculated by adding the message count to packet header's sequence number.

Heartbeat

A packet header with a message count of zero represents a heartbeat. Heartbeats never increment the sequence number, and their sequence effectively represents the next expected sequence number.

Heartbeat messages are sent if there was no activity for a given instrument for at least 1 second.

Message Header

Message Header describes the type of message that follows it. Please note that the header size may grow in the future protocol versions. Applications should ensure that they read lengths specified in headers.

Name Type Byte Length Description
Message Header Length Numeric 2 Length of this header.
Be sure to honor the length fields included in headers, in case the length increases between protocol versions.
Message Length Numeric 2 Length of the message, excluding this header.
Message Type Numeric 1 Known message types:
0 Clear Book
1 Add Order
2 Replace Order
3 Delete Order
4 Trading Status
5 Trade
6 Trade Break
7 Session End

Note: More message types may be added in the future. See a section on handling unknown messages.

Reserved Byte 27 Reserved for future use.

Unknown Messages

The Seed CX Market Data protocol is designed for extensibility. In addition to the message types currently specified in the Message Header, more messages may be added to the protocol in the future. Every message sent is always preceded by the Message Header, which list the message type and size. When a client application does not know how to handle a particular message type, it should simply ignore that message by skipping over its bytes. It is important to note that all messages, handled and unhandled, do increment the sequence number.

Add Order Message

An Add Order message represents a newly accepted visible order on the Seed CX book. It includes a session-specific Order ID which will subsequently be used to refer to this order.

Name Type Byte Length Description
Order ID Order Identifier 16 Unique order identifier for the current trading session.
Price Price 8 Order price.
Size Remaining Quantity 8 Order size resting in the book.
Side Numeric 1 Possible values:
0 Bid
1 Ask
Reserved Byte 7 Reserved for future use

Replace Order Message

Replace Order messages represent a modification of the order state. They refer to an Order ID previously sent via Add Order, Instrument Snapshot, or a Replace Order message.

Orders whose remaining size reaches zero should be removed from the book.

Name Type Byte Length Description
Original Order ID Order Identifier 16 Previously assigned unique order identifier.
New Order ID Order Identifier 16 New unique order identifier which will be used to refer to this order.
Price Price 8 Order price.
Size Quantity 8 Remaining order size.
Lost Priority Numeric 1 Possible values:
0 Did not lose priority
1 Lost priority
Reserved Byte 7 Reserved for future use

Delete Order Message

Delete Order messages are sent when an order is removed from the order book. This may occur due to a cancellation or when an order has been completely filled.

The Order ID field refers to an order identifier previously sent via Add Order, Instrument Snapshot, or a Replace Order message.

Name Type Byte Length Description
Order ID Order Identifier 16 Unique order identifier of the order being deleted.

Trade Message

Trade messages are sent whenever an order on Seed CX has been executed in whole or in part.

Applications that simply build a full depth order book can safely ignore Trade Break messages.

Name Type Byte Length Description
Execution ID Execution Identifier 16 Execution identifier for the trade.
Price Price 8 Execution price.
Size Executed Quantity 8 Executed order size.
Reserved Byte 16 Reserved for future use.

Trade Break Message

The Trade Break message is sent whenever an execution on Seed CX is busted. Trade breaks are very rare. Trade Break immediately followed by a new Trade Message with the same Execution ID indicates a trade correction.

Again, applications that simply build a full depth order book can safely ignore Trade Break messages.

Name Type Byte Length Description
Execution ID Execution Identifier 16 Execution identifier of the broken/cancelled trade

Trading Status Changed Message

Trading Status Changed messages are sent whenever an instrument's trading status is updated. This may happen due to the regular exchange schedule, or unforeseen events such as a market halt.

Name Type Byte Length Description
Trading Status Numeric 1 Possible values:
0 Closed
1 Available
2 Opening Auction
3 Open
4 Pre-Closed
5 Halted
Reserved Byte 7 Reserved for future use

Clear Book Message

This message type is sent to indicate that the order book for this instrument should be cleared. All cached orders need to be purged. A Clear Book message is usually sent at the beginning and the end of the trading session, as well as during a system fail-over.

The message body is empty.

Session End Message

This message type is sent when an instrument's trading session has ended. Reset an instrument's sequence numbers to zero.

The message body is empty.

TCP Binary Snapshot Protocol

A full list of an instrument's resting orders can be obtained by establishing a TCP connection to the Market Snapshot Service and sending an Instrument Snapshot Request. This section describes the snapshot request as well as the possible responses.

Note that snapshots are transmitted using TCP, which is a stream-oriented protocol. Applications should pay attention to lengths specified within headers and messages to ensure successful data processing.

Instrument Snapshot Request

This message should be transmitted by client applications in order to request a full depth market snapshot for a given instrument.

Please note that snapshot requests are rate-limited. The current limit is 10 requests per participant per second.

Name Type Byte Length Description
Length Numeric 2 Length of the request.
Message Type Numeric 1 20 Snapshot request
Protocol Version Numeric 1 Protocol version. Must be set to 2.
Sender Comp ID Text 12 Seed-assigned SenderCompId used for authentication and rate limiting. It must be left-justified and padded with 0x00 to the right.
Instrument ID Instrument Identifier 8 Seed Instrument ID.

Snapshot Response Header

The snapshot response header is transmitted as part of any reply by the Market Snapshot Service. It identifies and describes the response message.

Name Type Byte Length Description
Header Length Numeric 2 Length of this header.
Message Length Numeric 2 Length of the message that follows this header. Message Length does not include the header nor any data that may follow the message.
Protocol Version Numeric 1 Version of the market data protocol.
Message Type Numeric 1 Possible values:
21 Snapshot Failed Response
22 Snapshot Success Response
Reserved Byte 2 Reserved for future use.
Sending Time Timestamp 8 Time of message transmission. Nanoseconds since UNIX epoch UTC.
Reserved Byte 24 Reserved for future use.

Snapshot Failed Response

Instrument snapshot request has failed. Depending on the failure reason, application may choose to re-try.

Name Type Byte Length Description
Instrument ID Instrument Identifier 8 Seed Instrument ID.
Reason Numeric 1 Possible values:
0 Malformed Request
1 Invalid Instrument ID
2 Snapshot Not Available
3 Invalid Credentials
4 Quota Exceeded
5 Unsupported protocol
Reserved Byte 7 Reserved for future use

Snapshot Success Response

The snapshot request was successful, and the response contains the as of sequence number as well as the current trading status for the instrument. The response message also contains length of an individual Add Order message and the count of Add Order messages which follow the response message. Applications can multiply the message length by the number of messages to determine the number of bytes to be read from the TCP stream.

The Add Order messages will be sorted from most advantageous to least advantageous. Their ordering reflects their position within matching engine's execution queue.

Name Type Byte Length Description
Instrument ID Instrument Identifier 8 Seed Instrument ID.
Sequence Number Numeric 8 As of sequence number.
Trading Status Numeric 1 Possible values:
0 Closed
1 Available
2 Opening Auction
3 Open
4 Pre-Closed
5 Halted
Reserved Byte 1 Reserved for future use.
Order Message Length Numeric 2 Length of a single Add Order message.
Order Count Numeric 4 Number of Add Order messages to follow this response. Note the Add Order messages will be sorted from most advantageous to least advantageous.

Changes since previous version

The wire layout remains largely the same as the previous version. However, there were some changes to header and data type sizes. Please see the details below.

Version number

The new protocol version is 2. It occupies the same spot in the wire messages (except the snapshot messages), so applications coded to older versions should be able to detect it.

The Market Snapshot Service will now only process requests whose protocol version is set to 2.

Lengths in headers

Rather than using hard-coded message sizes, the lengths of messages and headers themselves are now explicitly specified in headers. This makes data parsing easier, and allows for future expansion of messages.

Applications should always honor lengths specified in headers instead of making assumptions about data length.

More padding

Several messages and headers have been padded to improve alignment. The padding is reserved and should be read - but ignored by the client application. Padded areas may be re-purposed in the future.

128-bit order and execution identifiers

Previously, the identifiers of orders and executions were encoded as 64-bit unsigned integers. This has been changed in order to improve uniqueness and ensure room for future expansion.

Order and execution identifiers are now 128-bit opaque binary objects. For backwards compatibility and for display purposes, they can still be interpreted as 128-bit unsigned integers; Seed CX reserves the right to switch the display format to UUIDs or base58-encoded UUIDs in the future.

Simplified snapshot recovery

Instead of a unified total payload length field, the Snapshot Response Header now contains two separate fields: header_length and message_length. This makes reasoning about response sizes and order counts much easier.

Similarly, the Snapshot Success Response now contains a new field with order message length. The order message length can be multiplied by the number of orders to obtain byte size of the trailing order messages. This greatly simplifies reading snapshot responses from the TCP stream.

Resources

Third party libraries

Below are links to unofficial, third-party client libraries which decode PitchFork market data format.