Page 1

Approaching Real-Time Business Intelligence Trading at the Speed of Light Sean McClure, Ph.D. Business Analytics, Excellerate Inc.

Overview Themes Introducing Excellerate Real-Time BI and HFT

Information at High Frequency Strategies at High Frequency


Data Mining Big Data

Developing and Deploying Models Meta Data

Executing and Monitoring Real-Time Systems Prediction


About Us!

Business Intelligence Service Providers • Dedicated to bringing top quality business intelligence expertise to successful growing organizations (SGOs);

• Aggressively researching industry best practices and best-in-breed software tools to deliver high-end analytics and data mining expertise; • Business Intelligence model supported by Subject Matter Experts (SMEs) in key business areas.

Real -Time Business Intelligence Defining “Real-Time” Three types of latency1: • Data latency: time taken to collect and store the data;

• Analysis latency: time taken to analyze the data and turn it into actionable information; and • Action latency: the time taken to react to the information and take action.

Approaching “zero” latency • Real-time business intelligence technologies are designed to reduce all three latencies as close to zero as possible;

• Traditional BI only seeks to reduce data latency.

Real -Time Business Intelligence Real Time BI in various industries Debit and Credit Fraud Detection Marketing

Inventory Control


Supply-chain Optimization Customer relationship management (CRM) Dynamic pricing and yield management Data validation Operational intelligence and risk management


Call center optimization Transportation industry Finance (biggest candidates)


High Frequency Trading (HFT) Best Case Study for “Real-Time” Intelligence

• Uses complex algorithms to analyze multiple markets and execute orders based on market conditions; • Traders with fastest execution speeds will be more profitable than traders with slower execution speeds (arbitrage opportunities).


Execution Latency

• Trading platform that uses powerful computers to transact a large number of orders at very fast speeds;

Traditional longterm Investing

Highfrequency Trading

Algorithmic/ electronic trading

Low Short

In the U.S., high-frequency trading accounts for ~73% of all equity trading volume5

Position Holding Period

Figure 1


High Frequency Information Properties of Tick Data – Quote, Trade, Price and Volume Information

Date/time quote originated (>20ms)

• Timestamp • Security ID • Bid Price • Ask Price • Available bid volume • Available ask volume • Last trade price • Last trade size • Option-specific data

Highest price available for sale of the security

Provided by other market participants through limit orders

Lowest price entered for buying the security

Total demand Total supply

Price at which the last trade in the security cleared Actual size of the last executed trade

High Frequency Information Recent microstructure research and advances in econometric modeling tell us there are unique characteristics to tick data. irregularly spaced quotes arriving randomly very short time intervals

Tick Data time

(low-frequency data is opposite) Irregularities provide a wealth of information not available in lowfrequency data. Inter-trade durations may signal changes in market volatility, liquidity, and other variables. Volume of data allows for statistically precise inferences. Number of observations in a single day of tick data = 30 years of daily observations

High Frequency Information Modeling the Arrivals of Tick Data creates a host of opportunities not available at low-frequency • Time distance between quote arrivals carries information time

quote processes trade processes price processes volume processes

Duration models Estimate the factors affecting the duration between ticks High Trade Duration Higher likelihood of unobserved bad news

Low Trade Duration Higher likelihood of unobserved good news Absence of Trade Lack of news, low levels of liquidity, trading halts, trader motivations

Low Price Duration Increased Volatility

Low Volume Duration Increased Liquidity

High Frequency Information Data sampling methods overcome irregularities in high-frequency data for ease of processing Traditional Approach

Minute 1

Minute 2

Minute 3

Linear Time-Weighted Interpolation

Minute 1

Figure A quote

ˆt  qt ,last q

Minute 2

Minute 3

Figure B

t  tlast tnexttlast

qˆt  qt ,last  (qt ,next  qt ,last )

Most modern computational techniques have been developed to work with regularly spaced data (easy to process) High frequency data-sampling methods developed to overcome irregularities in tick data by sampling at predetermined periods of time

High Frequency Information Security Price Adjustments to Information The price of the security in the inefficient market begins adjusting before/after the news becomes public ( “information leakage” and “overshooting”)

Many solid trading strategies exploit both the information leakage and overshooting to generate consistent profits Incorporation of information in efficient and inefficient markets Inefficient market response

Good News

Bad News

Efficient market response Information Arrival Time

Efficient market response

Information Arrival Time

Inefficient market response

High-Frequency Strategies Trading on High-Frequency Information Traders leverage state-of-the-art IT technology to implement trading strategies that have high-frequency opportunities; High-frequency trading strategies typically fall into four main categories. HFT-based Strategies

Electronic Liquidity Provision

Statistical Arbitrage

Liquidity Detection


Spread Capturing

Market Neutral Arbitrage

Sniffing/Pinging/ Sniping

Latency Arbitrage


Cross Asset, Cross Market & ETF

Quote Matching

Short Term Momentum

High-Frequency Strategies Liquidity Provision Strategies - Spread Capturing Liquidity providers profit from the spread between bid and ask prices by continuously buying and selling securities; Executed predominantly using limit orders Ask

Asking Price Market Buy Orders

Bid-Ask Spread

Market Sell Orders

Market Price Limit Buy Orders


Limit Sell Orders

Offer Price

High-speed transmission of orders and low-latency execution required for successful implementation of liquidity provision strategies.

Market Transactions

High-Frequency Strategies Predictions based on the Limit Order Book Direction of market price movement

• Shape of limit order book is predictive of impending changes in market price;

• Exploited by market-maker traders; buy


Direction of market price movement buy sell

• Depends on probability distribution for arriving market orders; • Shape can be estimated when book not observable.

High-Frequency Strategies Statistical Arbitrage “Stat-Arb� rests squarely on data mining. It finds statistical relationships in large amounts of data and builds a model of those relationships; Leverages states of the art technology to profit from small and short-lived discrepancies between securities; Arbitrageurs generate profits by selling the asset on the market where it is valued higher and simultaneously buying it on another market where it is valued lower.

High-Frequency Strategies Detecting Statistical Anomalies in Price Levels Once gap in prices reverse, close out position/stop loss Measure difference between prices of identified securities

Identify securities that trade in frequency unit

Sij ,t  Si,t  S j ,t ,t  1, T 

Monitor and act upon differences in security prices

St  Si ,  S j ,  ES   2 S 

St  Si ,  S j ,  ES   2 S 


1 T 2 distributional      St    S  E  S  t t properties of the T  1 t 1


Select most stable relationships

1 T ESt    St T t 1


min  S  T

i, j

t 1

ij ,t

High-Frequency Strategies Fundamental Arbitrage Strategies by Asset Class Asset Class

Fundamental Arbitrage Strategy

Foreign Exchange

Triangular Arbitrage

Foreign Exchange

Uncovered Interest Parity (UIP) Arbitrage


Different Equity Classes of the Same Issuer


Market Neutral Arbitrage


Liquidity Arbitrage


Large-to-Small Information Spillovers

Futures and the Underlying Asset

Basis Trading

Indexes and ETFs

Index Composition Arbitrage


Volatility Curve Arbitrage

Model Development/Deployment Model Development

Models used in HFT • Linear Econometric Models • Autoregressive (AR) Estimation • Moving Average (MA) Estimation • Autoregressive Moving Average (ARMA) • Cointegration Volatility Modeling • To model observed volatility clustering = ARMA or GARCH NonLinear Econometric Models Allows for modeling of complex nontrivial relationships in data • Taylor series expansion • Threshold autoregressive model • Markov switching model • Nonparametric estimation • Neural Networks

Ideas • Academic research and proprietary extensions

Tools • Modeling predominantly in Matlab /R, • c++ for back-tests and transition into production

Back Testing • Modeled relationships tested on lengthy spans of tick data • Forecasting validity • Various market situations

Model Development/Deployment Back-Testing Econometric Models

Point Forecasts

Directional Forecasts • makes decisions to enter into positions based on expectations of system going up or down (without target)

Accuracy Curves • compares the accuracy of probabilistic forecasts • HFT models done with TSA curves

Accuracy Curve Random Forecast 100 Model C

Model A Hit Rate (%)

• predict price will reach certain level /point • regression of realized values from historical data against out of sample forecasts

Model Accuracy Analysis

Model B 0.0 Miss Rate (%)

100 %

Executing Real-Time Systems Execution Optimization Algorithms • Algorithms spanning order-execution processes

• Designed to optimize trading execution once the buyand-sell decisions have been made elsewhere best way to route the order to the exchange best point in time to execute a submitted order (non-market order) best sequence of sizes in which the order should be optimally processed Common Types 1) 1) Market Aggressiveness Selection algorithms designed to choose between market and limit orders for optimal execution; 2) 1) Price-Scaling algorithms designed to select the best execution price according to the pre-specified trading benchmarks; and 3) 1) Size-optimization algorithms that determine the optimal ways to break down large trading lots into smaller parcels to minimize adverse costs (cost of market impact)

Executing Real-Time Systems Execution Optimization Algorithms Market Aggressiveness Selection • Balances passive and aggressive trading using optimization

min Cost ( )  Risk ( ) 

Cost ( )  Eo P( )  Pb  Risk ( )   ( ( )) P( )  P  f ( X ,  )  g ( X )   ( )


Benchmark execution price

Price-Scaling • Tries to obtain the best price for the strategy Strike Algorithm • Minimizes the cost of execution relative to a benchmark • Designed to capture gains in periods of favorable prices

min Et Pt 1 ( t )  Pb,t 



P (a ) Realized execution price Pt 1 ( t ) Realized price  (a ) Deviation of trading outcome  t Trading aggressiveness P Market price at order entry Pb ,t Benchmark price f ( X , a ) Temp impact due to liquidity Plus Algorithm g ( X ) Price impact due to info leak

Size Optimization

Wealth Algorithm

• Tries to trade with position undetected • Large order packets are broken up for least amount of market impact (“Stealth Trading”)

Executing Real-Time Systems HFT Business Cycle


1 – 4: run-time 5 – 6: post-trade

Receive/archive realtime tick data on securities of interest


6 Ensure trading costs incurred during execution are within acceptable ranges

5 Evaluate trading performance relative to predetermined benchmarks

Each functions built with independent alert systems that notify monitoring personnel of problems, unusual patterns etc.

Apply back-tested econometric models to the tick data obtained in 1

3 Send orders and keep track of open positions/P&L values


Monitor run-time trading behavior, compare with predefined parameters, manage the run-time trading risk

Summary Themes Introducing Excellerate


Real-Time BI and HFT Data Mining

Information at High Frequency

Strategies at High Frequency

Big Data

Developing and Deploying Models

Meta Data

Executing and Monitoring Real-Time Systems


Thank You Sean McClure, Ph.D. Business Analytics, Excellerate Inc.

References 1) Richard Hackathorn, "Active Data Warehousing: From Nice to Necessary," Teradata Magazine (June 2006), AR-4835 2) 3) 4) 5)

Sean McClure - Approaching Real-time Business Intelligence - Trading at the Speed of Light  
Read more
Read more
Similar to
Popular now
Just for you