The Stock Market May Have Grashed 18,000 Times Since 2006...

N&P: Discuss governments, nations, politics and recent related news here.

Moderators: Alyrium Denryle, Edi, K. A. Pital

Post Reply
User avatar
Zaune
Emperor's Hand
Posts: 7551
Joined: 2010-06-21 11:05am
Location: In Transit
Contact:

The Stock Market May Have Grashed 18,000 Times Since 2006...

Post by Zaune »

... but nobody noticed.

The Atlantic
What if someone told you the stock market crashed and spiked 18,000 times since 2006, and you had no idea?

That’s the contention of a group of scientists who study complex systems after analyzing market data, collected by Nanex, since the advent of high-speed trading. While the fallout of computerized algorithms has been seen before, including the infamous 2010 “flash crash,” when markets lost nearly 10% of value in just a few minutes, that same kind of sudden volatility is going on all the time, unseen.

In a new paper called “Abrupt rise of new machine ecology beyond human response time,” researchers found a new trading ecosystem that humans don’t even notice.

People can’t really respond to stimuli much faster than in one second. The benchmark comes from cognitive scientists who find that it takes 650 milliseconds for a chess grandmaster to realize that a king has been put in check after a move. Below that time period, you can find “ultrafast extreme events,” or UEEs, in which trading algorithms cause prices to change by 0.08% or more before returning to human-time market prices. This appears to be the case when many simple algorithms, operating on limited information, pile into a single trade.

“Down in the sub-second regime, they are the only game in town,” University of Miami Physics Professor Neil Johnson, who led the study, says. “It’s almost like you’re seeing them in pure form.”

This chart shows what an UEE crash looks like (box A), what a spike looks like (box B), and most interestingly, how the number of these events (in red and blue) has risen between 2006 and 2011 compared with the S&P index (in black). That list of stock symbols in green contains the equities that have the most extreme events, with the most likely at the bottom:

Image

If you’ve noticed that the number of extreme events spikes around the time of the financial crisis, and the stocks most likely to experience them are bank stocks, you’ll see why the researchers are so interested in this hidden market: This pattern suggests the coupling between extreme market behaviors and global instability—”how machine and human worlds can become entwined across timescales from milliseconds to months”—and is also are seen more often before and after the kinds of “flash crashes” that people actually notice.

Regulators, though, aren’t keeping track of these events. That’s a problem, not just because of any potential forewarning, but also because trading at that speed creates volatility that makes markets less efficient.

“Are these 18,000 lucky breaks for one of the algorithms or 18,000 examples of a new form of inside trading?” Johnson says. “In terms of the information availability, it’s really hard to tell. It’s sort of strange to have that going on and have nobody know.”

The researchers say there’s much more to learn, especially at the border where human traders and robotic ones interact. One question is whether moving at computer speeds is inefficient because there’s less information available at that time scale—data just can’t move that fast, even electronically. Laboratory experiments suggest computers are more efficient on a human time-scale than a sub-second one. And if sub-second trading does continue, do market participants need to come up with sub-second hedges and derivatives to protect from this kind volatility?

Regardless, the complexity emerging naturally from high-frequency trading tends to be hard to comprehend for market participants and regulators alike.

“It’s sort of a collective, in some sense they all share responsibility and yet nobody’s responsible,” Johnson says. “Am I responsible for the traffic jam out on US 1? No, I’m just in it, but if no one was in it, there wouldn’t be one.”
I'm rethinking my stance against classical Marxism right now...
There are hardly any excesses of the most crazed psychopath that cannot easily be duplicated by a normal kindly family man who just comes in to work every day and has a job to do.
-- (Terry Pratchett, Small Gods)


Replace "ginger" with "n*gger," and suddenly it become a lot less funny, doesn't it?
-- fgalkin


Like my writing? Tip me on Patreon

I Have A Blog
User avatar
Esquire
Jedi Council Member
Posts: 1583
Joined: 2011-11-16 11:20pm

Re: The Stock Market May Have Grashed 18,000 Times Since 200

Post by Esquire »

This particular thing is more on the 24-hour news cycle and automatic trading programs than on capitalism itself, I think.
“Heroes are heroes because they are heroic in behavior, not because they won or lost.” Nassim Nicholas Taleb
User avatar
K. A. Pital
Glamorous Commie
Posts: 20813
Joined: 2003-02-26 11:39am
Location: Elysium

Re: The Stock Market May Have Grashed 18,000 Times Since 200

Post by K. A. Pital »

Capitalism is innovation; automated trading is simply the answer to a particular demand in society.
Lì ci sono chiese, macerie, moschee e questure, lì frontiere, prezzi inaccessibile e freddure
Lì paludi, minacce, cecchini coi fucili, documenti, file notturne e clandestini
Qui incontri, lotte, passi sincronizzati, colori, capannelli non autorizzati,
Uccelli migratori, reti, informazioni, piazze di Tutti i like pazze di passioni...

...La tranquillità è importante ma la libertà è tutto!
Assalti Frontali
Grumman
Jedi Council Member
Posts: 2488
Joined: 2011-12-10 09:13am

Re: The Stock Market May Have Grashed 18,000 Times Since 200

Post by Grumman »

And if sub-second trading does continue, do market participants need to come up with sub-second hedges and derivatives to protect from this kind volatility?
No, and if the author isn't an idiot he'll see why. A momentary spike is in itself harmless - just wait another millisecond and the price will go back to normal. If it is going to cause any damage, it is because someone blew it out of proportion - such as by gambling on sub-second hedges and derivatives, or panic selling based on an insignificant hiccup.

And who thought graphing the cumulative number of UEEs was a good idea? The whole point is that they don't accumulate!
Simon_Jester
Emperor's Hand
Posts: 30165
Joined: 2009-05-23 07:29pm

Re: The Stock Market May Have Grashed 18,000 Times Since 200

Post by Simon_Jester »

It'd be better to graph d(UEEs)/dt, yes.

I will say that frequent UEEs are a sign that something screwy is going on, they reflect bugs in our system for assigning values to stocks. The bugs may appear harmless, but on the other hand they may be a "harmless" symptom of a more serious issue. I mean, suppose you inexplicably turn green every second Tuesday. It's harmless, purely cosmetic and periodic, but even if there are no other symptoms, something is not right with the state of your health.
This space dedicated to Vasily Arkhipov
User avatar
Zaune
Emperor's Hand
Posts: 7551
Joined: 2010-06-21 11:05am
Location: In Transit
Contact:

Re: The Stock Market May Have Grashed 18,000 Times Since 200

Post by Zaune »

I'm more concerned with the fact that we're letting people get stupidly rich from this... this glorified videogame that seems to be almost entirely disconnected from the real world. And it's not even the guys who design and build the fucking computer systems who are walking away with most of the cash anyway.
There are hardly any excesses of the most crazed psychopath that cannot easily be duplicated by a normal kindly family man who just comes in to work every day and has a job to do.
-- (Terry Pratchett, Small Gods)


Replace "ginger" with "n*gger," and suddenly it become a lot less funny, doesn't it?
-- fgalkin


Like my writing? Tip me on Patreon

I Have A Blog
Grumman
Jedi Council Member
Posts: 2488
Joined: 2011-12-10 09:13am

Re: The Stock Market May Have Grashed 18,000 Times Since 200

Post by Grumman »

Simon_Jester wrote:I will say that frequent UEEs are a sign that something screwy is going on, they reflect bugs in our system for assigning values to stocks. The bugs may appear harmless, but on the other hand they may be a "harmless" symptom of a more serious issue. I mean, suppose you inexplicably turn green every second Tuesday. It's harmless, purely cosmetic and periodic, but even if there are no other symptoms, something is not right with the state of your health.
This I agree with. If I had to make a wild stab-in-the-dark guess, I'd say these fluctuations are a result of a hundred programs making the same decision to trade at the same time based on the same information, and then ninety-nine of those attempted trades being cancelled because there's not enough to go around. But I might be wrong, and it's definitely worth finding out for certain.
User avatar
Starglider
Miles Dyson
Posts: 8709
Joined: 2007-04-05 09:44pm
Location: Isle of Dogs
Contact:

Re: The Stock Market May Have Grashed 18,000 Times Since 200

Post by Starglider »

'That humans don't even notice' is hyperbole. Analysts and developers spend a great deal of time analysing FIX message logs and strategy traces. Institutional investors don't notice the specifics but then generally they don't care, because standard (textbook / naive) pricing models already assume that price movement is (essentially) pink noise. Really this is just a comparison error by naive observers; instantaneous mid price inside an execution engine where depth replenishment is constrained by the speed of light is hardly the same thing as global market clearing price. Admittedly, everyone being so cagey with leaking information and iceberging to the hilt doesn't help with that, but it's just not that big a problem assuming you are doing major volume (sucks for small investors who use naive stops though). Quote stuffing and other abusive HFT gets the headlines, and it's true that some HFT shops outright bribe exchanges to allow them to frontrun (it's annoying that end users can't co-operatively pressure brokers enough to get exchanges to stop doing this). However most automated trading is in fact arbing away mispricings and providing shallow liquidity, with the net effect of minimising spreads and reducing some of the older (and larger) forms of rent seeking by banks & market makers.

Frankly I would rather the money was spent on GPU grids and 100 gigabit optical switches vs going solely on trader/quant/sales compensation. The former helps bring down the cost of supercomputing technology (early adoption & economies of scale) and supports fundamental work in computer science (the reason why I am working in finance). In fact the recent regulator push to central clearing and transparent brokers has only accelerated the automated trading trend; it is eating further and further into exotic and structured products that were last bastions of seat-of-the-pants trading and price-from-personal-spreadsheet. Latencies used to be a secondary concern in options but they're rapidly becoming as important as for spot. Fundamentally at the institutional scale is no reason why humans should be trading; they're slow, expensive and prone to errors, emotions and rogue trading.
User avatar
Arthur_Tuxedo
Sith Acolyte
Posts: 5637
Joined: 2002-07-23 03:28am
Location: San Francisco, California

Re: The Stock Market May Have Grashed 18,000 Times Since 200

Post by Arthur_Tuxedo »

Starglider wrote:'That humans don't even notice' is hyperbole. Analysts and developers spend a great deal of time analysing FIX message logs and strategy traces. Institutional investors don't notice the specifics but then generally they don't care, because standard (textbook / naive) pricing models already assume that price movement is (essentially) pink noise. Really this is just a comparison error by naive observers; instantaneous mid price inside an execution engine where depth replenishment is constrained by the speed of light is hardly the same thing as global market clearing price. Admittedly, everyone being so cagey with leaking information and iceberging to the hilt doesn't help with that, but it's just not that big a problem assuming you are doing major volume (sucks for small investors who use naive stops though). Quote stuffing and other abusive HFT gets the headlines, and it's true that some HFT shops outright bribe exchanges to allow them to frontrun (it's annoying that end users can't co-operatively pressure brokers enough to get exchanges to stop doing this). However most automated trading is in fact arbing away mispricings and providing shallow liquidity, with the net effect of minimising spreads and reducing some of the older (and larger) forms of rent seeking by banks & market makers.

Frankly I would rather the money was spent on GPU grids and 100 gigabit optical switches vs going solely on trader/quant/sales compensation. The former helps bring down the cost of supercomputing technology (early adoption & economies of scale) and supports fundamental work in computer science (the reason why I am working in finance). In fact the recent regulator push to central clearing and transparent brokers has only accelerated the automated trading trend; it is eating further and further into exotic and structured products that were last bastions of seat-of-the-pants trading and price-from-personal-spreadsheet. Latencies used to be a secondary concern in options but they're rapidly becoming as important as for spot. Fundamentally at the institutional scale is no reason why humans should be trading; they're slow, expensive and prone to errors, emotions and rogue trading.
That's all well and good, and I am in favor of replacing shoddy human decision making with machine logic in general, but what need is being served by having so many trades executed in a second? It seems to me that HFT simply rigs the game against individuals and small fund traders and increases volatility (ex. the flash crash of 2010), creating no wealth (perhaps even destroying a small amount) but instead transferring existing wealth to the top of Wall Street's food chain. Do you know of any evidence that there is a correlation between increased HFT (not automated trading in general) and the markets having become more efficient? If not, it strikes me as a form of legalized theft that should be stopped.
"I'm so fast that last night I turned off the light switch in my hotel room and was in bed before the room was dark." - Muhammad Ali

"Dating is not supposed to be easy. It's supposed to be a heart-pounding, stomach-wrenching, gut-churning exercise in pitting your fear of rejection and public humiliation against your desire to find a mate. Enjoy." - Darth Wong
User avatar
Terralthra
Requiescat in Pace
Posts: 4741
Joined: 2007-10-05 09:55pm
Location: San Francisco, California, United States

Re: The Stock Market May Have Grashed 18,000 Times Since 200

Post by Terralthra »

Starglider wrote:'That humans don't even notice' is hyperbole. Analysts and developers spend a great deal of time analysing FIX message logs and strategy traces. Institutional investors don't notice the specifics but then generally they don't care, because standard (textbook / naive) pricing models already assume that price movement is (essentially) pink noise. Really this is just a comparison error by naive observers; instantaneous mid price inside an execution engine where depth replenishment is constrained by the speed of light is hardly the same thing as global market clearing price. Admittedly, everyone being so cagey with leaking information and iceberging to the hilt doesn't help with that, but it's just not that big a problem assuming you are doing major volume (sucks for small investors who use naive stops though). Quote stuffing and other abusive HFT gets the headlines, and it's true that some HFT shops outright bribe exchanges to allow them to frontrun (it's annoying that end users can't co-operatively pressure brokers enough to get exchanges to stop doing this). However most automated trading is in fact arbing away mispricings and providing shallow liquidity, with the net effect of minimising spreads and reducing some of the older (and larger) forms of rent seeking by banks & market makers.

Frankly I would rather the money was spent on GPU grids and 100 gigabit optical switches vs going solely on trader/quant/sales compensation. The former helps bring down the cost of supercomputing technology (early adoption & economies of scale) and supports fundamental work in computer science (the reason why I am working in finance). In fact the recent regulator push to central clearing and transparent brokers has only accelerated the automated trading trend; it is eating further and further into exotic and structured products that were last bastions of seat-of-the-pants trading and price-from-personal-spreadsheet. Latencies used to be a secondary concern in options but they're rapidly becoming as important as for spot. Fundamentally at the institutional scale is no reason why humans should be trading; they're slow, expensive and prone to errors, emotions and rogue trading.
Not being too far into the HFT algorithm world, could you define a few of those terms for me? I know what pink noise is, but iceberging, naive stops, quote-stuffing, frontrunning, and arbing are unknown to me.
User avatar
salm
Rabid Monkey
Posts: 10296
Joined: 2002-09-09 08:25pm

Re: The Stock Market May Have Grashed 18,000 Times Since 200

Post by salm »

A stop or stop loss is a price you set at which the your broker automatically sells your stocks/currency/whatever in order to prevent too much losses. If the price falls bellow (or raises above in case of a short position) your set price it is automatically sold. It´s supposed to prevent you from losing money but if you set a stop naively you can lose money for no good reason.
That´s usually the case if you set the stop loss too high. The price of you stock drops bellow your stop loss for a very short time due to a spike because of noise. Your stock is automatically sold. After that the price goes up again to a "normal" level. So you´ve sold your stock at a low price because your stop loss was too high.

arbing -> from arbitrage: you use the price differences of the same thing in two differnet markets buy it in one market and directly afterwars sell it in the other market.
User avatar
Starglider
Miles Dyson
Posts: 8709
Joined: 2007-04-05 09:44pm
Location: Isle of Dogs
Contact:

Re: The Stock Market May Have Grashed 18,000 Times Since 200

Post by Starglider »

Terralthra wrote:Not being too far into the HFT algorithm world, could you define a few of those terms for me? I know what pink noise is, but iceberging, naive stops, quote-stuffing, frontrunning, and arbing are unknown to me.
Iceberging is not a HFT specific term, its general to all large orders. For any given security, the order book on an exchange consists of certain quantities ready to buy or sell at various prices away from the quoted (mid) price. This is the 'market depth'; the instantaneous liquidity available. As you try to buy/sell more and more in one go, you exhaust the orders closer to the mid and have to match against orders further and further away; more expensive offers if you are buying, cheaper bids if you are selling. This is the 'cost of liquidity'; just crossing the bid-offer spread for small orders, but increasing for larger orders. For very large orders, there may not be enough depth on the exchange to execute at any price; you'd have to do as much as you can then wait for people to put more liquidity on. Naturally since you are demonstrating strong demand this will move the price against you. Even if you don't want to execute right now, and simply put your big order on at the bid/ask and wait for people to hit you, you will still move the price against you. Other participants will see the demand, assume the instrument is worth more/less and start trading away.

Thus for large orders either the customer specifies an 'iceberg' or leaves it to the best judgement of their broker. This is the amount of liquidity to show on the exchange at any one time; the large trade is split up into numerous small trades which the customer hopes will be lost in the general trading volume. This reduces the chance of moving the market and hence improves the price. In fact these days when you do see massive orders on an open broker or exchange, particularly when there isn't an obvious move that would trigger hard stops, it is usually a sign of someone actively trying to manipulate the market. This happens constantly in commodities; it generally doesn't in major currencies since the liquidity is so high, but can happen in EM currencies. Iceberging orders takes longer and still leaks information, thus the existence of dark pools and a fair amount of OTC (over the counter; bilateral, not on an exchange) trading just to avoid tipping others off about your intentions.

A naive stop is what salm just described; a stop-loss order with a trigger set on simple last reported mid price. If you seriously set one of those on an open exchange without direct market access and close monitoring, you are asking to be picked off (traded against at a retrospectively bad price). The world has moved on and such simplistic rules are only going to get you in trouble. Decent modern brokers offer direct market access (low latency connections and/or hosted algo execution engines) that allow more sensible stops based on time-averaged and/or multi-venue pricing.

Quote stuffing is essentially a denial of service attack on exchange infrastructure, although sometimes it is just very short term speculation against incoming data from other exchanges. Algos constantly post and then cancel orders (a few milliseconds later); either tradeable (would match against the existing book if allowed to complete) or non-tradeable, both have their uses. Sometimes this is just to slow everything down and force discrepencies in pricing (between instruments e.g. future/option and future/underlying arb and between the same instrument at different venues), which can be exploited to earn a profit. Sometimes it takes advantage of the fact that older orders have preference and that execution is queued; effectively reserving lots of places in the queue in case you actually need them, or forcing competitors to wait so long that the market has moved against them by the time their order gets to the matching engine. Quote stuffing is essentially a 'tragedy of the commons' effect as exchange participants aren't billed for quotes. Fortunately it has been reducing steadily for the last couple of years as everyone else wakes up and realises the HFT funds were pissing on them.

Frontrunning is again not HFT specific, essentially any time you know that someone is trying to execute a large order, you can buy whatever they are trying to buy (or sell whatever they are trying to sell if you have it or can borrow it), then trade to them at a more advantageous price and capture the spread. For example the iceberging described above, you are hoping your orders will be lost in the noise, because if other market participants realise you want to buy something they will buy up all the liquidity then try to charge you a premium for it. Of course if the price moves too much you may no longer want to trade and the speculators will make a loss. HFTs frontrun very efficiently because they can work with small amounts of capital and spot patterns on much smaller timescales. This is merely annoying but the genuinely unfair part is that some HFT shops pay some exchanges to effectively jump other customers in the order queue, allowing them to frontrun almost without risk (technically the slower trader could cancel their orders while in the queue, but institutions generally don't and in fact can't do that). This is one of the few cases where a simple regulation would solve the problem and improve it for everyone.

Arbitrage is any situation where you execute a set of trades that lock in a riskless profit (greater than the cost of capital, execution fees and where applicable opportunity cost to be worth doing). There are lots of theoretical kinds of arbitrage you can do, all of which are hard to do in practice due to the real world not being a frictionless, reliable and predictable trading environment (although we are relatively close for coarse time scales). Hedge funds like to claim they are making profits from arbitrage because it makes them sound really smart (arbitrage is hard and complicated) and a sure bet (riskless profits), but they are usually major hidden risks. Credit risk if nothing else, as illustrated by the MF Global collapse.
Post Reply