How I made $500k with machine learning and high frequency trading

A lot of people are stating that this is like gambling - it is - but not in the sense that you think. Firstly he doesn't use his entire bankroll on each trade, secondly he goes long-short consistently over very short periods of time, thirdly he's too tiny to actually move markets, and fourthly he is in and out within a day - where his max var. is 100% on thousands of tiny trades. Think of it like this - he has $100 - he bets $10 of that each day. He can blow that $10 - no problems.

Worst case he runs out of capital over a period of weeks.

He can't blow up in the way that you think - but he can have large drawdowns over a period of weeks.

Markets are eventually consistent scalable systems - and that is why we prefer them over central planning. In the medium term they prices things correctly, cheaply and efficiently (decade+).

In the short term however (sub-decade) - they can't price jack.

Markets are inefficient period - if they weren't, well then P=NP and you could just put your NP-hard problems into a market and get back cheap, quick, accurate results. Oh - wait - protein folding is actually harder than that.

There are 2 major ways to make money in the markets. Value-Growth and statistical arb (often high frequency). The former (Buffett) is highly concentrated bets on the future of business (I'm value - long TSLA/GOOG/Samsung). The latter is looking for thousands of small diversified statistically significant correlations above 50% (random guesses) and trading costs between securities/price movements over short time intervals (aka statistical ghosts in the data - RenTech/Shaw).

Both work. Both work well. And will continue to do so as long as markets exist.

I don't have much experience with finance or working experience with machine learning, but I've always wondered how much room there was for a clever amateur to profit in this space, even as it's crowded with much more sophisticated professionals with much more sophisticated algorithms and machines.

I'm thinking back to Garry Kasparov's piece in the NY Book Review a couple years back: http://www.nybooks.com/articles/archives/2010/feb/11/the-chess-master-and-the-computer/?pagination=false ↗

He talks about a chess tournament in which it was "anything goes"...competitors could be human, computers, or humans with computers. The expected outcome was that a grandmaster using a Deep Blue-like computer would win, but the winners ended up being a couple of amateurs with three computers:

The surprise came at the conclusion of the event. The winner was revealed to be not a grandmaster with a state-of-the-art PC but a pair of amateur American chess players using three computers at the same time. Their skill at manipulating and “coaching” their computers to look very deeply into positions effectively counteracted the superior chess understanding of their grandmaster opponents and the greater computational power of other participants. Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.

So in HFT, how much room is there for an amateur to profit over professionals by having a sophisticated process?

Great work, very interesting to me. Counter to what we're constantly told through the media this stuff can be done. Doing it year after year seems to be the elusive part. Intuitively, once you've proven your technique on 1000+ trades it's not luck.

I developed a fully-automated low-frequency stat arb system that I ran in 2007 based on a perhaps even simpler algorithm. It traded various equities equally to the long and short side regardless of market conditions so widespread rally or collapse was irrelevant. I logged about 20-30 trades/day - much slower.

The results, using no leverage, were +90% in a year with a worst drawdown of 2% and a Sharpe ratio of 2. Total trades were 5000+. Month-to-month the results were very consistent until the uptick rule was nixed in July 2007. August 2007 was a record winner for me, but Sept-Dec 2007 fell flat, not losing, but with greatly diminished profits and the same variation and more frequently getting slammed all-long or all-short instead of a mix that was often near-neutral. Also getting fills better than my orders then completely disappeared, as this was the beginning of the HFT middlemen - including your own brokerage. I shut it down at the start of 2008, keeping the profits intact and moving on to other priorities.

I continued to monitor the theoretical results for a couple of years but the conditions didn't return so I eventually cancelled my data feed.

Pls do not follow the advice of the OP. I started a hedgefund in 2004 doing HF platform arbitrage and ran it for 5yrs and i can honestly tell you that this is just survivorship bias. This is a very complex field and being off slightly, having a slight bias, a fraction of a point off your execution pricing and a slightly flawed money management system is recipe for disaster.

The biggest issue is the confusion that you can apply machine learning to HF trading. HF trading sub 15min mark is more about playing the deal flow, and only the institutions have an edge on this. This is why goldman had to separate the buy and sell sides in the early 2000's. Above 15mins you are able to find an edge using time series analyses since the market is scaling invariant according to Benoit Mandelbrot and this does not apply to dealflow. Also having access to dealflow allows you to predict volatilty seconds ahead which allows you decrease your risk and increase you reward as well as handle your costs since the volatility will impact your transaction costs even if transaction costs themselves stay the same. There is just so much stuff to cover that a comment will not do justice in explaining what is wrong with this guys logic.

Being pedantic, 4000 trades a day isn't HFT. This is stil algo trading, of which HFT is a subset.

I consider HFT to be any strategy where speed itself is the what gives the edge. Colocation is usually a prerequisite, though not sufficient. It's a shame HFT gets all the attention, when it's really a tiny portion of trading activity. Algo-trading in general is 70%+ of market activity in the US.

Also limiting trades isn't really adequate risk management. The tech exists to very accurately model your exposures. This is something I see underdeveloped a lot, and what separates the top trading firms from the rest.

Still I commend you creating a model, working out how to test and execute it automatically and actually trading your own money.

I really think more hackers should be actively managing their money, (in general, not like in the article). We have these amazing liquid markets, all time low spreads/commissions, products like ETFs/derivatives to accurately and cheaply execute a given strategy, and a huge increase in tech to model risk, but personal personal investing is the same as the 60s.