The "COTs," as they are known, give data on trillions of dollars of futures and options holdings in over 90 markets - everything from crude oil to the U.S. dollar, gold and the NASDAQ. The reports are issued every week by the U.S. Commodity Futures Trading Commission. They can be downloaded for free from the CFTC's website each Friday at 3:30 p.m. (Eastern Standard Time).
But the data in its raw form is hard to make use of - or even understand. Each weekly report gives the total position held in each market by three groups of traders.
First are the commodity producers. Analysts call them the "smart money." Also known as commercial hedgers or just "the commercials," they are seen as the folks with the best market information. When they are increasing their positions in crude oil, for example, it's probably a good time to get some energy stocks.
Then there is the dumb money - the "non-commercial" category. These are the large speculative investment funds, usually called the "large speculators" or simply the "large specs" - or just the "dumb money." They are trend-followers. Analysts say they are usually wrong at market extremes. If they are buying oil, it's probably a good time to sell your energy stocks.
The third group is the "non-reportable" category. These guys are seen as the really dumb money. They are the small-time traders who apparently don't know what they're doing at all and don't provide any meaningful market information. Or so say most analysts.
WHAT TO DO?
To make any sense of all this data, analysts say, you have to compare each group's position in a market to what it held last week. So that's what I started to do.
I wrote down the weekly positions for all three groups for a couple of dozen markets - the S&P 500, the NASDAQ, crude oil, gold, silver, the U.S. dollar, the Yen, the 10-year Treasury bond and others.
But then I was stumped. What should I do with all this information? The price of gold, for example, rarely went up just because the commercials had bought more gold or because the large specs had sold it. And sometimes, the commercials and large specs were both buying the NASDAQ at the same time. What would I do then?
The data looked interesting, but there didn't seem to be any clear way to use it.
LOOKING FOR TRENDS
After doing this for a while, I started to download the data into Microsoft Excel and make charts out of it to see if I could spot trends. I still couldn't get much use out of it. I tried to find correlations between the COTs data and underlying cash prices. I couldn't find much of any use.
(One bizarre discovery I made was that some equity indexes have very high correlations with subsequent COTs data many months later. However intriguing, that didn't seem like very useful information for investing.)
To my knowledge - despite many studies by economists and statisticians - no one has found any correlations you can bank money on and made this research public. (If I'm wrong on that, let me know. I'd love to see the study.)
Just then, a couple of interesting books came out by COTs gurus Larry Williams (who wrote "Trade Stocks and Commodities with the Insiders") and Floyd Upperman (author of "Commitments of Traders"). Both analysts suggested the COTs were most useful when the commercials or large specs had accumulated historically extreme positions.
In other words, if the commercials have a multi-year high net long NASDAQ position, it's probably a good time to buy.
But when is an extreme position truly extreme? That's the real question. Commercials and large specs often accumulate record positions for weeks and even months, without any change in the underlying markets. Anyone relying only on such "signals" would probably lose lots of money.
The same problem exists with many technical analysis tools. A market can be overbought or oversold for weeks without a change in trend. You can be right, but if your timing is wrong, well, you won't be trading for long.
As a result, COTs analysts say the data can't be relied on solely to time trades. They suggest the COTs are most useful as guideposts for possible future market turns. But the data must be combined with technical analysis for a signal.
A RELEVATION
While I was on paternity leave with our second baby last summer, I had one of those revelations that sometimes comes when changing a poopy diaper or sitting in a rocking chair past midnight with a sleeping baby on your shoulder.
Why not use technical analysis to study the COTs? There must be a way, I thought, to prove once and for all if the COTs positions have any impact at all on subsequent cash prices. Could the tools of technical analysis help?
I looked again at the S&P 500 data going back to 1995 (when the COTs first came out for free in electronic form). I wanted to see what happened to the index after traders acquired an extreme net position. I defined "extreme" as two standard deviations or more from the 27-week moving average.
The results were pretty exciting. An extreme net position usually led to returns that beat the market. For example, if you bought the S&P 500 index when the commercials were at an extreme net long - or when the large specs or small traders were at an extreme net short - your returns over the next weeks and months were usually better on average than if you had just bought the market at any random moment.
For example, the return for the subsequent week was 0.6 percent, compared to the S&P 500's average one-week return of 0.2 since 1995. For the subsequent three weeks, the average return was 2.5 percent, compared to 0.5 percent for the S&P 500. Over the subsequent 10 weeks, the average return was 6.6 percent, compared to 1.7 percent for the S&P 500 since 1995.
The superior returns continued for up to 40 weeks after the extreme position was first registered - as far out as I measured.
And there were superior returns in most time frames for both the large specs and even the usually-ignored small traders.
What it meant was the data could be the basis for a trading system after all.
TIMING SYSTEM
There were still a bunch of questions to resolve. Which group of traders should I follow? What if they give contradictory signals? And I still needed to figure out the best way to measure an extreme position in the first place, since I had picked the two standard deviations and 27-week moving average arbitrarily. Were better results possible with another definition of "extreme"?
Using Excel, I started to test returns with various combinations of moving averages and standard deviations. I was stunned at the results. With many of the combinations, a simple switching system of buying the S&P 500 when the commercials were at an extreme net long position and selling when they were at an extreme net short would have given market-beating profits. The returns were in some cases more than double those of buying and holding the index.
The results were just as interesting fading - or trading opposite to - the large specs. But the biggest surprise was that the best results came from fading the small traders - the guys everyone was ignoring.
I found a way to automate the testing with help from a clever journalist colleague, Mike Gordon, who is an expert in computer-assisted reporting.
MARKET-BEATING
In every index or commodity I analyzed, I found that a market-beating trading system could be found. I discovered that it was possible to increase profitability, in some cases, by delaying the trade for one to four weeks after a signal was given.
Also, I found the most profitable results came from using not the net number of contracts of each group of traders, but rather their net-percentage-of-open-interest position. This makes sense, I believe, because of the explosion of futures and options trading in recent years. The net-percentage-of-open-interest numbers have probably preserved their internal consistency better during this time.
The best news for me is that the system requires only one or two trades a year in most of the markets. There can certainly be something exhilarating about swing trading or day trading. It feels like you can actually behold the entire world - all its people, their anxieties, their hopes, the events that shape their lives - in motion at once, like a raging and crashing ocean. It can be so absorbing, I find it's often a struggle to focus on anything else.
Now, I need no longer fret about daily market ups and downs. All I do is spend just a few minutes each week downloading the latest COTs data and executing any trades, then sit back and relax.