Introduction
The market prices movement doesn't caused by the subjective Elliott waves, Gann lines and Fibonacci levels or by technical indicators. Nothing of it does contain any new information bits compared to price chart of OHLCV format. The price movement is determined by fundamental law of supply and demand. No matter how trivial it may seems, but demand and supply determine the price.
The price is a result of buyers' and sellers' interaction. Many traders know about it, but just a few use it. The practical application of this comprehensive market law can be extremely useful for anyone who have ever taken a dare to predict future prices. This article is a kind of "conductor" from theoretical postulates of the economy to its application for trading on the futures and commodity markets.
The economic theory states about the possibility of the existence of the "overbought" and "oversold" phases of the market. These phases are critical to the market. A further upward price movement in "overbought" zone is becoming more difficult because of increasing competition of the supply. Further downward price movement in the "oversold" zone is also becoming increasingly difficult, but because of the increasing pressure of demand. In both cases, the market is close to its long-term reversal. These statements may seem simple and so useless, because of the fact that economic theory has not proposed any methods to determine these market phases. However, these methods exist and, as it often happens, were developed a long time ago by successful traders.
This article is a bridge between the theory and practice, it introduces the reader with the methods of the market phase determination, based on simple indicators, described in the book
"Trade Stocks and Commodities With the Insiders: Secrets of the COT Report", written by the famous trader Larry Williams. Nevertheless, the book did not become very popular, and the material didn't become widely used. One of the main reasons for this fact is that all the concepts presented in the book, have remained theoretical.
After more than two years since the Russian translation of the book, there hasn't been proposed any universal, useful, and opened software for the conception use in MetaTrader4, the most popular trading platform in Russia. This article fills this gap. Now anyone can test and use the Larry Williams' theory in practice using their own MetaTrader 4 client terminal.
The method of the current market analysis using the Commitments of Traders Report can be considered as a fundamental analysis. The fundamental analysis itself hasn't found a wide application for the traders. It's no secret that most of traders uses technical analysis for the real trading. This is due to the fact that fundamental analysis is often connected with the economic news release, and its impossible to predict the market reaction on the news.
Secondly, this analysis is non-parameterized, and thus it's very subjective.
The third, the fundamental analysis is potentially endless. It means that you can analyze just a certain part of the economic effects that can affect the market, and the other part is always not considered by the trader, but it will be considered by the market. The use of the CFTC reports automatically eliminates all the weaknesses of the fundamental analysis, and Larry Williams's method for these reports analysis gives the best features of technical analysis to the technology. This means that firstly, all the significant economic effects can be considered indirectly through observing the actions of three groups of market participants.
Secondly, all these observations can be reduced to the simple technical indicators, with reducing the subjectivity and ambiguity.
It is possible to create an Expert Advisor that uses the simple fundamental analysis rules for trading. One of such developments is presented here. I think that it can be useful for anyone, who will decide to use the CFTC report for a trading.
The first part of the article is about the economic theory of supply and demand. Using simple examples, the market model is described, the possible existence of the different market phases is proved. The second part is about data analysis methods, which are necessary to determine the current market phase. It includes the descriptions of a set of indicators needed to analyze the interaction of buyers and sellers. It's mostly based on Larry Williams's book, in which the practical use of these indicators has been shown in details. The third part is about the technical implementation of the Meta COT Project. It contains detailed information about the Expert Advisor use, the use of indicators and scripts that are included in the project and how to download necessary information. The fourth part is the final. We will examine the Expert Advisor and the effectiveness of the concept presented in the article. The final conclusions are presented, and some constructive suggestions are proposed.
1. The Market as a Balancing Function of Prices
1.1 The law of supply and demand
Market is an institution or mechanism, that brings together buyers ("demand") and sellers ("suppliers") of particular goods, services or resources. [3]
There is no more precise definition of the market, than that given above. In fact, the main task of any open market is bringing together sellers and buyers. The relationship between the buyers and sellers are regulated by the law of supply and demand. This fundamental law makes stock prices rise or fall, which in turn allows traders to profit or loss from the price difference between the future and the present prices. Trader should know the mechanism of the balance changing between the supply and demand, so let's consider the following table [3]:
Total supply | Price per bushel (U.S.) | Total demand per week | Surplus (+) or shortage (-) |
|---|
12 000 | 5 | 2 000 | +10 000 |
10 000 | 4 | 4 000 | +6 000 |
7 000 | 3 | 7 000 | 0 |
4 000 | 2 | 11 000 | -7 000 |
1 000 | 1 | 16 000 | -15 000 |
Table 1-1. The Market supply and demand of corn (in thousands of bushels)
Here is the connection between corn sellers and buyers, expressed in terms of its price and delivery volume. Sellers wish to sell their goods as expensive as possible, and buyers wish to buy it as cheap as possible. In this case, at corn cost of $ 5 per bushel, there are many sellers who would like to sell it. Their total supply will be 12 000 bushels. But the seller's problem is that there aren't so many buyers who are ready to buy it at such a high price. There are buyers who are ready to buy only 2 000 bushels at such a high price. The other 10 000 bushels (12 000 - 2 000 = 10 000) remain unfulfilled, causing goods surplus from sellers, and goods shortage from buyers. This price is unable to stay on the market. Many sellers will cut their prices just to get rid of goods surplus and get even smaller, but live money. They will join other sellers who wish to pre-empt his fellow competitors. The competition among sellers will begin, which will decrease prices [3].
Now let's consider the opposite case. At this time we assume that the corn price is very low - $ 1 per bushel. Such corn price will be very attractive for buyers, there are many buyers who wish to buy it. Their total demand is 16 000 bushels. But there would be much less sellers who want to sell at that price. The total supply (sales) amount is just 1 000 bushels, which again will cause goods shortage among the buyers in the amount of 15 000 bushels, and surpluses from sellers. Again, this price is not able to stay on the market. Some of the buyers will buy the corn at a higher price, to get the goods they need. These buyers will join their colleagues and anticipate further growth of the prices and want to be among the first who bought it relatively cheap. The competition among buyers will begin, which will increase the price [3].
To make it clear let's draw the functions of demand and supply on the graph:
Figure 1-2. Demand and Supply as function of the price
We have considered only the extreme distribution cases between the demand and supply. In the first case, the price will fall, in the second case, the price will rise. Both of these cases will be to the certain equilibrium point in which demand will equal supply. At this point there will be no shortage of goods among buyers, or its surplus from sellers. In our table such point is located at 3 dollars. At that price, the total supply, as well as total demand will be 7 000 bushels, at this price there will be the equilibrium exchange between the buyers and sellers with no unsold goods left or no deficit appeared.
In a real life, the supply and demand are changing under the influence of many factors, ranging from the fundamental reasons to random events. In any case, there will be the only one cross point of supply and demand. This point will be the current market price.
Consider an example when the supply will increase at a constant demand (point A-B on the Figure 1-3, a). In this case the price will fall (p1<p2), since the goods will become cheaper and the competition among the sellers will begin. On the other hand, the amount of buyers who want to buy goods at a better price will be more (q3>q2).
Now consider the case when the supply decreases (point A to C at Figure 1-3a). This will cause the higher prices, because there will be less goods and the competition among buyers will begin (p3>p2). Certainly the buyers who want to buy at the higher prices will become less.
Figure 1-3a. Supply change at a constant demand
The same is true for the demand - when it increases (points A and B at Figure 1-3b), the price starts to rise, and it will be many who want to sell at a higher price, which would result into increased total supply (q3>q2). In the opposite case, when demand begins to fall, the price will fall, sellers wonting to sell at a low price will be less, and thus, the cumulative number of supply will decrease (q1<q2).
Figure 1-3b. Demand change at a constant supply
Now let's consider the case where the demand and supply are changing simultaneously. Is clear that there are only four such cases [3]:
1. The demand decreases, the supply increases. Each of these factors separately leads to the lower prices and, consequently, the total price decline will be even greater than that caused by each of the factors separately.
2. The demand increases, the supply increases. These factors are mutually compensating, since the demand growth leads to the price increase, and the supply growth leads to the price decrease. Generally, the price movement will depend on the stronger factor.
3. The demand decreases, the supply decreases. Just as in the previous case, the result is uncertain. With the demand decrease the price will decline, while the supply decrease leads to the price increase. Generally, the price movement will depend on the stronger factor.
4. The demand increases, the supply decreases. The demand increase leads to the price increase, exactly as the supply decrease leads to the price increase. As a result, the total price increase will be even greater than that caused by each of the factors separately.
On the real market the supply and demand are changing every second, and all these four cases are reflected in the actual market price of the value. Thus, the cross point of supply and demand "walks" on the chart of prices and volume, at different time periods, being in different locations (see Figure 1-4):
Figure 1-4. The dynamics of demand and supply
However, and most importantly, the cross point of supply and demand can't have high volumes at low cost (ie, located in the upper left quarter of the chart), exactly as it can't have a high price for low volumes (ie, located on the right bottom quarter of the chart). For greater clarity, let us turn to the Figure 1-5:
Figure 1-5. The Market Phases
As it has been stated above, the suppliers will refuse to provide a large volume at low prices (to sell a lot and cheap), and the buyers will refuse to provide a high demand at high prices (to buy a lot and expensive). Therefore, it is possible to construct a model of any market. The Figure 1.5 shows a model. The dotted line shows the average cross of the supply and demand. The full market cycle can be divided into three periods or phases:
1. The Low Prices Phase. It is characterized by low volumes at low price. The real, but unrealized market demand from buyers will be high, while the market supply will be limited. There is a goods shortage among the buyers and unrealized goods surplus from the sellers. It will cause the buyers competition, which will lead ultimately to the price increase.
2. The Average Prices Phase. In this phase there are acceptable prices at sufficiently high volumes. There is no shortage of buyers for both and there is no surplus from the sellers. Here is the equilibrium point for the price and volume [3]. In Figure 1.5 it is presented by dot.
3. The High Prices Phase. It is characterized by high prices (P) with high volumes (Q). The actual supply from the producers of goods will be high, while the buyers demand will be limited. The buyers' reluctance to buy expensive goods and producers' desire to sell expensive goods will cause the producers' competition, which will lead ultimately to the price decrease.
In other words, the law of supply and demand determines the return-and-trending market movement. In the medium term (its time scale is about 1-1,5 years approximately) the market will go from the low prices phase to the high prices phase, or vice versa, so there will be a trend. In the long term (from one year or more) the market will go to the high or low prices phase, meeting with buyers' or sellers' resistance, and under their action will turn backwards, so it will be the flat. The market movement can be compared with the breathing. After a full exhalation it is very difficult to find even just a little air in the lungs, to continue exhalation.
Contrary, the pressure between the diaphragm and lungs is becoming less than atmospheric, and air come to the lungs easy. After the full breath the volume of lungs is completely used and there is no possibility to inhale it still at least a little. At this time the pressure between a diaphragm and lungs becomes more than atmospheric, superfluous pressure tries to push out air from the lungs. This process repeats again and again. The similar processes occurs in the market. The arising buyers' competition (the negative pressure) forces the market to increase prices (to inhale air) while the sellers' competition (the positive pressure) forces the market to decrease prices (to exhale air).
As we have seen above, the goods shortage and oversupply situations are possible at the market. But it is also true that the equal quantity is bought and sold every time on the market. Therefore, the conclusions of some technical analysts that the market is dominated by the bulls, when they see an uptrend or bear when they see a downward trend, are incorrect. The market always has an equal number of bulls (buyers) and bears (sellers) contracts. But, at low prices, there is a limited quantity of goods for sale on the market. The majority of goods owners wait out such periods.
In the other words, if it is wheat, it stays in the storehouse and stays in the bank if it is a currency. Thus, buyers are unable to buy goods in the quantities they want. Using the standard terminology this situation can be called as "overbought". The opposite case, where the prices are high and all of the goods surplus from the sellers are on the market, may be called as "oversold". Further we will mean it for the overbought and oversold cases.
To illustrate the examples of low and high prices, let's consider the weekly sugar bars:
Figure 1-6. Sugar, weekly bars
We know that at relatively low prices there will be little who want to sell, and thus the goods volume presented on the market is also relatively small. Contrary, at the high market prices the goods volume will be more. The price itself can tell us about the possibility of the oversupply or shortage of goods. However, this assumption is often rude and estimated. Although the goods price and volume are correlated between themselves, there is no direct linear relationship between them.
At relatively high prices there can be a relatively small volume of goods, while at the same time at relatively high volumes there may be relatively cheap prices. This is due to the fact that the functions of supply and demand are rarely linear, almost always they are subject to both price elasticity of demand and price elasticity of supply [5]. The second problem determining the volumes of the market using the prices is the "relativity" of such an approach. How can we determine, what prices are high, and what prices are low?
We know that the market is not counter-trended in the long term, so why not to create an indicator of overbought and oversold states of the market, and using it start to open short positions first, when the market reaches the high prices and to open long positions, when prices are low? The profit must be obtained with a simultaneous use of all of the market properties - when it is trended and when it is counter-trended.
Many technical indicators have been developed during last fifty years, that attempt to determine overbought and oversold market states. But all of them have one serious problem - the are based on only one variable of the demand and supply, and this variable is price. The change of the price is the last thing that happens on the market after the change of supply and demand balance.
Consequently, at best, a trader using any indicator, based on the price or the price itself, will be the last to keep up with the changing price, but will never be able to preempt it. The simple computer tests shows the inconsistency of this approach.
Here is a profit chart of the elementary system based on the RSI indicator:
Figure 1-7. The profit dynamics of the Expert Advisor, based on RSI indicator. EURUSD, daily bars, from 2000 to 2009.
The system is elementary. If 7-period RSI reaches 80 percents, it opens a sell deal. If the RSI reaches 20%, then it opens the buy deal.The custom period has chosen to open more deals. The stop is 100 points, and the exist from the trade condition is the profit of 200 points. Testing performed on EURUSD, from 2000 to 2009, the daily bars has used. The trading lot for all of the deals was constant and equal to 0.1.
The conclusions are unambiguous: using the price data only it is not enough to determine the phase of the market. We need a second variable of the equation of supply and demand - the market volume.
1.3 The Open Interest is the second variable of the demand and supply equation
It is surprising that for many markets, there is an opportunity to know their volume up to one contract, analyzing the open interest values.
The Open Interest is the number of contracts of the futures and/or options market and uncompensated by deals, delivery, etc. The Open Interest is the sum of all long or short positions [1]. For example, if the sum of all long positions in the market is equal to 1000 contracts, it means that the sum of all short positions in the same market, too, is equal to 1000 contracts and open interest of 1,000 contracts are the same. The formula for the open interest is presented below:
The buyer and the seller of one futures contract for the wheat will add one unit to the Open Interest value.
It is important to understand the difference between the Open Interest and the market volume. For example during a day 1 000 000 contracts can be traded, but it does not mean that there are goods for 1 000 000 contracts. It may be much smaller, just during the day they passed from hand to hand, increasing the daily volume. At the end of the day, not all of the positions are closed, many of them are rollovered to the next day. The number of open contracts is the Open Interest Indicator. The producers of goods are looking to hedge their risks by buying long futures, thus they give a commitment of the actual delivery of goods. It is reflected in the Open Interest values. Thus, in some sense, the Open Interest could be a measure of the volume of goods on the market.
The Open Interest is calculated daily at the end of each session in all futures markets. This information is open and it is usually available on the official websites of the futures exchanges. The same data on Open Interest can be obtained by subscribing to the paid signals like e-Signal. For the MetaTrader users, the Open Interest data for more than 40 markets can be obtained using the indicator «Meta COT: Net Position», which is part of Meta COT. project presented here. For more information on its using see the third part of the article.
Unfortunately, the information on Open Interest (as well as for all other types of data described in this article) is not available on the spot and stock markets, because of their structure and properties. However, due to the strong correlation between the spot and the futures markets, it becomes possible to trade on the spot, guided by the dynamics of supply and demand on the same futures market. For example, you can analyze the Open Interest dynamics for the EURO futures, but trading on the spot EURUSD. However, for a more accurate representation we will use the futures chart only.
Now let's look at the of Open Interest dynamics for Gold over a long time period:
Figure 1-8. Gold, weekly chart
In Figure 1-8 we see the Price chart (upper) and the Open Interest (lower) chart for the futures of Gold from 2003 to 2009. One can see that when the Open Interest values were relatively high as compared with its previous values, the market started to go downward. At the same time, when the Open Interest values were low, the market started to go upward. It isn't seen well on a weekly chart because of the long time period, including a recent high volatility.
What is true for the gold, it is true for any other commodity, the law of supply and demand works on any market:
Figure 1-9. Soybean, weekly chart
Figure 1-10. US Treasury bonds, weekly chart
As you can see, relatively low values of the Open Interest indicate the potential for the market growth, while relatively high Open Interest values indicate the potential for the prices decline. The market does not always reverse when Open Interest has minimal or maximal values, and the extremal values of Open Interest don't point to an immediate market reversal. The Open Interest is not a magic market "switcher", but it allows to measure the second dimension - the market volume, and estimate the current phase of the market.
Here is the first rule, that can be formulated from the law of supply and demand:
When Open Interest has extremely high levels, try to open short positions.
When Open Interest has extremely low levels, try to open long positions.
2. The Market Structure
We have found that the market - is an institution or mechanism, that brings together buyers and sellers of certain goods and services. Now it is time to consider these groups in details.
2.1 Hedgers and speculators are parts of the market structure.
So, the market always finds the cross point of supply and demand. This point is the consensus price of the goods, i.e., the price at which sellers are ready to sell a certain volume of goods and buyers are ready to buy this volume of goods. Every time the demand and supply is changing under the influence of the fundamental, political, random, and other factors. Based on the law of supply and demand, the price of goods is also changing. Consequently, there is some probability that the future price will be either more or less than the current one. Once the price may change over the time, the price has uncertainty in the future. Such price uncertainty about the price in the future is called a risk [6].
Thus, any owner of goods holds the risk of the adverse price changes in the future. However, the price change may be upward and downward. If the price changes downward it would mean the losses from the sale at a lower price in the future compared to the current market price. Contrary the increase in commodity prices should give the owner an additional profit. Typically the principal owners of goods are their producers.
If the product is wheat, its owner may be the General Mills company, and if it is gold - its owner can be Barrick Gold Corp, etc. While the high risk of the price changes is high, and the main income of these companies is obtained from the difference between the cost of their products and the price of its sale, they are interested in reducing the risk that arises in consequence of the goods owning. To do this, the manufacturers use hedging operations in commodity and financial markets. In fact, manufacturers sell their risk to others who want to buy it. As addition to the risk, its buyers have an opportunity to get an additional profit (risk premium) in case of the favorable price changes.
Thus, the exchange is not only a centralized market place. Exchange is also a centralized place for buying and selling risk and premiums for it.
The buyers of goods and the risks attendant to them are called speculators. The main objective of a speculator is to get a profit from the difference between the current and future prices. Speculators are like a "glue" between the manufacturer of the product and its end consumer. They provide a high market liquidity, with a smooth change of the price [4].
The hedging operations are not so much the reduction of risk to zero, it is a way to find the reasonable balance between risk and profit. As you can see further, competent hedging allows the use of an opportunity for speculation. By monitoring hedged positions dynamics it is possible to join the investment group of large manufacturing companies, who know almost everything about the market they work in.
In fact, this viewpoint is not devoid of meaning. These companies are direct goods suppliers, which are traded on the exchange. Who, if not the manufacturer, must know everything that happens in this field of the business? These guys have information hidden from the party eyes and ears, they know that nobody else knows. Or how to explain the fact that for a many decades, they remain successful players on the exchange, performing many millions of dollars daily transactions!
The monitoring of speculators is also not devoid of meaning. The speculators' buying levels indicates the moments when the market is "overheated", i.e. it has the third phase, or vice versa, the moments when is it cold (first phase) (see Part 1). In the end, the speculators usually are the main buyers of the goods, which means that when they built up their long positions to the maximum, and they no longer have the possibility to buy, the market reverses to downward direction.
2.2 Commodity Futures Trading Commission Reports analysis
Anyone has an opportunity to observe the positions of hedgers and speculators, thanks to the American Government Organization called "Commodity Futures Trading Commission"- CFTC. The fact is that any private or legal person shall submit a report on their committed deals in commodity exchanges, if the volume of these transactions amounts to or exceeds the level specified by the Commission. Once a week the commission reports on the combined positions of traders.
Each report is published on the official website:
www.cftc.gov. The report is generated for every Tuesday state and is published on Friday late night, Moscow time. The report itself is presented in several forms: Excel table, CSV text file format, as well as a simple tabular text. Also you can download the history records for a long time period in the Excel and CSV formats. Reports are prepared for both futures deals and futures and option deals. The reports have a brief and extended form. The extended form differs from the brief by existence of the additional statistics and data on crop yields for some crops. In practice, the Excel or CSV files used for the CFTC report analysis. Now let's look at a brief report on wheat for August 4, 2009:
Table 2-1. The wheat report of 04.08.2009 (futures only)
At the top of the report there is a product name, in this case, it is the wheat traded on the Chicago Board of Trade, the type of report - futures only positions, and the date - August 4, 2009. The matrix consists of four main columns. Each of them represents the combined short and long positions of each of the three groups of traders. The first group of traders includes major speculators. The reports are called non-commercial traders (NON-COMMERCIAL). We can see that on 4 August 2009 the big speculators had 75 933 contracts, while on the short side there were 97 574 contracts. This indicates that their total or net position was short and it was -21 641 contracts. Usually, it is not a typical situation for speculators.
In the most, but not all markets they are pure buyers, that is, their long position is always higher than short. Using only one report for analysis it is impossible to determine whether this situation is typical in this market or not. In any case, you must remember that the most important parameter here is the Net position of the group, but not long or short position separately. This net position is used when calculating the majority of indicators of the traders reports analysis. So, once again: the total or net position is equal to the difference between long and short positions, it can be both positive and negative, that's its formula:
where NetPosition is a pure or total net position of the traders, i is the category of traders, such as major non-commercial traders or large commercial traders.
The next is the number of major contracts for non-commercial traders which are in the so-called spreads, or coverage.
The Larry Williams writes about it the following: "If a non-commercial trader holds futures contracts on the euro/dollar 2 000 long contracts and 1 500 short contracts, 500 contracts will be included in the "Long" category and 1 500 contracts are "Coverage" [1]. Simply, the coverage shows the number of both long and short contracts.
It is important to note that we do not mean the locking. For example traders can keep one and the same commodity opposed positions in futures contracts for different months of delivery, or have opposed positions on the same commodity in futures and options. In the report, which includes positions on options, such cases are taken into account.
The second group reflects the hedger's positions. They are also called the operators or commercial traders (COMMERCIAL). Typically, but not always, the operators are pure sellers, because most of them are producers of commodities like wheat, gold, or pigs. Commercial traders are also other manufactures who use goods as a base for their products, for example, candy floss or a bread. Typically, but not always, they are sellers, i.e., their contracts are on the short side.
Generally, when prices are dropped to a low level, operators are reducing their sales to minimum, because it is not profitable to sell their goods cheaply. At the same time, operators who are producing their own products, on the contrary, increase their long position. Thus, the net position of operators can be long, and indicators based on the net position of operators will be located in the upper zone.
Let's consider the specific case using the example of our report on wheat. The long side of the operators has 166 518 contracts, while the short side has 130 979 contracts. Their net position is 35 539 contracts, i.e. it is long. Often, this means that the price of goods is at the low level, because operators are selling less than buying. In any case, to determine the exact situation it is necessary to use a chart of their net position for a long time period.
The «Total» column contains a set of long and short positions for the combined commercial and non-commercial traders. It is of no practical interest.
The «Nonreportable Positions» column contains a number of long and short positions of unaccountable traders. In fact, this column reflects the small speculators' position, whose positions are too small to get into the speculators' group. These positions are calculated by synthetic calculation - from the value of open interest (OI) the total long and total short positions of reported traders are subtracted. In our case, the OI is 322 431 contracts, while the long is 279 239 contracts, this means that 29 194 contracts are long side of unaccountable persons (322 431 - 279 239). The same calculation can be performed to the short side: 322 431 - 279 339 = 43 092 contracts.
Here are changes in the number of contracts for each group compared with the previous report. Following is the share of each category in the open interest, and at the bottom of the number of traders in each category. These data are not used in practice.
In addition to the report, as it was mentioned above, there are reports that include the Options position. All options positions are converted into futures equivalent and added directly to the futures positions. The same report on wheat, but option and futures combined positions looks as follows:
Table 2-2. The brief wheat report of 04.08.2009, futures and options
Despite of the different numbers, both types of reports will give similar charts of the position changes dynamics. Nevertheless, it's more preferable to use reports that include information on options positions, as they consider a more complete volume of the market. Take a look at the increased value of the "Spreads" column. Now values in this row are dominated by comparison with the long and short positions for non-commercial traders. This is due to the fact that commercial traders mostly use options to cover their futures positions.
Now let's examine an extended form of the report which includes positions of futures and options on the same wheat:
Figure 2-3. The full wheat futures and options report of 04.08.2009
As you can see, the report includes additional statistics on old positions for each of the three groups of traders. Since all of these reports are analyzed in the dynamics over a long time period, this information is of no interest.
2.3 Observing large hedgers' positions
So, it is time to compile statistics on buys and sales of hedgers and monitor their actions for a long time period. For example, we will use the wheat price chart already presented, with the indicator showing the long and short positions of operators. The chart consists of weekly bars and covers the period from the middle of 2001 to the middle of 2009:
Figure 2-4. Wheat, long an short positions of operators, weekly bars
The green line here shows a long position of operators, the red corresponds to the short positions of operators. All positions are specified in absolute terms, they don't have a sign. It is necessary to assume that the red line represents the absolute amount of short positions, and green - the absolute amount of long positions. In general, short positions of operators dominated until the spring of 2004 (red line above the green), after which the operators had more buys than sells (the green line above the red). Such a chart does not show key break points. Now let's take a look at this same chart, but with an indicator of operators' total net position:
Figure 2-5. Total net positions for the operators, weekly bars
Here you can see more interesting moments. Note that when the net position of operators was relatively high, after some time the market went upward. At the same time, when the net position of operators was relatively low, after some time the market went downward. Of course, not all of the indicator predictions come true, but even if it happened to enter at least one correct point, it would be a huge profit.
Let's pay attention to the extremely high net position of operators in the end of 2005! Never before or after, the operators had such high values of the net position. And now take a look at what happened to the market thereafter. It began the inexorable upward movement, which continued more than 2 years. As a result, the price increased by more than 700 (!) USD. There were some other moments, showing us a good opportunity for speculation.
For example, after a strong price fall at the end of 2007, many might think that the bull trend would finish. Apparently, the operators didn't think so. While others tried to reduce their long positions (we know it from the traders reports), the operators bought up the market! The rise was soon. A maximal price has been updated, and then the panic has began. During these four months, prices increased by another $400! The sale signals given by the operators were not so convincing. Nevertheless, in many cases, the trader was able to escape the market safe if he used a simple trailing stop techniques. In any case, all of the losses could be compensated by one good deal.
The key words here are "shortly thereafter". Do not forget that this indicator is not calculated by using the price. It is completely independent from it. We could remove the price chart, but the indicator of net positions of operators will still be showing the same values. Price is the recent market changes. The first to change is the balance of supply and demand forces, and even before that the minds of major market participants change. Observing that the operators are doing, we are witnessing the evolution of the market at its early stage, when the price changes hasn't appeared yet. It gives us a unique advantage to take place in another half-empty trailer of the future locomotive trend. When the movement begins, many will come too late there, but not the traders using the commission reports!
Let's observe gold and silver through the prism of operators:
Figure 2-6. The gold and the net position of the operators. Weekly bars
The chart of gold prices is full of favorable opportunities for speculation. Not all of them lead to excess profits, but many of them could bring a substantial income. If a trader correlated his entry points with the main bullish trend that continued for many years, he could be a very big winner. Let me remind you, we are considering the moments when the net position of operators is high enough, these points are potential points for buying. The same applies to sales: we're looking for relatively low levels of net position and trying to short, looking for an opportune moment to sale.
Note the madness that went to gold at the end of 2008. After the strong fall ended in early September, it began an unexpected rise. In fact, during one day the gold price has increased by 90 dollars! Then it was at a high price till the end of September, but it was not very long. Shortly after this there was a new fall, and in the end of October, the gold updated its year's minimum.
Could we predict that shaking at least partially? Yes we could predict it by using the reports on traders positions. Pay attention that not long before the bull Friday (it was the day when the gold's price "suddenly" has increased by $90), operators have stopped to sell the gold.
They are the producers in this market, and all these years there was not a single week, when the purchase would exceed the sale. In fact, their refusal of sales brought the panic among the gold bulls. There was a shortage of metal that moved the prices up.
Although the prices didn't stay on such Olympus for a long time, operators sold some of their metal to gold bulls, who where the last to respond to the price peak. Further the gold price fall has began, after which operators reduced their gold sales to the four-year minimum (their net position increased (see chart)). Further we saw the effect of the sellers' competition. The shortage commodity began to rise in price, and this could be a good idea to make money.
Now let's turn to the chart of silver:
Figure 2-7. The silver and the net position of the operators. Weekly bars
And again we see here the same as for the gold. Whenever the net position reached a relative maximum, the silver price made a new leap to the sun. The relatively low levels were indicating the recession in the uptrend of silver.
The conclusion is the following. The markets of precious metals are largely controlled by a small group of professionals. And any trader who wishes to trade profitably on these markets should take into account the influence of these forces.
2.4 Index of a net position
But how do we know what is the "relatively" high net position, and what is low? As noted by Larry Williams, the floor for one is the ceiling for another. Therefore it is necessary to use the normalized indicator, which would clearly state about the phases of our position. Such an indicator exists, it is called the COT Index. The index is a usual Stochastics calculated for the values of our positions.
Lets recall it's formula [1]:
The current price is compared to the maximal price for three years. The index can be calculated for any time period, not just for three years. It's better to use a period of not less than 26 weeks, i.e. half-year index. In the longer-term deals a 156-week(three-year) index is used. The index shows the relative current strength in percentages, compared with the selected period.
Here is an example of the index calculation for the operators listed below, the digits on the right side indicate the number of contracts [1]:
Current week value | 350 |
Minimal value of last 3 years | -150 |
Difference | 200 |
The maximal value of 3 last years | 750 |
The minimal value of 3 last years | -150 |
Difference | 600 |
Index = (200/600)x100=0,33x100=33%;
In this case the operators are bearish rather than bullish. If the index reaches extremely low levels, i.e. located in the bottom 20% range, the market tends to fall. If the index reaches extremely high levels, i.e. located in the upper 80% range, the market tends to the upward reversal.
To verify this, let's look at the same silver chart, but with 156-week Operators Index:
Figure 2-8. Silver and 156-week Operator's Index, weekly bars
You can use other levels, for example 75% and 25%. The meaning will be unchanged, the Indicator will show potential overbought and oversold zones.
Strangely enough, but the period of the index does not change the dynamics of monitored positions. Here is an example of several indexes with different averaging periods for silver:
Figure 2-9. Silver chart with different periods for the index of the silver operators, weekly bars
Here we have used (from top to bottom) 156-week, 104-week, 52-week and 26-week averaging for the indexes of the operators. Only 26-week index shows a more frequent movement amplitude. The other periods of averaging look almost similar and don't change the indicator's shape. The averaging period used depends on the personal preferences. For example you can use the 156-week averaging period, it shows the dynamics in positions change rather widely, and at the same time it gives a clear interest result for bullish positions.
The COT index is used to monitor not only the bullish operators, but also for non-commercial traders, as well as small speculators.
2.5 The Open Interest Structure
In the first part we have analyzed examples of the commodity volume influence on its price. To measure the volume of goods on the market, the use of the Open Interest (OI) is a very effective. Economic theory predicts that the market which characterized by relatively high levels of the Open Interest is most tends to the downward reversal. The same rule is true for the market with the relatively small values of the Open Interest. In this case, the price most likely to move upwards.
Now it's time to investigate its structure. From the traders deal reports we know that the main players on the futures markets are hedgers, also known as operators, and "non-commercial traders" - representing mostly large commodity funds. Also there is a third, minor group of traders - scalper or so-called "crowd". Because of the small deal volumes performed by them, they are not able to influence the market prices. The operations volume of that category is calculated indirectly, and represents the difference between the Open Interest and the total amount of operator's and non-commercial traders' open positions.
The weekly Open Interest values are presented in the CFTC reports. Since the OI is the cumulative number of opened long and short positions, as it always is, its level can be calculated in two ways, by counting the total long position, or by counting the total short position.
Here are the formulas:
OI = Noncommercial Traders Long + Noncommercial Traders Spreading + Operators Long + Nonreportable Long;
OI = Noncommercial Traders Short + Noncommercial Traders Spreading + Operators Short + Nonreportable Short;
Now let us turn to the report on the euro futures of the August 4, 2009, and calculate the open interest for this market by using these two formulas:
Table 2-10. Euro futures report of 4 August 2009
OI = 61 443 + 946 + 22 984 + 52 864 = 138 237;
OI = 34 337 + 946 + 72 454 + 30 500 = 138 237;
Regardless of the Open Interest calculating formula, the result is the same. However, as you can see, long positions are held mostly by large non-commercial traders (61 337 contracts), while the operators prefer short positions (52 864 contract). Calculation of share of each of these three groups in Open Interest would be a logical continuation of the analysis of the Open Interest structure. For example, the proportion of short position statements in favor of the period amounted to 52,4% of open interest (78 454 / 138,237 x 100%). For example, the share of the operator's short positions for the presented period has made 52,4 % of the Open Interest (78 454 / 138237 x 100 %). Its value is presented in section «Percent of Open Interest for each category of traders».
However it is interesting to consider the share change dynamics of each of three groups of traders. If we collect this information for the long period of time we can construct the corresponding chart. The indicator «Meta Cot: Percent Position in OI». It calculates this data for each of three groups of traders.
The figure 2-14 shows the long-term chart for the Japanese Yen futures. Every time when the share of operators' short position has made 70 % and more of the Open Interest the Ten was close to the downwards reverse. Almost every time the share of short positions operators was below 30% of open interest, the market was close to its bottom, and frequently thereafter began its long-term bullish trends in the movement. Such cases are represented by red dashed line. The same is true for the upwards reversal. Every time when the share of the short position was below 30 % of the Open Interest the Yen was close to the downwards reversal. Every time when the share of operators' short positions was below 30 % of the Open Interest, the market was close to its bottom, and frequently after that it began the long-term bull trends.
Figure 2-11. Share of the operator's short positions of the Open Interest. Japanese Yen, weekly bars
Larry Williams in his book
"Trade Stocks & Commodities with the Insiders: Secrets of the COT Report" [1] proposes to combine the Open Interest values with the net position of operators in one indicator. In fact, if the relative net position of operators is low enough and at the same time, they hold a significant market share it can be assumed that the market is close to its peak and soon will turn down.
This indicator is calculated by the formula:
Stochastic Oscillator (Net Operators / OI);
In other words, the net position of operators divides by the Open Interest, these data are collected over a long period of time, the Stochastics indicator calculates using these data. This indicator is called Williams Commercial Index, or simply WILLCO. Larry Williams recommends to use a 26-week or a six-month average, but you can use the other types of averaging, such as the annual (52-week) or three-year (156-week) averages.
Its using is the same as the COT index, whenever its value exceeds 80% - expect the market turns down, whenever its value is below 20% - expect the market reversal up. At figure 2-15 is presented the WILLCO indicator for the Japanese Yen. The red dotted lines show the same levels that were outlines by Larry Williams on that same chart [1]:
Figure 2-12. The averaged WILLCO (26 weeks) and the Japaneese Yen, weekly bars
As we see, the further recommendations of this indicator were not so accurate. However, the time period increase removes this problem. Take a look at the chart of the U.S. Treasury Bonds and the same indicator, but with a period averaging of 156 weeks:
Figure 2-13. Averaged WILLCO (156 week) and U.S. Treasury Bonds, weekly bars
Now we have marked extremal low levels as well as extremal high levels. The indicator accuracy is amazing, because of the fact that its data are not based on the price data! Undoubtedly, the U.S. Treasury notes are very sensitive to the actions of operators.
2.6 Momentum indicator
This indicator was proposed by Stefan Brice, in his book «The Commitments of Traders Bible». Its idea is simple, it represents the difference between the current COT-index and the same index 6 periods ago, so its formula is:
COT-Index (p) – COT-Index(p-n);
Where p is the current value of the index, n is a period equal to 6.
The index delta can be any, and not only 6-period. In addition, the index can be calculated for all participants of the futures market, and even for the Open Interest itself. This indicator is called the Movement Index. It is mainly used to confirm for correction completion to the long-term trends. Its interpretation is simple. If the Movement index rises above 40% - the current downward movement is coming to finish. The prices rise is expected. If the index falls below 40% - the current upward movement is coming to finish. The prices decrease is expected.
Let's try to apply this indicator to the futures of euro:
Figure 2-14. Movement Index for EURO futures
Over the past years Euro has a steady upward trend. The blue arrows showing the times, where it has crossed the 40% barrier. Look to the incredible accuracy of the indicator, each time after it has crossed the 40% level, the price correction has completed, and the euro has continued its rise. The sale signals of the indicator were not so accurate. However, often the correction has began after the movement index crossed the lower boundary of -40%. Such indicator reactivity should be taken into account. It's a kind of COT momentum barometer. It clearly shows the market turbulence. Its use is particularly useful for the aggressive and less long-term trade.
The next indicator is an experimental and it hasn't described before anywhere. In the ninth chapter of his book Larry Williams explores the Open Interest changes via the action of the operators. It is not clear exactly how to use this relationship. For some examples, Larry Williams shows the divergence/convergence. For the other examples he concludes that the sales increase or operator's buys also increases the level of Open Interest. The investigation of the index movement has led to the idea that it is possible to monitor the changes in the operators positions via the changes of the Open Interest level.
The most interesting interaction models are the following:
1. Open Interest level is falling - the operators net position level is growing.
2. Open Interest level isincreasing - the operators net position level is falling.
In other words, there is a some discrepancy between the actions of all market participants (Open Interest) and the operators actions. Such changes would be best to observe based on the movement index calculated as for Open Interest as for the operators index. Comparing their difference, we can determine the divergence of the hedgers actions and the rest market participants actions.
This indicator was named
Spread Movement Index, is its formula:
Movement Index (Operators) – Movement Index (Open Interest);
In general it looks like simple Movement Index, as well as WILLCO looks like COT Index. The approximate critical level values are 60% and -60% respectively. Its analysis is carried out by the same rules as for the ordinary movement index.
Lets consider the index behavior on the euro futures example:
Figure 2-15. Spread Movement Index EURO
One can see that its values also indicates the significant turning points and correction complete moments. However, this indicator should be used with caution - its efficiency hasn't been proved.
2.7. Observing the large hedge fund's positions
The main goal for the large commodity funds is a speculative profits on the commodity markets. The trading methods they used are based on the general trend following. Once the price exceeds a certain n-week high, some of the funds opens long position. It gives an additional upward prices movement. Perhaps one of the main reasons of the market trends in the medium term are the large commodity funds actions.
By some estimates the funds choses 26-week High/Low a most frequently [1]. Funds are in the market gradually, using the technique of building products, besides many of the funds use more long-term strategy for entry. The funds enters on the market gradually, using addition technique for their positions, besides many funds use more long-term strategy for entry. For example, if the price reached its 26-week high, some of the funds are open long positions. Then the price moves further upward and reaches its 52-week high. More cautious funds are join to the game and open long position, together with them the funds already have entered into long are adding to their position. Finally, the price reaches its 156-week high. At this point, who wanted to buy are already have bought. Thats's all. There is no buyers now.
The trend strategies of all of the funds are involved to the maximum. The market became extremely strained. Because of the fact that there are no more buyers, we know it from the traders reports, the price should soon go down ... and soon it does. The price begins its downward movement. At this point the market has an extremely high number of participants (we know it from the Open Interest level), in they are starts to panic.
The first, the buyers who have entered the market later than others starts to close their long positions. They have not yet managed to get a sufficient level of profit that is necessary for long-term position retention. It gives an additional price acceleration and it begins to fall faster. Soon the panic increases and more and more participants are trying to close their long positions. Everyone is trying to squeeze into a doorway. The panic is growing so much that price falls down, during a very short time period.
After the crowd has left the market and the price was at the bottom, just after that the operators start their game and come on the market again, by adding their long positions. Their idea is simple - the operators that use the commodities for production of its goods are interested to buy the goods at cheap prices. Operators, by contrast, reduce their sales to a minimum, is it not profitable to sell goods at low prices. As a result, the net position of operators will be extremely high. Indicators will show this fact accordingly. The demand competition will start, and the price will rise again. The circle closes and history will repeat again.
Let's examine the actions of large non-commercial traders on the basis of the above concepts using the three-year index applied to a non-commercial traders' net positions:
Figure 2-16. Cattle futures price and total net postion of non-commercial traders
As we see on the cattle futures exaple, the actions of the non-commercial traders looks like the current price trend. If the price goes up, the funds are buying, if it goes down, they are sold, it's simple. Let's note that every time when the funds have increased their positions up to maximal values, the cattle price has began it's downward movement. When the funds position was close to its minimal values, the price has began to rise.
I think that the financial success of the investors, who have decided to invest their money to such funds are under the question. The similar actions of the non-commercial traders are the same on the other markets:
Figure 2-17. Cotton chart and 156-week averaged net position of non-commercial traders
While the index tops clearly shows for the coming market reversal, its low values, as a rule, were premature. In any case, this chart shows that it is better to short when the "big guys" are buying.
2.8 Observing of the small speculator's positions
According to the CFTC specification, the small speculators - all those who have too small positions to reach any of the above two groups. Their exact values are unknown, although it can be assumed that the total number rather impressive. For example, you can apply to a brief report CFTC on wheat on the Chicago Mercantile Exchange:
Table 2-18. The share of the small speculators in the brief report on wheat
As we see from the report the total number of major market participants is small. On the short side - 286, on a long - 304. 286 of the market participants holds 91.8% of all long positions and 304 traders has 88.5% of all short positions. Can we call these major market participants as crowd? I think that we can't. But many traders that haven't been counted in the commission report are the very small speculators, creating a market crowd.
In this case, the crowd controls only 8.2% of long positions and 11.5% of short positions. It would be quite remarkable to observe the history of buys and sales of this category of traders. Let us turn to the GBP chart over a long time period, but this time we consider the total net positions of the small speculators:
Figure 2-19. Total net position of small speculators on GBP futures
Note that every time when the total net positions of the crowd are reached their relatively high values, the market price hardly began it's downward movement. Conversely, when the crowd were disappointed in GBP and began to sell it, the GBP price began a rally. Particularly noteworthy were the two final points which marked by red lines. After the grand fall of the pound, the small speculators have decided that the bottom has been reached and the upward movement is inevitable. Within one week, they escaped from net sellers to net buyers. However, the bottom hasn't yet been reached. Pound continued to fall about two months. During this time the crowd mood has changed, and it was again mainly sell. And crowd was wrong again. Pound showed a decent rally.
The general rule for the analysis of the trading activities of this group is simple. Try to act contrary to how the crowd. If unaccountable traders sharply increased their sales - try to buy if they are contrary started to buy - try short.
