Curious observations about efficient frontier calculations

author — Wed, 18 Jan 2023 15:13:41 GMT

I was using Portfolio Visualizer to try out a few efficient frontier calculations. You can run the same calculation using this link. I find it quite curious: 1) What causes Portfolio 2 to stop at σ ~ 15%? In some other portfolios I tried, the efficient frontier curves can sometimes be really short. 2) How/Why would Portfolio 2, which is a superset of Portfolio 1, cross 1 from below?

To illustrate this, I've attached four example runs below. Red is EF #2, while blue is EF #1. EF #2 = EF #1 + one extra asset class. The region I was talking about is where the two curves intersect. Most prominent are Run01 and Run02, where the computed curves already intersect, while for the other two runs if the EF #2 calculations were to extend to higher risk values, they would also be below EF #2. My point was that by simply setting that extra asset class to 0%, EF #2 would do at least equally well as EF #1 -- what am I missing here?

The app developer kindly replied on the brief inner workings of resampled efficient frontier calculations, which is what one should expect for Monte Carlo simulations. Since I didn't ask the permission, I'll just point to the FAQ where some of the same references are listed. My best understanding is this: The algorithm first generates say 1000 simulated return series that include assets/asset classes from both portfolios. Each return series is different, but sampled from the same joint normal distribution with the same mean and covariance matrix as found for the historical data. Now for each of the two portfolio, the algorithm computes the efficient frontier for each return series and then averages, at each risk value, the 1000 (risk, return) data to get the combined efficient frontier.

How should non-professionals invest in the stock market?

author — Thu, 01 Aug 2013 05:16:32 GMT

Comparison of simple periodic, periodic with MACD signaled buys, and periodic using AIM strategy.

After reading a few books on investing (my favorites include Benjamin Gramham's "The Intelligent Investor", Philip Fisher's "Common Stocks And Uncommon Profits", Jeremy Siegel's "Stocks for the Long Run", Burton Malkiel's "A Random Walk Down Wall Street", Robert Shille's "Irrational Exuberance", Peter Lynch's "One Up On Wall Street" and "Beating The Street", and William O'Neil's "How To Make Money In Stocks"), I feel increasingly certain about two things: 1) The equations of motion for the equity market are not known. Even if they do exist, when they are not known, the market is still sufficiently difficult to predict. A good analogy would be pseudo-random numbers, which although are indeed generated by well-defined procedures, are statistically indistinguishable to real random numbers for anyone who don't know the underlying algorithms. 2) With high confidence one can say that whoever has benefit greatly from the market has worked hard to achieve it. Think about Peter Lynch, who left the trade to get to know his kids because he hadn't had time when he managed Magellan.

What is then the purpose of non-professionals participating in the stock market? To individual investors/traders, undoubtedly the answer seems to be making money. But on what grounds can one logically expect the endeavor to be profitable for him? Above all, the stock market only serves as a place to facilitate public companies in raising capital (initial offering) or to facilitate people to trade with each other (so that the market becomes more fluid and thus more attractive). In initial public offering, people with unused money fund those with ideas but not enough money to expand. A simple transfer of money does not create value. It is the increase in efficiency in the use of all available money that is responsible for why the investors and the companies can both become richer due to the same facility. In subsequent exchanging of shares of stocks, traders are essentially purchasing residual claims on the underlying companies' cash flow that they think are under-priced. One therefore naturally comes to the conclusion: in order to benefit from the equity market, one has to contribute to it through stock-picking in which funds can be preferentially directed towards more worthwhile companies. If it was not for the randomness and uncertainties inherent in real life, the success in the market would entirely depend on the ability of stock-picking; no strategy or system would be needed nor useful. However, life is full of the unexpected and market does fluctuate, and we test various ideas that try to smooth out or even capitalize on the market fluctuation.

Back to a more interesting question: can a non-professional investor outperform the market consistently? By market we also loosely mean any broad index weighted by capitalization. Since an index is a composite of all market participants, for one who outperforms it, there must be another one who underperforms it. Without business growth, the market seems a zero-sum game, so who win and who lose? Admittedly there will be exceptions, but for the most part, I believe non-professionals lose while professionals win, understandably so because the latter have devoted their studies in college and graduate school to learning the trade and the theory, are spending endless hours a day studying the businesses and the financial environment, have access to more and non-public information, have dedicated crew of analysts and tools to help with decision-making, etc. It's often quoted that most mutual funds underperform the market, but they do so mainly because of the numerous restrictions, from how much they can own a company, the requirement on diversification, to the curse of size (to a degree that no single investment will make a big difference and that they cannot load/unload a meaningful number of shares without driving up/depressing the stock's price). I highly suspect that most fund managers' and analysts' personal portfolio (provided they can have one) mostly outperform the market; otherwise they seriously should consider a different career.

With the foregoing arguments, it is then obvious to any sane person that the two only alternatives for ordinary investors are: investing in mutual funds and developing strong faiths in one's own ability to pick winning stocks with less time commitment (syn. luck & gambling psychology). It's interesting to speculate what would happen if everyone followed the advice and took the first route to invest in mutual funds -- the burden of stock-picking would then become the burden of fund-picking, and meta-funds emerge...

So much for the reasoning, and to summarize, the conclusion is that the main work of stock-picking should be delegated to fund managers or index funds, and one should be content with a reasonable rate of return consistent with the overall economic growth. In addition, the market fluctuation may need to be dealt with using a sound investment plan. Let's now look at the some of the more popular/simple methods.

Back-Testing Different Investment Management Strategies

Tests are done using NinjaTrader. The free version supports everything except the ability to trade within the software, including charting, simulation, back-testing/optimizing strategies, and a few more. Its strategy analyzer supports both graphical wizards and a programming language called NinjaScript, which is based on C#. The programs I used for back-testing are provided. In the tables of results below, the total contribution is the face monetary value of all funds put into the market, fund value is the market price of all shares held as of 08/16/2013, cash reserve is the amount of uninvested cash, market net profit is the sum of net profits from all trades (sell price minus buy price), market net return is the ratio of market net profit to total contribution, portfolio value is the sum of fund value and cash reserve, and annualized return rate is the compound rate of return taking into account of when money is invested.

SP500 is chosen as the index, both the fund offered by Charles Schwab (SWPPX) and the pure index. Fund fees are not taken into account but they are typically very small (0.1%) for index funds, and in addition, dividends are not counted either as I don't know how to obtain them from the data source (one can multiply the average dividend rate with the compound rate afterwards to get a feeling of the total rate of nominal return). These two factors should compensate each other somewhat but the dividend rate is expected to be higher. The 16-year period from 06/18/1997 to 08/16/2013 includes two bear markets and has a return of 3.98%, or 6.5% after considering 2.4% dividend rate, shy of the nominal compound rates of 8.3% (1802-2006) and 8.9% (1871-2006) (data from Stocks for the Long Run). The longer SP500 history from 01/03/1950 to 08/16/2013 has a return of 7.5%, or 11.4% with dividends, almost identical with the 11.2% found for all New York Stock Exchange stocks. The problem is that one does not have all the funds available at day one, so the simplest buy-and-hold strategy is not suitable.

Table 1 SWPPX, 06/18/1997--08/16/2013

	Total Contribution\*	Fund Value	Cash Reserve	Market Net Profit	Market Net Return	Portfolio Value	Annualized Return Rate
SWPPX	13.88	26.09	0.00	12.21	87.97%	26.09	3.98%
Simple Calendar	65318.06	92515.14	0.00	27197.08	41.64%	92515.14	4.5%
Calendar+ MACD\\	66782.25	94628.43	0.00	27846.18	41.70%	94628.43	4.5%
AIM Periodic	65284.76	48788.30	44128.72	25326.28	38.79%	92917.02	4.5%
AIM Lump-sum\\\*	20000.00	19731.66	11288.96	9540.17	47.70%	31020.62	3.30%

*Dividends and fees not considered. Annual inflation rate: 3%. Bank APY on cash: 1%.

**Initial (will adjust to inflation) regular contribution $60/week, MACD signaled buy $200/trade.

***Tested for ETF: SPY from 1/2000 to 6/2013. Bank APY on cash: 2%. Ran out of cash once, but an extremely small amount ($-12).

Table 2 SP500, 01/03/1950--08/16/2013

	Total Contribution\*	Fund Value	Cash Reserve	Net Market Profit	Net Market Profitability	Portfolio Value	Annualized Return Rate
SP500	16.66	1655.83	0.00	1639.17	9838.96%	1655.83	7.50%
Simple Calendar	70177.87	680819.34	0.00	610641.48	870.13%	680819.34	7.07%
Calendar with MACD\\	69437.12	670180.63	0.00	600743.52	865.16%	670180.63	7.06%
AIM Periodic	73203.67	76557.30	129318.77	108700.72	148.49%	205876.07	3.72%

*Dividends and fees not considered. Annual inflation rate: 3%. Bank APY on cash: 1%.

**Initial (will adjust to inflation) regular contribution $13/week, MACD signaled buy $44/trade.

Simple Periodic Investment by Calendar and/or with Additional Buys on MACD Signals (file)

Time/Dollar-cost averaging is perhaps one of the simplest investment plans, and is often recommended to ordinary investors as one of their best options. The main merit of time averaging is its ability to smooth out any deviations of stock prices from their "intrinsic" trend. If you invest a constant amount (adjusted for inflation) periodically over a long time trajectory, it's very likely you will end up paying premium as many times as you will get bargains. It also fits well with most people's scenarios that a fixed portion of the pay check is saved and invested, and it gets the emotion out of the equation as any strategy will do. Sometimes, one may like to spice things up by employing some technical indicators for additional buy/sell signals. An example will be tested that uses the moving average convergence-divergence (MACD) indicator. It signals extra buys when MACD crosses above its exponential moving average.

The attached NinjaTrader script was used to back-test the two systems on historical/simulated data. The test settings assume operating on weekly data (Data series type: Week) starting in the 1990s with $68 (CalendarEntryQuantity = 68) contribution every one week (CalendarEntryInterval = 1). Contributions are adjusted for inflation (InflationRate = 5.68600096E-4 corresponding to 3% annually) every CalendarEntryInterval time periods. One can also optionally make additional buys when the exponential moving average of MACD(12, 26), EMA(MACD, 9), crosses above MACD itself. The amount of additional purchase is set to $200 and in this case CalendarEntryQuantity is set to $60, both are adjusted for inflation over time.

Testing on Schwab S&P 500 Index (SWPPX) from 6/18/1997 to 8/16/2013 without commission costs (since it's a no-load fund, but you can set "Include commission" to true), we obtained the 2nd and 3rd rows in the tables above. Here, dollar-cost averaging yields better results than the market index by 0.5%, but it is worse by the same percentage over the longer period. Interestingly, the apparently useful MACD indicator does not really help at all in both periods.

Lichello's AIM (Automatic Investment Management) System (file)

Robert Lichello's AIM (book) is basically a scale trading system with modifications. It's intended for stocks but in its original form didn't really offer much advice on stock selection, how to re-balance to the suggested stock/cash ratios, etc. (see discussions in this blog). People have used it with mutual funds and ETFs in order to avoid the danger of individual stocks going bankrupt (see this article by Doug Burkeaz). Investopedia has a tutorial by Braden Glett, which attacks a simpler form of scale trading, but presents a strategy which he calls "reverse scaling". Glett's method cannot be easily used for funds because it involves selling out the entire position when the instrument drops to a lower decision point.

The attached NinjaTrader script can be used to back-test/optimize the AIM system on historical/simulated data. There are separate SAFE (Stock Adjustment Factor Equalizer) settings for buys and sells (default: 0.1), and limits on the minimum number of shares for a trade are also added to prevent frequent trading (default: 10; set to 0 if you don't want them). Since there are cash reserves, an interest rate is assumed to be compounded every time period (InterestRate = 0.01). I've coded the script to handle periodic investment, as this is the most encountered scenario. The default settings assume operating on monthly data (Data series type: Month) starting in the 1990s with $3000 (CalendarEntryQuantity = 3000) contribution every 12 months (CalendarEntryInterval = 12). Contributions are adjusted for inflation (InflationRate = 0.03) every CalendarEntryInterval time periods. The contributions are treated as the original lump-sum investment in AIM, so a portion of them (StockRatio = 0.5) is used to purchase the same instrument and its entire amount is added to portfolio control, whereas only half the purchase value is added to portfolio control when it's triggered by AIM buy signals.

Testing the modified AIM system as described above on Schwab S&P 500 Index (SWPPX) from 6/18/1997 to 8/16/2013 without commission costs (since it's a no-load fund, but you can set "Include commission" to true), we obtained the 4th rows in the tables. The more complicated AIM performs almost the same as the simple periodic investment plan in the shorter period, but is significantly worse for the longer period. I suspect this is because no re-balancing was done and the chosen parameters might not be suitable for the different "price" levels of the pure market index, which led me to the following tests. Nonetheless, I think we can safely conclude that it's not worth the effort to use AIM.

Following the article by Doug Burkeaz, I also did some similar tests for lump-sum types of AIM (but still without considering dividends as I don't know how to get the data from the source and also one can take them into account afterwards by multiplying it with the annualized compound return rate). In my first test, I didn't implement the 10-share trade limit, and as a result I ran out of cash twice (worst at about $-2630) during the period 1/2000 -- 06/2013 for ETF: SPY. It seems to me that an absolute rather than a relative value (10) here for the share limit is somewhat arbitrary, since another index fund or ETF with different price per share would usually require changing this value accordingly, and besides, shouldn't SAFE supposed to be a regulator already? For my second test (5th row, Table 1), I added the minimum-share requirement, I still ran out of cash once, albeit at a much smaller amount, which in real-life may not be a big issue since I can always temporarily move some funds into the market and retrieve it the next time I sell. The result of 3.30%, or 5.2% with geometrically averaged dividend rate of 1.8% factored in, happens to be slightly better than the 4.56% compound return rate reported in the original article (the reason why trade decisions are not identical in the two supposedly same tests is that Microsoft Excel treats int() like floor(), so int(-9.5) becomes -10, while int(-9.5) in C# is -9). Note that in these tests, depositing the dividends into the cash reserve may dramatically reduce the chance of depleting the cash reserve, but running out of cash was non-consequential and wouldn't change any sell/buy decisions as the situation was simply reported but allowed. However, it does appear that AIM is not very robust and its behavior depends to some extent on the exact choice of relevant parameters and on different market conditions.

These tests have also reinforced one of my previous thought, which is: if a system does not include a stock-selection process, based "somehow" (through either fundamental or technical analysis) on the real business growth prospects, it cannot hope to outperform the market. If it did and it was a robust system, it must also do so under extreme conditions -- think of what would happen if everyone adopted the same strategy -- dollar-cost averaging only claims to help achieve market return rate.

Conclusion

The price movements in the equity market consist of two parts: 1) the intrinsic change associated with overall economic improvement and business growth of individual companies, and 2) the fluctuations or deviations from the intrinsic trend due to imperfection of the market and its participants. Therefore, winning in the stock market involves, for the first part, stock-picking that demands hard work and talent, which helps direct funds towards companies with better growth prospects and which we argue is best left to professionals or the use of index funds, and for the second part, the management of incremental investment whose purpose is to reduce the impact of bad market entry and for which we suggest the simple periodic investment by calendar (time/dollar-cost averaging).