Bayesian Estimation of Simultaneous Equation Models with Outliers and Multicollinearity Problem

Abstract

Outliers and multicollinearity are problems in the analysis of Simultaneous Equation Model (SEM). They can lead to bias or inefficiency of estimators. This study employed a Bayesian technique for estimation of SEM that is characterized with both multicollinearity and outliers. Monte Carlo experiment was applied while the data sets with specified outliers and multicollinearity were simulated for the SEM. The estimates of Bayesian and classical methods namely; two Stage Least Squares (2SLS), Three Stage Least Squares (3SLS), and Ordinary Least Squares (OLS) in simultaneous equation model were then compared. The criteria used for comparison were the Mean Square Error (MSE) and Absolute Bias (AB). The Bayesian method of estimation outperformed other classical methods followed by two stage least squares, three stage least squares, and limited information maximum likelihood in terms of MSE and AB. However, the classical method has the same performance with Bayesian method when there are no outliers and multicollinearity in the simultaneous equation model. Hence, Bayesian method of estimation is preferred than classical method when there is problem of outlier and multicollinearity in a just identified simultaneous equation model. Mathematical subject classification: 62C10, 62CO7.

Keywords

Bayesian; Multicollinearity; Outliers; Monte Carlo; Simultaneous equation model

Introduction

Researchers often face the problems of multicollinearity and outliers in applied works either planned or not planned. Multicollinearity and outliers can lead to poor predictive power of the model and statistical inferences. These can also lead to inefficiency of estimators of simultaneous equation model. When multicollinearity occurs in the simultaneous equation model, so many classical simultaneous equation estimators will be difficult to applied, especially the two stage least squares method. Outliers are typical observations that are greatly different from group of observations. They can also make the models to have high error rates and substantial distortions for both parameter and statistic estimates ^[1].

In recent times, attention has been given to the problem of outliers and multicollineaarity in both classical and Bayesian econometrics especially in regression models ^[2]. There is not so much research on outliers in simultaneous equation models but there are limited researches on multicollinearity in simultaneous equation models. Major works carried out on the outliers in simultaneous equation models are Mishra and Adepoju and Olaomi while researches on multicollinearity in simultaneous equation models are Schink and Chiu, Agunbiade and Iyaniwura, Agunbiade, Mishra, Ozbay and Toker ^[3].

Mishra proposed a robust method that generalizes the two Stage Least Squares (2SLS) to the Weighted two Stage Least Squares (W2SLS) to tackle the effect of outliers and perturbations in data matrix. Monte Carlo method experiment was conducted to examine the performance of the proposed method in simultaneous equations. It was found out that the robustness of the proposed method did not disrupt the magnitude of outliers but sensitive to the number of outliers in the data matrix ^[4].

The performance of five estimators; Ordinary Least Squares (OLS), two Stage Least Squares (2SLS), three Stage Least Squares (3SLS), Generalized Method of Moment (GMM), and Weighted two Stage Least Squares (W2SLS) of simultaneous equations model parameters with first order autocorrelation levels of error terms and there is outliers in the data at small sample sizes were considered by Adepoju and Olaomi. It was observed that the system method performed better than the single equation for all the cases of outliers considered ^[5].

Agunbiade and Iyaniwura investigated the performance of six different estimation techniques of a just identified simultaneous three equation model with three multi collinear exogenous variables. The estimators considered under the three levels of multicollinearity were Ordinary Least Squares (OLS), two Stage Least Squares (2SLS), three Stage Least Squares (3SLS), Limited Information Maximum Likelihood (LIML), Full Information Maximum Likelihood (FIML), and Indirect Least Squares (ILS). It was revealed that 2SLS, LIML and ILS estimators were the best for lower open interval negative level of multicollinearity while FIML and OLS were best for closed interval and upper categories level of multicollinearity ^[6].

Agunbiade investigated effect of multicollinearity and sensitivity of three estimators in three just identified simultaneous equation model with the aid of Monte Carlo approach. The study was estimates using mean of estimates and its bias. However, the study revealed that identical estimates as the assumed parameter was not produced but some estimates are quite close. The use of shapely value regression at the second stage of two-stage least squares for simultaneous equation when there is collinearity was also proposed by Mishra. It was observed that all the structural coefficients estimated with the proposed two-stage least squares have an expected sign and can help to overcome the problem of collinearity ^[7].

A biased estimation method was proposed by Ozbay and Toker to remedy the problem of multicollinearity that exists in simultaneous equations model. Two parameter estimation in linear regression model is carried out to the simultaneous equations model. Monte Carlo experiment and real life data were used to evaluate the proposed method. The performance of the estimation of the new method is better than the conventional two stage least squares estimator ^[8].

Many researchers have provided alternative solutions like M-estimator for outliers. However, these methods cannot be applied to outlier and multicollinearity when they occur in the data at the same time. In this study, we consider the two problems together, that is, multicollinearity and outliers in simultaneous equation model by using a Bayesian method of estimation. The performance of Bayesian method will be then be compared with some classical Simultaneous equation methods in the presence of outlier and multicollinearity ^[9,10].

Materials and Methods

Consider the following two equations structural model;

Where, y_1t and y_2t are the endogenous variables at time t and x_1t,x_2t and x_3t are the exogenous or predetermined variables. The u_1tand u_2t are the random disturbance terms assumed to be independently and identically normally distributed with zero means and finite variance-covariance matrix Σ i.e., u~NID(0,Σ). Also β₁₂, β₂₁, γ₁₁, γ₁₂, γ₂₂ and γ₂₃ are unknown population parameters of the model.

In this section, Bayesian method for solving multicollinearity and outlier problems will be given while different classical methods of solving simultaneous equation will be examined. The simultaneous equations given in (1) and (2) can further also be written as:

Rearranging, we have;

In matrix form;

This can be written in reduced from as:

Where

Classical methods

Ordinary Least Squares (OLS): This method can be applied in regression model. If OLS is applied to an equation in a simultaneous model, there will usually be more than one current endogenous variable in a relation, and whichever variable is selected as dependent variable, the remaining endogenous variables which are correlated with the disturbance term will appear in the equation as explanatory variables. Hence, OLS estimates will be biased and inconsistent when used in simultaneous equation model, but will be applied in this study for comparison purpose ^[11].

Two Stage Least Squares (2SLS): This method is the most common method used for estimating simultaneous equation models. It was developed by Theil and independently by Basmann. It is applicable to equations, which are over identified or exactly identified and it is a single equation method, being applied to one equation of the system at a time. It is to eliminate the simultaneous equation bias ^[12].

Three Stage Least Squares (3SLS): It is a systems method, that is, it is applied to all the equations of the model at the same time and gives estimates of all the parameters simultaneously. This method was developed by Theil and Zellner as a logical extension of Theil’s 2SLS. It involves the application of the method of Least Squares in three successive stages. It utilizes more information than the single-equation techniques, that is, it takes into account the entire structure of the model with all the restrictions that the structure imposes on the values of the parameters ^[13].

Limited Information Maximum Likelihood (LIML): This is a single equation method which makes use of the principle of Maximum Likelihood. It is a “limited information” method because it does not make full use of the information provided by the equations of the model other than those of the particular equation under consideration. The limited information it requires on the other equations of the model, is the specification of all the truly exogenous variables that are contained in those other equations. It is an appropriate method for estimating over identified models ^[14].

Bayesian method

In Bayesian method, it is better to work on reduced form rather than the structural form due to prior elicitation and identification problem.

Recall from equation (9),

The likelihood function is the principal to the process of estimation of unknown parameters in Bayesian analysis. Using the definition of multivariate Normal distribution, the likelihood can be written as:

Prior distribution

When there is absence of prior knowledge, using a non-informative prior in Bayesian inference can be of great value. Here, we use a diffuse prior introduced by Jeffreys, which is given as:

Posterior distribution

The posterior distribution summarizes what we know about uncertain quantities. It gathers all the evidence or information that has been taken into account by prior distribution. Hence, it combines both the likelihood and prior distribution ^[15].

Therefore, the joint posterior density is proportional to the likelihood times prior and can simply be written as:

While the conditionals densities forms are given in equations (18) and (19):

Equations in (18) and (19) are normal and inverse Wishart distributions

In order to obtain the point estimate from the posterior density functions, we will solve the conditional posteriors given in equations (18) and (19), these can be achieved by using the widely used method called Markov Chain Monte Carlo (MCMC).

Monte Carlo experiment

In this section, a Monte Carlo experiment will be setup to facilitate comparison between the Bayesian method and classical methods in simultaneous equation model that is characterized by multicollinearity and outliers. The steps of the experiment are outlined below:

• Generate the exogenous variables x_1t,x_2t and x_3t for each sample point. Here, the variables will be generated from the uniform distribution (0,1) Kmenta and Ojo and Adepoju. These exogenous variables are characterized by multicollinearity and outliers. The levels of multicollinearity are: High Multicollinearity (HM): 0.95 and 0.99 Low Multicollinearity (LM): 0.2, and 0.40 While the scenarios of outliers are: 0%, 10% and 20%
• The initial values of the parameters are chosen arbitrarily given as: β₂₁=0.5, β;₁₂=0.8, γ₁₁=1.5, γ₁₂=1.0, γ₂₂
=1.0, γ₂₃=1.0
• The disturbance terms, u_1t and u_2t will also be generated at each sample point.
• The disturbance terms and exogenous variables will be used to generate the endogenous variables.
• The sample sizes considered are 15, 50, and 100 while each of the samples is replicated 10000 times.

Results and Discussion

This Section discusses the results from the Monte Carlo experiment described in section 4. The MSE and ABIAS for estimators namely; Bayesian, two stage least squares, three stage least squares, limited information maximum likelihood, and ordinary least squares are obtained in Tables 1 and 2. It was observed that the estimates of two stage least squares, three stage least squares, and Limited information Maximum likelihood are the same, hence they are represented by 23LIML while Bayesian is represented by Bayes ^[16].

Table 1. Absolute Bias of the estimators with varying sample sizes for collinearity and outliers.

Table 2. MSE of the estimators with varying sample sizes for collinearity and outliers.

From Table 1, the Bayes method has the least absolute bias followed by 23SLS while the OLS method has the largest absolute bias for the two equations for all the levels of collinearity. It was also observed that the bias estimates decreases the sample sizes increases for all the methods considered across the levels of collinearity. The ABIAS obtained in both equations 1 and 2 for 10% and 20% are higher than when there no outlier ^[17-20]. The ABIAS estimates for equation 1 are smaller than equation 2 for all the sample sizes considered across both levels of outliers and collinearity ^[21].

In Table 2, it is observed Bayes method gives the minimum MSE for the entire sample sizes considered followed by 23LIML while OLS has the highest MSE. All the methods are not greatly affected by outliers; however when the percentage of contamination goes to 10% and 20%, the MSE of the estimators increases. For low level of collinearity, the MSE are minimal.

Conclusion

Multicollinearity and outliers are great problems in applied work. This work determined the best method of estimation when a just identified simultaneous equation model has both problem of multicollinearity and outliers. The method considered were Bayesian, two stage least squares; three stage least squares, and Limited information maximum likelihood. When there is no outliers, all the estimators have the same performances, however when the levels of outlier were 10% and20%, the estimates of the estimators increases. The Absolute bias and Mean squared estimates of the estimators increases as the level of collinearity also increases. Also, all the methods considered show consistent asymptotic pattern with values of absolute bias and mean squared error decreasing consistently. Bayesian method of estimation is considered the best estimator when a just identified simultaneous equation model has both problem of multicollinearity and outliers.

Conflicts of Interest

There is no conflict of interest

Funding

There is no funding.

References

1. Adepoju AA, et al. Bayesian method for solving the problem of multicollinearity in regression. Afr Stat. 2018;13(3):1823-1834.

   [Google Scholar]

2. Adepoju AA, et al. Evaluation of small sample estimators of outliers infested simultaneous equation model: A Monte Carlo approach. J Appl Econ Sci. 2012;7(1):8-16.

   [Google Scholar]

3. Agunbiade DA, et al. Effect of multicollinearity and sensitivity of estimation methods in simultaneous equation model. J Mod Math Stat. 2011;5(1);9-12.
4. Agunbiade D, et al. Estimation under multicollinearity: A comparative approach using Monte Carlo methods. J Math Stat. 2010;6(2):183–192.

   [Google Scholar]

5. Brikes D, et al. Alternative methods of regression. John Wiley and Sons. New York, USA. 1993.

   [Google Scholar]

6. Datta GS, et al. Some remarks on non-informative priors. J Am Stat Assoc. 1995;90(432):1357-1363.

   [Google Scholar]

7. Duzan H, et al. Ridge regression for solving the multicollinearity problem: Review of methods and models. J Appl Sci. 2015.

   [Google Scholar]

8. Hoerl AE, et al. Ridge regression: Biased estimation for non-orthogonal problems. Technometrics. 1970;12:55-67.

   [Crossref] [Google Scholar]

9. Jadhav NH, et al. A jack knifed ridge M-estimator for regression model with multicollinearity and outliers. J Stat Theory Pract. 2011;5(4):659-673.

   [Crossref] [Google Scholar]

10. Jeffreys H, et al. An invariant form for the prior probability in estimation problems. Proc R Soc Lond A Math Phys Sci. 1946;186:453-461.

    [Crossref] [Google Scholar] [PubMed]

11. Kang SG, et al. Noninformative priors for the common mean in log-normal distributions. J Korean Inf Sci Soc. 2011;22(6):1241-1250.

    [Google Scholar]

12. Kmenta J, et al. Elements of econometrics. Macmillian Press Ltd. New York, USA. 1971.

    [Google Scholar]

13. Mishra SK, et al. Robust two stage least squares: Some Monte Carlo experiments. J Appl Econ Sci. 2008;3:434-443.

    [Google Scholar]

14. Ojo OO, et al. A simulation study on the performance of Bayesian and L2 regularization methods in multicollinearity problem. Ann Comput Sci Ser J. 2020;18(1):143-147.
15. Ojo OO, et al. Evaluation of simultaneous equation in the presence of misspecification error: A Monte Carlo approach. Eur J Bus Manag. 2013;6(37):257-260.
16. Ozbay N, et al. Multicollinearity in simultaneous equations system: Evaluation of estimation performance of two parameter estimator. Comput Appl Math. 2018;37(4):5334-5357.

    [Crossref] [Google Scholar]

17. Percy DF, et al. Bayesian analysis in statistics and econometrics: Essays in Honor of Arnold Zellner. John Wiley and Sons. Newyork, USA. 1996.

    [Google Scholar]

18. Schink WA, et al. A Simulation study of effects of multicollinearity and autocorrelation on estimates of parameters. J Financial Quant Anal. 1996;1(2):36-67.

    [Google Scholar]

19. Shariff NS, et al. An application of robust ridge regression model in the presence of outliers to real data problem. J Phys Conf Ser. 2017;890:1-7.

    [Crossref] [Google Scholar]

20. Zellner A, et al. Gibbs sampler convergence criteria. J Am Stat Assoc. 1995;90:921–927.

    [Google Scholar]

21. Zimmerman DW, et al. Invalidation of parametric and nonparametric statistical tests by concurrent violation of two assumptions. J Exp Educ. 1998;67:55-68.

    [Crossref] [Google Scholar]