An Analysis for Measure of Effectiveness of an Unmanned Aerial Vehicle Using Simulation

Abstract

As an UAV (Unmanned Aerial Vehicle) costs less for producing and operating than a manned aircraft, and it is widely used and efficient, it has been developing by many countries. Its tactical worthiness is highly evaluated in the field of military ISR (Intelligence, Surveillance & Reconnaissance) operations. For applying the M&S to the acquisition, the previous methodologies to measure the effectiveness of the weapon system like an UAV showed limitations of containing subjective and qualitative factors. So, new methodology using simulation for measuring performances of weapon system inputted combat circumstances and operational concept is needed. This study is to develop a new methodology for measuring and analyzing the effectiveness of an UAV. In order to decide the factors for evaluating effectiveness, literature reviews, experts’ interviews, surveys and factor analysis were conducted. With the selected factors, an experiment design for simulation was organized. With the developed simulation, MOEs (Measures of Effectiveness) were drawn depending upon effectiveness on an UAV. In addition, meaningful results for better effectiveness of UAV were gotten by analyzing through multiple linear regression and Structural equation modeling. As applied the result of the analysis to the improved scenario, around 29% increase of operating effectiveness was shown in the simulation.

Keywords

Unmanned aerial vehicle, Effectiveness of measure, Simulation analysis, MOE (Measures of Effectiveness), Factor analysis, Multiple linear regression, Structural equation modeling.

Introduction

The definition of UAV is a powered, aerial vehicle that does not carry a human operator, uses aerodynamic forces to provide vehicle lift, can fly autonomously or be piloted remotely, can be expendable or recoverable, and can carry a lethal or non-lethal payload (US DOD Dictionary, 2010). It is recently used as a UAS (Unmanned Aircraft System) based on the acknowledge that is needed the total system comprised of the unmanned aircraft, payload, human element, control element, weapons system platform, display, communication architectures, life cycle logistics, and the supported soldiers after a variety uses of words, UAV (Unmanned Aerial Vehicle or Uninhabited Aerial Vehicle), RMS (Remotely Manned System), RPV (Remotely Piloted Vehicle), RCS (Remotely Controlled System), Drone and Flying Robot (US DOD, 2010).

As Unmanned Aerial vehicles cost less for producing and operating than manned Aerial Vehicle and it is widely used and efficient, it has been developed by many countries. Its tactical worthiness is highly evaluated in the field of military ISR (Intelligence, Surveillance and Reconnaissance) missions [1]. The effectiveness analysis of weapon systems has been studied for developing more efficient weapons in DoD, and various methodologies to quantify subjective concepts like maneuver, fire, survivability and C4I were obtained and applied for the studies. For instance, there were methodologies using WEI (Weapon Effectiveness Indices) and WUV (Weighted Unit Value) but they showed limitations of containing subjective and qualitative factors [2]. In order to avoid these limitations, several methods to quantify were created, and M&S using simulations for measuring performances of weapon systems inputted combat circumstances and operational concept are widely utilized. A new methodology using M&S is needed for acquiring weapon systems as “the guidance of applying M&S to acquisition systems” became law in September, 2010.

UAV has used for the “3D” missions because of the “Unmanned” characteristic. “3D” is an abbreviation of the “Dull, Dirty, Dangerous”. For example, UAV operates the long period Surveillance and Reconnaissance missions as a “Dull” missions, the observation and data collection missions under the air pollutions due to chemical or radiological as a “Dirty” missions and the ISR (Intelligence, Surveillance and Reconnaissance) and SEAD (Suppression of Enemy Air Defenses) operations over the enemy’s dominated area as a “Dangerous” missions. UAV systems which are already sufficiently demonstrated the usefulness in the "3D” missions gradually replaced out the mission area of the manned aircraft [3]. Since the first UAV have been succeeded in the joint experiments with the U.S. Army and civilian engineers in the military base, Iowa in 1914 as a mechanical device by the vehicle, it has been continuously expanding the utilization of the area and needs for an UAV by the development of science and technology [4]. In military, UAVs have proved its effectiveness as “Eyes of the battlefield” through several warfare like Lebanon (1982), Kosovo (1999), Afghanistan (2001), Iraq (2003) [5]. For civilian purpose, it’s potential and direct demands exist in a variety of fields like a scientific measurement, monitoring illegal activities, wildlife monitoring/protections, maritime missions and utilizing satellite replacements including agricultural support. And it is being used as well as the actual.

The definition of defense M&S (Modeling & Simulation) is the related models, tools and all activities in order to solve the Defense-related issues by the scientific analysis (KIDA, 2009). As the utilization of M&S is emphasized recently, the application of M&S is used for widely Weapon systems acquisition, Simulation game for training and combat power analysis. In the future, it will be the effective decision making tool [6]. The concept of M&S is used as SBA (Simulation Based Acquisition), Simulation Cased Design, SMART (Simulation & Modeling for Acquisition, Requirement and Training), STEP (Simulation based Test & Evaluation Program) and Battle Experimentation [7]. According to the legalizing “the Guidance of applying M&S to acquisition systems,” the application of M&S in the acquisition system is obligated. SBA (Simulation Based Acquisition) is positively developed in these circumstances. SBA is scientific and systematic acquisition activities performed the simulation analysis and verification using DM&S methods or tools Trough the entire life cycle of weapon systems. The expected effects are risk management, optimal Performance, reduction of the cost and period [8]. For applying the M&S in the acquisition systems as SBA concept, the Indicator used widely is MOE (Measures of Effectiveness). MOE is a qualitative or quantitative measure of the performance of a model or simulation or a characteristic that indicates the degree to which it performs the task or meets an operational objective or requirement under specified conditions (Defense Modeling and Simulation Office Online M&S Glossary, 2005).

As reviewing the list of simulation tools and analytic methods, each research uses the different tools and analytic methods according their own purposes. For the simulation tools, MANA (Map Aware Non-Uniform Automata) agent-based model used several times in international researches. And there are not any major simulation models in domestic model. Several models like a System Dynamics model and ARENA model used for their own purposes of each research. For analytic tools, Multiple Linear Regression used many times in both international and domestic researches. In order to apply the simulation model for our UAV, the simulation models which can reflect the Korean warfare condition like a terrain and weather are needed. The simulation model used in previous research was made by the foreign military or company. Some limitation should exist those simulation models. The advanced analytic methods are needed for analyzing the simulation data. In the previous research, they considered the effect of each attribute to MOE due to the capacities of analytic methods. Even though Multiple Linear regression is a useful and effective statistical method to analyze the effectiveness, it cannot consider how the patterns grouped by similar attributes affect the effectiveness of an UAV. So, simulation tool reflected Korean warfare condition and advanced analytic method which can measure the factor’s effectiveness should be developed for improving of a methodology to measure and analyze the MOE of an UAV. Therefore, the purpose of this paper is to develop a new methodology for measuring and analyzing an UAV operating effectiveness, which is required for developing a future essential Army Tactical UAV. Factors of quantifying UAV operating effectiveness were selected from previous studies, the model for assessing operating effectiveness was set, the UAV operating effectiveness was analyzed by simulation, and finally meaningful conclusions were drawn.

Research Methodology

Framework

In order to measure and analyze MOE of an UAV, the methodology is developed by the procedure like (Figure 1). The procedures are divided by 3 steps – Factor selection, Simulation & Data analysis and Validation. In step 1, MOE attributes are identified by the literature reviews and Experts’ interviews. Then, MOE factors are determined by the factor analysis by using MOE attributes reviewed in previous procedures. In step 2, the simulation is conducted through the experiment design using the MOE factors determined and data analysis is conducted by the advanced statistical method like multiple linear regression and Structural equation modeling using the simulated data. In step 3, the validation is conducted by comparing the result between previous scenario and improved scenario used the result of the data analysis.

Factor selection

Identifying the MOE attributes

There are many attributes affecting the UAV’s effectiveness used in the previous researches. Many attributes are also used to measure the effectiveness of an UAV. In US Army, they highlight areas of UAV testing that have proven problematic from DOT&E’s perspective in past UAV operational tests. Armed with this knowledge, they insisted “Value of Information”, “Timeliness”, “Quality of Imagery”, “Detection, Classification, Recognition Criteria”, “Target Location Error Criteria” and “Survivability Criteria” are main factors for UAV’s effectiveness [9]. But we need the attributes for adjusting the Korean battle condition (terrain, weather and so on) and implying our own development procedures for an UAV. There are several researches to measure and analyze the effectiveness of an UAV using simulation. From those researches 28 attributes are extracted as follows (Table 1) to identify the attributes affecting the UAV’s MOE [10-17]. Those attributes can be the pools of the MOE measuring factors. Through the selection process like the concept analysis or experts interviews those can be extracted and added to find essential attributes affecting the effectiveness of an UAV.

Table 1: Summaries of literature review for MOE of an UAV.

Determining the MOE factors

In order to determine the MOE factors to use for experiment design, the Factor analysis can be conducted. Exploratory factor analysis (EFA) is generally used to discover the factor structure of a measure and to examine its internal reliability. EFA is often recommended when researchers have no hypotheses about the nature of the underlying factor structure of their measure. Exploratory factor analysis has three basic decision points (1) deciding the number of factors, (2) choosing an extraction method, (3) choosing a rotation method (Kim & Mueller, 1978). Using the selected attributes by selection process the survey of the researchers and operator for the UAV could be conducted to extract the meaningless attributes and select the factors to explain the UAV’s attributes. The meaningless attributes are extracted by the Eigen-value and Factor loading from the results of the factor analysis. And the attributes showed the regular patterns made by groups and renaming the factors to represent the attributes in the factor.

Simulation and data analysis

Simulation: A simulation is a tool to evaluate the performance of the system existing or proposed under different configurations of interest and over long periods of real time. It is used before an existing system is altered or a new system is built to reduce the chances of failure to meet specifications, to eliminate unforeseen bottlenecks, to prevent under or over-utilization of resources, and to optimize system performance [18]. A simulation is conducted for getting the UAV’s data by using the attributes resulted by the Factor analysis.

Before conducting a simulation, designing the experiment is needed. An experiment design plays an integral role in the conduct of the simulation study. At first the variations of factors should be set. The simulation may use factors. Each factor can be set to two or more attributes. A design point consists of the specification of each factor in the study. There are numerous parameters in the simulation model that can be varied to explore the effects on the performance. Due to computational limitations, a subset of these parameters, that is deemed to be most likely to cause major effects on the selected MOEs, is chosen as factors for the experiment design. Reasonable ranges of these factors are then determined. Current capabilities of the agents are extended to determine the maximum ranges for the factors [19].

Secondly, a selecting of the experimental levels and points should be set. Minimum and maximum levels of these factors form the frame of the experimental region. To explore the main effects and the interactions of the factors, we need to select points to sample within that experimental region [20]. Despite the technological improvements on computers, it is still practically impossible to run a model for all possible points in an experimental region. Therefore, we need sophisticated techniques to find an efficient number of design points, which together allow for maximum information to be gained from the experiment. According to Sanchez, Latin hypercube (LH) designs provide a flexible way of constructing efficient designs for quantitative factors and have some of the space-filling properties of factorial designs, but require orders of magnitude less sampling [21].

• The UAV’s attributes (variations of factors, characteristic, etc.)

• The enemy’s attributes (variations of factors, characteristic, etc.)

• Target’s attributes (number, type, camouflage, etc.)

• The environmental factors (terrain, fog, temperature, etc.)

• The restriction by the computation (limitation of factors, etc.)

Data analysis

After running the model, data sets are acquired for MOEs. Several statistical methods can be used for the analysis of the data set. But, the appropriate methods for this analysis are MLR (Multiple Linear Regression) and SEM (Structural equation modeling). MLR can be used for the analyzing the effectiveness of each attribute to MOEs and SEM can be used for analyzing the effectiveness of the factors to MOEs. MLR is a common method of determining factor effects on a response variable. It involves applying linear combinations of the coefficients of the factors that predict the response variable by minimizing errors. Minimizing the error term produces an accurate fit of the response based on the factors. Various statistical packages are available for facilitating multiple regression analysis. Although MLR gives coefficient estimates for each significant factor, the focus here is on the relative importance of each factor and not value of the coefficient. The response is a probability and is therefore limited to values between zero and one. SEM grows out and serves purposes similar to multiple regression, but in a more powerful way which takes into account multiple latent independents each measured by multiple indicators, one or more latent dependents also each with multiple indicators, the modeling of mediators as both causes and effects, modeling of interactions, nonlinearities, correlated independents, measurement error, and correlated error terms. SEM may be used as a more powerful alternative to multiple regression, path analysis, factor analysis, time series analysis, and analysis of covariance [22].

Validation

In order to verify the result of data analysis, the validation should be conducted. Although the methods are various, the method presented in this paper is a comparison validation method. In this method, the improved scenario reflecting the results of data analysis is developed for comparison method. By the conducting of the simulation under the improved scenario, new results of MOEs came out. Comparing the result of the previous scenario with the result of the improved scenario, we can verify the data analysis.

Results

Case study–army tactical UAVCase study–army tactical UAV

This case study is conducted targeted on Army Tactical UAV for applying the new methodology developed in this research. The processes of the methodology are divided by three parts: factor selection, simulation, and data analysis and validation.

Factor Selection

In order to select factors for measuring UAV operating effectiveness, literature reviews on UAV operating effectiveness and experts interviews were conducted. The following (Figure 1) is the process of factor selection. For the first step, forty-three different attributes were chosen from the literature reviews on UAV operating effectiveness, and for the second, redundant thirteen attributes were eliminated and final twenty-eight attributes remained. For the third step, fourteen attributes were obtained by experts’ interviews with UAV researchers and the field operators and the simulation was set with sixteen attributes by adding two attributes. For the fourth step, statistical meaningful ten attributes were added by surveying UAV researchers and field operators and by analyzing factor analysis [23].

Identifying the MOE attributes

The forty-three attributes were selected and they were used for independent variables from former studies on UAV operating effectiveness. The following (Table 2) is for the factors. Fifteen attributes were gotten rid of as redundant or same attributes and the remaining attributes were “Quality of Imagery” was for “EO/IR range,” “Sensor range” and “Sensor System.” As the same concepts; “Classification range” for “Detection range,” “classification Probability” for “P(Identification),” “P(Classification)” and “Rate of Classification,” “Payload” for “Payload weight,” “altitude” for “Ceiling,” “Air speed” for “Max speed,” and “Loiter speed,” “Susceptibility to weather” for “weather,” “Endurance” for “Duration of flight,” and “Operation range” for “Mission plan” and “Mission radius.” The twenty-eight attributes after the first filtering was classified to seven categories by each purpose. Seven categories are “Vehicle ability” was for “Air Speed,” “Endurance,” “Altitude,” “Operational Range,” “Payload” and “Fuel Capacity.” “Sensor ability” was for “Quality of Imagery,” “Classification Range,” “Classification Probability,” “Sweep Width,” “FOV," and “Target Location Error.” “Survivability” was for “Accident rate,” “Stealth,” “MTBF (Mean Time between Failure),” “MTTR (Mean Time to Repair)” and “Survivability.” “Data link” was for “Link Reliability,” “Message Latency,” “Target Process Capacity” and “Data link.” “Environmental effect” was for “Susceptibility to Weather,” and “Terrain.” “Etc.” was for “Reactivity,” “Value of Information,” “Waypoint Attribute,” “Maneuverability” and “Timeliness.”

Table 2: The list of attributes affecting the effectiveness of an UAV.

The second filtering was done by experts’ interviews. It was conducted for two weeks from the 3rd to the 21st of October from the experts in DAPA (Defense Acquisition Program Administration), ADD (Agency for Defense Development), KIDA (Korea Institute for Defense Analyses), DTaQ (Defense Agency for Technology and Quality) and Corps-level Intelligence Battalion. Based on the interviews, fourteen attributes were added, two attributes were modified and two attributes were included. Attributes from the second filtering were “Fuel capacity,” “Sweep width,” “MTBF,” “MTTR,” “Survivability,” “Message latency,” “Target process capacity,” “Data link,” “Weather,” “Terrain,” “Reactivity,” “Value of information,” “Waypoint attribute,” “Maneuverability,” “UAV availability” and “Timeliness.” Modified two attributes were “Return rate” for “Accident rate” and “Radar detection rate” for “Stealth.” “Anti-EA capability” and “Temperature” were additionally included as they were considered important these days. The final sixteen attributes were like the Table 2 by filtering two times to get twenty-nine out of forty-three and adding two attributes.

Determining the MOE factors

The survey was conducted from 17th to 21st, OCT during one week by visiting. The subjects were UAV researchers and filed operators. Researchers were the people worked in Army intelligence school, DAPA, ADD, KIDA and DTaQ and Field operators were the soldiers in 3 of 6 UAV company in Army [24].

Survey was marked by 7-Likert Scales. “1” score means “not effective” and “7” score means “most effective.” We can get the 96 samples (86%) after eliminating the samples without reliability. Samples composed the 28 researchers (29%) and 68 field operators (71%). About the working period for UAV, “below 1 year” was 11%, “less than 2 years, more than 5 years” was 14%, “less than 6 years, more than 10 years” was 49%, and “more than 11 years” was 26%. The average of working period is about 7.3 years. Exploratory factor analysis was conducted with these survey results. See Figure 2 for the average scores of factors.

The ranges are minimum “5.81” and maximum “6.58,” the decending orders are “Quality of imagery,” “Endurance,” “Susceptibility to weather,” “Data Link,” “Classification Probability,” “Classification Range,” “Location Error,” “Altitude,” “Operational Range,” “Anti-EA capability,” “Return Rate,” “Radar detection rate,” “Air Speed,” “FOV,” “Payload” and “Temperature.” Six independent variables are eliminated by the refining process for factors. For exploratory factor analysis, Principle Component Analysis and Varimax Orthogonal rotation are used. The criteria of this analysis are Eigen value ≥1.0, communality ≥ 0.60, Factor loading ≥ 0.50. By the four times analysis, “Payload,” “Target location error,” “Temperature,” “Return rate,” “Link reliability” and “FOV” is eliminated. Ten of sixteen variables are used for exploratory factor analysis. The Measurement variables are grouped by 4 factors as followed (Table 3). “Quality of Imagery,” “Classification Range,” “Classification Probability,” and “Susceptibility to Weather” variables are converged as 1st factor. “Radar Receiving rate” and “Anti-EA capability” variables are converged as 2nd factor. “Air Speed” and “Altitude” variables are converged as 3th factor. “Operational Range” and “Endurance” variables are converged as 4th factor. The Eigen-values of all variables are over 0.80 except “Altitude (0.747)” and “Endurance (0.763).” the values of communality are also high (over 0.80). The accumulated Distributed Descriptions are 85.495%. Namely, the 10 variables are well explained the model as over 85%. By the result of factor analysis, 4 factors within 10 attributes are selected by the statistical method as the factor influencing the measures of effectiveness of UAV. 4 factors are named as Vehicle ability, Sensor ability, Tactical operation, Electronic warfare as followed (Figure 3). The attributes are used for the indicator of Measures of effectiveness of UAV. “Vehicle ability” factor can be used for measuring how much the aircraft’s performance effects to the UAV’s mission achievements. “Sensor ability” factor can be used for measuring how much the sensor’s resolution of UAV effects to the UAV’s mission achievements. “Tactical operation” factor can be used for measuring how much operational tactics like routing effects to the UAV’s mission achievements. “Electronic warfare” factor can be used for measuring how much the ability for survivability from the electronic warfare effects to the UAV’s mission achievements.

Table 3: The result of factor analysis.

Simulation and data analysis

The experiment design was conducted for the simulation. In the experiment design, the attributes extracted by factor selection processes are used as the independent variable and dependent variable. The simulation was conducted by the scenarios designed for measures of effectiveness of an Army tactical UAV. After the simulation, data analysis was conducted using statistical method. As the meaningful results from the data analysis, the improved scenario was designed for the validation, and compare with the previous scenario.

UAV simulation analysis model

UAV Simulation model was developed for measuring the operational effectiveness of Army tactical UAV based on the UAV characteristics, the terrain and weather of the Korean peninsula, warfare condition of Korean military in 2003. Army uses the simulation model to generate the ROC (Requirement of Capability) of Corps, Division and Battalion-level UAV and develop the tactic for an UAV’s routing and optimal altitude and way point under the specific condition.

The detection algorithm of UAV Simulation Model uses the one of the standard approved by U.S. DoD, “Air STD 80/15 <Minimum Resolved Object Sizes for Imagery Interpretation>”. This standard defined the process of image detection of satellite image and the target’s minimum size of the representative subjects. This standard set for satellite image but there is validity because it is as same as the process of UAV [25]. See the Figure 4 for the target detection algorithm of UAV simulation model.

Experiment design

The scenario: The simulation tool used in this research is UAV Simulation Analysis Model. It is used for Army tactical UAV to generate the ROC and Developed by Model Sim. This model applies the UAV’s flight characteristics by the strict physical model, implies the enemy forces’ arrangements that simulate the battle conditions, and reflects the Korean weather and terrain for using the scie ntific analysis tool.

The setting of this scenario is that a middle of peninsula, “Yeo-ju” mixed of the mountains, rivers and roads. The area of operation is 25,000 m × 25,000 m and there are 5 targets in each 5,000 m × 5,000 m. The target is “T-59”, the major Tanks in North Korean military. The assumptions for this scenario are follows.

• The area of operation for UAV to take the ISR missions is 25km square area, which distributed rivers, mountainous terrain, roads irregularly.

• The purpose of the ISR missions in this Area of operation is to obtain the information about enemy’s locations and imagery information.

• In this ISR mission, UAV takes the “Area reconnaissance” scout methods and has “2km” sweep widths.

• The flight time of UAV is the time to conduct the aerial reconnaissance over the enemy’s area of operation.

• The enemies in Area of Operation are to prepare the assault operations and concentrated in the tactical assembly area.

The variations

The simulation environment allows a model to manipulate a large range of factors, providing superior flexibility in design implementation. Each of the chosen factors is then varied to assess their impact on UAV effectiveness. The factors varied in this research can be broken into two groups, controlled and uncontrolled. Controlled factors are those that are directly controllable by the user. For example, UAV speed can be controlled in both the design stage and by the operator during use. Uncontrollable factors either not be influenced by or are unknown to the user. These are usually with enemy traits, such as camouflage, capability. Both controllable and uncontrollable factors are varied here [26].

The factors in this research are chosen by the result of selection process. “Air speed” and “Altitude” factors in vehicle ability and “Quality of imagery” and “susceptibility to weather” in sensor ability are chosen in this scenario. And as an uncontrolled factor, “enemy camouflage” factor is included in this scenario. The variations of factor are shown in the Table 4. The effectiveness of the screening experiment is measured according to the probability of classified enemy per each scenario. Since the number of enemy agents is 125, the sum of the classification percentages divided by 125 is the empirical probability of enemy classification in the area. The Army commander wants to maximize this probability.

Table 4: The variations of the factors for simulation.

Orthogonal latin hypercubes design

There are a large number of factors worthy of consideration between the controllable and uncontrollable factors. A problem arises in attempting to effectively vary these factors across a wide range of possible levels. A traditional factorial experimental design tests only a few factors at two or perhaps three levels such. To utilize this approach, some factors would have to be left out of the experimental design and linear relationships assumed. A smart design is required.

An Orthogonal Latin hypercube(OLH) design is chosen for its excellent space filling properties, the resulting low correlation between factor inputs, and ability to identify nonlinear relationship. OLHs can be used to design an experiment evaluating up to seven factors at 17 levels each in an efficient and effective manner. Nearly Orthogonal Latin hypercubes (NOLHs) have nearly the same properties with slightly higher, but negligible, correlation between factors. NOLHs can be utilized to evaluate from 8 to 22 factors at up to 129 levels.

The orthogonal nature of the design results in no significant design imposed correlation. This provides the ability to look at the effects of each variable independently as well as interactions during analysis. Through optimal chaining of OLHs or NOLHs, the space filling characteristics can be increased while maintaining the orthogonal nature of the design and no significant design point correlations. This provides a greater ability to analyze multidimensional data.

This study uses an orthogonal Latin hypercubes for factors controlled. This provides 17 design points for controlled factor optimized for great space filling. For uncontrolled factor 11 levels are applied. This results in 17 times 11=185 design points which are each run for the 5 factorial cases of 9 kinds of sensors (4.3mm, 6mm, 8mm, 12mm, 16mm, 20mm, 24mm, 68mm and 160mm). Finally these 1,665 design (Traditional way: 9,401,700 → OLH way: 1,665) points are each run for 5 iterations to take advantage of the stochastic nature of UAV simulation model. This results in 8,325 total runs of the scenario (Figure 5).

Data analysis

Multiple linear regression: Multiple Linear Regression (MLR) is a common method of determining factor effects on a response variable. It involves applying linear combinations of the coefficients of the factors that predict the response variable by minimizing error. Minimizing the error term produces an accurate fit of the response based on the factors. Various statistical packages are available for facilitating multiple regression analysis. Although MLR gives coefficient estimates for each significant factor, the focus here is on the relative importance of each factor and not value of the coefficient. The response is a probability and is therefore limited to values between zero and one. SPSS 18.0 is utilized for this research.

Using the simulated data, “classification rate” is used for the dependent variable and “Enemy’s camouflage,” “Altitude,” “Air speed,” “Wind speed,” “fog rate” and “Quality of imagery” factors are used for the independent variable. See the Table 5 for descriptive statistics of variables. For model fit, R-square and Durbin-Watson values are good. The R-square value in this model is 0.602. R-square is a measure of the variability explained by the regression model. This R-square value is not so high. But it achieved even though there are only six variables. The Durbin-Watson value is 1.522. The Durbin–Watson statistic is a test statistic used to detect the presence of a relationship between values separated from each other by a given time lag in the residuals from a regression analysis. The value is far from zero or four values and close to two values. And F value in this model is 63.641. Significant provability is 0.000. It can interpret that the model fit is good. See the Table 6 for the value of model fit. By the result of coefficients, the independent variables, “Air speed”, “Fog rate” and “Quality of imagery” are accepted (t>1.96, Sig<0.05). The others independent variables, “Enemy’s camouflage”, “Altitude” and “Wind speed” are not accepted due to the t-value and significant factor. There isn’t collinearity problem because all variables are over 0.10 values for the tolerance. “Enemy’s camouflage” has no meaning for this model. “Altitude” and “Air speed” have a negative effect to the Classification probability. But “Wind speed”, “Fog rate” and “Quality of imagery” have a positive effect for that. Especially “Quality of imagery” variable has quiet high relation with the classification probability. “Quality of imagery - Altitude - Wind speed – Fog rate - Air speed –enemy’s camouflage” is the descending order for the importance to classification probability. See the Table 7 for coefficient values.

Table 5: The descriptive statistics of variables.

Table 6: The Indicators for model fit of MRL.

Table 7:  The coefficients table (Quality of imagery < 16mm).

Multiple linear Regression equation is conducted by the coefficients values. The coefficient values can be used for the optimization problem for a UAV. For instance, the optimization of development costs for the payload equipment according to the vehicle or sensor ability (Classification range, FOV) with the MOE and operation costs (the fuel cost according the altitude or speed) with the MOE. Other finding in the comparing between the results of two groups for Quality of imagery. The coefficient of Q. imagery < 16 mm (Table 8) is a positive (0.027*) but that of Q. imagery>16mm is negative (-0.001**). So, the high quality of imagery does not produce the high classification probability. It can interpret that there is a pin point of quality of imagery to make highest classification probability

Table 8: The coefficients table (Quality of imagery > 16mm).

Structural equation modeling

Structural equation modeling (SEM) grows out and serves purposes similar to multiple regression, but in a more powerful way which takes into account multiple latent independents each measured by multiple indicators, one or more latent dependents also each with multiple indicators, the modeling of mediators as both causes and effects, modeling of interactions, nonlinearities, correlated independents, measurement error, and correlated error terms. SEM may be used as a more powerful alternative to multiple regression, path analysis, factor analysis, time series analysis, and analysis of covariance. AMOS 16.0 is utilized for this research [27].

The factors for Structural Equation Modeling in this research are extracted by the result of exploratory factor analysis in chapter 3. Two factors, “Vehicle ability” and “Sensor ability” are used for the latent variable. For “vehicle ability”, “Air speed”, “Altitude”, “Wind speed” and “Wind direction” was used for observed variable. For “Sensor ability”, “Enemy’s camouflage”, “fog rate” and “Quality of imagery” are used for observed variable. See Table 9 for the latent variable and observed variable.

Table 9: The latent variable and observed variable.

The simulated data used for this analysis. The typical indicator for the model fit was used for this research. The overall results of model fit were good except p-value. χ², cmin/df, TLI, CFI and RMSEA are above the optimum level. See the Table 10 for detailed result of model fit. The model for SEM was established as follows (Figure 6). For the model, “Vehicle ability” and “Sensor ability” factors are used for the latent variable. “Altitude,” “Air speed,” “Wind speed” and “Wind direction” factors are used for an observed variable for Vehicle ability” variable. “Enemy’s camouflage,” “Quality of imagery” and “Fog rate” factors are used for an observed variable for “Sensor ability” variable. Measure of effectiveness is defined by the “Classification probability” as the endogenous variable. The correlation relationship was set between two latent variables, “Vehicle ability” and “Sensor ability” variables. Each variable has the error term. And the variance value of error term “e8” is set as “0.005” due to the variance of “Classification Probability”. The model for this research is set for analyzing the degree of the effectiveness of “Vehicle ability” and “Sensor ability” to the measure of effectiveness of a UAV. As the result of SEM, the regression estimate value of “Vehicle ability” is -0.003 (p=0.850) and the regression estimate value of “Sensor ability” is 0.107 (p=0.059). So “Sensor ability” factors are more effective factors compared with “Vehicle ability” for the measure of effectiveness of an UAV. Even the standardized values of estimate of “Sensor ability” are higher than “Vehicle ability.” And the only “Sensor ability” has significant value in 95% confidence level. But SEM gives low coefficient estimates for each significant factor; the focus here is on the relative importance of each factor and not value of the coefficient. As values of the observed variable influenced the exogenous variables, the only “Altitude” variable in the “Vehicle ability” variable has significant values and the only “Quality of imagery” variable in the “Sensor ability” variable has significant value. The Squared multiple correlation of each variable are -0.284 (for “Altitude”) and 0.862 (for “Quality of imagery”). See the Table 11 for the coefficients of the variables.

Table 10: The indicators for model fit of SEM.

Table 11: The coefficients of the variables.

Summary of data analysis

To analyze the operating effectiveness of an UAV with simulated data, statistical methodologies like multiple linear regression and structural equation modeling were applied. The results can be explained as followings. For Multiple Linear Regression, the model fit is acceptable (R Square=0.362, Durbin-Watson=1.522, F=63.641 sig=0.000). The model also does not have collinearity (the tolerance values of all variables>0.90). A few findings were drawn through MLR results. First of all, quality of imagery was the factor that affects the operating effectiveness of an UAV most. According to the analysis result, there were three factors that were in statistical significant level, “Air speed,” “Fog rate” and “Quality of imagery.” However, “Quality of imagery” (0.027) was extreme high for the value of coefficient. That is, it can be assumed that higher “Quality of imagery” is the most significant factor to have more effective operating results of target classification probability of UAVs. Second, a proper rage of “Quality of imagery” is required to have the most effective operating results of an UAV. The values of coefficient of regressions with 16mm less of quality of imagery or with more were different. The value of coefficient of the former was 0.027 (p=19.168) and the latter was -0.001 (p=-15.413). That is, it is meaningful to increase quality of imagery to the certain level for more effective operating results of an UAV. It is not necessary, however, to increase the quality of imagery beyond the certain level. Therefore, a study is needed to find the certain level of the quality of imagery for the optimized results. Third, the optimization for the effectiveness of UAVs is possible using regression equation. New applications are possibly developed to optimize the UAV’s combat effectiveness in limited conditions by utilizing regression equation driven by data analysis [28].

For Structural Equation modeling, the model fit is acceptable (cmin/df=1.676, TLI=0.905, CFI =0.939, RMSEA=0.066) except (p-=0.036). A few findings also were drawn through SEM results. First, “Sensor ability” and “Vehicle ability” influence for enhanced operating effectiveness. The regression estimate of “Sensor ability” was 0.107, but the regression estimate of “Vehicle ability” was -0.003. It is anticipated that increased quality of imagery for improved operating effectiveness or usage of imaging infrared (I2R) for lowering camouflage level of enemies is more significant than “Air speed” or setting “Altitude.” Second, “Quality of imagery” and “Altitude” impacts on operating effectiveness of UAVs. Each value of SMC (Squared multiple correlations) was Quality of imagery (0.862) and Altitude (-0.284). Quality of imagery provides positive impact on the operating effectiveness of UAVs while Altitude negative impacts.

Validation

In order to verify results from data analysis, the simulation result from improved scenario was compared with previous results. It was noted that “Quality of imagery” affected critically on UAV operating effectiveness through data analysis by MLR. Furthermore, it is observed from the MLR analysis from simulations between the value of coefficient of “Quality of imagery” with dataset (Quality of imagery > 16mm) and another dataset (Quality of imagery > 16mm) that the higher Quality of imagery could not lead to more positive results. That is, proper “Quality of imagery” could lead to positive effects on UAV. In addition to that, it is noticed that “Quality of imagery” and “Altitude” affected to operating effectiveness through data analysis by SEM. The following Table 12 explains the simulation results. If “Quality of imagery” and altitude and air speed were decreased, around 29% increase of operating effectiveness was resulted in the simulation. Therefore, operating effectiveness could be improved by simulation applied to analyzed data results.

Table 12: The result of simulation for validation.

Discussion

This study is for the methodology of measuring and analyzing the effectiveness of an UAV. In order to decide the factors for evaluating effectiveness, literature reviews, experts’ interviews, surveys and factor analysis were conducted. With the selected factors, an experiment design for simulations was organized. With the developed simulations, the different MOEs (Measures of Effectiveness) were drawn depending upon effectiveness on UAVs. In addition, with the results, meaningful conclusions for better effectiveness of UAV were gotten by analyzing through Multiple Linear Regression and Structural Equation Modeling.

The contributions of this study can be summarized as followings. First, the new methodology through M&S was presented to obtain operating effectiveness of weapon systems. In the past they were subjective and qualitative, but this is more objective and quantitative based on simulations through Multiple Linear Regression and Structural Equation Modeling. Second, practical factors were achieved through simulations to calculate operating effectiveness. Through experts’ interviews rather than the judgmental analyses in the past, the factors were more generalized. As a result, practical factors were acquired and were suitable for Korean peninsula and effective UAV’s operational concept of Korean Military. They were applied to the simulations.

However, there are limitations on this study. First, even though a new methodology to measure the operating effectiveness of weapon systems was given through simulations, a new practical application was not developed. New applications are possibly developed to optimize the UAV’s combat effectiveness in limited conditions by utilizing regression equation driven by data analysis. Second, a few factors out of total selected factors were used in the step of experiment design. If not only “Classification probability” but also “Survivability” and “Electronic warfare” are reflected in the operating effectiveness with all selected factors, more reliable analysis can be done. Third, various simulation models for UAV have been developed in the domestic and international market, but this study is only limited to Korean Army UAVs. It will be more reliable; however, if simulation results from domestic and international various simulation models on UAVs are compared.

References

1. Kong JP, et al. Drop impact analysis of smart unmanned aerial vehicle (SUAV) landing gear and comparison with experimental data. Advanced Computational Engineering and Experimenting (ACE-X 2).2009;40:192-197.
2. Andreas B, et al. Safety, Security, and Rescue Missions with an Unmanned Aerial Vehicle (UAV), Aerial Mosaicking and Autonomous Flight. 2009.
3. David G. Unmanned Aerial Vehicles: Implications for military operations, Air University Occasional paper. 2000.
4. George V, et al. From mission planning to flight control of unmanned aerial vehicles: Strategies and implementation tools. Annual Reviews in Control.2005;29:101-115.
5. Elizabeth B and Christopher B. Unmanned Aerial Vehicles: Background and Issues for Congress. Congressional Research Service. 2003.
6. Jeffrey AD, et al. Innovative management in the DARPA High Altitude Endurance Unmanned Aerial Vehicle Program, Rand. 1999.
7. Cummings ML and Mitchell PJ. Operator scheduling strategies in supervisory control of multiple UAVs. Aerospace Science and Technology.2007;11:339-348.
8. Katrina H. Development of the unmanned aerial vehicle market: forecasts and trends. Air & Space Europe.2000;2:25-27.
9. Lee C, et al. Unmanned Aerial Vehicle Operational Test and Evaluation Lessons Learned. Institute for defense analyses. 2003.
10. Park, et al. A Study on the Real-Time Parameter Estimation of DURUMI-II for Control Surface Fault Using Flight Test Data. International Journal of Control, Automation and Systems.2007;5:410-418.
11. Robert AB. The Effective Use of Multiple Unmanned Aerial Vehicles in B Surface Search and Control. Naval Postgraduate School Master’s Thesis. 2004.
12. Lawrence AH and Liang. The Use of Agent-Based Simulation for Cooperative Sensing of the Battlefield. Naval Postgraduate School Master’s Thesis. 2005.
13. L. A. Ingham, T. Jones, A. Maneschijn (2006), Considerations for UAV design and operation in South African airspace.The Aeronautical Journal.2006;695:695-701.
14. Kuo-Ping L andKuo-Chen H. An efficient fuzzy weighted average algorithm for the military UAV selecting under group decision making. Knowledge-Based Systems. 2011;24:877-889.
15. Kevin LM. Unmanned Aerial Vehicle Survivability: The Impacts of Speed, Detectability, and Enemy Capabilities. Naval Postgraduate School Master’s Thesis. 2005.
16. Jennifer GW. Unmanned Aerial Vehicle Mission Level Simulation. Air Force Institute of Technology. 1999.
17. Bahri Y. Exploration of the Use of Unmanned Aerial Vehicles along with Other Assets to Enhance Border Protection. Naval Postgraduate School Master’s Thesis. 2009
18. Geoffrey S. The Global Hawk Unmanned Aerial Vehicle Acquisition Process: A Summary of Phase I Experience. Rand. 1997.
19. Mark R. Unmanned Aerial Vehicle Contributions to Intelligence, Surveillance, and Reconnaissance Missions for Expeditionary Operations. Naval Postgraduate School Master’s Thesis. 2004.
20. Michael EM. Human systems integration and automation issues in small unmanned aerial vehicles. Naval Postgraduate School Master’s Thesis. 2004
21. Dalamagkidis K, et al. On unmanned aircraft systems issues, Challenges and operational restrictions preventing integration. National Airspace System.2008;503-519.
22. Sujit PB and Ghose D. Search Using Multiple UAVs with Flight Time Constraints. IEEE Transactions on Aerospace and Electronic Systems. 2003;40:491-509.
23. Patrik L, et al. Operating Multiple Semi-autonomous UGVs: Navigation, Strategies, and Instantaneous Performance.Engineering Psychology and Cognitive Ergonomics.2007;732-740.
24. Unmanned aircraft systems roadmap 2010-2035, U.S. Army. 2010.
25. Yong L andJose BC. General Markov Modeling of Pop-Up Threats with Applications to Persistent Area Denial. Air Force Research Laboratory. 2006.
26. Vasileios L. Exploring Naval Tactics with UAVs in an Island Complex Using Agent-Based Simulation. Naval Postgraduate School Master’s Thesis. 2007.
27. Suk and Jin-Young. Ground Test and Evaluation of a Flight Control System for Unmanned Aerial Vehicles. KSAS international journal.2004;5:57-63.
28. Paul AN. Uninhabited Aerial Vehicles: Current and Future Use.Observing Systems for Atmospheric Composition. 2007;106-118.