Issue Brief

Are We in a Pentagon Wars 2.0?

Independent combat testing of new weapons is under threat

Members of Congress need independent analysis of operational testing results to properly oversee the weapons acquisition process, but new regulations from the Pentagon will likely prevent Congress from getting the data it needs.

Introduction

A new presidential administration has approximately 1,200 political appointments to consider. The public largely focuses its attention on high profile positions like Attorney General and Secretary of Defense, but offices farther down the hierarchy deserve equal consideration. This is especially true for the Director, Operational Test & Evaluation, the person charged with combat testing the various acquisition programs in the Pentagon’s $315 billion annual hardware portfolio.

For the past 41 years, the operational testing director’s office has been tasked with ensuring that new weapons are properly combat tested before entering service – despite ongoing efforts to neuter both the tests and the office itself. Now, efforts are quietly underway in Washington that threaten both the effective conduct of rigorous combat tests and the completeness and accuracy of the independence of the office’s reports to Congress. That makes the Trump transition team’s selection for the testing director’s position particularly important. The person selected for the role must be committed to maintaining the office’s integrity and independence.

Congress created the operational testing office in 1983 over vehement objections from both Pentagon leadership and the defense industry. The motivation for its creation emanated from a bipartisan coalition of members across the political spectrum in both houses of Congress who believed they were not being told the full truth about the performance of new weapons from the services and that tests were being compromised. There have been numerous efforts over the years to weaken or eliminate the office ever since. These undermining efforts have included the appointment and confirmation of directors who authorized compromised tests and incomplete and inaccurate reports.¹ The latest attempt to diminish the office will blur the lines between two completely different testing processes, thereby compromising test integrity, and potentially threatening Congress’s access to unfiltered reports about how effective new weapons programs really are.

The Defense Science Board released a report in August 2024 that included several proposals to merge developmental and operational testing and to create a new overall testing director position. Such a move would add a layer of bureaucracy between the operational testing director and the Secretary of Defense and Congress that would threaten the independence of the office. This would undo one of the fundamental reforms of the original act, which purposefully separated developmental testing – often heavily influenced by contractors and other program advocates –from the operational variety as federal law intends. Bad as that would be, specific new instructions from inside the testing community itself threaten the independence and integrity of the operational testing process.

Many senior officials in the Pentagon’s hierarchy do not want rigorous operational testing of any weapons because the results frequently contradict the optimistic statements made about the weapon’s capabilities and effectiveness which could threaten a program’s future.

Unless the operational testing office maintains its current highly independent status, Congress and the American people may not know if the weapons they purchase for the military actually work as intended.

Operational Testing Evolution

The 1998 movie “The Pentagon Wars” satirized the all-too-real story of Colonel James Burton, an officer overseeing weapon testing for the Pentagon and battling against the Department of Defense as officials rigged tests, manipulated data, and wasted money while developing the Bradley Fighting Vehicle.

Burton detailed his efforts to fully test the Bradley in his much more serious 1993 book. Burton believed it would be a good idea to see whether the Army’s new armored personnel carrier could stand up to the Soviet weapons that could be used against it in combat one day. Burton, an Air Force officer, worked as a testing supervisor for the Under Secretary of Defense for Research and Engineering. He believed Army officials were not fully testing the new Bradley Fighting Vehicle to hide critical vulnerabilities that might threaten the program’s budget.²

Burton did not believe the Bradley’s aluminum armor would be able to stop Soviet munitions and worried that such a hit would set fire to the vehicle’s fuel and cause secondary explosions of ammunition stored inside the crew compartment. In one famous case, he ordered a live-fire test using Soviet weapons against a Bradley loaded with fuel and ammunition to see how the vehicle would hold up under fire. But the night before the test was to take place, soldiers removed the ammunition and replaced it with water cans to prevent a catastrophic explosion and make the Bradley look safer than it would be in combat.³

Colonel Burton’s efforts eventually came to the attention of the Congressional Military Reform Caucus on Capitol Hill, especially the chairmen Senator David Pryor (D-AR) and Representative Denny Smith (R-WA). The bipartisan caucus was formed in the early 1980s by Senator Gary Hart of Colorado and Representative William Whitehurst of Virginia. Dozens of members from both sides of the aisle eventually joined the effort to push the Pentagon to adopt more creative warfighting doctrine and purchase simpler and less expensive weapons.⁴

Caucus members searched for a meaningful reform proposal that could garner the necessary political support for passage. They found the issue they needed in the newspapers delivered to their offices every morning. By 1982, a constant flow of news articles detailing failed weapon tests appeared on the pages of the Washington Post and the New York Times.⁵ A group of Pentagon insiders concerned about the state of affairs inside the building had teamed up with journalists to ensure the public knew how their tax dollars flowing into the Department of Defense were being spent.⁶ The partnership of insiders and journalists proved critical because they could show with evidence just how poorly many of the marquee weapon programs performed.

Some members of Congress already suspected they weren’t receiving all the information they needed to properly oversee the Pentagon. The barrage of bad news stories in the papers removed any remaining doubt. Senator Pryor directed his staff to meet with some of the people behind the press stories to craft legislation to ensure new weapon programs were subjected to realistic combat testing before the programs went into wide production.⁷ The amendment they drafted would create a new testing office outside of the acquisition bureaucracy, headed by the Director, Operational Test & Evaluation. The new office would approve testing plans, oversee the actual testing events, and analyze the raw data. The director of the office would write the results which would then be delivered directly to the Secretary of Defense and Congress without being subject to review by the services or the acquisition bureaucracy first.⁸

Pentagon leaders, their allies on Capitol Hill, and defense industry leaders recognized the threat an independent testing office posed to their usual business practices. They wanted to retain tight control of the information regarding the performance of their pet projects.⁹ Senator Pryor countered that his testing office amendment prevented “the students from writing their own exam, monitoring the exam, and then grading it themselves.”¹⁰

Opponents of the bill attempted a few legislative maneuvers to weaken the bill and reduce its chances of passage, but efforts to weaken the legislation during the reconciliation conference largely failed. The new Director, Operational Test & Evaluation position became codified into law when President Reagan signed the 1984 National Defense Authorization Act into law on September 24, 1983.¹¹ The new office proved its value in the first year when the director reported the Army’s troubled Sergeant York Division Air Defense system as not “operationally effective,” prompting the secretary of defense to cancel the program.¹²

Proposed Reforms

Efforts to neuter or abolish the independent operational testing office began nearly from the moment of its creation in 1984. The threats against operational testing have generally come from the defense industry, the military services and factions in the civilian bureaucracy. In a significant twist, however, the latest effort against operational testing started from within the testing community.

President Biden nominated Nickolas Guertin to be the Director, Operational Test & Evaluation (DOT&E) in September 2021.¹³ Guertin made it clear during his Senate confirmation hearing that he believed operational testing should happen in concert with developmental tests. “Injecting operationally realistic testing early into a program’s development allows the Department to implement affordable, comprehensive changes, making the best and most efficient use of taxpayer resources, and ultimately, achieving the greatest possible performance,” Guertin told the Senate Armed Services Committee on October 19, 2021.¹⁴

Guertin’s successor as DOT&E, Douglas Schmidt, recently approved changes to several DoD regulations pertaining to acquisition within the Department of Defense. For example, Schmidt’s draft regulations call for the testing community to perform “mission-based risk assessments” to test new weapons incrementally at the component level during the development process. Analysis of this kind, which may not include any actual testing and could include paper studies and computer simulations validated by no one other than system advocates, is far outside of the scope of operational testing which is supposed to evaluate finished products in realistic scenarios rather than just a series of subcomponents. Engineers already scrutinize new systems during the design phase for component-level vulnerabilities in a process called Failure Modes, Effects & Criticality Analysis.¹⁵

The inherent danger of a focus on component level testing is that it can obscure assessments of the overall system’s performance. The interaction of a system’s components is what really counts, and it is one of the key things the original law took care to uphold. It is possible that every component of a new system works exactly as designed, but if they don’t work together, then the overall system is a failure. For example, Navy officials argued that the new aircraft carrier USS Gerald R. Ford didn’t require full-ship shock trials because they had conducted component-level testing.¹⁶ When the shock trial tests, which involved a series of underwater detonations close to the ship, were eventually performed in 2021, the testing office reported several “survivability improvement opportunities,” but the specifics were classified.¹⁷

The new plan would also inject operational testing officials into the weapons development process at the earliest stages of a program as part of a proposed Live Fire Test & Evaluation organization. Individuals would be assigned to the program offices of the new systems to undergo testing where they would actively participate in the development process. Their job would be to test the survivability of a system at the component level during the development process. The draft regulations call this “full spectrum” testing. But this has the potential to compromise participants, including DOT&E, by involving them in, and seeking their collaboration with, a process very different from realistic combat testing.

The new regulations also assume survivability can be calculated with a simple formula. The authors want to calculate a system’s survivability by measuring its susceptibility and survivability.¹⁸

Susceptibility is a measure of a threat and how it affects the people operating the tested system while trying to accomplish their mission.¹⁹ For example, an enemy surface to air missile may be extremely effective at low altitude, but not be able to reach an aircraft flying above the missile’s operating ceiling. As long as a pilot flying above the threat’s effective range can still accomplish the mission, then the aircraft is not susceptible. However, flying at such higher altitudes may seriously compromise the aircraft’s ability to identify targets or attack ones that pose a danger to friendly forces. Thus, the entire system and the entire combat environment must be tested for well-informed conclusions to be drawn.

Vulnerability, as defined by the Defense Acquisition University, is the characteristics of a weapon that make it “suffer a definite degradation (loss or reduction of capability to perform the designated mission) as a result of having been subjected to a certain (defined) level of effects in an unnatural (man-made) hostile environment.”²⁰

The new regulations would have test officials collect data regarding a system’s susceptibility to potential threats and then separately collect data on its vulnerability. Officials would then later evaluate all the results on a spreadsheet to calculate the system’s survivability. All the data would be accurate, but it would not result in an accurate assessment of the weapon’s overall survivability. In a classic example of how this approach fails, when the Bradley Fighting Vehicle was tested for susceptibility and vulnerability, testers did not initially include stowed ammunition, fuels, and lubricants inside the vehicles. When the tests were forced to be more realistic by a combination of public and official pressure, some officials allowed the test teams to use less effective enemy weapons and aimed the shots to avoid flammables and explosives inside the vehicle. In one case, they even wanted to fill fuel tanks with water for the tests.²¹ When only laboratory results are sought, the result for real world events can be completely misleading.

Determining a weapon’s survivability requires evaluators to consider a range of observations and information far beyond a simple data set derived from live-fire tests and threat analysis charts. In the case of an aircraft, pilots develop tactics to neutralize many potential threats, like using terrain to hide their approach to a target. They can also take advantage of weather and daylight conditions. Such considerations are impossible for Excel to calculate. Only experienced operational testing officials using the gathered data and their own knowledge and experience of combat can consider all the available information to come to an accurate conclusion of a system’s survivability.

An even more profound issue with the proposed changes, and the one that certainly violates the intent behind the creation of the operational testing office in the first place, is that of integrating operational testing officials into the program’s development, which blurs the lines between two processes that are supposed to be reasonably adversarial. Operational testing should be like taking standardized tests in high school. Development of a weapon system can be thought of as the knowledge gained during the school year. The developmental test officials administer quizzes and tests throughout the process to check on the progress of the designers and engineers. Once the design is complete, the operational testing officials then administer their own final test confirm the design is both effective and suitable before the system goes into full-rate production.

Injecting operational testing personnel into the development process creates a potential Stockholm Syndrome risk. They would have a stake in the outcome because they will be involved in the development process when they should be neutral arbiters evaluating results right before graduation. In this case, however, rather than determining if a student gets to walk across the stage to receive their diploma, DoD’s testing results are supposed to inform the decision regarding a new program’s qualifications to proceed to full-rate production. The implications for the future, however, are very different. In one case, the result can be the qualified or unqualified graduate. In the other, the result can mean the life or death of uncounted troops and the unnecessary expenditure of billions in tax dollars.

Test Community Turmoil

Nickolas Guertin outlined his vision for the operational testing community in a 2022 Strategy Update shortly after taking office. “The goal is for the T&E community to enshrine routine engagement and close cooperation with everyone in the capability procurement chain: requirement and technology developers, buyers, users, the intelligence community and, when possible, our allies and partners.”²²

Guertin’s former counterpart on the developmental testing side, Executive Director Christopher Collins, responded by undertaking a consolidation effort of his own. Collins has argued that the discrete separation of developmental and operational testing “has resulted in late ‘discovery’ of issues” when realistic mission testing is performed in sequence.²³

Collins raises a valid point, but his ultimate solution is flawed. To use the F-35 as an example, operational testing of that program revealed hundreds of design flaws that all required corrections. DOT&E reported in 2020 that the program had 871 open deficiencies (design flaws).²⁴ The problem in this case isn’t that earlier testing did not reveal the design flaws; the problem is that both the needlessly complex design, combined with the program’s unrealistic budget and schedule estimates, forced the program into operational testing before it was ready. Moreover, the late discovery of design flaws, especially the dozens of very serious ones found in the F-35, suggests that early developmental testing was compromised and further crippled by blurring the lines between laboratory testing and realistic combat testing.

Much like Guertin’s and his successor Schmidt’s concept of “full spectrum” testing, Collins has proposed a “continuum” for testing where developmental and operational testing occurs simultaneously, and earlier in the acquisition process.

Guertin, Schmidt, and Collins received some official cover for their efforts when the Defense Science Board published a report on Test and Evaluation in August 2024. The wide-ranging report included proposals for reorganizing the Pentagon’s testing community aimed to speed up the acquisition process. “This disconnect between [developmental testing], which verifies that the system meets its requirements, and [operational testing], which validates that the system meets its intended capability in an operational environment,” the report warned, “can significantly impact program delivery costs and schedules.”²⁵

One of the report’s proposals would create an overall testing director who reports directly to the secretary of defense and would oversee both the developmental and operational testing offices. Another proposal would see both agencies fall under the supervision of the undersecretary of defense for research and engineering.²⁶ A glaring omission from the Defense Science Board’s report is any mention of DOT&E’s direct Congressional reporting requirements, although the authors acknowledge that independent operational testing evaluations “have been a concern of the Congress since DOT&E was established in 1983.”²⁷

Both of these proposals echo the consolidation efforts of Guertin and Collins. But these efforts very much violate the spirit of federal law, which enshrines the independence of the operational testing director. 10 U.S. Code, section 139, states the testing director is to communicate directly to the secretary of defense “without obtaining the approval or concurrence of any other official within the Department of Defense.”²⁸ This is extremely important. The bureaucratic separation of the DOT&E office from system advocates in the developmental community, including those in the office of the secretary of defense, is the heart and core of the initial testing office reform. If this interference with both testing and report writing is to be allowed, why bother with DOT&E at all? They will call the new process “independent,” but it will be anything but. DOT&E would just become another cog in the Pentagon’s advocacy system.

Developmental versus Operational Testing

Developmental testing and operational testing of weapons are two distinct processes with very different goals.

Developmental testing is conducted to determine if the system under review meets the contract specifications. Developmental tests are performed in a controlled environment by specially trained personnel like test pilots. Often only a single performance parameter is tested at a time. If the contract specifies an aircraft to fly to 35,000 feet, for example, a test pilot will take off in the prototype and try to fly it to that altitude. If it does, the test is deemed successful and then another test will be conducted to see if the prototype can achieve another design parameter (e.g. speed).

Developmental testing is an important part of the acquisition process to confirm specific technical requirements. But combat does not take place in a laboratory by technicians holding clipboards. To see how a new weapon performs in the hands of actual troops in a realistic environment, a different series of tests are needed. That process is called operational testing.

The services each have their own operational test agencies. The Air Force has the Air Force Operational Test & Evaluation Center. The Navy has the Operational Test & Evaluation Force. The Marine Corps and the Army have the Marine Corps Test and Evaluation Activity office and the Army Test and Evaluation Command, respectively.

During the early stages of the acquisition process, the program manager works with the service test agency and the Department of Defense Developmental and Operational testing offices to create a Test & Evaluation Master Plan. The plan outlines all the developmental, operational, and live-fire testing events necessary to properly evaluate new technology. The plan is used to create schedules and identify testing resources like ranges, equipment, and the number of prototypes outfitted with the instrumentation required for successful completion.

The actual testing events are performed by personnel from the service test agencies. The data collected by them is used by evaluators from the operational testing office to independently determine the program’s effectiveness and suitability. And it is only after those operational tests are completed, and any remaining issues or vulnerabilities identified and addressed, that the system can be put into service. To do anything less risks placing U.S. military personnel in harm’s way with flawed equipment.

Conclusion

The Pentagon has been given unprecedented levels of funding over the past 25 years. Defense spending in 2024 is more than 40% higher than it was at the turn of the century. That money has been spent developing a series of major acquisition programs like the Littoral Combat Ship, the Zumwalt-class destroyer, the Army’s Future Combat System, the Marine Corps’ Expeditionary Fighting Vehicle, the Crusader artillery system, and the Comanche helicopter. All of these — and more — have been serious disappointments because they failed to live up to the original promises made by their boosters and builders. Rigorous operational testing demonstrated the shortcomings of these systems.

Most, but unfortunately not all, of the people working within DOT&E have been dutifully evaluating the effectiveness and suitability of these programs. Their reporting is crucial for the top levels of the Pentagon and members of Congress to understand exactly how well new weapons perform under realistic conditions in the hands of the troops. Congress needs accurate information to perform its oversight role and the secretary of defense needs members to help him control an otherwise unaccountable acquisition community in the building.

The accuracy and completeness of the testing data that especially Congress receives will be endangered if the independence of the operational testing office is degraded. The operational testing director needs to have a direct communication link with Capitol Hill. Without that, members have a justifiable reason to suspect the testing data they receive has been manipulated through the acquisition chain of command.

The best way to avoid going back to the days of Jim Burton’s comic but sad opera of the “The Pentagon Wars” is for the Trump administration to nominate a strong individual to be the Director, Operational Test & Evaluation. This person should be dedicated to the office’s integrity and independence.

Notes

1
U.S. Government Accountability Office, Weapons Testing: Quality of DOD Operational Testing and Reporting, GAO PEMD-88-32BR (Washington, July 1988), 3, accessed December 9, 2024, https://www.gao.gov/assets/pemd-88-32br.pdf.
2
James G. Burton, The Pentagon Wars: Reformers Challenge the Old Guard (Annapolis: Naval Institute Press, 1993), 41.
3
Tim Weiner, “Corrupt From Top to Bottom,” New York Times, October 3, 1993, https://timesmachine.nytimes.com/timesmachine/1993/10/03/103993.html?pageNumber=316.
4
Richard Halloran, “Caucus Challenges Defense Concepts,” New York Times, January 12, 1982, https://www.nytimes.com/1982/01/12/us/caucus-challenges-defense-concepts.html.
5
Charles Mohr, “Tests of M-1 Tank Give Mixed Results,” New York Times, September 20, 1982, https://www.nytimes.com/1982/09/20/us/tests-of-m-1-tank-give-mixed-results.html
6
George C. Wilson, “The Birth of a Spending ‘Bow Wave,’” Washington Post, November 27, 1982, https://www.washingtonpost.com/archive/politics/1982/11/28/the-birth-of-a-spending-bow-wave/5790343d-e207-486f-9e16-ec2ffbeb514c/.
7
Winslow Wheeler, The Wastrels of Defense: How Congress Sabotages U.S. Security (Annapolis: Naval Institute Press, 2004), 21.
8
Helen Dewar, “Senate Rejects Pentagon Opposition,” Washington Post, July 14, 1983, https://www.washingtonpost.com/archive/politics/1983/07/15/senate-rejects-pentagon-opposition/de3acdd5-c9e3-4652-9f78-412950998b30/
9
Fred Hiatt, “Buy Arms Now, Test Them Later, Defense Aide Says,” Washington Post, June 28, 1983, https://www.washingtonpost.com/archive/politics/1983/06/29/buy-arms-now-test-them-later-defense-aide-says/738a649d-9f82-4049-912c-8a3c92be53ef/
10
Winslow Wheeler and Larry Korb, Military Reform: A Reference Handbook (Westport, Praeger Security International, 2007), 32.
11
Department of Defense Authorization Act, 1984, Public Law 98-94, https://www.govinfo.gov/app/details/COMPS-483/
12
Bruce Van Voorst and Amy Wilentz, “No More Time for Sergeant York,” Time, September 9, 1985, http://www.time.com/time/magazine/article/0,9171,959780,00.html.
13
Angeline Leishman, “Carnegie Mellon’s Nickolas Guertin in Line to Become Next Defense OT&E Director,” ExecutiveGov.com, September 17, 2021, https://executivegov.com/2021/09/carnegie-mellons-nickolas-guertin-in-line-to-become-next-defense-otande-director/.
14
United States Senate Committee on Armed Services, “Nomination – Guertin, Baker, Coffey, Bush,” October 19, 2021, 22:37, https://www.armed-services.senate.gov/hearings/nomination_-guertin-baker-coffey-bush
15
“Failure Modes & Effects Analysis (FMEA) and Failure Modes, Effects & Criticality Analysis (FMECA),” Defense Acquisition University, https://www.dau.edu/acquipedia-article/failure-modes-effects-analysis-fmea-and-failure-modes-effects-criticality
16
Tony Cappacio, “Delaying Shock Tests on Costliest Ship Opposed in Pentagon,” Bloomberg, June 10, 2015, https://www.bloomberg.com/news/articles/2015-06-10/delaying-shock-tests-on-costliest-ship-opposed-within-pentagon
17
Director, Operational Test & Evaluation, FY 2022 Annual Report, January 2023, 170, https://www.dote.osd.mil/Portals/97/pub/reports/FY2022/navy/2022cvn78_0.pdf?ver=xeSmdm4H07E79gO8tvdtdQ%3d%3d
18
Department of Defense, Office of the Director of Operational Test and Evaluation, DoD Instruction 5000.XF: Operational Test and Evaluation and Live Fire Test and Evaluation, 2024, https://www.documentcloud.org/documents/25508116-dod-instruction-5000-xf/?mode=document.
19
“Susceptibility,” Defense Acquisition University, https://www.dau.edu/glossary/susceptibility
20
“Vulnerability,” Defense Acquisition University, https://www.dau.edu/glossary/vulnerability.
21
James G. Burton, The Pentagon Wars: Reformers Challenge the Old Guard (Annapolis: Naval Institute Press, 1993), 142.
22
Director, Operational Test and Evaluation, DOT&E Strategy Update 2022, June 13, 2022, https://www.dote.osd.mil/Portals/97/pub/reports/FINAL%20DOTE%202022%20Strategy%20Update%2020220613a.pdf?ver=NkYa8WXfdonuh7M7HT3PBg%3D%3D
23
Christopher Collins and Kenneth Senechal, “Test and Evaluation as a Continuum,” The ITEA Journal of Test and Evaluation, March 2023, https://itea.org/journals/volume-44-1/test-and-evaluation-as-a-continuum/
24
Director, Operational Test & Evaluation, FY 2020 Annual Report, January 2021, 19, https://www.dote.osd.mil/Portals/97/pub/reports/FY2020/dod/2020f35jsf.pdf?ver=C5dAWLFs4_N3ZLrP-qB0QQ%3d%3d.
25
Defense Science Board, Test and Evaluation, Department of Defense Office of Prepublication and Security Review, (Washington,2024), 14, https://dsb.cto.mil/wp-content/uploads/2024/08/DSB_TE-Report_UNCLASS_FINAL_August-2024_Stamped.pdf.
26
Defense Science Board, Test and Evaluation, Department of Defense Office of Prepublication and Security Review, (Washington,2024), 58, https://dsb.cto.mil/wp-content/uploads/2024/08/DSB_TE-Report_UNCLASS_FINAL_August-2024_Stamped.pdf.
27
Defense Science Board, Test and Evaluation, Department of Defense Office of Prepublication and Security Review, (Washington,2024), 59, https://dsb.cto.mil/wp-content/uploads/2024/08/DSB_TE-Report_UNCLASS_FINAL_August-2024_Stamped.pdf.
28
Director of Operational Test and Evaluation, U.S. Code 10 (2011) § 139 https://www.law.cornell.edu/uscode/text/10/139.