Is the impending decision on Australia’s submarine competition based on any test and evaluation (T&E) report, or will there be any such reports before a development contract? Just as Government’s looking at the submarine record of Japan, Germany and France, it needs to adjust for its own tendencies, especially in inadequate use of T&E to inform decision-making at such key acquisition milestones.

The Collins-class had serious difficulties in its T&E because the majority of the testing occurred too late in the piece. In 2002, when five of the six boats had been built and operational release still not achieved, the Australian National Audit Office (ANAO) found test plans weren’t updated, resourced or enforced, that test coordination meetings weren’t being held, and that Defence was struggling to verify safety-critical software. A 1999 special report to the Defence Minister stated:

‘To our surprise, deficiencies have occurred in items that should have been relatively straightforward, had testing, even along the lines of that routinely applied to merchant ships, been undertaken.

Prescient to the case for preview activities on SEA 1000 today, the authors also stated:

‘What is required are terms which compel the contractor to demonstrate his capacity to deliver in advance of the execution of the contract itself … a series of tests or trials.’

The ANAO audit in 2001 covered 23 maritime projects as well as Collins, and found the T&E of some, like the Minehunter Coastal ship project, to be exemplary. It recommended Defence take steps similar to the US to better enforce its T&E policy for greater consistency and to avoid release of equipment into service with inadequate testing. Defence rejected this recommendation, claiming it didn’t do the sort of projects requiring such a T&E regime and preferred instead to front-load better planning and resourcing of T&E.

But in the decade following the Collins difficulties, the Australian Senate found in 2012 that there’d been insufficient use of preview T&E on many major acquisitions, mainly under the often false premise of acquisitions being off-the-shelf. The Government agreed to a policy of conducting preview T&E on all acquisition approvals. The resulting Defence capability planning process began use in 2013 and in 2015 was formally released, and endorsed by the ANAO as successfully implemented. The process develops the earliest possible T&E, prior to contract and preferably as part of solicitation, using systematic evaluation of the likely technical and operational risks.

Only last month, the Joint Committee for Parliamentary Accounts and Audit examined yet another multi-billion dollar project that was delayed by 3–4 years when paper-based down-selection proved inadequate. At the hearing Defence officials advised that the lesson on preview T&E had been learnt, the new T&E policy would ensure a repeat was unlikely, and that, interestingly, the current restructure of Defence should only strengthen use of this preview T&E policy.

How the last point would occur wasn’t explained. The two large projects currently undergoing down-selection (SEA 1000 & LAND 400), the outcomes of both possibly to be announced before the election, will seriously test this commitment. Working against the likelihood of the new T&E policy being followed for SEA 1000 is ANAO’s 2015 finding that there are 12 separate test organisations, not one of which is devoted to developmental testing in the maritime domain.

Australian Defence has independent test organisations for development of land and aerospace equipment, but for maritime projects this critical skill and governance is left to individual projects to assemble, with the associated loss of independence and disparity inherent in such a solution.

Given the recently announced investment in maritime capabilities, Defence must address this T&E deficiency. Otherwise, the lack of adherence to test policy observed by the ANAO in maritime projects in 2002, and again in 2011, will continue. Because of this chequered history, I’ve authored a working paper (PDF) on how the Australian and American T&E policies ought to apply to SEA 1000 at this critical decision time. The paper proposes:

• close alignment of SEA 1000 to US acquisition lifecycles and T&E governance checks;
  
• five preview test activities that could be done now for the preferred reference submarine design before the first developmental contract is signed; and,
• 11 test goals for the developmental T&E period.

Such activities and goals would shift SEA 1000 towards first building the necessary land-based test infrastructure for real submarine componentry, here in Australia or overseas, to validate the design before building a first-of-class submarine. These many facilities take about 7 years to develop and build, but without them we’ll repeat the Collins’ mistakes and certainly won’t achieve continuous spiral development and build. I commend my full working paper to anyone involved in the SEA 1000 decision and beg them to trust testing as the basis of their decisions and design iterations every step of the way.

When commentators argue that the government should determine the designer and builder of the Future Submarine Program (FSP) via a competitive Project Definition Study (PDS), they seem to forget—or don’t know—that the SEA 1114 PDS process (which led to the selection of Kockums to build the Collins class submarines) was less than perfect. At the time of submitting the proposals the design of the HDW and KAB submarines were less than 20% complete, with lump sum prices submitted on scant performance criteria. The government made a political decision on the data submitted and following assurances provided during contract negotiations, including the establishment of a domestic naval submarine industry.

In essence, the Competitive Evaluation Process (CEP) will be no less effective than the PDS tendering processes for the Collins class acquisition program. And the CEP will arguably deliver a better result for the government unless an enhanced PDS delivered a substantially completed design of the submarine or the Commonwealth could conflate the designs of the PDS contenders into its preferred outcome. These, however, are unlikely to materialise.

TKMS of Germany, DCNS of France and Mitsubishi-Kawasaki of Japan are accomplished submarine designers and builders. Each bidder would have made a persuasive case in their CEP response that they are committed to fulfilling the RAN’s unique operational requirements, that they’ll engage local industry extensively and that their submarine design and building concepts are safe and cost effective no matter where the future class will be built—in Europe, Japan or Australia. Thus, based on nuanced preferences the government’s CEP evaluation team can recommend any of the three contenders.

As the owner of the IP the Department of Defence has repaired, replaced, retro-engineered and improved the Collins class submarine systems to a high level of performance and reliability. And with more improvements planned, an augmented Collins class could provide the RAN with better value than any of the three current contenders for Australia’s future submarines.

Notwithstanding the present drafting of the Defence White Paper it is anticipated the National Security Committee of Cabinet (NSC) is likely to decide on the DOD’s recommendation of the preferred submarine designer by mid-2016. That would clear the way for the FSP Office to fully develop the scope of work with the selected international partner over the following three years. During that early design phase the operational and interoperability requirements with the key regional allies (USN and JMSDF) will be finalised, and the projected building and through-life costs will be better understood. And because the first of the new class is unlikely to be commissioned into the Australian Navy before 2030 and the Collins class will be operating well into the 2040s, synergies arising from commonality between the Collins and the new submarine class should also be considered during that period.

Whereas the RAN now rates the Collins class as peerless the three CEP contenders haven’t designed, let alone built, a submarine that approaches the RAN’s future submarine operational requirements. As a first hurdle the contenders for the FSP will have to demonstrate that their submarine design will exceed the performance and cost-competitiveness of the Collins class in its current configuration. Failing that the Commonwealth has the fall-back option of keeping the upgraded Collins boats in service until a solution for the future class is agreed to.

And in the event that the Commonwealth and the submarine design house are unable to conclude the three-year design and definition phase successfully, or the parties decide not to enter into a build contract, an evolved Collins class remains a realistic option for the RAN.

Under the current SEA 1439 Collins Class Submarine Reliability and Sustainability program major platform systems are upgraded. The electronic components and software for the automated Integrated Ship Control Management and Monitoring System (ISCMMS) are being updated to ensure the system continues to operate dependably into the future, and planning is underway to upgrade the communication systems for obsolescence and interoperability reasons. It must also be expected that the power-hungry AN/BYG-1 combat control system will evolve under the US/AUS Advanced Processor Build (APB) program to render it more suitable for diesel-electric submarines. But it will be progress in (quantum) computing and advancements in cyber-physical systems (CPS) that will determine the next-generation multi-disciplinary systems and construction methodology of the FSP.

The acoustic performances of the array suites have been enhanced while the hydrodynamic characteristics of the class can be further improved by reconfiguring the towed array handling system, by installing streamlined flank and distributed arrays on the hull, and by redesigning the cylindrical array situated in the fore section of the submarine casing. While the buoyant wire and the high frequency antenna and periscopes are to be updated under the sustainability program, the casing, fin and masts will also require augmentation to lessen mast wake vortices and vibration, and lower pressure drag and skin friction.

The Collins boats’ propulsion system should be replaced with low signature and fuel-efficient diesel generators and gearless permanent-magnetic synchronous motors in the CEP proposals. While propeller technology based on high damping material are likely to feature in the TKMS and Mitsubishi-Kawasaki designs, DCNS could suggest a propulsor for their new Australian class. Continuing with Swedish design philosophy the X-configured aft-control-surfaces that provide the Collins class with superior manoeuvrability will feature on the future submarines in augmented form.

On the energy front, lead-acid batteries will have been replaced almost entirely within 10 years. And by the time the first of the future RAN submarines is launched, lithium-ion battery technology may have been overtaken by lithium-sulphur (Li-S) Li-air, or Zinc-air battery technology. By providing much greater energy density at a fraction of the cost such an energy source would render submarine AIP technologies (PEM fuel-cell, MESMA, Stirling and closed cycle diesel engine) obsolete.

Those emerging technologies and the comprehensive program of continuous innovation and improvement enable the current class to be a serious and low risk contender for the FSP. As a minimum, the RAN should make one Collins boat available for prototyping systems that are designed to feature in the next generation of Australian submarines.

An additional conclusion that can be reached in light of those considerations is the contractual model for the FSP. Given the rapid rate of technological change and yet to be fully specified operational requirements, a fixed-price lump-sum contract is meaningless in any CEP or PDS offer as it’s unenforceable under a construction program over an anticipated 21 year timeline for 12 boats. Indeed a fixed-price model will in all likelihood lead to massive cost blow-outs through the traditional variations and extensions-of-time processes. Add in disruptive innovation and technological advancements, exchange rate movements and the inevitable changes in the RAN’s capability requirements and you have a recipe for a procurement disaster.

From an inauspicious and somewhat controversial start, the CEP has leapt into pride of place as the ‘process of choice’ for the Government when considering large maritime defence projects. Not only has it been used for the Future Submarine project (SEA 1000), but the Prime Minister has stated that it’ll be used for the Future Frigates (SEA 5000), and for the Offshore Patrol Vessels (OPVs) project (SEA 1180); collectively in the order of $90 billion.

This post looks at CEP in terms of 4Ps—process, politics and potential problems.

The process seems to be quite straightforward. Defence sets some broad issues to be responded to, selects a small number of potential suppliers, provides them with a few million dollars each to develop their responses, receives the responses and makes a recommendation to Government. The recommendation is for the selection of a long-term strategic partner with whom to develop the detailed design and ultimately have the ships/submarines constructed.

The SEA 1000 CEP appears to be thorough, even-handed and, as far as possible, transparent. The only issue is the decision that removed Sweden from the competition before it begun. It could be expected that the process for SEA 5000 will be similarly thorough, even-handed and, as far as possible, transparent. One issue for SEA 5000 will again be the potential limitation of the prospective international partners. The stated use of a CEP for the OPV project becomes more problematic as Australian shipyards other than Adelaide are potential builders, and there are many more potential designs. The way in which a CEP would be undertaken for this project is therefore unclear.

The politics are interesting—and somewhat opaque. First, the utilisation of a CEP for SEA 5000 and SEA 1180 means that the Government can argue that the CEP used for SEA 1000 wasn’t a ‘thought-bubble’, but a valid option within the normal way of doing business. The Prime Minister and the Defence Minister will also repeatedly say that the utilisation of a CEP for both submarines and ships is a necessity following the inactivity of the previous Government in this important area of national security, and is the only means to get the project acquisition schedules back on track. The Future Submarines experience has shown that the Government won’t shy away from limiting the number of runners at the starting gate.

The information provided from the CEP process will be of lesser fidelity than that of a typical tender process; arguably as the detailed design hasn’t been finalised. While Defence will no doubt do a good job of the evaluation process, the results will therefore be more amenable to interpretation. Simply put, the selection of the contestants and the lack of detail in the information provided means that it will be easier for the Government to make a decision other than the one that might be recommended by Defence.

The CEP for both frigates and submarines will circumvent the normal tendering processes and hence speed up the process to the point where a decision can be made. Not an acquisition decision based on the finer points of a complete tender process, but a politically-influenced decision on a long-term strategic maritime defence partner based on broad indications of cost, schedule, industrial issues, intellectual property, and capability. The decision on the selection of the international design partner for SEA 1000 is likely in the second quarter 2016. If the CEP for SEA 5000 commences in October as stated, then a decision for that project might be possible around the middle of 2016. That will clear the national security decks for the Abbott Government, and allow it to go to an election claiming that they’ve fixed the shipbuilding problem created by Labor, overcome years of inactivity on key defence issues, supported Australian manufacturing, saved jobs, etc.

Perhaps most importantly from the Government’s perspective, it provides a platform that might save some seats in South Australia.

The potential problems (read: risks) are associated with project cost, project schedule and whether the outcome (and the partner) will continue to provide the right capability into the future. As suggested by Mark Thomson in a previous post, we’ll only know what we’re really going to get, and the overall project cost, once the partner has been in place for some time—and there’ll be no turning back. There’s also a major problem in that political announcements seem to be driving the national security agenda, rather than real aspects of the nation’s security.

The decisions on SEA 1000, SEA 5000 and SEA 1180 will fundamentally alter the maritime capability landscape in Australia for the foreseeable future. The selection of the partner for each of these projects is critically important—and shouldn’t be made on a political whim.