The purpose of this handbook is to provide general guidelines for the personnel in the Cost Estimating Group (CEG) with respect to the development of
parametric cost estimates for proposed space systems at the Johnson Space Center (JSC). It is recognized that two
individuals performing a cost analysis on the same project may derive slightly different
results due to the degree of subjectivity inherent in the cost estimating process.
However, it is desirable to standardize costing methodologies and processes, where
possible, in order to provide consistency to the process and, thereby, increase the
credibility of the resulting cost estimates.
The guidelines contained in this handbook are just that, guidelines, and should be used
in the absence of a specific or compelling reason to use an alternative approach or
methodology. Since each new project or proposed new start is different, each cost analyst
is empowered to use his or her best judgment to decide when circumstances would require
deviation from these guidelines.
Most of this handbook will focus on guidelines for developing parametric cost estimates
for the DDT&E and production phases of a NASA program. Operations costs, which are generally
developed using a grass-roots approach, would be obtained from the cognizant organization
or center (e.g., KSC for ground operations costs and MOD for mission operations costs).
NASA has developed many cost models to fit a variety
of costing situations. Additionally, NASA has access to
many others developed by others. The selection of the appropriate cost model to use for a
particular project is, of course, a key consideration in the cost estimating process.
The most common situation is when a study involves a single spacecraft or vehicle and
the design data is being developed at the subsystem level. For this case, the NASA/Air Force Cost Model (NAFCOM)
is generally the most appropriate model to use. The NAFCOM
contains a comprehensive set of historical cost and technical data for completed NASA programs. These data have been broken down to the
subsystem level, normalized, and stratified by mission type, i.e., launch vehicles, manned
space vehicles, unmanned spacecraft, and scientific instruments. This facilitates the use
of the single point analog approach where the cost estimator builds up the spacecraft or
vehicle estimate by selecting the most analogous data point for each subsystem, adjusting
for weight and complexity differences, and applying overhead, or "wrap" factors.
The Spacecraft/Vehicle Level Cost Model (SVLCM) is a simple
online
cost model that provides ROM cost estimates for the
development and production of spacecraft, launch vehicle stages, engines and scientific
instruments. The SVLCM is a top-level implementation of the NAFCOM.
The Advanced Missions Cost Model (AMCM) is most appropriately used for situations early
in the conceptual stages where design data is available only at the total vehicle or
spacecraft level and where there are multiple elements for a given scenario. An example
would be a return to the moon scenario involving a set of launch vehicles, transfer and
landing vehicles, surface systems and hardware (such as habitats and power systems),
precursor satellites, communications systems, and similar elements.
There are several other cost models available that can be used as the primary
estimating technique or as a "sanity check" against the results of another
model's results. For example, GSFC's Multi-Variable
Instrument Cost Model (MICM), Scientific Instrument Cost Model (SICM), Mission System
Integration and Test (MSI&T) cost model, and Parametric Manpower Model are available.
We also have access to the Air Force's Unmanned Spacecraft Cost Model (USCM) and to
commercially available cost models such as PRICE and SEER.
The cost estimator should be prepared to defend the choice of cost models. As implied
above, the purpose and level of design detail available will often dictate the choice of
cost model or estimating technique.
The proper documentation of results is an important step in any analysis, especially
parametric cost estimating. The purpose of the Cost Analysis Requirements Document (CARD)
is to provide a standard format for this documentation. There are (at least) two reasons
why good documentation is important in a cost analysis.
One, experience from formal cost reviews, such as Non-Advocate Reviews (NARs) conducted
by Code B, indicates that poorly documented analyses do not fare well in these reviews.
The credibility of the total project suffers if the analyst is unable to explain clearly
the rationale used to derive each of the cost estimates. Conversely, if a reviewer
understands your inputs, approach, and assumptions, your estimate remains credible in
his/her eyes even if the reviewer disagrees with some aspect of it and recommends an
adjustment.
Two, if the basis of the estimates is explicitly documented, it is easier to update the
estimate and provide a verifiable trace to a new cost baseline as key assumptions change
during the course of the project lifetime. This is especially important with respect to
supporting the requirement imposed by NMI 7120.4
(Reference 1) to revalidate the Program Cost Commitment (PCC) annually. A well documented CARD not only facilitates the establishment of the baseline PCC, but also would aid the revalidation process and the
development of updated PCCs.
The CARD does not have to be an elaborately produced
document, especially for cost estimates developed in the early phases of a program, e.g.,
for Phase A or pre-Phase A studies. Physically, it could consist of a one or two-inch
binder with tabbed dividers separating the data for the individual cost elements. A more
formal document would be prepared to support a NAR or the
establishment of the PCC.
The format of the CARD could vary, but it would
typically contain the following information:
- Project description,
- Work Breakdown Structure (WBS),
- Project groundrules and assumptions,
- Project schedule,
- Cost summaries for each of the WBS elements, and
- Cost phasing summaries.
If the cost summaries identified in (5) above are for subsystems, the CARD should contain as much of the following information as
possible or appropriate for each subsystem or cost element:
- an overall technical description of the subsystem being costed. This would include such
things as the function of the subsystem, a listing of the components below the subsystem
level, quantities, design heritage, and an evaluation of the technology readiness level
and development risk,
- a schematic, picture, or diagram of the subsystem,
- a copy of the Cost Estimating Relationship (CER) graphs or identification of the
single-point analog and the rationale for its selection,
- the rationale for the selection of the complexity factor to be applied against the CER or analog,
- basis of estimate if the subsystem was costed using a non-parametric costing method
(e.g., grass-roots, throughput, vendor ROM, etc.). If a
throughput or vendor ROM was used, the source of the
estimate should be identified including any backup details available. If a grass-roots
estimate was used, the time (in terms of labor-hours or man-years), labor rates, and
materials estimates should be shown along with the source of these estimates,
- a listing of the key assumptions that drive or impact the subsystem cost, and
- point(s) of contact.
The derivation of the cost estimates for elements other than subsystems should also be
documented and would have similar information to that shown above in the CARD. Facilities, wraps, and reserve are examples of
non-subsystem cost elements.
The Systems and Cost Analysis Division in the Office
of the Chief Financial Officer at NASA Headquarters
provides an annual update of the NASA New Start inflation index to be used to prepare cost
estimates for new research and development projects. These are provided to the various NASA cost offices in April. These inflation indices are to
be used for a variety of purposes such as:
- inflating cost model results expressed in terms of constant year costs to real year
dollars for budgetary or POP purposes,
- converting from constant dollars expressed in one year to constant dollars expressed in
a different year, and
- normalizing historical cost data expressed in real year (as-spent) dollars to constant
year dollars.
Several on-line inflation calculators are available.
The traditional complexity factor is a linear multiplier that is applied to the
subsystem cost produced by a cost model. In its simplest terms, it is a measure of the
complexity of the subsystem being costed compared to the composite of the CER data base being used or compared to the single point
analog data point being used. The selection of an appropriate complexity factor is
undoubtedly the most controversial part of the cost estimating process because of the
subjectivity involved in its selection.
The most straightforward approach to determining an appropriate value for the
complexity factor of a subsystem is to work closely with the design engineer responsible
for that subsystem. The following steps would generally be followed to determine the
complexity factor. The design engineer (with the assistance of the cost analyst) would:
- become familiar with the historical data points that are candidates for selection as the
costing analog,
- select that data point that is most analogous to the new subsystem being designed,
- assess the complexity of the new subsystem compared to that of the selected analog. This
assessment would be in terms of design maturity of the new subsystem compared to the
design maturity of the analog when it was developed, technology readiness of the new
design compared to the technology readiness of the analog when it was developed, and
specific design differences that make the new subsystem more or less complex than the
analog (examples would be comparisons of pointing accuracy requirements for a guidance
system, data rate and storage requirements for a computer, differences in materials for
structural items, etc.),
- make a quantitative judgment for a value of the complexity factor based on the above
considerations, and
- document the rationale for the selection of the complexity factor.
If a CER is used instead of a single point analog, the
above process is still applicable. The only difference is that the design engineer would
make these assessments with respect to the total data base making up the CER for that subsystem rather than a single data point.
Tables have been prepared by various NASA cost offices
as guidelines to design engineers in making these judgments regarding selection of a
complexity factor. Although these are not absolute standards, they may be useful as
general guidance if the engineer is having difficulty quantifying his/her assessment of
the relative complexities.
Cost estimates developed by grassroots methods would not require use of a complexity
factor. These estimates would presumably have already taken complexity into account as the
time and materials estimates are made in the buildup of the total estimate.
It should be noted that some cost models, notably AMCM
and PRICE, do not use a complexity factor directly. These models incorporate complexity
factor as part of other variables within these models.
Reserves represent the additional funding required to account for the cost uncertainty
in the point estimates produced by the cost models. For CER-based
or analog-based cost models such as NAFCOM, the uncertainty
stems from several sources such as the statistical uncertainty from the least squares
regression analysis for CERs, the subjectivity inherent in
the selection of the complexity factor, and the engineer's estimate of the value of the
independent parameter (e.g., weight).
Similar cost uncertainty exists for estimates based on other cost estimating
techniques. For example, AMCM has statistical uncertainty
and some subjectivity inherent in its input parameters similar to that explained above.
There is subjectivity in estimating time and materials requirements in grassroots
estimating. Also, if a vendor ROM is used as an estimate,
the ROM may be based on informal verbal requests to the
contractor for an estimate and may not reflect a detailed understanding of the total
requirements. Therefore, there is uncertainty regardless of the cost estimating
methodology used. Reserves are required to account for this cost uncertainty.
It should be noted that reserves are not meant to cover major changes in scope or
content of a project. The funding required to cover major scope and content changes is
known as Allowance for Program Adjustment (APA). The decisions regarding the amount of APA to carry in the program would be determined at the
program level or higher.
There have been several approaches developed in calculating the amount of reserves
required for a project. Generally, the selection of the appropriate one depends primarily
on the program phase and the amount of information available. A few examples are described
below.
A straight percentage factor is often used early during a project (i.e., pre-Phase A or
Phase A) before detailed design information is available about each of the subsystems. The
percentage factor is applied to the total project cost (DDT&E
and production) and is typically in the range of 25% to 35%. A variation of this approach
is to apply one percentage to the total DDT&E
costs and another to the production costs. MSFC estimates
often have 30% applied to DDT&E costs and 20%
applied to production costs for reserve.
This approach requires the cost analyst to apply a different reserve percentage factor
to the DDT&E and production costs for each of
the cost elements in the WBS. The cost elements are
reviewed individually and a determination is made as to the appropriate reserve percentage
to be applied. For subsystems this determination is based on an assessment of its design
maturity. The following table is an example of this approach for subsystems.
- 10% - Off the shelf; hardware exists; no modifications required.
- 15% - Modifications required to existing hardware.
- 20% - New hardware, but design has been through Critical Design Review (CDR). Also,
vendor quotes.
- 25% - New hardware, but design has been through Preliminary Design Review (PDR).
- 35% - New design but within State of the Art (SOTA); CERs
or analogs used to estimate cost. Also vendor ROMs.
- 50% - New design; remote (or no) analogs to subsystem; beyond current SOTA (never been done before)
An alternative method to quantifying reserves is to use a cost risk simulation model.
These models are used to generate a cumulative probability distribution curve (an
"S" curve) for the total project cost. The cumulative probability distribution
curve indicates what the probability is of not exceeding the baseline, or point estimate.
An example might be that the point estimate results in a 35% probability of not being
exceeded. The analyst then can use the curve to determine the amount of cost reserves
required to raise this probability to some higher, and more acceptable, value such as 50%.
There are several commercial PC-based models available
which attempt to quantify cost risk using the simulation method. Some are separate,
stand-alone models (such as @RISK); others are incorporated as an option within the cost
model itself (such as PRICE). These programs use a statistical sampling technique, such as
Monte Carlo or Latin Hypercube, to generate the cumulative probability distribution curve
for the total project cost. In some cases the inputs to these risk models will be in terms
of cost (i.e., high, low, and most likely costs for each cost element). In other cases the
inputs will be in terms of the input parameters themselves (high, low, and most likely
values for weight, complexity factor, etc. for each cost element). As with the cost models
themselves, there is a measure of subjectivity involved in the cost risk models since
judgments must be made regarding the selection of these high and low values for each cost
element.
The point estimates produced by NASA's cost models
need to be spread over the project schedule in order to provide annual funding
requirements. If the estimates were derived using parametric cost models, the Beta curve
is typically the best technique to use in spreading the costs over the project schedule.
The Beta curve was developed at JSC in the 60's and has
become an industry standard for spreading parametrically derived cost estimates. A simple
on-line cost spreading calculator using the Beta curve is
available.
Although the actual mathematical formulation of the Beta curve is somewhat complicated,
its shape can be specified by two easy-to-understand parameters: C, cost fraction, or the
fraction of dollars spent by 50% time; and P, peakedness coefficient, a measure of the
peakedness of the curve shape. The cost fraction, C, is constrained to the interval,
0.1875<= C<=0.8125. The P value can be between zero and one (inclusive) where a P
value equal to or close to one indicates the maximum peakedness and a P value equal to or
close to zero indicates a flat curve shape. The following table presents typical values
for the parameters, C and P.
| Beta Curve Parameters |
|
| C |
P |
Comments |
| 0.5 |
1.0 |
Used for most subsystems. Indicates that 50% of the cost is expended at 50% time.
Development activity rises rapidly, peaks, and falls off rapidly. |
| 0.6 |
1.0 |
Used for subsystems known to be required to be developed earlier than most. Indicates
that 60% of the cost will be expended at 50% time. |
| 0.4 |
1.0 |
Used for subsystems known to be developed later than most. Indicates that 40% of the
cost will be expended at 50% time. |
| 0.5 |
0.0 |
Used for cost elements which would display a flatter funding profile indicating more
of a level of effort. Examples might be Program Management or Systems Engineering and
Integration. |
Since the output of a cost model is expressed in constant year dollars, appropriate inflation factors would need to be applied to the cost spreads
produced by the Beta curve to obtain real-year dollars.
If the estimates were derived using a grassroots approach, the phasing will typically
fall out of the estimating process. Usually a grassroots estimate is developed at a low
level of detail with the time and materials requirements phased by year over the project
schedule. As labor rates and other cost factors are applied to these time and materials
estimates, the cost phasing for the cost element is determined. Most often, the labor
rates and other cost factors that are applied to these time and materials estimates are
already expressed in terms of real year dollars. If this is not the case, inflation factors would have to be applied to convert to real year
dollars.
There are many program or project costs that must be accounted for in addition to the
costs for spacecraft vehicle and facilities.
Each program/project is required to contribute a portion of its funding to Headquarters
in order to pay for the Defense Contract Audit Service (DCAS), Small Business Innovative
Research (SBIR), and other agency-wide functions and services. For simplicity, these costs
are sometimes excluded in early cost estimates (during Phase A or pre-Phase A, for
example). However, they should be included in later estimates and cost commitments in
order to present the project's full funding requirement for budgeting purposes. The
following percentages were applicable during the Station Redesign in 1993:
- DCAS: 0.55% of the DDT&E
cost.
- SBIR/others: 2.5% of the DDT&E cost.
- Code B at Headquarters can be contacted to obtain the latest percentages.
The IMS is an institutional tax paid to each center to
pay for center-wide services (guard service, ADP, etc.).
It is based on a per capita amount levied against each civil servant on each program.
Currently, the Station program pays about $20K per civil servant to JSC for IMS while the Shuttle
program pays about $30K per civil servant. These rates are periodically adjusted, so check
with the Comptroller's Office for the latest rates and an estimate for the rates that
would apply to the new project being costed.
A Level II program office may be required if the new program is large enough to have
more than one Level III project office. If an integrating contractor would be required,
the cost of the Level II program office can be estimated as 5% of the sum of the DDT&E costs for the Level III project offices. This
percentage is based on the Shuttle program as the analog, i.e., the cost of the Shuttle
Level II program office as a percentage of the sum of the DDT&E costs for the Level III offices (Orbiter,
Solid Rocket Booster, External Tank, Main Engines, and KSC).
If the project is small enough not to require a program office, this cost element would
not be required.
Program Support consists of all of the activities, systems, and hardware developed
outside of the prime contract. It consists of such things as specific analysis tasks, use
of government test facilities, equipment and personnel, GFE,
project office support contractors, SR&QA
activities, and technical oversight of contractor activities. A factor of 14% can be used
to account for this cost element if a detailed, grassroots estimate is not available. This
factor would be applied against the total DDT&E
cost.
Operations Capability Development (OCD) consists of those activities associated with
preparation for the operations phase. This includes outfitting and modifications to the
mission operations control center, establishment of the logistics system, and
modifications to ground processing facilities and launch pads, purchase of ground handling
equipment, and similar activities. A factor of 5% can be used to account for this cost
element if a detailed, grassroots estimate is not available. This factor would be applied
against the total DDT&E cost.
The definition phase of a program typically consists of one or more Phase B study
contracts plus the advanced development work required to bring the subsystems to an
acceptable readiness level. Research has shown that there is a negative correlation
between the cost overruns of NASA's programs and the
amount spent up front in the definition phase. That is, the more NASA spends in the definition phase to understand its
requirements and bring critical technologies up to the proper levels, the lower the
average cost overrun will be at the end of Phase C/D. The research indicates that the
definition phase should be at least 5% of the DDT&E
costs.
Typical values to use for prime contract fee would be 8% for large, complex projects;
10% for smaller projects such as unmanned spacecraft.
Assumptions and groundrules are a major element of a cost analysis. Since the results
of the cost analysis are conditional upon each of the assumptions and groundrules being
true, they must be documented as completely as practical. The following is a checklist of
the types of information that should be addressed in the assumptions and groundrules
section of the CARD.
- What year dollars the cost results are expressed in, e.g., FY94$.
- Percentages (or approach) used for computing program level wraps: i.e., fee, reserves,
program support, OCD, Phase B/Advanced Development, PMS/IMS/ROS, HQ taxes, Level II
Program Office.
- Production unit quantities, including assumptions regarding spares.
- Quantity of development units, prototype or prototype units.
- Life cycle cost considerations: mission lifetimes, hardware replacement assumptions,
launch rates, number of flights per year.
- Schedule information: Development and production start and stop dates, Phase B
Authorization to Proceed (ATP), Phase C/D ATP, first
flight, Initial Operating Capability (IOC), time frame for life cycle cost computations,
etc.
- Use of existing facilities, modifications to existing facilities, and new facility
requirements.
- Cost sharing or joint funding arrangements with other government agencies, if any.
- Management concepts, especially if cost credit is taken for change in management
culture, New Ways of Doing Business (NWODB), in-house vs. contract, etc.
- Operations concept (e.g., launch vehicle utilized, location of Mission Control Center
(MCC), use of Tracking and Data Relay Satellite System (TDRSS), Deep Space Network (DSN),
or other communication systems, etc.).
- Commonality or design inheritance assumptions.
- Specific items excluded from the cost estimate.
More Cost Models:
|
|
