2007 PRICE INSight
ISPA System of Systems Renew Your Training Exercise
Overview
ISPA's Renew Your Training is an annual exercise for cost estimating in which estimating teams are invited to make a 30-minute presentation of their cost/benefit trade-off analysis to a Senior Review Panel. These presentations demonstrate, in an open conference workshop environment, each team's best skills in developing and presenting a cost estimate for a future net-centric program as compared to funding a traditional upgrade. The cost estimate is to be developed using an optimized mix (with rationale) of estimating methods. Emphasis should be placed on presenting a documented credible result, with indications of "best practice" in the estimating profession.
Download presentation | Download project models
The Problem
The cost exercise is to support the State of Florida in deciding on the type of implementation to pursue for an expanded state and federal-linked system of emergency response command centers to natural disasters. The lead government agency will be the Florida Division of Emergency Management with support from the Division State Fire Marshal and FEMA Region IV at the national level. The content providers will be weather satellites, air and ground weather stations, and federal, state and local agencies including the water and power companies. The team will be conducting a cost/benefit analysis on behalf of the lead agency to select between two options.
- Option A—Modify the existing system to use the federal government's Global Information Grid (GIG) for expanded delivery of all essential information to users and data providers while performing minimum maintenance on existing infrastructure where deemed necessary for backup
- Option B—Perform a major upgrade on all existing systems to support increased information flow expected from future sources and to have higher rates of interconnectivity between all users and data providers including assured backup capability
PRICE Systems' Approach
Benefits of Using TruePlanning 2007
PRICE Team Interview
PRICE Systems' Approach
Use TruePlanning © 2007 to model the software, hardware, IT and networking equipment as described in the problem statement for both options. More specifically, True S is used for software; True H for hardware; True IT for networking equipment; and True Systems for integration, test, system engineering, and project management costs at each level of the infrastructure. Take advantage of PRICE Systems' historical knowledge bases that are built into the TruePlanning models to simplify the estimation by selecting custom-developed and COTS software complexities; communication gear hardware weights and complexities; and IT equipment purchase prices, upgrade requirements, and lifetimes. Consult with PRICE Systems' subject matter experts to size the GIG core enterprise services by analogy using data from a completed estimate with similar communication service requirements.
Option A—Model the system as described in the problem statement, as in the case of the traditional system upgrade (see description of the approach to Option B below), but with an additional system component to represent the custom development of the GIG's core enterprise services. Use True Systems to capture the GIG integration effort required by the state of Florida by isolating the appropriate activities at the highest TruePlanning project level, and subtracting the GIG development costs, since Florida is using but not developing the communication services provided by the GIG.
Option B—Model the system as described in the problem statement in a straightforward way, by agency, using the information provided about licensed and custom developed software, communication hardware, IT and network equipment requirements throughout the system infrastructure.
Benefits of Using TruePlanning 2007
With TruePlanning 2007 the entire estimate for each option was able to be constructed as an integrated whole, including system engineering, integration and test, and project management costs. PRICE Systems' estimating team did not have to spend any effort combining hardware, software, and IT estimates from disparate sources, and did not have to identify potential overlapping of project management costs due to having multiple subsystem estimates. This simplification of the estimating process, enabled by TruePlanning 2007, allowed the PRICE Systems team to focus on understanding the underlying issues involved in the available upgrade options, and also resulted in more straightforward and easily understood end results for the senior source selection panel.
PRICE Team Interview
The PRICE Team of Angelina Zhou and Josh Wilson filled in the details of their impressive ISPA performance using TruePlanning 2007 in this interview with PRICE INSight following the event.
INSight: We understand that you recently teamed to represent PRICE Systems in a competitive decision making exercise related to cost estimating. What was the problem you were asked to address?
Angelina: Every year, ISPA poses a question based on the same assumptions and asks each team to use whatever software tools the team is comfortable with to solve the problem. They ask each team to attend the conference and present results. This year, Josh and I collaborated to represent PRICE Systems as the company team. The problem we were presented this year was to help the state of Florida decide an implementation approach for network linking into an emergency response system. Two approaches were postulated and we were asked to evaluate the cost of each.
INSight: How many other teams competed with you on this exercise, Josh?
Josh: There were 3 other teams besides ours; Boeing had a team, Northrop Grumman, and Galorath.
INSight: What can you tell us about the demographics of the other teams, Josh? What's their level of education and experience? Are they government or industry? Engineers? Estimators? Managers? Tell us anything at all that will help us understand what sort of competitors you faced.
Josh: One of the ideas behind the RYT exercise is that each team gets experience in presenting its solution to a panel of experts. That said, all of the teams were from industry; one was another tool company, like PRICE. It was a little difficult to determine the exact make up of each team as not all members of every team attended the conference. Northrop, for example, had a 4-person team, but only one attended and presented for them. From what I could tell, the Northrop presenter was either in general management or program management. Boeing's 3-person team of cost analysts all attended and presented in one way or another. In terms of work experience/seniority, I would rank the Galorath presenter as the most experienced, followed closely by the Northrop Grumman presenter; both were in the 25 to 35 years of experience range, I would say. By contrast, the Boeing team appeared to average between 5 to 10 years of experience.
INSight: Angelina, how about your team? How did you stack up against the competition as far as education, experience, and so on?
Angelina: I earned a Masters degree in Operations Research in 2005. I joined PRICE immediately after getting the degree and I am a certified ISPA cost analyst. I was the lead analyst for the RYT exercise.
INSight: How about you, Josh?
Josh: I joined PRICE about 6 months after Angelina. I graduated form Georgia Tech with 2 degrees - one in Math, the other in Industrial and Systems Engineering. PRICE was my first real job after college.
INSight: So, would we be accurate if we said that PRICE fielded the least experienced team of the four, but at least on average with the educational credentials of the others?
Angelina & Josh: Yes.
INSight: Tell us about the approach you took to solving the problem. How did you evaluate the two different configuration alternatives to the Florida Emergency Response system?
Josh: We were tasked to perform a cost analysis and include as much benefit analysis, risk analysis, or any other type of Figure of Merit (FOM) evaluation that made sense to us and which we thought would make sense to a Florida official involved in making the real life decision. Option 1 was the traditional upgrade path - buy more servers, buy more work stations, development more software. Option 2 involved using as much of the existing assets as possible and tap into the US Global Information Grid (GIG) for interconnectivity with other state and national emergency response systems.
INSight: We have heard of the GIG, but don't know much about it. Can you explain what it is?
Angelina: The GIG is like a special world-wide-web for official government use. Access to the GIG is strictly controlled; one needs authorization to be a GIG user.
Josh: A document called the NCOW gives the DoD vision for information sharing systems like the GIG.
Angelina: The GIG seems to be in the concept stage of its existence where the basic architecture is being established. We couldn't find any information to suggest that the GIG really exists at this time.
Josh: That situation - the vaguely defined state of the GIG - made addressing Option 2 somewhat of a bigger challenge than Option 1.
INSight: We interrupted your explanation with a side bar on the GIG. Please continue to describe your approach to solving the problem.
Angelina: The expectation of the solution was that it would be the lowest cost configuration over a 10 year operation period and that it meets or exceeds emergency response threshold times.
Josh: We had to do a lot of research to get our arms around the problem we were trying to solve, especially for the GIG option.
INSight: Exactly, how much information were you given?
Angelina: We were given some basic assumptions about the amount of hardware of various types needed, the length of cable needed, what kind of software was needed. But, things like weight, complexity, and acquisition cost were rarely presented as known.
Josh: We were given a table of current Florida IT equipment and software configuration by location and projections of what additional is needed to upgrade to the desired emergency response system performance.
INSight: Did you employ any special tools or methods in executing the solution to the problem?
Angelina: We used TruePlanning 2007to solve the problem.TruePlanning 2007 is the first product to integrate hardware, software, and IT system estimating within one framework. It also allows us to attach all of our research reference documents to the project database, which is very important to us as we had many Excel files that served as our basis of estimate - we need to keep those within easy access to our estimate. TruePlanning has risk analysis capability, which is a must for good decision making. Plus, the operation and maintenance capability meant we could address costs for all phases of the emergency response system life cycle.
INSight: What other tools besides TruePlanning did you use?
Angelina: None - TruePlanning met all of the estimating requirements.
INSight: What degree of satisfaction did you have with your final solution? This would be at the point where you had completed all of your internal PRICE Systems reviews, but before you or any of your competitors had presented your solution.
Angelina: We felt very confident in the goodness of our estimate. We made a point of seeking advice from the Subject Matter Experts (SME) within PRICE whenever we had a question. We also found information on a system from another state, Michigan, which we used as a sounding board.
Josh: I agree; we built the estimate from a solid base and didn't back into any pre-conceived ideas to make things fit. All of our data and assumptions came from research and represent industry averages. In addition to Michigan, we found unit prices for similar hardware for a Mississippi system after we had completed our solution. It was reassuring to note how similar those prices were to those we estimated for the Florida system.
INSight: You mentioned a panel of experts that heard and evaluated the solutions from all of the teams. How did the panel do their evaluation and what grade did your team receive?
Josh: Because this is a learning exercise, the panel was not overly critical of any team. The feedback to us was very good. The panel felt we did a really good job in researching and understanding the problem and that we had done more than simply present a cost for Option 1 and another cost for Option 2; here you are, now pick the lowest one. They appreciated that we sought to understand the problem and that we extended the solution beyond the cost dimension.
Angelina: Right; they liked our cross references to GSA tables and on-line resources. They also liked the risk report, charts, and graphics from TruePlanning; they served as pictures of the estimate story, dynamically responding to view any level of the system desired. The pictures quickly display the system differences. The panel also liked the TruePlanning estimating breakdown structure.
Josh: The panel made particular note that it was easy to see where Systems Engineering costs were being incurred; something that was pertinent to the exercise since both options are complex systems undertakings. Most of the other teams used tools that could only identify systems engineering at the project level. There was no way to see its build-up, which left the reviewers with no confidence in the estimate.
Angelina: We really have to thank TruePlanning 2007 for the quality of our analysis and the rousing acceptance of it by the panel. Everything we needed to solve the problem and to convey the solution to a panel of reviewers was provided by TruePlanning. It really did enable us to tell a credible story; actually, TruePlanning told the story.
Josh: Absolutely; we were able to drill down to answer any panel question about cost anywhere in the WBS we were provided. Furthermore, unlike all of the other teams, we were able to produce the prescribed cost breakout by WBS directly from one tool - TruePlanning. It was impressive.
INSight: As you talk more about the depth of your solution, we hear the word "system" a lot. 10 to 15 years or so ago the thought leader Peter Senge began promoting the notion of system thinking. Senge and others saw the collision of massive amounts of information and IT systems to process it all. Managing and making sense of all of this would require caretakers adept in understanding the interrelationships of things. This skill is the essence of systems thinking. Are we off base here by saying TruePlanning helped you with the systems thinking of this problem?
Josh: I don't think you are off base at all on that. In fact, I was surprised to see Option 1 being more systems engineering intensive than Option 2. My first reaction was that all of the new software required to work with the GIG, Option 2, would lead it to have higher systems engineering costs. Not seeing what I expected, I had to check why. That caused me to study what activities were required for each option. It made me realize that the conventional upgrade option, while less software intensive, had more things going on than did Option 2; it was a more complex system network, which you can only appreciate if you do think about and attempt to evaluate the impact of the interrelatedness of the system parts.
INSight: What was the format for panel feedback to you?
Angelina: First, there was about 10 minutes devoted to each team's solution, given in a general session for any conference attendee to attend. Then, each team had a private review session with the panel, where comments were usually more directed.
INSight: Was there any attempt to grade each team's performance?
Josh: Not explicitly. But the five panelists told us about our solution and presentation were outstanding.
INSight: In your opinion, what was the most outstanding part of your solution?
Josh: I think it was the clarity and credibility of our estimate, enabled by TruePlanning's capability to manage every aspect of our estimate in one place.
Angelina: Another strength, cited by the panel, was the richness of our data used as basis of estimate. The interesting thing is that we didn't know what we needed to know before we started the problem. The process that TruePlanning led us through actually helped us identify the nuances of the problem that simply were not evident at the start. It helped us develop systems thinking, as we discussed earlier.
INSight: Didn't the other teams have a totally integrated solution like yours?
Josh: No, none of them did. In most cases they used different tools for the software, IT, and hardware and then integrated the results with Excel or something similar. As far as system of systems integration, the others pretty much threw one number into whatever was used to integrate the results. I'm not saying the number was bad, but its basis was a mystery to the panel. I don't know what the other teams used to do risk analysis, but here is another area where the panel thought we stood out. We used the risk analysis within TruePlanning for that as well. Again, I think the fact that our risk analysis was totally integrated, while the competition had disparate analyses that needed something else to integrate them was an overwhelming advantage for us.
Josh: Kind of like the difference between viewing a uniform landscape versus a patchwork quilt?
Angelina: Yes; that's a good analogy. It's easier to take in everything at once with a uniform landscape than it is for a patch worked one.
INSight: OK, so what do you think you could have been better?
Angelina: The panel suggested that we select our colors for charts and graphs with more attention to those who are visually impaired. For instance, they said we should avoid green and red as they are indiscernible to those with color blindness.
Josh: I think we could have done a better job in assessing the 10 year support costs for the software, though it wasn't a weakness identified by the panel.
INSight: Before we end the interview, is there anything either of you would care to share with our readers?
Angelina: I was really glad to have the opportunity to do the RYT exercise. It gave me the opportunity to interact with SMEs within the company, which taught me a lot. It also enabled me to learn more about how to sell an estimate - I got some good pointers on that from one of the PRICE business development managers.
Josh: The problem was pretty easy to solve intuitively because TruePlanning conforms so well to input structures we were given and the output formats we were required to produce. A number of the things that the panel found impressive about our solution, like risk analysis and non-recurring costs to multiple levels were actually quite easy to do with TruePlanning. TruePlanning was awesome to use; even with our limited experience in estimating, we were able to produce a credible, well thought out solution.
INSight: You both exude much more confidence in your work than we would expect from the team with the least amount of experience of all RYT competitors. Why is that?
Angelina: TruePlanning; we would have been lost without it, I think. It is almost like a mentor guiding you to seek pertinent information for estimating as you work through it.
Josh: TruePlanning is a big part of it, as Angelina said. There is quite a bit of knowledge built into the tool; this helped us a lot. It was also helpful to have the guidance of the SMEs working at PRICE.
INSight: Angelina and Josh, congratulations on being identified as an outstanding RYT team this year and thank you for this interview.
Bottom Line: If innate intelligence, willingness to exert estimating data collection due diligence, and a really good estimating tool enables an analyst with less than 5 years experience to produce a high quality and credible estimate for a complex system problem, imagine the possibilities the tool presents for those with many years of experience.
