Developing complex systems is a risky business; around one third of all complex engineering projects will fail; two-thirds of engineering projects will fail to achieve all of their goals (INCOSE 2009). One of the biggest challenges with developing systems is delivering the project on budget and on time. Without thinking too hard I am sure that most readers can think of several examples of large projects that have suffered cost overruns and/or which have been delivered late. Often being able to accurately estimate the cost of developing an engineering solution is one of the most important factors in determining project success, often outweighing purely technical considerations. Inaccurate cost estimates for projects can lead to failure or cancellation of a project as it becomes unaffordable. In the worst case poor cost estimation can lead to companies losing money, or maybe even going bankrupt, if the company undercharges for the work they are performing. Estimating the cost of a project is often a critical part of most engineering activities, but is one that is not often discussed. In this blog post we are going to look at what it takes to estimate how much an engineering project should cost. The content of this blog is generic, but broadly describes the approach that we take at Adiuvo.
The Estimation Process
To be able to estimate the cost of a project there are a number of things that we require. These include:
A definition of the work that is to be performed, often embodied in a document such as a statement of Work (SoW) and a set of technical requirements.
A definition of the process that we intend to follow when executing the project, as this will define the activities that we expect to perform.
Some form of schedule that lays out how long we expect the work to take and how many engineers (or other resources) we will require.
A set of metrics that identify how much time we expect to spend executing each activity.
A way of overlaying our metrics onto our process to generate a predicted cost for each activity that we expect to perform.
A risk assessment that analyses the level of risk provision that we should include in the costings.
A set of pricing metrics that allow us to convert our estimate into the price that we will ultimately charge our customers.
We are going to discuss each of the above in more detail, however in broad terms you will need each of the above to be able to estimate a project with any degree of accuracy.
Project Requirements
Most engineering projects require two basic documents as the basis for estimation: the Statement of Work (SoW) and the technical requirements. Both documents are normally provided by the customer at the start of a project, although they may not always be provided as formal documentation. The technical requirements define the solution that needs to be developed, whereas the SoW defines the work that the customer expects the contractor to perform. The technical requirements define things like the functionality, performance, and physical characteristics of the technical solution. The SoW will define things like the deliverables from the project (such as documentation, software builds, physical assets etc.), standards that need to be applied, or specific activities that need to be performed (such as performing tests or analyses).
At the start of the cost estimation activity the first task is generally to convert the technical requirements into some form of solution. This will often be embodied in an architecture or a product breakdown structure that defines the solution that is to be developed. The SoW is then used to define the work that is performed when developing that solution. Generally, the SoW is mapped on to some form of internal company process, being used to tailor the approach that is to be taken when developing the solution. This then brings us on to the next item in our discussion, process.
Process
The second item that we require to be able to accurately estimate the cost of a project is some definition of the process we are going to follow. A process in this context is some definition of the work that is typically performed by the organization when executing an engineering project. In engineering organizations there can be technical processes (such as an FPGA development process) and non-technical processes (such as a project management process). However, in either case the processes defines the work that is to be performed when executing a project. For example, a fairly generic engineering development process may look like this:
Generate requirements.
Develop architecture.
Perform detailed design.
Implement design.
Integrate design.
Verify design.
Transition to production.
Each of the above is an activity that needs to be performed when developing our engineering solution. Often each activity will have more detail that sits underneath it, however even at a high-level we can get a sense of the tasks we are going to perform when executing our project. Without having a defined process we won’t have a repeatable set of activities which means that we cannot easily generate deterministic costings. If our process changes each time we execute a project then the work that we perform will vary from project to project which in turn will mean our cost also vary across projects. Having a repeatable process is therefore key to being able to estimate accurately.
When performing cost estimation, the SoW and technical solution are overlaid onto the process to tailor it to the specifics of the project that is being executed. For example, if the SoW doesn’t require a specific activity to be performed (e.g., verification) this could be tailored out of the process. Likewise, if the technical solution contains three FPGAs then three separate sets of FPGA activities may be required. The SoW and technical solution are therefore combined with the process to produce a definitive list of all the tasks that must be performed on the project in question.
Often the list of tasks that are to be executed on a project are expressed through a work-breakdown structure (WBS). A WBS is a hierarchical definition of all the work packages that are to be performed on a project. The WBS can be expressed as a tree-like diagram or can simply be a list of tasks. However, the WBS is expressed, the key thing is that the WBS contains a list of all the activities that are to be performed on the project and is generated based on tailoring the process to the SoW and technical solution. The WBS will ultimately form the ‘basis of estimate’.
Example of a simple WBS for a project with 2 FPGAs.
Schedule
The next stage in the process is to convert our WBS into some form of realistic schedule. The schedule effectively takes our estimate and time phases it in some way. Normally we would use a tool like Microsoft Project to generate the schedule. The schedule will contain a list of all the tasks along with the expected start and end dates for each task. The dependencies between tasks will also be captured as links between tasks. Advanced scheduling tools like Microsoft Project will automatically adjust the scheduling of tasks based on the dependencies between the tasks in the schedule. The other thing that a schedule will contain is some identification of the resources required to execute a task; typically, this is going to be personnel to perform the work. It should be noted that the duration of a task can be shortened (or extended) by deploying additional resources to work on a task. For example, a task with a 6 months of labor effort could be executed in 3 months by 2 engineers working in parallel. It isn’t always possible to execute tasks in parallel, but often deploying additional engineers onto a project can speed things up to a certain degree.
The generation of a schedule can often be an iterative process. Normally before we try and generate an estimate we tend to try and have some rough idea of what the schedule may look like as this can help to guide our estimate. Then, as we generate the estimate, we will refine the detail of the schedule. For example, we might initially think that our concept design task will take 1 month and reflect this in our initial schedule. However, when we generate the estimate we realize that there is 3 months’ worth of work involved in this activity. We therefore initially extend the task to three months however this then causes a problem with the project completion date extending out too far. We then look at whether we could deploy additional engineers onto the task to complete the work in a shorter period. For example, we might put 2 engineers onto the concept design task so that it will only take 6 weeks to execute.
The schedule is important as all projects have both a cost and a time dimension to them. The time dimension is captured within the schedule. We will next look at how we generate the cost element.
Metrics
The next step in our estimation process is to estimate the cost for the project. To do that we require metrics that identify how much we expect each task in our project to cost (either in monetary terms or in terms of hours of labor). It is worth pointing out that the term “cost” means the raw underlying expense incurred to execute some form of activity whereas “price” is the amount that we charge the customer taking into account things like profit. Metrics normally capture the cost incurred to execute some defined activity on prior projects. For most engineering activities metrics are often collected as the number of labor hours required to execute a given task. For material assets (like test equipment, PCBs etc.) metrics are collected as a monetary value. Normally metrics are collected over a number of projects by recording the cost to perform specific tasks on that project. Labor hour metrics for engineering tasks are often collected using a tool such as Jira or with a basic spreadsheet. If an organization has no metrics this can make generating an accurate estimate a difficult task. It is therefore often a good idea to build metric collection into the way an organization works to help ensure that good metrics are available when required. Even collecting metrics across small tasks can be useful. If no metrics are available then the estimator will have to make their best guess on how long a task will take. However, guessing the duration of a task often falls foul of the ‘planning fallacy’ where humans significantly underestimate how long future tasks will take. This can make estimates that rely on guessing (or “engineering judgement”) fairly inaccurate.
Estimate Generation
Now that we have a WBS, a schedule and a set of metrics we can then generate an estimate. At a high-level what we are going to do is overlay our metrics onto our project WBS taking into account our schedule. This process will generate a cost for each activity in the project. Tasks requiring employees to expend time will be expressed in labor hours whereas tasks that require buying something will be expressed as a monetary value.
When generating an estimate, metrics often need to be scaled up or down depending on the complexity of the work being performed. Metrics are therefore often expressed as the time taken to produce a specific output. For example, in many software projects the “cost per line of code” is often used as an estimation metric. Generally, most metrics are defined as some kind of “cost per design unit” where the design unit can be something like a line of code, a block of functionality, or even a larger element like a PCB. By expressing metrics as a “cost per” the metric can be scaled up or down as needed based on the project complexity.
To generate an estimate for a given activity in our WBS, we may therefore need to estimate how many “design units” the work will contain. For example, on a software project we may need to estimate how many lines of code a given piece of software will contain. This in itself may also rely on other metrics from previous projects. For example, the number of lines of code in a similar piece of software may be used or the average number of lines of code in a typical file could be used. However this is done, the overall aim of this phase of the estimation process is to generate a cost, using our metrics, for each activity in the WBS.
The schedule can also often impact our costs, particularly when considering tasks that are described as “level of effort” tasks. A level of effort task is a task that isn’t finite in scope, but which extends over some period of time. For example, if your customer requires you to support a weekly status meeting then the total cost of supporting these meetings will depend on the duration of the project. This is where our schedule comes in. For any level of effort tasks (often these are management related tasks) we scale the expected cost based on the schedule duration.
At the end of this phase of the estimation process we should have an estimate of the baseline cost that we expect to incur when executing the project. These costs then form the primary input into the final stage of our estimation process, pricing.
Risk Assessment
Before we move on to pricing, we also need to understand what level of risk our project contains. All engineering activities have some degree of cost or schedule risk within them. Novel, complex projects often contain more risk than routine, simple projects, however all engineering estimates should contain some level of risk provision. To try and estimate the required level of risk provision, a risk register is normally generated. A risk register contains a record of all the identified risks for a project along with an analysis of the impact and likelihood of each risk.
To perform a risk assessment the foreseen risks must first be identified. This is normally done by the estimation team as they go through the initial stages of the estimation process. Each risk that is identified is recorded in the risk register. Once the risk register has been populated with a set of risks the likelihood and impact of each risk must be analyzed. The likelihood of a risk occurring is generally expressed as a “low”, “medium” or “high” rating, where “low” means the risk has a low chance of occurring and “high” means the risk is likely to occur. The qualitative likelihood of occurrence is then converted into a numerical probability. For example, a “low” likelihood might equate to a 10% probability of occurrence whereas a “high” risk might equate to a 40% probability of occurrence. It should be noted that a risk that has more than a 50% probability of occurrence is actually classed as an issue and not a risk. An issue is something that is almost certain to occur and needs to be addressed within the baseline estimate.
Once the risks have been captured and their probability of occurrence defined, the next thing is work out what the impact of risk is were it to occur. Almost all risks will manifest as either a cost or schedule impact (or sometimes as both). The risk register will contain a definition of the expected “gross impact” were the risk to occur. The “gross impact” is the total cost (or schedule duration) that would be required to fully address the risk were it to occur. Often the total gross risk on a project can be significant; sometimes being larger than the baseline estimate. It is therefore often impractical (and unnecessary) to provision for the full costs of mitigating the identified risks. Instead, what we do is we factor the gross risk provision down to produce a “factored risk” provision. We do this by multiplying the probability of the risk occurring by the gross risk provision. This factored risk provision takes account how likely the risk is to occur. The idea is that some risks will occur whereas others won’t. The aim is that the overall factored risk provision should provide sufficient costs to mitigate those risks that do occur. The factored risk provision will be added to our estimate to provide some contingency to the project for addressing any risks that occur.
Pricing
The final stage in our estimation process is to convert our total cost into a price that we will charge our customer.
The first step in this process is to convert our baseline estimate from labor hours into a monetary value. To do this we need to understand how much it costs the project for each hour of labor that it expends. This cost includes the obvious cost of paying the workers’ salaries, but also includes less obvious costs such as paying for the working environment (office space, power, water etc.) and the costs associated with the tools required for the employee to do their job (IT equipment, software licenses etc.). This cost can also include other provisions such as paying for the employee’s pension, insurance for the company, or even paying for management overhead to supervise the employee. These costs are all generally called direct costs. There can also be indirect costs to each labor hour. These can include things like marketing, business development, or paying for research and development. These indirect costs are also often factored into the cost of an hour of labor.
Once we have converted our estimate into a monetary value we then need to add our level of risk provision. If the risk provision is expressed as a cost in terms of labor hours then this first needs to be converted into a monetary cost as we did with the baseline estimate. The sum of the baseline estimate and the risk provision then yields the total estimate for the activity.
Once we have the total estimate, we then need to convert this into a selling price. Projects can generally either be contracted as “firm, fixed price” or as “cost-plus”. A firm, fixed price project is one where the company will agree a firm price for the scope of work. If the project costs more than expected, then ultimately this additional cost will erode the company’s profit margin. In this way the company doing the work takes on the risk. A cost-plus project is one where the customer agrees to pay for all the costs on the project plus as agreed level of profit to the contractor. In this way the customer takes on the risk by agreeing to pay the costs even if they are larger than originally expected. Most commercial projects are normally bid as “firm, fixed price”. Cost-plus projects are normally used for large, high-risk government procurement projects where the level of risk would be too significant for a contractor to take on. The type of contracting route will impact how the project is priced.
For a cost-plus project the customer will normally be provided with the raw estimate and the cost associated with that estimate. The customer and the contractor will then agree a level of profit that will be paid on top of the costs incurred. Normally the customer will hold the risk provision for the project and as such this would be excluded from the agreed price.
For a firm, fixed price project the selling price that is given to the customer will be based on adding a level of profit to the estimated costs. The selling price will also take into account things like exchange rates (if selling to other countries) and the cost for things like warranties. The final selling price needs to account for every cost that the project will incur, as well as allowing for a reasonable risk provision, to ultimately allow the contracting company to make a profit from the agreed price.
One of the key considerations when pricing a project is “the price to win”. Whilst every company would like to have a large, low-risk estimate that guarantees they make a healthy level of profit, often these types of estimates can exceed what the customer is willing to pay. There is therefore a balancing act between ensuring that the company can execute the project and make a profit, versus charging what the customer is willing to pay. The amount that the customer is willing to pay (which can be impacted by the price that a competitor is expected to charge) is expressed as a “price to win”. The price-to-win is the price that the contracting company believes that they can reasonably charge and still win the work. If an estimate is above the price-to-win then the contracting company either needs to find a way of reducing the estimate (often by changing the scope of work) or by reducing the level of profit they expect to make.
Once the price has been generated this will then be provided to the customer. There may then be a period of negotiation, particularly for large drawn-out procurement processes. However, assuming that the customer accepts the price a purchase order will (hopefully!) be issued and the project will get underway. The next challenge is then ensuring that the project stays within the cost envelope defined by the estimate. This, however, is a significant topic and is one for another blog!