By William Jackson
THE COMMON CRITERIA Evaluation and Validation Scheme has been heavily
criticized lately (GCN.com, Quickfind 850). Devised as an independent
evaluation of security products against a set of standard criteria,
Common Criteria has been faulted for being expensive and not providing a
foolproof measure to increase security. Not everyone shares these views.
Mary Ann Davidson, chief security officer at Oracle, for one, feels
Common Criteria has a number of strengths.
GCN: It seems as if the perceived value of a Common Criteria evaluation
depends in large part on how a vendor approaches the process. Those that
put the most into it get the best value from the investment. Is this
DAVIDSON: The value of assurance is the extent to which a vendor
embraces it across its development processes. That said, since every
vendor of [information technology] products claims, Our product is
secure: trust us! having a third party validate the product against the
Common Criteria is tremendously valuable to customers, who otherwise
would have to rely on unproven security claims. Also, many vendors,
including Oracle, view the Common Criteria as the starting point for
assurance, not the ending point.
GCN: How do you use the Common Criteria evaluation to create a reliable,
repeatable development process?
DAVIDSON: The Common Criteria allows vendors to start their evaluations
with a lower Evaluation Assurance Level and improve their processes to
meet a higher assurance level over time. The higher in assurance levels
you go, the more aspects of your development process the evaluators
validate, and thus you need more process to meet the requirements. This
avoids an all-or-nothing benchmark that few vendors could meet and
allows them to improve their assurance over time.
GCN: How helpful are automated vulnerability assessment tools in
improving the quality of your products and in achieving evaluation?
DAVIDSON: Automated vulnerability assessment tools do not come into play
in Common Criteria until you reach those Evaluation Assurance Levels
that are higher than those mutually recognized under the Common Criteria
Recognition Arrangement. The national schemes that use such tools do not
release them to vendors, which means they are of no use in helping
improve product security.
The main value of automated vulnerability assessment tools is finding
and fixing problems during development, before products ship. Also,
automated vulnerability assessment tools are just one component of a
robust, comprehensive assurance program. Oracle uses multiple tools as
part of its Software Security Assurance program.
GCN: What are the weaknesses of these tools, and why are they not
required in Common Criteria?
DAVIDSON: There is no tool validation program. Anybody can create a tool
and even if wildly inaccurate claim they found a problem, and the burden
is on the product vendor to prove that there isnt a problem instead of
on the tool vendor to prove its tool is accurate.
Automated tools can find, at best, half the common security defects in
software, and they miss many design defects. Also, in a product with
millions of lines of code, if the tool has a 90 percent false-positive
rate, the vendor could spend thousands of hours in nonrecoverable time
chasing false alarms instead of actually improving security. Finally,
these tools do not validate whether a product has any useful security
GCN: How do you synchronize your development process with the National
Information Assurance Partnership evaluation process so products are not
outdated by the time they have been evaluated?
DAVIDSON: The National Information Assurance Partnership is merely one
of the national schemes under which vendors can evaluate their products.
Under the Common Criteria Recognition Arrangement, vendors can do an
evaluation up to Evaluation Assurance Level four that is accepted in
other venues. In Oracles experience, we can evaluate a large, complex
product like our database in about six months. The length of the
evaluation cycle has not been an impediment to customer adoption of the
product. A vendor going through an evaluation for the first time or who
does not have well-developed development processes may take longer to go
through an evaluation.
GCN: How do you select an evaluation laboratory? And is it possible to
shop for labs to get a favorable evaluation?
DAVIDSON: Labs do not have the final say on whether a product completes
an evaluation successfully; the national schemes that certify the labs
do. A lab doing substandard work would face scrutiny by the national
scheme. Generally, vendors shop for labs based on expertise and cost. We
do our evaluations primarily in the United Kingdom and in Germany
because we found these labs to have a higher level of expertise in
Oracle software and an acceptable cost versus labs in other countries.
GCN: A frequent complaint of Common Criteria is that it focuses on
process rather than the product. Has Common Criteria helped improve your
products? If so, how?
DAVIDSON: Oracle continues to invest significant resources in building
market-leading new functionality and products that we evaluate under the
Common Criteria so that our evaluation-aware customers will feel
comfortable using the products. We also continue to improve development
processes; for example, we have formal processes addressing security
vulnerabilities that we have included in our evaluations under flaw
GCN: What are the strengths and weaknesses in Common Criteria as it now
DAVIDSON: Common Criteria has a number of strengths. Common Criteria
considers both threats and the technical remedies needed to counter
those threats; a product must have actual security functionality and
assurance proof points for it. Common Criteria evaluation assurance
levels are graduated, so that vendors can improve their assurance level
over time. Also, secure development processes need to be demonstrable
Common Criteria is flexible. Vendors can assert a security claim, such
as the use of automated vulnerability tools, through the target of
evaluation they choose. And because of the Common Criteria Recognition
Arrangement, evaluations are cost-effective. A vendor can do one
evaluation that is recognized and accepted in multiple venues. Many
vendors who complain about Common Criteria did not experience the
pre-Common Criteria days when vendors evaluated the exact same product
in multiple countries.
As far as challenges, Common Criteria is a committee-led organization
representing 24 countries and is thus often slow to change. There is a
lack of transparency in its structure, its decision-making and its
technical review processes that the Common Criteria Vendors Forum has
raised. The organization is slowly starting to address these issues.
GCN: What changes would you like to see made in Common Criteria?
DAVIDSON: Any critical piece of software should be designed in
consideration of what kinds of threats the product is likely to face and
the appropriate technical remedy for those threats. As threats evolve,
so should Common Criteria.
That said, the National Cyber Security Partnership recommended the use
of automated vulnerability testing tools at lower assurance levels,
which could be a useful change, provided that all the criteria are met.
The tools themselves must be evaluated and validated for what they find
and how well they find it. The vendor can only make claims about use of
automated tools as part of an evaluation based only on the tools they
actually use in development. The vendor must assure that the Common
Criteria Recognition Arrangement still applies: The use of these tools
is mutually accepted in multiple venues. And, finally, the vendor must
continue to maintain control of its source code. That is, the company is
protected by contractual relationships it has with its labs but is not
required to give source to any other party.
GCN: How well is the National Information Assurance Partnership working
with industry to make needed changes in the criteria and processes?
DAVIDSON: Since the National Information Assurance Partnership is just
one of the national schemes under which vendors can do Common Criteria
evaluations, vendors who want the Common Criteria improved should work
within the Common Criteria Vendors Forum. Oracle also believes it is
duplicative and wasteful for individual schemes to want country-specific
variants that would require vendors to evaluate their products only
under those schemes.
1996-2007 1105 Media, Inc. All Rights Reserved.
CSI 2007 is the only conference that delivers a business-focused
overview of enterprise security. It will convene 1,500+ delegates,
80 exhibitors and features 100+ sessions/seminars providing a
roadmap for integrating policies and procedures with new tools
and techniques. Register now for savings on conference fees
and/or free exhibits admission. - www.csiannual.com