What’s the best way to judge cyber skills?
Most information security workers in the U.S. are judged by the certifications they hold, and those certifications are earned by taking exams administered by a handful of independent groups. Not everyone's convinced that asking questions is sufficient to measure the skills of thousands of cyber professionals at NSA, Cyber Command and other agencies.
Among them are experts at the Maryland information company TeleCommunication Systems. TCS has developed a software tool that lets agencies assess the performance of workers as they pluck their way through live test scenarios in carefully controlled cyber sandboxes.
Learning cyber skills at TeleCommunication Systems
The TCS software, called PerformanScore, is among a new breed of “keyloggers on steroids,” said NIST’s Ernest McDuffie, leader of the U.S. National Initiative for Cybersecurity Education, which is working to improve the U.S. cyber workforce.
The arrival of the TCS tool is likely to fire up the debate over written tests versus hands on exercises, a debate that McDuffie said shouldn’t be a debate at all.
“It’s really not an either or situation. You want to do both,” he said. An exam is best for testing knowledge, while a lab exercise is better for assessing a worker’s ability to work under time pressure, McDuffie said.
TCS hopes to license and install the new software for government customers.
“We can validate that the person’s not just book smart,” said Dan Callahan, product manager at TCS’ Art of Exploitation center in Hanover, Md. As an example, he said the software records the commands a worker would type to scan a network for vulnerabilities during penetration testing.
TCS won’t talk about specific customers, but its exploitation center is located about eight miles from Fort Meade, home to NSA and U.S. Cyber Command.
TCS says it has no visions of putting any certification organizations out of business. “Some of those certification bodies are our friends,” Callahan said.
But Callahan said TCS has gotten feedback that the certifications alone are “too broad” to prove “that these guys have the necessary skills and abilities to do that job.”
It’s not a consensus view.
The power of a detailed exam shouldn’t be underestimated, said W. Hord Tipton, executive director of the 90,000-member International Information Systems Security Certification Consortium, or (ISC)2.
“You get a slice of what the person is doing with your lab exercise,” but with “a well-thought-out, well-designed written exam, you get the full scope of what they’re capable of,” Tipton said. That's how his group certifies information security professionals.
Tipton said (ISC)2 just finished weighing whether to include lab exercises in its new Certified Cyber Forensics Professional certification designed for litigation consultants, cyber intelligence analysts and others.
(ISC)2 sought advice on the lab question from experts at FBI and CIA and from universities and organizations in Hong Kong, India, and South Korea. The experts concluded “we can gather all of that information through properly designed scenario-based questions,” Tipton said.
Tipton's organization conceived the certification with experts in South Korea who are now adapting the forensics exam to reflect South Korean laws and regulations. That should be done by October, Tipton said.
