News
Stomping cognitive bias
How technology could help
Imagine you’re the intelligence analyst of the future and you think you’ve just solved a vexing international puzzle for your leaders, when suddenly a gentle voice rises from your computer: “Excuse me Dave, other explanations with more weight are available. Would you like to see them?”
This kind of ghost in the machine is called a software agent. Technologists are almost certain to avoid any similarities in their designs to H.A.L, the all powerful computer in “2001: A Space Odyssey.” But software agents are, in fact, a serious idea for mitigating cognitive biases – the brain’s natural tendency to find shortcuts through complex information.
Intelligence managers took on the bias problem after the Sept. 11 terror attacks, and redoubled their efforts after the wildly wrong 2002 national intelligence estimate about Iraq’s weapons of mass destruction. For the most part, agencies addressed bias by training analysts to use new structured analytic techniques – software is now used to chart and visualize evidence, arguments and logic.It’s hard to train away all cognitive biases, though, because the human mind innately applies mental shortcuts when faced with big data dumps. The mind falls back easily on previous experiences, for example, which can lead to confirmation bias. Cognitive biases should not be confused with stereotypes or prejudices. They’re much more subtle, cognitive experts say.
Scientists such as those at Raytheon think the time is right for science and technology to play a larger role in combating bias – but with limits. If software agents were to be deployed – one researcher says their use in the community is “incipient” at this point – analysts would have to retain the power to take or leave the computer’s advice.
“The analyst’s reaction to a tool like that would depend on how intrusive it is -- whether it’s Big Brother,” says former intelligence analyst James Steiner, who retired from the CIA in 2005 and teaches at Rockefeller College, an arm of the State University of New York at Albany.
This article is based on interviews with cognitive researchers, former analysts and industry experts. It shows that the community has options it has yet to exercise. Software agents are just one. Experts applaud the structured analytics trend but they decry the lack of quantitative evidence about their effectiveness in different scenarios. What if information technology could be applied to monitor analytical strategies and outcomes over time? Some of the structured techniques have yet to be institutionalized out of concern that they could overburden time-pressed analysts who find old-fashioned intuition to be a lot faster.
Raytheon is emerging as the most outspoken advocate of applying information technology and experimentation to the cognitive bias problem. The company wants to monitor the decisions of analysts in lab exercises, during training, and eventually in real time while they’re working. The monitoring could help the community assess which techniques work best, and give analysts instant feedback so they can avoid mistakes, rather than simply leaving an audit trail for investigators.
“Awareness alone isn’t enough to overcome (bias) – this is what research has shown. You have to have some way to intervene when the bias is emerging,” says Raytheon researcher and PhD student Don Kretz, who studies cognitive bias in complex problem solving scenarios.
Kretz’s words echo those in a prepared statement from IARPA, the Intelligence Advanced Research Projects Activity: “Research to date has found that simply knowing about the bias is not sufficient to help the individual avoid the bias.”
IARPA wants to see if specialized video games can improve training.
Raytheon wants to do even more. Installation of software agents is one idea.
Rand’s Gregory Treverton, who oversaw the drafting of national intelligence estimates from 1993 to 19995, says software agents might actually empower analysts if they are implemented in a non-heavy handed way.
“One of the things that people learn best from, the psychologists tell us, is feedback. And the closer it is to the event the better,” he says.
GAME TIME
Last June, 27 Raytheon employees, mostly from the engineering ranks, volunteered to participate in a one-day intelligence analysis exercise. The purpose was to show how elaborate games might be used to compare different structured analytic techniques for effectiveness. Raytheon plans to release the paper Nov. 15 at the IEEE Homeland Security Technologies conference in Waltham, Mass.
The company hopes the exercise and a similar one in August will prompt its mission partners – meaning intelligence agencies – to assign working analysts to participate in one or more exercises late this year or early next. Talks are underway, Kretz says.
In the June pilot, the volunteers were given fictional intelligence reports, cell call transcripts, and embassy reports describing three bombings near the Green Zone in Baghdad. The information was drawn from a library of synthetic counterinsurgency, or SYNCOIN, scenarios developed by Penn State University with funds from the Army Research Office. Participants had to figure out who planted the bombs and who the intended targets were.
The game designers made things difficult for the participants by including facts intended to draw out anchoring biases – the tendency of human minds to glom onto evidence supporting an initial hypothesis – and confirmation biases, the tendency to put more weight on evidence that confirms an underlying belief.
The participants were briefed on three techniques for "debiasing" their analysis and given simple software tools. Raytheon calls these techniques analytic multipliers.
Some participants used link analysis software to chart the relationships among people, organizations, objects and events.
Others used an information extraction and weighting technique to assign numbers from 1 to 10 to specific pieces of evidence based on how critical they saw them – 10 being the most critical. The evidence and numbers were typed into spread sheets.
Others used software to draw up matrices documenting the competing explanations and the evidence supporting them. That technique is called Analysis of Competing Hypotheses or ACH.
EARLY RESULTS
How did the comparison turn out? The first thing Raytheon wants you to know is that picking a winner was not the main point. The purpose was to develop an experimental protocol to show how the community could, over time, assemble quantitative information to document the effectiveness of different techniques.
For CIA-funded researcher Richards Heuer, it’s about time someone gathered performance data about the structured techniques he advocates: “This is exactly what we need more of -- empirical testing and documentation of the effectiveness or the ineffectiveness of various analytic tools and a better understanding of which tools are best used for which type of problem assembled over time,” he said by email.
Raytheon has discussed its research with Heuer, but he is not affiliated with the Raytheon team.
To the degree we should care about the result, it was good news for advocates of ACH, a 30-year old structured technique developed by Heuer.
“The Analysis of Competing Hypotheses technique did in fact improve judgment the most,” says Kretz who is one of the paper’s authors and is scheduled to speak at the IEEE conference.
Heuer came up with ACH in the 1980s after retiring from the CIA and continuing his research as a contractor to the agency. His goal was to improve analysis of the Soviet Union:
“When analysts were asked if they thought the Soviets were deliberately trying to deceive us on a specific point, they would almost invariably say ‘No’, because they didn’t see any evidence of it,” Heuer recalls. “What they didn’t recognize was that if the deception was being done well, they shouldn’t expect to see evidence of it.”
ACH helped the CIA sort deception from reality. For about 15 years, that’s the only reason it was taught and used, says Heuer.
Then came the Sept. 11 attacks and the Iraq weapons fiasco. The agencies took a fresh look at ACH and other structured techniques. In 2010, Heuer and Randolph Pherson authored an unclassified book, “Structured Analytic Techniques for Intelligence Analysis,” which explained how the techniques could help analysts in intelligence, law enforcement and even the business world.
Heuer cautions that ACH is not a “cure-all” for all analysis problems even though it shined in the Raytheon exercise.
Some researchers suspect ACH has the most impact for novice or new analysts, which could explain why it came out on top among engineers pretending to be analysts.
If Raytheon convinces intelligence agencies to play, the June exercise could go down as one of the turning points in the fight against bias. The company insists its advocacy is not about winning the next big contract or selling the agencies more software.
“They’ve got shelves and shelves full of tools that they’ve bought and paid for that they don’t use,” Kretz says. “We’re not trying to develop more stuff like that. We want them to be partners in this.”
TEXT: Petraeus resignation letter to CIA employees
Gen. David H. Petraeus in 2011
President Obama accepted the resignation of CIA Director David H. Petraeus today in a phone call. Petraeus offered his resignation yesterday in a meeting with Obama at the White House, explaining "the circumstances behind" his decision, a White House official said. Petraeus had an extramarital affair. CIA posted this statement on the agency web site at 2:50 p.m.:
Message from General David H. Petraeus (US Army Retired)
Statement to Employees from General David H. Petraeus (US Army Retired)
November 9, 2012
Yesterday afternoon, I went to the White House and asked the President to be allowed, for personal reasons, to resign from my position as D/CIA. After being married for over 37 years, I showed extremely poor judgment by engaging in an extramarital affair. Such behavior is unacceptable, both as a husband and as the leader of an organization such as ours. This afternoon, the President graciously accepted my resignation.
As I depart Langley, I want you to know that it has been the greatest of privileges to have served with you, the officers of our Nation’s Silent Service, a work force that is truly exceptional in every regard. Indeed, you did extraordinary work on a host of critical missions during my time as director, and I am deeply grateful to you for that.
Teddy Roosevelt once observed that life’s greatest gift is the opportunity to work hard at work worth doing. I will always treasure my opportunity to have done that with you and I will always regret the circumstances that brought that work with you to an end.
Thank you for your extraordinary service to our country, and best wishes for continued success in the important endeavors that lie ahead for our country and our Agency.
With admiration and appreciation,
David H. Petraeus
IARPA is serious about video games
By next June, college students and professional intelligence analysts could be playing video games paid for by the intelligence community.
The Intelligence Advanced Research Projects Activity wants to see if it can root out cognitive biases by having future analysts and veterans play specialized video games during training or professional workshops.
Soldiers train with video games. Credit: U.S. Army
IARPA calls the program Sirius as a play on the term serious games, which is what the simulation industry calls video games that are both instructive and fun.
The agency sent Deep Dive a prepared statement explaining that “cognitive biases are notoriously resistant to training.”
Cognitive biases shouldn't be confused with easy to spot prejudices or stereotypes, brain experts say. They're the shortcuts that our minds naturally seek when we’re presented with massive data dumps.
In September, IARPA chose five teams to build competing versions of the games for assessment next year following trials by college students. IARPA is in talks with potential “transition partners in the U.S. Government” to try out the games next year together with students. Those tests will have to show a 50 percent reduction in bias for the developers to proceed any further.
IARPA won’t say how much it’ll pay for the games, but last November, Raytheon BBN Technologies put out a press release placing the potential value of its multiyear contract at $10.5 million. The company was not selected for the new phase that began last month.
The winning teams are led by:
Applied Research Associates
Boeing
SET Corporation (Now part of SAIC)
State University of New York at Albany
University of Oklahoma
OPINION: The dangers of Benghazi politicking
The Obama administration has entered dangerous territory by letting unnamed senior intelligence officials join the public fray over the Sept. 11 attack on the U.S. consulate in Benghazi.
The bodies of four slain Americans arrive at Joint Base Andrews outside Washington. Credit: U.S. State Department
The administration’s frustration at the media’s treatment of Benghazi is understandable. It used to take secret tapes or a source in a parking garage to allege a cover-up. Now all it takes is a TV segment and a Twitter account.
Allowing intelligence officials to rebut Republican talking points is not the right solution. The strategy threatens to bring down the protective wall erected between politics and the intelligence community after the intelligence abuses of the 1960s and ’70s. That wall took a hit in 2002 and 2003 when George Tenet let the Bush administration’s zest to invade Iraq color the community’s analytical processes.
The Obama administration risks damaging the wall further for little return. A few words from unnamed officials are unlikely to convince anyone of anything in today’s charged environment.
Careful readers will wonder why they should trust a prepared statement and timeline provided to news outlets but not posted on the public websites of the CIA, the Office of Director of National Intelligence or the White House, The comments also address only one aspect of the Benghazi controversy – whether the CIA delayed sending help to the consulate. Does this mean all other rumors of incompetence and cover-ups are true?
Perhaps most dangerous of all, the loosely attributed comments could inspire others to straighten out their bosses.
If you doubt that could happen, take a look at the epilogue of “No Easy Day,” the initially anonymous account of the Bin Laden killing by Matt Bissonnette, one of the Navy SEALs who fired the kill shots into Bin Laden’s convulsing chest.
“We always prided ourselves for being the quiet professionals, but the more I saw coverage of the raid, the more I wanted to set the record straight,” writes Bissonnette, then working under the pen name Mark Owen.
How could Bissonnette have the gall? Well, think back to the days after the raid. The White House gave background briefings to favored media outlets; CIA Director Leon Panetta and White House counterterrorism adviser John Brennan stood on separate stages and contradicted each other; clarifications were issued.
The result was a tangled mess, with no one sure whose words to trust. It was a lot like what we’re seeing in the Benghazi case.
There was another option available to the administration in both the Bin Laden and Benghazi cases.
The White House could have used the months and years preceding the raid to line up a retired prosecutor or investigator to quickly assembly an unclassified account in the event Bin Laden were apprehended. Bissonnette and others could have been interviewed, perhaps staving off publication of “No Easy Day” or at least making it a much less interesting read.
In the Benghazi case, something similar could have been done and might still be possible. The equivalent of a newspaper’s ombudsmen should be assigned to quickly assemble an unclassified account of what the administration knows and does not know about the attack and its circumstances.
Quickly releasing transparent, on-the-record information would be a radical departure from the typical Washington approach of dragging out official investigations for months or years.
It’s worth remembering President Obama came into office lambasting those who played “the same old Washington games” while “expecting different results.”
Whoever wins Tuesday will need an intelligence community that is trusted as nonpartisan. This is no time for Washington games, from either Democrats or Republicans.
Intel cuts deepen
The race is on between the budget knife and the intelligence community’s information technology overhaul.
The U.S. appropriated $75.4 billion toward intelligence in 2012, a drop of 4 percent compared to 2011, according to figures released by intelligence director Jim Clapper's office and the Pentagon.
The intelligence community is banking that it can roll out modernized information technologies quickly enough to avoid deep cuts to collections and analysis as it struggles to meet the White House budget reduction target.
If you happened to be watching the CNN news ticker, the 2012 reduction was not, in fact, the first time the intelligence budget has been cut since the Sept. 11 terror attacks. The U.S. appropriated $78.6 billion in 2011, down from $80.1 billion in 2010, a decline of 1.8 percent.
U.S. law requires the Office of the Director of National Intelligence to reveal its portion of the intelligence appropriation, called the National Intelligence Program, within 30 days of the close of the fiscal year on Sept. 30. The Pentagon traditionally follows by releasing its appropriation for the Military Intelligence Program. Taken together, these two categories constitute the top line intelligence appropriation.
One school of thought says that releasing these figures will ensure a vigorous public debate over the appropriate level of intelligence spending. The appropriations for specific technology and networking programs, however, remain disguised within the annual defense appropriation act, which makes it hard to publicly assess the community's spending priorities.
For 2012, the figures break down like this:
The U.S. appropriated $21.5 billion in 2012 for the Military Intelligence Program, down from $24 billion in 2011 and $27 billion in 2010.
The U.S. appropriated $53.9 billion in 2012 for the National Intelligence Program, down from $54.6 billion in 2011 and $53.1 billion in 2010.
Odierno talks technology
The Army’s chief of staff enjoys challenging the technological status quo.
Ask Gen. Raymond Odierno about radios and he gushes over the capabilities of "iPhones," a term he uses generically for smartphones. Ask him about the controversy over Palantir’s link analysis software, and he says the Army might yet figure out how to include it in the service’s intelligence production and sharing toolbox.
Army Gen. Raymond Odierno
Adding the Palantir software to the Distributed Common Ground System-Army would be a big breakthrough for the Palo Alto company. The Palantir software is popular among analysts outside the Army as an easy way to visualize the linkages among terror suspects or criminals.
The Army has tested the software in the lab and in Afghanistan with good results but has resisted embracing it in the DCGS-A network for reasons other than the software's performance. The Army says Palantir's unique design makes it hard to ingest or share data created by non-Palantir software. For two years, Palantir has sought to shoehorn the software into DCGS-A by appealing to Congress and to the media via proxies.
The Army wants all new software to follow a guide called DI2E, short for the Defense Intelligence Information Enterprise, which is the equivalent of a building code book for software developers.
“Right now, there’s some proprietary issues with Palantir. We’re working those now,” Odierno said after a press briefing at the Army Association’s annual conference in Washington, D.C.
At the same time, he warned against hyping the significance of whether or not Palantir’s software is added to DCGS-A:
“What I got frustrated over is -- it’s not DCGS versus Palantir. DCGS-A is much larger than what Palantir is delivering,” he said.
Odierno also defended the performance of DCGS-A: “It’s a huge network, which I think is very, very good.”
As for the smartphones, Odierno said getting more of them into the hands of soldiers could help commanders understand events in the field in close to real time.
“One of our weaknesses now (is) we have all these sensors out there -- squads of people who have information, and when they report it up, it takes a very long time. With these iPhones, it can almost be done instantaneously,” he said.
DIA chief faces surprising foe
The Defense Intelligence Agency director is getting serious push-back from within his agency over his plan to reorganize around one of the agency's core missions: collections of intelligence abroad.
U.S. Army Lt. Gen. Michael T. Flynn had a quick answer when asked what he would do if people in his agency resist the changes -- “Move them or fire them,” he told an audience at the GEOINT symposium in Orlando. The comment drew rare applause from the group of industry executives and intelligence officials.
Afterwards, in a brief interview with Deep Dive, Flynn showed no remorse about that simple answer. “It’s a bureaucracy,” he said. “People like their little worlds.”
Lt. Gen. Michael T. Flynn on stage at GEOINT symposium
In April, DIA began the process of converting the Defense Counterintelligence and HUMINT Center, known as DX, into the Defense Clandestine Service, a name chosen to mirror the CIA’s National Clandestine Service.
Flynn arrived at DIA in July promising big changes, and he used his GEOINT speech to explain how the clandestine service figures in. “I’m going to use (the Defense Clandestine Service) to integrate the entire agency. This is not a marginal adjustment for DIA. This is a major adjustment for national security,” he said.
Flynn said the U.S. will need to work abroad with partner nations to avoid strategic surprises in what he expects will remain a tumultuous century.
“I believe that for the rest of my time, we will be in this era of persistent conflict,” he said.
The changes won’t just involve members of the clandestine service. DIA analysts will be sent to embassies around the world, rather than being restricted to joint intelligence operations centers, he said.
Deploying DIA staff more widely would provide “better finger tip feel, better sense of what’s going on, whether it’s in Afghanistan, whether it’s in Central or South Africa, whether it’s in South America, or whether it’s in the Asia Pacific basin somewhere.”
Flynn took pains to emphasize that the changes would not come at the expense of work in Afghanistan.
“My number one priority is Afghanistan in terms of the tactical operational level of stuff that we get involved in,” he said.
The reorganization around the clandestine service is to be described in a document called Vision 2020.
“Vision 2020 is a team effort that will take a broad look at everything we do across the Defense Intelligence Agency and look at how we need to operate in the future. Certainly over the coarse of the rest of this decade and probably at least 2025, 2030 and as far out as we can see.”
After his talk, Flynn said news reports casting the clandestine service as an entirely new entity were overstated.
“It’s an evolution,” he said.
He said a formal notification to Congress about the reorganization would be forthcoming, but that the agency has conducted briefings on Capitol Hill.
“We have to keep congressional staffs and member in the loop along this path,” he added.
DigitalGlobe: Thumbs-up for GeoEye Analytics
DigitalGlobe plans to retain GeoEye’s special-operations oriented predictive analytics unit, should regulators approve the merger of the commercial satellite firms, said Walter Scott, DigitalGlobe’s founder and chief technology officer.
“We think (GeoEye Analytics) is highly complementary,” Scott told Deep Dive in an interview at the GEOINT symposium.
Walter Scott, DigitalGlobe's founder and CTO. Credit: DigitalGlobe
DigitalGlobe, which would lead the merged imaging company, has its own analytics group, but Scott said there would be room for both DigitalGlobe’s Analysis Center and GeoEye Analytics.
DigitalGlobe provides unclassified expertise whereas experts from GeoEye Analytics are “embedded in the customer footprint” to work on classified projects, Scott said.
“We think that there’s a high degree of synergy there,” he added.
GeoEye Analytics, formerly known as SPADAC, combines satellite imagery with intelligence data to try to anticipate where IED planters, domestic criminals or fugitives like Uganda’s Joseph Kony might go next. GeoEye bought SPADAC in 2010.
One of the company’s human terrain analysts, Kevin Stofan, received the Lt. Michael P. Murphy Award in Geospatial Intelligence at the 2010 GEOINT symposium.
GeoEye CEO Matt O’Connell talked up GeoEye Analytics last month in an address to the Washington Space Business Roundtable group.
“We helped apprehend a rapist in Philadelphia, by okay, ‘Where has he struck in the past?’,” he said. “We just got a commendation from a British special forces unit because we helped find a bunch of raw materials that people had built into a cache. It was obviously for IEDs.”
After the talk, O’Connell told Deep Dive that demand is growing for the unit’s services, especially in the special operations world. “We’ve expanded the Tampa office because there’s so much work,” he said.
Clapper blasts Benghazi “politicking”
Director of National Intelligence James R. Clapper fell back on the words of an old friend to defend his handling of intelligence about the attack on the U.S. consulate in Benghazi, Libya.
Speaking at the GEOINT symposium in Orlando, Clapper recalled arriving back in Washington from Australia and “reading media clips about the hapless, hopeless, helpless, incompetent DNI, because I acknowledged publicly that we didn’t instantly have the God’s eye, God’s ear certitude about the event.”
Credit: U.S. Geospatial Intelligence Foundation
He was referring to a Sept. 28 statement from his office announcing that the community had shifted its thinking about whether the Benghazi attack was preplanned, a change that became fodder for Republican critics of the Obama administration.
Intelligence officials initially told the White House the attack was the result of a spontaneous demonstration. Newer analysis suggests it was “a deliberate and organized terrorist attack,” according to the statement.
In a talk that was at times caustic, sarcastic, and humorous, Clapper suggested that it was dangerous to inject election year politics into the intelligence process which is just that – a process.
He said an article published last week by retired Georgetown professor Paul Pillar should be instructive to the legislative and executive branches, and to “the fourth estate – the media.”
Clapper read aloud Pillar’s entire 455 word article posted Oct. 1 on the “The National Interest” website. The article makes the case that “hindsight is cheap,” but Clapper put special emphasis on this passage:
“The demand for an explanation that is quick, definite and unchanging, reflects a naïve expectation, or in the present case, irresponsible politicking.”
Later, during the question and answer session, Clapper said it might be time to set a term limit for his position. A fixed term “would be a good thing,” he said, if the country wants to keep the position “functionally apolitical.”
Clapper joked that he might need a new job after Jan. 21. “I’ll have both feet in assisted living,” he said.
Interview: Jump starting GEOINT apps
Job number one for Mark Riccio is to speed up the apps revolution promised two years ago by his boss, National Geospatial-Intelligence Agency Director Letitia Long. NGA has released about a hundred apps so far, but the agency wants as many as thousands out there on all three intelligence networks. Apps for counterterrorism; apps for disaster relief; apps for things no one’s thought of yet. Industry buy-in has been tough to get. Lots of software developers are doing just fine under the old model of meeting rigidly defined requirements. Riccio must sort this out in his role as director of application services within NGA's Online GEOINT Services Directorate. His solution? Gather industry reps in one room and start brainstorming. He spoke to Deep Dive about the Sept. 27 “Industry Outreach Day,” and what comes next. Excerpts below>>
NGA's Mark Riccio speaks to developers. Credit: NGA
Fear of change >> “This is something brand new, and there’s a lot of trepidation as we step forward on this because it hasn’t been done before. If I hadn’t encountered some of that skepticism, or some of the good hard questions that we got, I don’t know if we would have been pushing it far enough…I am very confident that we’re going to be able to succeed in this.”
Buy-in >> “Of the emails and communiqués I’ve gotten after that (industry day), I haven’t gotten any that said, ‘We’re not behind you. We don’t want to work with you. We don’t think we can get this done.’ A hundred percent, to a T so far, everybody says, ‘We want to work with you. This is something we can get done. Let’s kind of collaborate and figure out how we can move forward on this.’
Turnout >> “We had 263 discrete companies (at the industry day) and we are estimating that when all is said and done we had over 420 attendees.”
Next step >> “We don’t want this to be where we got everybody together, and then it’s months and months and months before you hear anything. We are ready to go and we are proceeding at a very expedient pace to get this (acquisition process) in place.”
Competition >> “Our directorate’s mission is to create the best user experience, and I think competition will enable that, and that’s what we’re interested in. This is a big opportunity for a much broader segment of the market to begin working with NGA.”
How many apps? >> “The intent is to spur the marketplace, and to spur the development community to help us fill that app store with hundreds if not thousands of different applications.”
Flexibility >> “The FAR (Federal Acquisition Regulation) offers a lot of capability to do things differently, to do things in a much more agile manner, and we’re going to be exploiting some of that capability.”
Rolling solicitation >> “It’s really different in the sense that instead of being super specific in terms of what we need, and then contracting specifically for those items, it’s more of a broad-based, open-based type solicitation, where we don’t want to be so specific that we hamper innovation.”
Exceptions >> “I think there’s going to be certain instances where we may get proscriptive. Maybe we’ll put something out that says, ‘NGA is interested in an app that does this,’ and we provide additional specifics. But the intent of the compensation model, as it’s (being) built, is to have that flexibility and the ability to cover situations like that.”
Payday >> ”A big part of what industry day was about is to develop a business-to-business application compensation model. And that’s really about publishing that speculative model for app development…It was made pretty clear at industry day that we’re still working on it.”
Innovation >> “We certainly don’t own all those good ideas, and we don’t want to put in place a mechanism that restricts us from enjoying the innovation that’s out there.”
No lemons >> “In the sense of multiple apps competing, just like there’s multiple weather apps, they all provide weather, but maybe some do it better. Maybe some are more pleasing, maybe the interface and utility and the use of that app is a little bit better. We’re not going to put any duds out there in terms of functions not working.”
User friendly >> “There’s a lot of different ways to build an app that shows a helicopter landing zone. The winner in a case like that is going to be the one that provides not only the validated output, but does it in a way that provides the best possible experience.”
Validation >> “If we’ve got an application that generates coordinates, does some kind of coordinate conversion, or gives one a heading, or computes slope, or something like that, we’re going to make sure that the output from that app is accurate and that it’s validated.”
Vendors can adapt >> “There’s a number of examples of traditional desktop software vendors who have made successful implementation of both existing technologies and new technologies within that [consumer app] type of market.”
Briefing Congress >> “We haven’t at this time, but it’s something we hope happens.”
