The technology is reminiscent of the axed Total Information Awareness program.
Civil liberties and privacy advocates criticized the effort, called Tangram, which is being developed by contractors working for the Office of the Director of National Intelligence.
"They are misdirecting resources towards this kind of fanciful, science-fiction project," said ACLU attorney Tim Sparapani, "while neglecting the basics" of good counter-terrorist detective work.
The office of John Negroponte, the Director of National Intelligence, declined comment on the program, but it is described in some detail in a procurement document posted on the Web by the U.S. Air Force, and officials have said it is being tested without using any data about Americans.
The document says the system will build on previous work by U.S. intelligence agencies "developing systems, tools and algorithms to detect international terrorist activities and planned events" which have developed "methods of ... efficiently searching large data stores for evidence of known (terrorist) behaviors."
The document does not say what kind of information will be used, either to test and develop the system, or when it is deployed. An intelligence official who asked for anonymity told United Press International that the system was being tested using two data sets, one artificial, and the other consisting of intelligence data from the Department of Defense.
"There is nothing in there that does not comply with the regulations on U.S. persons," said the official, referring to the restrictive rules that govern what information U.S. intelligence agencies can collect, analyze and store on American citizens and legal residents.
Nonetheless, the new system is bound to attract controversy because of its similarity to the Total Information Awareness or TIA program, a project run by the Pentagon's Defense Advanced Research Projects Agency.
That program also aimed to detect patterns of suspect terrorist behavior by data-mining huge stores of information about everyday transactions like credit card purchases, telephone calls and travel records.
Alarmed by the privacy and civil liberties implications of the program, Congress in 2003 cut all funding for it, but research continued in different agencies, funded by appropriations in the classified intelligence annex to the Defense Department budget.
Most of that continuing research was conducted by the Advanced Research and Development Activity, a unit formerly based at the National Security Agency but now part of Negroponte's office. The National Journal, which first revealed the existence of Tangram last week, said that office would oversee the new program, too.
"The administration has flat out ignored Congress," said Sparapani. "They renamed it, re-tied the bow around it and off they went."
Some of the scientists that worked on TIA later said the project, headed by indicted Iran-Contra figure John Poindexter, who chose an all-seeing eye in a pyramid as its logo, became an undeserved lightning rod for privacy concerns.
Sparapani said he did not necessarily doubt the assurances about how the new Tangram system was being tested and developed, but said officials were "not making any commitments about the future."
"What is this tool they're developing for? What data is it going to be used on?" he asked.
The Tangram document, a technical guide for contractors, says that researchers have already "developed novel algorithms and methods for linking entities and activities using a guilt-by-association model" also known as link analysis.
"By relying on highly accurate and analyst-vetted knowledge about known terrorists, groups, affiliations and activities, these tools and methods have proven to be very effective at tracking terrorist suspects and detecting their threat event intentions," says the document.
However, despite this, the work is still far from ready for prime time. "Several fundamental challenges remain before the technology can be deployed broadly within the Intelligence Community."
These include that a single question may take "days and weeks" to answer, "Yet, to have any demonstrable improvement in the intelligence process we need to provide answers in hours or minutes."
The document says processing times and efficiency needs to be improved by two or three orders of magnitude, meaning they must be hundreds or even thousands of times faster.
"Despite all the millions they are throwing into this, they haven't got past square one," said Sparapani. The amounts spent on the continuing TIA research are classified, but the procurement document says $49 million has been put aside for the development of Tangram over the next four years.
The National Journal reported that the government last month awarded three contracts for that development at a cost of nearly $12 million. Two of the firms receiving awards -- Booz Allen Hamilton and 21st Century Technologies -- worked on TIA. The third, SRI International, worked on one of its predecessors, the so-called Genoa project.
The procurement document also makes clear how little progress has been made on what some consider the data-mining Holy Grail, the ability to recognize "anomalous and suspicious" behavior patterns, and distinguish them from the random acts of the innocent.
"In large measure, we cannot readily distinguish the absolute scale of normal behaviors," from which deviations would be suspicious, says the document.
"The underlying assumption of existing approaches is that behaviors are a constant that can be described as a graph. Yet, behaviors are not constant." It calls this problem "A recognized gap in current terrorist detection processes."
The Tangram program will also include "collective inferencing techniques" -- a way of scoring whole populations on a kind of suspicion index. "This technique is capable of making simultaneous inferences (scores) about large numbers of likely interrelated entities in large data collections," says the document, cautioning that its use "for real intelligence analysis is still a promise rather than a reality."
Sparapani said it was likely to remain that way, calling the effort "a wild goose chase for a hail-Mary scientific miracle technology that doesn't exist."
He said link analysis, the only approach to have produced any real world result so far, was "just another word for good old-fashioned gumshoe detective work. You have an event, you have a suspect, and then you look at who is connected to that."
He advocated more spending on FBI agents and translators instead.
"Every dollar spent on this is a dollar not spent on proven strategies" for fighting terrorism, he said.