Wednesday, August 17, 2005

Amassing Data.

Well, we have a name now to go along with all the talk on Able Danger, and he is certainly not being shy about the story.

Anyway, since I have been on a discussion of technology fit of late let us continue.

What Able Danger did was to amass huge amounts of publicly available data (whether that data was free or for purchase) and develop a computer system to analyze that data to pull out interesting nuggets of data from the tons of dirt and useless rocks of irrelevant data.

We are talking about terrabytes worth of data. What's a terrabyte? Same but different from megabyte and gigabyte. Mega is 1x106, giga is 1x109, and terra is 1x1012 (fyi, peta is 1x1015 and exa is 1x1018). So, we are talking about a huge quantity of data.

In the not too distant future I am quite sure we will be seeing terrabytes on our micros but right now that is an astonishingly huge amount of data.

So, what did the Able Danger team do? According to Lt. Col. Tony Shaffer (I heard his interview on Michael Savage's radio show) they studied the gang o goobers who bombed the World Trade Center in 1993 (remember that?) and sent their programs coming through their terrabytes of data looking for patterns similar to the 1993 bombing gang. That is the same behaviors that the original Trade Center bombers had were replicated by the successful WTC bombers.

Not surprising considering there were reports of airline pre-screening software systems that fingered a subset of the 9/11-19 for more thorough scrutiny and that program was not looking at nowhere near the data Able Danger's data miner did.

Of course it was asked if any such program is being thought of being run again. I would be upset if it were not already in full gear for sometime.