Speaking MetaData

Last updated 3 August 2013.

One Monsieur Jourdain, a comic character in a French play, is often quoted thus:  "Good heavens!  For more than forty years I have been speaking prose without knowing it."  You, gentle citizen, have been "speaking metadata" all your adult life as well.

A little extraneous background: meta is a Latin word meaning boundry that is used in English as a prefix.  My "Shorter Oxford English Dictionary" (3,742 pages in two volumes of 14 pounds total) defines "metadata noun [COMPUTING] a set of data that describes and gives information about other data."  In short, data about data is key to finding  the data you really want. ( Feel free to set that last sentence to music.  Hip hop maybe?)

An everyday example: if you write a label on the tab of a manila file folder and stick it into your household file of such folders in alphabetical order, you have created metadata out of the alphabet we all learned in childhood and that label you just wrote.  Neither you, Verison, the National Security Agency (NSA) or Google for that matter could easily find stuff without such data about data, organized and at the ready. 

A typical computer example: while switching my files to a new computer my list of email folders vanished.  I could no longer click on the folder name and get a list of the emails the folder contained.  That list of folder names is a file of metadata.  So is the list of email subjects in each folder.  Metadata, like Russian dolls, can be nested. (Yes, my list was finally found.)

What kind of metadata does Verison and the other communcations companies keep on our domestic phone calls that the secretive NSA lusts after?  Who knows for sure at this date?  But if I were NSA I would want as a minimum:

      1.  Call and answer phone numbers, including area codes.

      2.  Date and time and length of the call.

      3.  Location of the calling and answering cell phone towers or best land line equivalents.

What  other metadata does Verison and the others have?  Your name, your full address, customer and credit history with the company. So far there is nothing sinister here.  The companies need this stuff to calculate and mail your bill and clutter up your envelope with tempting offers.  But they also keep for a time in electronic storage the phone call itself, what the media has been calling your "content."  I can't imagine why they would store the lengthy babblings of every love sick teen ager for more than a minute unless the government told them to.  For sure NSA isn't attempting to listen to -- and evaluate -- every domestic phone call as it is being made or, even later.  They could hire India and still not keep up.

Nor can I see the companies keeping the call content of billions of calls for very long while waiting for the FBI to go to the FISA Court for an order to listen in.  Computer systems today have immense storage capacities at costs so low that  emailers like Google, AOL and Microsoft basically give it away in generous chunks to lure us to be customers.  But there is a limit to such freebies and our nation's massive phone call content volumes would easily exceed it in a matter of hours. In contrast, metadata about a call is but a tiny fraction of the size in bits and bytes of the content of even the shortest phone gab fest.)

So this is my guess as to how the system works.  The companies are transmitting their metadata -- less name and address -- to NSA for each phone call as it ends.  NSA receives each such record and adds it to their data base.  Separately NSA has in file suspect phone numbers, both domestic and international, constantly gleaned from intelligence agency suppliers and electronic intercepts from around the world.  With today's technologies it is a simple matter to match these suspect numbers to the phone call metadata from the companies as it comes in, or, as the computer pros term it, in "real time."

Picture these two data streams colliding like particles in an atom smasher.  Only a very few such hits are worthy of further study.  These are flagged and no doubt assigned to an analyst.  I would also expecct that a signal goes back to the phone company to keep the content of the matching calls until further notice.  New matches may widen the circle of linkages from the ceasless flow. For each match at some point the analyst must make a judgement: forget it; keep watching; or turn suspicions over to the FBI.

Let's be clear about one thing: for reasons both technical and economic, getting rid of irrelevent data is second only in importance to "connecting the dots" relevent to each hit.  I believe that the most that NSA can expect the phone companies to do is to archive (store all) its phone calls on high capacity magnetic tape or removable disk for a stated time.  I wouldn't even do that.  If I were the NSA's system's designer the contents of your phone calls not matched in real time to a suspect phone number would not be retrieved from Verison and others.

Most probably NSA's metadata is organized on hard drives in the form of what the techies call an inverted data base.  That means that most if not all of the data elements are indexed.  That in turn means that the NSA analyst sitting at her terminal can retrieve any metadata record by more than one -- and up to any -- of the data elements that are included in each record in the data base.  It's as if back home you kept a separrate card file of the names of your manila folders organized by the date each folder was created so you could find the oldest folder without looking at every folder in your file drawer.

NSA has said that the system doesn't let them tie metadata records to a name.  Only the FBI can do that after it asks the the FISA court via its boss, the Justice Department, for a warrant based on  probable cause.  The FBI takes the case from there.  How well this works in practice is another story.  For more info google "FISCA Court" and read the Wikipedia article.  Or read my blog entry, "FISCA: A Court or a Fig Leaf?"   Or both.