Sunday, May 11, 2008

The Infinity of Data

Reading Mr. Tony Judt's lovely elegy for history in the New York Review of Books, I was struck by this phrase: "Most people in the world outside of sub-Saharan Africa have access to a near infinity of data.".

Particularly the use of the word "near".

Is the amount of data to which we have access to best described as "near-infinite"? Or is "infinite" itself a better descriptor?

There are now well over 2 million articles on Wikipedia. There are 24,000 texts on Project Gutenberg. Little Canada published 20,000 new books last year; the United Kingdom published over 200,000.

Even assuming a great deal of overlap, there's a phenomenal amount of data being created on a daily basis in our world. If data is being created faster than anyone can ingest it, doesn't that mean that there actually is an infinite amount of data available to us?

What does that mean? What does it mean to say that nobody can grasp human knowledge? Or even keep up? How do we keep track of where we are if none of us can take in enough of the new data to maintain a "50,000-foot view" of the human race's progress?

I think it partly means abandoning the idea of progress in the first place. How can anything so unsupervisable be referred to as "progress"?

Mr. Judt's axes are sharpening in a different direction; he is taking on the idea that torture could ever be acceptable in a republic devoted to freedom. But I think part of what makes Guantanamo Bay and Abu Ghraib possible is the "infinity of data" that he tosses off -- in a world where we accept that we can never acquire ALL the information, aren't we in danger of being unable to ever trust that we have all the RELEVANT information?

And without all the relevant information, how does each one of us find our way? Step by step, through the fog, always uncertain of our path? It's strange to think that an infinity of data requires us to shrink our horizons, to minimize our worlds. But maybe stepping smaller and lighter isn't such a bad thing.

Bridge photo: Utpal Deka