Preserving the born-digital: "The future for digital storage is constant migration."

Since I'm not really done with the previous post, but think this deserves a dedicated one to itself, the article I linked to, on the question of preserving films that were 'born-digital':

The preservation of born-digital films is going to be the greatest challenge ever to face archivists.

Margaret Bodde, Executive Director of the Film Foundation

The new magical software has sometimes led to over-restoration. Grain has too often been polished out, creating a plastic sheen. Still, today no archivist can avoid using the new toolkit. The sadder story involves not restoration but conservation and preservation. A civilian might think: That’s simple. Just save film on film and digital on digital. But things are more complicated than that.

Let’s start with a movie like The Girl with the Dragon Tattoo, which was shot with digital capture. After production and post-production, it was made available to theatres as both a Digital Cinema Package (that batch of files on a hard drive) and some 35mm prints. But there are several digital versions of the movie.

The Digital Source Master: This is the original sound and image “content” captured in specific formats, either tape-based or file-based, and those may come in many flavors. The Girl was shot with the Red One camera on the company’s proprietary format R3D. That material, along with sound recording, was converted to other files in postproduction. Any major film nowadays is likely to use many digital video and audio formats. This entire set of materials forms the Digital Source Master for a film and these assets are usually stored in the studio’s vaults. Along with them are, usually, film-based copies of the final product, often as separation masters.

The Digital Cinema Distribution Master, in standards specified by the Digital Cinema Initiatives. This is the finished film unencrypted and uncompressed, providing “content” at 2K and/or 4K resolution. Roughly speaking, this is the digital counterpart of a 35mm film negative.

The DCP, compressed and encrypted for theatrical playback. It is, again in some respects, the digital counterpart of an analog film print.

Eventually, The Girl will show up on the optical disc formats DVD and Blu-ray, not to mention streaming video, cable transmission, and web-based platforms. (Actually, it’s probably already available for Darknet download.)

Many studio films are housed in nonprofit archives too, and until recently those movies have been deposited and stored as analogue copies. But what will those institutions now keep? There are only three minimally acceptable formats: the uncompressed and unencrypted DCDM, the DCP, and a 35mm print. Suppose your film archive is lucky enough to receive both a DCP and a 35mm print of The Girl.

First, how do you access the DCP files? A DCP is typically encrypted to block piracy. When The Girl played theatres digitally, each exhibitor was provided an alphanumeric password that would open the files for loading into the theatre’s server. By the time you the archivist get the files, that key may have expired or been lost. Without the key, the DCP is useless.

Then there’s the matter of storage. The 35 print of The Girl can simply be passively conserved, following the motto, “Store and ignore.” But all digital material, no matter how minor, requires proactive preservation. The future for digital storage is constant migration.

Archivists estimate the life of any digital platform to be less than ten years, sometimes less than five. All hard drives fail sooner or later, and they need to be run periodically to lubricate themselves. Tape degradation can be quite quick; one expert found that 40 % of tapes from digital intermediate houses had missing frames or corrupted data. Most of the tapes were only nine months old.

Moreover, hardware and software are constantly changing. One archivist estimates that over one hundred video playback systems have come and gone over the last sixty years. Archives currently recognize over two dozen video formats and over a dozen audio ones.

Periodically, then, the DCP files of The Girl will have to be checked for corruption and transferred to another tape or hard drive and eventually to another digital format. Such maintenance takes time; shifting a terabyte of data from one system to another may need at least three or four hours. Ideally, you’d want several copies for backup, and you’d want to store them in different locations.

There are hundreds of other films like The Girl awaiting processing at major archives. About 600-900 features are produced in the US each year. Currently the world is producing about 5500 films per year. At some point, they will all originate in digital capture.

Besides access and storage there’s the matter of cost. Storing 4K digital masters costs about 11 times as much as storing a film master. You can store the digital master for about $12,000 per year, while the film master averages about $1,100.

How do the overall costs of digitizing mount up? Look at the situation in Europe. The EU countries produce about 1100 features and 1400 shorts per year. An EU archival commission, the Digital Agenda for European Film Heritage, estimates that to conserve one year’s output would require 5.8 PB (petabytes) of storage. In 2015, the costs of archiving that year’s output (without restoration) are projected to be between 1.5 million and 3 million euros. Beyond initial conservation, long-term preservation of that year’s output would consume, though migration and backing up, about 1900 PB and cost about 290 million euros.

The access problem is soluble. Your archive could be given an unencrypted DCP of The Girl and then create its own key to prevent copying. Or the DCP could be assigned a generic key, perhaps for a specified time period, that will open the files in a secure milieu. They could then be migrated to an format under archive control. On the matter of software, archivists are working on establishing standard preservation file formats and codecs. To deal with the other problems, you’d have to press for increased budgets and personnel to cover the new duties that digital archiving creates. But the costs, including training personnel on ever-changing platforms, are of tidal-wave proportions.

So there you are: first there's the ease of technology and then - as any editor will tell you - the sheer migraine it induces.

**

Not entirely related, but this: "Once you’ve found a way to conserve-preserve The Girl with the Dragon Tattoo, what if you want to show it tomorrow? Or ten years from now? Or fifty?"

I feel they could have chosen another film as an example, because this just....makes me giggle.

The Archivist of the Ephemeral

The British Library here had a panel discussion and inauguration of an exhibition that's travelling across the country, about South Asians in Britain. Someone from the audience asked the panelists why they did their research only at the V&A, the National Archives of India and all the usual places. Why were they not looking for the more rare manuscripts, archives and material that was surely available in other places, the man wanted to know.

I was thinking of this when I read this article about the digital preservation of film (via The Valve), and about the fire that destroyed so much of the Film Archives at Pune. The whole enterprise of archiving anything seemed impossible, brave and quixotic.

Imagine you are on the threshold of your career and, with the thoughts of material success and respectability dinned into you, you still choose a life that is not just hard to justify as signifying 'success' in the usual way, but is actively futile and Sisyphian.

You become an archivist. You might as well have chosen to be a mortician.

**
There were children who had been invited to be placeholders yesterday. I don't know who was fooled by the tactic; certainly not the panelists, who took it with fairly good grace. (This is not meant to insult the intelligence of the children; they were clearly brought there with no preparation about the nature of the exhibition, or given a context for the discussion. I doubt they gave a hoot about Krishna Menon's Pelican Series of non-fiction, the first Asian woman to study law at Cambridge or the intricacies of a pre-Independence Indian being an  MP in the UK.)

It occurs to me that anyone interested in the archival is already a specialist with a specialist's peculiar interest in what is being resurrected.Who but someone interested in silent comedy is going to care that a 'not yet found' Harold Lloyd short was buried under the permafrost somewhere in Canada and has now been restored? Perhaps the only way to answer that question is by not putting the actual found film to the test, but to make, say, The Artist.

It's hard enough for a new generation to pay its respects to the one gone by via what is already available and accessible. I say this with a tinge of bitterness, because I'm finding it unexpectedly hard to get the kid to read Wodehouse. Wodehouse! This kind of behaviour is calculated to make me shake my head more in sorrow than in anger and moan about the present (de)generation....but I digress.

It's hard enough to get what someone older goes on about; how hard it is going to be to make anyone care about what is deeply past, unless they already care about it.

And therefore how doubly, triply admirable that so many people dedicate their lives to a monumentally impossible task: keeping the past alive in its primary forms.

**

In Praise Of Limestone

by W. H. Auden

 

If it form the one landscape that we, the inconstant ones,
Are consistently homesick for, this is chiefly
Because it dissolves in water. Mark these rounded slopes
With their surface fragrance of thyme and, beneath,
A secret system of caves and conduits; hear the springs
That spurt out everywhere with a chuckle,
Each filling a private pool for its fish and carving
Its own little ravine whose cliffs entertain
The butterfly and the lizard; examine this region
Of short distances and definite places:
What could be more like Mother or a fitter background
For her son, the flirtatious male who lounges
Against a rock in the sunlight, never doubting
That for all his faults he is loved; whose works are but
Extensions of his power to charm? From weathered outcrop
To hill-top temple, from appearing waters to
Conspicuous fountains, from a wild to a formal vineyard,
Are ingenious but short steps that a child's wish
To receive more attention than his brothers, whether
By pleasing or teasing, can easily take.

Watch, then, the band of rivals as they climb up and down
Their steep stone gennels in twos and threes, at times
Arm in arm, but never, thank God, in step; or engaged
On the shady side of a square at midday in
Voluble discourse, knowing each other too well to think
There are any important secrets, unable
To conceive a god whose temper-tantrums are moral
And not to be pacified by a clever line
Or a good lay: for accustomed to a stone that responds,
They have never had to veil their faces in awe
Of a crater whose blazing fury could not be fixed;
Adjusted to the local needs of valleys
Where everything can be touched or reached by walking,
Their eyes have never looked into infinite space
Through the lattice-work of a nomad's comb; born lucky,
Their legs have never encountered the fungi
And insects of the jungle, the monstrous forms and lives
With which we have nothing, we like to hope, in common.
So, when one of them goes to the bad, the way his mind works
Remains incomprehensible: to become a pimp
Or deal in fake jewellery or ruin a fine tenor voice
For effects that bring down the house, could happen to all
But the best and the worst of us...
That is why, I suppose,
The best and worst never stayed here long but sought
Immoderate soils where the beauty was not so external,
The light less public and the meaning of life
Something more than a mad camp. 'Come!' cried the granite wastes,
"How evasive is your humour, how accidental
Your kindest kiss, how permanent is death." (Saints-to-be
Slipped away sighing.) "Come!" purred the clays and gravels,
"On our plains there is room for armies to drill; rivers
Wait to be tamed and slaves to construct you a tomb
In the grand manner: soft as the earth is mankind and both
Need to be altered." (Intendant Caesars rose and
Left, slamming the door.) But the really reckless were fetched
By an older colder voice, the oceanic whisper:
"I am the solitude that asks and promises nothing;
That is how I shall set you free. There is no love;
There are only the various envies, all of them sad."

They were right, my dear, all those voices were right
And still are; this land is not the sweet home that it looks,
Nor its peace the historical calm of a site
Where something was settled once and for all: A back ward
And dilapidated province, connected
To the big busy world by a tunnel, with a certain
Seedy appeal, is that all it is now? Not quite:
It has a worldy duty which in spite of itself
It does not neglect, but calls into question
All the Great Powers assume; it disturbs our rights. The poet,
Admired for his earnest habit of calling
The sun the sun, his mind Puzzle, is made uneasy
By these marble statues which so obviously doubt
His antimythological myth; and these gamins,
Pursuing the scientist down the tiled colonnade
With such lively offers, rebuke his concern for Nature's
Remotest aspects: I, too, am reproached, for what
And how much you know. Not to lose time, not to get caught,
Not to be left behind, not, please! to resemble
The beasts who repeat themselves, or a thing like water
Or stone whose conduct can be predicted, these
Are our common prayer, whose greatest comfort is music
Which can be made anywhere, is invisible,
And does not smell. In so far as we have to look forward
To death as a fact, no doubt we are right: But if
Sins can be forgiven, if bodies rise from the dead,
These modifications of matter into
Innocent athletes and gesticulating fountains,
Made solely for pleasure, make a further point:
The blessed will not care what angle they are regarded from,
Having nothing to hide. Dear, I know nothing of
Either, but when I try to imagine a faultless love
Or the life to come, what I hear is the murmur
Of underground streams, what I see is a limestone landscape.