Archiving old data

Archiving old data

Home Forums The Tea Room Archiving old data

Viewing 25 posts - 1 through 25 (of 26 total)
  • Author
    Posts
  • #443277
    Danny M2Z
    Participant
      @dannym2z

      As mentioned in the Win 10 thread, I just built a new (old) Win XP PC, partly for the fun of it but mainly to archive a few hundred CD's and floppy discs to an SSD drive for posterity but also to de-clutter a book rack.

      Some of this data is important to me such as my first ever email (1996) – trivial to most people and also my first pcb design from 1979.

      An old collection of computer virii (about 174,600) with their disassembly notes and fingerprints might be of value to historians one day. The original 'marijuana/paki' virus still lives on a 5,25 floppy from 1989.

      So my problem is: How does one archive such data to last such that future generations might access what we were up to in our time?

      Floppy disc's and CD's are already obsolete, SSD's are maybe headed the same way and as for the 'cloud', this just means storing one's data on somebody else's computer (not really comfortable with that)!

      Too much info to chisel into stone, still have the old DOS 3 boxes to interrogate and then my 1983 Z-80 which still manages to boot up and talk through it's serial port.

      So what's the best way to archive all this data so that it it might be readable to future generations?

      * Danny M *

      #35698
      Danny M2Z
      Participant
        @dannym2z
        #443280
        peak4
        Participant
          @peak4

          I guess two separate hard drives from different manufacturers, stored with care in two different places.
          As the new technology comes along to replace HDDs, transfer the data onto that as well.
          I can remember a friend of mine storing stuff on video disks, and then a few years later couldn't retrieve it, a bit like Betamax etc.

          Bill

          #443281
          Enough!
          Participant
            @enough
            Posted by Danny M2Z on 25/12/2019 15:20:35:

            So my problem is: How does one archive such data to last such that future generations might access what we were up to in our time?

            Floppy disc's and CD's are already obsolete, SSD's are maybe headed the same way

            … and none of them are really "archival" anyway. Least of all CDR/DVR because of the dye-layers.

             

            There are, I believe, new forms of CD that don't use dye layers and are considered archival. Whether they are available commercially to the general public (and indeed, whether they require special writers) I don't know.

            It's a good question though. One, indirect, way to make something readable to future generations might be as simple as uploading it to the internet. I think there are sites for that.

             

            (Edit) Like my Uncle Bill said "Google first, reply second" (well he didn't actually – he way pre-dated the Internet – but it's the sort of thing he would have said). This is interesting.

            Edited By Bandersnatch on 25/12/2019 16:05:26

            #443282
            Hacksaw
            Participant
              @hacksaw

              I can see a hell of a lot of personal photos / images which will one day be historically important / interesting to someone just lost and forgotten…sad

              #443285
              asimpleparson
              Participant
                @asimpleparson

                Perhaps look at what the Government uses to store data. It's usually a format that's going to last a long time. Both Oxford and Cambridge house vast archives of data, it may be worth getting in touch.

                #443289
                Frances IoM
                Participant
                  @francesiom58905

                  I regularly handle 450 year old rag paper documents – still as readable as the day they were written – catch is their information density is somewhat low but would 450 old documents describing early computer viruses specific to long dead O/S be of any interest.

                  #443292
                  HOWARDT
                  Participant
                    @howardt

                    In days long gone gold coloured CDs were used for archiving as they were less susceptible to degradation by light. But whatever you use it should be checked at least yearly to ensure there is no data loss or failure of the media. I started using floppy disks back in 1990 on a daily basis and can say have never lost any data stored for archive. Also have only had two drive failures in PCs in daily use. What ever you use you should keep three backups and not keeping them in the same place.

                    #443296
                    JA
                    Participant
                      @ja

                      Earlier this year I went to a morning meeting held by the Gloucestershire County Archives for local history groups. It was really a sales talk but subject of digital archiving was included. It appears to be a very, very serious problem with formats changing every few years and unstable storage devices.

                      If you want to take this further I would suggest you contact your local county archives for advice. They may even welcome the query as a change from looking at parish records.

                      Today we can read day to day letters written by the Romans on clay tablets. In 2000 years time will anyone be able to read anything on an i-pad?

                      JA

                      #443298
                      John Haine
                      Participant
                        @johnhaine32865

                        Technically, the best solution is cloud storage. Any sensibly managed service would regularly back up data to whatever storage technology is best from time to time. That's the magic of digital data that as long as it is stored with some redundancy it can be exactly reconstituted and re-stored. The problem is having a provider that will be around for as long as there could be people who want to see the data. One solution might be a sort of "Long Now" movement that constructs and runs a civilisational repository.

                        #443300
                        Nick Clarke 3
                        Participant
                          @nickclarke3

                          +1 for cloud storage

                          A separate issue for me as a photographer is the lack of a permanent record of digital images. On the trashier channels of cable TV one sees Hitler, Churchill etc in otherwise insignificant photos. With digital these will be deleted.

                          How will historians of the future manage?

                          #443301
                          HOWARDT
                          Participant
                            @howardt

                            Remember also it is not only the media which may not be be able to be read it is also the file format. Some software like cad has changed file format during its lifetime which if not updated the file format becomes unsupported.

                            #443303
                            Andrew Evans
                            Participant
                              @andrewevans67134

                              It's an interesting and complex topic. I used to work on the multimillion £ Digital Preservation project at the British Library. As part of that we built a system that automatically copied each digital file to 3 remote storage sites, it automatically checked each file on a regular basis to ensure it hadn't changed and if it had changed recovered it from a remote site. Files on hard drives can spontaneously have bits changed and of course there can be bugs and physical problems like fires. We archived digitised books and newspapers as well as websites and other 'born digital' files such as the huge master files for Ordnance Survey maps. When one disk started to fail it was swapped for a new one and data automatically copied over. I am sure that project has evolved since then.

                              Files stored on older CDs can be unreadable as the surface degrades, generally they are poor way to store data long term.

                              90% of the data collected during the Apollo program is now unreadable – the data format was never recorded but the data itself is fine.

                              For files created with a specific program you need to store the program with the file or instructions on how to read the data – no point in having a well preserved file if you have no idea how to read it.

                              The BBC had a project in 1981 called the Doomesday Project to record census data and general information about life in Britain – the data was stored on Laser Disks. I understand there is only a single, working Laser Disk player in existence now that is capable of reading this and this is used as a classic example if digital obsolescence. So it can be a hardware problem as well as a software one.

                              Data loss isn't always accidental, governments can try to change data – look how Trotsky was erased from photos after he fell out of favour in the Soviet Union. So a good preservation strategy has to cope with deliberate data tampering as well.

                              #443316
                              Brian G
                              Participant
                                @briang
                                Posted by Danny M2Z on 25/12/2019 15:20:35:

                                Floppy disc's and CD's are already obsolete, …

                                * Danny M *

                                Perhaps your statement is untrue with regard to CDs as there are still large numbers of CD/DVD/BD disks sold, and that, so far at least, all 120mm optical disk readers are backward compatible, and that pressed (rather than burned) optical disks have a long lifespan. It may be there is so much material on these formats that cash-strapped future historians would be more likely to preserve or re-create hardware to read even fragments of material from these than any other current data format.

                                Whilst conventional writeable optical disks have a very short lifespan, M-DISC compatible DVD writers are available for under £15, and Blu-Ray for around £50. The disks can be read on most DVD/Blu-Ray readers have a claimed readable life of "up to" 1000 years, so it might be reasonable to expect a century or two under less than ideal conditions. Unfortunately the disks themselves are relatively expensive compared to other media (unless you compare the total cost of ownership over the next few hundred years.

                                I fitted one M-DISK and one LightScribe DVD writer to my PC, so I have perhaps achieved a 50% success rate in choosing drives

                                Brian G

                                #443317
                                John Haine
                                Participant
                                  @johnhaine32865
                                  Posted by Andrew Evans on 25/12/2019 23:28:17:

                                  …..

                                  The BBC had a project in 1981 called the Doomesday Project to record census data and general information about life in Britain – the data was stored on Laser Disks. I understand there is only a single, working Laser Disk player in existence now that is capable of reading this and this is used as a classic example if digital obsolescence. So it can be a hardware problem as well as a software one.

                                  …..

                                  A great example of a failure to think the problem through. When laser disks became obsolescent there should have been a plan in place to transfer the data to the next mass storage technology – preferably in the cloud. As always, this isn't a technology problem, it's a process and system failure. I've recently been reading up on Bletchley Park again, and what comes through from a couple of the better books is that, whilst the technology was key, it would not have worked without a carefully constructed and maintained process to industrialise codebreaking.

                                  #443322
                                  JA
                                  Participant
                                    @ja

                                    During the mid 1960s NASA spent an awful lot of money photographing the other side of the moon. In about 1995 someone at NASA actually wanted to have a look at the pictures. They found the tapes easily enough but the viewing equipment had gone. Five machines were finally found as scrap at Edwards Air Force Base.

                                    JA

                                    #443339
                                    Nick Clarke 3
                                    Participant
                                      @nickclarke3
                                      Posted by Andrew Evans on 25/12/2019 23:28:17:

                                      The BBC had a project in 1981 called the Domesday Project to record census data and general information about life in Britain – the data was stored on Laser Disks. I understand there is only a single, working Laser Disk player in existence now that is capable of reading this and this is used as a classic example if digital obsolescence. So it can be a hardware problem as well as a software one.

                                      The Wikipedia article on the BBC Doomesday Project makes interesting reading, not just to show the issues surrounding the preservation and access to this information, but also the copyright nightmare associated with trying to do so.

                                      BBC Domesday Project on Wikipedia

                                      #443341
                                      Ed Duffner
                                      Participant
                                        @edduffner79357

                                        I still use floppy disks on my Amiga computers. yes

                                        For the virii I would get rid of them, just my honest opinion. I'm sure there are copies of them archived at the companies who write software to tackle those. …and I doubt you'd be unable to store them in the cloud (a remote disk somewhere on the internet) because of the risk and possible breach of usage agreements.

                                        At HP our DAT tape backups were only guaranteed for 7 years by our IT Dept..

                                        You could use an optical disk, but I've had a heck of a job getting my Blu-ray DVD writers to read disks on windows 10, but are ok on Windows 7. I have to go into device manager, disable the drive, then re-enable it so the OS can then read from it.

                                        USB Thumb drives could also be an alternative but how long will the interface connector be available into the future?

                                        Ed.

                                        #443345
                                        Perko7
                                        Participant
                                          @perko7

                                          Not a big fan of 'cloud' storage. My previous employer engaged an outside organisation to archive all their records. For some years now they have been progressively converting all their paper filing to digital archives in-house but the task was getting too big. As a consulting engineering firm they had to make sure design information on each project was available for the life of the project in case of any future liability claims. The statue of limitations in most Australian States can be extended where civil suits pertain to engineering designs. Seems like their highly reputable international data storage contractor lost a big chunk of stuff pertaining to some quite notable projects from the last 10-15 years. They are not happy, and neither are their insurers as they now have no evidence to build a defence against any potential claims.

                                          #443400
                                          Enough!
                                          Participant
                                            @enough
                                            Posted by Nick Clarke 3 on 26/12/2019 11:34:55:

                                            The Wikipedia article on the BBC Doomesday Project makes interesting reading ….

                                            …. although correctly called the "Domesday Project" devil

                                            #443402
                                            Nick Clarke 3
                                            Participant
                                              @nickclarke3

                                              A fair cop.

                                              Spelt it wrong three times but only corrected it twice.

                                              But as Meatloaf sang 'Two out of three ain't bad'

                                              #443407
                                              Samsaranda
                                              Participant
                                                @samsaranda

                                                I think vellum with a quill pen, if it’s good enough for Parliament. 😇

                                                Dave W

                                                #443410
                                                Neil Wyatt
                                                Moderator
                                                  @neilwyatt
                                                  Posted by John Haine on 26/12/2019 09:41:40:

                                                  Posted by Andrew Evans on 25/12/2019 23:28:17:

                                                  …..

                                                  The BBC had a project in 1981 called the Doomesday Project to record census data and general information about life in Britain – the data was stored on Laser Disks. I understand there is only a single, working Laser Disk player in existence now that is capable of reading this and this is used as a classic example if digital obsolescence. So it can be a hardware problem as well as a software one.

                                                  …..

                                                  A great example of a failure to think the problem through. When laser disks became obsolescent there should have been a plan in place to transfer the data to the next mass storage technology – preferably in the cloud. As always, this isn't a technology problem, it's a process and system failure. I've recently been reading up on Bletchley Park again, and what comes through from a couple of the better books is that, whilst the technology was key, it would not have worked without a carefully constructed and maintained process to industrialise codebreaking.

                                                  See:

                                                  http://www.domesday86.com/

                                                  This is a project to create interfaces so that the original data (ported to a SCSI drive) can be viewed as was originally intended – on a BBC Master series microcomputer.

                                                  #443424
                                                  Danny M2Z
                                                  Participant
                                                    @dannym2z
                                                    Posted by Ed Duffner on 26/12/2019 11:47:11:

                                                    I still use floppy disks on my Amiga computers. yes

                                                    For the virii I would get rid of them, just my honest opinion. I'm sure there are copies of them archived at the companies who write software to tackle those. …and I doubt you'd be unable to store them in the cloud (a remote disk somewhere on the internet) because of the risk and possible breach of usage agreements.

                                                    The virus collection was archived from when I worked as an IT specialist.

                                                    They were used to test the accuracy and speed of various anti-viral products and dis-assembled to gain an understanding of their logic.

                                                    Part of one report that I wrote emphasised that the delivery mechanism (getting it into a victim's system) was the weak link and isolation coupled with education was recommended, including a total ban of people bringing media onto the sites.

                                                    Interestingly enough, NONE of the commercial a/v products had a 100% success rate and many gave false positives.

                                                    About 120 CD's read and backed up so far, some well over 20 years old with only a few CRC errors on one or two. The new box shall never be allowed online or even networked.

                                                    * Danny M *

                                                    #443459
                                                    SillyOldDuffer
                                                    Moderator
                                                      @sillyoldduffer
                                                      Posted by Danny M2Z on 26/12/2019 22:23:56:

                                                      Posted by Ed Duffner on 26/12/2019 11:47:11:.
                                                      …The virus collection was archived from when I worked as an IT specialist.

                                                      Part of one report that I wrote emphasised that the delivery mechanism (getting it into a victim's system) was the weak link and isolation coupled with education was recommended, including a total ban of people bringing media onto the sites.

                                                      Interestingly enough, NONE of the commercial a/v products had a 100% success rate and many gave false positives.

                                                      * Danny M *

                                                      Plenty of valid reasons for owning a large collection of computer viruses even if you aren't a super-criminal!

                                                      I agree with Danny's total ban, and have reinforced it in practice by configuring computers so they can't read or write media, and are fully encrypted, plus keeping virus and malware protections active, and with all the data in a physically secure computer room, not on local hard-drives etc. This is because people (including me) are unreliable.

                                                      Education is essential, but often ineffective. Unless someone has a particular interest in computer security, most people in my experience fail to pay attention or can't maintain their guard over long periods. There's another group who don't believe any of it applies to them: they see no harm in writing their password on a label under the keyboard, or copying all their customers bank details on a memory stick and then leaving it in a pub.

                                                      The level of protection required is related to the value of the data being protected and the risks to it. For example, the CIA has to manage the threat of well-funded professional attacks mounted by any of several foreign intelligence services, including Allies. Elderly gentlemen using XP only to browse this forum, never spending money online, viewing porn, or sharing private details willy-nilly on social media can be more relaxed about computer security. But no-one is immune!

                                                      Security gets really interesting when balancing usability against security needs. Security experts are only happy after the entire system has been disconnected, shredded and incinerated. Not possible, so real computer systems doing real work, always have loop-holes and vulnerabilities. It's the job of the security team to manage the risks, a particularly hard job when most of us don't understand security, are naive about the risks, get lazy, make stupid mistakes or take bribes.

                                                      Corporates often use as many as three or four different AVM products in parallel as a way of reducing virus risks: one on the internet gateway, another on all the workstations, and a third on the servers. Developers and system administrators loading new software may add a fourth. It's necessary to update all of them regularly; ideas about this vary, but weekly isn't enough! The most at risk systems will have someone monitoring virus threats continually, perhaps reacting 24×7 to dangerous issues by applying patches or by disconnecting services. Also important to keep a careful eye on AVM products because they aren't consistently effective over time. Untested loyalty to Brand-X is unwise because AVM products rely on an engine and a database of signatures. Both can get out of date, or – in the worst case – be unable to adapt to new threats without major rework. They're not like quality physical tools that stay reliable for decades.

                                                      Dave

                                                    Viewing 25 posts - 1 through 25 (of 26 total)
                                                    • Please log in to reply to this topic. Registering is free and easy using the links on the menu at the top of this page.

                                                    Latest Replies

                                                    Home Forums The Tea Room Topics

                                                    Viewing 25 topics - 1 through 25 (of 25 total)
                                                    Viewing 25 topics - 1 through 25 (of 25 total)

                                                    View full reply list.