Request for Comment: Archive Integrity

[linas]

I'm wondering if it's possible to modify and use FIM for something "completely different". I'm interested in monitoring the integrity of archives and backups, and searching backups for a valid version of a file, if the "primary" copy as been corrupted or lost.

I've got a large collection of files -- photos, MP3's, source code, websites. While moving some of these around, I found a thousand files, out of a million, were filled with all-zeros. Data corruption, in short. These are archive files; I wasn't monitoring them, the corruption could have happened decades ago. I probably have backups of these files. Maybe. Not sure where.

I'm thinking of coding up a content management system, something to check current data integrity, compute hashes, compare files, keep a record of filenames, timestamps, hashes for not just one computer, but half-a-dozen, including stuff on off-line, cold-storage disks. I figure running data integrity checks a couple times a year is enough. This is not for intrusion detection, but archive integrity.

Instead of starting from scratch, I'm hunting for something that already is half-way there. And so I ended up here. I'd like to have something modular, pluggable, w/ logging, search, a long list of cool features I ca day-dream about. Any comments on whether FIM could be or should be adapted to this? Maybe there are pieces-parts in FIM that I could recycle? Maybe you know of something else that does this already?

[okynos]

Hello @linas,

Thank you a lot for your patience and taking the time to write here. The holiday season made me forgot all my pending stuff.
Well, I will try to answer you section by section because it is a complex matter.
Short answer to your questions:

• Any comments on whether FIM could be or should be adapted to this? Yes FIM could perform something pretty similar to the use case you mentioned.
• Maybe there are pieces-parts in FIM that I could recycle? Currenly, it would be required the monitoring module (notify crate) + indexing module (Already present in FIM) + Event handler (Would need some modifications) + Hash database (Not present on FIM 0.5.1)
• Maybe you know of something else that does this already? Sadly, I am not so updated in the latest tools available but maybe there is something, try to search by File hash database or File tracker.

Long and ramble answer:
FIM could perform your use case with no so much effort but it will requires some modifications that are out of the scope of the tool but, maybe I could guide you through those modifications.
Firstly, you will requires the hash database, that it is currently under development but it will be out pretty soon.
FIM will create a database with the files of a given path, this database will store filename, hash, size and timestamp (when the file was scanned) in your case you do not need FIM running just run each 6 months or so, to refresh this database and trigger events of the files that changed. Those events should be ingested in an indexer, with that you could manage all the hosts or places where you run FIM and manage isolated database in case of corruption you will know which host is failing and which file is modified (Not corrupted). To detect file corruption it will require additional coding to detect the previous file data and the current one that probably differs in 75% or more (We will need to dig more into file corruption stages) and handle false positive cases (File replacement).
Probably I left something on the way but I guess this is a good starting point, please let me know what do you think.
Have a nice day!

[linas]

Thanks! Since then, I've made some progress. Not with FIM, but let me describe anyway. You might find it interesting.

• Writing a crawler, per se, to walk over a filesystem and compute a hash is not hard. To make it pluggable (collect attributes other than hash) is harder, but I have not tried to do that.
• Some filenames are not UTF-8 (they appear to be microsoft code pages of some kind) Turns out python, guile/scheme and sqlite3 just hates this. Don't know about rust, but the insistence on utf-8 strings in these systems creates truly ugly issues.
• I split data collection into two parts: file location (URL) and "witnesses" that attest "on this and such a date I saw that URL with this hash".
• I have multiple copies of files; in many cases in offline storage (backups) This means the file names are identical; even the last few directories in the path, but the root of the URL differs (hostname, first part of the path)
• Besides just the name fragment, I can see if files are identical based on the hash, the ctime timestamp.

This last bullet is the one driving me crazy: the data analysis, data visualization, data exploration part. Say I've got two directories; they've got 300 files, and 299 of them have the same hash, and one doesn't. Where are these directories? How do I find them? How do I present results? What's the user interface? I hacked something up to do all this, but am not happy with it. There's a bottomless-pit of what-if's in this dataset. I'm dealing with many millions of files, so trite solutions don't really help.

FWIW I found that some of my files have all-zero content, while others have runs of 4K or more zeros in the middle. How did this happen? My files are on RAID-1 mirrors, on Linux, ext4fs, consumer hardware (PC's) I'm guessing the corruption happened 10 or 20 years ago, when maybe I wasn't running ext4 but ext2, maybe I wasn't running RAID. with PATA ribbon cables not SATA. The files got copied from one disk to another; maybe the old disk was failing? Bad sectors? No clue. Its all distant past. But I've also got ancient backups. So... how do I find and repair, decades after the fact? That's the hard part. Collecting the data is relatively easy.

[okynos]

Good to know that you could improve your code and knowledge. Some comments about this:

• I remember some problem related to encoding on Windows related to the way Windows manage some file contents. Something related with (BOM) that my mind doesn't want to remember but in any case you can store all the strings in standard format and the encoding required then, at the moment to process this string convert it to the original encoding (Just a suggestion maybe is a bad approach depend on the contex).
• If you wouldn't perform any search by file name, maybe you can split the URL in something like container and file or filepath this is easier to filter on data processing and later search. On FIM I used an unique ID that is stored in the database at the same time the path and the hash but this could be a problem in environment where the files can be the same name because FIM perform searches by name in the case the file was deleted or touched I have the path and the action but not the database ID.
• To the data analysis I used ElasticSearch and Splunk as data indexer, these tools allows you to ingest large amount of files and present it in a better way or even filter them by some conditions. Remember to get a good labeled dataset to be able to parse/filter the data in an easy way.
• To the Zeroed files I'm not sure about the solution but maybe you can extract the list of corrupted files and compare them by name with the rest of your dataset maybe some of them are recoverable from one backup or isolated directory. That's the hard way (and time consuming) but it is the way that comes to my mind now. Having that dataset you can compare files byte by byte and replace the zeroed areas to get something consistent. In any case, is a marvelous way to spend the time.

Thanks for sharing your knowledge with me and keep up the good work, have a nice day!

[linas]

Bullet points:

• The filename issue is that I have no clue what the encoding is. At some point in the past, someone created a file on Windows. Now its in Linux. No one jotted down "here's the encoding that was used". That info was never kept. The Linux file systems (and C/C++) don't care: its just a null-terminated string. Python, guile and most SQL's do care. Of course python has byte-arrays but printing these gives you garbage, so you have to guess what the encoding might be. Its an unpleasant situation.
• I thought long and hard about using "unique IDs", and decided the best-possible unique ID is the URL itself. The problem with "unique id's" is that they have to be deterministic. Both you, and the guy in Australia, have to somehow magically come up with the ID. How? That rules out a central issuing authority. All other variations suffer from the "birthday paradox" (see wikipedia) The U in URL stands for "unique" so I decided to use that: it really is unique.
• I did say to myself "something like splunk". Didn't occur to me that it could literally be splunk. I will have to look into that.

Oh, the filename-encoding issue is even worse if the archive contains URL's from asian countries a decade or two ago, before standardization. Some of the japanese and korean websites did figure out how to use either iso8859-nn encoding schemes or something else in the URL, Me, I don't care. I just store the URL (as a linux filename). Python cares: it throws exceptions. Big guys (internet archive/wayback machine) have money and staff to throw at this. I don't.