FAT directory entries not cleared.

[Joel]

In DirectExecutor.EraseUnusedSpace, directory entries are not cleared. Look at Eraser.cpp:2348, r400.

Garrett: What do the equivalent functions in v5 do that are important? I see a lot of undocumented API use which is obviously dangerous because Windows undergo internal component rearrangement almost every service pack. I thought we wanted to remove the dependency on undocumented API calls?

[Joel]

Quite important, this one. It has the potential to be a regression from v5.

[Overwriter]

I am writing with limited knowledge here but is this related to ticket 23 ?

[Joel]

Hmm, NTFS is safe, FAT isn't because we're reimplementing the data structures ourselves.

[Joel]

I've looked at the FAT code again and I've come to the conclusion that techniques used in f5 to clean the FAT will not work in Vista. We'll need to find new ways of erasing the FAT in Vista. Garrett/Kaz? can you take charge of this?

I'll just need a document detailing what needs to be erased for FAT drives, the differences between FAT12/FAT16/FAT32 and how to do this without opening a Write handle to the raw disk sectors.

[Joel]

Slightly more accurate title.

[Joel]

I was reading up about FAT and I've developed a method to erase the stuff, which requires verification by someone else, before I implement it.

FAT entries that require erasing are basically the directory tables, as documented in ​Wikipedia. After a directory is deleted, the corresponding clusters are marked as available, and the referencing Directory Table Entry has byte 0x00 reset to read 0xE5, which means that the entry has been removed.

So this leaves us with two things that need to be erased. The actual directory table, and the referencing directory table which refers to the new deleted directory. The erasure of the Directory Table structure itself is a non-issue to me because the entire structure's is now marked as available; an unused space erase will clear it.

The referring directory table structure, however, must be erased. The structure that must be erased is the parent directory of the directory that was deleted. As mentioned earlier, the old entry is merely marked as deleted through the setting of the byte at offset 0x00 to read 0xE5. The problem is further exacerbated by the presence of Long File Names, as outlined in ​Wikipedia as well. The long names are stored as "invalid" DOS file entries in the directory table. So deleting a folder or a file will leave multiple directory table entries marked as "deleted", with the old information still intact.

My proposed erasure method is to do what we do with NTFS now:

1. Determine Directory Table Size.
2. Determine the number of valid entries in the directory table. This will let us know how many directory entries must be erased.
3. Erase the directory entries. This is a multi-step process. Repeat this for every entry that must be created.
   1. Create files inside the directory that needs entries to be erased with 8.3 names which fills up the first 11 bytes of the directory table entry structure.
   2. Set the Create, Access and Modified (CAM) date values to Jan 1, 1980
   3. Entry offset 0x14 and 0x1a should be reset to zero automatically, regardless of FAT variant
4. Delete all the created files

That should erase the directory entry such that file names are no longer recoverable. Please comment if you have relevant experience and tell me if this works.

[Overwriter]

Good grief !!

Well done Joel, stunning work !

I hope this fixes it, I will try to get help to verify it for you.

I think it will have to be Svante as this looks a bit complicated !

:o)

[Joel]

Note to self: while I'm at it refactor all filesystem-specific code into helper classes/interfaces.

[Joel]

Okay the note is no longer valid - I did that already.

I've now got a sample working: but it's hopelessly slow because we're practically forcing every directory file to fullness then clearing after ourselves. This leads to two bad effects:

1. Every directory entry grows by one cluster at every wipe. This is statistically significant as a drive can have hundreds of directories and consuming 100's of clusters per wipe is unsustainable
2. Because we're filling the directory file with filenames, this well, takes time. A lot of time.

Do we have alternative approaches?

P/s I've attached a patch created against r1032.

[Joel]

Part 1 of 2. This is the patch to apply to the source tree.

[Joel]

Part 2 of 2. The DirectoryStream? class which allows client code to determine the number of clusters a directory occupies

[Joel]

I've had a chat with Svante Seleborg (of AxCrypt?) and I've attached our chat log.

[Joel]

Chat Log with Svante Seleborg on FAT erasures

[Joel]

Okay I've spent the whole of today implementing FAT erasure (see r1206 r1224) and FAT32 erasure is now working, I'll implement FAT12 and FAT16 tomorrow.

[Joel]

Completed as of r1235. Finally!