Automated File Management System

Technical documentation
02/27/2009

1. Overview

File Management continues to be a major burden for users. In Version 1, I sought to remove it as an issue from MTTs by placing it all in the adviser's hands; but I've subsequently observed that it is a problem for advisers as well. A translation has a great many files, the management of which is only compounded when multiple computers are used in a given project.

In V1 I gave the adviser complete control over the location of files; on importing a Scripture Book he specifies the folder for that book. Some advisers place each book in a different folder; others place all of the books for a translation in a single folder. This flexibility originated because we wanted to support the existing Timor system, which was based on the need to accommodate working with Toolbox in their complex language situation. That is, they developed a system to reduce the complexity of file management for the many language projects they deal with, taking into account the limitations of the different software packages they were using. In developing OurWord, I wanted to remain compatible with the work they had done; and therefore supported a user-defined folder system.

The move to Distributed Version Source Control (DVCS) gives us both opportunity and motive to address these complexities of file management. A DVCS does not work well when file names are changing regularly (as they do in V1); nor when files are stored in different folder hierarchy strategies on different computers. So a prerequisite of moving to DVCS is for OurWord to standardize in its filenames and folder structures.

2. Note for Testers

The purposes of this document are
  • to give me a road map of what I need to implement
  • to give you an idea of the sorts of things that have changed, which I hope will guide you in knowing what to test.
  • to serve as a record for the reasons behind some of the decisions I've made.

Once I have checked the Feature off as Closed, I do not intend to further maintain this document. Furthermore, I do not expect to worry about keeping it in synch with changes I make to OurWord. Thus if the wording in a dialog differs from what is written here, it is the implementation that you should go by. Please, please, please don't enter bug reports of the nature "the wording in the implementation is not the same as the specification." I promise I'll be deleting those. Keeping a specification up-to-date with the implementation is a tremendous amount of work, and I can't afford to go there. Thanks for your help in testing, and your understanding in this matter.

3. Implementation

Model Changes

  • (Done) I have dropped the Abbreviation from the translation properties. This was typically the 3-letter code from the Ethnologue. I realized that I don't really use it anywhere, especially now that filenames now user the language's full name (see below.) So while it may be important in the overall management of the work, I do not currently see the need to complicate OurWord with it.
  • (Done) I removed the Relative/Absolute pathnames mechanism in favor of automatically computed ones. (This broke 24 unit tests.)

File Names

In V1 the file name was a combination of Language 3-Letter Code, Book 3-Letter Code, Localized Stage, Version Letter, and the Date of the save. Each day, a new filename was created, which on the positive side provided nice backup of work; but on the negative side created a lot of files, and was therefore incompatible with source control systems, where we need to maintain the same name. One nice thing about the V1 system is that from Windows Explorer you can tell at a glance the stage and status of a book; this is something we unfortunately lose with source control. But we can keep it with backup file names.

  • (Done) Book filenames will be (number) + (Book Code) + (Language Name) + ".db"
    • where Number is two digits, starting at 01 for Genesis, ending at 66 for Revelation. Beginning with this number causes the books to appear in cannonical Biblical order under Windows Explorer.
    • where the Book Code is the international three-letter code, which has become a worldwide standard.
    • where the Language Name is the full name of the language, entered by the adviser in the configuration.
    • spaces and hyphens added for readability
    • For example, 01 GEN - Huichol.db
  • (Done) Filenames for backup begin with the filename described above, but add the following information:
    • The abbreviation of the Stage (e.g., Draft, Rev, Team.) These are localized flavors, as set up in the Configuration.
    • Version as defined by the advisor, this can be a letter, number, or any short string.
    • The date of the backup, in year-month-day format, so that they sort properly in Windows Explorer.
    • For example, 01 GEN - Huichol - Team-B 2009-02-25.db

Folder Hierarchy

In V1 the adviser could set up any type of folder hierarchy desired. Some created a root folder at C:, others used MyDocuments. Some placed all of the books in a single folder, others used a folder for each book. I am removing all of these options from the user, benevolently dictating where all files are stored. Thus file locations are now completely automated, the user has no control over them.

  • (Done) All folders/files are stored under each user's MyDocuments. I considered the All Users MyDocuments, thinking this would make the files accessible to everyone; but on reflection decided that it is better if each person has his own user (and thus separate Mercurial repositories), so that the system can keep track of changes on a per-user bases.
  • Each Cluster is defined as a folder under MyDocuments. The default cluster when OurWord is installed is "Our Word." Thus new language projects are created under "Our Word", so in the case where the user just has a single language and not a cluster, the folders will have the reasonable default value of, e.g., "...My Documents\Our Word\Huichol." Users can rename their cluster (and therefore the folder) in the configuration dialog to something more appropriate like "Timor" or "Aboriginal"; there they can also add or remove clusters. Each cluster will thus have a root folder, under MyDocuments, of its name.
    Items remaining:
    • (High) Renaming a cluster in OurWord
    • (High) What do we do if the user renames the cluster folder in Windows Explorer?
    • (High) Handle if Exception fires in TeamSettings.OnLoad override (creating folders)?
    • (High) Adding / removing clusters
  • Each Cluster has a single .Backup folder under it. All projects make backups to that folder, rather than having separate backup folders for each project. (Following Mercurial's lead, all system folders like this start with a period, so that they sort together under Windows Explorer.) The backup system uses a weed-out algorithm, such that backups are stored daily for two weeks, then weekly, then monthly; so as to not place too many files on the storage medium.
    Items remaining:
    • (High) Weed-Out algorithm
  • (Done) Each Cluster has a single .Settings folder under it. All settings for the cluster (owt, owp, otrans) are stored there, rather than scattered as in V1.
  • Each Cluster has a single .Hg folder under it for those users which use Mercurial. Thus a single repository will be used for the entire cluster. (Not implemented here, because this is part of the upcoming Chorus/Mercurial feature.)
  • (Done) Each Project/Translation has its own folder, stored under the Cluster folder. The name of this folder is the translation (language's) name. All of the books of the translation are stored in this folder, not in subfolders.
    • The front/source is stored as a folder under the cluster
    • all of the targets are stored as folders under the cluster
    • all of the reference translations are stored as folders under the cluster
  • (Medium) Folder names can be localized, so that, e.g., ".Backup" becomes meaningful to the local context.

Dialogs

The following items given an idea of how much of the user interface was devoted to file management. Automating file management is proving to be a substantial simplification to the user interface.

  • (Done) The Import Book Wizard no longer needs to know about the destination filename. A number of simplifications made it possible to reduce this wizard to two pages: (1) browse for import file (which includes the logic that tests for, and tells the user about, any problems with the import file, and (2) a summary page.
    • I removed the "Tell Us The Abbreviation" page because (1) we can find it in the \id line of the book, and (2) as a fallback we know it from what was selected when the Import button was pressed.
    • The BrowseForFile now starts at MyDocuments, the root from which files should be findable.
    • I removed Stage and Version from the import process, figuring that this is more of an option than a requirement, and the user can just do this from the Properties button if he wants to fool with it.
  • (Done) The New Project Wizard changed as follows:
    • The former Team Settings page has become a Choose Cluster page. The page only shows up if there is more than one cluster, which is determined by scanning MyDocuments for folders that have an "owt" file in their ".Settings" subfolder. We present the user with a choice, selecting the top one by default (or the one chosen from the last time if available.) This page is present early in the sequence, as we need it for the other pages.
    • The Target Page is next. We only ask for the name of the project (no longer needing the abbreviation, see the Model Changes section above.) We display an error message (and disable the Next button) if the project already exists in the cluster.
    • The Front/source Information Page no longer needs a settings folder; it now just has a combobox to choose from existing translations. The user can enter something into the combo box to create a new translation, or can just choose from the combo's items. I populate the combo box by scanning the otrans files in the cluster's .settings folder. As a convenience to those setting up a large number of translations, I default to the front choice made the last time.
    • The Summary Page changed to match the above, which largely meant dropping information that was displayed in V1. I did add text to show if they were creating a new Front, vs using an existing Front. The cluster in which the project will be placed is not shown if we did not display the Choose Cluster page.
  • (Done) The Book Properties Dialog was reorganized into two tabs, one for the Comment, the other for the remaining properties, in the Literate Settings style. All of the former filename controls were removed. We are left with Locked for Editing, Copyright, Stage, and Version. The Version can now be any text.
  • (High) The Increment Book Status Dialog needs its future decided upon. It has not been used (that I know of) by the user, so I could delete it. On the other hand, it is probably along the lines of what I need for when a commit is being done into Mercurial. At any rate, all the filename stuff in it is now obsolete. Design needed.
  • The Configuration Dialog needed considerable rework, complicated by the fact that it is also undergoing the Literate Settings re-write.
    • The Translation Properties Page was reorganized into two tabs: (1) Books is a consolidation with the former Book Names tab, and is now first in the sequence because it is the most frequently used. (2) Other was formerly General, I renamed it to reflect that it is lesser importance/frequency. Implementation included:
      • Using a Grid in place of PropertyGrid for the Books tab; key for the consolidation
      • Adding the Show filter
      • Reviewed and updated logic for Create, Import, Export, Remove.
      • Properties dialog rework (see related section)
      • (High) Import can guess the book from the selection in the grid.
      • (High) Copy Book Names From working again
      • (High) Harvesting Book Name edits
      • (High) What to do when the user changes the name of the translation.
    • (High) The Cluster Page is where the user defines which cluster this project belongs to. If he chooses a different cluster, then the various folders of the project are moved to it. (Need to think this through.) He can add/remove/rename clusters on this page.

Upgrading V1 data

  • (High) Users must be able to easily (automatically?) upgrade their Version 1 data.
    • On loading a new TeamSettings, we create a Cluster folder under MyDocuments
    • On loading a Project, we move everything to the new folder and filenames
    • On loading a Project, we rename the old folder so that it is clear it is no longer active.
    • At any time, if we can't find a file that we think should be there, we let the user navigate to it, then copy it into the proper folder.

Miscellaneous

  • (Done) I no longer keep backups of the owt, otrans, owp files around. I create a temporary backup before starting the save, which I use to restore if the save files; but I delete that backup once the save is finished. I went this way because (1) these files were cluttering up the folder, (2) there were really not being used outside of this temporary process, and (3) most importantly, Mercurial will become that backup.
  • The message when there is no project is now larger and more prominent.
    • (Medium) I want to go to separate messages: "No Project Defined" for nothing; "No Books Defined" for incomplete project, with appropriate directions on what to do.

4. Bugs introduced and fixed during implementation

Files