Data Management: Data Organization and Storage

Setting up a file organization system may seem like a lot of up-front work, but it will save you time and productivity in the end. There are no "rules" of file organization, only guidelines. Work with your collaborators to devise an organization system that works best for you and your data.

Directory Structure Naming

The directory of folders is the top level of data organization. Some tips for creating a folder directory structure:

• The top-level folder should include at minimum the project name and timeframe.
• Folders within the substructure should have a commons theme, such as experimental run, person, or data type.
• e.g. Bridge Analysis/images/Watkins/03.18.2015 

File Naming

A good file naming system will facilitate quick and easy access to your data in the future, and should be intuitive to team members and colleagues. 

• Choose a naming system and ensure that it is followed consistently and systematically. 
• Reserve the 3-letter file extensions for the assigned filetype instead of modifying or changing them. 
• If you expect to end up with hundreds of files, use leading zeros in the earlier numbers (e.g. 001, 002, 003....999). 
• Possible elements for file names include date of creation, name of researcher, data collection method, and version number.
• e.g. image_webcam_watkins_023_04122015.jpg

Version Control

Tracking the version of a file is always important, but particularly so when multiple people are working on the same project or with the same data files. Some general file versioning principles include:

• Save every change as a new version, no matter how minor.
• Use ordinal numbers (1,2,3, etc.) for major version changes and the decimal for minor changes e.g. v1, v1.1, v2.6
• Avoid imprecise labels. Will you remember the differences between revision, final, final2, and definitive_copy a year from now?

Short-Term Data Storage

Short-term data storage typically refers to the period of time covering the beginning of the project to publication. The short-term storage of data should be maintained in ALL of the following locations:

1) Original copy on lab or PI computer

2) External local (e.g. saved to an external hard drive)

3) External remote (e.g. cloud-based storage such as Dropbox, Mega, OneDrive, or Google Drive)

Long-Term Data Storage

Long-term data storage refers to permanent archiving after the completion of a project.

There are many third-party data repositories suitable for long-term storage. Some of these are broad and general while others are subject-specific, so make sure you select the right repository for your data.

Many funding agency data management plans include a requirement that your data will be freely and publicly available in a repository.