Storage

Backup, Backup, Backup!

Systems and Storage (drives, HD’s, SSD’s, Flash, SD Cards, optical media, etc.), can fail without warning. Making regular and up-to-date backups of your data, photos, videos, etc., is your insurance against data loss.

  1. Data should exist in a minimum of three devices/places. The 3-2-1 backup strategy means you have at least three copies of your files. Two of which are local, but on different physical devices/physical drives. And one additional copy offsite that’s not live synced.
  2. One of the places should be off-site.
  3. Your off-site backup should not be live synced.
  4. Test your backup procedures and medium to make sure,
    1. You’re backing up what you expected.
    2. You can successfully restore from the backup.
  5. RAID is not a backup.

Backup Software and Services

  • BackupPC is a high-performance, enterprise-grade system for backing up Linux, Windows and macOS PCs and laptops to a server’s disk. BackupPC is highly configurable and easy to install and maintain.
  • FBackup – Free backup software with simple features.
  • Veeam Backup & Replication Community Edition – Free Backup & Recovery Software for Virtual, Physical and Cloud Workloads

Cloud storage options:

Note: Any files with sensitive information, should be encrypted locally, before they’re uploaded/synchronized to the cloud.

Reference

File Synchronizing, Copy & Backup

  • FreeFileSync is a folder comparison and synchronization software that creates and manages backup copies of all your important files. Instead of copying every file every time, FreeFileSync determines the differences between a source and a target folder and transfers only the minimum amount of data needed.
    • FreeFileSync Tips/Notes:
      • Use Synchronizing: Mirror mode if you’re just backing up, and not changing anything on the destination needing to sync back to the source.
      • If you’re syncing Lightroom Classic (LrC) catalogs, exclude *Previews.lrdata\ and *Sync.lrdata\. The previews and sync will automatically be recreated per your settings in LrC.
  • TeraCopy – Streamline your file management tasks by enhancing file copy and move functions. TeraCopy can verify files after they have been copied to ensure they are identical.
  • Syncthing is a free, open-source peer-to-peer file synchronization application available for Windows, macOS, Linux, Android, Solaris, Darwin, and BSD. It can sync files between devices on a local network, or between remote devices over the Internet. Data security and data safety are built into its design.
  • ShotPut Pro – The industry standard for offloading video, audio and photo files. Make multiple, verified copies from one source.
  • OffShoot allows you to create multiple copies, backups, and archives from multiple sources simultaneously. Built with video workflows in mind, OffShoot performs verified transfers using checksums to/from any type of storage medium.
  • Allway Sync uses algorithms to synchronize your data between desktop PCs, laptops, USB drives, remote FTP/SFTP and WebDAV servers, various online data storage, and more.

Partition Managers/Editors

Drive Diagnostic & Data Recovery

S.M.A.R.T. (Self-Monitoring, Analysis, and Reporting Technology), monitors different drive attributes in an attempt to detect a failing disk.

  •  S.M.A.R.T. Diagnostics
    • Windows Command Prompt: wmic diskdrive get model,status
    • MacOS: Click the Apple icon at the top of the screen and choose About This Mac. Click System Report and select Storage from the list. Make sure the correct drive is selected.
    • CrystalDiskInfo – Reads HDD/SSD/NVMe S.M.A.R.T. Health Status.
    • For a deeper, more accurate picture into your drive’s health, check its manufacturer’s website for a dedicated tool.
  • Recuva can recover pictures, music, documents, videos, emails or any other file type you’ve lost. And it can recover from any rewriteable media you have: memory cards, external hard drives, USB sticks and more.
  • Disk Drill data recovery software. Disk Drill’s interface is designed so it has everything you need right at your fingertips – easy-to-use and clean of any clutter.
  • TestDisk is powerful free data recovery software. It was primarily designed to help recover lost partitions and/or make non-booting disks bootable again when these symptoms are caused by faulty software: certain types of viruses or human error (such as accidentally deleting a Partition Table). Partition table recovery using TestDisk is really easy.
  • DMDE is a powerful tool for data searching, editing, and recovery on disks. It may recover directory structure and files in some complicated cases through the use of special algorithms when other software can’t help.
  • CrystalDiskMark is a simple disk benchmark software.

Hard Drive Reliability

Flash Drive & SD Card Diagnostics & Verification

  • f3 – Fight Flash Fraud – Tests flash cards capacity and performance to see if they live up to claimed specifications.
  • h2testw is a program for checking the speed and memory capacity of flash drives, SD cards and HDD
  • ValiDrive – Quickly spot-check any USB mass storage drive for fraudulent deliberately missing storage.
  • CrystalDiskMark is a simple disk benchmark software.

Direct Attached Storage (USB/Firewire)

Windows Storage Spaces

Storage Spaces helps protect your data from drive failures and extend storage over time as you add drives to your PC. You can use Storage Spaces to group two or more drives together in a storage pool and then use capacity from that pool to create virtual drives. These storage spaces drives typically store two copies of your data so if one of your drives fails, you still have an intact copy of your data. If you run low on capacity, just add more drives to the storage pool.

  • I use Storage Spaces to mirror (RAID 1) two external drives into a storage pool. The storage pool shows up as one drive.
  • If you’re running BitLocker on the storage pool, you’ll need to reboot the computer if the drive shows the locked icon, but when you click on it you get “The drive protected by BitLocker Drive Encryption is already unlocked.”

Resilient File System (ReFS)

When a mirror space is formatted with the Resilient File System (ReFS), Windows will automatically maintain your data integrity, which makes your files even more resilient to drive failure.

  • ReFS in only available in Windows 10/11 Workstation, Server, and Enterprise.
  • ReFS creation is no longer supported on standard Windows 10/11 Pro. This is why ReFS may not be listed when setting up Storage Spaces, or when formatting a drive.

Reference

Disk Sets on macOS

On macOS you can create disk sets using Disk Utility to mirror two external drives into RAID 1.

NAS (Network Attached Storage)

A NAS (Network Attached Storage) system is a data storage device connected to a network that allows storage and retrieval of data from a centralized location for authorized network.

Best Practices

  • Use RAID Level 6 or ZFS as the file system on NAS/SAN devices.
  • Do not use RAID 5, if RAID 6 or ZFS is available!
  • Try and get drives from different batches/lots, by not purchasing all the hard drives from the same vendor at the same time. Why? If a drive in the NAS fails, and is replaced, the NAS has to rebuild the RAID volume. The added stress of rebuilding may cause another drive to fail, thus, there may be insufficient data on the remaining drives to complete the rebuild of the replaced drive. Drives purchased at the same time, are probably from the same batch, and would’ve been in service for the same length of time. When a drive fails, what are the chances another from the same batch may also fail?

CMR (Conventional Magnetic Recording) vs SMR (Shingled Magnetic Recording) Drives for NAS

References

NAS Hardware

SATA Expansion Cards

To add additional SATA ports, use a Serial Attached SCSI (SAS) Host Bus Adapter (HBA). Instead of SATA ports, these cards have SAS ports, to which you connect four SATA drives per port. A SAS HBA with only two SAS ports will allow you to hook up eight SATA hard drives using Mini SAS to SATA Breakout Cables (x2).

Certain cards will work out of the box, such as the LSI SAS 9300-8i HBA (SAS3008) and LSI SAS 9201-8i HBA (SAS2008). Other cards will have to be flashed with an alternative firmware to enable IT-mode. That way, the hard drives connected to the card will appear in Unraid as individual disks and the hardware RAID that an HBA supports, will be completely disabled.

NAS Operating Systems

TrueNAS (formerly FreeNAS)

TrueNAS is an operating system, based on FreeBSD, that can be installed on virtually any hardware platform to share data over a network.

  • TrueNAS combines RAID redundancy, ZFS, Replication, Snapshots, Automatic Corruption Repair, and optional High Availability.
  • ZFS Primer
  • It is highly recommend you use ECC RAM with “mission-critical” ZFS with TrueNAS.
    • Systems with ECC RAM will correct single bit errors on the fly, and will halt the system before they can do any damage to the array if multiple bit errors are detected.
    • The ZFS filesystem (which TrueNAS uses) requires 1GB of ECC RAM for every Terabyte of storage.
Reference

Unraid

Unraid OS allows sophisticated media aficionados, gamers, and other intensive data-users to have ultimate control over their data, media, applications, and desktops, using just about any combination of hardware.

SnapRAID is a backup program for disk arrays. It stores parity information of your data and it recovers from up to six disk failures.

Decentralized Storage

“We put a lot of trust in cloud storage providers, presuming that they’ll keep our data safe. But what if they’re hacked? Or their servers go down? Or what if they decide to just kick us out of our accounts, and we lose all our emails, files, contacts, photos…” NBTV

  • Decentralized Storage by NBTV (Naomi Brockwell TV)
  • Arweave: Permanent information storage. A permanent and decentralized web inside an open ledger.
  • Sia : Decentralized data storage. Rent storage space on the Sia network.
  • Storj is a cloud storage platform that lets you store your data securely on thousands of nodes worldwide with CDN-like performance and privacy.
  • LBRY (pronounced “library”) is a blockchain-based file-sharing and payment network that powers decentralized platforms, primarily social networks and video platforms.
  • Filecoin is an open-source, public cryptocurrency and digital payment system intended to be a blockchain-based cooperative digital storage and data retrieval method.
  • InterPlanetary File System (IPFS) is a protocol, hypermedia and file sharing peer-to-peer network for storing and sharing data in a distributed file system.

Long-term Archival Storage

Standard writable optical media, CDs & DVDs, etc., are not reliable for long term storage. The die used with optical media degrade with exposure to light, temperature extremes, moisture, etc. Because of this, you may not be able read parts or whole files/data from the media within several months to 5 years.

M-DISC (Millennial Disc) by Millenniata, now Verbatim, is the standard for digital archival storage. Unlike traditional optical media, which utilize dyes that can break down over time, data stored on an M DISC is engraved on a patented inorganic write layer – it will not fade or deteriorate. This unique engraving process renders these archival grade discs practically impervious to environmental exposure, including light, temperature and humidity.

  • M-DISC optical media are readable in conventional optical drives.
  • ISO/IEC 16963 standard longevity tests have proven the durability of M DISC technology, and it withstood rigorous testing by the US Department of Defense. Based on ISO/IEC 16963 testing, M DISC media has a projected lifetime of several hundred years.

Drives with M-DISC support and M-DISC Media

Cloud Storage

  • Cyberduck is a libre server and cloud storage browser for Mac and Windows with support for FTP, SFTP, WebDAV, Amazon S3, OpenStack Swift, Backblaze B2, Microsoft Azure & OneDrive, Google Drive and Dropbox.
  • ownCloud is a suite of client–server software for creating file hosting services and using them. ownCloud is functionally similar to the Dropbox, with the primary functional difference being the Server Edition of ownCloud is free and open-source. It also supports extensions that allow it to work like Google Drive, with online document editing, calendar and contact synchronization, and more.

Drive Encryption

Storage Media Creation & Boot Menus

RG Adguard – Download Windows ISO’s – RG Adguard generates links which point to Microsoft servers

Microsoft Windows and Office ISO Download Tool

  • Use the Microsoft Windows and Office ISO Download Tool to download ISO’s directly from Microsoft’s servers.
  • The Windows 10 ISO gotten from with this tool is better than the one created via Microsoft’s Media Creation Tool, as it includes “install.wim” which can be used to deploy Windows 10.

Rufus

  • Create bootable USB drives from ISO images of CD/DVD discs.
  • Can be used to create a live bootable “Windows to Go” USB Flash Drive on Windows 10 Professional.
  • FreeDOS is built in. Nothing else needed to create a bootable FreeDOS USB Flash Drive.
  • UEFI: When using GPT UEFI:NTFS format/bootloader, to boot from the media, you must disable disable Secure Boot.

YUMI (Your Universal Multiboot Integrator)

  • It can be used to create a Multiboot USB Flash Drive containing multiple operating systems, antivirus utilities, disc cloning, diagnostic tools, and more.
  • Contrary to MultiBootISOs which used grub to boot ISO files directly from USB, YUMI uses syslinux to boot extracted distributions stored on the USB device.

XBoot

  • An easy-to-use utility for creating Multiboot USB/ISO.
  • You may have seen many bootable ISO files like Linux live CDs , Antivirus rescue CDs etc. XBoot can combine these ISO files in to one Multiboot ISO file or create Multiboot USB in just few clicks.

ImageUSB

  • Lets you create an image of a bootable USB flash drive, then create a bootable USB flash drive from that image.
  • Warning: Due to the forensic nature of image duplication by ImageUSB, please ensure that you select UFDs with a storage size similar to the image you wish to duplicate. For example, if a 2GB image is copied to an 8GB USB Flash Drive, the drive will only be able to use two out of the eight gigabytes of storage space.

ImgBurn

  • A lightweight CD / DVD / HD DVD / Blu-ray burning application.
  • Supports a wide range of image file formats – including BIN, DI, DVD, GI, IMG, ISO, MDS, NRG and PDI.

AOMEI Partition Assistant

  • Standard is FREE disk partition management software.

Disposal, Destruction, Sanitization of Storage Media

Review Your Cart
0
Add Coupon Code
Subtotal
Checkout