.. _disk: | :Last modified: |today| | Data management =============== The following topics relate to disk access. .. _backup: data backups ------------ backups are available on Neuro cluster nodes nx10 through nx28. These nodes aren't accessible from the internet, so you must login to a x2go login server first. For example, to log into nx10, open a terminal window in your x2go session, and type:: ssh nx10 Your lab's backups are in the directory /home/lab-backup (replace 'lab' with your lab's name). The following table displays the name and age of backups (replace 'lab' with your lab's name): ======================== ====================== Directory Backup established ======================== ====================== /home/lab-backup/lab.1 less than 24 hours ago /home/lab-backup/lab.2 1-2 days ago /home/lab-backup/lab.3 2-3 days ago /home/lab-backup/lab.4 3-4 days ago /home/lab-backup/lab.5 4-5 days ago /home/lab-backup/lab.6 5-6 days ago /home/lab-backup/lab.7 6-7 days ago ======================== ====================== When you're finished with the backup, please log out so data will be refreshed. user backups ^^^^^^^^^^^^ I recommend that you backup important files to off-site storage. Berkeley Box is free to staff, faculty and students. It provides unlimited off-site storage, with an option to sync files. For more information, see: * documentation from UC Berkeley: https://bconnected.berkeley.edu/collaboration-services/box * login page: https://berkeley.app.box.com/login .. _howto-quota: disk quotas ----------- Run the 'quota' command from the :ref:`command line` to view your disk quota and usage:: quota Run 'dsum' to summarize your disk usage:: dsum For more options, run:: dsum --help It may take a while to complete, depending on your disk usage. Review the output, and #. delete temporary data #. delete redundant data #. compress data that you no longer use If you exceed disk quota, you will receive email with instructions every day during the 10-day grace period. If the grace period expires, you may log in via :ref:`secure shell` if you have a SSH key installed. Remote desktop (X2Go) logins will fail. If you want a quota increase, email support-neuro@berkeley.edu before the grace period expires. Please don't wait until the last day of your grace period to request a quota increase. .. _howto-compress: data compression ---------------- I recommend compressing: #. files that you can analyze in a compressed format (e.g. afni BRIK) #. files you want to archive (files you don't expect to use on the cluster) | Do not compress/uncompress files frequently to stay under quota. This causes lag, and reduces the efficiency of the backups. | | Below are examples using 'gzip' from the :ref:`command line`: * **Compress a file** This example will compress a file named 'unused.txt', and rename it 'unused.txt.gz'. Modify 'unused.txt' to specify the name of the file you want to compress:: gzip unused.txt * **Uncompress a file** This example will uncompress a file named 'unused.txt.gz', and rename it 'unused.txt'. Modify 'unused.txt.gz' to specify the name of the file you want to uncompress:: gunzip unused.txt.gz * **Compress files in a directory** This example will compress all files in /location/of/files (and subdirectories). Modify '/location/of/files/' to specify the directory containing the files you want to compress:: gzip -r /location/of/files * **Useful in shared directories - compress files you own in a directory** This example will compress files you own in /location/of/files (and its subdirectories). Modify '/location/of/files/' to specify the directory containing the files you want to compress. Don't modify the second line that begins with 'find' - it should be run as written:: cd /location/of/files find . -user $USER -type f ! -name "*gz" -exec gzip {} \; * **Compress files you own based on the filename** This example will compress files you own in /location/of/files that end with the string '.dcm'. Modify '/location/of/files/' to specify the directory containing the files you want to compress. Modify '.dcm' to specify the string in the filename that you want to match:: cd /location/of/files find . -user $USER -name "*.dcm" -exec gzip {} \; * **Compress big files you own** This example will compress big files you own. It runs at low priority, so it won't cause network lag. Modify '/location/of/files/' to specify the directory containing the files you want to compress. Don't modify the second line that begins with 'find' - it should be run as written:: cd /location/of/files ionice -c2 -n7 find . -user $USER -size +500k ! \( -iname "*.zip" -o -iname "*.gz" -o -iname "*.tgz" -o -iname "*.bz2" \) -type f -exec gzip {} \; * **Uncompress files based on the filename (recursively)** This example will uncompress files you own in /location/of/dicoms that end with the string '.dcm.gz'. Modify '/location/of/dicoms/' to specify the directory containing the files you want to uncompress. Modify '.dcm.gz' to specify the string in the filename that you want to match:: cd /location/of/dicoms find . -user $USER -name "*.dcm.gz" -exec gunzip {} \;