You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* Update htc quota guide
* New hpc quota page; remove broken link
* Apply suggestions from code review
Change "more" to "higher"
Co-authored-by: Christina K. <[email protected]>
---------
Co-authored-by: Christina K. <[email protected]>
The following commands will allow you to monitor the amount of disk
12
-
space you are using in your home directory on the access point and to determine the
13
-
amount of disk space you have been allotted (your quota).
11
+
## Introduction
12
+
13
+
This guide shows you how to check the amount of disk space and number of files you are using on the Access Point in your `/home` and `/staging` directories. You can also check your limit on your disk space and number of files (i.e., your "quota"). This guide will help you manage your disk space and give tips on what do when you've reached your quota.
14
+
15
+
{% capture content %}
16
+
-[Introduction](#introduction)
17
+
-[Default quotas](#default-quotas)
18
+
-[Check your quota](#check-your-quota)
19
+
-[Other ways to check your quota](#other-ways-to-check-your-quota)
20
+
-[Check the size of a directory and its contents](#check-the-size-of-a-directory-and-its-contents)
21
+
-[What to do when you've reached your quota](#what-to-do-when-youve-reached-your-quota)
22
+
-[Related pages](#related-pages)
23
+
{% endcapture %}
24
+
{% include /components/directory.html title="Table of Contents" %}
For the most up-to-date information about your quota, use the `get_quotas` command on the Access Point.
47
+
```
48
+
get_quotas
49
+
```
50
+
{:.term}
14
51
15
-
The default quota allotment in your `/home` directory is 20 GB with a hard
16
-
limit of 30 GB (at which point you cannot write more files).
52
+
This will print a table with your `/home` and `/staging` quotas. An example output is shown below.
17
53
18
-
**Note: The CHTC access points are not backed up, so you should
19
-
copy completed jobs to a secure location as soon as a batch completes,
20
-
and then delete them on the submit node in order to make room for future
21
-
jobs.** Disk space provided is intended for *active* calculations only, not permanent storage.
22
-
If you need more disk space to run a single batch or concurrent
23
-
batches of jobs, please contact us ([Get Help!](get-help.html)). We have multiple ways of dealing with large disk space requirements to make things easier for you.
Your `/staging` directory has a disk and item quota. In the example above, the disk quota is
64
-
20 GB, and the items quota is 5 items. The current usage is printed in the following columns;
65
-
in the example, the user has used 3.19 GB and 5 items.
129
+
However, we recommend [using the `get_quotas` command](#check-your-quota) for the most up-to-date information.
66
130
67
-
To request a quota increase, [fill out our quota request form](quota-request).
131
+
## Check the size of a directory and its contents
68
132
69
-
**3. Checking the Size of Directories and Contents**
70
-
------------------------------------------------
133
+
If you want to check the size of specific directories and their contents, you can do this with the `du` command.
71
134
72
-
Move to the directory you'd like to check and type `du`. After several
135
+
Move to the directory you'd like to check and type `du`. After several
73
136
moments (longer if the contents of your directory are large), the command
74
137
will add up the sizes of directory contents and output the total size of
75
-
each contained directory in units of kilobytes with the total size of
138
+
each contained directory in units of kilobytes (KB) with the total size of
76
139
that directory listed last. See the example below:
77
140
78
141
```
79
-
[alice@submit]$ du ./
142
+
[user@ap2002 ~]$ du ./
80
143
4096 ./dir/subdir/file.txt
81
144
4096 ./dir/subdir
82
145
7140 ./dir
83
146
74688 .
84
147
```
85
148
{:.term}
86
149
87
-
As for quota usage above, you can divide each value by 1024 to get
88
-
megabytes, and again for gigabytes.
150
+
You can divide each value by 1024 to get megabytes, and again for gigabytes. Below are extra options for the `du` command.
89
151
90
-
Using `du` with the `-h` or `--human-readable` flags will display the
91
-
same values with only two significant digits and a K, M, or G to denote
92
-
the byte units. The `-s` or `--summarize` flags will total up the size
93
-
of the current directory without listing the size of directory contents
94
-
. You can also specify which directory you\'d like to query, without
95
-
moving to it, by adding the relative filepath after the flags. See the
96
-
below example from the `home` directory which contains the directory
97
-
`dir`:
152
+
| Command | Usage |
153
+
| --- | --- |
154
+
| `du -h` or `du --human-readable` | Prints disk usage in a human-readable format.<br>(K, M, G denote kilo-, mega-, and gigabytes, respectively.)
155
+
|`du -s` or `du --summarize`| Prints total disk usage of the directory without printing its contents. |
156
+
|`du <path/to/directory>`| Prints disk usage of the specified directory. |
98
157
99
-
```
100
-
[alice@submit]$ du -sh dir
101
-
7.1K dir
102
-
```
103
-
{:.term}
158
+
## What to do when you've reached your quota
159
+
160
+
When you've reached your quota, you may encounter error messages such as `Disk quota exceeded`. When you encounter the error message, we recommend the following steps:
161
+
162
+
1. Check your quota with `get_quotas`. At which data location did you reach your quota? Did you reach your disk quota or your items quota?
163
+
1. If possible, remove any files you no longer need from the system.
164
+
1. If you've reached your quota for the **number of files** in `/staging`, we recommend compressing your dataset into zip files or tarballs, because `/staging` is intended for storing few, large files. [Read more about this here.](file-avail-largedata#reduce-file-counts)
165
+
1. If you still need more disk space, [request a higher quota.](quota-request)
166
+
167
+
> ### ⚠️ CHTC is not a storage service
168
+
{:.tip-header}
169
+
170
+
> CHTC data locations are intended for temporarily storing files used in **active calculations only**. Once you are done with the files, please remove them from the system to clear disk space.<br><br>
171
+
> **We do not back up any of the data you place on our system. It is your responsibility to back up your own files.**
172
+
{:.tip}
173
+
174
+
## Related pages
175
+
176
+
*[Use and transfer data in jobs on the HTC system](htc-job-file-transfer)
177
+
*[Manage large data in `/staging`](file-avail-largedata)
This guide shows you how to check the amount of disk space and number of files you are using in your `/home` and `/scratch` directories. You can also check your limit on your disk space and number of files (i.e., your "quota"). This guide will help you manage your disk space and give tips on what do when you've reached your quota.
14
+
15
+
{% capture content %}
16
+
-[Introduction](#introduction)
17
+
-[Default quotas](#default-quotas)
18
+
-[Check your quota](#check-your-quota)
19
+
-[Check your Message of the Day (MOTD)](#check-your-message-of-the-day-motd)
20
+
-[Check the size of a directory and its contents](#check-the-size-of-a-directory-and-its-contents)
21
+
-[What to do when you've reached your quota](#what-to-do-when-youve-reached-your-quota)
22
+
-[Related pages](#related-pages)
23
+
{% endcapture %}
24
+
{% include /components/directory.html title="Table of Contents" %}
However, we recommend [using the `get_quotas` command](#check-your-quota) for the most up-to-date information.
106
+
107
+
## Check the size of a directory and its contents
108
+
109
+
If you want to check the size of specific directories and their contents, you can do this with the `ncdu` utility.
110
+
111
+
```
112
+
[user@spark-login ~]$ ncdu /home/username
113
+
[user@spark-login ~]$ ncdu /scratch/username
114
+
```
115
+
{:.term}
116
+
117
+
When `ncdu` has finished running, the output will give you a total file count and allow you to navigate between subdirectories for even more details. Type `q `when you're ready to exit the output viewer.
118
+
119
+
## What to do when you've reached your quota
120
+
121
+
When you've reached your quota, you may encounter error messages such as `Disk quota exceeded`. When you encounter the error message, we recommend the following steps:
122
+
123
+
1. Check your quota with `get_quotas`. At which data location did you reach your quota? Did you reach your disk quota or your items quota?
124
+
1. If possible, remove any files you no longer need from the system.
125
+
1. If you still need more disk space, [request more quota.](quota-request)
126
+
127
+
> ### ⚠️ CHTC is not a storage service
128
+
{:.tip-header}
129
+
130
+
> CHTC data locations are intended for temporarily storing files used in **active calculations only**. Once you are done with the files, please remove them from the system to clear disk space.<br><br>
131
+
> **We do not back up any of the data you place on our system. It is your responsibility to back up your own files.**
0 commit comments