Merge Split Branch

Author: actions-user

preview-staging-transition/assets/search/index.json

@@ -209,14 +209,14 @@ <h1 class="uw-mini-bar mb-4">
 
       <h2 id="featured-fellow">Featured Fellow</h2>
 
-      <div id="wil-cram" class="bg-white team-card justify-content-center mb-3 flex-grow-1">
+      <div id="william-jiang" class="bg-white team-card justify-content-center mb-3 flex-grow-1">
     <div class="uw-card-content shadow">
         <div class="row">
             <div class="col-3">
                 <div class="overflow-hidden d-flex justify-content-center">
 
 
-                    <img src="/web-preview/preview-staging-transition/staff-list/images/wil_cram.jpg" alt="Headshot for Wil Cram" />
+                    <img src="/web-preview/preview-staging-transition/staff-list/images/william_jiang.png" alt="Headshot for William Jiang" />
 
                 </div>
             </div>
@@ -225,27 +225,32 @@ <h2 id="featured-fellow">Featured Fellow</h2>
                     <div>
                         <h2 class="uw-mini-bar mb-0 mt-3">
 
-                            Wil Cram
+                            William Jiang
 
                         </h2>
-                        <div><h6 class="mt-2">Mentor(s):</h6> Greg Thain</div>
+                        <div><h6 class="mt-2">Mentor(s):</h6> Emma Turetsky and Justin Hiemstra</div>
                     </div>
                 </div>
             </div>
             <div class="col-12">
-                <h4 class="mt-3">Schedd performance analysis for human</h4>
+                <h4 class="mt-3">Integrating Pelican and Globus</h4>
                 <div class="pt-2">
-                    <p>The condor_schedd is a single threaded program, and when it is overloaded, 
-it is difficult for administrators to understand why.  There are some
-statistics about what it is doing, but there is no clear way to present
-this information in a useful way to an administrator. Students working 
-on this project would build visualizations of complex data, and work
-with end users and facilitators to tune output for real world human 
-consumption.</p>
+                    <p>The Pelican Platform is an open-source software platform designed to create
+data federations by unifying diverse data repositories (e.g. POSIX
+filesystems, S3) under a single architecture. It differs from the Globus
+project, whose main goal is to facilitate peer-to-peer data transfer, and it
+thus lacks the means to plug into cluster scheduling frameworks like HTCondor.</p>
+
+<p>This project focused on bridging the gap between Pelican and Globus by
+integrating Globus transfer capabilities into Pelican. This will allow users
+to leverage Globus’s robust data transfer features while benefiting from
+Pelican’s unified data access model. The integration involved developing a
+Globus backend for Pelican and enabling seamless data transfers between
+Pelican-managed repositories and external systems via Globus.</p>
 
                 </div>
                 <div>
-                    <a class="btn btn-primary" href="./reports/2024/wil-cram">Read Final Report</a>
+                    <a class="btn btn-primary" href="./reports/2024/william-jiang">Read Final Report</a>
                 </div>
             </div>
         </div>

preview-staging-transition/assets/search/metadata.json

@@ -360,7 +360,7 @@ <h4 id="5-move-the-container-sif-file-to-staging-">5. Move the container .sif fi
 
 <p>Once you are satisfied that your container is built correctly, copy your <code>.sif</code> file to your staging directory.</p>
 
-<pre class="term"><code>mv my-container.sif /staging/$USER
+<pre class="term"><code>mv my-container.sif /staging/u/username
 </code></pre>
 
 <p>Once the file has transferred, exit the interactive job with</p>
@@ -530,7 +530,7 @@ <h2 id="move-the-container-sif-file-to-staging">Move the Container .sif File to
 <p>Since Apptainer <code>.sif</code> files are routinely more than 1GB in size, we recommend that you transfer <code>my-container.sif</code> to your <code>/staging</code> directory.
 It is usually easiest to move the container file directly to staging while still in the interactive build job:</p>
 
-<pre class="term"><code>mv my-container.sif /staging/$USER
+<pre class="term"><code>mv my-container.sif /staging/u/username
 </code></pre>
 
 <p>If you do not have a <code>/staging</code> directory, you can skip this step and the <code>.sif</code> file will be automatically transferred back to the login server when you exit the interactive job.

preview-staging-transition/fellowships/index.html

@@ -310,9 +310,9 @@ <h2 id="check-your-quota">Check your quota</h2>
 <p>This will print a table with your <code>/home</code> and <code>/staging</code> quotas. An example output is shown below.</p>
 
 <pre class="term"><code>[user@ap2002 ~]$ get_quotas
-Path            Disk_Used(GB)  Disk_Limit(GB)  Files_Used  File_Limit
-/home/user      16.0711        40              8039        N/A
-/staging/user   13.4731        100             12          1000
+Path              Disk_Used(GB)  Disk_Limit(GB)  Files_Used  File_Limit
+/home/username        16.0711        40              8039        N/A
+/staging/u/username   13.4731        100             12          1000
 </code></pre>
 
 <p>To print the quota for any path in <code>/staging</code>, <code>/software</code>, and <code>/projects</code> that you have access to, (i.e., a group directory), use the <code>-p</code> option:</p>
@@ -368,10 +368,10 @@ <h3 id="option-2-check-your-message-of-the-day-motd">Option 2: Check your Messag
  == NOTICE: THIS NODE IS ON PUPPET ENVIRONMENT "puppet8" ==
 
 Filesystem quota report (last updated 10:33 AM)
-Storage               Used (GB)    Limit (GB)    Files (#)    File Cap (#)    Quota (%)
-------------------  -----------  ------------  -----------  --------------  -----------
-/home/user                29.38            40           94               0        73.46
-/staging/user             50.23          1000          110           10000         5.02
+Storage                     Used (GB)    Limit (GB)    Files (#)    File Cap (#)    Quota (%)
+------------------        -----------  ------------  -----------  --------------  -----------
+/home/username                  29.38            40           94               0        73.46
+/staging/u/username             50.23          1000          110           10000         5.02
 </code></pre>
 
 <p>However, we recommend <a href="#check-your-quota">using the <code>get_quotas</code> command</a> for the most up-to-date information.</p>

preview-staging-transition/uw-research-computing/apptainer-htc.html

@@ -338,7 +338,7 @@ <h2>Table of Contents</h2>
         </li>
         <li><a href="#stage-large-data">Stage large data</a>
           <ul>
-            <li><a href="#request-a-staging-directory">Request a <code>/staging</code> directory</a></li>
+            <li><a href="#your-personal-staging-directory">Your personal <code>/staging</code> directory</a></li>
             <li><a href="#reduce-file-counts">Reduce file counts</a></li>
             <li><a href="#use-the-transfer-server">Use the transfer server</a></li>
             <li><a href="#remove-files-after-jobs-complete">Remove files after jobs complete</a></li>
@@ -391,19 +391,39 @@ <h2 id="stage-large-data">Stage large data</h2>
 <p>In order to stage large data for use on CHTC’s HTC system, users must:</p>
 
 <ol>
-  <li><strong>Request a <code>/staging</code> directory</strong>: Use our quota request form.</li>
   <li><strong>Reduce file counts</strong>: Combine and compress files that are used together.</li>
   <li><strong>Transfer files to the HTC system via the transfer server</strong>: Upload your data via our dedicated file transfer server.</li>
   <li><strong>Remove files after jobs complete</strong>: Our data staging space is quota-controlled and not backed up.</li>
 </ol>
 
-<h3 id="request-a-staging-directory">Request a <code>/staging</code> directory</h3>
+<h3 id="your-personal-staging-directory">Your personal <code>/staging</code> directory</h3>
 
-<p>Any one with a CHTC account whose data meets the intended use above can request space in our large data staging area by filling out a quota request form. The default quota is 100 GB / 1000 items; if a larger quota is needed, request a higher quota. The created directory will exist at this path: <code>/staging/username</code></p>
+<p>Each user should have a personal <code>/staging</code> directory. The created directory will exist in an alphabetized subdirectory based on the <strong>first letter</strong> of your NetID. For example:</p>
+
+<table>
+  <thead>
+    <tr>
+      <th>NetID</th>
+      <th>Path to your personal <code>/staging</code> directory</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>alice</code></td>
+      <td><code>/staging/a/alice</code></td>
+    </tr>
+    <tr>
+      <td><code>bucky</code></td>
+      <td><code>/staging/b/bucky</code></td>
+    </tr>
+  </tbody>
+</table>
 
 <p>We can also create group or shared spaces by request.</p>
 
-<p style="text-align: center; margin-bottom: 0; font-weight: bold;">Need a <code>/staging</code> directory or higher quota?</p>
+<p>The default quota is 100 GB / 1000 items; if a larger quota is needed, request a higher quota.</p>
+
+<p style="text-align: center; margin-bottom: 0; font-weight: bold;">Need a group <code>/staging</code> directory or higher quota?</p>
 <div class="d-flex mb-3">
 	<div class="p-3 m-auto">
 		<a class="btn btn-primary" style="text-align: center" href="quota-request">Quota request form</a>
@@ -412,7 +432,7 @@ <h3 id="request-a-staging-directory">Request a <code>/staging</code> directory</
 
 <h3 id="reduce-file-counts">Reduce file counts</h3>
 
-<p>The file system backing our <code>/staging</code>space is optimized to handle small numbers of large files. If your job requires many small files, we recommend placing these files in the <code>/home</code> directory or compressing multiple files into a single zip file or tarball. See <a href="htc-job-file-transfer#data-storage-locations">this table</a> for more information on the differences between <code>/staging</code> and <code>/home</code>.</p>
+<p>The file system backing our <code>/staging</code> space is optimized to handle small numbers of large files. If your job requires many small files, we recommend placing these files in the <code>/home</code> directory or compressing multiple files into a single zip file or tarball. See <a href="htc-job-file-transfer#data-storage-locations">this table</a> for more information on the differences between <code>/staging</code> and <code>/home</code>.</p>
 
 <p>Data placed in our large data <code>/staging</code> location should be stored in as few files as possible (ideally, one file per job), and will be used by a job only after being copied from <code>/staging</code> into the job working directory. Similarly, large output should first be written to the job’s working directory then compressed in to a single file before being copied to <code>/staging</code> at the end of the job.</p>
 
@@ -426,7 +446,7 @@ <h3 id="use-the-transfer-server">Use the transfer server</h3>
 <p>Uploading or downloading data to <code>/staging</code> should only be performed via CHTC’s transfer server, <code>transfer.chtc.wisc.edu</code>, and <strong>not via a CHTC Access Point.</strong></p>
 
 <p>For example, you can use <code>scp</code> to transfer files into your <code>/staging</code> directory:</p>
-<pre class="term"><code>$ scp large.file netid@transfer.chtc.wisc.edu:/staging/netid/ 
+<pre class="term"><code>$ scp large.file username@transfer.chtc.wisc.edu:/staging/u/username/ 
 </code></pre>
 
 <h3 id="remove-files-after-jobs-complete">Remove files after jobs complete</h3>
@@ -442,7 +462,7 @@ <h3 id="transfer-large-input-files">Transfer large input files</h3>
 <p>Staged files should be specified in the job submit file using the <code>osdf:///</code> or <code>file:///</code> syntax,
 depending on the size of the files to be transferred. <a href="htc-job-file-transfer#transfer-input-data-to-jobs-with-transfer_input_files">See this table for more information</a>.</p>
 
-<pre class="sub"><code>transfer_input_files = osdf:///chtc/staging/username/file1, file:///staging/username/file2, file3 
+<pre class="sub"><code>transfer_input_files = osdf:///chtc/staging/u/username/file1, file:///staging/u/username/file2, file3 
 </code></pre>
 
 <h3 id="transfer-large-output-files">Transfer large output files</h3>
@@ -452,7 +472,7 @@ <h3 id="transfer-large-output-files">Transfer large output files</h3>
 <p>Large outputs should be transferred to staging using the same file transfer protocols in HTCondor’s <code>transfer_output_remaps</code> or <code>output_destination</code> option:</p>
 
 <pre class="sub"><code>transfer_output_files = file1, file2, file3
-transfer_output_remaps = "file1 = osdf:///chtc/staging/username/file1; file2 = file:///staging/username/file2"
+transfer_output_remaps = "file1 = osdf:///chtc/staging/u/username/file1; file2 = file:///staging/u/username/file2"
 </code></pre>
 
 <p>In the example above, <code>file1</code> and <code>file2</code> are transferred to <code>/staging</code> via two different file transfer protocols (which depends on their file size), and <code>file3</code> is transferred to the submit directory on <code>/home</code>.</p>
@@ -484,14 +504,14 @@ <h2 id="submit-jobs-using-staged-data">Submit jobs using staged data</h2>
 
 <pre><code class="language-{.sub}">### Example submit file for a single job that stages large data
 # Files for the below lines MUST all be somewhere within /home/username,
-# and not within /staging/username
+# and not within /staging/u/username
 
 executable = run_myprogram.sh
 log = myprogram.log
 output = $(Cluster).out
 error = $(Cluster).err
 
-transfer_input_files = osdf:///chtc/staging/username/myprogram, file:///staging/username/largedata.tar.gz
+transfer_input_files = osdf:///chtc/staging/u/username/myprogram, file:///staging/u/username/largedata.tar.gz
 
 # IMPORTANT! Require execute servers that can access /staging
 Requirements = (Target.HasCHTCStaging == true)

preview-staging-transition/uw-research-computing/check-quota.html

@@ -436,16 +436,13 @@ <h2 id="a-submit-file">A. Submit File</h2>
 
 <pre><code class="language-{.sub}">### Example submit file for a single staging-dependent job
 
-universe = vanilla
-
 # Files for the below lines will all be somewhere within /home/username,
-# and not within /staging/username
+# and not within /staging/u/username
 log = run_myprogram.log
 executable = run_Trinity.sh
 output = $(Cluster).out
 error = $(Cluster).err
 transfer_input_files = trinityrnaseq-2.0.1.tar.gz
-should_transfer_files = YES
 
 # Require execute servers that have large data staging
 Requirements = (Target.HasCHTCStaging == true)
@@ -495,7 +492,7 @@ <h2 id="c-executable-script">C. "Executable" script</h2>
 <pre class="file"><code>#!/bin/bash
 # Copy input data from /staging to the present directory of the job
 # and un-tar/un-zip them.  
-cp /staging/username/reads.tar.gz ./
+cp /staging/u/username/reads.tar.gz ./
 tar -xzvf reads.tar.gz
 rm reads.tar.gz
 
@@ -514,7 +511,7 @@ <h2 id="c-executable-script">C. "Executable" script</h2>
 # Trinity will write output to the working directory by default, 
 # so when the job finishes, it needs to be moved back to /staging
 tar -czvf trinity_out_dir.tar.gz trinity_out_dir
-cp trinity_out_dir.tar.gz trinity_stdout.txt /staging/username/
+cp trinity_out_dir.tar.gz trinity_stdout.txt /staging/u/username/
 rm reads_*.fq trinity_out_dir.tar.gz trinity_stdout.txt
 
 ### END

preview-staging-transition/uw-research-computing/file-avail-largedata.html

@@ -305,7 +305,7 @@ <h3 id="move-the-image-to-your-staging-directory">Move the image to your staging
 
 <p>Because container images are generally large, we require users to move these images into their staging directories. While you are still in your interactive job, move the image to your staging directory.</p>
 
-<pre class="term"><code>mv container.sif /staging/username/
+<pre class="term"><code>mv container.sif /staging/u/username/
 </code></pre>
 
 <h2 id="use-the-apptainer-image-in-a-job">Use the Apptainer image in a job</h2>