FRONTERA ALLOCATION SUBMISSION GUIDELINES

Introduction

TACC’s Frontera High Performance Computing resource is provided through funding from the National Science Foundation (NSF). Frontera provides a uniquely balanced set of capabilities that supports both capability and capacity simulation, data intensive science, visualization, and data analysis, as well as emerging applications in AI and deep learning.  Representing a substantial public investment, the compute time provided by Frontera is vast, yet still finite, and the process for allocating this time requires rigor to insure the investment is protected and used as effectively as possible. This page defines the policies and submission guidelines governing allocations on the Frontera system. Frontera is a system for discovery and new use cases may emerge — please don’t hesitate to contact the project team if you feel you have special circumstances not covered herein.

Apply for an Allocation

Please thoroughly read this entire page so that you will be prepared to submit a high-quality proposal containing the documentation pertaining to eligibility, a description of the research to be performed using the testbed and sources of reearch support, and a justification for the amount of node hours requested. In addition to the guidelines below, you can view recordings of webinars that describe the various allocation tracks and that provide detailed guidance on how to properly execute and document code performance and scaling tests in your proposal.   You will submit your allocation request via the Resource Allocation System.

• Only Principal Investigators (PIs) may submit a Frontera allocation request.
• The quarterly Frontera Pathways (max 250K node-hours) and LSCP (max 1M node-hours) allocations request periods are: 
  • November 15 - December 15 for March 1 start.
  • February 15 - March 15 for June 1 start.
  • May 15 - June 15 for September 1 start.
  • August 15 - September 15 for December 1 start.
• Frontera LRAC (max 5M node-hours) annual allocation request period: 
  • December 1 - January 15 for April 1 start.

Eligibility

Frontera is broadly available to any US researcher (or their non-US collaborators) working in open research. By “open”, the expectation is that any research performed on Frontera will result in publication in a broadly available journal or conference.

An allocation on Frontera is headed by a project PI, typically a faculty member or researcher scientist at a US-based academic institution. The PI may then add additional investigators, including students, to their allocation at their discretion. PIs are responsible for ensuring that any users added to their allocation comply with the terms and conditions for use of the Frontera resource. Collaborative projects involving non-U.S. researchers are allowed as long as they include substantive intellectual participation by the U.S. researchers. 

While a PI is typically a research scientist or faculty member at an academic institution, there are a number of other cases where an individual may be eligible to be a PI:

• NSF Graduate Student Fellows: While in most cases, a graduate student is ineligible to be PI of an allocation request, an exception is made for NSF Graduate Student Fellows. Recipients of these NSF awards can submit requests for Startup allocations. Include supporting documentation (grant number or an award letter) as part of the request submission.

• Other federal agencies:  Research staff employed by federal agencies or non-NSF FFRDCs are eligible to apply for an allocation if their agency or center does not typically provide research staff with access to resources of adequate scope for the planned research.

• State educational offices or organizations and local school districts may submit allocation requests intended to broaden the impact, accelerate the pace, and increase the effectiveness of improvements in science, mathematics, and engineering education in both K-12 and post-secondary levels. A teacher or educator at an accredited public or private K-12 school is eligible to apply for an allocation as PI.

• Independent museums, observatories, libraries, research laboratories, professional societies and similar organizations in the United States that are directly associated with educational or research activities are eligible.

• U.S. commercial organizations, especially small businesses with strong capabilities in scientific or engineering research or education may apply for an allocation. The Frontera Project is interested in supporting projects that couple industrial research resources and perspectives with those of universities; therefore, it especially welcomes requests from cooperative projects involving both universities and the private commercial sector. It is necessary for these projects to submit their work in an open forum, and make the results of this work readily available to the public.

Allocation Metrics
How is my Time on Frontera Charged?

Compute time on Frontera is allocated in node hours. A typical Frontera node has 56 processor cores and 192GB of RAM. Note that while this is consistent with allocations on most modern systems, it breaks from the historical system of allocating per core-hour, so requests should be scaled accordingly. 

Other node types on Frontera are charged as multiples of the basic rate.  For instance, nodes with four single-precision GPUs (RTX5000) are charged at 3x SU per hour (triple the rate of a “normal” compute node). Lonestar6-GPU nodes with three double-precision GPUs (A100 ) are charged at 4x SU per hour, reflecting the approximate cost ratio of the GPU nodes to the CPU nodes. Using three NVIDIA A100 GPUs, the Lonestar6 GPU nodes provide similar peak performance and almost twice as much GPU memory as the now retired Longhorn V100 nodes originally offered via the Frontera project. Note that this rate is for allocation of the node, not for resources actually consumed – if you run a 2 hour job on A100 GPU node and use only 2 of the 4 GPUs on the node, you are still charged the full rate (8x), as the idle GPUs were not available to another user during this period.

Allocation Tracks

Frontera allocations are divided into four classes, or “tracks” of allocations through which projects may receive access to portions of Frontera’s available computational resources: Leadership Computing Resource Allocations (which for simplicity we will refer to as “LRACs”), “Pathways” allocations to enable projects to mature into computing at the largest scale, Leadership Community Partnerships (or, LSCPs), and discretionary allocations. The table below provides a preview of the allocation tracks.

Allocation Track	Description
LRAC (max 5M node-hours)	Large allocations, ready to immediately use up to ~5% of the annual capacity of the system.
Pathway (max 250K node-hours)	Projects that require, or will soon evolve to require, LRACs but which are not yet ready to compute (effectively) at that scale.
Large-Scale Community Partnerships (LSCP) (max 1M node-hours)	Allocations where we can’t strictly characterize the set of experiments to run in the allocation period, but that line up with a large multi-institution team or community who collectively justify use of Frontera, such as research based on data from a large instrument, or other large NSF investment.
Discretionary	Allocations managed directly by the PI for local, educational, industrial (and so on) use that does not fit well into the other three allocation tracks.

Leadership Computing Resource Allocations (LRAC)

Description and Need. Ultimately all activities of the Frontera project support the overriding project goal as articulated in the NSF program solicitation: “serve as a national resource for providing predictable and sustained long-term leadership computing capabilities for science and engineering to push the frontiers of knowledge….” In the context of Frontera, this means enabling computation-based scientific discovery, with a particular focus on “big science.”

Fraction of Total Allocations. A minimum of 65% of allocated time on the system will be dedicated to this track.

Mechanism of Award. Potential awardees will submit an allocation proposal to a peer review panel of independent experts that the Frontera team will invite and organize. Recommendations for award will be sent to NSF for their review and consensus agreement. Frontera allocations are highly competitive, and the highest ranked proposals are awarded as close to their requested amount as possible depending on availability. Accordingly, lower ranked proposals will be reduced significantly on the order of a Pathways allocation level. Deadlines will be at least once per year, possibly twice depending on availability of time. In an attempt to align allocations with research funding, while awards will be made for one year, annual renewals will be available until the end of the associated grant with a progress report verifying level of usage, publications, and progress towards research goals and also a computational plan for the coming year. There will be no extensions for existing allocations, thus a renewal submission is required annually regardless of remaining allocation balance.

Allocation Size Limits. LRAC requests can be submitted for up to 5M node hours per year for Frontera. Note that this represents nearly 10% of the full system annual capacity, so we expect typical requests to be much smaller.   The minimum size for a LRAC request is 250,000 node hours per year. For Frontera GPU and Lonestar6-GPU, requests can be submitted for up to 200K node hours per year for Frontera GPU and Lonestar6-GPU, with no minimum. For significantly underutilized allocations, we reserve the right to reduce the remaining allocation balance during the second half of the award period.

Pathways Allocations

Description and Need. Experience suggests that not all user teams can seamlessly make the transition  from more general-use resources, such as Stampede2, to a system that emphasizes larger scale computing. It is here that latent scaling bugs (such as race conditions, hard-coded array bounds, and so on) are often encountered for the first time. The “Pathways” allocations track is for projects that believe they are ready to begin scaling up, but have not yet fully tested their applications at scale.  Less stringent scaling data will need to be provided to receive a pathways allocation, and the total award size will be smaller, with a focus on preparing to reach the LRAC scale.

Fraction of Total Allocations. Pathways allocations will be available for up to 15% of system capacity.

Mechanism of Award. Pathways award should be submitted through the Frontera portal, and will be awarded quarterly.  The award duration will be for 1 year. There will be no extensions for existing allocations, thus a renewal submission is required annually regardless of remaining allocation balance.

Allocation Size Limits.  Pathways awards may be between 10,000-250,000 node hours per year for Frontera. For Frontera GPU and Lonestar6-GPU, requests can be submitted  for up to 12K node hours per year, with no minimum. For significantly underutilized allocations, we reserve the right to reduce the remaining allocation balance during the second half of the award period.

Large Scale Community Partnerships (LSCP)

Description and Need. These allocations support a large community of users, typically multi-institutional, working on the same research topic, who collectively justify the need for this scale of HPC resource but individually would not. A shortcoming of the current LRAC and XSEDE allocation processes is that the 12-month limitation on allocation requests means that very large science investments by the NSF and other agencies (e.g. MREFC projects like LIGO or LHC) cannot count on being able to “compute to completion.” This mismatch is felt most acutely at the largest computational scale, where researchers are attempting fundamental breakthroughs in their simulations and data analyses. Extended allocations, punctuated by merit reviews to assure appropriate progress, will ensure uninterrupted availability of critical computational resources, and encourage rational allocation of science expertise.

Fraction of Total Allocations. Up to fifteen percent of available system time will be awarded in LSCP allocations, subject to the availability of quality proposals.

Mechanism of Award. Application should be made through the Frontera portal. LSCP awards will be reviewed at least twice per year. The LSCP process will make merit-based allocation awards annually for allocations spanning to three years, with years 2 and 3 awarded after successful completion of a progress review and an updated computational plan for the next year. There will be no extensions for existing allocations, thus a renewal submission is required annually regardless of remaining allocation balance.

Allocation Size Limits.  LSCP awards may be between 25,000-1,000,000 node hours per year for Frontera. For Frontera GPU and Lonestar6-GPU, requests can be submitted  for up to 30K node hours per year, with no minimum. For significantly underutilized allocations, we reserve the right to reduce the remaining allocation balance during the second half of the award period.

Discretionary Allocations

The remaining 5% of Frontera’s resources will be allocated by TACC, to be awarded at the PI’s discretion to areas of urgent need, industry projects, or for UT use. To be considered for a discretionary allocation, please contact the project team directly. 

Education Allocations

Within the discretionary pool, or from unused time in other pools, time on Frontera may be awarded for classroom, training, or other educational uses.  TACC reserves the right to direct these to other computing resources if it deems appropriate.  An Educational allocation may request up to 15,000 node hours on Frontera. This would be approximately equivalent to a 50 student class allocating 8 nodes per student for a 40 hour work week. For Frontera GPU and Lonestar6-GPU requests can be submitted for up to 300 node hours. To request an Education Allocation please contact the project team directly.

Startup Allocations

A startup allocation may be requested to gather benchmark data for a larger allocation request, or to test the suitability of the system for a particular research project.  Startup awards are less than 6 month duration and may be for up to 5,000 node hours for Frontera, and 100 node hours for Frontera GPU and Lonestar6-GPU. To request a Startup Allocation please contact the project team directly.

Proposals and Review Criteria

Frontera allocation are reviewed for merit by a committee consisting of members of the resource allocation board. The board is made up of researchers in the academic community with support from members of the project team. The board makes recommendations to the project based solely on the merit of the proposal, and not on overall availability of the resource. The board will review the proposal and make a recommendation to the project team based on the following criteria:

• Intellectual Merit: Is the proposed work supported by a grant or grants that have undergone review for intellectual merit and/or broader impact (if relevant to the allocation)? If so, is the allocation request consistent with the objectives of the supporting grants and is the scale of resource use commensurate with the level and purpose of support? If a request is not supported by a merit-reviewed award, reviewers will assess the intellectual merit of the proposed work and factor that into their overall recommendation. Reviewers will also consider whether the identified support provides necessary and sufficient human resources to complete the proposed work.

• Appropriateness of Methodology: Does the request describe appropriate tools, methods, and approaches for addressing the research objectives? These methodologies may be community codes or models, data analysis methods, or algorithmic formulations expressed in user-developed scripts or tools. Is this methodology appropriate for use on Frontera?

• Appropriateness of Research Plan: Does the request describe necessary and sufficient computational experiments to answer the research questions posed? In some cases, the research plan may be more reasonably expressed as estimates of resource use, supported by past or early experience. Serious concerns about the research plan will be documented in reviews and may lead to reduced allocation awards.

• Efficient Use of Resources: Has the request identified appropriate resources for undertaking the research plan with the proposed methodology? And will those resources be used as efficiently as is reasonably possible?

• For renewal requests, success of prior or other existing allocations in terms of significant results, published research and new funding.

• Demonstrated need for a large scale resource.

Proposals that are deemed competitive by the project advisory board are then reviewed by the project team for technical feasibility (i.e, can the project be implemented in the Frontera environment?) and for the availability of time on the resource. Frontera allocations are highly competitive, and the highest ranked proposals are awarded as close to their requested amount as possible depending on availability. Accordingly, lower ranked proposals will be reduced significantly. Recommended proposals will then be sent to NSF for final approval.   

Conflict of Interest and Confidentiality

Every effort is made to avoid conflicts of interest. Reviewers are not allowed to review or be present for the discussion of requests from their home institution, former students, postdocs, advisors, or current and recent collaborators. If in the opinion of a PI, a certain individual has a conflict of interest, the PI may request that the individual not act as reviewer on their request.   All reviews remain confidential, and are made available only to:  The PI and Co-PIs submitting the request, assigned reviewers from the Advisory Committee, project staff involved in the allocation review process, and NSF program officers for the project.

While the contents of reviews and details of allocation requests and experiments remain confidential, a list of projects receiving allocations and the general area of research may be made public on the project web site and used in project reports and presentations.

Supplemental Requests

A supplement is a request for additional resources during an existing allocation’s one year time frame. Its purpose is to support changes in the original computational research plan that are required to achieve the scientific goals of the project. This may include altered or new goals, or support for projects proceeding more rapidly than anticipated or that require more resources than anticipated. Supplement awards are highly dependent upon availability of resources and limited to when allocation awards at the previous allocation period have been reduced to eliminate oversubscriptions. Supplements are not a mechanism to acquire additional resources for awards that were recommended for less than the amount originally requested.   Supplemental requests require the filing of a progress report that documents what has been done with the originally awarded allocation, and makes the case for additional resources.

Proposal Format

LRAC, Pathways, and LSCP Submissions Requests

When writing your LRAC request, be clear and concise. We strive to have domain experts review every request, but they may not have deep expertise in your specific subdomain. Someone outside of your area should be able to understand the scientific objectives and understand why the chosen technique is preferred over another.

Renewal requests require less documentation, as described later.

The documents required of a new LRAC request ensures reviewers can effectively determine how each request satisfies the Frontera allocation Review Criteria. LRAC requests are limited to 10 pages.

Research requests must include a well-documented resource-use plan that describes how the requested allocations are necessary and sufficient to accomplish the project’s research objectives. An effective resource-use plan must address the Frontera allocation Review Criteria and, in particular, must include the following elements:

• Scientific Background
• Research Objectives and specific questions to be pursued
• Resource Usage Plan to achieve the research objectives
• Justification of the allocation amounts for all resources and resource types
• Access to other CI resources and why those resources are not available or sufficient for the work proposed in this request

Scientific Background and Support

Succinctly state the scientific objectives that will be facilitated by the allocation. The existing merit-reviewed supporting grants should be listed and briefly described; please include the end date for each grant listed. Requests with merit-reviewed supporting grants will not be subject to further scientific review by the LRAC; however, the stated scientific objectives must match or be sub-goals of those described in the listed funding award(s). The description of the objectives should be sufficient to allow the reviewers to assess the resource usage plan.

Research Questions

Identify the specific research questions that are covered by the allocation request. Within the context of the scientific background and supporting grants, these objectives and questions should be stated so that the reviewers can understand how elements of the resource plan will contribute to relevant answers. If you consider the allocation request in terms of computational experiments, the research objectives and questions define the experiments to be conducted.

Resource Usage Plan

The bulk of the document should focus on the resource usage plan and allocation request. Inadequate justification for requested resources is the primary reason for most reduced or denied allocations. Once again, the PI should keep in mind the Frontera allocation Review Criteria when describing and justifying the choice of resources and allocation amounts.

Justifying Allocation Amounts

This section should contain code performance timings, resource usage details, and scaling information to support the calculation of the resource request. Ideally, the code performance data should be provided from benchmark runs on the resource requested, using the model configuration(s) needed for the computational plan proposed and demonstrating the scaling efficiency to the job size(s) planned. If applicable, the panel may accept well-justified performance data from architecturally equivalent systems. PI’s should contact the Frontera Allocations Coordinator to obtain a small allocation for benchmarking or to discuss non-Frontera performance data to be used in their submission.

The justification of allocation should take the quantitative parameters from the resource usage plan and combine them with benchmark cost information to calculate the allocation needed. Where possible for computational experiments, the justification should tabulate and calculate the costs of conducting each experiment for each resource and resource type.

Disclosure of Access to Other Compute Resources

Because of the high demand for NSF-funded resources, Frontera and the LRAC closely consider whether the PI has access to other, less constrained resources on which the proposed work could be conducted. All PIs must describe their local resources and other non-Frontera resources available to them, including both local and other national resources.

Page limit: 10 pages

References

A PI may use this OPTIONAL document to separate a lengthy bibliography of cited work to take full advantage of the page limits in the Main Document. This bibliography is not the publications resulting from prior Frontera support, which should be entered into the Frontera publications database, but rather other citations referenced in describing or supporting the intellectual merit of the proposed work or the appropriateness of the proposed approach for addressing the research objectives.

Page limit: No limit.

Prepare and submit your New allocation request.

 

Renewal Submission Requirements

For science teams that are renewing their Frontera LRAC, Pathways, or LSCP allocation to continue the same research topic, a progress report may be submitted in lieu of a new request. The progress report should describe:

• How the PI’s current or prior allocation was used and summarize the findings or results.
• A list of any relevant  publications and other communications resulting from the work.
• Explanation of  any significant digressions from the originally proposed resource usage plan for the prior period.  Include an updated resource usage plan for the upcoming year, and a justification of the allocation amounts. Previous benchmarking data from the original request can be referenced.
• If a PI’s allocation was significantly underutilized, the Progress Report should briefly describe the reasons for the underutilization and any mitigation by the PI to avoid the situation with the current allocation.

Page limit: 8 pages.

Prepare and submit your Renewal request.

 

A Startup Allocation request consists of the following components:

• PI Name, Title, and Organization
• (optional) Co-investigator names, titles, and organizations
• Summary of the proposed experiments (1 page maximum).
• Estimated resources required (include a short justification of the number and type of nodes you will request, as well as any storage requirements). (1 page maximum)
• List any sources of funding (or pending funding) that support the proposed research, including funding agency and grant name(s) (Startup Allocations may be for unfunded exploratory research)

An Education Allocation request consists of the following components:

• PI Name, Title, and Organization
• (optional) Co-investigator names, titles, and organizations
• Summary of the proposed course or training session to be supported, including the expected number of students who will participate (1 page maximum).
• Estimated resources required (include a short justification of the number and type of nodes you will request, and any specific software, hardware, or networking requirements. (1 page maximum)
• If applicable, list any sources of funding that support the proposed course, including funding agency and grant name(s) (Educational Allocations do not require any external funding, but in cases where it does exist, the project would like to record any grants that are impacted by Frontera).

Document Formatting

While readability is of greatest importance, documents must satisfy the following minimum requirements. Documents that conform to NSF proposal format guidelines will satisfy these guidelines.

• Margins: Documents must have 2.5-cm (1-inch) margins at the top, bottom, and sides.
• Fonts and Spacing: The type size used throughout the documents must conform to the following three requirements:

Use one of the following typefaces identified below:

• Arial 11, Courier New, or Palatino Linotype at a font size of 10 points or larger;
• Times New Roman at a font size of 11 points or larger; or
• Computer Modern family of fonts at a font size of 11 points or larger.

A font size of less than 10 points may be used for mathematical formulas or equations, figures, table or diagram captions and when using a Symbol font to insert Greek letters or special characters. PIs are cautioned, however, that the text must still be readable.

Page Numbering: Page numbers should be included by the submitter.

One Project per PI

An individual may be a PI on only one active Frontera allocation request/project at a given time. Several distinct research activities can be combined in a single resource request, however, the resource request for each activity must be justified, and any allocation-size limits apply to the aggregate request.

The single-project rule is designed to minimize the effort required by PIs for submitting resource requests and the overhead to the process for reviewing those requests. While PIs may have several different funded grants that require computational support, these should be included as sub-projects within a single request.

There are several exceptions to this rule:

• A PI may submit a Research request at any time during an active Startup allocation. A successful Research allocation will supersede any Startup allocations and start a new 12-month allocation period.

• If a PI is leading an individual research activity (LRAC or Pathways) as well as a community consortium activity or community service activity (LSCP), separate requests for these distinct activities will be considered.

• A PI with active Startup or Research projects may submit requests for any number of Education projects to keep classroom and training users separate from research activities.

Similarly, to minimize the effort required to gain access to Frontera, closely collaborating researchers should submit a single collaborative request rather than several individual requests. For example, a PI and associated post-doctoral researchers; investigators supported by the same funding grant; and researchers in the same lab group should consider submitting a request describing and justifying the various subactivities. One of the collaborators is designated as the PI, and others can be designated as co-PIs.

Acknowledgement of Support

An acknowledgement of support from the Frontera project and the National Science Foundation should appear in any publication of material, whether copyrighted or not, that describes work which benefited from access to TACC’s Frontera resource. For suggested language, see TACC Citation.

Other Requests and Feedback

The project team is interested in receiving as many excellent requests as possible, and encourages any potential investigators who wish to use the resource to not hesitate in contacting the team for additional help in preparing this request, with questions about the process, or to discuss any type of project that may not fit the structure described here. The project team also welcomes input in improving this process, and encourages feedback on both this document and the process.