Rules and Requirements

A. CHALLENGE-SPECIFIC RULES

CHALLENGE TITLE: RATER Competition

CHALLENGE HOST: The Learning Agency

CHALLENGE HOST ADDRESS: 700 12th St. N.W, Suite 700 PMB 93369, Washington, D.C. 20002, USA

CHALLENGE WEBSITE: https://the-learning-agency.com (the “Site”)

SUBMISSION LICENSE TYPE: Open Source MIT

ENTRY IN THIS CHALLENGE CONSTITUTES YOUR ACCEPTANCE OF THESE OFFICIAL CHALLENGE RULES.

These Challenge-Specific Rules and the General Challenge Rules in Section B. below are, collectively, the “Challenge Rules”. The Challenge Rules are subject to change at any time by the Challenge Host. The Challenge Host will use commercially reasonable efforts to notify Participants of any changes to the Challenge Rules, including by posting the modified Challenge Rules on the Site.

The Challenge is a skills-based challenge to promote and further the field of data science. Your Challenge submissions (“Submissions”) must conform to the requirements stated on the Site. Your Submissions will be scored based on the evaluation metric described on the Site

1. TEAMS, TEAM LEADS AND TEAM MEMBERS

Each participant team (“Team”) will be comprised of a single Team Lead (as defined herein) and any number of Team Members (as defined herein). A team lead (“Team Lead”) is the individual selected by the Challenge Host for a Team, who will be the primary point of contact between the Challenge Host and such Team. Team Leads must meet the eligibility requirements of the Challenge Rules. The platform for the Challenge will have one account for every Team Lead, which is the only valid submission point for Submissions from such Team Lead’s Team. Team Leads are the only individuals eligible to receive the Prizes.

A team member (“Team Member”) is any individual working on a Team under a Team Lead. There is no limit on how many Team Members may be part of a Team. There is no maximum Team size, but the size of a Prize remains constant regardless of the number of people within the winning Team. Team Members do not have an account on the platform for the Challenge, and will not receive a prize for the Challenge.

The Challenge is open to residents of the United States and worldwide, except residents of Cuba, Iran, Syria, North Korea, Russia, Sudan, Syria, or the Crimea, Donetsk, or Luhansk regions of Ukraine. In addition, if you are subject to U.S. export controls or sanctions, you may not enter the Challenge. Other local rules and regulations may apply to you, so please check your local laws to ensure that you are eligible to participate in skills-based competitions.

2. PARTICIPANT LICENSE

Under Section 9 (Participants’ Obligations) of the General Challenge Rules below, you hereby grant and will grant Challenge Host the following license(s) with respect to your Submission as a Challenge Participant:

The “License”:

You hereby license and will license to the Challenge Host your winning Submission and the source code used to generate the Submission under an Open Source MIT license (see https://opensource.org/licenses/MIT) that in no event limits commercial use of such Submission or code, or model containing or depending on such Submission or code. To the extent your Submission makes use of generally commercially available software not owned by you that you used to generate your Submission, but that can be procured by the Challenge Host without undue expense, you do not grant the license in the preceding sentence to that software.

3. AUTOMATED MACHINE LEARNING TOOLS (AMLT).

Individual Participants and Teams may use automated machine learning tool(s) (“AMLT”) (e.g., Google AutoML, H2O Driverless AI, etc.) to create a Submission, provided that the Participant or Team ensures that they have an appropriate license to the AMLT such that they are able to comply with the Challenge Rules, including without limitation Section 2 (Participant License) of these Challenge-Specific Rules

B. GENERAL CHALLENGE RULES

1. BINDING AGREEMENT.

To enter the Challenge, you (both Team Leads and Team Members) must agree to the Challenge Rules, which include these General Challenge Rules and incorporate by reference the provisions and content of the Challenge-Specific Rules. Please read the Challenge Rules carefully to ensure you understand and agree. You further agree that submission of a Submission in the Challenge constitutes your agreement to the Challenge Rules. Team Leads may not submit a Submission to the Challenge, and are not eligible to receive the prize associated with the Challenge (“Prize”) unless they agree to the Challenge Rules. The Challenge Rules form a binding legal agreement between you and the Challenge Host with respect to the Challenge.

2. ELIGIBILITY.

A. To be eligible to enter the Challenge, you must be:
(i) the older of 18 years old or the age of majority in your jurisdiction of residence (unless otherwise agreed to by Challenge Host and appropriate parental/guardian consents have been submitted to, received and accepted by Challenge Host);
(ii) not a resident of Cuba, Iran, Syria, North Korea, Russia, Sudan, Syria, or the Crimea, Donetsk or Luhansk region of Ukraine; and
(iii) not a person or representative of an entity under U.S. export controls or sanctions (see https://www.treasury.gov/resource-center/sanctions/Programs/Pages/Programs.aspx).

If you are entering as a representative of a company, educational institution or other legal entity, or on behalf of your employer, the Challenge Rules are binding on you, individually, and the entity you represent or are an employee of. If you are acting within the scope of your employment or representation, as an employee, contractor, or agent of another party, you warrant that such party has full knowledge of your actions and has consented thereto, including your receipt of a Prize (if participating as a Team Lead). You further warrant that your actions do not violate your employer’s or entity’s policies and procedures.
The Challenge Host reserves the right to verify eligibility and to adjudicate on any dispute at any time. If you provide any false information relating to the Challenge concerning your identity, residency, mailing address, telephone number, email address, ownership of right, or information required for entering the Challenge, you may be immediately disqualified from the Challenge.

B. Unless otherwise stated in the Rules or prohibited by internal policies of the Challenge Entities, employees, interns, contractors, officers and directors of Challenge Entities may enter and participate in the Challenge, but are not eligible to receive the Prize. “Challenge Entities” means the Challenge Host, and their respective parent companies, subsidiaries and affiliates. If you are such a Participant from a Challenge Entity, you are subject to all applicable internal policies of your employer with respect to your participation.

3. CHALLENGE PERIOD.

The Challenge will run from the Start Date (February 26, 2024) to the Final Submission Deadline (April 27, 2024) (such duration, the “Challenge Period”). The Challenge Period is subject to change by the Challenge Host at any time, and Challenge Host may introduce additional hurdle deadlines during the Challenge Period. Any updated or additional deadlines will be publicized on the Site. It is your responsibility to check the Site regularly to stay informed of any deadline additions or changes.

4. CHALLENGE PARTICIPATION.

NO PURCHASE NECESSARY TO ENTER OR RECEIVE A PRIZE OR AN HONORARIUM (IN THE CASE OF TEAM LEADS). To participate in the Challenge as a Team Lead, you must be selected by the Challenge Host to participate as a Team Lead for a Team, and you must accept the terms of the prize agreement. Team Leads may choose the Team Members for his or her Team, subject to the Eligibility Requirements set forth in Section 2 of these General Challenge Rules.

Submission Limits

There are no Submission limits. However, Teams are required to make at least one Submission per week during the Challenge Period. Submissions to the leaderboard will be evaluated as they are submitted and the leaderboard will be updated following each Submission.
Each Team may only select one final Submission for judging.

5. CHALLENGE DATA.

“Challenge Data” means the data or datasets available from the Site for the purpose of use in the Challenge, including any prototype or executable code provided on the Site. The Challenge Data will contain private and public test sets.

A. Data Access and Use. You may access and use the Challenge Data only for participating in the Challenge and on the official Discord channel that the Challenge Host shared. You may not use data from the Challenge Data’s test set to design, train, or evaluate the models associated with your Submissions. The Challenge Host reserves the right to disqualify any Participant who uses the Challenge Data other than as permitted by the Site and the Challenge Rules.

B. External Data. You may use data other than the Challenge Data (“External Data”) to develop and test your Submissions. However, you will ensure the External Data is publicly available and equally accessible to use by all Participants of the Challenge, for purposes of the Challenge, at no cost to the other Participants. The ability to use External Data under this Section 5.B (External Data) does not limit your other obligations under these Challenge Rules, including but not limited to Section 9 (Participants’ Obligations).

6. SUBMISSION CODE REQUIREMENTS.

A. Private Code Sharing. Unless otherwise specifically permitted under the Site or the Challenge Rules, during the Challenge Period, you are not allowed to privately share source or executable code developed in connection with or based upon the Challenge Data or other source or executable code relevant to the Challenge (“Challenge Code”). This prohibition includes sharing Challenge Code between separate Teams. Any such private sharing of Challenge Code is a breach of the Challenge Rules and may result in disqualification.

B. Public Code Sharing. You are permitted to publicly share Challenge Code, provided that such public sharing does not violate the intellectual property rights of any third party. If you do choose to share Challenge Code or other such code, you are required to share it on the hosting platform, on the discussion forum, or via notebooks associated specifically with the Challenge for the benefit of all Participants. By so sharing, you are deemed to have licensed the shared code under the License.

7. DETERMINING WINNERS.

Each Submission will be scored and ranked by the evaluation metric stated on the Site. During the Challenge Period, the current ranking will be visible on the Site’s public leaderboard. The potential winner(s) are determined solely by the leaderboard ranking on the private leaderboard, subject to compliance with the Challenge Rules. The public leaderboard will be based on the public test set, and the private leaderboard will be based on the private test set.

In the event of a tie, the Submission that was entered first to the Challenge will be the winner. In the event a potential winner is disqualified for any reason, the Submission that received the next highest score rank will be chosen as the potential winner.

8. NOTIFICATION OF WINNERS; DISQUALIFICATION.

The potential winner(s) will be notified by email.
Challenge Host reserves the right to disqualify any Participant from the Challenge if the Challenge Host reasonably believes that the Participant has attempted to undermine the legitimate operation of the Challenge by cheating, deception, or other unfair playing practices or abuses, threatens or harasses any other Participants or Challenge Host.
A disqualified Participant may be removed from the Challenge leaderboard, at Challenge Host’s sole discretion.
The final leaderboard list will be publicly displayed at https://the-learning-agency.com/. Determinations of Challenge Host as to the Challenge and the Challenge Rules are final and binding.

9. PARTICIPANTS’ OBLIGATIONS.

As a condition to being awarded a Prize, a Team Lead must fulfill all of the following obligations:

A. Place in one of Prize winning positions OR fulfill one of the other monetary reward requirements (such as submitting a solution that has a score that meets the competition ballpark score);

B. Deliver to the Challenge Host the final model’s software code as used to generate the Submission, and all associated documentation. The delivered software code should follow the documentation guidelines, provided in Appendix A to the Challenge Rules, must be capable of generating the Submission, and must contain a description of resources required to build and/or run the executable code successfully. To the extent that the final model’s software code includes generally commercially available software that is not owned by you, but that can be procured by the Challenge Host without undue expense, then instead of delivering the code for that software to the Challenge Host, you must identify that software, the method for procuring it, and any parameters or other information necessary to replicate the Submission;

C. Grant to the Challenge Host the License to the Submission stated in these Challenge Rules, and represent that you have the unrestricted right to grant that License;

D. Sign and return all prize acceptance documents as may be required by Challenge Host including without limitation:
(i) eligibility certifications; (ii) licenses, releases and other agreements required under the Rules; and (iii) U.S. tax forms (such as IRS Form W-9 if U.S. resident, IRS Form W-8BEN if foreign resident, or future equivalents).

10. PRIZES AND HONORARIUMS.

Prizes are as described on the Site and shall be awarded to each Team Lead.

All Prizes will be disseminated upon completion of the Challenge and review of required Participant obligations. Prizes are subject to Challenge Host’s review and verification of the Participant’s eligibility and compliance with the Challenge Rules, and the compliance of the Submissions with the Submissions Requirements, all in Challenge Host’s sole determination. In the event that the Participant or the Submission demonstrates non-compliance with these Challenge Rules, Challenge Host may at its discretion take either of the following actions: (i) disqualify the Participant and Submission(s); or (ii) require the Participant to remediate, within one week after notice, all issues identified in the Submission(s) (including, without limitation, the resolution of license conflicts, the fulfillment of all obligations required by software licenses, and the removal of any software that violates the software restrictions).

The distribution of the Prize among Team Members, if any, is at the sole discretion of the Team Lead. Team Leads may choose to share or not share the prize with Team Members. The Competition Host is not responsible or liable for any direct payment to Team Members.

A potential Challenge winner may decline to be chosen as a Challenge winner.

You are not eligible to receive any Prize if you do not meet the Eligibility requirements in Section 2 (Eligibility) of these General Challenge Rules.

Additionally, each Team Lead will be awarded an honorarium of $500 at the start of the competition to use for competition resources.

The distribution of the honorarium among Team Members, if any, is at the sole discretion of the Team Lead. Team Leads may choose to share or not share the honorarium with Team Members. The Competition Host is not responsible or liable for any direct payment to Team Members.

Participants may decline the honorarium if they so choose by emailing the Competition Host.

11. TAXES.

ALL TAXES IMPOSED ON PRIZES AND HONORARIUMS ARE THE SOLE RESPONSIBILITY OF THE TEAM LEADS. Payments to Team Leads are subject to the express requirement that they submit all documentation requested by Challenge Host for compliance with applicable state, federal, local and foreign (including provincial) tax reporting and withholding requirements. Prizes and honorariums will be net of any taxes that Challenge Host is required by law to withhold. If a Team Lead fails to provide any required documentation or comply with applicable laws, the Prize or honorarium may be forfeited. Any Team Leads who are U.S. residents will receive an IRS Form-1099 in the amount of their Prize.

12. GENERAL CONDITIONS.

All federal, state, provincial and local laws and regulations apply.

13. PUBLICITY.

You agree that Challenge Host and its affiliates may use your name and likeness for advertising and promotional purposes without additional compensation, unless prohibited by law.

14. PRIVACY.

You acknowledge and agree that Challenge Host may collect, store, share and otherwise use personally identifiable information provided by you during the registration process and throughout the Challenge, including but not limited to, name, mailing address, phone number, and email address (“Personal Information”).

As a controller of such Personal Information, Challenge Host agrees to comply with all U.S. and foreign data protection obligations with regard to your Personal Information. Your Personal Information will be transferred to Challenge Host in the country specified in the Challenge Host Address listed above, which may be a country outside the country of your residence. Such country may not have privacy laws and regulations similar to those of the country of your residence.

15. WARRANTY, INDEMNITY AND RELEASE.

You warrant that your Submission is your own original work and, as such, you are the sole and exclusive owner and rights holder of the Submission, and you have the right to make the Submission and grant all required licenses including without limitation the License. You agree not to make any Submission that: (i) infringes any third party proprietary rights, intellectual property rights, industrial property rights, personal or moral rights or any other rights, including without limitation, copyright, trademark, patent, trade secret, privacy, publicity or confidentiality obligations, or defames any person; or (ii) otherwise violates any applicable U.S. or foreign state or federal law.

To the maximum extent permitted by law, you indemnify and agree to keep indemnified Challenge Host, its affiliates, and its and their directors, officers, employees and agents (collectively, the “Challenge Entities”) at all times from and against any liability, claims, demands, losses, damages, costs and expenses resulting from any of your acts, defaults or omissions and/or a breach of any warranty set forth herein. To the maximum extent permitted by law, you agree to defend, indemnify and hold harmless the Challenge Entities from and against any and all claims, actions, suits or proceedings, as well as any and all losses, liabilities, damages, costs and expenses (including reasonable attorneys fees) arising out of or accruing from: (a) your Submission or other material uploaded or otherwise provided by you that infringes any third party proprietary rights, intellectual property rights, industrial property rights, personal or moral rights or any other rights, including without limitation, copyright, trademark, patent, trade secret, privacy, publicity or confidentiality obligations, or defames any person; (b) any misrepresentation made by you in connection with the Challenge; (c) any non-compliance by you with the Challenge Rules or any applicable U.S. or foreign state or federal law; (d) claims brought by persons or entities other than the parties to the Challenge Rules arising from or related to your involvement with the Challenge; and (e) your acceptance, possession, misuse or use of any Prize, or your participation in the Challenge and any Challenge-related activity.

You hereby release Challenge Entities from any liability associated with: (a) any malfunction or other problem with the Site; (b) any error in the collection, processing, or retention of any Submission; or (c) any typographical or other error in the printing, offering or announcement of winners.

16. INTERNET.

Challenge Entities are not responsible for any malfunction of the Site or any late, lost, damaged, misdirected, incomplete, illegible, undeliverable, or destroyed Submissions or entry materials due to system errors, failed, incomplete or garbled computer or other telecommunication transmission malfunctions, hardware or software failures of any kind, lost or unavailable network connections, typographical or system/human errors and failures, technical malfunction(s) of any telephone network or lines, cable connections, satellite transmissions, servers or providers, or computer equipment, traffic congestion on the Internet or at the Site, or any combination thereof, which may limit a participant’s ability to participate.

17. RIGHT TO CANCEL, MODIFY OR DISQUALIFY.

If for any reason the Challenge is not capable of running as planned, Challenge Host reserves the right to cancel, terminate, modify or suspend the Challenge.

18. NOT AN OFFER OR CONTRACT OF EMPLOYMENT.

Under no circumstances will the entry of a Submission, the awarding of a honorarium or a Prize, or anything in the Challenge Rules be construed as an offer or contract of employment with Challenge Host or any of the Challenge Entities. You acknowledge that you have submitted your Submission voluntarily and not in confidence or in trust. You acknowledge that no confidential, fiduciary, agency, employment or other similar relationship is created between you and Challenge Host or any of the Challenge Entities by your acceptance of these Rules or your entry of your Submission.

19. GOVERNING LAW.

Unless otherwise provided in these Rules, all claims arising out of or relating to these Rules will be governed by Maryland law, excluding its conflict of laws rules, and will be litigated exclusively in the Federal or State courts of Montgomery County, Maryland, USA. The parties consent to personal jurisdiction in those courts. If any provision of these Rules is held to be invalid or unenforceable, all remaining provisions of the Rules will remain in full force and effect.

APPENDIX A. FINAL MODEL DOCUMENTATION

Below are the standard expectations for FINAL Model Documentation. There are two standard components:

1. Model Summary: Requirements detailed in Section A of this Appendix A

2. Submission Model: Requirements detailed in Section B of this Appendix A

A. MODEL SUMMARY

General Guidelines:

Documentation should be in Word or PDF format. It should be in English.

The below should be considered helpful guidance. You can ignore any questions that are not relevant to your Team. You should also add any useful details that are not covered by the questions.

A1. Background on you/your team

[For each team member]

A2. Background on you/your team

If part of a Team, please answer these questions for each Team Member. For larger Teams (3+ Team Members), please give shorter responses.

What is your academic/professional background?
Did you have any prior experience that helped you succeed in this Challenge?
What made you decide to enter this Challenge?
How much time did you spend on the Challenge?
If part of a Team, how did you decide to team up?
If you competed as part of a Team, who did what?

A3. Summary

In four to six sentences, summarize the most important aspects of your model and analysis, such as:

A4. Features Selection / Engineering

What were the most important features? (if applicable)
How did you select features? (if applicable)
Did you make any important feature transformations? (if applicable)
Did you find any interesting interactions between features? (if applicable)
Did you use external data?

A5. Training Method(s)

A6. Interesting findings

A7. Simple Features and Methods

A8. Model Execution Time

How long does it take to train your model?
How long does it take to generate predictions using your model?
How long does it take to train the simplified model (referenced in section A7)?
How long does it take to generate predictions from the simplified model?
What computing resources did you use to get these execution times?

A9. References

Citations to references, websites, blog posts, and external sources of information where appropriate.

B. SUBMISSION MODEL

The final Submission for evaluation must be shared by the Final Submission Deadline and meet the following requirements:

Training notebook. A Jupyter Notebook file that contains the source code for developing the model trained on the Challenge Data’s training set (and External Data if applicable).
Inference notebook. A Jupyter Notebook file that contains the source code for the trained model to generate new predictions on new data.
External Data repository. A folder archive of any External Data used to train the model. Note these files will be used to determine the Team’s final score and leaderboard ranking..
Overall, the notebook should be well organized, documented, and executable.