Add challenge configuration information

.gitignore

@@ -0,0 +1,2 @@
+challenge_config.zip
+evaluation_script.zip

README.md

@@ -0,0 +1,40 @@
+## EvalAI Starter
+
+If you are looking for a simple challenge configuration that you can replicate to create a challenge on EvalAI, then you are at the right place. Follow the instructions given below to get started.
+
+## Directory Structure
+
+```
+.
+├── README.md
+├── annotations                                 # Contains the annotations for Dataset splits
+│   ├── test_annotations_devsplit.json          # Annotations of dev split
+│   └── test_annotations_testsplit.json         # Annotations for test split
+├── challenge_config.yaml                       # Configuration file to define challenge setup
+├── evaluation_script                           # Contains the evaluation script
+│   ├── __init__.py                             # Imports the modules that involve annotations loading etc
+│   └── main.py                                 # Contains the main `evaluate()` method
+├── logo.jpg                                    # Logo image of the challenge
+├── run.sh                                      # Script to create the challenge configuration zip to be uploaded on EvalAI website
+└── templates									# Contains challenge related HTML templates
+    ├── challenge_phase_1_description.html      # Challenge Phase 1 description template
+    ├── challenge_phase_2_description.html      # Challenge Phase 2 description template
+    ├── description.html                        # Challenge description template
+    ├── evaluation_details.html                 # Contains description about how submissions will be evalauted for each challenge phase
+    ├── submission_guidelines.html              # Contains information about how to make submissions to the challenge
+    └── terms_and_conditions.html               # Contains terms and conditions related to the challenge
+```
+
+### Steps involved
+
+1. Fork this repository.
+
+2. Read [EvalAI challenge creation documentation](http://evalai.readthedocs.io/en/latest/challenge_creation.html#challenge-creation-using-zip-configuration) to know more about how you want to structure your challenge. Once you are ready, start making changes in the yaml file, HTML templates, evaluation script according to your need. 
+
+3. Once you are done making changes, run the command `./run.sh` to generate the `challenge_config.zip`
+
+4. Upload the `challenge_config.zip` on [EvalAI](https://evalai.cloudcv.org) to create a challenge on EvalAI. Challenge will be available publicly once EvalAI Admin approves the challenge. 
+
+### Facing problems in creating a challenge?
+
+Please feel free to open issues on our [Github Repository](https://github.com/Cloud-CV/EvalAI-Starter/issues) or contact us at [email protected] if you have issues.

annotations/test_annotations_devsplit.json

@@ -0,0 +1,3 @@
+{
+	"foo": "bar"
+}

annotations/test_annotations_testsplit.json

@@ -0,0 +1,3 @@
+{
+	"foo": "bar"
+}

challenge_config.yaml

@@ -0,0 +1,63 @@
+# If you are not sure what all these fields mean, please refer our documentation here:
+# http://evalai.readthedocs.io/en/latest/challenge_creation.html#challenge-creation-using-zip-configuration
+title: Random Number Generator Challenge
+short_description: Random number generation challenge for each submission
+description: templates/description.html
+evaluation_details: templates/evaluation_details.html
+terms_and_conditions: templates/terms_and_conditions.html
+image: logo.jpg
+submission_guidelines: templates/submission_guidelines.html
+evaluation_script: evaluation_script.zip
+start_date: 2018-01-19 00:00:00
+end_date: 2099-05-31 23:59:59
+published: True
+
+leaderboard:
+  - id: 1
+    schema: { "labels": ["Metric1", "Metric2", "Metric3", "Total"], "default_order_by": "Total" }
+
+challenge_phases:
+  - id: 1
+    name: Dev Phase
+    description: templates/challenge_phase_1_description.html
+    leaderboard_public: False
+    is_public: True
+    start_date: 2018-01-19 00:00:00
+    end_date: 2099-04-25 23:59:59
+    test_annotation_file: annotations/test_annotations_devsplit.json
+    codename: dev
+    max_submissions_per_day: 5
+    max_submissions: 50
+  - id: 2
+    name: Test Phase
+    description: templates/challenge_phase_2_description.html
+    leaderboard_public: True
+    is_public: True
+    start_date: 2018-01-19 00:00:00
+    end_date: 2018-05-24 23:59:59
+    test_annotation_file: annotations/test_annotations_testsplit.json
+    codename: test
+    max_submissions_per_day: 5
+    max_submissions: 50
+
+dataset_splits:
+  - id: 1
+    name: Train Split
+    codename: train_split
+  - id: 2
+    name: Test Split
+    codename: test_split
+
+challenge_phase_splits:
+  - challenge_phase_id: 1
+    leaderboard_id: 1
+    dataset_split_id: 1
+    visibility: 1
+  - challenge_phase_id: 2
+    leaderboard_id: 1
+    dataset_split_id: 1
+    visibility: 3
+  - challenge_phase_id: 2
+    leaderboard_id: 1
+    dataset_split_id: 2
+    visibility: 1

evaluation_script/__init__.py

@@ -0,0 +1 @@
+from .main import evaluate

evaluation_script/main.py

@@ -0,0 +1,81 @@
+import random
+
+
+def evaluate(test_annotation_file, user_submission_file, phase_codename, **kwargs):
+    """
+    Evaluates the submission for a particular challenge phase adn returns score
+    Arguments:
+
+        `test_annotations_file`: Path to test_annotation_file on the server
+        `user_submission_file`: Path to file submitted by the user
+        `phase_codename`: Phase to which submission is made
+
+        `**kwargs`: keyword arguments that contains additional submission
+        metadata that challenge hosts can use to send slack notification.
+        You can access the submission metadata
+        with kwargs['submission_metadata']
+
+        Example: A sample submission metadata can be accessed like this:
+        >>> print(kwargs['submission_metadata'])
+        {
+            'status': u'running',
+            'when_made_public': None,
+            'participant_team': 5,
+            'input_file': 'https://abc.xyz/path/to/submission/file.json',
+            execution_time': u'123',
+            'publication_url': u'ABC',
+            'challenge_phase': 1,
+            'created_by': u'ABC',
+            'stdout_file': 'https://abc.xyz/path/to/stdout/file.json',
+            'method_name': u'Test',
+            'stderr_file': 'https://abc.xyz/path/to/stderr/file.json',
+            'participant_team_name': u'Test Team',
+            'project_url': u'http://foo.bar',
+            'method_description': u'ABC',
+            'is_public': False,
+            'submission_result_file': 'https://abc.xyz/path/result/file.json',
+            'id': 123,
+            'submitted_at': u'2017-03-20T19:22:03.880652Z'
+        }
+    """
+
+    print("Starting Evaluation.....")
+    print("Submission related metadata:")
+    print(kwargs['submission_metadata'])
+
+    output = {}
+    if phase_codename == "dev":
+        print("Evaluating for Dev Phase")
+        output['result'] = [
+            {
+                'train_split': {
+                    'Metric1': random.randint(0, 99),
+                    'Metric2': random.randint(0, 99),
+                    'Metric3': random.randint(0, 99),
+                    'Total': random.randint(0, 99),
+                }
+            },
+        ]
+        print("Completed evaluation for Dev Phase")
+    elif phase_codename == "test":
+        print("Evaluating for Test Phase")
+        output['result'] = [
+            {
+                'train_split': {
+                    'Metric1': random.randint(0, 99),
+                    'Metric2': random.randint(0, 99),
+                    'Metric3': random.randint(0, 99),
+                    'Total': random.randint(0, 99),
+                }
+            },
+            {
+                'test_split': {
+                    'Metric1': random.randint(0, 99),
+                    'Metric2': random.randint(0, 99),
+                    'Metric3': random.randint(0, 99),
+                    'Total': random.randint(0, 99),
+                }
+            }
+        ]
+        print("Completed evaluation for Test Phase")
+    return output

run.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+
+# Remove already existing zip files
+rm evaluation_script.zip
+rm challenge_config.zip
+
+# Create new zip configuration according the updated code
+zip -r -j evaluation_script.zip evaluation_script/*  -x "*.DS_Store"
+zip -r challenge_config.zip *  -x "*.DS_Store" -x "evaluation_script/*" -x "*.git" -x "run.sh"

templates/challenge_phase_1_description.html

@@ -0,0 +1 @@
+<p>"Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo. Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt. Neque porro quisquam est, qui dolorem ipsum quia dolor sit amet, consectetur, adipisci velit, sed quia non numquam eius modi tempora incidunt ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit qui in ea voluptate velit esse quam nihil molestiae consequatur, vel illum qui dolorem eum fugiat quo voluptas nulla pariatur?"</p>

templates/challenge_phase_2_description.html

@@ -0,0 +1 @@
+"Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo. Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt. Neque porro quisquam est, qui dolorem ipsum quia dolor sit amet, consectetur, adipisci velit, sed quia non numquam eius modi tempora incidunt ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit qui in ea voluptate velit esse quam nihil molestiae consequatur, vel illum qui dolorem eum fugiat quo voluptas nulla pariatur?"

templates/description.html

@@ -0,0 +1,3 @@
+<p>"Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo. Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt. Neque porro quisquam est, qui dolorem ipsum quia dolor sit amet, consectetur, adipisci velit, sed quia non numquam eius modi tempora incidunt ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit qui in ea voluptate velit esse quam nihil molestiae consequatur, vel illum qui dolorem eum fugiat quo voluptas nulla pariatur?"</p>
+
+<p>"Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo. Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt. Neque porro quisquam est, qui dolorem ipsum quia dolor sit amet, consectetur, adipisci velit, sed quia non numquam eius modi tempora incidunt ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit qui in ea voluptate velit esse quam nihil molestiae consequatur, vel illum qui dolorem eum fugiat quo voluptas nulla pariatur?"</p>

templates/evaluation_details.html

@@ -0,0 +1 @@
+<p>"Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum."</p>

templates/submission_guidelines.html

@@ -0,0 +1 @@
+<p>Submit any blank file here to see a random number generated for your submission. If you get lucky, you might reach the top of the leaderboard.</p>

templates/terms_and_conditions.html

@@ -0,0 +1 @@
+<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p>