gonito/templates/challenge-how-to.hamlet

218 lines
8.0 KiB
Plaintext
Raw Normal View History

2015-11-11 22:10:41 +01:00
<h1>Challenge How To
2017-09-28 20:44:44 +02:00
<h2>Install GEval (recommended, but not obligatory)
2015-11-11 22:10:41 +01:00
<p>In order to evaluate your work locally, you need the GEval tool. There are two ways to get it.
2018-04-09 11:38:34 +02:00
<h4>option A — just download the GEval fully static binary (simpler if you have a standard 64-bit Linux)
<pre>
wget <a href="https://gonito.net/get/bin/geval">https://gonito.net/get/bin/geval</a>
chmod u+x geval
./geval --help
<h4>option B — install GEval with Haskell Stack (needs some time and a couple of gigabytes of hard disk space)
<p>First, install <a href="https://docs.haskellstack.org/en/stable/README/">Haskell Stack</a>:
2017-09-23 15:45:14 +02:00
<pre>
curl -sSL https://get.haskellstack.org/ | sh
<p>and then install GEval itself:
2015-11-11 22:10:41 +01:00
<pre>
git clone git://gonito.net/geval
cd geval
stack setup
stack install
2018-04-09 11:38:34 +02:00
~/.local/bin/geval --help
2015-11-11 22:10:41 +01:00
<p>For troubleshooting, see <a href="https://gitlab.com/filipg/geval#installing">GEval readme</a>.
2017-09-28 20:44:44 +02:00
<p>(GEval will make your life easier, but is not obligatory, so if you have any trouble with installing/running GEval, you can skip this step, for the time being.)
2017-09-23 15:45:14 +02:00
2015-11-11 22:10:41 +01:00
<h2>Get your repo
$case (appRepoScheme settings)
$of SelfHosted
$if (not isIDSet || not isSSHUploaded) && isNothing mAltRepoScheme
<p>You need to
$if not isIDSet
\ set up your ID
$if not isSSHUploaded
\ and
$if not isSSHUploaded
\ upload your SSH public key
\ (see <a href="@{YourAccountR}">your account</a>) if you want to have a repo hosted on this instance, then:
2015-11-11 22:10:41 +01:00
$maybe altRepoScheme <- mAltRepoScheme
2021-08-21 12:11:40 +02:00
<p>(Depending on your setup, you might need to create the repo at your git server first; remember to initiate it without any commits!)
<pre>
git clone #{altRepoScheme}#{challengeName challenge}
$nothing
<pre>
git clone --single-branch #{appRepoHost settings}#{shownId}/#{challengeName challenge}
# --single-branch was used to speed up the process, but now we need to fix remote branches
git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"
2017-10-03 22:19:18 +02:00
<p>(Warning about empty repository is expected, don't worry about it.)
$maybe altRepoScheme <- mAltRepoScheme
^{externalRepoInfo settings}
2017-10-03 22:19:18 +02:00
<pre>
cd #{challengeName challenge}
2020-02-22 19:12:18 +01:00
git pull #{repoUrl repo}
$if isNothing mAltRepoScheme
<h4>Using an external repository
2019-11-25 23:09:09 +01:00
<p>Alternatively, you can use any other Git repo, e.g. GitLab, GitHub or your own repo.
^{externalRepoInfo settings}
$of NoInternalGitServer
$maybe altRepoScheme <- mAltRepoScheme
<p>Make sure the repo #{altRepoScheme}#{challengeName challenge} exists. Note that it should be created as empty (do <em>not</em> create any default `README.md` file).
$nothing
<p>You need to have a repo at some external Git server (e.g. GitHub, GitLab or your own git server) for storing your solution.
<p>Create a repo at your server. It should be created as empty (do **not** create any default `README.md` file).
<p>The repo should contain <tt>#{challengeName challenge}</tt> as part of its URL (preferably it should be the last part of the URL)
^{externalRepoInfo settings}
<pre>
$maybe altRepoScheme <- mAltRepoScheme
git clone #{altRepoScheme}#{challengeName challenge}
$nothing
git clone URL_TO_YOUR_REPO
<pre>
cd #{challengeName challenge}
git pull #{repoUrl repo}
2019-11-25 23:09:09 +01:00
$of Branches
<p>Clone the repo:
<pre>
2018-10-15 14:19:20 +02:00
git clone --single-branch #{repoUrl repo}
# --single-branch was used to speed up the process, but now we need to fix remote branches
git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"
2015-11-11 22:10:41 +01:00
$maybe gitAnnexRemote <- (repoGitAnnexRemote repo)
<pre>
git annex init
git annex initremote storage #{gitAnnexRemote}
git annex get --from storage
2015-11-11 22:10:41 +01:00
<h2>Work on your solution
<p>You need to generate your solution for the test set as <tt>test-A/out.tsv</tt>. It is also recommended to generate the output for the dev set (<tt>dev-0/out.tsv</tt>).
2015-11-11 22:10:41 +01:00
<p>You can evaluate results for the dev set locally:
<pre>
cd #{challengeName challenge}
geval --test-name dev-0
2017-09-28 16:51:10 +02:00
<h2>Push your solution to the git repo
2015-11-11 22:10:41 +01:00
2017-09-28 16:51:10 +02:00
<p>Commit and push <tt>out.tsv</tt> files to your repo. It is also recommended to push your source codes along with <tt>out.tsv</tt> files.
2015-11-11 22:10:41 +01:00
<pre>
cd #{challengeName challenge}
git checkout -b #{myBranch} # switch to some other branch
git add foo.py build.sh # add your source codes
git add gonito.yaml # it's a good practice to add metadata file, see below
2018-07-28 21:36:45 +02:00
$if isJust (repoGitAnnexRemote repo)
<pre>
# if your output files are large or if they should not pushed to the regular repo (e.g. contain sensitive information):
git annex add dev-0/out.tsv test-A/out.tsv
git annex copy --to storage
# otherwise (e.g. they are just class labels), add the output files in a regular manner:
git add dev-0/out.tsv test-A/out.tsv
2018-07-28 21:36:45 +02:00
$else
<pre>
git add dev-0/out.tsv test-A/out.tsv # add your output files
<pre>
2015-11-11 22:10:41 +01:00
git commit -m 'my brilliant solution'
$case (appRepoScheme settings)
$of SelfHosted
<pre>
git push origin #{myBranch}
<h2>Submit your solution to Gonito
<h3>Repos hosted on this instance
2018-09-01 11:06:42 +02:00
<p>If you use a repo hosted here, a submission and evaluation is triggered automatically. You'll see the evaluation results in your console while pushing.
$of _
<pre>
git push origin #{myBranch}
<h2>Submit your solution to Gonito
2015-11-11 22:10:41 +01:00
<h3>Integration with external repos
2017-09-28 16:51:10 +02:00
2018-09-01 11:06:42 +02:00
<p>If you use an external repo (e.g. at your own of Gitolite or at GitLab/GitHub), you can configure a webhook.
2017-09-28 16:51:10 +02:00
$maybe token <- mToken
<p>Your webook is:
<pre>
wget --timeout=0 --quiet -O - '#{appRoot settings}/trigger-remotely' --post-data 'token=#{token}&branch=#{myBranch}&challenge=#{challengeName challenge}&url=#{urlToYourRepo}'
2020-02-21 22:56:39 +01:00
<p>or you can just use the following URL:
<pre>
#{appRoot settings}/trigger-remotely-simple/#{token}/#{challengeName challenge}/#{urlToYourRepo}/#{myBranch}
2017-09-28 16:51:10 +02:00
<h3>Manual submission
<p>In case other methods fail, you can submit your solution manually — go to the <a href="@{ChallengeSubmissionR $ challengeName challenge}">submit form</a>.
<h2>Submission metadata
2018-11-03 18:02:27 +01:00
<p>Gonito can take the metadata (description, tags, parameters) of a submission from a number of sources (in order of precedence):
<ol>
2018-11-03 18:02:27 +01:00
<li>the YAML files specified in the <tt>param-files</tt> field of the <tt>gonito.yaml</tt> file
<ul>
<li>only applicable for parameters,
<li><tt>gonito.yaml</tt> will be always skipped even if matches a mask given in the <tt>param-files</tt> field,
<li>parameters blacklisted in the <tt>unwanted-params</tt> field of the <tt>gonito.yaml</tt> file will be discarded;
<li><tt>gonito.yaml</tt> file committed to the repository
<ul>
<li>description given in the <tt>description</tt> field,
<li>tags given in <tt>tags</tt> field,
<li>parameters given in <tt>params</tt> field,
<li>names of output files (only for parameters)
2018-11-03 18:02:27 +01:00
<ul>
<li>e.g. if the output file is <tt>out-epochs=10,learning-rate=0.01.tsv</tt>, then parameters <i>epochs</i>=10 and <i>learning-rare</i>=0.01 will be extracted;
<li>submission form (when submitting manually);
<li>git commit message
<ul>
<li>description taken from the first paragraph
<li>tags taken from a line starting with <tt>tags:</tt>
2021-07-28 21:00:02 +02:00
<p>Here is an example of <tt>gonito.yaml</tt>, in which all metadata could be given (note that you can also add links to external resources using the `links` section):
2018-11-03 18:02:27 +01:00
<pre>
description: This my brilliant solution
tags:
\ - neural-network
\ - left-to-right
params:
\ epochs: 10
\ learning-rate: 0.01
unwanted-params:
\ - model-file
\ - vocab-file
param-files:
\ - "*.yaml"
\ - config/*.yaml
2021-07-28 21:00:02 +02:00
links:
\ - title: "Some external link"
\ url: "https://example.com/foo-bar-baz-123"
\ - title: "Yet another link"
\ url: "https://example.org/xyz"
\ - url: "https://example.net/bare-link-without-text"
<p>It might seem a little bit complicated, but you could simply use the method which is the most convenient for you.