Duckietown Challenges Home Challenges Submissions

Evaluator 233

ID233
evaluatoridsc-rudolf-14652
ownerAndrea Censi 🇨🇭
machineidsc-rudolf
processidsc-rudolf-14652
versionDC:3.1.52;DCR:3.2.12;DTS:3.0.30
first heard
last heard
statusinactive
# evaluating
# success10 2901
# timeout6 2962
# failed3 2896
# error8 2917
# aborted
# host-error
arm
x86_64
Mac
gpu available
Number of processors
Processor frequency (MHz)
Free % of processors
RAM total (MB)
RAM free (MB)
Disk (MB)
Disk available (MB)
Docker Hub
P1
P2
Cloud simulations
PI Camera
# Duckiebots
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
3036330Benjamin Ramtoula 🇨🇦PyTorch DDPG templateaido1_LF1_r3-v3step1-simulationtimeoutnoidsc-rudolf-146520:11:29
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
3032343Bhairav MehtaPyTorch DDPG templateaido1_LF1_r3-v3step1-simulationerrornoidsc-rudolf-146520:00:26
Uncaught exception w [...]
Uncaught exception while running Docker Compose:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 526, in run_one
    cr = read_challenge_results(wd)
  File "/project/src/duckietown-challenges/src/duckietown_challenges/challenge_results.py", line 63, in read_challenge_results
    raise Exception(msg)
Exception: File u'/tmp/duckietown/DT18/evaluator/executions/aido1_LF1_r3-v3/submission343/step1-simulation-idsc-rudolf-14652-job3032/challenge-results/challenge_results.yaml' does not exist.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
3030345Bhairav MehtaPyTorch DDPG templateaido1_LF1_r3-v3step1-simulationtimeoutnoidsc-rudolf-146520:11:04
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
3028347Anton Mashikhin 🇷🇺AI DL RL MML XXXL 2k18 yooaido1_LF1_r3-v3step1-simulationtimeoutnoidsc-rudolf-146520:11:02
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
3023350Manfred DiazTensorflow templateaido1_LF1_r3-v3step1-simulationtimeoutnoidsc-rudolf-146520:11:13
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
3022350Manfred DiazTensorflow templateaido1_LF1_r3-v3step1-simulationtimeoutnoidsc-rudolf-146520:11:01
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
3007306Andrea Censi 🇨🇭Random executionaido1_LF1_r3-v3step1-simulationfailednoidsc-rudolf-146520:02:13
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 486, in wrap_evaluator
    raise InvalidSubmission(out[SPECIAL_INVALID_SUBMISSION])
InvalidSubmission: Invalid solution:
Traceback (most recent call last):
  File "/notebooks/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 586, in wrap_solution
    raise InvalidSubmission(msg)
InvalidSubmission: Uncaught exception in solution:
Traceback (most recent call last):
  File "/notebooks/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 581, in wrap_solution
    solution.run(cis)
  File "solution.py", line 64, in run
    solve(gym_environment, cis)  # let's try to solve the challenge, exciting ah?
  File "solution.py", line 39, in solve
    observation, reward, done, info = env.step(action)
  File "/usr/local/lib/python2.7/dist-packages/gym/wrappers/time_limit.py", line 31, in step
    observation, reward, done, info = self.env.step(action)
  File "/notebooks/src/gym-duckietown-agent/gym_duckietown_agent/envs/simplesimagent_env.py", line 107, in step
    obs, rew, done, misc = self.sim.step(action, with_observation=True)
  File "/notebooks/src/duckietown-slimremote/duckietown_slimremote/pc/robot.py", line 55, in step
    return self._failsafe_observe(msg)
  File "/notebooks/src/duckietown-slimremote/duckietown_slimremote/pc/robot.py", line 86, in _failsafe_observe
    raise Exception(msg)
Exception: Giving up to connect to the gym duckietown server at host: evaluator


Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2999335Andrea Censi 🇨🇭Tensorflow templateaido1_LF1_r3-v3step3-videossuccessnoidsc-rudolf-146520:00:56
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
2996335Andrea Censi 🇨🇭Tensorflow templateaido1_LF1_r3-v3step2-scoringsuccessnoidsc-rudolf-146520:00:17
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median7.166666666666651


other stats
episodes
details{"ep000": {"nsteps": 49, "reward": -21.013977104645907, "good_angle": 3.1363606392236787, "survival_time": 1.633333333333335, "traveled_tiles": 3, "valid_direction": 1.2666666666666684}, "ep001": {"nsteps": 122, "reward": -8.268821754410755, "good_angle": 0.5527441703687996, "survival_time": 4.066666666666662, "traveled_tiles": 2, "valid_direction": 0.8999999999999978}, "ep002": {"nsteps": 500, "reward": 0.054930976391846346, "good_angle": 0.4941867446231046, "survival_time": 16.666666666666654, "traveled_tiles": 11, "valid_direction": 1.1333333333333293}, "ep003": {"nsteps": 215, "reward": -4.849887623503631, "good_angle": 5.355869516405512, "survival_time": 7.166666666666651, "traveled_tiles": 5, "valid_direction": 2.76666666666666}, "ep004": {"nsteps": 500, "reward": 0.03539378764200956, "good_angle": 1.3090289511173765, "survival_time": 16.666666666666654, "traveled_tiles": 10, "valid_direction": 1.8999999999999964}}
good_angle_max5.355869516405512
good_angle_mean2.169638004347694
good_angle_median1.3090289511173765
good_angle_min0.4941867446231046
reward_max0.054930976391846346
reward_mean-6.808472343705287
reward_median-4.849887623503631
reward_min-21.013977104645907
survival_time_max16.666666666666654
survival_time_mean9.239999999999991
survival_time_min1.633333333333335
traveled_tiles_max11
traveled_tiles_mean6.2
traveled_tiles_median5
traveled_tiles_min2
valid_direction_max2.76666666666666
valid_direction_mean1.5933333333333306
valid_direction_median1.2666666666666684
valid_direction_min0.8999999999999978
No reset possible
2986335Andrea Censi 🇨🇭Tensorflow templateaido1_LF1_r3-v3step1-simulationsuccessnoidsc-rudolf-146520:02:26
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation-passed1
No reset possible
2984352Anton Mashikhin 🇷🇺AI DL RL MML XXXL 2k18 yooaido1_LF1_r3-v3step1-simulationerrornoidsc-rudolf-146520:10:30
Timeout: Waited 602 [...]
Timeout:

Waited 602.906049967 for container to finish. Giving up. 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2982351Pravish Sainath 🇨🇦Solution templateaido1_luck-v3step1successyesidsc-rudolf-146520:00:52
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
score186.62134721742594


No reset possible
2978349Bhairav MehtaPyTorch DDPG templateaido1_LF1_r3-v3step3-videossuccessnoidsc-rudolf-146520:00:47
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
2973346Bhairav MehtaPyTorch DDPG templateaido1_LF1_r3-v3step3-videossuccessnoidsc-rudolf-146520:00:55
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
2969346Bhairav MehtaPyTorch DDPG templateaido1_LF1_r3-v3step2-scoringsuccessnoidsc-rudolf-146520:00:18
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median2.500000000000001


other stats
episodes
details{"ep000": {"nsteps": 75, "reward": -14.334179728428524, "good_angle": 5.501256831651871, "survival_time": 2.500000000000001, "traveled_tiles": 2, "valid_direction": 1.7666666666666675}, "ep001": {"nsteps": 76, "reward": -14.7360854501787, "good_angle": 1.242759194817416, "survival_time": 2.533333333333334, "traveled_tiles": 1, "valid_direction": 1.800000000000001}, "ep002": {"nsteps": 89, "reward": -12.475425450068512, "good_angle": 1.68633432212303, "survival_time": 2.966666666666666, "traveled_tiles": 2, "valid_direction": 2.066666666666666}, "ep003": {"nsteps": 63, "reward": -17.92272880909935, "good_angle": 0.7645386987529875, "survival_time": 2.1000000000000023, "traveled_tiles": 2, "valid_direction": 1.4333333333333356}, "ep004": {"nsteps": 48, "reward": -23.43614997466405, "good_angle": 0.35779319893925954, "survival_time": 1.6000000000000016, "traveled_tiles": 2, "valid_direction": 0.933333333333335}}
good_angle_max5.501256831651871
good_angle_mean1.9105364492569128
good_angle_median1.242759194817416
good_angle_min0.35779319893925954
reward_max-12.475425450068512
reward_mean-16.580913882487827
reward_median-14.7360854501787
reward_min-23.43614997466405
survival_time_max2.966666666666666
survival_time_mean2.3400000000000007
survival_time_min1.6000000000000016
traveled_tiles_max2
traveled_tiles_mean1.8
traveled_tiles_median2
traveled_tiles_min1
valid_direction_max2.066666666666666
valid_direction_mean1.600000000000001
valid_direction_median1.7666666666666675
valid_direction_min0.933333333333335
No reset possible
2968343Bhairav MehtaPyTorch DDPG templateaido1_LF1_r3-v3step1-simulationerrornoidsc-rudolf-146520:00:25
Uncaught exception w [...]
Uncaught exception while running Docker Compose:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 526, in run_one
    cr = read_challenge_results(wd)
  File "/project/src/duckietown-challenges/src/duckietown_challenges/challenge_results.py", line 63, in read_challenge_results
    raise Exception(msg)
Exception: File u'/tmp/duckietown/DT18/evaluator/executions/aido1_LF1_r3-v3/submission343/step1-simulation-idsc-rudolf-14652-job2968/challenge-results/challenge_results.yaml' does not exist.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2963344Bhairav MehtaPyTorch DDPG templateaido1_LF1_r3-v3step1-simulationerrornoidsc-rudolf-146520:00:25
Uncaught exception w [...]
Uncaught exception while running Docker Compose:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 526, in run_one
    cr = read_challenge_results(wd)
  File "/project/src/duckietown-challenges/src/duckietown_challenges/challenge_results.py", line 63, in read_challenge_results
    raise Exception(msg)
Exception: File u'/tmp/duckietown/DT18/evaluator/executions/aido1_LF1_r3-v3/submission344/step1-simulation-idsc-rudolf-14652-job2963/challenge-results/challenge_results.yaml' does not exist.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2962343Bhairav MehtaPyTorch DDPG templateaido1_LF1_r3-v3step1-simulationtimeoutnoidsc-rudolf-146520:11:02
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2959342Claudio RuchJava templateaido1_amod1-v3step1-simulationerroryesidsc-rudolf-146520:10:28
Timeout: Waited 602 [...]
Timeout:

Waited 602.972043991 for container to finish. Giving up. 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2949194Florian GolemoPyTorch DDPG templateaido1_LF1_r3-v3step1-simulationerrornoidsc-rudolf-146520:10:28
Timeout: Waited 602 [...]
Timeout:

Waited 602.89296484 for container to finish. Giving up. 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2945217Bhairav MehtaROS-based Lane Followingaido1_LF1_r3-v3step1-simulationfailednoidsc-rudolf-146520:00:27
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 486, in wrap_evaluator
    raise InvalidSubmission(out[SPECIAL_INVALID_SUBMISSION])
InvalidSubmission: Invalid solution:
Env Duckietown-Lf-Lfv-Navv-Silent-v1 not found (valid versions include ['Duckietown-Lf-Lfv-Navv-Silent-v0'])
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2926280Anton Mashikhin 🇷🇺AI DL RL MML XXXL 2k18 yooaido1_LF1_r3-v3step1-simulationerrornoidsc-rudolf-146520:10:28
Timeout: Waited 603 [...]
Timeout:

Waited 603.022886038 for container to finish. Giving up. 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2917282Anton Mashikhin 🇷🇺AI DL RL MML XXXL 2k18 yooaido1_LF1_r3-v3step1-simulationerrornoidsc-rudolf-146520:10:28
Timeout: Waited 602 [...]
Timeout:

Waited 602.942661047 for container to finish. Giving up. 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2914334Andrea Censi 🇨🇭Tensorflow templateaido1_LFV_r1-v3step4-vizsuccessnoidsc-rudolf-146520:01:10
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
2913334Andrea Censi 🇨🇭Tensorflow templateaido1_LFV_r1-v3step3-videossuccessnoidsc-rudolf-146520:00:36
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
2901334Andrea Censi 🇨🇭Tensorflow templateaido1_LFV_r1-v3step2-scoringsuccessnoidsc-rudolf-146520:01:32
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median3.766666666666663


other stats
episodes
details{"ep000": {"nsteps": 48, "reward": -21.42554907328061, "good_angle": 2.8207892741930283, "survival_time": 1.6000000000000016, "traveled_tiles": 3, "valid_direction": 1.2000000000000015}, "ep001": {"nsteps": 113, "reward": -9.126021487457034, "good_angle": 0.4336047256271916, "survival_time": 3.766666666666663, "traveled_tiles": 2, "valid_direction": 0.6333333333333311}, "ep002": {"nsteps": 243, "reward": -4.33631397868864, "good_angle": 0.5664875769269541, "survival_time": 8.099999999999982, "traveled_tiles": 5, "valid_direction": 0.5999999999999999}, "ep003": {"nsteps": 41, "reward": -26.565000374142716, "good_angle": 0.35119644914517983, "survival_time": 1.3666666666666676, "traveled_tiles": 1, "valid_direction": 1.0000000000000004}, "ep004": {"nsteps": 144, "reward": -7.17783817061637, "good_angle": 0.320953404745754, "survival_time": 4.799999999999993, "traveled_tiles": 4, "valid_direction": 1.0333333333333328}}
good_angle_max2.8207892741930283
good_angle_mean0.8986062861276215
good_angle_median0.4336047256271916
good_angle_min0.320953404745754
reward_max-4.33631397868864
reward_mean-13.726144616837072
reward_median-9.126021487457034
reward_min-26.565000374142716
survival_time_max8.099999999999982
survival_time_mean3.9266666666666614
survival_time_min1.3666666666666676
traveled_tiles_max5
traveled_tiles_mean3
traveled_tiles_median3
traveled_tiles_min1
valid_direction_max1.2000000000000015
valid_direction_mean0.8933333333333332
valid_direction_median1.0000000000000004
valid_direction_min0.5999999999999999
No reset possible
2896332Andrea Censi 🇨🇭Random executionaido1_LFV_r1-v3step1-simulationfailednoidsc-rudolf-146520:00:40
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 486, in wrap_evaluator
    raise InvalidSubmission(out[SPECIAL_INVALID_SUBMISSION])
InvalidSubmission: Invalid solution:
Traceback (most recent call last):
  File "/notebooks/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 586, in wrap_solution
    raise InvalidSubmission(msg)
InvalidSubmission: Uncaught exception in solution:
Traceback (most recent call last):
  File "/notebooks/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 581, in wrap_solution
    solution.run(cis)
  File "solution.py", line 73, in run
    solve(gym_environment, cis)  # let's try to solve the challenge, exciting ah?
  File "solution.py", line 46, in solve
    observation, reward, done, info = env.step(action)
  File "/usr/local/lib/python2.7/dist-packages/gym/wrappers/time_limit.py", line 31, in step
    observation, reward, done, info = self.env.step(action)
  File "/notebooks/src/gym-duckietown-agent/gym_duckietown_agent/envs/simplesimagent_env.py", line 109, in step
    obs, rew, done, misc = self.sim.step(action, with_observation=True)
  File "/notebooks/src/duckietown-slimremote/duckietown_slimremote/pc/robot.py", line 55, in step
    return self._failsafe_observe(msg)
  File "/notebooks/src/duckietown-slimremote/duckietown_slimremote/pc/robot.py", line 86, in _failsafe_observe
    raise Exception(msg)
Exception: Giving up to connect to the gym duckietown server at host: evaluator


Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible