Duckietown Challenges Home Challenges Submissions

Evaluator 2186

ID2186
evaluatormont01-6ec4790ce633-1
ownerI don't have one 😀
machineaido-ms-7b09
processmont01-6ec4790ce633-1
versiond-c:6.1.20;d-c-r:6.0.49
first heard
last heard
statusinactive
# evaluating
# success1 37998
# timeout
# failed1 38005
# error1 38011
# aborted
# host-error4 38107
arm
x86_64
Mac
gpu available
Number of processors
Processor frequency (MHz)
Free % of processors
RAM total (MB)
RAM free (MB)
Disk (MB)
Disk available (MB)
Docker Hub
P1
P2
Cloud simulations
PI Camera
# Duckiebots
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
381249370Yishu Malhotra 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simhost-errornomont01-6ec4790ce633-10:00:00
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner_daffy-6.0.49-py3.8.egg/duckietown_challenges_runner/runner.py", line 654, in get_cr
    os.makedirs(wd)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/media/sdb/duckietown-tmp-evals/production/aido5-LF-sim-validation'
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
381199371Yishu Malhotra 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simhost-errornomont01-6ec4790ce633-10:00:00
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner_daffy-6.0.49-py3.8.egg/duckietown_challenges_runner/runner.py", line 654, in get_cr
    os.makedirs(wd)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/media/sdb/duckietown-tmp-evals/production/aido5-LF-sim-validation'
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
381119367Jean-Sébastien Grondin 🇨🇦sim-exercise-1aido5-LF-sim-validationLFv-simhost-errornomont01-6ec4790ce633-10:00:00
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner_daffy-6.0.49-py3.8.egg/duckietown_challenges_runner/runner.py", line 654, in get_cr
    os.makedirs(wd)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/media/sdb/duckietown-tmp-evals/production/aido5-LF-sim-validation'
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
381079369Yishu Malhotra 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simhost-errornomont01-6ec4790ce633-10:00:00
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner_daffy-6.0.49-py3.8.egg/duckietown_challenges_runner/runner.py", line 654, in get_cr
    os.makedirs(wd)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/media/sdb/duckietown-tmp-evals/production/aido5-LF-sim-validation'
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
380119970Bea Baselines 🐤template-pytorchaido5-LFV-sim-validationLFVv-simerrornomont01-6ec4790ce633-11:00:27
Waited 3600 seconds [...]
Waited 3600 seconds for container to finish. Giving up.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
3800510796Ayman Shams 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simfailednomont01-6ec4790ce633-10:03:31
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 216, in read_one
    msgs = read_reply(self.fpout, timeout=timeout, waiting_for=waiting_for, nickname=self.nickname,)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 299, in read_reply
    others = read_until_over(fpout, timeout=timeout, nickname=nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 337, in read_until_over
    raise TimeoutError(msg) from None
TimeoutError: Timeout while reading output of node "ego".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "experiment_manager.py", line 512, in run_episode
    r: MsgReceived = await loop.run_in_executor(executor, f)
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 133, in write_topic_and_expect
    ob: MsgReceived = self.read_one(expect_topic=expect, timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 279, in read_one
    raise TimeoutError(msg) from e
TimeoutError: Timeout detected on /fifos/ego-out after 99 messages. Expected topic "commands".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 678, in scoring_context
    yield cie
  File "experiment_manager.py", line 655, in go
    wrap(cie)
  File "experiment_manager.py", line 640, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "experiment_manager.py", line 253, in main
    length_s = await run_episode(
  File "experiment_manager.py", line 516, in run_episode
    raise dc.InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Trouble with communication to the agent.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
3799810811Francois Hebertsim-exercise-1aido5-LF-sim-validationLFv-simsuccessnomont01-6ec4790ce633-10:09:45
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median2.9942131666223073
survival_time_median14.950000000000076
deviation-center-line_median0.573893412155418
in-drivable-lane_median0.025000000000000355


other stats
agent_compute-ego_max0.01939761718114217
agent_compute-ego_mean0.019162761701477897
agent_compute-ego_median0.019231801297929552
agent_compute-ego_min0.01878982702891032
complete-iteration_max0.25570335388183596
complete-iteration_mean0.23919348425335352
complete-iteration_median0.23980354997846817
complete-iteration_min0.22146348317464193
deviation-center-line_max0.8107925822404541
deviation-center-line_mean0.5424726384782681
deviation-center-line_min0.21131114736178225
deviation-heading_max2.857021921225547
deviation-heading_mean1.7213779624411258
deviation-heading_median1.6211450158185423
deviation-heading_min0.7861998969018711
driven_any_max3.0631257213697927
driven_any_mean2.866685684632695
driven_any_median3.063059585455704
driven_any_min2.2774978462495787
driven_lanedir_consec_max3.038398713484349
driven_lanedir_consec_mean2.615626994815017
driven_lanedir_consec_min1.4356829325311051
driven_lanedir_max3.038398713484349
driven_lanedir_mean2.615626994815017
driven_lanedir_median2.9942131666223073
driven_lanedir_min1.4356829325311051
get_duckie_state_max2.774132622612847e-06
get_duckie_state_mean2.654923333062066e-06
get_duckie_state_median2.624193827311198e-06
get_duckie_state_min2.5971730550130207e-06
get_robot_state_max0.016972803274790445
get_robot_state_mean0.016671154300371808
get_robot_state_median0.016644229888916014
get_robot_state_min0.016423354148864745
get_state_dump_max0.014214359124501546
get_state_dump_mean0.01398543828063541
get_state_dump_median0.01395074036386278
get_state_dump_min0.013825913270314537
get_ui_image_max0.04288197093539768
get_ui_image_mean0.041498013867272275
get_ui_image_median0.042045359611511235
get_ui_image_min0.03901936531066894
in-drivable-lane_max3.900000000000043
in-drivable-lane_mean0.9875000000000108
in-drivable-lane_min0.0
per-episodes
details{"ETHZ_autolab_technical_track-sc0-0-ego": {"driven_any": 2.2774978462495787, "get_ui_image": 0.04288197093539768, "step_physics": 0.10489231215582952, "survival_time": 11.250000000000025, "driven_lanedir": 1.4356829325311051, "get_state_dump": 0.013956850899590386, "sim_render-ego": 0.006905891630384657, "get_robot_state": 0.016790316899617512, "get_duckie_state": 2.774132622612847e-06, "in-drivable-lane": 3.900000000000043, "agent_compute-ego": 0.01923523161146376, "deviation-heading": 0.7861998969018711, "complete-iteration": 0.24183522012498643, "set_robot_commands": 0.005540388955010308, "deviation-center-line": 0.21131114736178225, "driven_lanedir_consec": 1.4356829325311051, "sim_compute_sim_state": 0.026884322696261936, "sim_compute_performance-ego": 0.004588925043741862}, "ETHZ_autolab_technical_track-sc1-0-ego": {"driven_any": 3.063094341397713, "get_ui_image": 0.03901936531066894, "step_physics": 0.0909128761291504, "survival_time": 14.950000000000076, "driven_lanedir": 3.0150282556349524, "get_state_dump": 0.013825913270314537, "sim_render-ego": 0.006668095588684082, "get_robot_state": 0.016423354148864745, "get_duckie_state": 2.636909484863282e-06, "in-drivable-lane": 0.0, "agent_compute-ego": 0.01878982702891032, "deviation-heading": 1.898060025156063, "complete-iteration": 0.22146348317464193, "set_robot_commands": 0.004964284896850586, "deviation-center-line": 0.5238158006453122, "driven_lanedir_consec": 3.0150282556349524, "sim_compute_sim_state": 0.026282722155253093, "sim_compute_performance-ego": 0.0044191074371337895}, "ETHZ_autolab_technical_track-sc2-0-ego": {"driven_any": 3.0630248295136946, "get_ui_image": 0.04137129227320353, "step_physics": 0.10266769329706828, "survival_time": 14.950000000000076, "driven_lanedir": 2.9733980776096627, "get_state_dump": 0.013944629828135172, "sim_render-ego": 0.006909072399139404, "get_robot_state": 0.01649814287821452, "get_duckie_state": 2.5971730550130207e-06, "in-drivable-lane": 0.05000000000000071, "agent_compute-ego": 0.019228370984395345, "deviation-heading": 2.857021921225547, "complete-iteration": 0.23777187983194983, "set_robot_commands": 0.005188454786936442, "deviation-center-line": 0.8107925822404541, "driven_lanedir_consec": 2.9733980776096627, "sim_compute_sim_state": 0.027147082487742107, "sim_compute_performance-ego": 0.0046554152170817055}, "ETHZ_autolab_technical_track-sc3-0-ego": {"driven_any": 3.0631257213697927, "get_ui_image": 0.04271942694981893, "step_physics": 0.11283562819163004, "survival_time": 14.950000000000076, "driven_lanedir": 3.038398713484349, "get_state_dump": 0.014214359124501546, "sim_render-ego": 0.007144513130187988, "get_robot_state": 0.016972803274790445, "get_duckie_state": 2.6114781697591145e-06, "in-drivable-lane": 0.0, "agent_compute-ego": 0.01939761718114217, "deviation-heading": 1.3442300064810215, "complete-iteration": 0.25570335388183596, "set_robot_commands": 0.00542549212773641, "deviation-center-line": 0.6239710236655239, "driven_lanedir_consec": 3.038398713484349, "sim_compute_sim_state": 0.03207021395365397, "sim_compute_performance-ego": 0.004757972558339437}}
set_robot_commands_max0.005540388955010308
set_robot_commands_mean0.005279655191633436
set_robot_commands_median0.005306973457336425
set_robot_commands_min0.004964284896850586
sim_compute_performance-ego_max0.004757972558339437
sim_compute_performance-ego_mean0.004605355064074198
sim_compute_performance-ego_median0.004622170130411784
sim_compute_performance-ego_min0.0044191074371337895
sim_compute_sim_state_max0.03207021395365397
sim_compute_sim_state_mean0.028096085323227778
sim_compute_sim_state_median0.02701570259200202
sim_compute_sim_state_min0.026282722155253093
sim_render-ego_max0.007144513130187988
sim_render-ego_mean0.006906893187099032
sim_render-ego_median0.0069074820147620305
sim_render-ego_min0.006668095588684082
simulation-passed1
step_physics_max0.11283562819163004
step_physics_mean0.10282712744341956
step_physics_median0.10378000272644892
step_physics_min0.0909128761291504
survival_time_max14.950000000000076
survival_time_mean14.025000000000064
survival_time_min11.250000000000025
No reset possible