Duckietown Challenges Home Challenges Submissions

Evaluator 2185

ID2185
evaluatormont02-0e39d124a735-1
ownerI don't have one 😀
machineaido-ms-7b09
processmont02-0e39d124a735-1
versiond-c:6.1.20;d-c-r:6.0.49
first heard
last heard
statusinactive
# evaluating
# success1 38003
# timeout1 38128
# failed1 37997
# error1 38012
# aborted
# host-error4 38105
arm
x86_64
Mac
gpu available
Number of processors
Processor frequency (MHz)
Free % of processors
RAM total (MB)
RAM free (MB)
Disk (MB)
Disk available (MB)
Docker Hub
P1
P2
Cloud simulations
PI Camera
# Duckiebots
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
381289371Yishu Malhotra 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simtimeoutnomont02-0e39d124a735-11:05:14
I can see how the jo [...]
I can see how the job 38128 is timeout because passed 3914 seconds and the timeout is 3600.0.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
381239371Yishu Malhotra 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simhost-errornomont02-0e39d124a735-10:00:00
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner_daffy-6.0.49-py3.8.egg/duckietown_challenges_runner/runner.py", line 654, in get_cr
    os.makedirs(wd)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/media/sdb/duckietown-tmp-evals/production/aido5-LF-sim-validation'
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
381159364Yishu Malhotra 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simhost-errornomont02-0e39d124a735-10:00:00
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner_daffy-6.0.49-py3.8.egg/duckietown_challenges_runner/runner.py", line 654, in get_cr
    os.makedirs(wd)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/media/sdb/duckietown-tmp-evals/production/aido5-LF-sim-validation'
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
381149369Yishu Malhotra 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simhost-errornomont02-0e39d124a735-10:00:00
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner_daffy-6.0.49-py3.8.egg/duckietown_challenges_runner/runner.py", line 654, in get_cr
    os.makedirs(wd)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/media/sdb/duckietown-tmp-evals/production/aido5-LF-sim-validation'
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
381059367Jean-Sébastien Grondin 🇨🇦sim-exercise-1aido5-LF-sim-validationLFv-simhost-errornomont02-0e39d124a735-10:00:00
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner_daffy-6.0.49-py3.8.egg/duckietown_challenges_runner/runner.py", line 654, in get_cr
    os.makedirs(wd)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/media/sdb/duckietown-tmp-evals/production/aido5-LF-sim-validation'
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
380129971Bea Baselines 🐤template-pytorchaido5-LFV-sim-testingLFVv-simerrornomont02-0e39d124a735-11:00:35
Waited 3603 seconds [...]
Waited 3603 seconds for container to finish. Giving up.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
3800310807Fernanda Custodio Pereira do Carmo 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simsuccessnomont02-0e39d124a735-10:09:31
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median2.362846551666863
survival_time_median14.950000000000076
deviation-center-line_median0.7087653701966847
in-drivable-lane_median1.5250000000000217


other stats
agent_compute-ego_max0.018842915693918865
agent_compute-ego_mean0.01827520767847697
agent_compute-ego_median0.01840962012608846
agent_compute-ego_min0.017438674767812093
complete-iteration_max0.23904859860738115
complete-iteration_mean0.2204632560412089
complete-iteration_median0.2179837334156036
complete-iteration_min0.20683695872624716
deviation-center-line_max0.7645873466276818
deviation-center-line_mean0.6224400306175452
deviation-center-line_min0.30764203544912916
deviation-heading_max0.9468104973789764
deviation-heading_mean0.7731298362705422
deviation-heading_median0.7584151778673323
deviation-heading_min0.6288784919685279
driven_any_max2.6547302363435334
driven_any_mean2.6525533667367123
driven_any_median2.6547055231615486
driven_any_min2.646072184280218
driven_lanedir_consec_max2.646765507203206
driven_lanedir_consec_mean2.190980117811665
driven_lanedir_consec_min1.391461860709727
driven_lanedir_max2.646765507203206
driven_lanedir_mean2.190980117811665
driven_lanedir_median2.362846551666863
driven_lanedir_min1.391461860709727
get_duckie_state_max4.0777524312337236e-06
get_duckie_state_mean3.894766171773275e-06
get_duckie_state_median3.8695335388183595e-06
get_duckie_state_min3.762245178222656e-06
get_robot_state_max0.017414627075195314
get_robot_state_mean0.016297470132509866
get_robot_state_median0.0161473015944163
get_robot_state_min0.015480650266011557
get_state_dump_max0.01660522778828939
get_state_dump_mean0.014424843589464824
get_state_dump_median0.013787909746170043
get_state_dump_min0.013518327077229818
get_ui_image_max0.04328966061274211
get_ui_image_mean0.038421855966250106
get_ui_image_median0.03745716412862142
get_ui_image_min0.03548343499501546
in-drivable-lane_max6.900000000000098
in-drivable-lane_mean2.4875000000000353
in-drivable-lane_min0.0
per-episodes
details{"ETHZ_autolab_technical_track-sc0-0-ego": {"driven_any": 2.646072184280218, "get_ui_image": 0.04328966061274211, "step_physics": 0.10121314922968548, "survival_time": 14.950000000000076, "driven_lanedir": 2.634874146962394, "get_state_dump": 0.013770413398742676, "sim_render-ego": 0.0069739977518717446, "get_robot_state": 0.017414627075195314, "get_duckie_state": 3.933111826578776e-06, "in-drivable-lane": 0.0, "agent_compute-ego": 0.018842915693918865, "deviation-heading": 0.9468104973789764, "complete-iteration": 0.23904859860738115, "set_robot_commands": 0.005215038458506266, "deviation-center-line": 0.30764203544912916, "driven_lanedir_consec": 2.634874146962394, "sim_compute_sim_state": 0.02733186642328898, "sim_compute_performance-ego": 0.0048258344332377115}, "ETHZ_autolab_technical_track-sc1-0-ego": {"driven_any": 2.654718474191242, "get_ui_image": 0.03926607131958008, "step_physics": 0.09207716941833496, "survival_time": 14.950000000000076, "driven_lanedir": 2.646765507203206, "get_state_dump": 0.01660522778828939, "sim_render-ego": 0.006930681069691976, "get_robot_state": 0.01666032870610555, "get_duckie_state": 4.0777524312337236e-06, "in-drivable-lane": 0.0, "agent_compute-ego": 0.01876946767171224, "deviation-heading": 0.8640362906952694, "complete-iteration": 0.22572224299112956, "set_robot_commands": 0.005375416278839112, "deviation-center-line": 0.7645873466276818, "driven_lanedir_consec": 2.646765507203206, "sim_compute_sim_state": 0.02543442090352376, "sim_compute_performance-ego": 0.004432556629180909}, "ETHZ_autolab_technical_track-sc2-0-ego": {"driven_any": 2.6546925721318546, "get_ui_image": 0.03548343499501546, "step_physics": 0.08425358454386393, "survival_time": 14.950000000000076, "driven_lanedir": 1.391461860709727, "get_state_dump": 0.013805406093597412, "sim_render-ego": 0.006291337013244629, "get_robot_state": 0.015480650266011557, "get_duckie_state": 3.762245178222656e-06, "in-drivable-lane": 6.900000000000098, "agent_compute-ego": 0.01804977258046468, "deviation-heading": 0.6288784919685279, "complete-iteration": 0.20683695872624716, "set_robot_commands": 0.004874214331309001, "deviation-center-line": 0.7393491798602306, "driven_lanedir_consec": 1.391461860709727, "sim_compute_sim_state": 0.024348498980204265, "sim_compute_performance-ego": 0.0040888023376464844}, "ETHZ_autolab_technical_track-sc3-0-ego": {"driven_any": 2.6547302363435334, "get_ui_image": 0.03564825693766276, "step_physics": 0.08540121634801229, "survival_time": 14.950000000000076, "driven_lanedir": 2.0908189563713315, "get_state_dump": 0.013518327077229818, "sim_render-ego": 0.006189579168955485, "get_robot_state": 0.01563427448272705, "get_duckie_state": 3.805955251057942e-06, "in-drivable-lane": 3.0500000000000433, "agent_compute-ego": 0.017438674767812093, "deviation-heading": 0.6527940650393952, "complete-iteration": 0.2102452238400777, "set_robot_commands": 0.004794066747029623, "deviation-center-line": 0.6781815605331388, "driven_lanedir_consec": 2.0908189563713315, "sim_compute_sim_state": 0.02733460505803426, "sim_compute_performance-ego": 0.004128438631693522}}
set_robot_commands_max0.005375416278839112
set_robot_commands_mean0.005064683953921001
set_robot_commands_median0.0050446263949076335
set_robot_commands_min0.004794066747029623
sim_compute_performance-ego_max0.0048258344332377115
sim_compute_performance-ego_mean0.004368908007939657
sim_compute_performance-ego_median0.004280497630437216
sim_compute_performance-ego_min0.0040888023376464844
sim_compute_sim_state_max0.02733460505803426
sim_compute_sim_state_mean0.026112347841262817
sim_compute_sim_state_median0.02638314366340637
sim_compute_sim_state_min0.024348498980204265
sim_render-ego_max0.0069739977518717446
sim_render-ego_mean0.006596398750940959
sim_render-ego_median0.006611009041468302
sim_render-ego_min0.006189579168955485
simulation-passed1
step_physics_max0.10121314922968548
step_physics_mean0.09073627988497417
step_physics_median0.08873919288317363
step_physics_min0.08425358454386393
survival_time_max14.950000000000076
survival_time_mean14.950000000000076
survival_time_min14.950000000000076
No reset possible
3799710810Ayman Shams 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simfailednomont02-0e39d124a735-10:03:56
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 216, in read_one
    msgs = read_reply(self.fpout, timeout=timeout, waiting_for=waiting_for, nickname=self.nickname,)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 299, in read_reply
    others = read_until_over(fpout, timeout=timeout, nickname=nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 337, in read_until_over
    raise TimeoutError(msg) from None
TimeoutError: Timeout while reading output of node "ego".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "experiment_manager.py", line 512, in run_episode
    r: MsgReceived = await loop.run_in_executor(executor, f)
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 133, in write_topic_and_expect
    ob: MsgReceived = self.read_one(expect_topic=expect, timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 279, in read_one
    raise TimeoutError(msg) from e
TimeoutError: Timeout detected on /fifos/ego-out after 125 messages. Expected topic "commands".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 678, in scoring_context
    yield cie
  File "experiment_manager.py", line 655, in go
    wrap(cie)
  File "experiment_manager.py", line 640, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "experiment_manager.py", line 253, in main
    length_s = await run_episode(
  File "experiment_manager.py", line 516, in run_episode
    raise dc.InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Trouble with communication to the agent.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible