Evaluator 906

ID	906
evaluator	`ip-172-31-40-253-32125`
owner	I don't have one 😀
machine	ip-172-31-40-253
process	ip-172-31-40-253-32125
version	d-c:4.0.25;d-c-r:4.0.32;d-s:4.0.21
first heard	2019-05-07 19:25:18+00:00
last heard	2019-05-13 14:29:51+00:00
status	inactive
# evaluating
# success	2 `21859`
# timeout
# failed	2 `21695`
# error	3 `21702`
# aborted
# host-error	6 `21712`
arm
x86_64
Mac
gpu available
Number of processors
Processor frequency (MHz)
Free % of processors
RAM total (MB)
RAM free (MB)
Disk (MB)
Disk available (MB)
Docker Hub
P1
P2
Cloud simulations
PI Camera
# Duckiebots
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job ID

submission

user

user label

challenge

step

status

up to date

evaluator

date started

date completed

duration

message

21923

3353

FANG MEIYI 🇸🇬

test

aido2-LFV-sim-testing

step1-simulation

success

ip-172-31-40-253-32125

2019-05-13 05:02:26+00:00

2019-05-13 05:44:59+00:00

0:42:33

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

driven_lanedir_consec_median	2.57416760465345
survival_time_median	13.100000000000051
deviation-center-line_median	0.4795837561836091
in-drivable-lane_median	0

other stats

agent_compute-ego_max	0.09426415920257568
agent_compute-ego_mean	0.0925199244224921
agent_compute-ego_median	0.09247880299886067
agent_compute-ego_min	0.0912354598611088
deviation-center-line_max	1.059103768311453
deviation-center-line_mean	0.5330486848023112
deviation-center-line_min	0.21878770952411455
deviation-heading_max	3.525150739177256
deviation-heading_mean	1.858759301244596
deviation-heading_median	1.9576596595368891
deviation-heading_min	0.7115916052211718
driven_any_max	3.111028856000527
driven_any_mean	2.038461425684416
driven_any_median	2.793776749154716
driven_any_min	0.5254545550160427
driven_lanedir_consec_max	3.076192919129867
driven_lanedir_consec_mean	1.944918141784537
driven_lanedir_consec_min	0.4813675204845296
driven_lanedir_max	3.076192919129867
driven_lanedir_mean	1.944918141784537
driven_lanedir_median	2.57416760465345
driven_lanedir_min	0.4813675204845296
in-drivable-lane_max	1.050000000000015
in-drivable-lane_mean	0.22000000000000164
in-drivable-lane_min	0
per-episodes	details {"ETHZ_autolab_technical_track-0-0": {"driven_any": 0.5579967122962197, "sim_physics": 0.10054920083385405, "survival_time": 2.9499999999999975, "driven_lanedir": 0.5362204582186902, "sim_render-ego": 0.03670798317860749, "in-drivable-lane": 0, "agent_compute-ego": 0.0932210946487168, "deviation-heading": 0.7230954101375587, "set_robot_commands": 0.055990368632946985, "deviation-center-line": 0.21878770952411455, "driven_lanedir_consec": 0.5362204582186902, "sim_compute_sim_state": 0.022607645746004785, "sim_compute_performance-ego": 0.03901341405965514, "sim_compute_robot_state-ego": 0.0417167534262447, "sim_compute_robot_state-npc0": 0.04098817049446753, "sim_compute_robot_state-npc1": 0.04008247488636082, "sim_compute_robot_state-npc2": 0.03992880805064056, "sim_compute_robot_state-npc3": 0.03988977610054663}, "ETHZ_autolab_technical_track-1-0": {"driven_any": 3.111028856000527, "sim_physics": 0.09546915451685588, "survival_time": 14.950000000000076, "driven_lanedir": 3.076192919129867, "sim_render-ego": 0.03633246103922526, "in-drivable-lane": 0, "agent_compute-ego": 0.09247196356455484, "deviation-heading": 1.9576596595368891, "set_robot_commands": 0.05655972719192505, "deviation-center-line": 0.808753660048207, "driven_lanedir_consec": 3.076192919129867, "sim_compute_sim_state": 0.022823288440704345, "sim_compute_performance-ego": 0.03933812697728475, "sim_compute_robot_state-ego": 0.04175350268681844, "sim_compute_robot_state-npc0": 0.04106300115585327, "sim_compute_robot_state-npc1": 0.04027830521265666, "sim_compute_robot_state-npc2": 0.03974238952000936, "sim_compute_robot_state-npc3": 0.039635377724965414}, "ETHZ_autolab_technical_track-2-0": {"driven_any": 1.10259113813053, "sim_physics": 0.09334883916945684, "survival_time": 5.249999999999989, "driven_lanedir": 1.0873383876362102, "sim_render-ego": 0.03664747646876744, "in-drivable-lane": 0, "agent_compute-ego": 0.09170457295009068, "deviation-heading": 0.7115916052211718, "set_robot_commands": 0.05543870698838007, "deviation-center-line": 0.32706910969091485, "driven_lanedir_consec": 1.0873383876362102, "sim_compute_sim_state": 0.02287928717476981, "sim_compute_performance-ego": 0.039221484320504325, "sim_compute_robot_state-ego": 0.041571991784232, "sim_compute_robot_state-npc0": 0.040987646012079146, "sim_compute_robot_state-npc1": 0.0397549379439581, "sim_compute_robot_state-npc2": 0.0396658079964774, "sim_compute_robot_state-npc3": 0.039637490681239534}, "ETHZ_autolab_technical_track-3-0": {"driven_any": 1.1289860824444455, "sim_physics": 0.09378117712858682, "survival_time": 5.349999999999989, "driven_lanedir": 1.1060522970612805, "sim_render-ego": 0.03641311030521571, "in-drivable-lane": 0, "agent_compute-ego": 0.0939927680470119, "deviation-heading": 0.8122228059480999, "set_robot_commands": 0.055875800480352386, "deviation-center-line": 0.2956262600288367, "driven_lanedir_consec": 1.1060522970612805, "sim_compute_sim_state": 0.02323582907703435, "sim_compute_performance-ego": 0.03936747524225823, "sim_compute_robot_state-ego": 0.04172977554463895, "sim_compute_robot_state-npc0": 0.04133167222281483, "sim_compute_robot_state-npc1": 0.040481400266986024, "sim_compute_robot_state-npc2": 0.040369468314625394, "sim_compute_robot_state-npc3": 0.03976149202507233}, "ETHZ_autolab_technical_track-4-0": {"driven_any": 1.851347516508506, "sim_physics": 0.08378587166468303, "survival_time": 9.6, "driven_lanedir": 1.6027959402355272, "sim_render-ego": 0.036184498419364296, "in-drivable-lane": 1.050000000000015, "agent_compute-ego": 0.09189100190997124, "deviation-heading": 1.6133179515287897, "set_robot_commands": 0.056664422154426575, "deviation-center-line": 0.4795837561836091, "driven_lanedir_consec": 1.6027959402355272, "sim_compute_sim_state": 0.023015784720579784, "sim_compute_performance-ego": 0.03914984439810117, "sim_compute_robot_state-ego": 0.04144884025057157, "sim_compute_robot_state-npc0": 0.0409107506275177, "sim_compute_robot_state-npc1": 0.04000304266810417, "sim_compute_robot_state-npc2": 0.039664462208747864, "sim_compute_robot_state-npc3": 0.03978478411833445}, "ETHZ_autolab_technical_track-5-0": {"driven_any": 1.0776237924477423, "sim_physics": 0.08830017176541415, "survival_time": 5.4999999999999885, "driven_lanedir": 0.9895342580832172, "sim_render-ego": 0.03673955960707231, "in-drivable-lane": 0.29999999999999893, "agent_compute-ego": 0.09195692105726762, "deviation-heading": 0.9940146560093256, "set_robot_commands": 0.057061535661870784, "deviation-center-line": 0.3428943357127203, "driven_lanedir_consec": 0.9895342580832172, "sim_compute_sim_state": 0.022652409293434837, "sim_compute_performance-ego": 0.03913775574077259, "sim_compute_robot_state-ego": 0.041346255215731535, "sim_compute_robot_state-npc0": 0.04114778908816251, "sim_compute_robot_state-npc1": 0.039736359769647775, "sim_compute_robot_state-npc2": 0.039786698601462625, "sim_compute_robot_state-npc3": 0.03966302871704101}, "ETHZ_autolab_technical_track-6-0": {"driven_any": 2.8414775319243484, "sim_physics": 0.09052671670913696, "survival_time": 14.950000000000076, "driven_lanedir": 2.795399672482862, "sim_render-ego": 0.036608780225118, "in-drivable-lane": 0, "agent_compute-ego": 0.09426415920257568, "deviation-heading": 2.252262604981998, "set_robot_commands": 0.05581378698348999, "deviation-center-line": 0.5891125866981072, "driven_lanedir_consec": 2.795399672482862, "sim_compute_sim_state": 0.02276226838429769, "sim_compute_performance-ego": 0.03906535943349203, "sim_compute_robot_state-ego": 0.04126808245976766, "sim_compute_robot_state-npc0": 0.04079168081283569, "sim_compute_robot_state-npc1": 0.04062102476755778, "sim_compute_robot_state-npc2": 0.03959442377090454, "sim_compute_robot_state-npc3": 0.039867177804311114}, "ETHZ_autolab_technical_track-7-0": {"driven_any": 2.8017569866090746, "sim_physics": 0.09017577027916011, "survival_time": 13.300000000000054, "driven_lanedir": 2.57416760465345, "sim_render-ego": 0.03811391224538473, "in-drivable-lane": 0.8000000000000114, "agent_compute-ego": 0.092708963200562, "deviation-heading": 2.38015135583649, "set_robot_commands": 0.05632229407030837, "deviation-center-line": 0.5881285856218341, "driven_lanedir_consec": 2.57416760465345, "sim_compute_sim_state": 0.022970347476184816, "sim_compute_performance-ego": 0.03979093418981796, "sim_compute_robot_state-ego": 0.04172709382566294, "sim_compute_robot_state-npc0": 0.04140139612040125, "sim_compute_robot_state-npc1": 0.04021085742720984, "sim_compute_robot_state-npc2": 0.0403726423593392, "sim_compute_robot_state-npc3": 0.04000379447650192}, "ETHZ_autolab_technical_track-8-0": {"driven_any": 2.929709524909851, "sim_physics": 0.0985369356473287, "survival_time": 14.950000000000076, "driven_lanedir": 2.814048182972654, "sim_render-ego": 0.0364825701713562, "in-drivable-lane": 0, "agent_compute-ego": 0.09284500678380332, "deviation-heading": 3.525150739177256, "set_robot_commands": 0.05637880166371664, "deviation-center-line": 0.9279061528370056, "driven_lanedir_consec": 2.814048182972654, "sim_compute_sim_state": 0.02288838307062785, "sim_compute_performance-ego": 0.03943959633509318, "sim_compute_robot_state-ego": 0.04138147989908854, "sim_compute_robot_state-npc0": 0.04096012910207113, "sim_compute_robot_state-npc1": 0.039805467128753665, "sim_compute_robot_state-npc2": 0.03987889766693115, "sim_compute_robot_state-npc3": 0.039669482707977294}, "ETHZ_autolab_technical_track-9-0": {"driven_any": 2.793776749154716, "sim_physics": 0.09428142136289872, "survival_time": 13.100000000000051, "driven_lanedir": 2.6742317135710563, "sim_render-ego": 0.03619972895119936, "in-drivable-lane": 0.20000000000000284, "agent_compute-ego": 0.09164827015563733, "deviation-heading": 2.365448406152338, "set_robot_commands": 0.05572979413826047, "deviation-center-line": 0.6495834308660879, "driven_lanedir_consec": 2.6742317135710563, "sim_compute_sim_state": 0.02286205400947396, "sim_compute_performance-ego": 0.03894915016552874, "sim_compute_robot_state-ego": 0.04126793191633152, "sim_compute_robot_state-npc0": 0.040852824240240435, "sim_compute_robot_state-npc1": 0.03999413242776885, "sim_compute_robot_state-npc2": 0.03958163461612381, "sim_compute_robot_state-npc3": 0.03975827730338992}, "ETHZ_autolab_technical_track-10-0": {"driven_any": 2.8009394604230633, "sim_physics": 0.0901082436243693, "survival_time": 14.950000000000076, "driven_lanedir": 2.7430584187677765, "sim_render-ego": 0.0364675768216451, "in-drivable-lane": 0, "agent_compute-ego": 0.09265875577926636, "deviation-heading": 2.693844020138569, "set_robot_commands": 0.055343449115753174, "deviation-center-line": 0.6802874480951538, "driven_lanedir_consec": 2.7430584187677765, "sim_compute_sim_state": 0.022944425741831465, "sim_compute_performance-ego": 0.03923181931177775, "sim_compute_robot_state-ego": 0.04120957136154175, "sim_compute_robot_state-npc0": 0.0408998966217041, "sim_compute_robot_state-npc1": 0.040033578872680664, "sim_compute_robot_state-npc2": 0.03968730529149373, "sim_compute_robot_state-npc3": 0.03954227844874064}, "ETHZ_autolab_technical_track-11-0": {"driven_any": 1.0404857247106216, "sim_physics": 0.08486330913285077, "survival_time": 5.899999999999987, "driven_lanedir": 0.8238069266492873, "sim_render-ego": 0.03573649414515091, "in-drivable-lane": 0.8999999999999968, "agent_compute-ego": 0.0912354598611088, "deviation-heading": 1.4360793606870097, "set_robot_commands": 0.058360546322192175, "deviation-center-line": 0.340942542225096, "driven_lanedir_consec": 0.8238069266492873, "sim_compute_sim_state": 0.02308752981282897, "sim_compute_performance-ego": 0.039044976234436035, "sim_compute_robot_state-ego": 0.04139793929407152, "sim_compute_robot_state-npc0": 0.04591548644890219, "sim_compute_robot_state-npc1": 0.03994828361575886, "sim_compute_robot_state-npc2": 0.04016871775610972, "sim_compute_robot_state-npc3": 0.03954483896999036}, "ETHZ_autolab_technical_track-12-0": {"driven_any": 0.5254545550160427, "sim_physics": 0.10233536599174378, "survival_time": 3.149999999999997, "driven_lanedir": 0.4813675204845296, "sim_render-ego": 0.03635956370641315, "in-drivable-lane": 0.04999999999999982, "agent_compute-ego": 0.09264581165616474, "deviation-heading": 0.9604657875587174, "set_robot_commands": 0.055677796167040626, "deviation-center-line": 0.2409813945500072, "driven_lanedir_consec": 0.4813675204845296, "sim_compute_sim_state": 0.02282516918485127, "sim_compute_performance-ego": 0.039065273981245735, "sim_compute_robot_state-ego": 0.04143305051894415, "sim_compute_robot_state-npc0": 0.04073841988094269, "sim_compute_robot_state-npc1": 0.040096737089611235, "sim_compute_robot_state-npc2": 0.04029557818458194, "sim_compute_robot_state-npc3": 0.03931043261573428}, "ETHZ_autolab_technical_track-13-0": {"driven_any": 2.981026946021316, "sim_physics": 0.09029695749282836, "survival_time": 14.950000000000076, "driven_lanedir": 2.8805007322675253, "sim_render-ego": 0.036612123648325605, "in-drivable-lane": 0, "agent_compute-ego": 0.09247880299886067, "deviation-heading": 3.20215165851837, "set_robot_commands": 0.05600699424743652, "deviation-center-line": 1.059103768311453, "driven_lanedir_consec": 2.8805007322675253, "sim_compute_sim_state": 0.02257347504297892, "sim_compute_performance-ego": 0.03927194039026896, "sim_compute_robot_state-ego": 0.041469775040944415, "sim_compute_robot_state-npc0": 0.04083489100138346, "sim_compute_robot_state-npc1": 0.03966626882553101, "sim_compute_robot_state-npc2": 0.03976341803868612, "sim_compute_robot_state-npc3": 0.03978372017542521}, "ETHZ_autolab_technical_track-14-0": {"driven_any": 3.032719808669233, "sim_physics": 0.09623352924982707, "survival_time": 14.950000000000076, "driven_lanedir": 2.9890570945541213, "sim_render-ego": 0.0364319101969401, "in-drivable-lane": 0, "agent_compute-ego": 0.09207531452178956, "deviation-heading": 2.253933497236358, "set_robot_commands": 0.05546858310699463, "deviation-center-line": 0.44696953164151865, "driven_lanedir_consec": 2.9890570945541213, "sim_compute_sim_state": 0.02286589066187541, "sim_compute_performance-ego": 0.04011486609776815, "sim_compute_robot_state-ego": 0.04148960510889689, "sim_compute_robot_state-npc0": 0.04100425720214844, "sim_compute_robot_state-npc1": 0.040081835587819414, "sim_compute_robot_state-npc2": 0.039664398034413656, "sim_compute_robot_state-npc3": 0.03979611237843832}}
set_robot_commands_max	0.058360546322192175
set_robot_commands_mean	0.056179507128339634
set_robot_commands_median	0.055990368632946985
set_robot_commands_min	0.055343449115753174
sim_compute_performance-ego_max	0.04011486609776815
sim_compute_performance-ego_mean	0.03928013445853365
sim_compute_performance-ego_median	0.039221484320504325
sim_compute_performance-ego_min	0.03894915016552874
sim_compute_robot_state-ego_max	0.04175350268681844
sim_compute_robot_state-ego_mean	0.04148077655556577
sim_compute_robot_state-ego_median	0.04144884025057157
sim_compute_robot_state-ego_min	0.04120957136154175
sim_compute_robot_state-npc0_max	0.04591548644890219
sim_compute_robot_state-npc0_mean	0.04132186740210163
sim_compute_robot_state-npc0_median	0.040987646012079146
sim_compute_robot_state-npc0_min	0.04073841988094269
sim_compute_robot_state-npc1_max	0.04062102476755778
sim_compute_robot_state-npc1_mean	0.04005298043269366
sim_compute_robot_state-npc1_median	0.040033578872680664
sim_compute_robot_state-npc1_min	0.03966626882553101
sim_compute_robot_state-npc2_max	0.0403726423593392
sim_compute_robot_state-npc2_mean	0.03987764336070313
sim_compute_robot_state-npc2_median	0.03976341803868612
sim_compute_robot_state-npc2_min	0.03958163461612381
sim_compute_robot_state-npc3_max	0.04000379447650192
sim_compute_robot_state-npc3_mean	0.03970987094984722
sim_compute_robot_state-npc3_median	0.03975827730338992
sim_compute_robot_state-npc3_min	0.03931043261573428
sim_compute_sim_state_max	0.02323582907703435
sim_compute_sim_state_mean	0.02286625252249855
sim_compute_sim_state_median	0.02286589066187541
sim_compute_sim_state_min	0.02257347504297892
sim_physics_max	0.10233536599174378
sim_physics_mean	0.0928395109712663
sim_physics_median	0.09334883916945684
sim_physics_min	0.08378587166468303
sim_render-ego_max	0.03811391224538473
sim_render-ego_mean	0.03653584994198571
sim_render-ego_median	0.0364675768216451
sim_render-ego_min	0.03573649414515091
simulation-passed	1
survival_time_max	14.950000000000076
survival_time_mean	10.253333333333368
survival_time_min	2.9499999999999975

No reset possible

21884

3333

FANG MEIYI 🇸🇬

test

aido2-LFV-sim-validation

step1-simulation

host-error

ip-172-31-40-253-32125

2019-05-11 15:42:40+00:00

2019-05-11 16:00:55+00:00

0:18:15

Uncaught exception: [...]

Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 488, in get_cr
    submission_id=submission_id, timeout_sec=timeout_sec)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/project/src/duckietown_challenges_runner/runner.py", line 347, in setup_logging
    convert(f_stdout)
  File "/project/src/duckietown_challenges_runner/runner.py", line 343, in convert
    with open(fn, 'w') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/duckietown/DT18/evaluator/executions/aido2-LFV-sim-validation/submission3333/step1-simulation-ip-172-31-40-253-32125-job21884/logs/challenges-runner/stdout.html'

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

21863

3326

jiang peng

RL - TH -xyz

aido2-LFV-sim-validation

step1-simulation

failed

ip-172-31-40-253-32125

2019-05-10 09:25:45+00:00

2019-05-10 09:27:08+00:00

0:01:23

InvalidSubmission: T [...]

InvalidSubmission:
Traceback (most recent call last):
  File "experiment_manager.py", line 297, in run_episode
    r: MsgReceived = agent.write_topic_and_expect('get_commands', expect='commands')
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 93, in write_topic_and_expect
    ob: MsgReceived = self.read_one(expect_topic=expect, timeout=timeout)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 174, in read_one
    nickname=self.nickname)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 255, in read_reply
    others = read_until_over(fpout, timeout=timeout, nickname=nickname)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 283, in read_until_over
    raise RemoteNodeAborted(m)
zuper_nodes.structures.RemoteNodeAborted: External node "agent" aborted:

error in agent |Exception while handling a message on topic "get_commands".
               |
               || Traceback (most recent call last):
               ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 274, in loop
               ||     handle_message_node(parsed, receiver0, context0)
               ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 428, in handle_message_node
               ||     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
               ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/utils.py", line 18, in call_if_fun_exists
               ||     f(**kwargs)
               ||   File "solution.py", line 50, in on_received_get_commands
               ||     action = self.compute_action(self.current_image)
               ||   File "solution.py", line 47, in compute_action
               ||     return action.astype(float)
               || AttributeError: 'list' object has no attribute 'astype'
               ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 610, in scoring_context
    yield cie
  File "experiment_manager.py", line 439, in <module>
    wrap(cie)
  File "experiment_manager.py", line 427, in wrap
    main(cie, logdir, attempts)
  File "experiment_manager.py", line 135, in main
    physics_dt=config.physics_dt)
  File "experiment_manager.py", line 301, in run_episode
    raise dc.InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Trouble with communication to the agent.

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

21859

3323

Angel Woo 🇭🇰

Roasted duck

aido2-LF-sim-validation

step1-simulation

success

ip-172-31-40-253-32125

2019-05-10 08:44:13+00:00

2019-05-10 08:53:46+00:00

0:09:33

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

driven_lanedir_consec_median	0.34850121842387116
survival_time_median	5.299999999999989
deviation-center-line_median	0.30191457913726444
in-drivable-lane_median	0.1999999999999993

other stats

agent_compute-ego_max	0.12578569412231444
agent_compute-ego_mean	0.12505074736905047
agent_compute-ego_median	0.12538391278113847
agent_compute-ego_min	0.12400828089032856
deviation-center-line_max	0.8986167349059359
deviation-center-line_mean	0.4276804557327886
deviation-center-line_min	0.280838835191478
deviation-heading_max	2.091625344780871
deviation-heading_mean	1.7692674638256385
deviation-heading_median	1.8881579462934055
deviation-heading_min	1.0759558126539868
driven_any_max	1.0992041960508545
driven_any_mean	0.5637986305715273
driven_any_median	0.3968794954567439
driven_any_min	0.19175842346583125
driven_lanedir_consec_max	1.0667102504045491
driven_lanedir_consec_mean	0.5007238341097251
driven_lanedir_consec_min	0.17526362213856306
driven_lanedir_max	1.0667102504045491
driven_lanedir_mean	0.5007238341097251
driven_lanedir_median	0.34850121842387116
driven_lanedir_min	0.17526362213856306
in-drivable-lane_max	1.550000000000022
in-drivable-lane_mean	0.4200000000000047
in-drivable-lane_min	0
per-episodes	details {"ETHZ_autolab_technical_track-0-0": {"driven_any": 1.0992041960508545, "sim_physics": 0.06400335947672527, "survival_time": 14.950000000000076, "driven_lanedir": 1.0667102504045491, "sim_render-ego": 0.037285063266754154, "in-drivable-lane": 0.20000000000000284, "agent_compute-ego": 0.12578569412231444, "deviation-heading": 2.091625344780871, "set_robot_commands": 0.05663000345230103, "deviation-center-line": 0.8986167349059359, "driven_lanedir_consec": 1.0667102504045491, "sim_compute_sim_state": 0.023875393867492677, "sim_compute_performance-ego": 0.03996642033259074, "sim_compute_robot_state-ego": 0.041828501224517825}, "ETHZ_autolab_technical_track-1-0": {"driven_any": 0.8282291581257205, "sim_physics": 0.06031756110034638, "survival_time": 10.650000000000016, "driven_lanedir": 0.6685241870977788, "sim_render-ego": 0.03599658034776858, "in-drivable-lane": 1.550000000000022, "agent_compute-ego": 0.12456741243460928, "deviation-heading": 1.8881579462934055, "set_robot_commands": 0.055817361169018094, "deviation-center-line": 0.36712641166856913, "driven_lanedir_consec": 0.6685241870977788, "sim_compute_sim_state": 0.023529731052022582, "sim_compute_performance-ego": 0.039880795098246546, "sim_compute_robot_state-ego": 0.041706526223482664}, "ETHZ_autolab_technical_track-2-0": {"driven_any": 0.19175842346583125, "sim_physics": 0.055225678852626255, "survival_time": 2.799999999999998, "driven_lanedir": 0.17526362213856306, "sim_render-ego": 0.03560296978269305, "in-drivable-lane": 0, "agent_compute-ego": 0.12400828089032856, "deviation-heading": 1.0759558126539868, "set_robot_commands": 0.056518767561231344, "deviation-center-line": 0.30191457913726444, "driven_lanedir_consec": 0.17526362213856306, "sim_compute_sim_state": 0.02397323931966509, "sim_compute_performance-ego": 0.03937755737985883, "sim_compute_robot_state-ego": 0.04136959995542254}, "ETHZ_autolab_technical_track-3-0": {"driven_any": 0.30292187975848667, "sim_physics": 0.05852556228637695, "survival_time": 4.049999999999994, "driven_lanedir": 0.2446198924838634, "sim_render-ego": 0.03719732496473524, "in-drivable-lane": 0.1999999999999993, "agent_compute-ego": 0.12538391278113847, "deviation-heading": 1.9995077861887471, "set_robot_commands": 0.057101293846412944, "deviation-center-line": 0.280838835191478, "driven_lanedir_consec": 0.2446198924838634, "sim_compute_sim_state": 0.023677460941267603, "sim_compute_performance-ego": 0.039527304378556616, "sim_compute_robot_state-ego": 0.04222586714191201}, "ETHZ_autolab_technical_track-4-0": {"driven_any": 0.3968794954567439, "sim_physics": 0.06219720615530914, "survival_time": 5.299999999999989, "driven_lanedir": 0.34850121842387116, "sim_render-ego": 0.03720920715691908, "in-drivable-lane": 0.14999999999999947, "agent_compute-ego": 0.1255084366168616, "deviation-heading": 1.791090429211181, "set_robot_commands": 0.05631740363139027, "deviation-center-line": 0.2899057177606957, "driven_lanedir_consec": 0.34850121842387116, "sim_compute_sim_state": 0.02352150206295949, "sim_compute_performance-ego": 0.04010434195680438, "sim_compute_robot_state-ego": 0.04274810260196902}}
set_robot_commands_max	0.057101293846412944
set_robot_commands_mean	0.05647696593207073
set_robot_commands_median	0.056518767561231344
set_robot_commands_min	0.055817361169018094
sim_compute_performance-ego_max	0.04010434195680438
sim_compute_performance-ego_mean	0.03977128382921142
sim_compute_performance-ego_median	0.039880795098246546
sim_compute_performance-ego_min	0.03937755737985883
sim_compute_robot_state-ego_max	0.04274810260196902
sim_compute_robot_state-ego_mean	0.04197571942946081
sim_compute_robot_state-ego_median	0.041828501224517825
sim_compute_robot_state-ego_min	0.04136959995542254
sim_compute_sim_state_max	0.02397323931966509
sim_compute_sim_state_mean	0.02371546544868149
sim_compute_sim_state_median	0.023677460941267603
sim_compute_sim_state_min	0.02352150206295949
sim_physics_max	0.06400335947672527
sim_physics_mean	0.06005387357427679
sim_physics_median	0.06031756110034638
sim_physics_min	0.055225678852626255
sim_render-ego_max	0.037285063266754154
sim_render-ego_mean	0.03665822910377402
sim_render-ego_median	0.03719732496473524
sim_render-ego_min	0.03560296978269305
simulation-passed	1
survival_time_max	14.950000000000076
survival_time_mean	7.550000000000016
survival_time_min	2.799999999999998

No reset possible

21858

3322

jiang peng

RL - TH -xyz

aido2-LFV-sim-validation

step1-simulation

error

ip-172-31-40-253-32125

2019-05-10 08:39:54+00:00

2019-05-10 08:41:23+00:00

0:01:29

The container "solut [...]

The container "solution" exited with code 1.


Look at the logs for the container to know more about the error.

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

21845

3314

Peter Almasi 🇭🇺

Baseline solution using reinforcement learning

aido2-LFVI-sim-testing

step1-simulation

host-error

ip-172-31-40-253-32125

2019-05-09 13:49:38+00:00

2019-05-09 14:01:46+00:00

0:12:08

Uncaught exception: [...]

Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 488, in get_cr
    submission_id=submission_id, timeout_sec=timeout_sec)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/project/src/duckietown_challenges_runner/runner.py", line 347, in setup_logging
    convert(f_stdout)
  File "/project/src/duckietown_challenges_runner/runner.py", line 343, in convert
    with open(fn, 'w') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/duckietown/DT18/evaluator/executions/aido2-LFVI-sim-testing/submission3314/step1-simulation-ip-172-31-40-253-32125-job21845/logs/challenges-runner/stdout.html'

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

21779

2981

Julian Zilly

Baseline-IL-logs-tensorflow

aido2-LFVI-sim-testing

step1-simulation

host-error

ip-172-31-40-253-32125

2019-05-07 22:02:27+00:00

2019-05-07 23:01:37+00:00

0:59:10

Uncaught exception: [...]

Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 488, in get_cr
    submission_id=submission_id, timeout_sec=timeout_sec)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/project/src/duckietown_challenges_runner/runner.py", line 347, in setup_logging
    convert(f_stdout)
  File "/project/src/duckietown_challenges_runner/runner.py", line 343, in convert
    with open(fn, 'w') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/duckietown/DT18/evaluator/executions/aido2-LFVI-sim-testing/submission2981/step1-simulation-ip-172-31-40-253-32125-job21779/logs/challenges-runner/stdout.html'

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

21750

2907

Julian Zilly

Baseline-IL-logs-tensorflow

aido2-LFVI-sim-testing

step1-simulation

host-error

ip-172-31-40-253-32125

2019-05-07 21:00:30+00:00

2019-05-07 22:00:06+00:00

0:59:36

Uncaught exception: [...]

Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 1201, in upload
    aws_object.load()
  File "/usr/local/lib/python3.6/dist-packages/boto3/resources/factory.py", line 505, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
  File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 491, in get_cr
    uploaded = upload_files(wd, aws_config, copy_to_machine_cache=copy_to_machine_cache)
  File "/project/src/duckietown_challenges_runner/runner.py", line 1032, in upload_files
    uploaded = upload(aws_config, toupload, copy_to_machine_cache=copy_to_machine_cache)
  File "/project/src/duckietown_challenges_runner/runner.py", line 1211, in upload
    aws_object.upload_file(realfile, ExtraArgs={'ContentType': mime_type})
  File "/usr/local/lib/python3.6/dist-packages/boto3/s3/inject.py", line 280, in object_upload_file
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File "/usr/local/lib/python3.6/dist-packages/boto3/s3/inject.py", line 131, in upload_file
    extra_args=ExtraArgs, callback=Callback)
  File "/usr/local/lib/python3.6/dist-packages/boto3/s3/transfer.py", line 279, in upload_file
    future.result()
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/upload.py", line 722, in _main
    Body=body, **extra_args)
  File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 642, in _make_api_call
    request_signer=self._request_signer, context=request_context)
  File "/usr/local/lib/python3.6/dist-packages/botocore/hooks.py", line 360, in emit_until_response
    return self._emitter.emit_until_response(aliased_event_name, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/botocore/hooks.py", line 243, in emit_until_response
    responses = self._emit(event_name, kwargs, stop_on_response=True)
  File "/usr/local/lib/python3.6/dist-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.6/dist-packages/botocore/handlers.py", line 212, in conditionally_calculate_md5
    calculate_md5(params, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/botocore/handlers.py", line 190, in calculate_md5
    binary_md5 = _calculate_md5_from_file(body)
  File "/usr/local/lib/python3.6/dist-packages/botocore/handlers.py", line 203, in _calculate_md5_from_file
    for chunk in iter(lambda: fileobj.read(1024 * 1024), b''):
  File "/usr/local/lib/python3.6/dist-packages/botocore/handlers.py", line 203, in <lambda>
    for chunk in iter(lambda: fileobj.read(1024 * 1024), b''):
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/utils.py", line 448, in read
    data = self._fileobj.read(amount_to_read)
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/upload.py", line 86, in read
    return self._fileobj.read(amount)
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/utils.py", line 327, in read
    self._open_if_needed()
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/utils.py", line 318, in _open_if_needed
    self._fileobj = self._open_function(self._filename, self._mode)
  File "/usr/local/lib/python3.6/dist-packages/s3transfer/utils.py", line 244, in open
    return open(filename, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/duckietown/DT18/evaluator/executions/aido2-LFVI-sim-testing/submission2907/step1-simulation-ip-172-31-40-253-32125-job21750/tmp/tmpvrlykxmg/episodes/udem1-8-0/drawing.html'

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

21727

2698

Konstantin Chaika

challenge-aido_LF-baseline-duckietown

aido2-LFVI-sim-testing

step1-simulation

host-error

ip-172-31-40-253-32125

2019-05-07 20:03:38+00:00

2019-05-07 21:00:22+00:00

0:56:44

Uncaught exception: [...]

Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 488, in get_cr
    submission_id=submission_id, timeout_sec=timeout_sec)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/project/src/duckietown_challenges_runner/runner.py", line 347, in setup_logging
    convert(f_stdout)
  File "/project/src/duckietown_challenges_runner/runner.py", line 343, in convert
    with open(fn, 'w') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/duckietown/DT18/evaluator/executions/aido2-LFVI-sim-testing/submission2698/step1-simulation-ip-172-31-40-253-32125-job21727/logs/challenges-runner/stdout.html'

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

21722

2536

Liam Paull 🇨🇦

baseline-IL-sim-tensorflow

aido2-LFVI-sim-testing

step1-simulation

error

ip-172-31-40-253-32125

2019-05-07 20:00:55+00:00

2019-05-07 20:03:22+00:00

0:02:27

The container "solut [...]

The container "solution" exited with code 1.


Look at the logs for the container to know more about the error.

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

21712

2636

Andrea Censi 🇨🇭

challenge-aido_LF-template-random

aido2-LFVI-sim-testing

step1-simulation

host-error

ip-172-31-40-253-32125

2019-05-07 19:40:13+00:00

2019-05-07 20:00:29+00:00

0:20:16

Uncaught exception: [...]

Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 488, in get_cr
    submission_id=submission_id, timeout_sec=timeout_sec)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/project/src/duckietown_challenges_runner/runner.py", line 347, in setup_logging
    convert(f_stdout)
  File "/project/src/duckietown_challenges_runner/runner.py", line 343, in convert
    with open(fn, 'w') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/duckietown/DT18/evaluator/executions/aido2-LFVI-sim-testing/submission2636/step1-simulation-ip-172-31-40-253-32125-job21712/logs/challenges-runner/stdout.html'

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

21702

2581

Alexander Karavaev

challenge-aido_LF-template-ros - Template solution using ROS

aido2-LFVI-sim-testing

step1-simulation

error

ip-172-31-40-253-32125

2019-05-07 19:28:31+00:00

2019-05-07 19:31:06+00:00

0:02:35

The container "solut [...]

The container "solution" exited with code 1.


Look at the logs for the container to know more about the error.

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

21695

2738

Bhairav Mehta

challenge-aido_LF-template-ros - Template solution using ROS

aido2-LFVI-sim-testing

step1-simulation

failed

ip-172-31-40-253-32125

2019-05-07 19:25:28+00:00

2019-05-07 19:28:05+00:00

0:02:37

InvalidSubmission: T [...]

InvalidSubmission:
Traceback (most recent call last):
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 174, in read_one
    nickname=self.nickname)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 255, in read_reply
    others = read_until_over(fpout, timeout=timeout, nickname=nickname)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 293, in read_until_over
    raise TimeoutError(msg) from None
TimeoutError: Timeout while reading output of node "agent".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "experiment_manager.py", line 297, in run_episode
    r: MsgReceived = agent.write_topic_and_expect('get_commands', expect='commands')
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 93, in write_topic_and_expect
    ob: MsgReceived = self.read_one(expect_topic=expect, timeout=timeout)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 236, in read_one
    raise TimeoutError(msg) from e
TimeoutError: Timeout detected on /fifos/agent-out after 1 messages. Expected topic "commands".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 610, in scoring_context
    yield cie
  File "experiment_manager.py", line 439, in <module>
    wrap(cie)
  File "experiment_manager.py", line 427, in wrap
    main(cie, logdir, attempts)
  File "experiment_manager.py", line 135, in main
    physics_dt=config.physics_dt)
  File "experiment_manager.py", line 301, in run_episode
    raise dc.InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Trouble with communication to the agent.

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible