Duckietown Challenges Home Challenges Submissions

Evaluator 1654

ID1654
evaluatorMyEval
ownerGianmarco Bernasconi
machineduckietown21
processMyEval
versiond-c:5.1.1;d-c-r:5.0.12;d-s:5.1.2
first heard
last heard
statusinactive
# evaluating
# success6 31063
# timeout
# failed
# error9 30969
# aborted
# host-error
arm
x86_64
Mac
gpu available
Number of processors
Processor frequency (MHz)
Free % of processors
RAM total (MB)
RAM free (MB)
Disk (MB)
Disk available (MB)
Docker Hub
P1
P2
Cloud simulations
PI Camera
# Duckiebots
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
311516355Amaury Camus 🇨🇭challenge-aido_LF-template-randomaido3-off-LF-real-validationeval0-visualizeerrornoMyEval0:01:35
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "main.py", line 19, in wrap
    stats = evaluate_robotarium_log(logdir, outd=outd)
  File "/project/evaluate_robotarium_log.py", line 41, in evaluate_robotarium_log
    d = yaml.load(open(mapfile).read())
FileNotFoundError: [Errno 2] No such file or directory: '/challenges/previous-steps/eval0/trajectories/map.yaml'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/duckietown_challenges/cie_concrete.py", line 676, in scoring_context
    yield cie
  File "main.py", line 34, in <module>
    wrap(cie)
  File "main.py", line 24, in wrap
    raise dc.InvalidEvaluator() from e
duckietown_challenges.exceptions.InvalidEvaluator
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
311126294Rey Wiyatno 🇨🇦challenge-aido_LF-baseline-duckietownaido3-off-LF-sim-validationstep1-simulationsuccessyesMyEval0:07:23
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median2.989390939479974
survival_time_median14.950000000000076
deviation-center-line_median0.6208176912586028
in-drivable-lane_median0


other stats
agent_compute-ego_max0.01878458579381307
agent_compute-ego_mean0.017017368793487548
agent_compute-ego_median0.016646543343861897
agent_compute-ego_min0.01546119451522827
deviation-center-line_max0.870542544660809
deviation-center-line_mean0.676638461937978
deviation-center-line_min0.572040341697189
deviation-heading_max2.3701441574852384
deviation-heading_mean1.902083318016055
deviation-heading_median2.020185012301047
deviation-heading_min1.4013815211853973
driven_any_max3.587766476950398
driven_any_mean3.044560760433511
driven_any_median3.0218742630559117
driven_any_min2.271725515587425
driven_lanedir_consec_max3.5653494016831635
driven_lanedir_consec_mean3.0144114404695346
driven_lanedir_consec_min2.2477225830986063
driven_lanedir_max3.5653494016831635
driven_lanedir_mean3.0144114404695346
driven_lanedir_median2.989390939479974
driven_lanedir_min2.2477225830986063
in-drivable-lane_max0
in-drivable-lane_mean0
in-drivable-lane_min0
per-episodes
details{"ETHZ_autolab_technical_track-0-0": {"driven_any": 2.271725515587425, "sim_physics": 0.0521248467763265, "survival_time": 14.950000000000076, "driven_lanedir": 2.2477225830986063, "sim_render-ego": 0.00714099645614624, "in-drivable-lane": 0, "agent_compute-ego": 0.01878458579381307, "deviation-heading": 2.020185012301047, "set_robot_commands": 0.009670936266581215, "deviation-center-line": 0.572040341697189, "driven_lanedir_consec": 2.2477225830986063, "sim_compute_sim_state": 0.004193565050760905, "sim_compute_performance-ego": 0.004700158437093099, "sim_compute_robot_state-ego": 0.005772005716959635}, "ETHZ_autolab_technical_track-1-0": {"driven_any": 2.757213485152363, "sim_physics": 0.04723353703816731, "survival_time": 14.950000000000076, "driven_lanedir": 2.7376124646514053, "sim_render-ego": 0.006975277264912923, "in-drivable-lane": 0, "agent_compute-ego": 0.016646543343861897, "deviation-heading": 1.6403481780947269, "set_robot_commands": 0.008690992991129557, "deviation-center-line": 0.617641578145205, "driven_lanedir_consec": 2.7376124646514053, "sim_compute_sim_state": 0.0037564659118652344, "sim_compute_performance-ego": 0.004401423136393229, "sim_compute_robot_state-ego": 0.0054229315121968585}, "ETHZ_autolab_technical_track-2-0": {"driven_any": 3.587766476950398, "sim_physics": 0.04758253812789917, "survival_time": 14.950000000000076, "driven_lanedir": 3.5319818134345207, "sim_render-ego": 0.006927462418874105, "in-drivable-lane": 0, "agent_compute-ego": 0.01546119451522827, "deviation-heading": 2.3701441574852384, "set_robot_commands": 0.009029197692871091, "deviation-center-line": 0.870542544660809, "driven_lanedir_consec": 3.5319818134345207, "sim_compute_sim_state": 0.0038524762789408367, "sim_compute_performance-ego": 0.004382890860239665, "sim_compute_robot_state-ego": 0.005415594577789307}, "ETHZ_autolab_technical_track-3-0": {"driven_any": 3.584224061421459, "sim_physics": 0.04888757546742757, "survival_time": 14.950000000000076, "driven_lanedir": 3.5653494016831635, "sim_render-ego": 0.006977479457855224, "in-drivable-lane": 0, "agent_compute-ego": 0.016435712178548178, "deviation-heading": 1.4013815211853973, "set_robot_commands": 0.009162846406300863, "deviation-center-line": 0.6208176912586028, "driven_lanedir_consec": 3.5653494016831635, "sim_compute_sim_state": 0.003877971172332763, "sim_compute_performance-ego": 0.004427278836568196, "sim_compute_robot_state-ego": 0.005543184280395508}, "ETHZ_autolab_technical_track-4-0": {"driven_any": 3.0218742630559117, "sim_physics": 0.05070802529652913, "survival_time": 14.950000000000076, "driven_lanedir": 2.989390939479974, "sim_render-ego": 0.00706391970316569, "in-drivable-lane": 0, "agent_compute-ego": 0.017758808135986327, "deviation-heading": 2.078357721013867, "set_robot_commands": 0.009229853947957356, "deviation-center-line": 0.7021501539280846, "driven_lanedir_consec": 2.989390939479974, "sim_compute_sim_state": 0.00401114543279012, "sim_compute_performance-ego": 0.004437133471171062, "sim_compute_robot_state-ego": 0.005523326396942138}}
set_robot_commands_max0.009670936266581215
set_robot_commands_mean0.009156765460968018
set_robot_commands_median0.009162846406300863
set_robot_commands_min0.008690992991129557
sim_compute_performance-ego_max0.004700158437093099
sim_compute_performance-ego_mean0.00446977694829305
sim_compute_performance-ego_median0.004427278836568196
sim_compute_performance-ego_min0.004382890860239665
sim_compute_robot_state-ego_max0.005772005716959635
sim_compute_robot_state-ego_mean0.005535408496856689
sim_compute_robot_state-ego_median0.005523326396942138
sim_compute_robot_state-ego_min0.005415594577789307
sim_compute_sim_state_max0.004193565050760905
sim_compute_sim_state_mean0.003938324769337972
sim_compute_sim_state_median0.003877971172332763
sim_compute_sim_state_min0.0037564659118652344
sim_physics_max0.0521248467763265
sim_physics_mean0.049307304541269935
sim_physics_median0.04888757546742757
sim_physics_min0.04723353703816731
sim_render-ego_max0.00714099645614624
sim_render-ego_mean0.007017027060190836
sim_render-ego_median0.006977479457855224
sim_render-ego_min0.006927462418874105
simulation-passed1
survival_time_max14.950000000000076
survival_time_mean14.950000000000076
survival_time_min14.950000000000076
No reset possible
311016302Rey Wiyatno 🇨🇦challenge-aido_LF-baseline-duckietownaido3-off-LF-sim-validationstep1-simulationsuccessyesMyEval0:07:21
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median2.8837465483863416
survival_time_median14.950000000000076
deviation-center-line_median0.7656541604983518
in-drivable-lane_median0


other stats
agent_compute-ego_max0.018660576343536375
agent_compute-ego_mean0.017155965010325114
agent_compute-ego_median0.017465937932332358
agent_compute-ego_min0.01559280792872111
deviation-center-line_max0.8620634690698115
deviation-center-line_mean0.720186603848141
deviation-center-line_min0.5207063014276943
deviation-heading_max2.9892232325639556
deviation-heading_mean2.308221829198647
deviation-heading_median2.5204870992726915
deviation-heading_min1.239634713370836
driven_any_max3.7487480096654577
driven_any_mean3.0302890941645684
driven_any_median2.9556616511500047
driven_any_min2.5015569337614987
driven_lanedir_consec_max3.730322238358534
driven_lanedir_consec_mean2.9743747852737257
driven_lanedir_consec_min2.4205107344826367
driven_lanedir_max3.730322238358534
driven_lanedir_mean2.9743747852737257
driven_lanedir_median2.8837465483863416
driven_lanedir_min2.4205107344826367
in-drivable-lane_max0
in-drivable-lane_mean0
in-drivable-lane_min0
per-episodes
details{"ETHZ_autolab_technical_track-0-0": {"driven_any": 2.5015569337614987, "sim_physics": 0.04740603685379028, "survival_time": 14.950000000000076, "driven_lanedir": 2.4205107344826367, "sim_render-ego": 0.006762684981028239, "in-drivable-lane": 0, "agent_compute-ego": 0.017796527544657388, "deviation-heading": 2.9892232325639556, "set_robot_commands": 0.008743464946746826, "deviation-center-line": 0.7656541604983518, "driven_lanedir_consec": 2.4205107344826367, "sim_compute_sim_state": 0.003745003541310628, "sim_compute_performance-ego": 0.004367448488871256, "sim_compute_robot_state-ego": 0.005372262795766194}, "ETHZ_autolab_technical_track-1-0": {"driven_any": 2.9556616511500047, "sim_physics": 0.04696698506673177, "survival_time": 14.950000000000076, "driven_lanedir": 2.8837465483863416, "sim_render-ego": 0.006882426738739014, "in-drivable-lane": 0, "agent_compute-ego": 0.017465937932332358, "deviation-heading": 2.693399780298901, "set_robot_commands": 0.008577689329783122, "deviation-center-line": 0.8052193090832434, "driven_lanedir_consec": 2.8837465483863416, "sim_compute_sim_state": 0.003649803797403971, "sim_compute_performance-ego": 0.00427301565806071, "sim_compute_robot_state-ego": 0.005296006202697754}, "ETHZ_autolab_technical_track-2-0": {"driven_any": 3.0615761525740544, "sim_physics": 0.046471025149027505, "survival_time": 14.950000000000076, "driven_lanedir": 2.988648985461994, "sim_render-ego": 0.006828200022379557, "in-drivable-lane": 0, "agent_compute-ego": 0.01559280792872111, "deviation-heading": 2.5204870992726915, "set_robot_commands": 0.008993215560913086, "deviation-center-line": 0.8620634690698115, "driven_lanedir_consec": 2.988648985461994, "sim_compute_sim_state": 0.003845154444376627, "sim_compute_performance-ego": 0.004300076961517334, "sim_compute_robot_state-ego": 0.005325399239857992}, "ETHZ_autolab_technical_track-3-0": {"driven_any": 3.7487480096654577, "sim_physics": 0.04809330781300863, "survival_time": 14.950000000000076, "driven_lanedir": 3.730322238358534, "sim_render-ego": 0.006980834007263184, "in-drivable-lane": 0, "agent_compute-ego": 0.016263975302378338, "deviation-heading": 1.239634713370836, "set_robot_commands": 0.009037465254465738, "deviation-center-line": 0.5207063014276943, "driven_lanedir_consec": 3.730322238358534, "sim_compute_sim_state": 0.003841888109842936, "sim_compute_performance-ego": 0.004399154186248779, "sim_compute_robot_state-ego": 0.005519095261891683}, "ETHZ_autolab_technical_track-4-0": {"driven_any": 2.8839027236718273, "sim_physics": 0.04833846966425578, "survival_time": 14.950000000000076, "driven_lanedir": 2.848645419679123, "sim_render-ego": 0.007107581297556559, "in-drivable-lane": 0, "agent_compute-ego": 0.018660576343536375, "deviation-heading": 2.098364320486851, "set_robot_commands": 0.008864172299702962, "deviation-center-line": 0.6472897791616044, "driven_lanedir_consec": 2.848645419679123, "sim_compute_sim_state": 0.003836554686228434, "sim_compute_performance-ego": 0.004481995105743408, "sim_compute_robot_state-ego": 0.005598954359690348}}
set_robot_commands_max0.009037465254465738
set_robot_commands_mean0.008843201478322345
set_robot_commands_median0.008864172299702962
set_robot_commands_min0.008577689329783122
sim_compute_performance-ego_max0.004481995105743408
sim_compute_performance-ego_mean0.004364338080088298
sim_compute_performance-ego_median0.004367448488871256
sim_compute_performance-ego_min0.00427301565806071
sim_compute_robot_state-ego_max0.005598954359690348
sim_compute_robot_state-ego_mean0.005422343571980795
sim_compute_robot_state-ego_median0.005372262795766194
sim_compute_robot_state-ego_min0.005296006202697754
sim_compute_sim_state_max0.003845154444376627
sim_compute_sim_state_mean0.00378368091583252
sim_compute_sim_state_median0.003836554686228434
sim_compute_sim_state_min0.003649803797403971
sim_physics_max0.04833846966425578
sim_physics_mean0.047455164909362794
sim_physics_median0.04740603685379028
sim_physics_min0.046471025149027505
sim_render-ego_max0.007107581297556559
sim_render-ego_mean0.006912345409393311
sim_render-ego_median0.006882426738739014
sim_render-ego_min0.006762684981028239
simulation-passed1
survival_time_max14.950000000000076
survival_time_mean14.950000000000076
survival_time_min14.950000000000076
No reset possible
310946308Rey Wiyatno 🇨🇦challenge-aido_LF-baseline-duckietownaido3-off-LF-sim-validationstep1-simulationsuccessyesMyEval0:07:22
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median2.787792570059231
survival_time_median14.950000000000076
deviation-center-line_median0.887071824845663
in-drivable-lane_median0


other stats
agent_compute-ego_max0.02000847180684407
agent_compute-ego_mean0.019219390869140627
agent_compute-ego_median0.019109227657318116
agent_compute-ego_min0.01842610915501913
deviation-center-line_max1.0644523866769888
deviation-center-line_mean0.8542193495354276
deviation-center-line_min0.6254732535936984
deviation-heading_max3.252821516952374
deviation-heading_mean2.9294176345504255
deviation-heading_median2.84628413210395
deviation-heading_min2.4741905848165535
driven_any_max2.9209296448187443
driven_any_mean2.804959637119363
driven_any_median2.8587694425018157
driven_any_min2.612750940742663
driven_lanedir_consec_max2.8271512141300335
driven_lanedir_consec_mean2.7299518324842835
driven_lanedir_consec_min2.5257871257056945
driven_lanedir_max2.8271512141300335
driven_lanedir_mean2.7299518324842835
driven_lanedir_median2.787792570059231
driven_lanedir_min2.5257871257056945
in-drivable-lane_max0
in-drivable-lane_mean0
in-drivable-lane_min0
per-episodes
details{"ETHZ_autolab_technical_track-0-0": {"driven_any": 2.612750940742663, "sim_physics": 0.048211587270100914, "survival_time": 14.950000000000076, "driven_lanedir": 2.5257871257056945, "sim_render-ego": 0.007036640644073487, "in-drivable-lane": 0, "agent_compute-ego": 0.01842610915501913, "deviation-heading": 3.252821516952374, "set_robot_commands": 0.008874514102935792, "deviation-center-line": 0.9910285597106778, "driven_lanedir_consec": 2.5257871257056945, "sim_compute_sim_state": 0.003690110047658285, "sim_compute_performance-ego": 0.0044718623161315914, "sim_compute_robot_state-ego": 0.005456126530965169}, "ETHZ_autolab_technical_track-1-0": {"driven_any": 2.770180478929309, "sim_physics": 0.050099507967631025, "survival_time": 14.950000000000076, "driven_lanedir": 2.7031882035934904, "sim_render-ego": 0.007216227054595947, "in-drivable-lane": 0, "agent_compute-ego": 0.019104133447011312, "deviation-heading": 2.8247114869974967, "set_robot_commands": 0.008844610850016275, "deviation-center-line": 0.887071824845663, "driven_lanedir_consec": 2.7031882035934904, "sim_compute_sim_state": 0.0037657411893208823, "sim_compute_performance-ego": 0.0044766759872436526, "sim_compute_robot_state-ego": 0.005564608573913574}, "ETHZ_autolab_technical_track-2-0": {"driven_any": 2.9209296448187443, "sim_physics": 0.05104555686314901, "survival_time": 14.950000000000076, "driven_lanedir": 2.8271512141300335, "sim_render-ego": 0.007547637621561686, "in-drivable-lane": 0, "agent_compute-ego": 0.019109227657318116, "deviation-heading": 3.249080451881752, "set_robot_commands": 0.009464761416117353, "deviation-center-line": 1.0644523866769888, "driven_lanedir_consec": 2.8271512141300335, "sim_compute_sim_state": 0.003976021607716878, "sim_compute_performance-ego": 0.00483592430750529, "sim_compute_robot_state-ego": 0.005965252717336019}, "ETHZ_autolab_technical_track-3-0": {"driven_any": 2.8587694425018157, "sim_physics": 0.05170296748479208, "survival_time": 14.950000000000076, "driven_lanedir": 2.8058400489329687, "sim_render-ego": 0.007595949172973633, "in-drivable-lane": 0, "agent_compute-ego": 0.0194490122795105, "deviation-heading": 2.4741905848165535, "set_robot_commands": 0.00946940819422404, "deviation-center-line": 0.6254732535936984, "driven_lanedir_consec": 2.8058400489329687, "sim_compute_sim_state": 0.0040676164627075195, "sim_compute_performance-ego": 0.004823048909505208, "sim_compute_robot_state-ego": 0.006017458438873291}, "ETHZ_autolab_technical_track-4-0": {"driven_any": 2.8621676786042816, "sim_physics": 0.05077802658081055, "survival_time": 14.950000000000076, "driven_lanedir": 2.787792570059231, "sim_render-ego": 0.00728938897450765, "in-drivable-lane": 0, "agent_compute-ego": 0.02000847180684407, "deviation-heading": 2.84628413210395, "set_robot_commands": 0.00920387585957845, "deviation-center-line": 0.70307072285011, "driven_lanedir_consec": 2.787792570059231, "sim_compute_sim_state": 0.004064633051554362, "sim_compute_performance-ego": 0.004595167636871338, "sim_compute_robot_state-ego": 0.0057439915339152015}}
set_robot_commands_max0.00946940819422404
set_robot_commands_mean0.00917143408457438
set_robot_commands_median0.00920387585957845
set_robot_commands_min0.008844610850016275
sim_compute_performance-ego_max0.00483592430750529
sim_compute_performance-ego_mean0.004640535831451416
sim_compute_performance-ego_median0.004595167636871338
sim_compute_performance-ego_min0.0044718623161315914
sim_compute_robot_state-ego_max0.006017458438873291
sim_compute_robot_state-ego_mean0.005749487559000651
sim_compute_robot_state-ego_median0.0057439915339152015
sim_compute_robot_state-ego_min0.005456126530965169
sim_compute_sim_state_max0.0040676164627075195
sim_compute_sim_state_mean0.0039128244717915855
sim_compute_sim_state_median0.003976021607716878
sim_compute_sim_state_min0.003690110047658285
sim_physics_max0.05170296748479208
sim_physics_mean0.05036752923329672
sim_physics_median0.05077802658081055
sim_physics_min0.048211587270100914
sim_render-ego_max0.007595949172973633
sim_render-ego_mean0.007337168693542481
sim_render-ego_median0.00728938897450765
sim_render-ego_min0.007036640644073487
simulation-passed1
survival_time_max14.950000000000076
survival_time_mean14.950000000000076
survival_time_min14.950000000000076
No reset possible
310866314Rey Wiyatno 🇨🇦challenge-aido_LF-baseline-duckietownaido3-off-LF-sim-validationstep1-simulationsuccessyesMyEval0:07:20
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median3.076713635507828
survival_time_median14.950000000000076
deviation-center-line_median0.7145369124326418
in-drivable-lane_median0


other stats
agent_compute-ego_max0.021255333423614502
agent_compute-ego_mean0.019075394471486408
agent_compute-ego_median0.01888407309850057
agent_compute-ego_min0.01570566733678182
deviation-center-line_max0.960601420726504
deviation-center-line_mean0.7411394310355301
deviation-center-line_min0.5891849152797769
deviation-heading_max3.308378169231783
deviation-heading_mean1.916747216174866
deviation-heading_median1.605428557447187
deviation-heading_min1.183970635131239
driven_any_max3.486225119126711
driven_any_mean3.0195417781453258
driven_any_median3.0923930947042932
driven_any_min2.593015414317566
driven_lanedir_consec_max3.4725342663339167
driven_lanedir_consec_mean2.9825584769893236
driven_lanedir_consec_min2.5589401514369974
driven_lanedir_max3.4725342663339167
driven_lanedir_mean2.9825584769893236
driven_lanedir_median3.076713635507828
driven_lanedir_min2.5589401514369974
in-drivable-lane_max0
in-drivable-lane_mean0
in-drivable-lane_min0
per-episodes
details{"ETHZ_autolab_technical_track-0-0": {"driven_any": 2.593015414317566, "sim_physics": 0.05101423104604085, "survival_time": 14.950000000000076, "driven_lanedir": 2.5589401514369974, "sim_render-ego": 0.00715985377629598, "in-drivable-lane": 0, "agent_compute-ego": 0.02080286105473836, "deviation-heading": 2.021321897315301, "set_robot_commands": 0.00906137466430664, "deviation-center-line": 0.8503953463080062, "driven_lanedir_consec": 2.5589401514369974, "sim_compute_sim_state": 0.003826134204864502, "sim_compute_performance-ego": 0.004678115049997966, "sim_compute_robot_state-ego": 0.005686423778533936}, "ETHZ_autolab_technical_track-1-0": {"driven_any": 3.0923930947042932, "sim_physics": 0.05290771802266439, "survival_time": 14.950000000000076, "driven_lanedir": 3.076713635507828, "sim_render-ego": 0.007510777314503988, "in-drivable-lane": 0, "agent_compute-ego": 0.021255333423614502, "deviation-heading": 1.4646368217488193, "set_robot_commands": 0.00927293618520101, "deviation-center-line": 0.7145369124326418, "driven_lanedir_consec": 3.076713635507828, "sim_compute_sim_state": 0.004013783137003581, "sim_compute_performance-ego": 0.004707010587056478, "sim_compute_robot_state-ego": 0.00582356055577596}, "ETHZ_autolab_technical_track-2-0": {"driven_any": 2.7076625024752787, "sim_physics": 0.046506327788035075, "survival_time": 14.950000000000076, "driven_lanedir": 2.6083438715961793, "sim_render-ego": 0.006950419743855794, "in-drivable-lane": 0, "agent_compute-ego": 0.01570566733678182, "deviation-heading": 3.308378169231783, "set_robot_commands": 0.009014410972595217, "deviation-center-line": 0.960601420726504, "driven_lanedir_consec": 2.6083438715961793, "sim_compute_sim_state": 0.00372053861618042, "sim_compute_performance-ego": 0.004390749136606852, "sim_compute_robot_state-ego": 0.005417338212331136}, "ETHZ_autolab_technical_track-3-0": {"driven_any": 3.486225119126711, "sim_physics": 0.047238152821858725, "survival_time": 14.950000000000076, "driven_lanedir": 3.4725342663339167, "sim_render-ego": 0.00696592648824056, "in-drivable-lane": 0, "agent_compute-ego": 0.018729037443796795, "deviation-heading": 1.183970635131239, "set_robot_commands": 0.00853155771891276, "deviation-center-line": 0.5909785604307218, "driven_lanedir_consec": 3.4725342663339167, "sim_compute_sim_state": 0.0036406572659810382, "sim_compute_performance-ego": 0.00435383955637614, "sim_compute_robot_state-ego": 0.005441530545552572}, "ETHZ_autolab_technical_track-4-0": {"driven_any": 3.218412760102779, "sim_physics": 0.04796937783559163, "survival_time": 14.950000000000076, "driven_lanedir": 3.1962604600716973, "sim_render-ego": 0.00710783322652181, "in-drivable-lane": 0, "agent_compute-ego": 0.01888407309850057, "deviation-heading": 1.605428557447187, "set_robot_commands": 0.00866278092066447, "deviation-center-line": 0.5891849152797769, "driven_lanedir_consec": 3.1962604600716973, "sim_compute_sim_state": 0.0037118903795878098, "sim_compute_performance-ego": 0.004441097577412923, "sim_compute_robot_state-ego": 0.005499724547068278}}
set_robot_commands_max0.00927293618520101
set_robot_commands_mean0.008908612092336019
set_robot_commands_median0.009014410972595217
set_robot_commands_min0.00853155771891276
sim_compute_performance-ego_max0.004707010587056478
sim_compute_performance-ego_mean0.0045141623814900726
sim_compute_performance-ego_median0.004441097577412923
sim_compute_performance-ego_min0.00435383955637614
sim_compute_robot_state-ego_max0.00582356055577596
sim_compute_robot_state-ego_mean0.005573715527852376
sim_compute_robot_state-ego_median0.005499724547068278
sim_compute_robot_state-ego_min0.005417338212331136
sim_compute_sim_state_max0.004013783137003581
sim_compute_sim_state_mean0.0037826007207234697
sim_compute_sim_state_median0.00372053861618042
sim_compute_sim_state_min0.0036406572659810382
sim_physics_max0.05290771802266439
sim_physics_mean0.049127161502838135
sim_physics_median0.04796937783559163
sim_physics_min0.046506327788035075
sim_render-ego_max0.007510777314503988
sim_render-ego_mean0.007138962109883627
sim_render-ego_median0.00710783322652181
sim_render-ego_min0.006950419743855794
simulation-passed1
survival_time_max14.950000000000076
survival_time_mean14.950000000000076
survival_time_min14.950000000000076
No reset possible
310716322Rey Wiyatno 🇨🇦challenge-aido_LF-baseline-duckietownaido3-off-LF-sim-validationstep1-simulationsuccessyesMyEval0:07:24
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median3.3822661545444843
survival_time_median14.950000000000076
deviation-center-line_median0.9566875661775468
in-drivable-lane_median0


other stats
agent_compute-ego_max0.02035071055094401
agent_compute-ego_mean0.01926685333251953
agent_compute-ego_median0.019600438276926677
agent_compute-ego_min0.01767043352127075
deviation-center-line_max1.055156914832329
deviation-center-line_mean0.9566579522347376
deviation-center-line_min0.8917917279991175
deviation-heading_max3.537610156953064
deviation-heading_mean3.00264776922766
deviation-heading_median2.9570580584708765
deviation-heading_min2.352402175086589
driven_any_max4.131779463461953
driven_any_mean3.5506791047776525
driven_any_median3.4543675781173593
driven_any_min2.986997837264857
driven_lanedir_consec_max4.053582358731587
driven_lanedir_consec_mean3.453883721614563
driven_lanedir_consec_min2.877043900337325
driven_lanedir_max4.053582358731587
driven_lanedir_mean3.453883721614563
driven_lanedir_median3.3822661545444843
driven_lanedir_min2.877043900337325
in-drivable-lane_max0
in-drivable-lane_mean0
in-drivable-lane_min0
per-episodes
details{"ETHZ_autolab_technical_track-0-0": {"driven_any": 2.986997837264857, "sim_physics": 0.05196542819341024, "survival_time": 14.950000000000076, "driven_lanedir": 2.877043900337325, "sim_render-ego": 0.007091030279795329, "in-drivable-lane": 0, "agent_compute-ego": 0.020119907855987548, "deviation-heading": 3.5098621520171056, "set_robot_commands": 0.009344953695933024, "deviation-center-line": 1.055156914832329, "driven_lanedir_consec": 2.877043900337325, "sim_compute_sim_state": 0.004062480926513672, "sim_compute_performance-ego": 0.004537130196889241, "sim_compute_robot_state-ego": 0.005598161220550537}, "ETHZ_autolab_technical_track-1-0": {"driven_any": 3.4543675781173593, "sim_physics": 0.04988369226455688, "survival_time": 14.950000000000076, "driven_lanedir": 3.3822661545444843, "sim_render-ego": 0.007064514954884847, "in-drivable-lane": 0, "agent_compute-ego": 0.019600438276926677, "deviation-heading": 2.6563063036106636, "set_robot_commands": 0.009058087666829428, "deviation-center-line": 0.9566875661775468, "driven_lanedir_consec": 3.3822661545444843, "sim_compute_sim_state": 0.00407756487528483, "sim_compute_performance-ego": 0.004431037902832031, "sim_compute_robot_state-ego": 0.005561001300811767}, "ETHZ_autolab_technical_track-2-0": {"driven_any": 4.131779463461953, "sim_physics": 0.050957260131835935, "survival_time": 14.950000000000076, "driven_lanedir": 4.053582358731587, "sim_render-ego": 0.007303698062896729, "in-drivable-lane": 0, "agent_compute-ego": 0.01767043352127075, "deviation-heading": 2.352402175086589, "set_robot_commands": 0.009603611628214518, "deviation-center-line": 0.8917917279991175, "driven_lanedir_consec": 4.053582358731587, "sim_compute_sim_state": 0.004088157018025716, "sim_compute_performance-ego": 0.004709060986836751, "sim_compute_robot_state-ego": 0.005860195954640706}, "ETHZ_autolab_technical_track-3-0": {"driven_any": 3.830894575268508, "sim_physics": 0.05152835210164388, "survival_time": 14.950000000000076, "driven_lanedir": 3.720557640630595, "sim_render-ego": 0.0072316948572794595, "in-drivable-lane": 0, "agent_compute-ego": 0.01859277645746867, "deviation-heading": 2.9570580584708765, "set_robot_commands": 0.0094252347946167, "deviation-center-line": 0.9785948085667976, "driven_lanedir_consec": 3.720557640630595, "sim_compute_sim_state": 0.004075928529103597, "sim_compute_performance-ego": 0.004558758735656738, "sim_compute_robot_state-ego": 0.005727011362711589}, "ETHZ_autolab_technical_track-4-0": {"driven_any": 3.3493560697755855, "sim_physics": 0.05094654719034831, "survival_time": 14.950000000000076, "driven_lanedir": 3.2359685538288216, "sim_render-ego": 0.007121841112772624, "in-drivable-lane": 0, "agent_compute-ego": 0.02035071055094401, "deviation-heading": 3.537610156953064, "set_robot_commands": 0.009029051462809246, "deviation-center-line": 0.9010587435978972, "driven_lanedir_consec": 3.2359685538288216, "sim_compute_sim_state": 0.00390167236328125, "sim_compute_performance-ego": 0.004561842282613119, "sim_compute_robot_state-ego": 0.0057741657892862955}}
set_robot_commands_max0.009603611628214518
set_robot_commands_mean0.009292187849680583
set_robot_commands_median0.009344953695933024
set_robot_commands_min0.009029051462809246
sim_compute_performance-ego_max0.004709060986836751
sim_compute_performance-ego_mean0.004559566020965576
sim_compute_performance-ego_median0.004558758735656738
sim_compute_performance-ego_min0.004431037902832031
sim_compute_robot_state-ego_max0.005860195954640706
sim_compute_robot_state-ego_mean0.005704107125600179
sim_compute_robot_state-ego_median0.005727011362711589
sim_compute_robot_state-ego_min0.005561001300811767
sim_compute_sim_state_max0.004088157018025716
sim_compute_sim_state_mean0.004041160742441813
sim_compute_sim_state_median0.004075928529103597
sim_compute_sim_state_min0.00390167236328125
sim_physics_max0.05196542819341024
sim_physics_mean0.05105625597635906
sim_physics_median0.050957260131835935
sim_physics_min0.04988369226455688
sim_render-ego_max0.007303698062896729
sim_render-ego_mean0.007162555853525797
sim_render-ego_median0.007121841112772624
sim_render-ego_min0.007064514954884847
simulation-passed1
survival_time_max14.950000000000076
survival_time_mean14.950000000000076
survival_time_min14.950000000000076
No reset possible
310636329YUNG-CHIEN CHOU 🇹🇼challenge-aido_LF-template-randomaido3-off-LF-sim-validationstep1-simulationsuccessyesMyEval0:02:48
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.4838163264350306
survival_time_median1.7000000000000008
deviation-center-line_median0.08751730474052305
in-drivable-lane_median0


other stats
agent_compute-ego_max0.014212085650517393
agent_compute-ego_mean0.013367453782707688
agent_compute-ego_median0.013197765630834244
agent_compute-ego_min0.0125480055809021
deviation-center-line_max0.10683019887032848
deviation-center-line_mean0.08719800587294524
deviation-center-line_min0.07065220523066396
deviation-heading_max0.5050851849885492
deviation-heading_mean0.37472646794797326
deviation-heading_median0.3711400756311837
deviation-heading_min0.242154099114489
driven_any_max1.224736399271231
driven_any_mean0.7001682289165533
driven_any_median0.6334906616120762
driven_any_min0.311949329092399
driven_lanedir_consec_max0.6642632200434848
driven_lanedir_consec_mean0.47031102562542887
driven_lanedir_consec_min0.28855079430839536
driven_lanedir_max0.6642632200434848
driven_lanedir_mean0.47031102562542887
driven_lanedir_median0.4838163264350306
driven_lanedir_min0.28855079430839536
in-drivable-lane_max1.5999999999999963
in-drivable-lane_mean0.4499999999999989
in-drivable-lane_min0
per-episodes
details{"ETHZ_autolab_technical_track-0-0": {"driven_any": 1.0078896084088962, "sim_physics": 0.05147772568922777, "survival_time": 2.5999999999999988, "driven_lanedir": 0.6642632200434848, "sim_render-ego": 0.007833274511190562, "in-drivable-lane": 0.6499999999999977, "agent_compute-ego": 0.014212085650517393, "deviation-heading": 0.5050851849885492, "set_robot_commands": 0.005973462875072773, "deviation-center-line": 0.07065220523066396, "driven_lanedir_consec": 0.6642632200434848, "sim_compute_sim_state": 0.003070803788992075, "sim_compute_performance-ego": 0.0049063379947955794, "sim_compute_robot_state-ego": 0.005852942283336933}, "ETHZ_autolab_technical_track-1-0": {"driven_any": 0.311949329092399, "sim_physics": 0.05028897523880005, "survival_time": 1.0000000000000002, "driven_lanedir": 0.3030566281970668, "sim_render-ego": 0.007065272331237793, "in-drivable-lane": 0, "agent_compute-ego": 0.0125480055809021, "deviation-heading": 0.242154099114489, "set_robot_commands": 0.005612635612487793, "deviation-center-line": 0.08751730474052305, "driven_lanedir_consec": 0.3030566281970668, "sim_compute_sim_state": 0.0029543280601501466, "sim_compute_performance-ego": 0.004218459129333496, "sim_compute_robot_state-ego": 0.0051962494850158695}, "ETHZ_autolab_technical_track-2-0": {"driven_any": 0.6334906616120762, "sim_physics": 0.04870745013741886, "survival_time": 1.7000000000000008, "driven_lanedir": 0.6118681591431665, "sim_render-ego": 0.007423583198996151, "in-drivable-lane": 0, "agent_compute-ego": 0.013197765630834244, "deviation-heading": 0.3711400756311837, "set_robot_commands": 0.005790310747483198, "deviation-center-line": 0.10683019887032848, "driven_lanedir_consec": 0.6118681591431665, "sim_compute_sim_state": 0.0030317236395443186, "sim_compute_performance-ego": 0.0044124687419218175, "sim_compute_robot_state-ego": 0.0054301794837502875}, "ETHZ_autolab_technical_track-3-0": {"driven_any": 1.224736399271231, "sim_physics": 0.05109729141485496, "survival_time": 3.049999999999997, "driven_lanedir": 0.4838163264350306, "sim_render-ego": 0.007836478655455543, "in-drivable-lane": 1.5999999999999963, "agent_compute-ego": 0.01403570566021028, "deviation-heading": 0.29149367303980006, "set_robot_commands": 0.006136608905479556, "deviation-center-line": 0.10020114979294129, "driven_lanedir_consec": 0.4838163264350306, "sim_compute_sim_state": 0.0031950278360335553, "sim_compute_performance-ego": 0.00476673782848921, "sim_compute_robot_state-ego": 0.005893210895725938}, "ETHZ_autolab_technical_track-4-0": {"driven_any": 0.32277514619816383, "sim_physics": 0.05185674537311901, "survival_time": 1.1000000000000003, "driven_lanedir": 0.28855079430839536, "sim_render-ego": 0.007243665781888095, "in-drivable-lane": 0, "agent_compute-ego": 0.01284370639107444, "deviation-heading": 0.4637593069658444, "set_robot_commands": 0.005725080316716974, "deviation-center-line": 0.07078917073026944, "driven_lanedir_consec": 0.28855079430839536, "sim_compute_sim_state": 0.0030702569267966533, "sim_compute_performance-ego": 0.004487124356356534, "sim_compute_robot_state-ego": 0.005600571632385254}}
set_robot_commands_max0.006136608905479556
set_robot_commands_mean0.005847619691448059
set_robot_commands_median0.005790310747483198
set_robot_commands_min0.005612635612487793
sim_compute_performance-ego_max0.0049063379947955794
sim_compute_performance-ego_mean0.004558225610179328
sim_compute_performance-ego_median0.004487124356356534
sim_compute_performance-ego_min0.004218459129333496
sim_compute_robot_state-ego_max0.005893210895725938
sim_compute_robot_state-ego_mean0.005594630756042856
sim_compute_robot_state-ego_median0.005600571632385254
sim_compute_robot_state-ego_min0.0051962494850158695
sim_compute_sim_state_max0.0031950278360335553
sim_compute_sim_state_mean0.00306442805030335
sim_compute_sim_state_median0.0030702569267966533
sim_compute_sim_state_min0.0029543280601501466
sim_physics_max0.05185674537311901
sim_physics_mean0.05068563757068413
sim_physics_median0.05109729141485496
sim_physics_min0.04870745013741886
sim_render-ego_max0.007836478655455543
sim_render-ego_mean0.0074804548957536285
sim_render-ego_median0.007423583198996151
sim_render-ego_min0.007065272331237793
simulation-passed1
survival_time_max3.049999999999997
survival_time_mean1.8899999999999997
survival_time_min1.0000000000000002
No reset possible
310576345Susanne KellerPredictorLastaido3-off-LF-sim-validationstep1-simulationerroryesMyEval0:00:42
Unexpected exception [...]
Unexpected exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/zuper_nodes/compatibility.py", line 20, in check_compatible_protocol
    raise IncompatibleProtocol(msg)
zuper_nodes.compatibility.IncompatibleProtocol: First protocol misses input "get_commands".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/duckietown_challenges/cie_concrete.py", line 676, in scoring_context
    yield cie
  File "experiment_manager.py", line 446, in <module>
    wrap(cie)
  File "experiment_manager.py", line 434, in wrap
    main(cie, logdir, attempts)
  File "experiment_manager.py", line 79, in main
    agent_ci._get_node_protocol(timeout=config.timeout_initialization)
  File "/usr/local/lib/python3.7/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 88, in _get_node_protocol
    check_compatible_protocol(self.node_protocol, self.expect_protocol)
  File "/usr/local/lib/python3.7/site-packages/zuper_nodes/compatibility.py", line 43, in check_compatible_protocol
    raise IncompatibleProtocol(msg) from e
zuper_nodes.compatibility.IncompatibleProtocol: Cannot say that p1 is a sub-protocol of p2
p1: |InteractionProtocol(description='An estimator receives a stream of values and must predict the next value.', inputs={'seed': <class 'int'>, 'observations': <class 'float'>, 'get_prediction': <class 'NoneType'>}, outputs={'prediction': <class 'float'>}, language='(in:seed)? ; ((in:observations | (in:get_prediction ; out:prediction))*)')
p2: |InteractionProtocol(description='Generic protocol for an agent that receives "observations" and responds \nwith "commands".\n\n"episode_start" marks the beginning of an episode.', inputs={'observations': typing.Any, 'seed': <class 'int'>, 'get_commands': <class 'NoneType'>, 'episode_start': <class 'aido_schemas.protocol_agent.EpisodeStart'>}, outputs={'commands': typing.Any}, language='(in:seed)? ; ((in:episode_start ; ((in:observations | (in:get_commands ; out:commands))*))*)')
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
310386295Rey Wiyatno 🇨🇦challenge-aido_LF-baseline-duckietownaido3-off-LF-sim-validationstep1-simulationerrornoMyEval0:00:38
The container "simul [...]
The container "simulator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
310266300Rey Wiyatno 🇨🇦challenge-aido_LF-baseline-duckietownaido3-off-LF-sim-validationstep1-simulationerrornoMyEval0:00:39
The container "simul [...]
The container "simulator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
310116303Rey Wiyatno 🇨🇦challenge-aido_LF-baseline-duckietownaido3-off-LF-sim-validationstep1-simulationerrornoMyEval0:00:40
The container "simul [...]
The container "simulator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
310076320Rey Wiyatno 🇨🇦challenge-aido_LF-baseline-duckietownaido3-off-LF-sim-validationstep1-simulationerrornoMyEval0:00:39
The container "simul [...]
The container "simulator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
309926321Rey Wiyatno 🇨🇦challenge-aido_LF-baseline-duckietownaido3-off-LF-sim-validationstep1-simulationerrornoMyEval0:00:58
The container "simul [...]
The container "simulator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
309806353Amaury Camus 🇨🇭challenge-aido_LF-template-randomaido3-off-LF-sim-validationstep1-simulationerrornoMyEval0:00:36
The container "simul [...]
The container "simulator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
309696348Susanne KellerPredictorLastaido3-off-LF-sim-validationstep1-simulationerrornoMyEval0:00:40
The container "simul [...]
The container "simulator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible