Duckietown Challenges Home Challenges Submissions

Submission 2814

Submission2814
Competingyes
Challengeaido2-LF-sim-testing
UserLiam Paull 🇨🇦
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
Jobsstep1-simulation: 22758
Next
User labelchallenge-aido_LF-template-random
Admin priority50
Blessingn/a
User priority50

The highlights are available only to the owner and the admins.

Evaluation jobs for this submission

Show only up-to-date jobs
Job IDstepstatusup to datedate starteddate completeddurationmessage
22758step1-simulationsuccessyes0:12:21
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.5723310224856362
survival_time_median2.549999999999999
deviation-center-line_median0.10099507669212504
in-drivable-lane_median0.4499999999999984


other stats
agent_compute-ego_max0.08406064033508301
agent_compute-ego_mean0.07988255274132959
agent_compute-ego_median0.07933453952564913
agent_compute-ego_min0.0790115197499593
deviation-center-line_max0.2838635700333795
deviation-center-line_mean0.12786165368054708
deviation-center-line_min0.07112529346797256
deviation-heading_max0.5289022616823588
deviation-heading_mean0.3391682685652287
deviation-heading_median0.32928385103533875
deviation-heading_min0.19401296908406307
driven_any_max2.1706243860917107
driven_any_mean0.9759490378276146
driven_any_median1.0117660515775693
driven_any_min0.23626560261560617
driven_lanedir_consec_max1.919400421063832
driven_lanedir_consec_mean0.6896171461819246
driven_lanedir_consec_min0.2264965947033466
driven_lanedir_max1.919400421063832
driven_lanedir_mean0.6896171461819246
driven_lanedir_median0.5723310224856362
driven_lanedir_min0.2264965947033466
in-drivable-lane_max1.7999999999999936
in-drivable-lane_mean0.5899999999999982
in-drivable-lane_min0
per-episodes
details{"ETHZ_autolab_technical_track-0-0": {"driven_any": 1.0243289631429708, "sim_physics": 0.04049600053716589, "survival_time": 2.6999999999999984, "driven_lanedir": 0.5723310224856362, "sim_render-ego": 0.031716991353918006, "in-drivable-lane": 0.9999999999999976, "agent_compute-ego": 0.0811302970956873, "deviation-heading": 0.19401296908406307, "set_robot_commands": 0.04913304028687654, "deviation-center-line": 0.15907928712407893, "driven_lanedir_consec": 0.5723310224856362, "sim_compute_sim_state": 0.01991147465176053, "sim_compute_performance-ego": 0.03488934481585467, "sim_compute_robot_state-ego": 0.036982571637188946}, "ETHZ_autolab_technical_track-1-0": {"driven_any": 0.9933867744747804, "sim_physics": 0.03971535551781748, "survival_time": 2.549999999999999, "driven_lanedir": 0.8010406009604816, "sim_render-ego": 0.032250380983539655, "in-drivable-lane": 0.29999999999999893, "agent_compute-ego": 0.07924856391607546, "deviation-heading": 0.5289022616823588, "set_robot_commands": 0.048740022322710824, "deviation-center-line": 0.13005513204193506, "driven_lanedir_consec": 0.8010406009604816, "sim_compute_sim_state": 0.020247945598527495, "sim_compute_performance-ego": 0.035019977419984104, "sim_compute_robot_state-ego": 0.03704035515878715}, "ETHZ_autolab_technical_track-2-0": {"driven_any": 0.23626560261560617, "sim_physics": 0.04346682043636546, "survival_time": 0.8500000000000002, "driven_lanedir": 0.2264965947033466, "sim_render-ego": 0.0321493008557488, "in-drivable-lane": 0, "agent_compute-ego": 0.0793310614193187, "deviation-heading": 0.23048898560984332, "set_robot_commands": 0.048760259852689856, "deviation-center-line": 0.0788839760023766, "driven_lanedir_consec": 0.2264965947033466, "sim_compute_sim_state": 0.01981369186850155, "sim_compute_performance-ego": 0.03513449781081256, "sim_compute_robot_state-ego": 0.0368345344767851}, "ETHZ_autolab_technical_track-3-0": {"driven_any": 0.4346971317320432, "sim_physics": 0.04195998191833496, "survival_time": 1.2500000000000004, "driven_lanedir": 0.40985475638935265, "sim_render-ego": 0.0321730899810791, "in-drivable-lane": 0, "agent_compute-ego": 0.08406064033508301, "deviation-heading": 0.4138862827398061, "set_robot_commands": 0.048550891876220706, "deviation-center-line": 0.08421324241865966, "driven_lanedir_consec": 0.40985475638935265, "sim_compute_sim_state": 0.019958391189575195, "sim_compute_performance-ego": 0.03463831901550293, "sim_compute_robot_state-ego": 0.037802114486694335}, "ETHZ_autolab_technical_track-4-0": {"driven_any": 1.618214457700942, "sim_physics": 0.04047008405757856, "survival_time": 3.949999999999994, "driven_lanedir": 0.7890999076769956, "sim_render-ego": 0.032347522204435326, "in-drivable-lane": 1.7999999999999936, "agent_compute-ego": 0.07919234263746044, "deviation-heading": 0.23606825392187256, "set_robot_commands": 0.04951326454742045, "deviation-center-line": 0.2072381278624381, "driven_lanedir_consec": 0.7890999076769956, "sim_compute_sim_state": 0.02020062977754617, "sim_compute_performance-ego": 0.035157816319526, "sim_compute_robot_state-ego": 0.03743522076667109}, "ETHZ_autolab_technical_track-5-0": {"driven_any": 1.0038514956833002, "sim_physics": 0.039853100873986066, "survival_time": 2.4499999999999993, "driven_lanedir": 0.7531703123735944, "sim_render-ego": 0.03202736621000329, "in-drivable-lane": 0.4499999999999984, "agent_compute-ego": 0.07905399556062659, "deviation-heading": 0.4596437246381204, "set_robot_commands": 0.0493373627565345, "deviation-center-line": 0.08968283443958727, "driven_lanedir_consec": 0.7531703123735944, "sim_compute_sim_state": 0.02004294979329012, "sim_compute_performance-ego": 0.03492199644750478, "sim_compute_robot_state-ego": 0.03737764942402742}, "ETHZ_autolab_technical_track-6-0": {"driven_any": 1.0542898001504375, "sim_physics": 0.03865353124482291, "survival_time": 2.799999999999998, "driven_lanedir": 0.5389398655375865, "sim_render-ego": 0.03234850083078657, "in-drivable-lane": 1.1499999999999972, "agent_compute-ego": 0.07902286733899798, "deviation-heading": 0.4109275195546752, "set_robot_commands": 0.05145572338785444, "deviation-center-line": 0.10786249131209884, "driven_lanedir_consec": 0.5389398655375865, "sim_compute_sim_state": 0.02010388033730643, "sim_compute_performance-ego": 0.035251736640930176, "sim_compute_robot_state-ego": 0.03658502868243626}, "ETHZ_autolab_technical_track-7-0": {"driven_any": 0.3164475681403097, "sim_physics": 0.043501240866524835, "survival_time": 1.0500000000000005, "driven_lanedir": 0.3093025166124813, "sim_render-ego": 0.03156669934590658, "in-drivable-lane": 0, "agent_compute-ego": 0.07913316999162946, "deviation-heading": 0.2242052814634115, "set_robot_commands": 0.04841228893824986, "deviation-center-line": 0.09758615464743424, "driven_lanedir_consec": 0.3093025166124813, "sim_compute_sim_state": 0.019844021115984236, "sim_compute_performance-ego": 0.03483070646013532, "sim_compute_robot_state-ego": 0.03708867799668085}, "ETHZ_autolab_technical_track-8-0": {"driven_any": 1.2018528489510765, "sim_physics": 0.040710578530521714, "survival_time": 2.9499999999999975, "driven_lanedir": 0.6922532395203, "sim_render-ego": 0.03310443587222342, "in-drivable-lane": 1.0499999999999965, "agent_compute-ego": 0.0804105007042319, "deviation-heading": 0.32928385103533875, "set_robot_commands": 0.04984473778029619, "deviation-center-line": 0.07112529346797256, "driven_lanedir_consec": 0.6922532395203, "sim_compute_sim_state": 0.02015957185777567, "sim_compute_performance-ego": 0.03570150520841954, "sim_compute_robot_state-ego": 0.03691873712054754}, "ETHZ_autolab_technical_track-9-0": {"driven_any": 1.0435481553257333, "sim_physics": 0.040827699141068895, "survival_time": 2.7499999999999982, "driven_lanedir": 0.5204405548974873, "sim_render-ego": 0.032064628601074216, "in-drivable-lane": 1.1499999999999977, "agent_compute-ego": 0.0796168717471036, "deviation-heading": 0.33296447766287096, "set_robot_commands": 0.04901751171458851, "deviation-center-line": 0.0950267943453044, "driven_lanedir_consec": 0.5204405548974873, "sim_compute_sim_state": 0.020396601070057262, "sim_compute_performance-ego": 0.03529777093367143, "sim_compute_robot_state-ego": 0.03697701800953258}, "ETHZ_autolab_technical_track-10-0": {"driven_any": 1.5954548672193265, "sim_physics": 0.04015307302598829, "survival_time": 3.8499999999999943, "driven_lanedir": 1.1042463557848754, "sim_render-ego": 0.03219540707476727, "in-drivable-lane": 1.0499999999999965, "agent_compute-ego": 0.08070238534506265, "deviation-heading": 0.3566810710676767, "set_robot_commands": 0.049953943723207946, "deviation-center-line": 0.22513914436082189, "driven_lanedir_consec": 1.1042463557848754, "sim_compute_sim_state": 0.020804346381843863, "sim_compute_performance-ego": 0.03524313034949365, "sim_compute_robot_state-ego": 0.03696220261710031}, "ETHZ_autolab_technical_track-11-0": {"driven_any": 0.49030923107489305, "sim_physics": 0.04150243316377912, "survival_time": 1.4000000000000006, "driven_lanedir": 0.4826393189617785, "sim_render-ego": 0.032655341284615655, "in-drivable-lane": 0, "agent_compute-ego": 0.07958959681647164, "deviation-heading": 0.23322336656472228, "set_robot_commands": 0.04946574994495937, "deviation-center-line": 0.10099507669212504, "driven_lanedir_consec": 0.4826393189617785, "sim_compute_sim_state": 0.02003248248781477, "sim_compute_performance-ego": 0.03582922901426043, "sim_compute_robot_state-ego": 0.03692769152777536}, "ETHZ_autolab_technical_track-12-0": {"driven_any": 1.0117660515775693, "sim_physics": 0.03949393945581773, "survival_time": 2.549999999999999, "driven_lanedir": 0.791717039833255, "sim_render-ego": 0.03157952720043706, "in-drivable-lane": 0.3999999999999986, "agent_compute-ego": 0.0790115197499593, "deviation-heading": 0.320449356593834, "set_robot_commands": 0.049213591743918025, "deviation-center-line": 0.1017166571540705, "driven_lanedir_consec": 0.791717039833255, "sim_compute_sim_state": 0.020080739376591702, "sim_compute_performance-ego": 0.03513287562949985, "sim_compute_robot_state-ego": 0.036904162051630954}, "ETHZ_autolab_technical_track-13-0": {"driven_any": 0.44419823353351984, "sim_physics": 0.041203048494127065, "survival_time": 1.3500000000000003, "driven_lanedir": 0.4333246859278659, "sim_render-ego": 0.03177572179723669, "in-drivable-lane": 0, "agent_compute-ego": 0.07939993893658673, "deviation-heading": 0.2988926743481924, "set_robot_commands": 0.04966685507032606, "deviation-center-line": 0.08545702330592322, "driven_lanedir_consec": 0.4333246859278659, "sim_compute_sim_state": 0.01999495647571705, "sim_compute_performance-ego": 0.034668163016990376, "sim_compute_robot_state-ego": 0.03711348109775119}, "ETHZ_autolab_technical_track-14-0": {"driven_any": 2.1706243860917107, "sim_physics": 0.03860734958274692, "survival_time": 5.09999999999999, "driven_lanedir": 1.919400421063832, "sim_render-ego": 0.03195470454646092, "in-drivable-lane": 0.4999999999999982, "agent_compute-ego": 0.07933453952564913, "deviation-heading": 0.5178939525116442, "set_robot_commands": 0.04893013309029972, "deviation-center-line": 0.2838635700333795, "driven_lanedir_consec": 1.919400421063832, "sim_compute_sim_state": 0.020009875297546387, "sim_compute_performance-ego": 0.03540206890480191, "sim_compute_robot_state-ego": 0.03710757751090854}}
set_robot_commands_max0.05145572338785444
set_robot_commands_mean0.04933302513574354
set_robot_commands_median0.049213591743918025
set_robot_commands_min0.04841228893824986
sim_compute_performance-ego_max0.03582922901426043
sim_compute_performance-ego_mean0.03514127586582585
sim_compute_performance-ego_median0.03513449781081256
sim_compute_performance-ego_min0.03463831901550293
sim_compute_robot_state-ego_max0.037802114486694335
sim_compute_robot_state-ego_mean0.03707046817096784
sim_compute_robot_state-ego_median0.036982571637188946
sim_compute_robot_state-ego_min0.03658502868243626
sim_compute_sim_state_max0.020804346381843863
sim_compute_sim_state_mean0.02010677048532256
sim_compute_sim_state_median0.02004294979329012
sim_compute_sim_state_min0.01981369186850155
sim_physics_max0.043501240866524835
sim_physics_mean0.04070761578977639
sim_physics_median0.04049600053716589
sim_physics_min0.03860734958274692
sim_render-ego_max0.03310443587222342
sim_render-ego_mean0.03212730787614884
sim_render-ego_median0.0321493008557488
sim_render-ego_min0.03156669934590658
simulation-passed1
survival_time_max5.09999999999999
survival_time_mean2.503333333333331
survival_time_min0.8500000000000002
No reset possible
21006step1-simulationsuccessno0:17:33
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20993step1-simulationhost-errorno0:16:11
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 486, in get_cr
    uploaded = upload_files(wd, aws_config, copy_to_machine_cache=copy_to_machine_cache)
  File "/project/src/duckietown_challenges_runner/runner.py", line 1024, in upload_files
    uploaded = upload(aws_config, toupload, copy_to_machine_cache=copy_to_machine_cache)
  File "/project/src/duckietown_challenges_runner/runner.py", line 1181, in upload
    copy_to_cache(realfile, sha256hex)
  File "/project/src/duckietown_challenges_runner/runner_cache.py", line 39, in copy_to_cache
    msg = 'Copying %s to cache %s' % (friendly_size(os.stat(fn).st_size), have)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/duckietown/DT18/evaluator/executions/aido2-LF-sim-testing/submission2814/step1-simulation-ip-172-31-42-167-7194-job20993/tmp/tmpx3zywuo2/episodes/ETHZ_autolab_technical_track-6-0/drawing.html'
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20315step1-simulationsuccessno0:05:26
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19341step1-simulationsuccessno0:05:19
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19330step1-simulationhost-errorno0:00:22
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 990, in run_docker
    return subprocess.check_output(cmd0, cwd=cwd, stderr=sys.stderr)
  File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['docker-compose', '-p', 'job19330-275617', 'ps', '-q', 'solution']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 478, in get_cr
    submission_id=submission_id, timeout_sec=timeout_sec)
  File "/project/src/duckietown_challenges_runner/runner.py", line 573, in run_single
    write_logs(wd, project, services=config['services'])
  File "/project/src/duckietown_challenges_runner/runner.py", line 899, in write_logs
    container_id = get_id_for_service(wd, project, service)
  File "/project/src/duckietown_challenges_runner/runner.py", line 585, in get_id_for_service
    o = run_docker(wd, project, cmd, get_output=True)
  File "/project/src/duckietown_challenges_runner/runner.py", line 998, in run_docker
    raise DockerComposeFail(msg) from e
duckietown_challenges_runner.runner.DockerComposeFail: Could not run ['docker-compose', '-p', 'job19330-275617', 'ps', '-q', 'solution']:

   >  Command '['docker-compose', '-p', 'job19330-275617', 'ps', '-q', 'solution']' returned non-zero exit status 1.

stdout | 

stderr | 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19196step1-simulationerrorno0:02:19
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "experiment_manager.py", line 173, in main
    make_video1(fn, out_video)
  File "/project/src/aido-protocols/src/aido_schemas/utils_video.py", line 56, in make_video1
    pg('video_aido', dict(filename=fn, output=output))
  File "/project/src/procgraph/src/procgraph/scripts/pgmain.py", line 244, in pg
    model.init()
  File "/project/src/procgraph/src/procgraph/core/model.py", line 218, in init
    block.init()
  File "/project/src/aido-protocols/src/aido_schemas/utils_video.py", line 16, in init
    self.log = read_simulator_log_cbor(self.get_config('filename'))
  File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 179, in read_simulator_log_cbor
    render_time = read_perfomance(ld)
  File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 74, in read_perfomance
    for i, ob in enumerate(read_topic2(ld, 'timing_information')):
  File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 49, in read_topic2
    for ob in ld.objects:
AttributeError: 'str' object has no attribute 'objects'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 610, in scoring_context
    yield cie
  File "experiment_manager.py", line 435, in <module>
    wrap(cie)
  File "experiment_manager.py", line 423, in wrap
    main(cie, logdir, attempts)
  File "experiment_manager.py", line 186, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

 > Traceback (most recent call last):
 >   File "experiment_manager.py", line 173, in main
 >     make_video1(fn, out_video)
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_video.py", line 56, in make_video1
 >     pg('video_aido', dict(filename=fn, output=output))
 >   File "/project/src/procgraph/src/procgraph/scripts/pgmain.py", line 244, in pg
 >     model.init()
 >   File "/project/src/procgraph/src/procgraph/core/model.py", line 218, in init
 >     block.init()
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_video.py", line 16, in init
 >     self.log = read_simulator_log_cbor(self.get_config('filename'))
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 179, in read_simulator_log_cbor
 >     render_time = read_perfomance(ld)
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 74, in read_perfomance
 >     for i, ob in enumerate(read_topic2(ld, 'timing_information')):
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 49, in read_topic2
 >     for ob in ld.objects:
 > AttributeError: 'str' object has no attribute 'objects'
 > 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18614step1-simulationsuccessno0:05:45
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18284step1-simulationsuccessno0:05:45
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18183step1-simulationsuccessno0:06:00
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18153step1-simulationsuccessno0:05:39
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible