Duckietown Challenges Home Challenges Submissions

Submission 2944

Submission2944
Competingyes
Challengeaido2-LF-sim-testing
UserBhairav Mehta
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
Jobsstep1-simulation: 22700
Next
User labelBaseline solution using reinforcement learning
Admin priority50
Blessingn/a
User priority50

The highlights are available only to the owner and the admins.

Evaluation jobs for this submission

Show only up-to-date jobs
Job IDstepstatusup to datedate starteddate completeddurationmessage
22700step1-simulationsuccessyes0:09:29
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.13998674151769164
survival_time_median1.2500000000000004
deviation-center-line_median0.0821211947957373
in-drivable-lane_median0


other stats
agent_compute-ego_max0.1903850374550655
agent_compute-ego_mean0.17140917339505102
agent_compute-ego_median0.1692565849849156
agent_compute-ego_min0.15736076831817628
deviation-center-line_max0.12453681068801284
deviation-center-line_mean0.08680728335754546
deviation-center-line_min0.05451259282861306
deviation-heading_max1.006309431095703
deviation-heading_mean0.668921526640926
deviation-heading_median0.6716965842159365
deviation-heading_min0.38654298316916985
driven_any_max0.2421650671538297
driven_any_mean0.1890324860727193
driven_any_median0.19960349426687984
driven_any_min0.12592456874761618
driven_lanedir_consec_max0.17062660549205988
driven_lanedir_consec_mean0.13975103757286492
driven_lanedir_consec_min0.0971325873167519
driven_lanedir_max0.17062660549205988
driven_lanedir_mean0.13975103757286492
driven_lanedir_median0.13998674151769164
driven_lanedir_min0.0971325873167519
in-drivable-lane_max0.05000000000000005
in-drivable-lane_mean0.003333333333333336
in-drivable-lane_min0
per-episodes
details{"ETHZ_autolab_technical_track-0-0": {"driven_any": 0.2421650671538297, "sim_physics": 0.06907137235005696, "survival_time": 1.5000000000000009, "driven_lanedir": 0.16694425313161432, "sim_render-ego": 0.07042219638824462, "in-drivable-lane": 0, "agent_compute-ego": 0.1760260025660197, "deviation-heading": 0.8586149264758972, "set_robot_commands": 0.08247480392456055, "deviation-center-line": 0.10238020172854675, "driven_lanedir_consec": 0.16694425313161432, "sim_compute_sim_state": 0.043665130933125816, "sim_compute_performance-ego": 0.07120664119720459, "sim_compute_robot_state-ego": 0.06886041164398193}, "ETHZ_autolab_technical_track-1-0": {"driven_any": 0.20306395746429096, "sim_physics": 0.06479026531351023, "survival_time": 1.4500000000000006, "driven_lanedir": 0.13998674151769164, "sim_render-ego": 0.06802875420142865, "in-drivable-lane": 0, "agent_compute-ego": 0.17308546756875923, "deviation-heading": 0.7959348341955912, "set_robot_commands": 0.08100922354336443, "deviation-center-line": 0.06401973200665578, "driven_lanedir_consec": 0.13998674151769164, "sim_compute_sim_state": 0.04516407539104593, "sim_compute_performance-ego": 0.06880814453651166, "sim_compute_robot_state-ego": 0.06706092275422194}, "ETHZ_autolab_technical_track-2-0": {"driven_any": 0.12592456874761618, "sim_physics": 0.061483466625213624, "survival_time": 1.0000000000000002, "driven_lanedir": 0.0971325873167519, "sim_render-ego": 0.06457990407943726, "in-drivable-lane": 0, "agent_compute-ego": 0.15736076831817628, "deviation-heading": 0.5202615191440357, "set_robot_commands": 0.08082555532455445, "deviation-center-line": 0.09333126518764127, "driven_lanedir_consec": 0.0971325873167519, "sim_compute_sim_state": 0.03909521102905274, "sim_compute_performance-ego": 0.06719260215759278, "sim_compute_robot_state-ego": 0.06142675876617432}, "ETHZ_autolab_technical_track-3-0": {"driven_any": 0.21647975739143327, "sim_physics": 0.06584975549152919, "survival_time": 1.4000000000000006, "driven_lanedir": 0.15150303650062735, "sim_render-ego": 0.06630735737936837, "in-drivable-lane": 0, "agent_compute-ego": 0.1692565849849156, "deviation-heading": 0.9004343255838757, "set_robot_commands": 0.0826265982219151, "deviation-center-line": 0.08151718419711848, "driven_lanedir_consec": 0.15150303650062735, "sim_compute_sim_state": 0.04483503954751151, "sim_compute_performance-ego": 0.06634068489074707, "sim_compute_robot_state-ego": 0.06688661234719413}, "ETHZ_autolab_technical_track-4-0": {"driven_any": 0.20376674224739735, "sim_physics": 0.0630560907824286, "survival_time": 1.4500000000000006, "driven_lanedir": 0.1388830285337943, "sim_render-ego": 0.06839428276851259, "in-drivable-lane": 0, "agent_compute-ego": 0.1903850374550655, "deviation-heading": 0.7686573472587278, "set_robot_commands": 0.07749462949818578, "deviation-center-line": 0.06864101621049243, "driven_lanedir_consec": 0.1388830285337943, "sim_compute_sim_state": 0.04092005203510153, "sim_compute_performance-ego": 0.06751614603502996, "sim_compute_robot_state-ego": 0.06604301518407361}, "ETHZ_autolab_technical_track-5-0": {"driven_any": 0.2389444028717898, "sim_physics": 0.06353001594543457, "survival_time": 1.5000000000000009, "driven_lanedir": 0.16434875648206182, "sim_render-ego": 0.06686468919118245, "in-drivable-lane": 0, "agent_compute-ego": 0.1863058646519979, "deviation-heading": 1.006309431095703, "set_robot_commands": 0.08705532550811768, "deviation-center-line": 0.10001371263720148, "driven_lanedir_consec": 0.16434875648206182, "sim_compute_sim_state": 0.043173750241597496, "sim_compute_performance-ego": 0.07047764460245769, "sim_compute_robot_state-ego": 0.06516396999359131}, "ETHZ_autolab_technical_track-6-0": {"driven_any": 0.19960349426687984, "sim_physics": 0.06791055644000019, "survival_time": 1.3500000000000003, "driven_lanedir": 0.16642884359665056, "sim_render-ego": 0.07012295722961426, "in-drivable-lane": 0, "agent_compute-ego": 0.17056872226573802, "deviation-heading": 0.5264805011144537, "set_robot_commands": 0.08052042678550438, "deviation-center-line": 0.10010297470940584, "driven_lanedir_consec": 0.16642884359665056, "sim_compute_sim_state": 0.04451495629769785, "sim_compute_performance-ego": 0.07105051146613227, "sim_compute_robot_state-ego": 0.0687703291575114}, "ETHZ_autolab_technical_track-7-0": {"driven_any": 0.1643888752200286, "sim_physics": 0.06743944209554921, "survival_time": 1.1500000000000004, "driven_lanedir": 0.13515921895594296, "sim_render-ego": 0.0676465656446374, "in-drivable-lane": 0, "agent_compute-ego": 0.16611752302750296, "deviation-heading": 0.5335441723083946, "set_robot_commands": 0.08441255403601605, "deviation-center-line": 0.10830637169889312, "driven_lanedir_consec": 0.13515921895594296, "sim_compute_sim_state": 0.045752307643061096, "sim_compute_performance-ego": 0.07249457939811375, "sim_compute_robot_state-ego": 0.068083089330922}, "ETHZ_autolab_technical_track-8-0": {"driven_any": 0.16264817153985764, "sim_physics": 0.06382164955139161, "survival_time": 1.2500000000000004, "driven_lanedir": 0.10125437654025936, "sim_render-ego": 0.062477407455444334, "in-drivable-lane": 0, "agent_compute-ego": 0.16048384666442872, "deviation-heading": 0.793794164441544, "set_robot_commands": 0.07723977088928223, "deviation-center-line": 0.05930822246895032, "driven_lanedir_consec": 0.10125437654025936, "sim_compute_sim_state": 0.04162369728088379, "sim_compute_performance-ego": 0.07111862182617187, "sim_compute_robot_state-ego": 0.06464032173156738}, "ETHZ_autolab_technical_track-9-0": {"driven_any": 0.1529165694476567, "sim_physics": 0.06748750805854797, "survival_time": 1.2000000000000004, "driven_lanedir": 0.11812950467301107, "sim_render-ego": 0.06361338496208191, "in-drivable-lane": 0, "agent_compute-ego": 0.16353351871172586, "deviation-heading": 0.5119286906333682, "set_robot_commands": 0.07964089512825012, "deviation-center-line": 0.0821211947957373, "driven_lanedir_consec": 0.11812950467301107, "sim_compute_sim_state": 0.044695258140563965, "sim_compute_performance-ego": 0.06516467531522115, "sim_compute_robot_state-ego": 0.06596250335375468}, "ETHZ_autolab_technical_track-10-0": {"driven_any": 0.16033267702232462, "sim_physics": 0.06933227829311205, "survival_time": 1.1500000000000004, "driven_lanedir": 0.13419003140046093, "sim_render-ego": 0.06671853687452234, "in-drivable-lane": 0, "agent_compute-ego": 0.16376571033311926, "deviation-heading": 0.4070165387823732, "set_robot_commands": 0.0796140898828921, "deviation-center-line": 0.10008131586583537, "driven_lanedir_consec": 0.13419003140046093, "sim_compute_sim_state": 0.04336607974508534, "sim_compute_performance-ego": 0.06399519547172215, "sim_compute_robot_state-ego": 0.0667342414026675}, "ETHZ_autolab_technical_track-11-0": {"driven_any": 0.21177001660167843, "sim_physics": 0.07090342521667481, "survival_time": 1.2500000000000004, "driven_lanedir": 0.17062660549205988, "sim_render-ego": 0.07717686653137207, "in-drivable-lane": 0.05000000000000005, "agent_compute-ego": 0.17961515426635744, "deviation-heading": 0.4925079082312659, "set_robot_commands": 0.07996814727783202, "deviation-center-line": 0.08170291092232379, "driven_lanedir_consec": 0.17062660549205988, "sim_compute_sim_state": 0.044295825958251954, "sim_compute_performance-ego": 0.06905444145202637, "sim_compute_robot_state-ego": 0.07036013603210449}, "ETHZ_autolab_technical_track-12-0": {"driven_any": 0.2208883337937632, "sim_physics": 0.06390087604522705, "survival_time": 1.5000000000000009, "driven_lanedir": 0.15077053692012332, "sim_render-ego": 0.06716914176940918, "in-drivable-lane": 0, "agent_compute-ego": 0.15879749457041423, "deviation-heading": 0.8600989729635549, "set_robot_commands": 0.0786885658899943, "deviation-center-line": 0.08153374441775428, "driven_lanedir_consec": 0.15077053692012332, "sim_compute_sim_state": 0.04265157381693522, "sim_compute_performance-ego": 0.06820595264434814, "sim_compute_robot_state-ego": 0.06801048119862875}, "ETHZ_autolab_technical_track-13-0": {"driven_any": 0.1616404054116576, "sim_physics": 0.06424889357193657, "survival_time": 1.1500000000000004, "driven_lanedir": 0.11239702340706216, "sim_render-ego": 0.06521238451418669, "in-drivable-lane": 0, "agent_compute-ego": 0.16827180074608844, "deviation-heading": 0.6716965842159365, "set_robot_commands": 0.07903860962909201, "deviation-center-line": 0.05451259282861306, "driven_lanedir_consec": 0.11239702340706216, "sim_compute_sim_state": 0.04322238590406335, "sim_compute_performance-ego": 0.06897133329640263, "sim_compute_robot_state-ego": 0.06843849887018619}, "ETHZ_autolab_technical_track-14-0": {"driven_any": 0.17095425191058528, "sim_physics": 0.06619905432065327, "survival_time": 1.2000000000000004, "driven_lanedir": 0.1485110191248622, "sim_render-ego": 0.06674706935882568, "in-drivable-lane": 0, "agent_compute-ego": 0.18756410479545593, "deviation-heading": 0.38654298316916985, "set_robot_commands": 0.08310418327649434, "deviation-center-line": 0.12453681068801284, "driven_lanedir_consec": 0.1485110191248622, "sim_compute_sim_state": 0.044567505518595375, "sim_compute_performance-ego": 0.07183470328648885, "sim_compute_robot_state-ego": 0.07059342662493388}}
set_robot_commands_max0.08705532550811768
set_robot_commands_mean0.0809142252544037
set_robot_commands_median0.08052042678550438
set_robot_commands_min0.07723977088928223
sim_compute_performance-ego_max0.07249457939811375
sim_compute_performance-ego_mean0.06889545850507806
sim_compute_performance-ego_median0.06897133329640263
sim_compute_performance-ego_min0.06399519547172215
sim_compute_robot_state-ego_max0.07059342662493388
sim_compute_robot_state-ego_mean0.06713564789276756
sim_compute_robot_state-ego_median0.06706092275422194
sim_compute_robot_state-ego_min0.06142675876617432
sim_compute_sim_state_max0.045752307643061096
sim_compute_sim_state_mean0.043436189965504864
sim_compute_sim_state_median0.043665130933125816
sim_compute_sim_state_min0.03909521102905274
sim_physics_max0.07090342521667481
sim_physics_mean0.06593497667341773
sim_physics_median0.06584975549152919
sim_physics_min0.061483466625213624
sim_render-ego_max0.07717686653137207
sim_render-ego_mean0.06743209988988452
sim_render-ego_median0.06686468919118245
sim_render-ego_min0.062477407455444334
simulation-passed1
survival_time_max1.5000000000000009
survival_time_mean1.3000000000000005
survival_time_min1.0000000000000002
No reset possible
20913step1-simulationsuccessno0:16:02
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20779step1-simulationsuccessno0:11:22
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20239step1-simulationsuccessno0:04:01
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19200step1-simulationsuccessno0:06:47
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19182step1-simulationerrorno0:01:45
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "experiment_manager.py", line 173, in main
    make_video1(fn, out_video)
  File "/project/src/aido-protocols/src/aido_schemas/utils_video.py", line 56, in make_video1
    pg('video_aido', dict(filename=fn, output=output))
  File "/project/src/procgraph/src/procgraph/scripts/pgmain.py", line 244, in pg
    model.init()
  File "/project/src/procgraph/src/procgraph/core/model.py", line 218, in init
    block.init()
  File "/project/src/aido-protocols/src/aido_schemas/utils_video.py", line 16, in init
    self.log = read_simulator_log_cbor(self.get_config('filename'))
  File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 179, in read_simulator_log_cbor
    render_time = read_perfomance(ld)
  File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 74, in read_perfomance
    for i, ob in enumerate(read_topic2(ld, 'timing_information')):
  File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 49, in read_topic2
    for ob in ld.objects:
AttributeError: 'str' object has no attribute 'objects'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 610, in scoring_context
    yield cie
  File "experiment_manager.py", line 435, in <module>
    wrap(cie)
  File "experiment_manager.py", line 423, in wrap
    main(cie, logdir, attempts)
  File "experiment_manager.py", line 186, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

 > Traceback (most recent call last):
 >   File "experiment_manager.py", line 173, in main
 >     make_video1(fn, out_video)
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_video.py", line 56, in make_video1
 >     pg('video_aido', dict(filename=fn, output=output))
 >   File "/project/src/procgraph/src/procgraph/scripts/pgmain.py", line 244, in pg
 >     model.init()
 >   File "/project/src/procgraph/src/procgraph/core/model.py", line 218, in init
 >     block.init()
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_video.py", line 16, in init
 >     self.log = read_simulator_log_cbor(self.get_config('filename'))
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 179, in read_simulator_log_cbor
 >     render_time = read_perfomance(ld)
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 74, in read_perfomance
 >     for i, ob in enumerate(read_topic2(ld, 'timing_information')):
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 49, in read_topic2
 >     for ob in ld.objects:
 > AttributeError: 'str' object has no attribute 'objects'
 > 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19174step1-simulationhost-errorno0:02:49
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 990, in run_docker
    return subprocess.check_output(cmd0, cwd=cwd, stderr=sys.stderr)
  File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['docker-compose', '-p', 'job19174-517981', 'ps', '-q', 'solution']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 478, in get_cr
    submission_id=submission_id, timeout_sec=timeout_sec)
  File "/project/src/duckietown_challenges_runner/runner.py", line 573, in run_single
    write_logs(wd, project, services=config['services'])
  File "/project/src/duckietown_challenges_runner/runner.py", line 899, in write_logs
    container_id = get_id_for_service(wd, project, service)
  File "/project/src/duckietown_challenges_runner/runner.py", line 585, in get_id_for_service
    o = run_docker(wd, project, cmd, get_output=True)
  File "/project/src/duckietown_challenges_runner/runner.py", line 998, in run_docker
    raise DockerComposeFail(msg) from e
duckietown_challenges_runner.runner.DockerComposeFail: Could not run ['docker-compose', '-p', 'job19174-517981', 'ps', '-q', 'solution']:

   >  Command '['docker-compose', '-p', 'job19174-517981', 'ps', '-q', 'solution']' returned non-zero exit status 1.

stdout | 

stderr | 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19111step1-simulationsuccessno0:04:39
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible