Duckietown Challenges Home Challenges Submissions

Submission 11748

Submission11748
Competingyes
Challengeaido5-LF-sim-validation
UserJean-Sébastien Grondin 🇨🇦
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
JobsLFv-sim: 53849
Next
User labelreal-exercise-2
Admin priority50
Blessingn/a
User priority50

53849

Click the images to see detailed statistics about the episode.

LF-norm-loop-000

LF-norm-small_loop-000

LF-norm-techtrack-000

LF-norm-zigzag-000

Evaluation jobs for this submission

Show only up-to-date jobs
Job IDstepstatusup to datedate starteddate completeddurationmessage
53849LFv-simsuccessyes0:20:56
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.8675222582783844
survival_time_median34.19999999999947
deviation-center-line_median0.4760046632977747
in-drivable-lane_median17.624999999999694


other stats
agent_compute-ego0_max0.012777995019063465
agent_compute-ego0_mean0.01262942653634428
agent_compute-ego0_median0.0126934176131903
agent_compute-ego0_min0.01235287589993305
complete-iteration_max0.2516677257788442
complete-iteration_mean0.22780889246290845
complete-iteration_median0.2358855295369056
complete-iteration_min0.18779678499897845
deviation-center-line_max2.4240341917770003
deviation-center-line_mean0.89630459580951
deviation-center-line_min0.2091748648654904
deviation-heading_max9.720047484159844
deviation-heading_mean3.653164809166144
deviation-heading_median1.881591569485372
deviation-heading_min1.1294286135339868
driven_any_max10.217520184336951
driven_any_mean5.658927032917715
driven_any_median5.683158375277124
driven_any_min1.051871196779657
driven_lanedir_consec_max3.4558300915697884
driven_lanedir_consec_mean1.3924075779750975
driven_lanedir_consec_min0.3787557037738325
driven_lanedir_max4.788173486401501
driven_lanedir_mean1.7254934266830255
driven_lanedir_median0.8675222582783844
driven_lanedir_min0.3787557037738325
get_duckie_state_max1.493953328275885e-06
get_duckie_state_mean1.415260910747742e-06
get_duckie_state_median1.4278416420975714e-06
get_duckie_state_min1.31140703051994e-06
get_robot_state_max0.003923312788869178
get_robot_state_mean0.003815164477110107
get_robot_state_median0.003810827107205486
get_robot_state_min0.0037156909051602777
get_state_dump_max0.004903636705269921
get_state_dump_mean0.004818460515162663
get_state_dump_median0.004835278728485466
get_state_dump_min0.004699647898409801
get_ui_image_max0.03681935881176134
get_ui_image_mean0.031598281797552336
get_ui_image_median0.031579963057992676
get_ui_image_min0.02641384226246266
in-drivable-lane_max57.34999999999873
in-drivable-lane_mean23.14999999999953
in-drivable-lane_min0.0
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 10.217520184336951, "get_ui_image": 0.02998758474059347, "step_physics": 0.1521282593078359, "survival_time": 59.99999999999873, "driven_lanedir": 0.3787557037738325, "get_state_dump": 0.004903636705269921, "get_robot_state": 0.0038460194320106985, "sim_render-ego0": 0.003963610810304463, "get_duckie_state": 1.4408343439793012e-06, "in-drivable-lane": 57.34999999999873, "deviation-heading": 1.2253548400845191, "agent_compute-ego0": 0.012751468115305524, "complete-iteration": 0.2208849073548202, "set_robot_commands": 0.0022894794200481127, "deviation-center-line": 0.24316549929414585, "driven_lanedir_consec": 0.3787557037738325, "sim_compute_sim_state": 0.008842259819164165, "sim_compute_performance-ego0": 0.002080389303926822}, "LF-norm-zigzag-000-ego0": {"driven_any": 1.051871196779657, "get_ui_image": 0.03681935881176134, "step_physics": 0.17466694073085368, "survival_time": 6.799999999999984, "driven_lanedir": 0.9140824879456656, "get_state_dump": 0.004852707368614029, "get_robot_state": 0.0037156909051602777, "sim_render-ego0": 0.003927277822564118, "get_duckie_state": 1.4148489402158416e-06, "in-drivable-lane": 0.0, "deviation-heading": 2.537828298886225, "agent_compute-ego0": 0.012777995019063465, "complete-iteration": 0.2516677257788442, "set_robot_commands": 0.002285652787145907, "deviation-center-line": 0.7088438273014035, "driven_lanedir_consec": 0.9140824879456656, "sim_compute_sim_state": 0.010443302836731404, "sim_compute_performance-ego0": 0.00208372095205488}, "LF-norm-techtrack-000-ego0": {"driven_any": 1.7851502438795697, "get_ui_image": 0.033172341375391885, "step_physics": 0.17848905166331278, "survival_time": 11.60000000000003, "driven_lanedir": 0.8209620286111033, "get_state_dump": 0.004817850088356902, "get_robot_state": 0.003923312788869178, "sim_render-ego0": 0.003916609440750319, "get_duckie_state": 1.493953328275885e-06, "in-drivable-lane": 6.700000000000039, "deviation-heading": 1.1294286135339868, "agent_compute-ego0": 0.012635367111075078, "complete-iteration": 0.250886151718991, "set_robot_commands": 0.002367289792826247, "deviation-center-line": 0.2091748648654904, "driven_lanedir_consec": 0.8209620286111033, "sim_compute_sim_state": 0.009343898347518986, "sim_compute_performance-ego0": 0.0021251240513355434}, "LF-norm-small_loop-000-ego0": {"driven_any": 9.58116650667468, "get_ui_image": 0.02641384226246266, "step_physics": 0.12612297268635156, "survival_time": 56.79999999999891, "driven_lanedir": 4.788173486401501, "get_state_dump": 0.004699647898409801, "get_robot_state": 0.003775634782400274, "sim_render-ego0": 0.003810486252515486, "get_duckie_state": 1.31140703051994e-06, "in-drivable-lane": 28.549999999999347, "deviation-heading": 9.720047484159844, "agent_compute-ego0": 0.01235287589993305, "complete-iteration": 0.18779678499897845, "set_robot_commands": 0.002283068005201999, "deviation-center-line": 2.4240341917770003, "driven_lanedir_consec": 3.4558300915697884, "sim_compute_sim_state": 0.006226896296075907, "sim_compute_performance-ego0": 0.0020236258141912698}}
set_robot_commands_max0.002367289792826247
set_robot_commands_mean0.0023063725013055666
set_robot_commands_median0.00228756610359701
set_robot_commands_min0.002283068005201999
sim_compute_performance-ego0_max0.0021251240513355434
sim_compute_performance-ego0_mean0.0020782150303771288
sim_compute_performance-ego0_median0.0020820551279908507
sim_compute_performance-ego0_min0.0020236258141912698
sim_compute_sim_state_max0.010443302836731404
sim_compute_sim_state_mean0.008714089324872616
sim_compute_sim_state_median0.009093079083341575
sim_compute_sim_state_min0.006226896296075907
sim_render-ego0_max0.003963610810304463
sim_render-ego0_mean0.003904496081533596
sim_render-ego0_median0.0039219436316572185
sim_render-ego0_min0.003810486252515486
simulation-passed1
step_physics_max0.17848905166331278
step_physics_mean0.15785180609708846
step_physics_median0.16339760001934478
step_physics_min0.12612297268635156
survival_time_max59.99999999999873
survival_time_mean33.799999999999415
survival_time_min6.799999999999984
No reset possible
53846LFv-simsuccessyes0:27:27
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
48208LFv-simerrorno0:04:55
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140499183755024
- M:video_aido:cmdline(in:/;out:/) 140499182041984
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
48176LFv-simabortedno0:00:31
KeyboardInterrupt: T [...]
KeyboardInterrupt:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1169, in run_one
    heartbeat()
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 628, in heartbeat
    raise KeyboardInterrupt(msg_)
KeyboardInterrupt: The server told us to abort the job because: The challenge has been updated.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
48158LFv-simabortedno0:00:42
KeyboardInterrupt: T [...]
KeyboardInterrupt:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1169, in run_one
    heartbeat()
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 628, in heartbeat
    raise KeyboardInterrupt(msg_)
KeyboardInterrupt: The server told us to abort the job because: The challenge has been updated.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
43803LFv-simsuccessno0:05:10
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
43802LFv-simfailedno0:09:51
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 216, in read_one
    msgs = read_reply(self.fpout, timeout=timeout, waiting_for=waiting_for, nickname=self.nickname,)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 299, in read_reply
    others = read_until_over(fpout, timeout=timeout, nickname=nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 337, in read_until_over
    raise TimeoutError(msg) from None
TimeoutError: Timeout while reading output of node "ego".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "experiment_manager.py", line 540, in run_episode
    r: MsgReceived = await loop.run_in_executor(executor, f)
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 133, in write_topic_and_expect
    ob: MsgReceived = self.read_one(expect_topic=expect, timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 279, in read_one
    raise TimeoutError(msg) from e
TimeoutError: Timeout detected on /fifos/ego-out after 837 messages. Expected topic "commands".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 678, in scoring_context
    yield cie
  File "experiment_manager.py", line 683, in go
    wrap(cie)
  File "experiment_manager.py", line 668, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "experiment_manager.py", line 281, in main
    length_s = await run_episode(
  File "experiment_manager.py", line 544, in run_episode
    raise dc.InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Trouble with communication to the agent.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible