Duckietown Challenges Home Challenges Submissions

Submission 11733

Submission11733
Competingyes
Challengeaido5-LF-sim-validation
UserJean-Sébastien Grondin 🇨🇦
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
JobsLFv-sim: 53966
Next
User labelsim-exercise-2
Admin priority50
Blessingn/a
User priority50

53966

Click the images to see detailed statistics about the episode.

LF-norm-loop-000

LF-norm-small_loop-000

LF-norm-techtrack-000

LF-norm-zigzag-000

Evaluation jobs for this submission

Show only up-to-date jobs
Job IDstepstatusup to datedate starteddate completeddurationmessage
53966LFv-simsuccessyes0:30:00
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median5.047992066445484
survival_time_median59.09999999999878
deviation-center-line_median2.53926541361945
in-drivable-lane_median21.249999999999524


other stats
agent_compute-ego0_max0.014570125830759116
agent_compute-ego0_mean0.013440203943429506
agent_compute-ego0_median0.01324436601681925
agent_compute-ego0_min0.012701957909320402
complete-iteration_max0.24626972377300263
complete-iteration_mean0.2168642814462543
complete-iteration_median0.2233833483939773
complete-iteration_min0.17442070522405995
deviation-center-line_max3.457794624804717
deviation-center-line_mean2.3049858300499255
deviation-center-line_min0.6836178681560844
deviation-heading_max14.023896582136004
deviation-heading_mean9.738306176570587
deviation-heading_median11.340122191179107
deviation-heading_min2.2490837417881333
driven_any_max15.347571565009297
driven_any_mean11.116547815697874
driven_any_median13.246000104920558
driven_any_min2.6266194879410936
driven_lanedir_consec_max7.128190439382005
driven_lanedir_consec_mean4.627266776549353
driven_lanedir_consec_min1.284892533924444
driven_lanedir_max12.037639602189405
driven_lanedir_mean7.098164497151261
driven_lanedir_median7.314058520367399
driven_lanedir_min1.7269013456808415
get_duckie_state_max1.515944798787435e-06
get_duckie_state_mean1.4808089822521925e-06
get_duckie_state_median1.4779132434853905e-06
get_duckie_state_min1.4514646432505534e-06
get_robot_state_max0.0042014245089642905
get_robot_state_mean0.004050365336123764
get_robot_state_median0.00401656991073634
get_robot_state_min0.003966897014058084
get_state_dump_max0.005177424015550192
get_state_dump_mean0.004985849968418917
get_state_dump_median0.004998106513191108
get_state_dump_min0.00476976283174326
get_ui_image_max0.039025863739572696
get_ui_image_mean0.03431807632556355
get_ui_image_median0.03583711751891141
get_ui_image_min0.026572206524858703
in-drivable-lane_max34.19999999999867
in-drivable-lane_mean20.32499999999944
in-drivable-lane_min4.600000000000054
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 11.644280791039714, "get_ui_image": 0.03352756735867192, "step_physics": 0.1349563313089623, "survival_time": 59.64999999999875, "driven_lanedir": 6.336013608706313, "get_state_dump": 0.004953922338821181, "get_robot_state": 0.004030211087646995, "sim_render-ego0": 0.0040507893666150785, "get_duckie_state": 1.4836265014443964e-06, "in-drivable-lane": 34.19999999999867, "deviation-heading": 10.011388430961146, "agent_compute-ego0": 0.013579593830971263, "complete-iteration": 0.21016234768495287, "set_robot_commands": 0.002484707976106423, "deviation-center-line": 2.088540099443277, "driven_lanedir_consec": 5.480612859667266, "sim_compute_sim_state": 0.010228944583554164, "sim_compute_performance-ego0": 0.002260693752985304}, "LF-norm-zigzag-000-ego0": {"driven_any": 2.6266194879410936, "get_ui_image": 0.0381466676791509, "step_physics": 0.16459385255972545, "survival_time": 11.950000000000037, "driven_lanedir": 1.7269013456808415, "get_state_dump": 0.0050422906875610355, "get_robot_state": 0.004002928733825684, "sim_render-ego0": 0.004105861981709798, "get_duckie_state": 1.515944798787435e-06, "in-drivable-lane": 4.600000000000054, "deviation-heading": 2.2490837417881333, "agent_compute-ego0": 0.012909138202667236, "complete-iteration": 0.24626972377300263, "set_robot_commands": 0.0023728509744008384, "deviation-center-line": 0.6836178681560844, "driven_lanedir_consec": 1.284892533924444, "sim_compute_sim_state": 0.01280034879843394, "sim_compute_performance-ego0": 0.0022056639194488526}, "LF-norm-techtrack-000-ego0": {"driven_any": 15.347571565009297, "get_ui_image": 0.039025863739572696, "step_physics": 0.15123375131129027, "survival_time": 59.99999999999873, "driven_lanedir": 12.037639602189405, "get_state_dump": 0.005177424015550192, "get_robot_state": 0.0042014245089642905, "sim_render-ego0": 0.004289066066948401, "get_duckie_state": 1.4721999855263843e-06, "in-drivable-lane": 16.299999999999645, "deviation-heading": 12.668855951397068, "agent_compute-ego0": 0.014570125830759116, "complete-iteration": 0.23660434910300177, "set_robot_commands": 0.0026747707919613905, "deviation-center-line": 3.457794624804717, "driven_lanedir_consec": 7.128190439382005, "sim_compute_sim_state": 0.012956055078180902, "sim_compute_performance-ego0": 0.002377363962495853}, "LF-norm-small_loop-000-ego0": {"driven_any": 14.8477194188014, "get_ui_image": 0.026572206524858703, "step_physics": 0.11167552853607074, "survival_time": 58.54999999999881, "driven_lanedir": 8.292103432028485, "get_state_dump": 0.00476976283174326, "get_robot_state": 0.003966897014058084, "sim_render-ego0": 0.003950827561140874, "get_duckie_state": 1.4514646432505534e-06, "in-drivable-lane": 26.19999999999941, "deviation-heading": 14.023896582136004, "agent_compute-ego0": 0.012701957909320402, "complete-iteration": 0.17442070522405995, "set_robot_commands": 0.002350129771964949, "deviation-center-line": 2.989990727795623, "driven_lanedir_consec": 4.615371273223701, "sim_compute_sim_state": 0.006244205980984424, "sim_compute_performance-ego0": 0.002102146579950743}}
set_robot_commands_max0.0026747707919613905
set_robot_commands_mean0.0024706148786084
set_robot_commands_median0.0024287794752536306
set_robot_commands_min0.002350129771964949
sim_compute_performance-ego0_max0.002377363962495853
sim_compute_performance-ego0_mean0.002236467053720188
sim_compute_performance-ego0_median0.0022331788362170784
sim_compute_performance-ego0_min0.002102146579950743
sim_compute_sim_state_max0.012956055078180902
sim_compute_sim_state_mean0.010557388610288355
sim_compute_sim_state_median0.011514646690994052
sim_compute_sim_state_min0.006244205980984424
sim_render-ego0_max0.004289066066948401
sim_render-ego0_mean0.004099136244103538
sim_render-ego0_median0.004078325674162438
sim_render-ego0_min0.003950827561140874
simulation-passed1
step_physics_max0.16459385255972545
step_physics_mean0.1406148659290122
step_physics_median0.1430950413101263
step_physics_min0.11167552853607074
survival_time_max59.99999999999873
survival_time_mean47.537499999999085
survival_time_min11.950000000000037
No reset possible
53965LFv-simsuccessyes0:21:03
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
53940LFv-simhost-erroryes0:02:01
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/docker_compose.py", line 53, in get_services_id
    container = client.containers.get(container_id)
  File "/usr/local/lib/python3.8/dist-packages/docker/models/containers.py", line 880, in get
    resp = self.client.api.inspect_container(container_id)
  File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 16, in wrapped
    raise errors.NullResource(
docker.errors.NullResource: Resource ID was not provided

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 777, in get_cr
    cr = run_single(
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 991, in run_single
    write_logs(wd, project, services=config["services"])
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/docker_compose.py", line 120, in write_logs
    services2id: Dict[ServiceName, ContainerID] = get_services_id(wd, project, services)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/docker_compose.py", line 63, in get_services_id
    raise DockerComposeFail(msg, output=output.decode(), names=names) from e
duckietown_challenges_runner.docker_compose.DockerComposeFail: Cannot get process ids
│ output: ''
│  names: {}
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
53914LFv-simfailedyes0:23:15
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 218, in read_one
    msgs = read_reply(self.fpout, timeout=timeout, waiting_for=waiting_for, nickname=self.nickname,)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 304, in read_reply
    others = read_until_over(fpout, timeout=timeout, nickname=nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 341, in read_until_over
    raise TimeoutError(msg) from None
TimeoutError: Timeout while reading output of node "ego0".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 563, in run_episode
    r: MsgReceived = await loop.run_in_executor(executor, f)
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 135, in write_topic_and_expect
    ob: MsgReceived = self.read_one(expect_topic=expect, timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 281, in read_one
    raise TimeoutError(msg) from e
TimeoutError: Timeout detected on /fifos/ego0-out after 2902 messages. Expected topic "commands".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 286, in main
    length_s = await run_episode(
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 567, in run_episode
    raise dc.InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Trouble with communication to the agent.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
48782LFv-simerrorno0:06:47
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140399883079296
- M:video_aido:cmdline(in:/;out:/) 140399883958064
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
48139LFv-simabortedno0:00:42
KeyboardInterrupt: T [...]
KeyboardInterrupt:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1169, in run_one
    heartbeat()
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 628, in heartbeat
    raise KeyboardInterrupt(msg_)
KeyboardInterrupt: The server told us to abort the job because: The challenge has been updated.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
43758LFv-simsuccessno0:11:26
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
43756LFv-simsuccessno0:09:12
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
43754LFv-simsuccessno0:08:12
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible