Duckietown Challenges Home Challenges Submissions

Submission 11664

Submission11664
Competingyes
Challengeaido5-LF-sim-validation
UserMo Kleit 🇨🇦
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
JobsLFv-sim: 54101
Next
User labelsim-exercise-2
Admin priority50
Blessingn/a
User priority50

54101

Click the images to see detailed statistics about the episode.

LF-norm-loop-000

LF-norm-small_loop-000

LF-norm-techtrack-000

LF-norm-zigzag-000

Evaluation jobs for this submission

Show only up-to-date jobs
Job IDstepstatusup to datedate starteddate completeddurationmessage
54101LFv-simsuccessyes0:17:21
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median1.737599949640607
survival_time_median16.850000000000104
deviation-center-line_median0.5486974622126177
in-drivable-lane_median10.850000000000108


other stats
agent_compute-ego0_max0.013742348379340054
agent_compute-ego0_mean0.01325262053540728
agent_compute-ego0_median0.013164240927524024
agent_compute-ego0_min0.012939651907241017
complete-iteration_max0.2898316673907755
complete-iteration_mean0.2454598597329643
complete-iteration_median0.24138492142441673
complete-iteration_min0.20923792869224833
deviation-center-line_max2.210439303942148
deviation-center-line_mean0.9256546196356812
deviation-center-line_min0.39478425017534174
deviation-heading_max11.47768560016671
deviation-heading_mean4.392162987349466
deviation-heading_median2.1607531710554806
deviation-heading_min1.7694600071201922
driven_any_max15.839668373791664
driven_any_mean6.78628217330108
driven_any_median4.191079899128265
driven_any_min2.923300521156128
driven_lanedir_consec_max3.433652357053002
driven_lanedir_consec_mean1.9354989376966936
driven_lanedir_consec_min0.8331434944525578
driven_lanedir_max5.934635031372139
driven_lanedir_mean2.5717940584356276
driven_lanedir_median1.737599949640607
driven_lanedir_min0.8773413030891577
get_duckie_state_max1.5704099797020274e-06
get_duckie_state_mean1.4342009477448409e-06
get_duckie_state_median1.4060761299786882e-06
get_duckie_state_min1.3542415513199584e-06
get_robot_state_max0.0040515257307320585
get_robot_state_mean0.003881400178572818
get_robot_state_median0.003864868006679504
get_robot_state_min0.0037443389702002073
get_state_dump_max0.00516919164873791
get_state_dump_mean0.0049997277179968675
get_state_dump_median0.005061196560168839
get_state_dump_min0.00470732610291188
get_ui_image_max0.03819787032288035
get_ui_image_mean0.032181475424260766
get_ui_image_median0.03171809090958906
get_ui_image_min0.02709184955498459
in-drivable-lane_max33.749999999999716
in-drivable-lane_mean15.062499999999996
in-drivable-lane_min4.800000000000054
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 2.923300521156128, "get_ui_image": 0.03092918021619813, "step_physics": 0.1675858349839518, "survival_time": 12.050000000000036, "driven_lanedir": 1.7633274741589031, "get_state_dump": 0.0051546993334431294, "get_robot_state": 0.0040515257307320585, "sim_render-ego0": 0.004089146606193101, "get_duckie_state": 1.5704099797020274e-06, "in-drivable-lane": 4.800000000000054, "deviation-heading": 1.7694600071201922, "agent_compute-ego0": 0.013742348379340054, "complete-iteration": 0.24097722226923163, "set_robot_commands": 0.002467162352948149, "deviation-center-line": 0.5383412097653907, "driven_lanedir_consec": 1.7633274741589031, "sim_compute_sim_state": 0.010596078289441828, "sim_compute_performance-ego0": 0.00226682276765177}, "LF-norm-zigzag-000-ego0": {"driven_any": 3.414472211117795, "get_ui_image": 0.03819787032288035, "step_physics": 0.2086223453603765, "survival_time": 13.900000000000064, "driven_lanedir": 1.7118724251223107, "get_state_dump": 0.004967693786894549, "get_robot_state": 0.003925185049733808, "sim_render-ego0": 0.003994157237391318, "get_duckie_state": 1.4159841776748711e-06, "in-drivable-lane": 6.65000000000008, "deviation-heading": 2.131859477246518, "agent_compute-ego0": 0.013340668866284005, "complete-iteration": 0.2898316673907755, "set_robot_commands": 0.0023835517172317777, "deviation-center-line": 0.5590537146598448, "driven_lanedir_consec": 1.7118724251223107, "sim_compute_sim_state": 0.012195608521875088, "sim_compute_performance-ego0": 0.002113191884905634}, "LF-norm-techtrack-000-ego0": {"driven_any": 4.967687587138736, "get_ui_image": 0.03250700160297999, "step_physics": 0.16869257859679254, "survival_time": 19.800000000000143, "driven_lanedir": 0.8773413030891577, "get_state_dump": 0.00516919164873791, "get_robot_state": 0.003804550963625199, "sim_render-ego0": 0.003808365660890825, "get_duckie_state": 1.3542415513199584e-06, "in-drivable-lane": 15.050000000000134, "deviation-heading": 2.189646864864444, "agent_compute-ego0": 0.012939651907241017, "complete-iteration": 0.24179262057960185, "set_robot_commands": 0.0026146116424868027, "deviation-center-line": 0.39478425017534174, "driven_lanedir_consec": 0.8331434944525578, "sim_compute_sim_state": 0.010168264134404642, "sim_compute_performance-ego0": 0.002001676511404196}, "LF-norm-small_loop-000-ego0": {"driven_any": 15.839668373791664, "get_ui_image": 0.02709184955498459, "step_physics": 0.146068494583943, "survival_time": 59.99999999999873, "driven_lanedir": 5.934635031372139, "get_state_dump": 0.00470732610291188, "get_robot_state": 0.0037443389702002073, "sim_render-ego0": 0.003860157197162968, "get_duckie_state": 1.3961680822825055e-06, "in-drivable-lane": 33.749999999999716, "deviation-heading": 11.47768560016671, "agent_compute-ego0": 0.012987812988764042, "complete-iteration": 0.20923792869224833, "set_robot_commands": 0.002242210604169784, "deviation-center-line": 2.210439303942148, "driven_lanedir_consec": 3.433652357053002, "sim_compute_sim_state": 0.006432842751724535, "sim_compute_performance-ego0": 0.002014918093081815}}
set_robot_commands_max0.0026146116424868027
set_robot_commands_mean0.002426884079209128
set_robot_commands_median0.002425357035089963
set_robot_commands_min0.002242210604169784
sim_compute_performance-ego0_max0.00226682276765177
sim_compute_performance-ego0_mean0.0020991523142608534
sim_compute_performance-ego0_median0.0020640549889937247
sim_compute_performance-ego0_min0.002001676511404196
sim_compute_sim_state_max0.012195608521875088
sim_compute_sim_state_mean0.009848198424361524
sim_compute_sim_state_median0.010382171211923237
sim_compute_sim_state_min0.006432842751724535
sim_render-ego0_max0.004089146606193101
sim_render-ego0_mean0.003937956675409553
sim_render-ego0_median0.003927157217277143
sim_render-ego0_min0.003808365660890825
simulation-passed1
step_physics_max0.2086223453603765
step_physics_mean0.17274231338126594
step_physics_median0.16813920679037217
step_physics_min0.146068494583943
survival_time_max59.99999999999873
survival_time_mean26.437499999999744
survival_time_min12.050000000000036
No reset possible
54100LFv-simsuccessyes0:25:05
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
49184LFv-simerrorno0:04:12
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 139647933672032
- M:video_aido:cmdline(in:/;out:/) 139647933634688
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
49178LFv-simhost-errorno0:09:17
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
49171LFv-simerrorno0:04:34
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140472146832064
- M:video_aido:cmdline(in:/;out:/) 140472146830768
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
49165LFv-simerrorno0:07:55
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140463708406064
- M:video_aido:cmdline(in:/;out:/) 140463708432176
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
43369LFv-simsuccessno0:07:44
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
43368LFv-simhost-errorno0:00:34
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 687, in get_cr
    cr = run_single(
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 888, in run_single
    write_logs(wd, project, services=config["services"])
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1388, in write_logs
    services2id: Dict[str, str] = get_services_id(wd, project, services)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 937, in get_services_id
    container = client.containers.get(container_id)
  File "/usr/local/lib/python3.8/dist-packages/docker/models/containers.py", line 880, in get
    resp = self.client.api.inspect_container(container_id)
  File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 16, in wrapped
    raise errors.NullResource(
docker.errors.NullResource: Resource ID was not provided
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
43364LFv-simsuccessno0:08:29
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
43363LFv-simtimeoutno0:00:51
Timeout because eval [...]
Timeout because evaluator contacted us
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
43360LFv-simsuccessno0:12:48
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
43356LFv-simsuccessno0:13:35
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
43355LFv-simhost-errorno0:00:24
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 687, in get_cr
    cr = run_single(
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 888, in run_single
    write_logs(wd, project, services=config["services"])
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1388, in write_logs
    services2id: Dict[str, str] = get_services_id(wd, project, services)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 937, in get_services_id
    container = client.containers.get(container_id)
  File "/usr/local/lib/python3.8/dist-packages/docker/models/containers.py", line 880, in get
    resp = self.client.api.inspect_container(container_id)
  File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 16, in wrapped
    raise errors.NullResource(
docker.errors.NullResource: Resource ID was not provided
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible