Duckietown Challenges Home Challenges Submissions

Submission 11405

Submission11405
Competingyes
Challengeaido5-LF-sim-validation
UserThomas Wiggers 🇳🇱
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
JobsLFv-sim: 54655
Next
User labelppo_v1
Admin priority50
Blessingn/a
User priority50

54655

Click the images to see detailed statistics about the episode.

LF-norm-loop-000

LF-norm-small_loop-000

LF-norm-techtrack-000

LF-norm-zigzag-000

Evaluation jobs for this submission

Show only up-to-date jobs
Job IDstepstatusup to datedate starteddate completeddurationmessage
54655LFv-simsuccessyes0:04:46
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.44392017858149013
survival_time_median4.699999999999991
deviation-center-line_median0.12752386950009362
in-drivable-lane_median2.8249999999999917


other stats
agent_compute-ego0_max0.0380968811747792
agent_compute-ego0_mean0.02891635356445411
agent_compute-ego0_median0.026327992307728733
agent_compute-ego0_min0.024912548467579773
complete-iteration_max0.21116404218988105
complete-iteration_mean0.17946765328411668
complete-iteration_median0.17557325603155588
complete-iteration_min0.15556005888347385
deviation-center-line_max0.18720571322887256
deviation-center-line_mean0.1321168927991333
deviation-center-line_min0.08621411896747339
deviation-heading_max1.3575789622827008
deviation-heading_mean0.8421028574727152
deviation-heading_median0.8265781111286582
deviation-heading_min0.35767624535084325
driven_any_max3.683848852394706
driven_any_mean1.857151918161192
driven_any_median1.3360645479059563
driven_any_min1.0726297244381495
driven_lanedir_consec_max0.600605977227856
driven_lanedir_consec_mean0.4411549341790114
driven_lanedir_consec_min0.2761734023252096
driven_lanedir_max0.600605977227856
driven_lanedir_mean0.4411549341790114
driven_lanedir_median0.44392017858149013
driven_lanedir_min0.2761734023252096
get_duckie_state_max1.3067752500123615e-06
get_duckie_state_mean1.225911283479095e-06
get_duckie_state_median1.2167984083196254e-06
get_duckie_state_min1.1632730672647666e-06
get_robot_state_max0.003574209983902748
get_robot_state_mean0.0035448724161555516
get_robot_state_median0.0035483323554709612
get_robot_state_min0.003508614969777537
get_state_dump_max0.004834129352762242
get_state_dump_mean0.00458484014181475
get_state_dump_median0.004538595182101626
get_state_dump_min0.004428040850293505
get_ui_image_max0.03439356992532919
get_ui_image_mean0.02870809497500529
get_ui_image_median0.027448316319755774
get_ui_image_min0.025542177335180417
in-drivable-lane_max10.000000000000025
in-drivable-lane_mean4.425000000000001
in-drivable-lane_min2.0499999999999927
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 3.683848852394706, "get_ui_image": 0.02593206655124069, "step_physics": 0.07980801686958924, "survival_time": 11.800000000000033, "driven_lanedir": 0.2761734023252096, "get_state_dump": 0.004539662775611073, "get_robot_state": 0.0035547073380353582, "sim_render-ego0": 0.003638065313991112, "get_duckie_state": 1.3067752500123615e-06, "in-drivable-lane": 10.000000000000025, "deviation-heading": 1.3575789622827008, "agent_compute-ego0": 0.024912548467579773, "complete-iteration": 0.15556005888347385, "set_robot_commands": 0.002005558979662159, "deviation-center-line": 0.18720571322887256, "driven_lanedir_consec": 0.2761734023252096, "sim_compute_sim_state": 0.009192088485267091, "sim_compute_performance-ego0": 0.0018886765347251408}, "LF-norm-zigzag-000-ego0": {"driven_any": 1.2631565054633116, "get_ui_image": 0.03439356992532919, "step_physics": 0.11392902803945015, "survival_time": 4.499999999999992, "driven_lanedir": 0.3310807579058219, "get_state_dump": 0.004428040850293505, "get_robot_state": 0.003508614969777537, "sim_render-ego0": 0.0037605552882938593, "get_duckie_state": 1.1632730672647666e-06, "in-drivable-lane": 2.749999999999994, "deviation-heading": 1.276843561240056, "agent_compute-ego0": 0.0380968811747792, "complete-iteration": 0.21116404218988105, "set_robot_commands": 0.0021081258962442587, "deviation-center-line": 0.12695999172199976, "driven_lanedir_consec": 0.3310807579058219, "sim_compute_sim_state": 0.00896091775579767, "sim_compute_performance-ego0": 0.0018952469249348064}, "LF-norm-techtrack-000-ego0": {"driven_any": 1.0726297244381495, "get_ui_image": 0.028964566088270867, "step_physics": 0.10426300695572777, "survival_time": 4.299999999999993, "driven_lanedir": 0.600605977227856, "get_state_dump": 0.004537527588592179, "get_robot_state": 0.003541957372906564, "sim_render-ego0": 0.0036346254677608095, "get_duckie_state": 1.2222377733252514e-06, "in-drivable-lane": 2.0499999999999927, "deviation-heading": 0.37631266101726046, "agent_compute-ego0": 0.02614892488238455, "complete-iteration": 0.18284507181452608, "set_robot_commands": 0.002011255286205774, "deviation-center-line": 0.08621411896747339, "driven_lanedir_consec": 0.600605977227856, "sim_compute_sim_state": 0.007737406368913321, "sim_compute_performance-ego0": 0.00191958745320638}, "LF-norm-small_loop-000-ego0": {"driven_any": 1.4089725903486017, "get_ui_image": 0.025542177335180417, "step_physics": 0.09482262351296164, "survival_time": 4.899999999999991, "driven_lanedir": 0.5567595992571583, "get_state_dump": 0.004834129352762242, "get_robot_state": 0.003574209983902748, "sim_render-ego0": 0.0037960139187899504, "get_duckie_state": 1.2113590433139993e-06, "in-drivable-lane": 2.8999999999999897, "deviation-heading": 0.35767624535084325, "agent_compute-ego0": 0.02650705973307292, "complete-iteration": 0.1683014402485857, "set_robot_commands": 0.002203481365935971, "deviation-center-line": 0.12808774727818747, "driven_lanedir_consec": 0.5567595992571583, "sim_compute_sim_state": 0.005060600511955492, "sim_compute_performance-ego0": 0.0018759448118884157}}
set_robot_commands_max0.002203481365935971
set_robot_commands_mean0.0020821053820120404
set_robot_commands_median0.0020596905912250163
set_robot_commands_min0.002005558979662159
sim_compute_performance-ego0_max0.00191958745320638
sim_compute_performance-ego0_mean0.0018948639311886855
sim_compute_performance-ego0_median0.0018919617298299735
sim_compute_performance-ego0_min0.0018759448118884157
sim_compute_sim_state_max0.009192088485267091
sim_compute_sim_state_mean0.0077377532804833934
sim_compute_sim_state_median0.008349162062355496
sim_compute_sim_state_min0.005060600511955492
sim_render-ego0_max0.0037960139187899504
sim_render-ego0_mean0.003707314997208933
sim_render-ego0_median0.003699310301142486
sim_render-ego0_min0.0036346254677608095
simulation-passed1
step_physics_max0.11392902803945015
step_physics_mean0.0982056688444322
step_physics_median0.09954281523434472
step_physics_min0.07980801686958924
survival_time_max11.800000000000033
survival_time_mean6.375000000000003
survival_time_min4.299999999999993
No reset possible
54646LFv-simsuccessyes0:05:30
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
54633LFv-simsuccessyes0:07:53
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
54630LFv-simsuccessyes0:07:04
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
54620LFv-simhost-erroryes0:01:44
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/docker_compose.py", line 53, in get_services_id
    container = client.containers.get(container_id)
  File "/usr/local/lib/python3.8/dist-packages/docker/models/containers.py", line 880, in get
    resp = self.client.api.inspect_container(container_id)
  File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 16, in wrapped
    raise errors.NullResource(
docker.errors.NullResource: Resource ID was not provided

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 777, in get_cr
    cr = run_single(
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 991, in run_single
    write_logs(wd, project, services=config["services"])
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/docker_compose.py", line 120, in write_logs
    services2id: Dict[ServiceName, ContainerID] = get_services_id(wd, project, services)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/docker_compose.py", line 63, in get_services_id
    raise DockerComposeFail(msg, output=output.decode(), names=names) from e
duckietown_challenges_runner.docker_compose.DockerComposeFail: Cannot get process ids
│ output: ''
│  names: {}
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
49803LFv-simerrorno0:02:16
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140453091945728
- M:video_aido:cmdline(in:/;out:/) 140453091771536
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
49794LFv-simerrorno0:03:31
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140262797605328
- M:video_aido:cmdline(in:/;out:/) 140262797604416
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
49792LFv-simhost-errorno0:04:04
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
49777LFv-simhost-errorno0:01:19
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/docker_compose.py", line 53, in get_services_id
    container = client.containers.get(container_id)
  File "/usr/local/lib/python3.8/dist-packages/docker/models/containers.py", line 880, in get
    resp = self.client.api.inspect_container(container_id)
  File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 16, in wrapped
    raise errors.NullResource(
docker.errors.NullResource: Resource ID was not provided

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 777, in get_cr
    cr = run_single(
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 991, in run_single
    write_logs(wd, project, services=config["services"])
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/docker_compose.py", line 120, in write_logs
    services2id: Dict[ServiceName, ContainerID] = get_services_id(wd, project, services)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/docker_compose.py", line 63, in get_services_id
    raise DockerComposeFail(msg, output=output.decode(), names=names) from e
duckietown_challenges_runner.docker_compose.DockerComposeFail: Cannot get process ids
│ output: ''
│  names: {}
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
42243LFv-simsuccessno0:23:34
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
42242LFv-simsuccessno0:25:02
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
42241LFv-simsuccessno0:08:13
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
42240LFv-simsuccessno0:08:47
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
42239LFv-simsuccessno0:11:11
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
42238LFv-simsuccessno0:11:46
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
42237LFv-simsuccessno0:12:22
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible