Submission 9264

Submission

9264

Competing

yes

Challenge

aido5-LF-sim-validation

User

Jerome Labonte 🇨🇦

Date submitted

2020-10-24 04:29:23+00:00

Last status update

2021-12-04 20:52:32+00:00

Complete

complete

Details

Evaluation is complete.

Sisters

Result

💚

Jobs

LFv-sim: 58457

User label

exercise_ros_template

Admin priority

Blessing

n/a

User priority

58457

Click the images to see detailed statistics about the episode.

LF-norm-loop-000

LF-norm-small_loop-000

LF-norm-techtrack-000

LF-norm-zigzag-000

Evaluation jobs for this submission

Show only up-to-date jobs

Job ID

step

status

up to date

date started

date completed

duration

message

58457

LFv-sim

success

yes

2020-12-03 23:19:47+00:00

2020-12-03 23:50:14+00:00

0:30:27

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

driven_lanedir_consec_median	5.426090735704689
survival_time_median	59.99999999999873
deviation-center-line_median	3.3821727953102103
in-drivable-lane_median	13.699999999999704

other stats

agent_compute-ego0_max	0.04544854521453629
agent_compute-ego0_mean	0.02875741831591032
agent_compute-ego0_median	0.028274677560564565
agent_compute-ego0_min	0.013031772927975874
complete-iteration_max	0.26809116962251817
complete-iteration_mean	0.2127787752845011
complete-iteration_median	0.20396712499296135
complete-iteration_min	0.1750896815295633
deviation-center-line_max	3.980228442744924
deviation-center-line_mean	2.7897223489436325
deviation-center-line_min	0.41431536240918354
deviation-heading_max	27.032506059007975
deviation-heading_mean	13.647093483435045
deviation-heading_median	13.151397465424283
deviation-heading_min	1.2530729438836283
driven_any_max	14.794701357596178
driven_any_mean	10.709744532069692
driven_any_median	12.587103540318466
driven_any_min	2.870069690045656
driven_lanedir_consec_max	7.922930634110642
driven_lanedir_consec_mean	5.179291200185732
driven_lanedir_consec_min	1.9420526952229096
driven_lanedir_max	12.998368392256552
driven_lanedir_mean	8.131732976521892
driven_lanedir_median	8.793255409304056
driven_lanedir_min	1.9420526952229096
get_duckie_state_max	1.3290694313779858e-06
get_duckie_state_mean	1.3010853725657562e-06
get_duckie_state_median	1.3121955102925295e-06
get_duckie_state_min	1.2508810382999786e-06
get_robot_state_max	0.004016255855957336
get_robot_state_mean	0.003907087941335604
get_robot_state_median	0.003909974669933716
get_robot_state_min	0.0037921465695176498
get_state_dump_max	0.004874873923384436
get_state_dump_mean	0.004817208657098591
get_state_dump_median	0.004839699433109941
get_state_dump_min	0.0047145618387900426
get_ui_image_max	0.03904242142352534
get_ui_image_mean	0.03276560901795561
get_ui_image_median	0.03223426877398304
get_ui_image_min	0.027551477100331023
in-drivable-lane_max	18.199999999999516
in-drivable-lane_mean	12.43749999999974
in-drivable-lane_min	4.150000000000036
per-episodes	details {"LF-norm-loop-000-ego0": {"driven_any": 2.870069690045656, "get_ui_image": 0.02894013004215885, "step_physics": 0.10611565036860776, "survival_time": 10.90000000000002, "driven_lanedir": 1.9420526952229096, "get_state_dump": 0.004874873923384436, "get_robot_state": 0.0037921465695176498, "sim_render-ego0": 0.003981128675208244, "get_duckie_state": 1.2508810382999786e-06, "in-drivable-lane": 4.150000000000036, "deviation-heading": 1.2530729438836283, "agent_compute-ego0": 0.01312427215924546, "complete-iteration": 0.1750896815295633, "set_robot_commands": 0.0022989985061018436, "deviation-center-line": 0.41431536240918354, "driven_lanedir_consec": 1.9420526952229096, "sim_compute_sim_state": 0.00977802058877466, "sim_compute_performance-ego0": 0.0020980998261334147}, "LF-norm-zigzag-000-ego0": {"driven_any": 10.885631775913089, "get_ui_image": 0.03904242142352534, "step_physics": 0.1526540705405306, "survival_time": 59.99999999999873, "driven_lanedir": 6.408734856889801, "get_state_dump": 0.004807144874934849, "get_robot_state": 0.003932454802412276, "sim_render-ego0": 0.004102624326224728, "get_duckie_state": 1.3290694313779858e-06, "in-drivable-lane": 17.649999999999583, "deviation-heading": 27.032506059007975, "agent_compute-ego0": 0.04544854521453629, "complete-iteration": 0.26809116962251817, "set_robot_commands": 0.002580678036171233, "deviation-center-line": 3.980228442744924, "driven_lanedir_consec": 3.3880543403467103, "sim_compute_sim_state": 0.013187117223239364, "sim_compute_performance-ego0": 0.0022426320551634828}, "LF-norm-techtrack-000-ego0": {"driven_any": 14.794701357596178, "get_ui_image": 0.035528407505807234, "step_physics": 0.12973307431687126, "survival_time": 59.99999999999873, "driven_lanedir": 12.998368392256552, "get_state_dump": 0.004872253991285033, "get_robot_state": 0.004016255855957336, "sim_render-ego0": 0.0040619696903784605, "get_duckie_state": 1.3088207260754383e-06, "in-drivable-lane": 9.749999999999826, "deviation-heading": 13.682465494547849, "agent_compute-ego0": 0.013031772927975874, "complete-iteration": 0.2084046357080998, "set_robot_commands": 0.002463380065587637, "deviation-center-line": 3.5767337016747573, "driven_lanedir_consec": 7.464127131062668, "sim_compute_sim_state": 0.01238283606790484, "sim_compute_performance-ego0": 0.002223269925526437}, "LF-norm-small_loop-000-ego0": {"driven_any": 14.288575304723844, "get_ui_image": 0.027551477100331023, "step_physics": 0.10501542496343735, "survival_time": 59.99999999999873, "driven_lanedir": 11.17777596171831, "get_state_dump": 0.0047145618387900426, "get_robot_state": 0.003887494537455157, "sim_render-ego0": 0.003910409123772487, "get_duckie_state": 1.315570294509621e-06, "in-drivable-lane": 18.199999999999516, "deviation-heading": 12.620329436300723, "agent_compute-ego0": 0.043425082961883674, "complete-iteration": 0.19952961427782295, "set_robot_commands": 0.002427588692314917, "deviation-center-line": 3.1876118889456633, "driven_lanedir_consec": 7.922930634110642, "sim_compute_sim_state": 0.006411809706866592, "sim_compute_performance-ego0": 0.002093617068440789}}
set_robot_commands_max	0.002580678036171233
set_robot_commands_mean	0.0024426613250439076
set_robot_commands_median	0.002445484378951277
set_robot_commands_min	0.0022989985061018436
sim_compute_performance-ego0_max	0.0022426320551634828
sim_compute_performance-ego0_mean	0.002164404718816031
sim_compute_performance-ego0_median	0.002160684875829926
sim_compute_performance-ego0_min	0.002093617068440789
sim_compute_sim_state_max	0.013187117223239364
sim_compute_sim_state_mean	0.010439945896696364
sim_compute_sim_state_median	0.01108042832833975
sim_compute_sim_state_min	0.006411809706866592
sim_render-ego0_max	0.004102624326224728
sim_render-ego0_mean	0.00401403295389598
sim_render-ego0_median	0.004021549182793353
sim_render-ego0_min	0.003910409123772487
simulation-passed	1
step_physics_max	0.1526540705405306
step_physics_mean	0.12337955504736174
step_physics_median	0.11792436234273952
step_physics_min	0.10501542496343735
survival_time_max	59.99999999999873
survival_time_mean	47.72499999999905
survival_time_min	10.90000000000002

No reset possible

52454

LFv-sim

error

2020-12-02 11:41:59+00:00

2020-12-02 11:54:37+00:00

0:12:38

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140316162107904
- M:video_aido:cmdline(in:/;out:/) 140316160202544
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

52447

LFv-sim

error

2020-12-02 11:40:44+00:00

2020-12-02 11:48:13+00:00

0:07:29

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140392463837600
- M:video_aido:cmdline(in:/;out:/) 140392463396768
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

52446

LFv-sim

host-error

2020-12-02 11:40:39+00:00

2020-12-02 11:53:13+00:00

0:12:34

The container "evalu [...]

The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

52444

LFv-sim

error

2020-12-02 11:40:07+00:00

2020-12-02 11:49:07+00:00

0:09:00

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140372170834320
- M:video_aido:cmdline(in:/;out:/) 140372424936320
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

52380

LFv-sim

host-error

2020-12-02 11:31:53+00:00

2020-12-02 11:37:21+00:00

0:05:28

The container "evalu [...]

The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

41771

LFv-sim

success

2020-11-08 14:09:39+00:00

2020-11-08 14:16:48+00:00

0:07:09

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

41769

LFv-sim

success

2020-11-08 14:09:09+00:00

2020-11-08 14:17:21+00:00

0:08:12

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

41768

LFv-sim

success

2020-11-08 14:08:02+00:00

2020-11-08 14:15:30+00:00

0:07:28

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

38278

LFv-sim

success

2020-11-04 01:26:41+00:00

2020-11-04 01:33:21+00:00

0:06:40

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

36381

LFv-sim

success

2020-11-02 04:58:42+00:00

2020-11-02 05:09:23+00:00

0:10:41

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

35810

LFv-sim

success

2020-11-01 04:19:29+00:00

2020-11-01 04:20:34+00:00

0:01:05

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

35404

LFv-sim

error

2020-10-31 04:45:54+00:00

2020-10-31 05:07:44+00:00

0:21:50

The result file is n [...]

The result file is not found in working dir /tmp/duckietown/DT18/evaluator/executions/aido5-LF-sim-validation/submission9264/LFv-sim-reg02-1b92df2e7e91-1-job35404:

File '/tmp/duckietown/DT18/evaluator/executions/aido5-LF-sim-validation/submission9264/LFv-sim-reg02-1b92df2e7e91-1-job35404/challenge-results/challenge_results.yaml' does not exist.

This usually means that the evaluator did not finish and some times that there was an import error.
Check the evaluator log to see what happened.

List of all files:

 -/tmp/duckietown/DT18/evaluator/executions/aido5-LF-sim-validation/submission9264/LFv-sim-reg02-1b92df2e7e91-1-job35404/docker-compose.original.yaml
-/tmp/duckietown/DT18/evaluator/executions/aido5-LF-sim-validation/submission9264/LFv-sim-reg02-1b92df2e7e91-1-job35404/docker-compose.yaml
-/tmp/duckietown/DT18/evaluator/executions/aido5-LF-sim-validation/submission9264/LFv-sim-reg02-1b92df2e7e91-1-job35404/logs/challenges-runner/stdout.log
-/tmp/duckietown/DT18/evaluator/executions/aido5-LF-sim-validation/submission9264/LFv-sim-reg02-1b92df2e7e91-1-job35404/logs/challenges-runner/stderr.log

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

35047

LFv-sim

success

2020-10-29 05:48:41+00:00

2020-10-29 06:11:44+00:00

0:23:03

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

35046

LFv-sim

success

2020-10-29 05:48:07+00:00

2020-10-29 06:10:22+00:00

0:22:15

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

34562

LFv-sim

success

2020-10-24 04:29:50+00:00

2020-10-24 04:52:34+00:00

0:22:44

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible