Submission 11066

Submission

11066

Competing

yes

Challenge

aido5-LF-sim-validation

User

Fernanda Custodio Pereira do Carmo 🇨🇦

Date submitted

2020-11-04 23:53:56+00:00

Last status update

2021-12-04 20:52:39+00:00

Complete

complete

Details

Evaluation is complete.

Sisters

Result

💚

Jobs

LFv-sim: 56901

User label

sim-exercise-1

Admin priority

Blessing

n/a

User priority

56901

Click the images to see detailed statistics about the episode.

LF-norm-loop-000

LF-norm-small_loop-000

LF-norm-techtrack-000

LF-norm-zigzag-000

Evaluation jobs for this submission

Show only up-to-date jobs

Job ID

step

status

up to date

date started

date completed

duration

message

56901

LFv-sim

success

yes

2020-12-03 07:49:42+00:00

2020-12-03 08:07:09+00:00

0:17:27

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

driven_lanedir_consec_median	3.764768255318903
survival_time_median	29.874999999999662
deviation-center-line_median	1.3294215407423762
in-drivable-lane_median	4.674999999999921

other stats

agent_compute-ego0_max	0.02216680844624837
agent_compute-ego0_mean	0.014914433720594047
agent_compute-ego0_median	0.012811540822842114
agent_compute-ego0_min	0.011867844790443591
complete-iteration_max	0.23191119115287012
complete-iteration_mean	0.1800551317214547
complete-iteration_median	0.167704202940478
complete-iteration_min	0.15290092985199255
deviation-center-line_max	3.4663751133922593
deviation-center-line_mean	1.5658348194178704
deviation-center-line_min	0.13812108279446955
deviation-heading_max	7.048091197978931
deviation-heading_mean	3.6151318568063737
deviation-heading_median	3.392016629416844
deviation-heading_min	0.6284029704128733
driven_any_max	11.30211367903968
driven_any_mean	5.535569201279966
driven_any_median	5.058365793452627
driven_any_min	0.7234315391749288
driven_lanedir_consec_max	10.349309919264805
driven_lanedir_consec_mean	4.534174781986233
driven_lanedir_consec_min	0.2578526980423237
driven_lanedir_max	10.349309919264805
driven_lanedir_mean	4.534174781986233
driven_lanedir_median	3.764768255318903
driven_lanedir_min	0.2578526980423237
get_duckie_state_max	1.3255198067481365e-06
get_duckie_state_mean	1.2384651027951485e-06
get_duckie_state_median	1.214522365352027e-06
get_duckie_state_min	1.1992958737284035e-06
get_robot_state_max	0.0039650847058777415
get_robot_state_mean	0.0036833026018408327
get_robot_state_median	0.003644372108218357
get_robot_state_min	0.003479381485048877
get_state_dump_max	0.005030901060191864
get_state_dump_mean	0.0047220558819033726
get_state_dump_median	0.004701482116960672
get_state_dump_min	0.004454358233500281
get_ui_image_max	0.03746674913878834
get_ui_image_mean	0.02972240212665552
get_ui_image_median	0.02799431424904169
get_ui_image_min	0.02543423086975036
in-drivable-lane_max	11.24999999999944
in-drivable-lane_mean	5.987499999999817
in-drivable-lane_min	3.349999999999988
per-episodes	details {"LF-norm-loop-000-ego0": {"driven_any": 11.30211367903968, "get_ui_image": 0.026699491583437446, "step_physics": 0.09023998837784664, "survival_time": 59.99999999999873, "driven_lanedir": 10.349309919264805, "get_state_dump": 0.004454358233500281, "get_robot_state": 0.003479381485048877, "sim_render-ego0": 0.003589686108667785, "get_duckie_state": 1.2159149017461035e-06, "in-drivable-lane": 4.7999999999998195, "deviation-heading": 5.8806337544371265, "agent_compute-ego0": 0.011867844790443591, "complete-iteration": 0.15290092985199255, "set_robot_commands": 0.002018205529942699, "deviation-center-line": 3.4663751133922593, "driven_lanedir_consec": 10.349309919264805, "sim_compute_sim_state": 0.008650125015982979, "sim_compute_performance-ego0": 0.001820558711551409}, "LF-norm-zigzag-000-ego0": {"driven_any": 0.7234315391749288, "get_ui_image": 0.03746674913878834, "step_physics": 0.15311865194128194, "survival_time": 5.399999999999989, "driven_lanedir": 0.2578526980423237, "get_state_dump": 0.005030901060191864, "get_robot_state": 0.0039650847058777415, "sim_render-ego0": 0.004075667180052591, "get_duckie_state": 1.3255198067481365e-06, "in-drivable-lane": 3.349999999999988, "deviation-heading": 0.9033995043965616, "agent_compute-ego0": 0.01302981814113232, "complete-iteration": 0.23191119115287012, "set_robot_commands": 0.0023022726041461348, "deviation-center-line": 0.13812108279446955, "driven_lanedir_consec": 0.2578526980423237, "sim_compute_sim_state": 0.010739768316986364, "sim_compute_performance-ego0": 0.002092278331791589}, "LF-norm-techtrack-000-ego0": {"driven_any": 1.612485508758424, "get_ui_image": 0.029289136914645925, "step_physics": 0.10948320697335635, "survival_time": 10.15000000000001, "driven_lanedir": 0.966660048937341, "get_state_dump": 0.004683730649013145, "get_robot_state": 0.003667861807580088, "sim_render-ego0": 0.003846444335638308, "get_duckie_state": 1.2131298289579505e-06, "in-drivable-lane": 4.550000000000021, "deviation-heading": 0.6284029704128733, "agent_compute-ego0": 0.012593263504551906, "complete-iteration": 0.1764478975651311, "set_robot_commands": 0.002205488728541954, "deviation-center-line": 0.41716780700911366, "driven_lanedir_consec": 0.966660048937341, "sim_compute_sim_state": 0.00862792543336457, "sim_compute_performance-ego0": 0.0019617548175886565}, "LF-norm-small_loop-000-ego0": {"driven_any": 8.504246078146831, "get_ui_image": 0.02543423086975036, "step_physics": 0.08917309102214956, "survival_time": 49.59999999999932, "driven_lanedir": 6.562876461700465, "get_state_dump": 0.0047192335849082, "get_robot_state": 0.0036208824088566254, "sim_render-ego0": 0.0037521074665757223, "get_duckie_state": 1.1992958737284035e-06, "in-drivable-lane": 11.24999999999944, "deviation-heading": 7.048091197978931, "agent_compute-ego0": 0.02216680844624837, "complete-iteration": 0.15896050831582492, "set_robot_commands": 0.002146455336433403, "deviation-center-line": 2.2416752744756385, "driven_lanedir_consec": 6.562876461700465, "sim_compute_sim_state": 0.005957647392756989, "sim_compute_performance-ego0": 0.00190703026117515}}
set_robot_commands_max	0.0023022726041461348
set_robot_commands_mean	0.002168105549766048
set_robot_commands_median	0.0021759720324876783
set_robot_commands_min	0.002018205529942699
sim_compute_performance-ego0_max	0.002092278331791589
sim_compute_performance-ego0_mean	0.001945405530526701
sim_compute_performance-ego0_median	0.0019343925393819031
sim_compute_performance-ego0_min	0.001820558711551409
sim_compute_sim_state_max	0.010739768316986364
sim_compute_sim_state_mean	0.008493866539772726
sim_compute_sim_state_median	0.008639025224673774
sim_compute_sim_state_min	0.005957647392756989
sim_render-ego0_max	0.004075667180052591
sim_render-ego0_mean	0.0038159762727336018
sim_render-ego0_median	0.003799275901107015
sim_render-ego0_min	0.003589686108667785
simulation-passed	1
step_physics_max	0.15311865194128194
step_physics_mean	0.1105037345786586
step_physics_median	0.0998615976756015
step_physics_min	0.08917309102214956
survival_time_max	59.99999999999873
survival_time_mean	31.28749999999951
survival_time_min	5.399999999999989

No reset possible

56897

LFv-sim

success

yes

2020-12-03 07:48:59+00:00

2020-12-03 08:04:27+00:00

0:15:28

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

56885

LFv-sim

success

yes

2020-12-03 07:46:39+00:00

2020-12-03 08:05:05+00:00

0:18:26

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

50772

LFv-sim

error

2020-12-02 08:17:42+00:00

2020-12-02 08:20:45+00:00

0:03:03

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 139886825948832
- M:video_aido:cmdline(in:/;out:/) 139890976564512
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

50766

LFv-sim

error

2020-12-02 08:17:13+00:00

2020-12-02 08:23:39+00:00

0:06:26

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 139984850074160
- M:video_aido:cmdline(in:/;out:/) 139984849923856
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40329

LFv-sim

success

2020-11-07 10:55:06+00:00

2020-11-07 11:04:18+00:00

0:09:12

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40328

LFv-sim

success

2020-11-07 10:54:44+00:00

2020-11-07 11:04:01+00:00

0:09:17

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40327

LFv-sim

success

2020-11-07 10:53:15+00:00

2020-11-07 11:02:42+00:00

0:09:27

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

39688

LFv-sim

success

2020-11-06 16:33:56+00:00

2020-11-06 16:43:06+00:00

0:09:10

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

39686

LFv-sim

success

2020-11-06 16:33:22+00:00

2020-11-06 16:42:24+00:00

0:09:02

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

38843

LFv-sim

success

2020-11-04 23:55:55+00:00

2020-11-05 00:07:28+00:00

0:11:33

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

38842

LFv-sim

success

2020-11-04 23:55:26+00:00

2020-11-05 00:05:30+00:00

0:10:04

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible