Submission 11079

Submission

11079

Competing

yes

Challenge

aido5-LF-sim-validation

User

Étienne Boucher 🇨🇦

Date submitted

2020-11-05 00:34:17+00:00

Last status update

2021-12-04 20:52:39+00:00

Complete

complete

Details

Evaluation is complete.

Sisters

Result

💚

Jobs

LFv-sim: 56817

User label

exercise_ros_template

Admin priority

Blessing

n/a

User priority

56817

Click the images to see detailed statistics about the episode.

LF-norm-loop-000

LF-norm-small_loop-000

LF-norm-techtrack-000

LF-norm-zigzag-000

Evaluation jobs for this submission

Show only up-to-date jobs

Job ID

step

status

up to date

date started

date completed

duration

message

56817

LFv-sim

success

yes

2020-12-03 07:34:44+00:00

2020-12-03 07:39:21+00:00

0:04:37

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

driven_lanedir_consec_median	0.5421232533589626
survival_time_median	5.474999999999989
deviation-center-line_median	0.14258752155014676
in-drivable-lane_median	2.9749999999999894

other stats

agent_compute-ego0_max	0.013074104602520283
agent_compute-ego0_mean	0.012038237580355372
agent_compute-ego0_median	0.012038293171089853
agent_compute-ego0_min	0.011002259376721503
complete-iteration_max	0.1956741505480827
complete-iteration_mean	0.1719852204764062
complete-iteration_median	0.17369792246083332
complete-iteration_min	0.1448708864358755
deviation-center-line_max	0.2264256960389139
deviation-center-line_mean	0.14911646769757775
deviation-center-line_min	0.0848651316511036
deviation-heading_max	2.296491387142445
deviation-heading_mean	1.155194197174774
deviation-heading_median	0.9622225466145684
deviation-heading_min	0.39984030832751527
driven_any_max	3.584892340967327
driven_any_mean	1.981996613978529
driven_any_median	1.540872526162007
driven_any_min	1.2613490626227757
driven_lanedir_consec_max	0.6935156511660254
driven_lanedir_consec_mean	0.5011603005189074
driven_lanedir_consec_min	0.22687904419167904
driven_lanedir_max	0.6935156511660254
driven_lanedir_mean	0.5048368725998398
driven_lanedir_median	0.5494763975208273
driven_lanedir_min	0.22687904419167904
get_duckie_state_max	1.3731993161714995e-06
get_duckie_state_mean	1.264459449853148e-06
get_duckie_state_median	1.2544238564000267e-06
get_duckie_state_min	1.175790770441039e-06
get_robot_state_max	0.0039176459495837875
get_robot_state_mean	0.0035549533495037805
get_robot_state_median	0.0034583617378525907
get_robot_state_min	0.0033854439727261537
get_state_dump_max	0.004921019077301025
get_state_dump_mean	0.004641133192925258
get_state_dump_median	0.004611182574848615
get_state_dump_min	0.004421148544702775
get_ui_image_max	0.03387240653342389
get_ui_image_mean	0.029363049681270993
get_ui_image_median	0.029355403332538985
get_ui_image_min	0.02486898552658211
in-drivable-lane_max	9.750000000000016
in-drivable-lane_mean	4.424999999999997
in-drivable-lane_min	1.9999999999999951
per-episodes	details {"LF-norm-loop-000-ego0": {"driven_any": 3.584892340967327, "get_ui_image": 0.02727050508171451, "step_physics": 0.0950988987994089, "survival_time": 11.300000000000026, "driven_lanedir": 0.22687904419167904, "get_state_dump": 0.004716238786470523, "get_robot_state": 0.0035171991928033366, "sim_render-ego0": 0.00381502285927928, "get_duckie_state": 1.2330546778204157e-06, "in-drivable-lane": 9.750000000000016, "deviation-heading": 1.2376476894062516, "agent_compute-ego0": 0.01202858282080831, "complete-iteration": 0.15967566536386632, "set_robot_commands": 0.0020546545541233955, "deviation-center-line": 0.15769364294070565, "driven_lanedir_consec": 0.22687904419167904, "sim_compute_sim_state": 0.009169875787743389, "sim_compute_performance-ego0": 0.0019152500555903904}, "LF-norm-zigzag-000-ego0": {"driven_any": 1.2613490626227757, "get_ui_image": 0.03387240653342389, "step_physics": 0.12556611730697306, "survival_time": 4.6499999999999915, "driven_lanedir": 0.4127998105203827, "get_state_dump": 0.004506126363226708, "get_robot_state": 0.003399524282901845, "sim_render-ego0": 0.003661591955955992, "get_duckie_state": 1.2757930349796375e-06, "in-drivable-lane": 1.9999999999999951, "deviation-heading": 2.296491387142445, "agent_compute-ego0": 0.012048003521371396, "complete-iteration": 0.1956741505480827, "set_robot_commands": 0.0019755617101141747, "deviation-center-line": 0.2264256960389139, "driven_lanedir_consec": 0.3980935221966533, "sim_compute_sim_state": 0.008761367899306278, "sim_compute_performance-ego0": 0.001795413646292179}, "LF-norm-techtrack-000-ego0": {"driven_any": 1.4185786724850264, "get_ui_image": 0.03144030158336346, "step_physics": 0.11716584517405584, "survival_time": 5.14999999999999, "driven_lanedir": 0.6935156511660254, "get_state_dump": 0.004921019077301025, "get_robot_state": 0.0039176459495837875, "sim_render-ego0": 0.0040986033586355355, "get_duckie_state": 1.3731993161714995e-06, "in-drivable-lane": 2.549999999999991, "deviation-heading": 0.686797403822885, "agent_compute-ego0": 0.013074104602520283, "complete-iteration": 0.1877201795578003, "set_robot_commands": 0.002273282179465661, "deviation-center-line": 0.12748140015958784, "driven_lanedir_consec": 0.6935156511660254, "sim_compute_sim_state": 0.008621509258563701, "sim_compute_performance-ego0": 0.0021084730441753683}, "LF-norm-small_loop-000-ego0": {"driven_any": 1.6631663798389875, "get_ui_image": 0.02486898552658211, "step_physics": 0.08911847049354488, "survival_time": 5.799999999999987, "driven_lanedir": 0.6861529845212719, "get_state_dump": 0.004421148544702775, "get_robot_state": 0.0033854439727261537, "sim_render-ego0": 0.0035882994659945495, "get_duckie_state": 1.175790770441039e-06, "in-drivable-lane": 3.399999999999988, "deviation-heading": 0.39984030832751527, "agent_compute-ego0": 0.011002259376721503, "complete-iteration": 0.1448708864358755, "set_robot_commands": 0.0019115994119236611, "deviation-center-line": 0.0848651316511036, "driven_lanedir_consec": 0.6861529845212719, "sim_compute_sim_state": 0.004691792349530081, "sim_compute_performance-ego0": 0.001799923741919363}}
set_robot_commands_max	0.002273282179465661
set_robot_commands_mean	0.0020537744639067233
set_robot_commands_median	0.002015108132118786
set_robot_commands_min	0.0019115994119236611
sim_compute_performance-ego0_max	0.0021084730441753683
sim_compute_performance-ego0_mean	0.001904765121994325
sim_compute_performance-ego0_median	0.0018575868987548768
sim_compute_performance-ego0_min	0.001795413646292179
sim_compute_sim_state_max	0.009169875787743389
sim_compute_sim_state_mean	0.0078111363237858625
sim_compute_sim_state_median	0.00869143857893499
sim_compute_sim_state_min	0.004691792349530081
sim_render-ego0_max	0.0040986033586355355
sim_render-ego0_mean	0.003790879409966339
sim_render-ego0_median	0.003738307407617636
sim_render-ego0_min	0.0035882994659945495
simulation-passed	1
step_physics_max	0.12556611730697306
step_physics_mean	0.10673733294349567
step_physics_median	0.10613237198673237
step_physics_min	0.08911847049354488
survival_time_max	11.300000000000026
survival_time_mean	6.724999999999999
survival_time_min	4.6499999999999915

No reset possible

56815

LFv-sim

success

yes

2020-12-03 07:34:25+00:00

2020-12-03 07:41:22+00:00

0:06:57

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

56814

LFv-sim

success

yes

2020-12-03 07:34:08+00:00

2020-12-03 07:39:13+00:00

0:05:05

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

56798

LFv-sim

success

yes

2020-12-03 07:30:50+00:00

2020-12-03 07:35:46+00:00

0:04:56

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

50749

LFv-sim

error

2020-12-02 08:15:28+00:00

2020-12-02 08:17:04+00:00

0:01:36

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140679829095712
- M:video_aido:cmdline(in:/;out:/) 140684547214784
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

50738

LFv-sim

error

2020-12-02 08:14:47+00:00

2020-12-02 08:17:23+00:00

0:02:36

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140490131857616
- M:video_aido:cmdline(in:/;out:/) 140490131299920
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

50732

LFv-sim

host-error

2020-12-02 08:14:22+00:00

2020-12-02 08:16:19+00:00

0:01:57

The container "evalu [...]

The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

50729

LFv-sim

error

2020-12-02 08:13:51+00:00

2020-12-02 08:16:03+00:00

0:02:12

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140410853050064
- M:video_aido:cmdline(in:/;out:/) 140410853027952
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40292

LFv-sim

success

2020-11-07 10:24:18+00:00

2020-11-07 10:29:29+00:00

0:05:11

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40291

LFv-sim

success

2020-11-07 10:24:03+00:00

2020-11-07 10:30:20+00:00

0:06:17

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40290

LFv-sim

success

2020-11-07 10:23:19+00:00

2020-11-07 10:26:37+00:00

0:03:18

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40289

LFv-sim

success

2020-11-07 10:22:27+00:00

2020-11-07 10:29:04+00:00

0:06:37

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

39671

LFv-sim

success

2020-11-06 16:16:32+00:00

2020-11-06 16:19:58+00:00

0:03:26

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

38867

LFv-sim

success

2020-11-05 00:36:55+00:00

2020-11-05 00:40:27+00:00

0:03:32

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

38866

LFv-sim

success

2020-11-05 00:36:23+00:00

2020-11-05 00:39:50+00:00

0:03:27

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

38865

LFv-sim

success

2020-11-05 00:35:57+00:00

2020-11-05 00:40:40+00:00

0:04:43

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

38864

LFv-sim

success

2020-11-05 00:35:36+00:00

2020-11-05 00:40:09+00:00

0:04:33

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible