Submission 9774

Submission

9774

Competing

yes

Challenge

aido5-LF-sim-validation

User

Yishu Malhotra 🇨🇦

Date submitted

2020-10-30 02:30:20+00:00

Last status update

2021-12-04 20:52:33+00:00

Complete

complete

Details

Evaluation is complete.

Sisters

Result

💚

Jobs

LFv-sim: 58091

User label

exercise_ros_template

Admin priority

Blessing

n/a

User priority

58091

Click the images to see detailed statistics about the episode.

LF-norm-loop-000

LF-norm-small_loop-000

LF-norm-techtrack-000

LF-norm-zigzag-000

Evaluation jobs for this submission

Show only up-to-date jobs

Job ID

step

status

up to date

date started

date completed

duration

message

58091

LFv-sim

success

yes

2020-12-03 21:06:50+00:00

2020-12-03 21:13:11+00:00

0:06:21

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

driven_lanedir_consec_median	0.4384584331043765
survival_time_median	10.725000000000016
deviation-center-line_median	0.16430406830755806
in-drivable-lane_median	7.600000000000017

other stats

agent_compute-ego0_max	0.014918078909387122
agent_compute-ego0_mean	0.01322786384017525
agent_compute-ego0_median	0.01291822136007
agent_compute-ego0_min	0.012156933731173877
complete-iteration_max	0.2141993792660265
complete-iteration_mean	0.17281583907325596
complete-iteration_median	0.1665525990858871
complete-iteration_min	0.1439587788552231
deviation-center-line_max	0.223461692314485
deviation-center-line_mean	0.16219810946653318
deviation-center-line_min	0.0967226089365316
deviation-heading_max	1.9549907815321208
deviation-heading_mean	1.137185012068172
deviation-heading_median	1.0988853580458875
deviation-heading_min	0.3959785506487917
driven_any_max	3.4868753439111426
driven_any_mean	2.2657448479496995
driven_any_median	2.465552128867132
driven_any_min	0.6449997901533923
driven_lanedir_consec_max	0.6957957263013493
driven_lanedir_consec_mean	0.44505994135727334
driven_lanedir_consec_min	0.20752717291899092
driven_lanedir_max	0.6957957263013493
driven_lanedir_mean	0.44505994135727334
driven_lanedir_median	0.4384584331043765
driven_lanedir_min	0.20752717291899092
get_duckie_state_max	1.1047221118618982e-06
get_duckie_state_mean	1.0877159857621016e-06
get_duckie_state_median	1.0866291126147384e-06
get_duckie_state_min	1.0728836059570312e-06
get_robot_state_max	0.003619419111238493
get_robot_state_mean	0.003555363957329428
get_robot_state_median	0.00355574121852565
get_robot_state_min	0.0034905542810279204
get_state_dump_max	0.007099321089595197
get_state_dump_mean	0.005103494103013868
get_state_dump_median	0.004448416547372656
get_state_dump_min	0.004417822227714965
get_ui_image_max	0.035618038062589716
get_ui_image_mean	0.030743994895479018
get_ui_image_median	0.030578611008832565
get_ui_image_min	0.026200719501661217
in-drivable-lane_max	12.30000000000007
in-drivable-lane_mean	7.562500000000024
in-drivable-lane_min	2.749999999999993
per-episodes	details {"LF-norm-loop-000-ego0": {"driven_any": 3.4868753439111426, "get_ui_image": 0.02848191778142969, "step_physics": 0.09062401064625988, "survival_time": 14.250000000000068, "driven_lanedir": 0.22813062331527512, "get_state_dump": 0.004456399204014065, "get_robot_state": 0.003619419111238493, "sim_render-ego0": 0.003674519645584213, "get_duckie_state": 1.0728836059570312e-06, "in-drivable-lane": 12.30000000000007, "deviation-heading": 1.4856170213324444, "agent_compute-ego0": 0.014918078909387122, "complete-iteration": 0.15990704923242957, "set_robot_commands": 0.0021584775898006413, "deviation-center-line": 0.19447135376268967, "driven_lanedir_consec": 0.22813062331527512, "sim_compute_sim_state": 0.00999847658864268, "sim_compute_performance-ego0": 0.0019036339713143305}, "LF-norm-zigzag-000-ego0": {"driven_any": 0.6449997901533923, "get_ui_image": 0.035618038062589716, "step_physics": 0.13772039815603968, "survival_time": 4.099999999999993, "driven_lanedir": 0.20752717291899092, "get_state_dump": 0.007099321089595197, "get_robot_state": 0.0034905542810279204, "sim_render-ego0": 0.00361607448164239, "get_duckie_state": 1.0800648884600905e-06, "in-drivable-lane": 2.749999999999993, "deviation-heading": 0.7121536947593305, "agent_compute-ego0": 0.012692129755594644, "complete-iteration": 0.2141993792660265, "set_robot_commands": 0.0020567968667271628, "deviation-center-line": 0.0967226089365316, "driven_lanedir_consec": 0.20752717291899092, "sim_compute_sim_state": 0.009985685348510742, "sim_compute_performance-ego0": 0.0018484649888004163}, "LF-norm-techtrack-000-ego0": {"driven_any": 3.0767103075719566, "get_ui_image": 0.03267530423623544, "step_physics": 0.10115753809611004, "survival_time": 13.450000000000056, "driven_lanedir": 0.6957957263013493, "get_state_dump": 0.004440433890731246, "get_robot_state": 0.0035954007395991574, "sim_render-ego0": 0.003752121218928584, "get_duckie_state": 1.0931933367693866e-06, "in-drivable-lane": 9.800000000000054, "deviation-heading": 1.9549907815321208, "agent_compute-ego0": 0.013144312964545357, "complete-iteration": 0.17319814893934465, "set_robot_commands": 0.0022354708777533637, "deviation-center-line": 0.223461692314485, "driven_lanedir_consec": 0.6957957263013493, "sim_compute_sim_state": 0.010203409194946288, "sim_compute_performance-ego0": 0.0019202179378933375}, "LF-norm-small_loop-000-ego0": {"driven_any": 1.8543939501623072, "get_ui_image": 0.026200719501661217, "step_physics": 0.08511515581829948, "survival_time": 7.99999999999998, "driven_lanedir": 0.6487862428934779, "get_state_dump": 0.004417822227714965, "get_robot_state": 0.0035160816974521424, "sim_render-ego0": 0.0036001916257491023, "get_duckie_state": 1.1047221118618982e-06, "in-drivable-lane": 5.399999999999981, "deviation-heading": 0.3959785506487917, "agent_compute-ego0": 0.012156933731173877, "complete-iteration": 0.1439587788552231, "set_robot_commands": 0.002056789694365507, "deviation-center-line": 0.13413678285242642, "driven_lanedir_consec": 0.6487862428934779, "sim_compute_sim_state": 0.004955382080552, "sim_compute_performance-ego0": 0.0018665568428750364}}
set_robot_commands_max	0.0022354708777533637
set_robot_commands_mean	0.002126883757161669
set_robot_commands_median	0.002107637228263902
set_robot_commands_min	0.002056789694365507
sim_compute_performance-ego0_max	0.0019202179378933375
sim_compute_performance-ego0_mean	0.0018847184352207805
sim_compute_performance-ego0_median	0.001885095407094683
sim_compute_performance-ego0_min	0.0018484649888004163
sim_compute_sim_state_max	0.010203409194946288
sim_compute_sim_state_mean	0.008785738303162927
sim_compute_sim_state_median	0.009992080968576712
sim_compute_sim_state_min	0.004955382080552
sim_render-ego0_max	0.003752121218928584
sim_render-ego0_mean	0.003660726742976072
sim_render-ego0_median	0.0036452970636133017
sim_render-ego0_min	0.0036001916257491023
simulation-passed	1
step_physics_max	0.13772039815603968
step_physics_mean	0.10365427567917726
step_physics_median	0.09589077437118496
step_physics_min	0.08511515581829948
survival_time_max	14.250000000000068
survival_time_mean	9.950000000000024
survival_time_min	4.099999999999993

No reset possible

52148

LFv-sim

error

2020-12-02 11:01:47+00:00

2020-12-02 11:05:45+00:00

0:03:58

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140540875009856
- M:video_aido:cmdline(in:/;out:/) 140540875590576
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

52134

LFv-sim

error

2020-12-02 10:59:56+00:00

2020-12-02 11:05:37+00:00

0:05:41

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140518555044304
- M:video_aido:cmdline(in:/;out:/) 140518555981712
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

52130

LFv-sim

error

2020-12-02 10:59:33+00:00

2020-12-02 11:02:33+00:00

0:03:00

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140188697102224
- M:video_aido:cmdline(in:/;out:/) 140188697101648
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

52126

LFv-sim

timeout

2020-12-02 10:59:01+00:00

2020-12-02 11:07:29+00:00

0:08:28

Timeout because eval [...]

Timeout because evaluator contacted us

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

41646

LFv-sim

success

2020-11-08 11:08:23+00:00

2020-11-08 11:16:40+00:00

0:08:17

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

41645

LFv-sim

success

2020-11-08 11:05:46+00:00

2020-11-08 11:14:05+00:00

0:08:19

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

41644

LFv-sim

success

2020-11-08 11:04:07+00:00

2020-11-08 11:12:13+00:00

0:08:06

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

38036

LFv-sim

success

2020-11-03 22:19:49+00:00

2020-11-03 22:28:49+00:00

0:09:00

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

38035

LFv-sim

success

2020-11-03 22:19:13+00:00

2020-11-03 22:29:09+00:00

0:09:56

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

36197

LFv-sim

success

2020-11-02 02:27:01+00:00

2020-11-02 02:35:19+00:00

0:08:18

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

36195

LFv-sim

success

2020-11-02 02:26:36+00:00

2020-11-02 02:35:28+00:00

0:08:52

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

35607

LFv-sim

success

2020-11-01 03:23:34+00:00

2020-11-01 03:24:27+00:00

0:00:53

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

35605

LFv-sim

success

2020-11-01 03:23:03+00:00

2020-11-01 03:23:57+00:00

0:00:54

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

35250

LFv-sim

success

2020-10-30 19:01:24+00:00

2020-10-30 19:18:54+00:00

0:17:30

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

35225

LFv-sim

success

2020-10-30 02:31:06+00:00

2020-10-30 02:47:02+00:00

0:15:56

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible