Submission 6850

Submission

6850

Competing

yes

Challenge

aido5-LF-sim-validation

User

Himanshu Arora 🇨🇦

Date submitted

2020-10-18 23:07:01+00:00

Last status update

2021-12-04 20:52:37+00:00

Complete

complete

Details

Evaluation is complete.

Sisters

Result

💚

Jobs

LFv-sim: 58522

User label

exercise_ros_template

Admin priority

Blessing

n/a

User priority

58522

Click the images to see detailed statistics about the episode.

LF-norm-loop-000

LF-norm-small_loop-000

LF-norm-techtrack-000

LF-norm-zigzag-000

Evaluation jobs for this submission

Show only up-to-date jobs

Job ID

step

status

up to date

date started

date completed

duration

message

58522

LFv-sim

success

yes

2020-12-04 00:12:28+00:00

2020-12-04 00:55:51+00:00

0:43:23

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

driven_lanedir_consec_median	0.0
survival_time_median	59.99999999999873
deviation-center-line_median	1.2422730096440104
in-drivable-lane_median	0.0

other stats

agent_compute-ego0_max	0.01335668941024539
agent_compute-ego0_mean	0.01255814290662094
agent_compute-ego0_median	0.01272423191928149
agent_compute-ego0_min	0.011427418377675382
complete-iteration_max	0.34877707301131095
complete-iteration_mean	0.2986952401418471
complete-iteration_median	0.29875832126102875
complete-iteration_min	0.24848724503402012
deviation-center-line_max	4.053503393024394
deviation-center-line_mean	1.731069028434027
deviation-center-line_min	0.386226701423694
deviation-heading_max	27.859809596736422
deviation-heading_mean	14.925250437835436
deviation-heading_median	14.309269950178932
deviation-heading_min	3.22265225424745
driven_any_max	2.6645352591003757e-13
driven_any_mean	1.9984014443252818e-13
driven_any_median	2.6645352591003757e-13
driven_any_min	0.0
driven_lanedir_consec_max	0.000286102294921875
driven_lanedir_consec_mean	7.152557373046875e-05
driven_lanedir_consec_min	0.0
driven_lanedir_max	0.000286102294921875
driven_lanedir_mean	7.152557373046875e-05
driven_lanedir_median	0.0
driven_lanedir_min	0.0
get_duckie_state_max	2.0276497643158696e-06
get_duckie_state_mean	1.991470092341465e-06
get_duckie_state_median	1.9879464205853848e-06
get_duckie_state_min	1.9623377638792216e-06
get_robot_state_max	0.0037624716857986382
get_robot_state_mean	0.003748689761864553
get_robot_state_median	0.0037454404402136503
get_robot_state_min	0.0037414064812322737
get_state_dump_max	0.004793865099040595
get_state_dump_mean	0.004730207189532938
get_state_dump_median	0.004726821735042219
get_state_dump_min	0.00467332018900672
get_ui_image_max	0.034940667394595185
get_ui_image_mean	0.03046631277054176
get_ui_image_median	0.03010337874851655
get_ui_image_min	0.026717826190538748
in-drivable-lane_max	0.0
in-drivable-lane_mean	0.0
in-drivable-lane_min	0.0
per-episodes	details {"LF-norm-loop-000-ego0": {"driven_any": 2.6645352591003757e-13, "get_ui_image": 0.02801447923137782, "step_physics": 0.21519152230763813, "survival_time": 59.99999999999873, "driven_lanedir": 0.0, "get_state_dump": 0.00469688015317639, "get_robot_state": 0.0037436888279466208, "sim_render-ego0": 0.0038047225945795904, "get_duckie_state": 1.9786161348087207e-06, "in-drivable-lane": 0.0, "deviation-heading": 22.66279310353771, "agent_compute-ego0": 0.011427418377675382, "complete-iteration": 0.2797032951018296, "set_robot_commands": 0.002253900261147632, "deviation-center-line": 4.053503393024394, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.008438817269597622, "sim_compute_performance-ego0": 0.0020470025636671386}, "LF-norm-zigzag-000-ego0": {"driven_any": 0.0, "get_ui_image": 0.034940667394595185, "step_physics": 0.27234006483886364, "survival_time": 59.99999999999873, "driven_lanedir": 0.000286102294921875, "get_state_dump": 0.00467332018900672, "get_robot_state": 0.0037414064812322737, "sim_render-ego0": 0.00386026002882323, "get_duckie_state": 2.0276497643158696e-06, "in-drivable-lane": 0.0, "deviation-heading": 27.859809596736422, "agent_compute-ego0": 0.01335668941024539, "complete-iteration": 0.34877707301131095, "set_robot_commands": 0.002286069299855101, "deviation-center-line": 1.0457540566888746, "driven_lanedir_consec": 0.000286102294921875, "sim_compute_sim_state": 0.01142896402884681, "sim_compute_performance-ego0": 0.0020638153416032497}, "LF-norm-techtrack-000-ego0": {"driven_any": 2.6645352591003757e-13, "get_ui_image": 0.03219227826565529, "step_physics": 0.2475417748180456, "survival_time": 59.99999999999873, "driven_lanedir": 0.0, "get_state_dump": 0.004756763316908049, "get_robot_state": 0.00374719205248068, "sim_render-ego0": 0.0038100523714419705, "get_duckie_state": 1.9972767063620485e-06, "in-drivable-lane": 0.0, "deviation-heading": 3.22265225424745, "agent_compute-ego0": 0.01259682140779138, "complete-iteration": 0.31781334742022793, "set_robot_commands": 0.0022375339473117697, "deviation-center-line": 1.4387919625991463, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.008854205761225793, "sim_compute_performance-ego0": 0.0019911145488983586}, "LF-norm-small_loop-000-ego0": {"driven_any": 2.6645352591003757e-13, "get_ui_image": 0.026717826190538748, "step_physics": 0.18611795360301556, "survival_time": 59.99999999999873, "driven_lanedir": 0.0, "get_state_dump": 0.004793865099040595, "get_robot_state": 0.0037624716857986382, "sim_render-ego0": 0.0038285787456934896, "get_duckie_state": 1.9623377638792216e-06, "in-drivable-lane": 0.0, "deviation-heading": 5.955746796820156, "agent_compute-ego0": 0.0128516424307716, "complete-iteration": 0.24848724503402012, "set_robot_commands": 0.0022561460807857467, "deviation-center-line": 0.386226701423694, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.006081110035549294, "sim_compute_performance-ego0": 0.0019923511095388444}}
set_robot_commands_max	0.002286069299855101
set_robot_commands_mean	0.0022584123972750622
set_robot_commands_median	0.0022550231709666893
set_robot_commands_min	0.0022375339473117697
sim_compute_performance-ego0_max	0.0020638153416032497
sim_compute_performance-ego0_mean	0.002023570890926898
sim_compute_performance-ego0_median	0.0020196768366029915
sim_compute_performance-ego0_min	0.0019911145488983586
sim_compute_sim_state_max	0.01142896402884681
sim_compute_sim_state_mean	0.00870077427380488
sim_compute_sim_state_median	0.008646511515411707
sim_compute_sim_state_min	0.006081110035549294
sim_render-ego0_max	0.00386026002882323
sim_render-ego0_mean	0.00382590343513457
sim_render-ego0_median	0.00381931555856773
sim_render-ego0_min	0.0038047225945795904
simulation-passed	1
step_physics_max	0.27234006483886364
step_physics_mean	0.2302978288918907
step_physics_median	0.2313666485628419
step_physics_min	0.18611795360301556
survival_time_max	59.99999999999873
survival_time_mean	59.99999999999873
survival_time_min	59.99999999999873

No reset possible

58521

LFv-sim

success

yes

2020-12-04 00:11:56+00:00

2020-12-04 00:55:13+00:00

0:43:17

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

58518

LFv-sim

success

yes

2020-12-04 00:10:09+00:00

2020-12-04 00:51:19+00:00

0:41:10

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

58515

LFv-sim

success

yes

2020-12-04 00:09:19+00:00

2020-12-04 00:52:40+00:00

0:43:21

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

52461

LFv-sim

error

2020-12-02 11:43:05+00:00

2020-12-02 11:57:53+00:00

0:14:48

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140103820572080
- M:video_aido:cmdline(in:/;out:/) 140103820571024
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

52455

LFv-sim

error

2020-12-02 11:42:03+00:00

2020-12-02 11:53:18+00:00

0:11:15

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140270423253968
- M:video_aido:cmdline(in:/;out:/) 140270423254784
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

52450

LFv-sim

error

2020-12-02 11:41:36+00:00

2020-12-02 11:52:36+00:00

0:11:00

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 139674177573264
- M:video_aido:cmdline(in:/;out:/) 139674177576816
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

41790

LFv-sim

success

2020-11-08 14:44:48+00:00

2020-11-08 14:54:04+00:00

0:09:16

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

38338

LFv-sim

success

2020-11-04 02:25:50+00:00

2020-11-04 02:35:01+00:00

0:09:11

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

36440

LFv-sim

success

2020-11-02 05:31:28+00:00

2020-11-02 05:41:33+00:00

0:10:05

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

35857

LFv-sim

success

2020-11-01 04:30:00+00:00

2020-11-01 04:31:05+00:00

0:01:05

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

35439

LFv-sim

error

2020-10-31 07:18:21+00:00

2020-10-31 07:40:54+00:00

0:22:33

The result file is n [...]

The result file is not found in working dir /tmp/duckietown/DT18/evaluator/executions/aido5-LF-sim-validation/submission6850/LFv-sim-reg03-0c28c9d61367-1-job35439:

File '/tmp/duckietown/DT18/evaluator/executions/aido5-LF-sim-validation/submission6850/LFv-sim-reg03-0c28c9d61367-1-job35439/challenge-results/challenge_results.yaml' does not exist.

This usually means that the evaluator did not finish and some times that there was an import error.
Check the evaluator log to see what happened.

List of all files:

 -/tmp/duckietown/DT18/evaluator/executions/aido5-LF-sim-validation/submission6850/LFv-sim-reg03-0c28c9d61367-1-job35439/docker-compose.original.yaml
-/tmp/duckietown/DT18/evaluator/executions/aido5-LF-sim-validation/submission6850/LFv-sim-reg03-0c28c9d61367-1-job35439/docker-compose.yaml
-/tmp/duckietown/DT18/evaluator/executions/aido5-LF-sim-validation/submission6850/LFv-sim-reg03-0c28c9d61367-1-job35439/logs/challenges-runner/stdout.log
-/tmp/duckietown/DT18/evaluator/executions/aido5-LF-sim-validation/submission6850/LFv-sim-reg03-0c28c9d61367-1-job35439/logs/challenges-runner/stderr.log

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

35123

LFv-sim

success

2020-10-29 13:01:08+00:00

2020-10-29 13:24:15+00:00

0:23:07

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

35122

LFv-sim

success

2020-10-29 13:00:41+00:00

2020-10-29 13:23:32+00:00

0:22:51

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

33459

LFv-sim

success

2020-10-19 20:42:17+00:00

2020-10-19 21:08:37+00:00

0:26:20

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

33448

LFv-sim

success

2020-10-18 23:08:16+00:00

2020-10-18 23:31:44+00:00

0:23:28

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

33447

LFv-sim

success

2020-10-18 23:07:35+00:00

2020-10-18 23:31:01+00:00

0:23:26

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible