Submission 11054

Submission

11054

Competing

yes

Challenge

aido5-LF-sim-validation

User

Ayman Shams 🇨🇦

Date submitted

2020-11-04 22:30:39+00:00

Last status update

2021-12-04 20:52:39+00:00

Complete

complete

Details

Evaluation is complete.

Sisters

Result

💚

Jobs

LFv-sim: 56947

User label

exercise_ros_template

Admin priority

Blessing

n/a

User priority

56947

Click the images to see detailed statistics about the episode.

LF-norm-loop-000

LF-norm-small_loop-000

LF-norm-techtrack-000

LF-norm-zigzag-000

Evaluation jobs for this submission

Show only up-to-date jobs

Job ID

step

status

up to date

date started

date completed

duration

message

56947

LFv-sim

success

yes

2020-12-03 07:58:55+00:00

2020-12-03 08:09:00+00:00

0:10:05

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

driven_lanedir_consec_median	0.3838181412687161
survival_time_median	13.475000000000056
deviation-center-line_median	0.20641474008397864
in-drivable-lane_median	10.350000000000035

other stats

agent_compute-ego0_max	0.013330657883445816
agent_compute-ego0_mean	0.013079458707522966
agent_compute-ego0_median	0.013021874165227492
agent_compute-ego0_min	0.012943428616191063
complete-iteration_max	0.232691859090051
complete-iteration_mean	0.1883820264837523
complete-iteration_median	0.17782115995205716
complete-iteration_min	0.1651939269408439
deviation-center-line_max	0.4195223219775499
deviation-center-line_mean	0.230579047789394
deviation-center-line_min	0.0899643890120688
deviation-heading_max	2.8221523610338606
deviation-heading_mean	1.613592794945695
deviation-heading_median	1.533078845539745
deviation-heading_min	0.566061127669429
driven_any_max	3.712763565439082
driven_any_mean	2.3166642722557795
driven_any_median	2.468870921500827
driven_any_min	0.6161516805823831
driven_lanedir_consec_max	0.6592337619078826
driven_lanedir_consec_mean	0.40151292481547624
driven_lanedir_consec_min	0.1791816548165901
driven_lanedir_max	0.6592337619078826
driven_lanedir_mean	0.40151292481547624
driven_lanedir_median	0.3838181412687161
driven_lanedir_min	0.1791816548165901
get_duckie_state_max	1.5890714034293465e-06
get_duckie_state_mean	1.4412043428146332e-06
get_duckie_state_median	1.4651378892463331e-06
get_duckie_state_min	1.2454701893365206e-06
get_robot_state_max	0.00409707861039245
get_robot_state_mean	0.0038468495111771512
get_robot_state_median	0.0038028799502768896
get_robot_state_min	0.003684559533762377
get_state_dump_max	0.005276825821515426
get_state_dump_mean	0.004902260411118357
get_state_dump_median	0.004850055033398092
get_state_dump_min	0.004632105756161818
get_ui_image_max	0.03516465841337692
get_ui_image_mean	0.030424346766291153
get_ui_image_median	0.030026105076660346
get_ui_image_min	0.02648051849846701
in-drivable-lane_max	14.9500000000001
in-drivable-lane_mean	9.56250000000004
in-drivable-lane_min	2.599999999999992
per-episodes	details {"LF-norm-loop-000-ego0": {"driven_any": 3.712763565439082, "get_ui_image": 0.028584748211473523, "step_physics": 0.10034196506632435, "survival_time": 20.15000000000015, "driven_lanedir": 0.6592337619078826, "get_state_dump": 0.005001980479400937, "get_robot_state": 0.0038778534029970074, "sim_render-ego0": 0.004086424218546046, "get_duckie_state": 1.5579827941290222e-06, "in-drivable-lane": 14.9500000000001, "deviation-heading": 2.1574547188219, "agent_compute-ego0": 0.013330657883445816, "complete-iteration": 0.1706877008523091, "set_robot_commands": 0.002346948821945946, "deviation-center-line": 0.4195223219775499, "driven_lanedir_consec": 0.6592337619078826, "sim_compute_sim_state": 0.010898359931341492, "sim_compute_performance-ego0": 0.00212262231524628}, "LF-norm-zigzag-000-ego0": {"driven_any": 0.6161516805823831, "get_ui_image": 0.03516465841337692, "step_physics": 0.15702470790508183, "survival_time": 4.249999999999993, "driven_lanedir": 0.1791816548165901, "get_state_dump": 0.004698129587395247, "get_robot_state": 0.003684559533762377, "sim_render-ego0": 0.00409514959468398, "get_duckie_state": 1.3722929843636443e-06, "in-drivable-lane": 2.599999999999992, "deviation-heading": 0.9087029722575902, "agent_compute-ego0": 0.012943428616191063, "complete-iteration": 0.232691859090051, "set_robot_commands": 0.002321977948033533, "deviation-center-line": 0.0899643890120688, "driven_lanedir_consec": 0.1791816548165901, "sim_compute_sim_state": 0.010597536730211837, "sim_compute_performance-ego0": 0.0020717243815577308}, "LF-norm-techtrack-000-ego0": {"driven_any": 3.02092500108308, "get_ui_image": 0.03146746194184716, "step_physics": 0.11362194516765538, "survival_time": 16.700000000000102, "driven_lanedir": 0.5031505319508306, "get_state_dump": 0.004632105756161818, "get_robot_state": 0.003727906497556772, "sim_render-ego0": 0.003910185685798304, "get_duckie_state": 1.2454701893365206e-06, "in-drivable-lane": 12.200000000000063, "deviation-heading": 2.8221523610338606, "agent_compute-ego0": 0.013066136659081302, "complete-iteration": 0.1849546190518052, "set_robot_commands": 0.0021892234460631414, "deviation-center-line": 0.3041871845771305, "driven_lanedir_consec": 0.5031505319508306, "sim_compute_sim_state": 0.010295650140563054, "sim_compute_performance-ego0": 0.001958309714473895}, "LF-norm-small_loop-000-ego0": {"driven_any": 1.9168168419185745, "get_ui_image": 0.02648051849846701, "step_physics": 0.10225618348538296, "survival_time": 10.25000000000001, "driven_lanedir": 0.2644857505866016, "get_state_dump": 0.005276825821515426, "get_robot_state": 0.00409707861039245, "sim_render-ego0": 0.004131425931615737, "get_duckie_state": 1.5890714034293465e-06, "in-drivable-lane": 8.50000000000001, "deviation-heading": 0.566061127669429, "agent_compute-ego0": 0.012977611671373682, "complete-iteration": 0.1651939269408439, "set_robot_commands": 0.002355109140710923, "deviation-center-line": 0.1086422955908267, "driven_lanedir_consec": 0.2644857505866016, "sim_compute_sim_state": 0.0054630550366003535, "sim_compute_performance-ego0": 0.002061907527516189}}
set_robot_commands_max	0.002355109140710923
set_robot_commands_mean	0.0023033148391883858
set_robot_commands_median	0.002334463384989739
set_robot_commands_min	0.0021892234460631414
sim_compute_performance-ego0_max	0.00212262231524628
sim_compute_performance-ego0_mean	0.0020536409846985235
sim_compute_performance-ego0_median	0.00206681595453696
sim_compute_performance-ego0_min	0.001958309714473895
sim_compute_sim_state_max	0.010898359931341492
sim_compute_sim_state_mean	0.009313650459679183
sim_compute_sim_state_median	0.010446593435387444
sim_compute_sim_state_min	0.0054630550366003535
sim_render-ego0_max	0.004131425931615737
sim_render-ego0_mean	0.004055796357661017
sim_render-ego0_median	0.004090786906615013
sim_render-ego0_min	0.003910185685798304
simulation-passed	1
step_physics_max	0.15702470790508183
step_physics_mean	0.11831120040611114
step_physics_median	0.10793906432651916
step_physics_min	0.10034196506632435
survival_time_max	20.15000000000015
survival_time_mean	12.837500000000064
survival_time_min	4.249999999999993

No reset possible

56945

LFv-sim

success

yes

2020-12-03 07:58:13+00:00

2020-12-03 08:11:08+00:00

0:12:55

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

56944

LFv-sim

success

yes

2020-12-03 07:57:57+00:00

2020-12-03 08:10:31+00:00

0:12:34

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

56942

LFv-sim

success

yes

2020-12-03 07:57:45+00:00

2020-12-03 08:11:04+00:00

0:13:19

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

56935

LFv-sim

success

yes

2020-12-03 07:55:15+00:00

2020-12-03 08:04:35+00:00

0:09:20

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

50801

LFv-sim

error

2020-12-02 08:20:20+00:00

2020-12-02 08:23:35+00:00

0:03:15

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 139928054002928
- M:video_aido:cmdline(in:/;out:/) 139928054042784
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

50798

LFv-sim

error

2020-12-02 08:20:18+00:00

2020-12-02 08:27:59+00:00

0:07:41

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140008558706064
- M:video_aido:cmdline(in:/;out:/) 140008558677200
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

50789

LFv-sim

error

2020-12-02 08:19:24+00:00

2020-12-02 08:26:06+00:00

0:06:42

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140087046311120
- M:video_aido:cmdline(in:/;out:/) 140087046263664
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40360

LFv-sim

success

2020-11-07 11:21:32+00:00

2020-11-07 11:33:11+00:00

0:11:39

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40358

LFv-sim

success

2020-11-07 11:21:09+00:00

2020-11-07 11:32:49+00:00

0:11:40

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40356

LFv-sim

success

2020-11-07 11:20:07+00:00

2020-11-07 11:31:31+00:00

0:11:24

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40351

LFv-sim

success

2020-11-07 11:18:18+00:00

2020-11-07 11:30:36+00:00

0:12:18

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40346

LFv-sim

success

2020-11-07 11:12:53+00:00

2020-11-07 11:22:16+00:00

0:09:23

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

39718

LFv-sim

success

2020-11-06 17:06:11+00:00

2020-11-06 17:19:35+00:00

0:13:24

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

39717

LFv-sim

success

2020-11-06 17:05:25+00:00

2020-11-06 17:14:36+00:00

0:09:11

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

39708

LFv-sim

success

2020-11-06 16:55:58+00:00

2020-11-06 17:05:08+00:00

0:09:10

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

39707

LFv-sim

success

2020-11-06 16:55:25+00:00

2020-11-06 17:04:29+00:00

0:09:04

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

38817

LFv-sim

success

2020-11-04 22:32:14+00:00

2020-11-04 22:40:49+00:00

0:08:35

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible