Duckietown Challenges Home Challenges Submissions

Submission 11731

Submission11731
Competingyes
Challengeaido5-LF-sim-validation
UserYishu Malhotra 🇨🇦
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
JobsLFv-sim: 53976
Next
User labelsim-exercise-2
Admin priority50
Blessingn/a
User priority50

53976

Click the images to see detailed statistics about the episode.

LF-norm-loop-000

LF-norm-small_loop-000

LF-norm-techtrack-000

LF-norm-zigzag-000

Evaluation jobs for this submission

Show only up-to-date jobs
Job IDstepstatusup to datedate starteddate completeddurationmessage
53976LFv-simsuccessyes0:24:35
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median2.959500031093986
survival_time_median39.474999999999575
deviation-center-line_median1.2855910031735545
in-drivable-lane_median19.94999999999959


other stats
agent_compute-ego0_max0.013052778226973438
agent_compute-ego0_mean0.012415218299938292
agent_compute-ego0_median0.012365811670920542
agent_compute-ego0_min0.01187647163093864
complete-iteration_max0.2300221853963598
complete-iteration_mean0.20462260221476591
complete-iteration_median0.2056346123569275
complete-iteration_min0.1771989987488488
deviation-center-line_max2.1613880395990472
deviation-center-line_mean1.2886103836782703
deviation-center-line_min0.4218714887669255
deviation-heading_max10.434138993679731
deviation-heading_mean5.984467030667154
deviation-heading_median5.781612095357342
deviation-heading_min1.9405049382741988
driven_any_max13.17439129702074
driven_any_mean7.965385293602357
driven_any_median7.14251124647881
driven_any_min4.402127384431073
driven_lanedir_consec_max6.200725341845583
driven_lanedir_consec_mean3.408573700564621
driven_lanedir_consec_min1.5145693982249282
driven_lanedir_max7.994032374750411
driven_lanedir_mean3.858079029731987
driven_lanedir_median2.961857172976305
driven_lanedir_min1.5145693982249282
get_duckie_state_max1.4493705628813551e-06
get_duckie_state_mean1.338144036640301e-06
get_duckie_state_median1.3186582306536334e-06
get_duckie_state_min1.2658891223725818e-06
get_robot_state_max0.003982772588186764
get_robot_state_mean0.003885365483679768
get_robot_state_median0.003938417805207251
get_robot_state_min0.003681853736117805
get_state_dump_max0.004989331912738708
get_state_dump_mean0.0048456304691725655
get_state_dump_median0.004932185369708211
get_state_dump_min0.004528819224535129
get_ui_image_max0.036694085959232216
get_ui_image_mean0.03277830733743511
get_ui_image_median0.03290651201955516
get_ui_image_min0.02860611935139789
in-drivable-lane_max42.499999999999
in-drivable-lane_mean21.74999999999956
in-drivable-lane_min4.600000000000065
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 13.17439129702074, "get_ui_image": 0.030390507573390586, "step_physics": 0.11919038539921412, "survival_time": 59.99999999999873, "driven_lanedir": 7.994032374750411, "get_state_dump": 0.004938728704142829, "get_robot_state": 0.003910528630837116, "sim_render-ego0": 0.004009766542941307, "get_duckie_state": 1.4493705628813551e-06, "in-drivable-lane": 25.849999999998985, "deviation-heading": 10.434138993679731, "agent_compute-ego0": 0.01211342486017054, "complete-iteration": 0.18916790272969192, "set_robot_commands": 0.0023564954879976728, "deviation-center-line": 2.1613880395990472, "driven_lanedir_consec": 6.200725341845583, "sim_compute_sim_state": 0.010028279294181526, "sim_compute_performance-ego0": 0.0021362084333942295}, "LF-norm-zigzag-000-ego0": {"driven_any": 4.629218673584049, "get_ui_image": 0.036694085959232216, "step_physics": 0.14637443358763988, "survival_time": 23.050000000000193, "driven_lanedir": 3.7380873889692943, "get_state_dump": 0.004528819224535129, "get_robot_state": 0.003681853736117805, "sim_render-ego0": 0.003718011306993889, "get_duckie_state": 1.2658891223725818e-06, "in-drivable-lane": 4.600000000000065, "deviation-heading": 5.798335189934956, "agent_compute-ego0": 0.01187647163093864, "complete-iteration": 0.2221013219841631, "set_robot_commands": 0.002218073064630682, "deviation-center-line": 1.5015203275657212, "driven_lanedir_consec": 3.7377136028363216, "sim_compute_sim_state": 0.010877978234064011, "sim_compute_performance-ego0": 0.002047968633247144}, "LF-norm-techtrack-000-ego0": {"driven_any": 9.65580381937357, "get_ui_image": 0.035422516465719735, "step_physics": 0.15178967267940682, "survival_time": 55.89999999999896, "driven_lanedir": 2.185626956983316, "get_state_dump": 0.004989331912738708, "get_robot_state": 0.0039663069795773855, "sim_render-ego0": 0.004062695626811964, "get_duckie_state": 1.3393200967224504e-06, "in-drivable-lane": 42.499999999999, "deviation-heading": 5.764889000779728, "agent_compute-ego0": 0.013052778226973438, "complete-iteration": 0.2300221853963598, "set_robot_commands": 0.0023399136554353253, "deviation-center-line": 1.069661678781388, "driven_lanedir_consec": 2.18128645935165, "sim_compute_sim_state": 0.012081050787577149, "sim_compute_performance-ego0": 0.0022203159502726566}, "LF-norm-small_loop-000-ego0": {"driven_any": 4.402127384431073, "get_ui_image": 0.02860611935139789, "step_physics": 0.11170569313414276, "survival_time": 21.900000000000176, "driven_lanedir": 1.5145693982249282, "get_state_dump": 0.004925642035273593, "get_robot_state": 0.003982772588186764, "sim_render-ego0": 0.004053260307920278, "get_duckie_state": 1.2979963645848164e-06, "in-drivable-lane": 14.050000000000196, "deviation-heading": 1.9405049382741988, "agent_compute-ego0": 0.012618198481670544, "complete-iteration": 0.1771989987488488, "set_robot_commands": 0.002393584588122531, "deviation-center-line": 0.4218714887669255, "driven_lanedir_consec": 1.5145693982249282, "sim_compute_sim_state": 0.006703173109502076, "sim_compute_performance-ego0": 0.002120653969280269}}
set_robot_commands_max0.002393584588122531
set_robot_commands_mean0.0023270166990465527
set_robot_commands_median0.0023482045717164992
set_robot_commands_min0.002218073064630682
sim_compute_performance-ego0_max0.0022203159502726566
sim_compute_performance-ego0_mean0.0021312867465485747
sim_compute_performance-ego0_median0.002128431201337249
sim_compute_performance-ego0_min0.002047968633247144
sim_compute_sim_state_max0.012081050787577149
sim_compute_sim_state_mean0.009922620356331189
sim_compute_sim_state_median0.010453128764122768
sim_compute_sim_state_min0.006703173109502076
sim_render-ego0_max0.004062695626811964
sim_render-ego0_mean0.003960933446166859
sim_render-ego0_median0.004031513425430793
sim_render-ego0_min0.003718011306993889
simulation-passed1
step_physics_max0.15178967267940682
step_physics_mean0.1322650462001009
step_physics_median0.132782409493427
step_physics_min0.11170569313414276
survival_time_max59.99999999999873
survival_time_mean40.212499999999515
survival_time_min21.900000000000176
No reset possible
53973LFv-simsuccessyes0:19:12
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
53971LFv-simsuccessyes0:22:10
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
53970LFv-simsuccessyes0:16:54
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
53968LFv-simsuccessyes0:19:33
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
53938LFv-simhost-erroryes0:02:02
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/docker_compose.py", line 53, in get_services_id
    container = client.containers.get(container_id)
  File "/usr/local/lib/python3.8/dist-packages/docker/models/containers.py", line 880, in get
    resp = self.client.api.inspect_container(container_id)
  File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 16, in wrapped
    raise errors.NullResource(
docker.errors.NullResource: Resource ID was not provided

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 777, in get_cr
    cr = run_single(
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 991, in run_single
    write_logs(wd, project, services=config["services"])
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/docker_compose.py", line 120, in write_logs
    services2id: Dict[ServiceName, ContainerID] = get_services_id(wd, project, services)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/docker_compose.py", line 63, in get_services_id
    raise DockerComposeFail(msg, output=output.decode(), names=names) from e
duckietown_challenges_runner.docker_compose.DockerComposeFail: Cannot get process ids
│ output: ''
│  names: {}
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
53913LFv-simsuccessyes0:19:43
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
48795LFv-simerrorno0:07:55
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140577696388528
- M:video_aido:cmdline(in:/;out:/) 140577695987168
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
48791LFv-simerrorno0:07:27
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140392235494656
- M:video_aido:cmdline(in:/;out:/) 140392235492064
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
48141LFv-simabortedno0:00:32
KeyboardInterrupt: T [...]
KeyboardInterrupt:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1169, in run_one
    heartbeat()
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 628, in heartbeat
    raise KeyboardInterrupt(msg_)
KeyboardInterrupt: The server told us to abort the job because: The challenge has been updated.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
43753LFv-simsuccessno0:06:50
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
43752LFv-simsuccessno0:11:28
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible