Duckietown Challenges Home Challenges Submissions

Submission 2861

Submission2861
Competingyes
Challengeaido2-LFV-sim-validation
UserAlexander Karavaev
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
Jobsstep1-simulation: 22734
Next
User labelchallenge-aido_LF-baseline-duckietown
Admin priority50
Blessingn/a
User priority50

22734

Click the images to see detailed statistics about the episode.

ETHZ_autolab_technical_track-0-0

ETHZ_autolab_technical_track-1-0

ETHZ_autolab_technical_track-2-0

ETHZ_autolab_technical_track-3-0

ETHZ_autolab_technical_track-4-0

Evaluation jobs for this submission

Show only up-to-date jobs
Job IDstepstatusup to datedate starteddate completeddurationmessage
22734step1-simulationsuccessyes0:14:29
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.47836629091548977
survival_time_median6.999999999999983
deviation-center-line_median0.20433366980906725
in-drivable-lane_median0.5999999999999979


other stats
agent_compute-ego_max0.09199162661019018
agent_compute-ego_mean0.08828414076225752
agent_compute-ego_median0.08755681276321411
agent_compute-ego_min0.08486901124318441
deviation-center-line_max0.28844415094366993
deviation-center-line_mean0.2239502979920882
deviation-center-line_min0.1624146827548989
deviation-heading_max1.9220264916359744
deviation-heading_mean1.7590353372017369
deviation-heading_median1.8527944533600644
deviation-heading_min1.4242247726612312
driven_any_max1.1707162632485568
driven_any_mean0.5686705564943999
driven_any_median0.5356965595175749
driven_any_min0.21022919986359032
driven_lanedir_consec_max0.495496351283172
driven_lanedir_consec_mean0.3752627679813157
driven_lanedir_consec_min0.17424171691909784
driven_lanedir_max0.495496351283172
driven_lanedir_mean0.3752627679813157
driven_lanedir_median0.47836629091548977
driven_lanedir_min0.17424171691909784
in-drivable-lane_max7.800000000000094
in-drivable-lane_mean1.8400000000000176
in-drivable-lane_min0
per-episodes
details{"ETHZ_autolab_technical_track-0-0": {"driven_any": 1.1707162632485568, "sim_physics": 0.10158046086629231, "survival_time": 14.950000000000076, "driven_lanedir": 0.495496351283172, "sim_render-ego": 0.06051742076873779, "in-drivable-lane": 7.800000000000094, "agent_compute-ego": 0.08755681276321411, "deviation-heading": 1.8527944533600644, "set_robot_commands": 0.07726306041081746, "deviation-center-line": 0.28844415094366993, "driven_lanedir_consec": 0.495496351283172, "sim_compute_sim_state": 0.0360496441523234, "sim_compute_performance-ego": 0.06226936022440593, "sim_compute_robot_state-ego": 0.06228128353754679, "sim_compute_robot_state-npc0": 0.06931014458338419, "sim_compute_robot_state-npc1": 0.0647650154431661, "sim_compute_robot_state-npc2": 0.06398614088694254, "sim_compute_robot_state-npc3": 0.06397000789642333}, "ETHZ_autolab_technical_track-1-0": {"driven_any": 0.3508483993889199, "sim_physics": 0.10416061827476988, "survival_time": 4.699999999999991, "driven_lanedir": 0.23682562715238875, "sim_render-ego": 0.05845534040572795, "in-drivable-lane": 0.7999999999999972, "agent_compute-ego": 0.08564724820725461, "deviation-heading": 1.711197107599585, "set_robot_commands": 0.0786243753230318, "deviation-center-line": 0.20093089734150887, "driven_lanedir_consec": 0.23682562715238875, "sim_compute_sim_state": 0.03629558137122621, "sim_compute_performance-ego": 0.06339593897474573, "sim_compute_robot_state-ego": 0.06326496601104736, "sim_compute_robot_state-npc0": 0.06761231574606388, "sim_compute_robot_state-npc1": 0.06090057657120076, "sim_compute_robot_state-npc2": 0.0617787685800106, "sim_compute_robot_state-npc3": 0.06363348250693464}, "ETHZ_autolab_technical_track-2-0": {"driven_any": 0.5758623604533578, "sim_physics": 0.10391751607259114, "survival_time": 7.499999999999981, "driven_lanedir": 0.47836629091548977, "sim_render-ego": 0.05686238447825114, "in-drivable-lane": 0.5999999999999979, "agent_compute-ego": 0.08486901124318441, "deviation-heading": 1.8849338607518291, "set_robot_commands": 0.07573312282562256, "deviation-center-line": 0.20433366980906725, "driven_lanedir_consec": 0.47836629091548977, "sim_compute_sim_state": 0.034448825518290205, "sim_compute_performance-ego": 0.060205397605895994, "sim_compute_robot_state-ego": 0.06039220650990804, "sim_compute_robot_state-npc0": 0.06685725053151449, "sim_compute_robot_state-npc1": 0.06173037687937419, "sim_compute_robot_state-npc2": 0.06210443973541259, "sim_compute_robot_state-npc3": 0.062376969655354816}, "ETHZ_autolab_technical_track-3-0": {"driven_any": 0.5356965595175749, "sim_physics": 0.10589817762374878, "survival_time": 6.999999999999983, "driven_lanedir": 0.4913838536364299, "sim_render-ego": 0.06528695140566144, "in-drivable-lane": 0, "agent_compute-ego": 0.0913560049874442, "deviation-heading": 1.9220264916359744, "set_robot_commands": 0.07713988678795951, "deviation-center-line": 0.26362808911129604, "driven_lanedir_consec": 0.4913838536364299, "sim_compute_sim_state": 0.037812404973166326, "sim_compute_performance-ego": 0.06638285602842059, "sim_compute_robot_state-ego": 0.06438945872443062, "sim_compute_robot_state-npc0": 0.07351224422454834, "sim_compute_robot_state-npc1": 0.06887644018445696, "sim_compute_robot_state-npc2": 0.06686156136648995, "sim_compute_robot_state-npc3": 0.06684942075184414}, "ETHZ_autolab_technical_track-4-0": {"driven_any": 0.21022919986359032, "sim_physics": 0.1032134799633996, "survival_time": 2.9499999999999975, "driven_lanedir": 0.17424171691909784, "sim_render-ego": 0.06337611958131951, "in-drivable-lane": 0, "agent_compute-ego": 0.09199162661019018, "deviation-heading": 1.4242247726612312, "set_robot_commands": 0.07640050225338693, "deviation-center-line": 0.1624146827548989, "driven_lanedir_consec": 0.17424171691909784, "sim_compute_sim_state": 0.03771425909915213, "sim_compute_performance-ego": 0.0672359870652021, "sim_compute_robot_state-ego": 0.06650322170580848, "sim_compute_robot_state-npc0": 0.07463279821104922, "sim_compute_robot_state-npc1": 0.0698024580034159, "sim_compute_robot_state-npc2": 0.06923918400780629, "sim_compute_robot_state-npc3": 0.06759670225240416}}
set_robot_commands_max0.0786243753230318
set_robot_commands_mean0.07703218952016366
set_robot_commands_median0.07713988678795951
set_robot_commands_min0.07573312282562256
sim_compute_performance-ego_max0.0672359870652021
sim_compute_performance-ego_mean0.06389790797973408
sim_compute_performance-ego_median0.06339593897474573
sim_compute_performance-ego_min0.060205397605895994
sim_compute_robot_state-ego_max0.06650322170580848
sim_compute_robot_state-ego_mean0.06336622729774825
sim_compute_robot_state-ego_median0.06326496601104736
sim_compute_robot_state-ego_min0.06039220650990804
sim_compute_robot_state-npc0_max0.07463279821104922
sim_compute_robot_state-npc0_mean0.07038495065931202
sim_compute_robot_state-npc0_median0.06931014458338419
sim_compute_robot_state-npc0_min0.06685725053151449
sim_compute_robot_state-npc1_max0.0698024580034159
sim_compute_robot_state-npc1_mean0.06521497341632279
sim_compute_robot_state-npc1_median0.0647650154431661
sim_compute_robot_state-npc1_min0.06090057657120076
sim_compute_robot_state-npc2_max0.06923918400780629
sim_compute_robot_state-npc2_mean0.0647940189153324
sim_compute_robot_state-npc2_median0.06398614088694254
sim_compute_robot_state-npc2_min0.0617787685800106
sim_compute_robot_state-npc3_max0.06759670225240416
sim_compute_robot_state-npc3_mean0.06488531661259221
sim_compute_robot_state-npc3_median0.06397000789642333
sim_compute_robot_state-npc3_min0.062376969655354816
sim_compute_sim_state_max0.037812404973166326
sim_compute_sim_state_mean0.03646414302283165
sim_compute_sim_state_median0.03629558137122621
sim_compute_sim_state_min0.034448825518290205
sim_physics_max0.10589817762374878
sim_physics_mean0.10375405056016034
sim_physics_median0.10391751607259114
sim_physics_min0.10158046086629231
sim_render-ego_max0.06528695140566144
sim_render-ego_mean0.06089964332793957
sim_render-ego_median0.06051742076873779
sim_render-ego_min0.05686238447825114
simulation-passed1
survival_time_max14.950000000000076
survival_time_mean7.420000000000006
survival_time_min2.9499999999999975
No reset possible
20959step1-simulationsuccessno0:18:01
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20271step1-simulationerrorno0:01:42
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "experiment_manager.py", line 87, in main
    sim_ci.write_topic_and_expect_zero('seed', config.seed)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 102, in write_topic_and_expect_zero
    nickname=self.nickname)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 247, in read_reply
    wm: WireMessage = read_next_cbor(fpout, timeout=timeout, waiting_for=waiting_for)
  File "/project/src/zuper-utils/src/zuper_json/json2cbor.py", line 161, in read_next_cbor
    wait_for_data(f, timeout, waiting_for)
  File "/project/src/zuper-utils/src/zuper_json/json2cbor.py", line 146, in wait_for_data
    raise TimeoutError(msg)
TimeoutError: Timeout after 30.0 s.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 610, in scoring_context
    yield cie
  File "experiment_manager.py", line 439, in <module>
    wrap(cie)
  File "experiment_manager.py", line 427, in wrap
    main(cie, logdir, attempts)
  File "experiment_manager.py", line 186, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

 > Traceback (most recent call last):
 >   File "experiment_manager.py", line 87, in main
 >     sim_ci.write_topic_and_expect_zero('seed', config.seed)
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 102, in write_topic_and_expect_zero
 >     nickname=self.nickname)
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 247, in read_reply
 >     wm: WireMessage = read_next_cbor(fpout, timeout=timeout, waiting_for=waiting_for)
 >   File "/project/src/zuper-utils/src/zuper_json/json2cbor.py", line 161, in read_next_cbor
 >     wait_for_data(f, timeout, waiting_for)
 >   File "/project/src/zuper-utils/src/zuper_json/json2cbor.py", line 146, in wait_for_data
 >     raise TimeoutError(msg)
 > TimeoutError: Timeout after 30.0 s.
 >
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19877step1-simulationerrorno0:00:58
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "experiment_manager.py", line 135, in main
    physics_dt=config.physics_dt)
  File "experiment_manager.py", line 250, in run_episode
    sim_ci.write_topic_and_expect_zero('episode_start', EpisodeStart(episode_name))
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 102, in write_topic_and_expect_zero
    nickname=self.nickname)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 255, in read_reply
    others = read_until_over(fpout, timeout=timeout, nickname=nickname)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 283, in read_until_over
    raise RemoteNodeAborted(m)
zuper_nodes.structures.RemoteNodeAborted: External node "simulator" aborted:

error in simulator |Exception while handling a message on topic "episode_start".
                   |
                   || Traceback (most recent call last):
                   ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 274, in loop
                   ||     handle_message_node(parsed, receiver0, context0)
                   ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 428, in handle_message_node
                   ||     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
                   ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/utils.py", line 18, in call_if_fun_exists
                   ||     f(**kwargs)
                   ||   File "gym_bridge.py", line 209, in on_received_episode_start
                   ||     self.update_observations(context)
                   ||   File "gym_bridge.py", line 274, in update_observations
                   ||     jpg_data = rgb2jpg(obs)
                   ||   File "gym_bridge.py", line 352, in rgb2jpg
                   ||     bgr = cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR)
                   || cv2.error: OpenCV(4.1.0) /io/opencv/modules/imgproc/src/color.simd_helpers.hpp:94: error: (-2:Unspecified error) in function 'cv::impl::{anonymous}::CvtHelper<VScn, VDcn, VDepth, sizePolicy>::CvtHelper(cv::InputArray, cv::OutputArray, int) [with VScn = cv::impl::{anonymous}::Set<1>; VDcn = cv::impl::{anonymous}::Set<3, 4>; VDepth = cv::impl::{anonymous}::Set<0, 2, 5>; cv::impl::{anonymous}::SizePolicy sizePolicy = (cv::impl::<unnamed>::SizePolicy)2u; cv::InputArray = const cv::_InputArray&; cv::OutputArray = const cv::_OutputArray&]'
                   || > Unsupported depth of input image:
                   || >     'VDepth::contains(depth)'
                   || > where
                   || >     'depth' is 6 (CV_64F)
                   ||
                   ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 610, in scoring_context
    yield cie
  File "experiment_manager.py", line 439, in <module>
    wrap(cie)
  File "experiment_manager.py", line 427, in wrap
    main(cie, logdir, attempts)
  File "experiment_manager.py", line 186, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

 > Traceback (most recent call last):
 >   File "experiment_manager.py", line 135, in main
 >     physics_dt=config.physics_dt)
 >   File "experiment_manager.py", line 250, in run_episode
 >     sim_ci.write_topic_and_expect_zero('episode_start', EpisodeStart(episode_name))
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 102, in write_topic_and_expect_zero
 >     nickname=self.nickname)
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 255, in read_reply
 >     others = read_until_over(fpout, timeout=timeout, nickname=nickname)
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 283, in read_until_over
 >     raise RemoteNodeAborted(m)
 > zuper_nodes.structures.RemoteNodeAborted: External node "simulator" aborted:
 > 
 > error in simulator |Exception while handling a message on topic "episode_start".
 >                    |
 >                    || Traceback (most recent call last):
 >                    ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 274, in loop
 >                    ||     handle_message_node(parsed, receiver0, context0)
 >                    ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 428, in handle_message_node
 >                    ||     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
 >                    ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/utils.py", line 18, in call_if_fun_exists
 >                    ||     f(**kwargs)
 >                    ||   File "gym_bridge.py", line 209, in on_received_episode_start
 >                    ||     self.update_observations(context)
 >                    ||   File "gym_bridge.py", line 274, in update_observations
 >                    ||     jpg_data = rgb2jpg(obs)
 >                    ||   File "gym_bridge.py", line 352, in rgb2jpg
 >                    ||     bgr = cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR)
 >                    || cv2.error: OpenCV(4.1.0) /io/opencv/modules/imgproc/src/color.simd_helpers.hpp:94: error: (-2:Unspecified error) in function 'cv::impl::{anonymous}::CvtHelper<VScn, VDcn, VDepth, sizePolicy>::CvtHelper(cv::InputArray, cv::OutputArray, int) [with VScn = cv::impl::{anonymous}::Set<1>; VDcn = cv::impl::{anonymous}::Set<3, 4>; VDepth = cv::impl::{anonymous}::Set<0, 2, 5>; cv::impl::{anonymous}::SizePolicy sizePolicy = (cv::impl::<unnamed>::SizePolicy)2u; cv::InputArray = const cv::_InputArray&; cv::OutputArray = const cv::_OutputArray&]'
 >                    || > Unsupported depth of input image:
 >                    || >     'VDepth::contains(depth)'
 >                    || > where
 >                    || >     'depth' is 6 (CV_64F)
 >                    ||
 >                    ||
 >
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19299step1-simulationsuccessno0:14:10
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18591step1-simulationerrorno0:21:36
Waited 1201 seconds [...]
Waited 1201 seconds for container to finish. Giving up.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18588step1-simulationhost-errorno0:00:06
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 1041, in download_artefacts
    get_file_from_cache(fn, sha256hex)
  File "/project/src/duckietown_challenges_runner/runner_cache.py", line 27, in get_file_from_cache
    raise KeyError(msg)
KeyError: 'Hash not in cache'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 478, in get_cr
    submission_id=submission_id, timeout_sec=timeout_sec)
  File "/project/src/duckietown_challenges_runner/runner.py", line 518, in run_single
    prepare_dir(wd, aws_config, steps2artefacts)
  File "/project/src/duckietown_challenges_runner/runner.py", line 785, in prepare_dir
    download_artefacts(aws_config, steps2artefacts, previous_steps_dir)
  File "/project/src/duckietown_challenges_runner/runner.py", line 1062, in download_artefacts
    raise ValueError(msg)
ValueError: Corrupt cache or download for {'rpath': 'tmp/tmpkcoyldy0/attempts/ETHZ_autolab_technical_track-0-0.attempt0/log.gs2.cbor.tmp', 'mime_type': 'binary/octet-stream', 'sha256hex': '352e46035a8805bdf34665f8b50beeb9db3a1a2867eff8732ff286606285cf7d', 'size': 29822541, 'storage': {'s3': {'bucket_name': 'duckietown-ai-driving-olympics-1', 'object_key': 'v3/frankfurt/by-value/sha256/352e46035a8805bdf34665f8b50beeb9db3a1a2867eff8732ff286606285cf7d', 'url': 'http://duckietown-ai-driving-olympics-1.s3.amazonaws.com/v3/frankfurt/by-value/sha256/352e46035a8805bdf34665f8b50beeb9db3a1a2867eff8732ff286606285cf7d'}}} at /tmp/duckietown/DT18/evaluator/executions/aido2-LFV-sim-validation/submission2861/step1-simulation-ip-172-31-25-98-9205-job18588/previous-steps/step1-simulation/tmp/tmpkcoyldy0/attempts/ETHZ_autolab_technical_track-0-0.attempt0/log.gs2.cbor.tmp.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18582step1-simulationhost-errorno0:02:36
Uncaught exception w [...]
Uncaught exception while running Docker Compose:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 691, in run_one
    container = client.containers.get(container_id)
  File "/usr/local/lib/python3.6/dist-packages/docker/models/containers.py", line 862, in get
    resp = self.client.api.inspect_container(container_id)
  File "/usr/local/lib/python3.6/dist-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/docker/api/container.py", line 755, in inspect_container
    self._get(self._url("/containers/{0}/json", container)), True
  File "/usr/local/lib/python3.6/dist-packages/docker/api/client.py", line 267, in _result
    self._raise_for_status(response)
  File "/usr/local/lib/python3.6/dist-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.6/dist-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/containers/a9b78c1313bb4482e274c5ab8ec0779473d8b8f3df2fe977fd4f8c5c3fcab1eb/json
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18568step1-simulationerrorno0:24:45
Waited 1203 seconds [...]
Waited 1203 seconds for container to finish. Giving up.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18564step1-simulationerrorno0:21:28
Waited 1201 seconds [...]
Waited 1201 seconds for container to finish. Giving up.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible