Duckietown Challenges Home Challenges Submissions

Submission 2396

Submission2396
Competingyes
Challengeaido2-LFV-sim-validation
UserLiam Paull 🇨🇦
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
Jobsstep1-simulation: 23007
Next
User labelrandom_agent
Admin priority50
Blessingn/a
User priority50

23007

Click the images to see detailed statistics about the episode.

ETHZ_autolab_technical_track-0-0

ETHZ_autolab_technical_track-1-0

ETHZ_autolab_technical_track-2-0

ETHZ_autolab_technical_track-3-0

ETHZ_autolab_technical_track-4-0

Evaluation jobs for this submission

Show only up-to-date jobs
Job IDstepstatusup to datedate starteddate completeddurationmessage
23007step1-simulationsuccessyes0:09:19
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.5723310224856362
survival_time_median2.6999999999999984
deviation-center-line_median0.11519544839797546
in-drivable-lane_median0.5999999999999979


other stats
agent_compute-ego_max0.15397053956985474
agent_compute-ego_mean0.1480917702151118
agent_compute-ego_median0.14782419083993645
agent_compute-ego_min0.13978649139404298
deviation-center-line_max0.1743913456502906
deviation-center-line_mean0.12651756552531995
deviation-center-line_min0.09047885846140293
deviation-heading_max0.5020407942599273
deviation-heading_mean0.3641979127291371
deviation-heading_median0.34764983431811214
deviation-heading_min0.19401296908406307
driven_any_max1.6169932854589035
driven_any_mean1.0272530755098266
driven_any_median1.0243289631429708
driven_any_min0.4119037580118998
driven_lanedir_consec_max1.2433038249708377
driven_lanedir_consec_mean0.7800316850006626
driven_lanedir_consec_min0.3792793194470208
driven_lanedir_max1.2433038249708377
driven_lanedir_mean0.7800316850006626
driven_lanedir_median0.5723310224856362
driven_lanedir_min0.3792793194470208
in-drivable-lane_max0.9999999999999976
in-drivable-lane_mean0.4899999999999984
in-drivable-lane_min0
per-episodes
details{"ETHZ_autolab_technical_track-0-0": {"driven_any": 1.0243289631429708, "sim_physics": 0.18380938635932076, "survival_time": 2.6999999999999984, "driven_lanedir": 0.5723310224856362, "sim_render-ego": 0.0684723280094288, "in-drivable-lane": 0.9999999999999976, "agent_compute-ego": 0.14674325784047446, "deviation-heading": 0.19401296908406307, "set_robot_commands": 0.101191485369647, "deviation-center-line": 0.15907928712407893, "driven_lanedir_consec": 0.5723310224856362, "sim_compute_sim_state": 0.0430073031672725, "sim_compute_performance-ego": 0.07618929721690991, "sim_compute_robot_state-ego": 0.08006982450132016, "sim_compute_robot_state-npc0": 0.07376652735250967, "sim_compute_robot_state-npc1": 0.07581934222468624, "sim_compute_robot_state-npc2": 0.07599559978202537, "sim_compute_robot_state-npc3": 0.07800661192999946}, "ETHZ_autolab_technical_track-1-0": {"driven_any": 0.4119037580118998, "sim_physics": 0.2189486503601074, "survival_time": 1.2500000000000004, "driven_lanedir": 0.3792793194470208, "sim_render-ego": 0.06621910095214843, "in-drivable-lane": 0, "agent_compute-ego": 0.13978649139404298, "deviation-heading": 0.47860435001913393, "set_robot_commands": 0.10495659828186037, "deviation-center-line": 0.09344288799285194, "driven_lanedir_consec": 0.3792793194470208, "sim_compute_sim_state": 0.044588470458984376, "sim_compute_performance-ego": 0.07639209747314453, "sim_compute_robot_state-ego": 0.07870147705078125, "sim_compute_robot_state-npc0": 0.07579483032226562, "sim_compute_robot_state-npc1": 0.07830813407897949, "sim_compute_robot_state-npc2": 0.07472768783569336, "sim_compute_robot_state-npc3": 0.07954666137695313}, "ETHZ_autolab_technical_track-2-0": {"driven_any": 1.579862082650716, "sim_physics": 0.18910220735951477, "survival_time": 3.7999999999999945, "driven_lanedir": 1.2433038249708377, "sim_render-ego": 0.06501994634929456, "in-drivable-lane": 0.5999999999999979, "agent_compute-ego": 0.15213437143125033, "deviation-heading": 0.5020407942599273, "set_robot_commands": 0.1013270428306178, "deviation-center-line": 0.1743913456502906, "driven_lanedir_consec": 1.2433038249708377, "sim_compute_sim_state": 0.04057472944259643, "sim_compute_performance-ego": 0.07470105510008962, "sim_compute_robot_state-ego": 0.07754342493258025, "sim_compute_robot_state-npc0": 0.07065721562034205, "sim_compute_robot_state-npc1": 0.07254885999779952, "sim_compute_robot_state-npc2": 0.07330545939897236, "sim_compute_robot_state-npc3": 0.07025598224840666}, "ETHZ_autolab_technical_track-3-0": {"driven_any": 1.6169932854589035, "sim_physics": 0.2117893575113031, "survival_time": 3.949999999999994, "driven_lanedir": 1.2126117310470583, "sim_render-ego": 0.0669332274907752, "in-drivable-lane": 0.849999999999997, "agent_compute-ego": 0.14782419083993645, "deviation-heading": 0.34764983431811214, "set_robot_commands": 0.10539818111854264, "deviation-center-line": 0.11519544839797546, "driven_lanedir_consec": 1.2126117310470583, "sim_compute_sim_state": 0.04455892043777659, "sim_compute_performance-ego": 0.07426412196099004, "sim_compute_robot_state-ego": 0.08008031905451908, "sim_compute_robot_state-npc0": 0.07798314396339127, "sim_compute_robot_state-npc1": 0.07417843009852156, "sim_compute_robot_state-npc2": 0.07611581645434416, "sim_compute_robot_state-npc3": 0.07669291918790794}, "ETHZ_autolab_technical_track-4-0": {"driven_any": 0.5031772882846431, "sim_physics": 0.18715801409312657, "survival_time": 1.4000000000000006, "driven_lanedir": 0.4926325270527603, "sim_render-ego": 0.06525789839880806, "in-drivable-lane": 0, "agent_compute-ego": 0.15397053956985474, "deviation-heading": 0.29868161596444953, "set_robot_commands": 0.0991625019482204, "deviation-center-line": 0.09047885846140293, "driven_lanedir_consec": 0.4926325270527603, "sim_compute_sim_state": 0.04875373840332031, "sim_compute_performance-ego": 0.07301745244434901, "sim_compute_robot_state-ego": 0.07318781954901558, "sim_compute_robot_state-npc0": 0.07049942016601562, "sim_compute_robot_state-npc1": 0.07682983364377703, "sim_compute_robot_state-npc2": 0.07074805668422154, "sim_compute_robot_state-npc3": 0.07320293358394078}}
set_robot_commands_max0.10539818111854264
set_robot_commands_mean0.10240716190977764
set_robot_commands_median0.1013270428306178
set_robot_commands_min0.0991625019482204
sim_compute_performance-ego_max0.07639209747314453
sim_compute_performance-ego_mean0.07491280483909661
sim_compute_performance-ego_median0.07470105510008962
sim_compute_performance-ego_min0.07301745244434901
sim_compute_robot_state-ego_max0.08008031905451908
sim_compute_robot_state-ego_mean0.07791657301764326
sim_compute_robot_state-ego_median0.07870147705078125
sim_compute_robot_state-ego_min0.07318781954901558
sim_compute_robot_state-npc0_max0.07798314396339127
sim_compute_robot_state-npc0_mean0.07374022748490486
sim_compute_robot_state-npc0_median0.07376652735250967
sim_compute_robot_state-npc0_min0.07049942016601562
sim_compute_robot_state-npc1_max0.07830813407897949
sim_compute_robot_state-npc1_mean0.07553692000875277
sim_compute_robot_state-npc1_median0.07581934222468624
sim_compute_robot_state-npc1_min0.07254885999779952
sim_compute_robot_state-npc2_max0.07611581645434416
sim_compute_robot_state-npc2_mean0.07417852403105137
sim_compute_robot_state-npc2_median0.07472768783569336
sim_compute_robot_state-npc2_min0.07074805668422154
sim_compute_robot_state-npc3_max0.07954666137695313
sim_compute_robot_state-npc3_mean0.07554102166544159
sim_compute_robot_state-npc3_median0.07669291918790794
sim_compute_robot_state-npc3_min0.07025598224840666
sim_compute_sim_state_max0.04875373840332031
sim_compute_sim_state_mean0.04429663238199004
sim_compute_sim_state_median0.04455892043777659
sim_compute_sim_state_min0.04057472944259643
sim_physics_max0.2189486503601074
sim_physics_mean0.1981615231366745
sim_physics_median0.18910220735951477
sim_physics_min0.18380938635932076
sim_render-ego_max0.0684723280094288
sim_render-ego_mean0.06638050024009101
sim_render-ego_median0.06621910095214843
sim_render-ego_min0.06501994634929456
simulation-passed1
survival_time_max3.949999999999994
survival_time_mean2.6199999999999974
survival_time_min1.2500000000000004
No reset possible
21327step1-simulationsuccessno0:09:18
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20599step1-simulationsuccessno0:06:41
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20020step1-simulationerrorno0:01:57
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "experiment_manager.py", line 135, in main
    physics_dt=config.physics_dt)
  File "experiment_manager.py", line 250, in run_episode
    sim_ci.write_topic_and_expect_zero('episode_start', EpisodeStart(episode_name))
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 102, in write_topic_and_expect_zero
    nickname=self.nickname)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 255, in read_reply
    others = read_until_over(fpout, timeout=timeout, nickname=nickname)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 283, in read_until_over
    raise RemoteNodeAborted(m)
zuper_nodes.structures.RemoteNodeAborted: External node "simulator" aborted:

error in simulator |Exception while handling a message on topic "episode_start".
                   |
                   || Traceback (most recent call last):
                   ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 274, in loop
                   ||     handle_message_node(parsed, receiver0, context0)
                   ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 428, in handle_message_node
                   ||     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
                   ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/utils.py", line 18, in call_if_fun_exists
                   ||     f(**kwargs)
                   ||   File "gym_bridge.py", line 209, in on_received_episode_start
                   ||     self.update_observations(context)
                   ||   File "gym_bridge.py", line 274, in update_observations
                   ||     jpg_data = rgb2jpg(obs)
                   ||   File "gym_bridge.py", line 352, in rgb2jpg
                   ||     bgr = cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR)
                   || cv2.error: OpenCV(4.1.0) /io/opencv/modules/imgproc/src/color.simd_helpers.hpp:94: error: (-2:Unspecified error) in function 'cv::impl::{anonymous}::CvtHelper<VScn, VDcn, VDepth, sizePolicy>::CvtHelper(cv::InputArray, cv::OutputArray, int) [with VScn = cv::impl::{anonymous}::Set<1>; VDcn = cv::impl::{anonymous}::Set<3, 4>; VDepth = cv::impl::{anonymous}::Set<0, 2, 5>; cv::impl::{anonymous}::SizePolicy sizePolicy = (cv::impl::<unnamed>::SizePolicy)2u; cv::InputArray = const cv::_InputArray&; cv::OutputArray = const cv::_OutputArray&]'
                   || > Unsupported depth of input image:
                   || >     'VDepth::contains(depth)'
                   || > where
                   || >     'depth' is 6 (CV_64F)
                   ||
                   ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 610, in scoring_context
    yield cie
  File "experiment_manager.py", line 439, in <module>
    wrap(cie)
  File "experiment_manager.py", line 427, in wrap
    main(cie, logdir, attempts)
  File "experiment_manager.py", line 186, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

 > Traceback (most recent call last):
 >   File "experiment_manager.py", line 135, in main
 >     physics_dt=config.physics_dt)
 >   File "experiment_manager.py", line 250, in run_episode
 >     sim_ci.write_topic_and_expect_zero('episode_start', EpisodeStart(episode_name))
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 102, in write_topic_and_expect_zero
 >     nickname=self.nickname)
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 255, in read_reply
 >     others = read_until_over(fpout, timeout=timeout, nickname=nickname)
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 283, in read_until_over
 >     raise RemoteNodeAborted(m)
 > zuper_nodes.structures.RemoteNodeAborted: External node "simulator" aborted:
 > 
 > error in simulator |Exception while handling a message on topic "episode_start".
 >                    |
 >                    || Traceback (most recent call last):
 >                    ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 274, in loop
 >                    ||     handle_message_node(parsed, receiver0, context0)
 >                    ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 428, in handle_message_node
 >                    ||     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
 >                    ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/utils.py", line 18, in call_if_fun_exists
 >                    ||     f(**kwargs)
 >                    ||   File "gym_bridge.py", line 209, in on_received_episode_start
 >                    ||     self.update_observations(context)
 >                    ||   File "gym_bridge.py", line 274, in update_observations
 >                    ||     jpg_data = rgb2jpg(obs)
 >                    ||   File "gym_bridge.py", line 352, in rgb2jpg
 >                    ||     bgr = cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR)
 >                    || cv2.error: OpenCV(4.1.0) /io/opencv/modules/imgproc/src/color.simd_helpers.hpp:94: error: (-2:Unspecified error) in function 'cv::impl::{anonymous}::CvtHelper<VScn, VDcn, VDepth, sizePolicy>::CvtHelper(cv::InputArray, cv::OutputArray, int) [with VScn = cv::impl::{anonymous}::Set<1>; VDcn = cv::impl::{anonymous}::Set<3, 4>; VDepth = cv::impl::{anonymous}::Set<0, 2, 5>; cv::impl::{anonymous}::SizePolicy sizePolicy = (cv::impl::<unnamed>::SizePolicy)2u; cv::InputArray = const cv::_InputArray&; cv::OutputArray = const cv::_OutputArray&]'
 >                    || > Unsupported depth of input image:
 >                    || >     'VDepth::contains(depth)'
 >                    || > where
 >                    || >     'depth' is 6 (CV_64F)
 >                    ||
 >                    ||
 > 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19648step1-simulationsuccessno0:07:03
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18898step1-simulationsuccessno0:09:27
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18527step1-simulationsuccessno0:11:30
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
17908step1-simulationsuccessno0:07:41
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
17863step1-simulationsuccessno0:07:40
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
17588step1-simulationerrorno0:07:14
The result file is n [...]
The result file is not found. This usually means that the evaluator did not finish
and some times that there was an import error.

Check the evaluator log to see what happened.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
17578step1-simulationhost-errorno0:09:37
Uncaught exception w [...]
Uncaught exception while running Docker Compose:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 671, in run_one
    container = client.containers.get(container_id)
  File "/usr/local/lib/python3.6/dist-packages/docker/models/containers.py", line 862, in get
    resp = self.client.api.inspect_container(container_id)
  File "/usr/local/lib/python3.6/dist-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/docker/api/container.py", line 755, in inspect_container
    self._get(self._url("/containers/{0}/json", container)), True
  File "/usr/local/lib/python3.6/dist-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/docker/api/client.py", line 230, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
17573step1-simulationsuccessno0:07:20
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible