Duckietown Challenges Home Challenges Submissions

Submission 2953

Submission2953
Competingyes
Challengeaido2-LFVI-sim-validation
UserBhairav Mehta
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
Jobsstep1-simulation: 22695
Next
User labelBaseline solution using reinforcement learning
Admin priority50
Blessingn/a
User priority50

22695

Click the images to see detailed statistics about the episode.

udem1-0-0

udem1-1-0

udem1-2-0

udem1-3-0

udem1-4-0

Evaluation jobs for this submission

Show only up-to-date jobs
Job IDstepstatusup to datedate starteddate completeddurationmessage
22695step1-simulationsuccessyes0:15:29
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.07806043292455866
survival_time_median1.2500000000000004
deviation-center-line_median0.0974612829139978
in-drivable-lane_median0.05000000000000005


other stats
agent_compute-ego_max0.11540439854497496
agent_compute-ego_mean0.1139448890993967
agent_compute-ego_median0.11375650405883787
agent_compute-ego_min0.11286111701937283
deviation-center-line_max0.45046446332249823
deviation-center-line_mean0.2288050953144846
deviation-center-line_min0.057760376532252355
deviation-heading_max5.452227182841201
deviation-heading_mean2.559962304890861
deviation-heading_median0.8379291556002636
deviation-heading_min0.6234928932391447
driven_any_max1.7944200888459858
driven_any_mean0.7789390415104016
driven_any_median0.13916374471935433
driven_any_min0.0969027879260704
driven_lanedir_consec_max0.6132689425266915
driven_lanedir_consec_mean0.2661019796961642
driven_lanedir_consec_min0.052385922314882816
driven_lanedir_max0.6132689425266915
driven_lanedir_mean0.2661019796961642
driven_lanedir_median0.07806043292455866
driven_lanedir_min0.052385922314882816
in-drivable-lane_max6.700000000000028
in-drivable-lane_mean2.690000000000011
in-drivable-lane_min0
per-episodes
details{"udem1-0-0": {"driven_any": 1.7944200888459858, "sim_physics": 0.0964497345081274, "survival_time": 13.80000000000006, "driven_lanedir": 0.6132689425266915, "sim_render-ego": 0.05226804467215054, "in-drivable-lane": 6.700000000000028, "agent_compute-ego": 0.11316810742668484, "deviation-heading": 5.235858749479385, "set_robot_commands": 0.05031072661496591, "deviation-center-line": 0.4411302928645919, "driven_lanedir_consec": 0.6132689425266915, "sim_compute_sim_state": 0.03175011665924736, "sim_compute_performance-ego": 0.05597750691400058, "sim_compute_robot_state-ego": 0.05502185959746872, "sim_compute_robot_state-npc0": 0.06090877453486125, "sim_compute_robot_state-npc1": 0.054908218591109566, "sim_compute_robot_state-npc2": 0.054350204225899514, "sim_compute_robot_state-npc3": 0.05423062780629034}, "udem1-1-0": {"driven_any": 1.7511809371591127, "sim_physics": 0.0983053875320098, "survival_time": 13.600000000000058, "driven_lanedir": 0.5145281474410162, "sim_render-ego": 0.053224956288057214, "in-drivable-lane": 6.650000000000025, "agent_compute-ego": 0.11286111701937283, "deviation-heading": 5.452227182841201, "set_robot_commands": 0.05024545420618618, "deviation-center-line": 0.45046446332249823, "driven_lanedir_consec": 0.5145281474410162, "sim_compute_sim_state": 0.03235205306726344, "sim_compute_performance-ego": 0.05699467133073246, "sim_compute_robot_state-ego": 0.05528670111123253, "sim_compute_robot_state-npc0": 0.06345752670484431, "sim_compute_robot_state-npc1": 0.05606062534977408, "sim_compute_robot_state-npc2": 0.055441098178134245, "sim_compute_robot_state-npc3": 0.05581708778353298}, "udem1-2-0": {"driven_any": 0.0969027879260704, "sim_physics": 0.11724071917326552, "survival_time": 1.1500000000000004, "driven_lanedir": 0.052385922314882816, "sim_render-ego": 0.05333792644998302, "in-drivable-lane": 0, "agent_compute-ego": 0.11540439854497496, "deviation-heading": 0.6503035432943092, "set_robot_commands": 0.04919950858406399, "deviation-center-line": 0.0974612829139978, "driven_lanedir_consec": 0.052385922314882816, "sim_compute_sim_state": 0.032383970592332924, "sim_compute_performance-ego": 0.05653250735739003, "sim_compute_robot_state-ego": 0.05597405848295792, "sim_compute_robot_state-npc0": 0.06428108008011528, "sim_compute_robot_state-npc1": 0.05564420119575832, "sim_compute_robot_state-npc2": 0.05447220802307129, "sim_compute_robot_state-npc3": 0.05546675557675569}, "udem1-3-0": {"driven_any": 0.11302764890148503, "sim_physics": 0.10800381660461426, "survival_time": 1.2500000000000004, "driven_lanedir": 0.07226645327367187, "sim_render-ego": 0.05265570640563965, "in-drivable-lane": 0.05000000000000005, "agent_compute-ego": 0.11375650405883787, "deviation-heading": 0.6234928932391447, "set_robot_commands": 0.05121260643005371, "deviation-center-line": 0.09720906093908274, "driven_lanedir_consec": 0.07226645327367187, "sim_compute_sim_state": 0.0328909969329834, "sim_compute_performance-ego": 0.05930389404296875, "sim_compute_robot_state-ego": 0.05660543441772461, "sim_compute_robot_state-npc0": 0.06221333503723145, "sim_compute_robot_state-npc1": 0.05376955986022949, "sim_compute_robot_state-npc2": 0.052627363204956056, "sim_compute_robot_state-npc3": 0.05165313720703125}, "udem1-4-0": {"driven_any": 0.13916374471935433, "sim_physics": 0.09793612360954285, "survival_time": 1.2000000000000004, "driven_lanedir": 0.07806043292455866, "sim_render-ego": 0.05424729983011881, "in-drivable-lane": 0.05000000000000005, "agent_compute-ego": 0.11453431844711304, "deviation-heading": 0.8379291556002636, "set_robot_commands": 0.0495841900507609, "deviation-center-line": 0.057760376532252355, "driven_lanedir_consec": 0.07806043292455866, "sim_compute_sim_state": 0.03286592165629069, "sim_compute_performance-ego": 0.05883944034576416, "sim_compute_robot_state-ego": 0.05425083637237549, "sim_compute_robot_state-npc0": 0.0656888484954834, "sim_compute_robot_state-npc1": 0.057235111792882286, "sim_compute_robot_state-npc2": 0.05812508861223856, "sim_compute_robot_state-npc3": 0.056497931480407715}}
set_robot_commands_max0.05121260643005371
set_robot_commands_mean0.050110497177206136
set_robot_commands_median0.05024545420618618
set_robot_commands_min0.04919950858406399
sim_compute_performance-ego_max0.05930389404296875
sim_compute_performance-ego_mean0.0575296039981712
sim_compute_performance-ego_median0.05699467133073246
sim_compute_performance-ego_min0.05597750691400058
sim_compute_robot_state-ego_max0.05660543441772461
sim_compute_robot_state-ego_mean0.05542777799635186
sim_compute_robot_state-ego_median0.05528670111123253
sim_compute_robot_state-ego_min0.05425083637237549
sim_compute_robot_state-npc0_max0.0656888484954834
sim_compute_robot_state-npc0_mean0.06330991297050714
sim_compute_robot_state-npc0_median0.06345752670484431
sim_compute_robot_state-npc0_min0.06090877453486125
sim_compute_robot_state-npc1_max0.057235111792882286
sim_compute_robot_state-npc1_mean0.05552354335795075
sim_compute_robot_state-npc1_median0.05564420119575832
sim_compute_robot_state-npc1_min0.05376955986022949
sim_compute_robot_state-npc2_max0.05812508861223856
sim_compute_robot_state-npc2_mean0.05500319244885994
sim_compute_robot_state-npc2_median0.05447220802307129
sim_compute_robot_state-npc2_min0.052627363204956056
sim_compute_robot_state-npc3_max0.056497931480407715
sim_compute_robot_state-npc3_mean0.05473310797080359
sim_compute_robot_state-npc3_median0.05546675557675569
sim_compute_robot_state-npc3_min0.05165313720703125
sim_compute_sim_state_max0.0328909969329834
sim_compute_sim_state_mean0.03244861178162356
sim_compute_sim_state_median0.032383970592332924
sim_compute_sim_state_min0.03175011665924736
sim_physics_max0.11724071917326552
sim_physics_mean0.10358715628551196
sim_physics_median0.0983053875320098
sim_physics_min0.0964497345081274
sim_render-ego_max0.05424729983011881
sim_render-ego_mean0.053146786729189856
sim_render-ego_median0.053224956288057214
sim_render-ego_min0.05226804467215054
simulation-passed1
survival_time_max13.80000000000006
survival_time_mean6.200000000000023
survival_time_min1.1500000000000004
No reset possible
20931step1-simulationsuccessno0:22:44
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20909step1-simulationhost-errorno0:01:18
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 483, in get_cr
    submission_id=submission_id, timeout_sec=timeout_sec)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/project/src/duckietown_challenges_runner/runner.py", line 343, in setup_logging
    convert(f_stdout)
  File "/project/src/duckietown_challenges_runner/runner.py", line 332, in convert
    data = open(log).read().strip()
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/duckietown/DT18/evaluator/executions/aido2-LFVI-sim-validation/submission2953/step1-simulation-ip-172-31-38-104-5197-job20909/logs/challenges-runner/stdout.log'
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20234step1-simulationsuccessno0:07:28
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19239step1-simulationerrorno0:01:33
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "experiment_manager.py", line 88, in main
    sim_ci.write_topic_and_expect_zero('seed', config.seed)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 102, in write_topic_and_expect_zero
    nickname=self.nickname)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 247, in read_reply
    wm: WireMessage = read_next_cbor(fpout, timeout=timeout, waiting_for=waiting_for)
  File "/project/src/zuper-utils/src/zuper_json/json2cbor.py", line 161, in read_next_cbor
    wait_for_data(f, timeout, waiting_for)
  File "/project/src/zuper-utils/src/zuper_json/json2cbor.py", line 146, in wait_for_data
    raise TimeoutError(msg)
TimeoutError: Timeout after 30.0 s.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 610, in scoring_context
    yield cie
  File "experiment_manager.py", line 435, in <module>
    wrap(cie)
  File "experiment_manager.py", line 423, in wrap
    main(cie, logdir, attempts)
  File "experiment_manager.py", line 186, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

 > Traceback (most recent call last):
 >   File "experiment_manager.py", line 88, in main
 >     sim_ci.write_topic_and_expect_zero('seed', config.seed)
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 102, in write_topic_and_expect_zero
 >     nickname=self.nickname)
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 247, in read_reply
 >     wm: WireMessage = read_next_cbor(fpout, timeout=timeout, waiting_for=waiting_for)
 >   File "/project/src/zuper-utils/src/zuper_json/json2cbor.py", line 161, in read_next_cbor
 >     wait_for_data(f, timeout, waiting_for)
 >   File "/project/src/zuper-utils/src/zuper_json/json2cbor.py", line 146, in wait_for_data
 >     raise TimeoutError(msg)
 > TimeoutError: Timeout after 30.0 s.
 >
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19233step1-simulationtimeoutno1:05:02
I can see how the jo [...]
I can see how the job 19233 is timeout because passed 3902 seconds and the timeout is 3600.0.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19222step1-simulationhost-errorno0:02:16
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/usr/local/lib/python3.6/dist-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/usr/local/lib/python3.6/dist-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/project/src/duckietown-shell/lib/dt_shell/env_checks.py", line 60, in check_docker_environment
    containers = client.containers.list(filters=dict(status='running'))
  File "/usr/local/lib/python3.6/dist-packages/docker/models/containers.py", line 918, in list
    since=since)
  File "/usr/local/lib/python3.6/dist-packages/docker/api/container.py", line 208, in containers
    res = self._result(self._get(u, params=params), True)
  File "/usr/local/lib/python3.6/dist-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/docker/api/client.py", line 230, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 478, in get_cr
    submission_id=submission_id, timeout_sec=timeout_sec)
  File "/project/src/duckietown_challenges_runner/runner.py", line 573, in run_single
    write_logs(wd, project, services=config['services'])
  File "/project/src/duckietown_challenges_runner/runner.py", line 892, in write_logs
    client = check_docker_environment()
  File "/project/src/duckietown-shell/lib/dt_shell/env_checks.py", line 67, in check_docker_environment
    raise InvalidEnvironment(msg)
dt_shell.exceptions.InvalidEnvironment: I cannot communicate with Docker:
('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

Make sure the docker service is running.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19116step1-simulationsuccessno0:07:16
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible