Duckietown Challenges Home Challenges Submissions

Evaluator 5348

ID5348
evaluatorgpu-production-spot-0-05
ownerI don't have one 😀
machinegpu-production-spot-0_f526516025fd
processgpu-production-spot-0-05_f526516025fd
version6.2.7
first heard
last heard
statusinactive
# evaluating
# success2 79655
# timeout
# failed1 79684
# error
# aborted1 79689
# host-error
arm0
x86_641
Mac0
gpu available1
Number of processors64
Processor frequency (MHz)0.0 GHz
Free % of processors95%
RAM total (MB)249.0 GB
RAM free (MB)245.6 GB
Disk (MB)969.3 GB
Disk available (MB)882.1 GB
Docker Hub
P11
P2
Cloud simulations1
PI Camera0
# Duckiebots0
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
7968913798Nicholas Kostelniktemplate-randomaido-hello-sim-validation370abortednogpu-production-spot-0-050:00:22
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/images/create?tag=sha256%3Ab13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691&fromImage=docker.io%2Fnitaigao%2Faido-submissions

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 65, in docker_pull
    pulling = client.api.pull(repository=repository, tag=br.tag, stream=True, decode=True)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 415, in pull
    self._raise_for_status(response)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("pull access denied for nitaigao/aido-submissions, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 38, in docker_pull_retry
    return docker_pull(client, image_name, quiet=quiet)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 84, in docker_pull
    raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: Cannot pull repo  docker.io/nitaigao/aido-submissions@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691  tag  None

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 745, in get_cr
    cr = run_single(
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 944, in run_single
    docker_pull_retry(client, image, ntimes=4, wait=5)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 42, in docker_pull_retry
    raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: After trying 4 I still could not pull docker.io/nitaigao/aido-submissions@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
7968416087Matteo Razzanellicollision-check-exercisemooc-collision-check-valistep1failednogpu-production-spot-0-050:00:16
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/dt_protocols/checker.py", line 82, in run_checker
    msg = agent_ci.write_topic_and_expect("query", q, expect="response")
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 135, in write_topic_and_expect
    ob: MsgReceived = self.read_one(expect_topic=expect, timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 218, in read_one
    msgs = read_reply(self.fpout, timeout=timeout, waiting_for=waiting_for, nickname=self.nickname,)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 304, in read_reply
    others = read_until_over(fpout, timeout=timeout, nickname=nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 331, in read_until_over
    raise RemoteNodeAborted(m)
zuper_nodes.structures.RemoteNodeAborted: External node "checker" aborted:

error in checker |Exception while handling a message on topic "query".
                 |
                 || Traceback (most recent call last):
                 ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
                 ||     f(**kwargs)
                 ||   File "/project/collision_checker.py", line 30, in on_received_query
                 ||     collided = check_collision(Wcoll=self.params.environment, robot_body=self.params.body, robot_pose=data.pose)
                 ||   File "/project/collision_checker.py", line 45, in check_collision
                 ||     rototranslated_robot.append(np.dot(T_or,p))
                 ||   File "<__array_function__ internals>", line 5, in dot
                 || TypeError: unsupported operand type(s) for *: 'float' and 'PlacedPrimitive'
                 ||
                 || The above exception was the direct cause of the following exception:
                 ||
                 || Traceback (most recent call last):
                 ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 385, in loop
                 ||     handle_message_node(parsed, receiver0, context0)
                 ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 561, in handle_message_node
                 ||     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
                 ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/utils.py", line 24, in call_if_fun_exists
                 ||     raise ZTypeError(msg, f=f, args=kwargs, argspec=a) from e
                 || zuper_commons.types.exceptions.ZTypeError: Cannot call function <bound method CollisionChecker.on_received_query of <collision_checker.CollisionChecker object at 0x7f44ecfb0ca0>>.
                 || │       f: <bound method CollisionChecker.on_received_query of <collision_checker.CollisionChecker object at 0x7f44ecfb0ca0>>
                 || │    args: dict[2]
                 || │          │ data:
                 || │          │ CollisionCheckQuery
                 || │          │ │ pose:
                 || │          │ │ FriendlyPose
                 || │          │ │ │ x: 3.357616573732855
                 || │          │ │ │ y: 3.885123846583553
                 || │          │ │ │ theta_deg: 224.5250538878613
                 || │          │ context: <zuper_nodes_wrapper.wrapper.ConcreteContext object at 0x7f43bef0dd60>
                 || │ argspec: <class 'inspect.FullArgSpec'>[7]
                 || │          #0 [self, context, data]
                 || │          #1 None
                 || │          #2 None
                 || │          #3 None
                 || │          #4 []
                 || │          #5 None
                 || │          #6 dict[2]
                 || │             │ context: <class 'zuper_nodes_wrapper.interface.Context'>
                 || │             │ data:
                 || │             │ dataclass dt_protocols.collision_protocol.CollisionCheckQuery
                 || │             │  field pose : dataclass aido_schemas.protocol_simulator.FriendlyPose
                 || │             │                field         x : float
                 || │             │                field         y : float
                 || │             │                field theta_deg : float
                 ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/dt_collision/planning_checker.py", line 89, in planning_checker_main
    sessions = run_checker(
  File "/usr/local/lib/python3.8/site-packages/dt_protocols/checker.py", line 95, in run_checker
    raise dc.InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: The remote node has violated protocol
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
7967716091Matteo Razzanellicollision-check-exercisemooc-collision-check-valistep1successnogpu-production-spot-0-050:00:25
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
false_negative_rate0.4
false_positive_rate0.03


other stats
nsamples400
No reset possible
7965516081Evanthia Fasoulaexercises_braitenbergmooc-BV1sim-3of5successnogpu-production-spot-0-050:04:13
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
distance-from-start_mean6.153299480171918


other stats
agent_compute-ego0_max0.01196851176345002
agent_compute-ego0_mean0.01196851176345002
agent_compute-ego0_median0.01196851176345002
agent_compute-ego0_min0.01196851176345002
complete-iteration_max0.1975824635058518
complete-iteration_mean0.1975824635058518
complete-iteration_median0.1975824635058518
complete-iteration_min0.1975824635058518
deviation-center-line_max0.0
deviation-center-line_mean0.0
deviation-center-line_median0.0
deviation-center-line_min0.0
deviation-heading_max0.0
deviation-heading_mean0.0
deviation-heading_median0.0
deviation-heading_min0.0
distance-from-start_max6.153299480171918
distance-from-start_median6.153299480171918
distance-from-start_min6.153299480171918
driven_any_max6.244177031237488
driven_any_mean6.244177031237488
driven_any_median6.244177031237488
driven_any_min6.244177031237488
driven_lanedir_consec_max0.0
driven_lanedir_consec_mean0.0
driven_lanedir_consec_median0.0
driven_lanedir_consec_min0.0
driven_lanedir_max0.0
driven_lanedir_mean0.0
driven_lanedir_median0.0
driven_lanedir_min0.0
get_duckie_state_max0.06332919805376362
get_duckie_state_mean0.06332919805376362
get_duckie_state_median0.06332919805376362
get_duckie_state_min0.06332919805376362
get_robot_state_max0.0038966044350778415
get_robot_state_mean0.0038966044350778415
get_robot_state_median0.0038966044350778415
get_robot_state_min0.0038966044350778415
get_state_dump_max0.014811835348358789
get_state_dump_mean0.014811835348358789
get_state_dump_median0.014811835348358789
get_state_dump_min0.014811835348358789
get_ui_image_max0.01418980127548281
get_ui_image_mean0.01418980127548281
get_ui_image_median0.01418980127548281
get_ui_image_min0.01418980127548281
in-drivable-lane_max12.000000000000036
in-drivable-lane_mean12.000000000000036
in-drivable-lane_median12.000000000000036
in-drivable-lane_min12.000000000000036
per-episodes
details{"d30-ego0": {"driven_any": 6.244177031237488, "get_ui_image": 0.01418980127548281, "step_physics": 0.07322102562520513, "survival_time": 12.000000000000036, "driven_lanedir": 0.0, "get_state_dump": 0.014811835348358789, "get_robot_state": 0.0038966044350778415, "sim_render-ego0": 0.003725213133942537, "get_duckie_state": 0.06332919805376362, "in-drivable-lane": 12.000000000000036, "deviation-heading": 0.0, "agent_compute-ego0": 0.01196851176345002, "complete-iteration": 0.1975824635058518, "set_robot_commands": 0.002375062570532328, "distance-from-start": 6.153299480171918, "deviation-center-line": 0.0, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.007981498211745899, "sim_compute_performance-ego0": 0.0019883496138070134}}
set_robot_commands_max0.002375062570532328
set_robot_commands_mean0.002375062570532328
set_robot_commands_median0.002375062570532328
set_robot_commands_min0.002375062570532328
sim_compute_performance-ego0_max0.0019883496138070134
sim_compute_performance-ego0_mean0.0019883496138070134
sim_compute_performance-ego0_median0.0019883496138070134
sim_compute_performance-ego0_min0.0019883496138070134
sim_compute_sim_state_max0.007981498211745899
sim_compute_sim_state_mean0.007981498211745899
sim_compute_sim_state_median0.007981498211745899
sim_compute_sim_state_min0.007981498211745899
sim_render-ego0_max0.003725213133942537
sim_render-ego0_mean0.003725213133942537
sim_render-ego0_median0.003725213133942537
sim_render-ego0_min0.003725213133942537
simulation-passed1
step_physics_max0.07322102562520513
step_physics_mean0.07322102562520513
step_physics_median0.07322102562520513
step_physics_min0.07322102562520513
survival_time_max12.000000000000036
survival_time_mean12.000000000000036
survival_time_median12.000000000000036
survival_time_min12.000000000000036
No reset possible