InvalidSubmission:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
robot_ci.write_topic_and_expect_zero("seed", config.seed)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:
error in ego0 |Unexpected error:
|
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 60, in init
|| self.anomaly_model = cbcNetv2.get_anomaly_inference(anomaly_log)
|| File "/submission/cbcNetv2.py", line 92, in get_anomaly_inference
|| model.load_weights(weigths)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 2227, in load_weights
|| with h5py.File(filepath, 'r') as f:
|| File "/usr/local/lib/python3.8/dist-packages/h5py/_hl/files.py", line 406, in __init__
|| fid = make_fid(name, mode, userblock_size,
|| File "/usr/local/lib/python3.8/dist-packages/h5py/_hl/files.py", line 173, in make_fid
|| fid = h5f.open(name, flags, fapl=fapl)
|| File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
|| File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
|| File "h5py/h5f.pyx", line 88, in h5py.h5f.open
|| OSError: Unable to open file (unable to open file: name = 'cbcNetv2-Anomaly-Best_Validation_mar28_apr6.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
||
|| The above exception was the direct cause of the following exception:
||
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 339, in loop
|| raise Exception(msg) from e
|| Exception: Exception while calling the node's init() function.
||
|| | Traceback (most recent call last):
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| | call_if_fun_exists(node, "init", context=context_data)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| | f(**kwargs)
|| | File "solution.py", line 60, in init
|| | self.anomaly_model = cbcNetv2.get_anomaly_inference(anomaly_log)
|| | File "/submission/cbcNetv2.py", line 92, in get_anomaly_inference
|| | model.load_weights(weigths)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 2227, in load_weights
|| | with h5py.File(filepath, 'r') as f:
|| | File "/usr/local/lib/python3.8/dist-packages/h5py/_hl/files.py", line 406, in __init__
|| | fid = make_fid(name, mode, userblock_size,
|| | File "/usr/local/lib/python3.8/dist-packages/h5py/_hl/files.py", line 173, in make_fid
|| | fid = h5f.open(name, flags, fapl=fapl)
|| | File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
|| | File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
|| | File "h5py/h5f.pyx", line 88, in h5py.h5f.open
|| | OSError: Unable to open file (unable to open file: name = 'cbcNetv2-Anomaly-Best_Validation_mar28_apr6.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
|| |
||
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 249, in main
raise InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Getting agent protocol
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
InvalidSubmission:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 190, in main
raise InvalidSubmission(msg)
duckietown_challenges.exceptions.InvalidSubmission: Timeout during connection to ego0: <SignalTimeout in state: 2>
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
Uncaught exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 261, in _raise_for_status
response.raise_for_status()
File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/images/create?tag=sha256%3A929b5188a9b5978ea78814123e2e9dc311d26d3ce658c7d75a975d0e04f6503d&fromImage=docker.io%2Fgoldbattle%2Faido-submissions
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 65, in docker_pull
pulling = client.api.pull(repository=repository, tag=br.tag, stream=True, decode=True)
File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 415, in pull
self._raise_for_status(response)
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("pull access denied for goldbattle/aido-submissions, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 38, in docker_pull_retry
return docker_pull(client, image_name, quiet=quiet)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 84, in docker_pull
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: Cannot pull repo docker.io/goldbattle/aido-submissions@sha256:929b5188a9b5978ea78814123e2e9dc311d26d3ce658c7d75a975d0e04f6503d tag None
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 777, in get_cr
cr = run_single(
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 976, in run_single
docker_pull_retry(client, image, ntimes=4, wait=5)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 42, in docker_pull_retry
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: After trying 4 I still could not pull docker.io/goldbattle/aido-submissions@sha256:929b5188a9b5978ea78814123e2e9dc311d26d3ce658c7d75a975d0e04f6503d
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
Uncaught exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 261, in _raise_for_status
response.raise_for_status()
File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/images/create?tag=sha256%3A2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809&fromImage=docker.io%2Fh0mlab%2Faido-submissions
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 65, in docker_pull
pulling = client.api.pull(repository=repository, tag=br.tag, stream=True, decode=True)
File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 415, in pull
self._raise_for_status(response)
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("pull access denied for h0mlab/aido-submissions, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 38, in docker_pull_retry
return docker_pull(client, image_name, quiet=quiet)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 84, in docker_pull
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: Cannot pull repo docker.io/h0mlab/aido-submissions@sha256:2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809 tag None
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 777, in get_cr
cr = run_single(
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 976, in run_single
docker_pull_retry(client, image, ntimes=4, wait=5)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 42, in docker_pull_retry
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: After trying 4 I still could not pull docker.io/h0mlab/aido-submissions@sha256:2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
Uncaught exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 261, in _raise_for_status
response.raise_for_status()
File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/images/create?tag=sha256%3A2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809&fromImage=docker.io%2Fh0mlab%2Faido-submissions
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 65, in docker_pull
pulling = client.api.pull(repository=repository, tag=br.tag, stream=True, decode=True)
File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 415, in pull
self._raise_for_status(response)
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("pull access denied for h0mlab/aido-submissions, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 38, in docker_pull_retry
return docker_pull(client, image_name, quiet=quiet)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 84, in docker_pull
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: Cannot pull repo docker.io/h0mlab/aido-submissions@sha256:2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809 tag None
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 777, in get_cr
cr = run_single(
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 976, in run_single
docker_pull_retry(client, image, ntimes=4, wait=5)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 42, in docker_pull_retry
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: After trying 4 I still could not pull docker.io/h0mlab/aido-submissions@sha256:2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
Uncaught exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 261, in _raise_for_status
response.raise_for_status()
File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/images/create?tag=sha256%3A2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809&fromImage=docker.io%2Fh0mlab%2Faido-submissions
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 65, in docker_pull
pulling = client.api.pull(repository=repository, tag=br.tag, stream=True, decode=True)
File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 415, in pull
self._raise_for_status(response)
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("pull access denied for h0mlab/aido-submissions, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 38, in docker_pull_retry
return docker_pull(client, image_name, quiet=quiet)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 84, in docker_pull
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: Cannot pull repo docker.io/h0mlab/aido-submissions@sha256:2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809 tag None
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 777, in get_cr
cr = run_single(
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 976, in run_single
docker_pull_retry(client, image, ntimes=4, wait=5)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 42, in docker_pull_retry
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: After trying 4 I still could not pull docker.io/h0mlab/aido-submissions@sha256:2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
Uncaught exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 261, in _raise_for_status
response.raise_for_status()
File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/images/create?tag=sha256%3A2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809&fromImage=docker.io%2Fh0mlab%2Faido-submissions
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 65, in docker_pull
pulling = client.api.pull(repository=repository, tag=br.tag, stream=True, decode=True)
File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 415, in pull
self._raise_for_status(response)
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("pull access denied for h0mlab/aido-submissions, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 38, in docker_pull_retry
return docker_pull(client, image_name, quiet=quiet)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 84, in docker_pull
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: Cannot pull repo docker.io/h0mlab/aido-submissions@sha256:2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809 tag None
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 777, in get_cr
cr = run_single(
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 976, in run_single
docker_pull_retry(client, image, ntimes=4, wait=5)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 42, in docker_pull_retry
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: After trying 4 I still could not pull docker.io/h0mlab/aido-submissions@sha256:2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
Uncaught exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 261, in _raise_for_status
response.raise_for_status()
File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/images/create?tag=sha256%3A2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809&fromImage=docker.io%2Fh0mlab%2Faido-submissions
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 65, in docker_pull
pulling = client.api.pull(repository=repository, tag=br.tag, stream=True, decode=True)
File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 415, in pull
self._raise_for_status(response)
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("pull access denied for h0mlab/aido-submissions, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 38, in docker_pull_retry
return docker_pull(client, image_name, quiet=quiet)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 84, in docker_pull
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: Cannot pull repo docker.io/h0mlab/aido-submissions@sha256:2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809 tag None
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 777, in get_cr
cr = run_single(
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 976, in run_single
docker_pull_retry(client, image, ntimes=4, wait=5)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 42, in docker_pull_retry
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: After trying 4 I still could not pull docker.io/h0mlab/aido-submissions@sha256:2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
Uncaught exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 261, in _raise_for_status
response.raise_for_status()
File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/images/create?tag=sha256%3A2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809&fromImage=docker.io%2Fh0mlab%2Faido-submissions
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 65, in docker_pull
pulling = client.api.pull(repository=repository, tag=br.tag, stream=True, decode=True)
File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 415, in pull
self._raise_for_status(response)
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("pull access denied for h0mlab/aido-submissions, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 38, in docker_pull_retry
return docker_pull(client, image_name, quiet=quiet)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 84, in docker_pull
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: Cannot pull repo docker.io/h0mlab/aido-submissions@sha256:2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809 tag None
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 777, in get_cr
cr = run_single(
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 976, in run_single
docker_pull_retry(client, image, ntimes=4, wait=5)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 42, in docker_pull_retry
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: After trying 4 I still could not pull docker.io/h0mlab/aido-submissions@sha256:2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
Uncaught exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 261, in _raise_for_status
response.raise_for_status()
File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/images/create?tag=sha256%3A2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809&fromImage=docker.io%2Fh0mlab%2Faido-submissions
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 65, in docker_pull
pulling = client.api.pull(repository=repository, tag=br.tag, stream=True, decode=True)
File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 415, in pull
self._raise_for_status(response)
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("pull access denied for h0mlab/aido-submissions, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 38, in docker_pull_retry
return docker_pull(client, image_name, quiet=quiet)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 84, in docker_pull
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: Cannot pull repo docker.io/h0mlab/aido-submissions@sha256:2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809 tag None
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 777, in get_cr
cr = run_single(
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 976, in run_single
docker_pull_retry(client, image, ntimes=4, wait=5)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 42, in docker_pull_retry
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: After trying 4 I still could not pull docker.io/h0mlab/aido-submissions@sha256:2a8b625a92c565f6505a4c3cc11e9240bd29d6c68007963ac067d49418699809
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
Uncaught exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 261, in _raise_for_status
response.raise_for_status()
File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/images/create?tag=sha256%3Ab13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691&fromImage=docker.io%2Fnitaigao%2Faido-submissions
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 65, in docker_pull
pulling = client.api.pull(repository=repository, tag=br.tag, stream=True, decode=True)
File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 415, in pull
self._raise_for_status(response)
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("pull access denied for nitaigao/aido-submissions, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 38, in docker_pull_retry
return docker_pull(client, image_name, quiet=quiet)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 84, in docker_pull
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: Cannot pull repo docker.io/nitaigao/aido-submissions@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691 tag None
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 777, in get_cr
cr = run_single(
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 976, in run_single
docker_pull_retry(client, image, ntimes=4, wait=5)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 42, in docker_pull_retry
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: After trying 4 I still could not pull docker.io/nitaigao/aido-submissions@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
Uncaught exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 261, in _raise_for_status
response.raise_for_status()
File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/images/create?tag=sha256%3Ab13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691&fromImage=docker.io%2Fnitaigao%2Faido-submissions
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 65, in docker_pull
pulling = client.api.pull(repository=repository, tag=br.tag, stream=True, decode=True)
File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 415, in pull
self._raise_for_status(response)
File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("pull access denied for nitaigao/aido-submissions, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 38, in docker_pull_retry
return docker_pull(client, image_name, quiet=quiet)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 84, in docker_pull
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: Cannot pull repo docker.io/nitaigao/aido-submissions@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691 tag None
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 777, in get_cr
cr = run_single(
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 976, in run_single
docker_pull_retry(client, image, ntimes=4, wait=5)
File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 42, in docker_pull_retry
raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: After trying 4 I still could not pull docker.io/nitaigao/aido-submissions@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
InvalidSubmission:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 190, in main
raise InvalidSubmission(msg)
duckietown_challenges.exceptions.InvalidSubmission: Timeout during connection to ego0: <SignalTimeout in state: 2>
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
KeyboardInterrupt:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1169, in run_one
heartbeat()
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 628, in heartbeat
raise KeyboardInterrupt(msg_)
KeyboardInterrupt: The server told us to abort the job because: The challenge has been updated.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
KeyboardInterrupt:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1169, in run_one
heartbeat()
File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 628, in heartbeat
raise KeyboardInterrupt(msg_)
KeyboardInterrupt: The server told us to abort the job because: The challenge has been updated.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
InvalidSubmission:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 190, in main
raise InvalidSubmission(msg)
duckietown_challenges.exceptions.InvalidSubmission: Timeout during connection to ego0: <SignalTimeout in state: 2>
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
InvalidSubmission:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 190, in main
raise InvalidSubmission(msg)
duckietown_challenges.exceptions.InvalidSubmission: Timeout during connection to ego0: <SignalTimeout in state: 2>
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
InvalidEnvironment:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
robot_ci.write_topic_and_expect_zero("seed", config.seed)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:
error in ego0 |Unexpected error:
|
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 45, in init
|| limit_gpu_memory()
|| File "solution.py", line 29, in limit_gpu_memory
|| logical_gpus = tf.config.experimental.list_logical_devices('GPU')
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/config.py", line 439, in list_logical_devices
|| return context.context().list_logical_devices(device_type=device_type)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 1368, in list_logical_devices
|| self.ensure_initialized()
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 526, in ensure_initialized
|| context_handle = pywrap_tfe.TFE_NewContext(opts)
|| tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version
||
|| The above exception was the direct cause of the following exception:
||
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 339, in loop
|| raise Exception(msg) from e
|| Exception: Exception while calling the node's init() function.
||
|| | Traceback (most recent call last):
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| | call_if_fun_exists(node, "init", context=context_data)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| | f(**kwargs)
|| | File "solution.py", line 45, in init
|| | limit_gpu_memory()
|| | File "solution.py", line 29, in limit_gpu_memory
|| | logical_gpus = tf.config.experimental.list_logical_devices('GPU')
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/config.py", line 439, in list_logical_devices
|| | return context.context().list_logical_devices(device_type=device_type)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 1368, in list_logical_devices
|| | self.ensure_initialized()
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 526, in ensure_initialized
|| | context_handle = pywrap_tfe.TFE_NewContext(opts)
|| | tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version
|| |
||
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
robot_ci.write_topic_and_expect_zero("seed", config.seed)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:
error in ego0 |Unexpected error:
|
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 45, in init
|| limit_gpu_memory()
|| File "solution.py", line 29, in limit_gpu_memory
|| logical_gpus = tf.config.experimental.list_logical_devices('GPU')
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/config.py", line 439, in list_logical_devices
|| return context.context().list_logical_devices(device_type=device_type)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 1368, in list_logical_devices
|| self.ensure_initialized()
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 526, in ensure_initialized
|| context_handle = pywrap_tfe.TFE_NewContext(opts)
|| tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version
||
|| The above exception was the direct cause of the following exception:
||
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 339, in loop
|| raise Exception(msg) from e
|| Exception: Exception while calling the node's init() function.
||
|| | Traceback (most recent call last):
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| | call_if_fun_exists(node, "init", context=context_data)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| | f(**kwargs)
|| | File "solution.py", line 45, in init
|| | limit_gpu_memory()
|| | File "solution.py", line 29, in limit_gpu_memory
|| | logical_gpus = tf.config.experimental.list_logical_devices('GPU')
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/config.py", line 439, in list_logical_devices
|| | return context.context().list_logical_devices(device_type=device_type)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 1368, in list_logical_devices
|| | self.ensure_initialized()
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 526, in ensure_initialized
|| | context_handle = pywrap_tfe.TFE_NewContext(opts)
|| | tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version
|| |
||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
InvalidSubmission:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
robot_ci.write_topic_and_expect_zero("seed", config.seed)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego2" aborted with the following error:
error in ego2 |Unexpected error:
|
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
|| return fn(*args)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
|| return self._call_tf_sessionrun(options, feed_dict, fetch_list,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
|| return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
|| tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| [[default_policy/strided_slice_1/_3]]
|| 0 successful operations.
|| 0 derived errors ignored.
||
|| During handling of the above exception, another exception occurred:
||
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 29, in init
|| self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| File "/submission/model.py", line 55, in __init__
|| self.model = PPOTrainer(config=config["rllib_config"])
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| Trainer.__init__(self, config, env, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| super().__init__(config, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| self._setup(copy.deepcopy(self.config))
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| self._init(self.config, self.env_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| self.workers = self._make_workers(env_creator, self._policy,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| return WorkerSet(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| self._local_worker = self._make_worker(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| worker = cls(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| self._build_policy_map(policy_dict, policy_config)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| policy_map[name] = cls(obs_space, act_space, merged_conf)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| DynamicTFPolicy.__init__(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| self._initialize_loss()
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| postprocessed_batch = self.postprocess_trajectory(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| return postprocess_fn(self, sample_batch, other_agent_batches,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 86, in call
|| return session_or_none.run(symbolic_out[0], feed_dict)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 957, in run
|| result = self._run(None, fetches, feed_dict, options_ptr,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1180, in _run
|| results = self._do_run(handle, final_targets, final_fetches,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1358, in _do_run
|| return self._do_call(_run_fn, feeds, fetches, targets, options,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1384, in _do_call
|| raise type(e)(node_def, op, message)
|| tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| [[default_policy/strided_slice_1/_3]]
|| 0 successful operations.
|| 0 derived errors ignored.
||
|| Original stack trace for 'default_policy/functional_1_1/conv_value_1/Relu':
|| File "solution.py", line 127, in <module>
|| main()
|| File "solution.py", line 123, in main
|| wrap_direct(node=node, protocol=protocol)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/interface.py", line 24, in wrap_direct
|| run_loop(node, protocol, args)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 243, in run_loop
|| loop(node_name, fi, fo, node, protocol, tin, tout, config=config, fi_desc=fin, fo_desc=fout)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 29, in init
|| self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| File "/submission/model.py", line 55, in __init__
|| self.model = PPOTrainer(config=config["rllib_config"])
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| Trainer.__init__(self, config, env, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| super().__init__(config, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| self._setup(copy.deepcopy(self.config))
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| self._init(self.config, self.env_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| self.workers = self._make_workers(env_creator, self._policy,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| return WorkerSet(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| self._local_worker = self._make_worker(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| worker = cls(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| self._build_policy_map(policy_dict, policy_config)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| policy_map[name] = cls(obs_space, act_space, merged_conf)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| DynamicTFPolicy.__init__(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| self._initialize_loss()
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| postprocessed_batch = self.postprocess_trajectory(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| return postprocess_fn(self, sample_batch, other_agent_batches,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 84, in call
|| symbolic_out[0] = fn(*placeholders)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 235, in value
|| model_out, _ = self.model({
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/modelv2.py", line 150, in __call__
|| res = self.forward(restored, state or [], seq_lens)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py", line 103, in forward
|| model_out, self._value_out = self.base_model(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| outputs = call_fn(cast_inputs, *args, **kwargs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 385, in call
|| return self._run_internal_graph(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 508, in _run_internal_graph
|| outputs = node.layer(*args, **kwargs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| outputs = call_fn(cast_inputs, *args, **kwargs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/layers/convolutional.py", line 269, in call
|| return self.activation(outputs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 10435, in relu
|| _, _, _op, _outputs = _op_def_library._apply_op_helper(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/op_def_library.py", line 742, in _apply_op_helper
|| op = g._create_op_internal(op_type_name, inputs, dtypes=None,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 3477, in _create_op_internal
|| ret = Operation(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1949, in __init__
|| self._traceback = tf_stack.extract_stack()
||
||
|| The above exception was the direct cause of the following exception:
||
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 339, in loop
|| raise Exception(msg) from e
|| Exception: Exception while calling the node's init() function.
||
|| | Traceback (most recent call last):
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
|| | return fn(*args)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
|| | return self._call_tf_sessionrun(options, feed_dict, fetch_list,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
|| | return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
|| | tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| | (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| | (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| | [[default_policy/strided_slice_1/_3]]
|| | 0 successful operations.
|| | 0 derived errors ignored.
|| |
|| | During handling of the above exception, another exception occurred:
|| |
|| | Traceback (most recent call last):
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| | call_if_fun_exists(node, "init", context=context_data)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| | f(**kwargs)
|| | File "solution.py", line 29, in init
|| | self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| | File "/submission/model.py", line 55, in __init__
|| | self.model = PPOTrainer(config=config["rllib_config"])
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| | Trainer.__init__(self, config, env, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| | super().__init__(config, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| | self._setup(copy.deepcopy(self.config))
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| | self._init(self.config, self.env_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| | self.workers = self._make_workers(env_creator, self._policy,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| | return WorkerSet(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| | self._local_worker = self._make_worker(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| | worker = cls(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| | self._build_policy_map(policy_dict, policy_config)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| | policy_map[name] = cls(obs_space, act_space, merged_conf)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| | DynamicTFPolicy.__init__(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| | self._initialize_loss()
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| | postprocessed_batch = self.postprocess_trajectory(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| | return postprocess_fn(self, sample_batch, other_agent_batches,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| | last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 86, in call
|| | return session_or_none.run(symbolic_out[0], feed_dict)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 957, in run
|| | result = self._run(None, fetches, feed_dict, options_ptr,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1180, in _run
|| | results = self._do_run(handle, final_targets, final_fetches,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1358, in _do_run
|| | return self._do_call(_run_fn, feeds, fetches, targets, options,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1384, in _do_call
|| | raise type(e)(node_def, op, message)
|| | tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| | (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| | (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| | [[default_policy/strided_slice_1/_3]]
|| | 0 successful operations.
|| | 0 derived errors ignored.
|| |
|| | Original stack trace for 'default_policy/functional_1_1/conv_value_1/Relu':
|| | File "solution.py", line 127, in <module>
|| | main()
|| | File "solution.py", line 123, in main
|| | wrap_direct(node=node, protocol=protocol)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/interface.py", line 24, in wrap_direct
|| | run_loop(node, protocol, args)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 243, in run_loop
|| | loop(node_name, fi, fo, node, protocol, tin, tout, config=config, fi_desc=fin, fo_desc=fout)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| | call_if_fun_exists(node, "init", context=context_data)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| | f(**kwargs)
|| | File "solution.py", line 29, in init
|| | self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| | File "/submission/model.py", line 55, in __init__
|| | self.model = PPOTrainer(config=config["rllib_config"])
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| | Trainer.__init__(self, config, env, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| | super().__init__(config, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| | self._setup(copy.deepcopy(self.config))
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| | self._init(self.config, self.env_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| | self.workers = self._make_workers(env_creator, self._policy,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| | return WorkerSet(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| | self._local_worker = self._make_worker(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| | worker = cls(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| | self._build_policy_map(policy_dict, policy_config)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| | policy_map[name] = cls(obs_space, act_space, merged_conf)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| | DynamicTFPolicy.__init__(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| | self._initialize_loss()
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| | postprocessed_batch = self.postprocess_trajectory(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| | return postprocess_fn(self, sample_batch, other_agent_batches,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| | last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 84, in call
|| | symbolic_out[0] = fn(*placeholders)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 235, in value
|| | model_out, _ = self.model({
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/modelv2.py", line 150, in __call__
|| | res = self.forward(restored, state or [], seq_lens)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py", line 103, in forward
|| | model_out, self._value_out = self.base_model(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| | outputs = call_fn(cast_inputs, *args, **kwargs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 385, in call
|| | return self._run_internal_graph(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 508, in _run_internal_graph
|| | outputs = node.layer(*args, **kwargs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| | outputs = call_fn(cast_inputs, *args, **kwargs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/layers/convolutional.py", line 269, in call
|| | return self.activation(outputs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 10435, in relu
|| | _, _, _op, _outputs = _op_def_library._apply_op_helper(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/op_def_library.py", line 742, in _apply_op_helper
|| | op = g._create_op_internal(op_type_name, inputs, dtypes=None,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 3477, in _create_op_internal
|| | ret = Operation(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1949, in __init__
|| | self._traceback = tf_stack.extract_stack()
|| |
|| |
||
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 249, in main
raise InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Getting agent protocol
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
InvalidSubmission:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
robot_ci.write_topic_and_expect_zero("seed", config.seed)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego2" aborted with the following error:
error in ego2 |Unexpected error:
|
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
|| return fn(*args)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
|| return self._call_tf_sessionrun(options, feed_dict, fetch_list,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
|| return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
|| tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| [[default_policy/strided_slice_1/_3]]
|| 0 successful operations.
|| 0 derived errors ignored.
||
|| During handling of the above exception, another exception occurred:
||
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 29, in init
|| self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| File "/submission/model.py", line 55, in __init__
|| self.model = PPOTrainer(config=config["rllib_config"])
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| Trainer.__init__(self, config, env, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| super().__init__(config, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| self._setup(copy.deepcopy(self.config))
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| self._init(self.config, self.env_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| self.workers = self._make_workers(env_creator, self._policy,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| return WorkerSet(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| self._local_worker = self._make_worker(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| worker = cls(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| self._build_policy_map(policy_dict, policy_config)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| policy_map[name] = cls(obs_space, act_space, merged_conf)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| DynamicTFPolicy.__init__(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| self._initialize_loss()
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| postprocessed_batch = self.postprocess_trajectory(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| return postprocess_fn(self, sample_batch, other_agent_batches,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 86, in call
|| return session_or_none.run(symbolic_out[0], feed_dict)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 957, in run
|| result = self._run(None, fetches, feed_dict, options_ptr,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1180, in _run
|| results = self._do_run(handle, final_targets, final_fetches,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1358, in _do_run
|| return self._do_call(_run_fn, feeds, fetches, targets, options,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1384, in _do_call
|| raise type(e)(node_def, op, message)
|| tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| [[default_policy/strided_slice_1/_3]]
|| 0 successful operations.
|| 0 derived errors ignored.
||
|| Original stack trace for 'default_policy/functional_1_1/conv_value_1/Relu':
|| File "solution.py", line 127, in <module>
|| main()
|| File "solution.py", line 123, in main
|| wrap_direct(node=node, protocol=protocol)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/interface.py", line 24, in wrap_direct
|| run_loop(node, protocol, args)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 243, in run_loop
|| loop(node_name, fi, fo, node, protocol, tin, tout, config=config, fi_desc=fin, fo_desc=fout)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 29, in init
|| self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| File "/submission/model.py", line 55, in __init__
|| self.model = PPOTrainer(config=config["rllib_config"])
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| Trainer.__init__(self, config, env, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| super().__init__(config, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| self._setup(copy.deepcopy(self.config))
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| self._init(self.config, self.env_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| self.workers = self._make_workers(env_creator, self._policy,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| return WorkerSet(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| self._local_worker = self._make_worker(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| worker = cls(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| self._build_policy_map(policy_dict, policy_config)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| policy_map[name] = cls(obs_space, act_space, merged_conf)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| DynamicTFPolicy.__init__(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| self._initialize_loss()
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| postprocessed_batch = self.postprocess_trajectory(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| return postprocess_fn(self, sample_batch, other_agent_batches,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 84, in call
|| symbolic_out[0] = fn(*placeholders)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 235, in value
|| model_out, _ = self.model({
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/modelv2.py", line 150, in __call__
|| res = self.forward(restored, state or [], seq_lens)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py", line 103, in forward
|| model_out, self._value_out = self.base_model(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| outputs = call_fn(cast_inputs, *args, **kwargs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 385, in call
|| return self._run_internal_graph(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 508, in _run_internal_graph
|| outputs = node.layer(*args, **kwargs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| outputs = call_fn(cast_inputs, *args, **kwargs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/layers/convolutional.py", line 269, in call
|| return self.activation(outputs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 10435, in relu
|| _, _, _op, _outputs = _op_def_library._apply_op_helper(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/op_def_library.py", line 742, in _apply_op_helper
|| op = g._create_op_internal(op_type_name, inputs, dtypes=None,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 3477, in _create_op_internal
|| ret = Operation(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1949, in __init__
|| self._traceback = tf_stack.extract_stack()
||
||
|| The above exception was the direct cause of the following exception:
||
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 339, in loop
|| raise Exception(msg) from e
|| Exception: Exception while calling the node's init() function.
||
|| | Traceback (most recent call last):
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
|| | return fn(*args)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
|| | return self._call_tf_sessionrun(options, feed_dict, fetch_list,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
|| | return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
|| | tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| | (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| | (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| | [[default_policy/strided_slice_1/_3]]
|| | 0 successful operations.
|| | 0 derived errors ignored.
|| |
|| | During handling of the above exception, another exception occurred:
|| |
|| | Traceback (most recent call last):
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| | call_if_fun_exists(node, "init", context=context_data)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| | f(**kwargs)
|| | File "solution.py", line 29, in init
|| | self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| | File "/submission/model.py", line 55, in __init__
|| | self.model = PPOTrainer(config=config["rllib_config"])
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| | Trainer.__init__(self, config, env, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| | super().__init__(config, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| | self._setup(copy.deepcopy(self.config))
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| | self._init(self.config, self.env_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| | self.workers = self._make_workers(env_creator, self._policy,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| | return WorkerSet(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| | self._local_worker = self._make_worker(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| | worker = cls(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| | self._build_policy_map(policy_dict, policy_config)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| | policy_map[name] = cls(obs_space, act_space, merged_conf)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| | DynamicTFPolicy.__init__(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| | self._initialize_loss()
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| | postprocessed_batch = self.postprocess_trajectory(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| | return postprocess_fn(self, sample_batch, other_agent_batches,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| | last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 86, in call
|| | return session_or_none.run(symbolic_out[0], feed_dict)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 957, in run
|| | result = self._run(None, fetches, feed_dict, options_ptr,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1180, in _run
|| | results = self._do_run(handle, final_targets, final_fetches,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1358, in _do_run
|| | return self._do_call(_run_fn, feeds, fetches, targets, options,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1384, in _do_call
|| | raise type(e)(node_def, op, message)
|| | tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| | (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| | (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| | [[default_policy/strided_slice_1/_3]]
|| | 0 successful operations.
|| | 0 derived errors ignored.
|| |
|| | Original stack trace for 'default_policy/functional_1_1/conv_value_1/Relu':
|| | File "solution.py", line 127, in <module>
|| | main()
|| | File "solution.py", line 123, in main
|| | wrap_direct(node=node, protocol=protocol)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/interface.py", line 24, in wrap_direct
|| | run_loop(node, protocol, args)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 243, in run_loop
|| | loop(node_name, fi, fo, node, protocol, tin, tout, config=config, fi_desc=fin, fo_desc=fout)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| | call_if_fun_exists(node, "init", context=context_data)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| | f(**kwargs)
|| | File "solution.py", line 29, in init
|| | self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| | File "/submission/model.py", line 55, in __init__
|| | self.model = PPOTrainer(config=config["rllib_config"])
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| | Trainer.__init__(self, config, env, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| | super().__init__(config, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| | self._setup(copy.deepcopy(self.config))
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| | self._init(self.config, self.env_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| | self.workers = self._make_workers(env_creator, self._policy,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| | return WorkerSet(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| | self._local_worker = self._make_worker(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| | worker = cls(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| | self._build_policy_map(policy_dict, policy_config)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| | policy_map[name] = cls(obs_space, act_space, merged_conf)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| | DynamicTFPolicy.__init__(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| | self._initialize_loss()
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| | postprocessed_batch = self.postprocess_trajectory(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| | return postprocess_fn(self, sample_batch, other_agent_batches,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| | last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 84, in call
|| | symbolic_out[0] = fn(*placeholders)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 235, in value
|| | model_out, _ = self.model({
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/modelv2.py", line 150, in __call__
|| | res = self.forward(restored, state or [], seq_lens)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py", line 103, in forward
|| | model_out, self._value_out = self.base_model(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| | outputs = call_fn(cast_inputs, *args, **kwargs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 385, in call
|| | return self._run_internal_graph(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 508, in _run_internal_graph
|| | outputs = node.layer(*args, **kwargs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| | outputs = call_fn(cast_inputs, *args, **kwargs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/layers/convolutional.py", line 269, in call
|| | return self.activation(outputs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 10435, in relu
|| | _, _, _op, _outputs = _op_def_library._apply_op_helper(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/op_def_library.py", line 742, in _apply_op_helper
|| | op = g._create_op_internal(op_type_name, inputs, dtypes=None,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 3477, in _create_op_internal
|| | ret = Operation(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1949, in __init__
|| | self._traceback = tf_stack.extract_stack()
|| |
|| |
||
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 249, in main
raise InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Getting agent protocol
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
InvalidSubmission:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
robot_ci.write_topic_and_expect_zero("seed", config.seed)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego2" aborted with the following error:
error in ego2 |Unexpected error:
|
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
|| return fn(*args)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
|| return self._call_tf_sessionrun(options, feed_dict, fetch_list,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
|| return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
|| tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| [[default_policy/strided_slice_1/_3]]
|| 0 successful operations.
|| 0 derived errors ignored.
||
|| During handling of the above exception, another exception occurred:
||
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 29, in init
|| self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| File "/submission/model.py", line 55, in __init__
|| self.model = PPOTrainer(config=config["rllib_config"])
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| Trainer.__init__(self, config, env, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| super().__init__(config, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| self._setup(copy.deepcopy(self.config))
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| self._init(self.config, self.env_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| self.workers = self._make_workers(env_creator, self._policy,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| return WorkerSet(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| self._local_worker = self._make_worker(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| worker = cls(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| self._build_policy_map(policy_dict, policy_config)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| policy_map[name] = cls(obs_space, act_space, merged_conf)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| DynamicTFPolicy.__init__(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| self._initialize_loss()
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| postprocessed_batch = self.postprocess_trajectory(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| return postprocess_fn(self, sample_batch, other_agent_batches,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 86, in call
|| return session_or_none.run(symbolic_out[0], feed_dict)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 957, in run
|| result = self._run(None, fetches, feed_dict, options_ptr,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1180, in _run
|| results = self._do_run(handle, final_targets, final_fetches,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1358, in _do_run
|| return self._do_call(_run_fn, feeds, fetches, targets, options,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1384, in _do_call
|| raise type(e)(node_def, op, message)
|| tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| [[default_policy/strided_slice_1/_3]]
|| 0 successful operations.
|| 0 derived errors ignored.
||
|| Original stack trace for 'default_policy/functional_1_1/conv_value_1/Relu':
|| File "solution.py", line 127, in <module>
|| main()
|| File "solution.py", line 123, in main
|| wrap_direct(node=node, protocol=protocol)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/interface.py", line 24, in wrap_direct
|| run_loop(node, protocol, args)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 243, in run_loop
|| loop(node_name, fi, fo, node, protocol, tin, tout, config=config, fi_desc=fin, fo_desc=fout)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 29, in init
|| self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| File "/submission/model.py", line 55, in __init__
|| self.model = PPOTrainer(config=config["rllib_config"])
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| Trainer.__init__(self, config, env, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| super().__init__(config, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| self._setup(copy.deepcopy(self.config))
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| self._init(self.config, self.env_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| self.workers = self._make_workers(env_creator, self._policy,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| return WorkerSet(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| self._local_worker = self._make_worker(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| worker = cls(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| self._build_policy_map(policy_dict, policy_config)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| policy_map[name] = cls(obs_space, act_space, merged_conf)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| DynamicTFPolicy.__init__(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| self._initialize_loss()
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| postprocessed_batch = self.postprocess_trajectory(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| return postprocess_fn(self, sample_batch, other_agent_batches,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 84, in call
|| symbolic_out[0] = fn(*placeholders)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 235, in value
|| model_out, _ = self.model({
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/modelv2.py", line 150, in __call__
|| res = self.forward(restored, state or [], seq_lens)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py", line 103, in forward
|| model_out, self._value_out = self.base_model(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| outputs = call_fn(cast_inputs, *args, **kwargs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 385, in call
|| return self._run_internal_graph(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 508, in _run_internal_graph
|| outputs = node.layer(*args, **kwargs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| outputs = call_fn(cast_inputs, *args, **kwargs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/layers/convolutional.py", line 269, in call
|| return self.activation(outputs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 10435, in relu
|| _, _, _op, _outputs = _op_def_library._apply_op_helper(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/op_def_library.py", line 742, in _apply_op_helper
|| op = g._create_op_internal(op_type_name, inputs, dtypes=None,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 3477, in _create_op_internal
|| ret = Operation(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1949, in __init__
|| self._traceback = tf_stack.extract_stack()
||
||
|| The above exception was the direct cause of the following exception:
||
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 339, in loop
|| raise Exception(msg) from e
|| Exception: Exception while calling the node's init() function.
||
|| | Traceback (most recent call last):
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
|| | return fn(*args)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
|| | return self._call_tf_sessionrun(options, feed_dict, fetch_list,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
|| | return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
|| | tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| | (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| | (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| | [[default_policy/strided_slice_1/_3]]
|| | 0 successful operations.
|| | 0 derived errors ignored.
|| |
|| | During handling of the above exception, another exception occurred:
|| |
|| | Traceback (most recent call last):
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| | call_if_fun_exists(node, "init", context=context_data)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| | f(**kwargs)
|| | File "solution.py", line 29, in init
|| | self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| | File "/submission/model.py", line 55, in __init__
|| | self.model = PPOTrainer(config=config["rllib_config"])
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| | Trainer.__init__(self, config, env, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| | super().__init__(config, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| | self._setup(copy.deepcopy(self.config))
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| | self._init(self.config, self.env_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| | self.workers = self._make_workers(env_creator, self._policy,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| | return WorkerSet(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| | self._local_worker = self._make_worker(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| | worker = cls(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| | self._build_policy_map(policy_dict, policy_config)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| | policy_map[name] = cls(obs_space, act_space, merged_conf)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| | DynamicTFPolicy.__init__(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| | self._initialize_loss()
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| | postprocessed_batch = self.postprocess_trajectory(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| | return postprocess_fn(self, sample_batch, other_agent_batches,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| | last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 86, in call
|| | return session_or_none.run(symbolic_out[0], feed_dict)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 957, in run
|| | result = self._run(None, fetches, feed_dict, options_ptr,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1180, in _run
|| | results = self._do_run(handle, final_targets, final_fetches,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1358, in _do_run
|| | return self._do_call(_run_fn, feeds, fetches, targets, options,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1384, in _do_call
|| | raise type(e)(node_def, op, message)
|| | tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| | (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| | (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| | [[default_policy/strided_slice_1/_3]]
|| | 0 successful operations.
|| | 0 derived errors ignored.
|| |
|| | Original stack trace for 'default_policy/functional_1_1/conv_value_1/Relu':
|| | File "solution.py", line 127, in <module>
|| | main()
|| | File "solution.py", line 123, in main
|| | wrap_direct(node=node, protocol=protocol)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/interface.py", line 24, in wrap_direct
|| | run_loop(node, protocol, args)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 243, in run_loop
|| | loop(node_name, fi, fo, node, protocol, tin, tout, config=config, fi_desc=fin, fo_desc=fout)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| | call_if_fun_exists(node, "init", context=context_data)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| | f(**kwargs)
|| | File "solution.py", line 29, in init
|| | self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| | File "/submission/model.py", line 55, in __init__
|| | self.model = PPOTrainer(config=config["rllib_config"])
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| | Trainer.__init__(self, config, env, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| | super().__init__(config, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| | self._setup(copy.deepcopy(self.config))
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| | self._init(self.config, self.env_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| | self.workers = self._make_workers(env_creator, self._policy,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| | return WorkerSet(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| | self._local_worker = self._make_worker(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| | worker = cls(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| | self._build_policy_map(policy_dict, policy_config)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| | policy_map[name] = cls(obs_space, act_space, merged_conf)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| | DynamicTFPolicy.__init__(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| | self._initialize_loss()
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| | postprocessed_batch = self.postprocess_trajectory(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| | return postprocess_fn(self, sample_batch, other_agent_batches,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| | last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 84, in call
|| | symbolic_out[0] = fn(*placeholders)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 235, in value
|| | model_out, _ = self.model({
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/modelv2.py", line 150, in __call__
|| | res = self.forward(restored, state or [], seq_lens)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py", line 103, in forward
|| | model_out, self._value_out = self.base_model(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| | outputs = call_fn(cast_inputs, *args, **kwargs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 385, in call
|| | return self._run_internal_graph(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 508, in _run_internal_graph
|| | outputs = node.layer(*args, **kwargs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| | outputs = call_fn(cast_inputs, *args, **kwargs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/layers/convolutional.py", line 269, in call
|| | return self.activation(outputs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 10435, in relu
|| | _, _, _op, _outputs = _op_def_library._apply_op_helper(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/op_def_library.py", line 742, in _apply_op_helper
|| | op = g._create_op_internal(op_type_name, inputs, dtypes=None,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 3477, in _create_op_internal
|| | ret = Operation(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1949, in __init__
|| | self._traceback = tf_stack.extract_stack()
|| |
|| |
||
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 249, in main
raise InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Getting agent protocol
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
InvalidSubmission:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
robot_ci.write_topic_and_expect_zero("seed", config.seed)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego2" aborted with the following error:
error in ego2 |Unexpected error:
|
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
|| return fn(*args)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
|| return self._call_tf_sessionrun(options, feed_dict, fetch_list,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
|| return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
|| tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| [[default_policy/strided_slice_1/_3]]
|| 0 successful operations.
|| 0 derived errors ignored.
||
|| During handling of the above exception, another exception occurred:
||
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 29, in init
|| self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| File "/submission/model.py", line 55, in __init__
|| self.model = PPOTrainer(config=config["rllib_config"])
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| Trainer.__init__(self, config, env, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| super().__init__(config, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| self._setup(copy.deepcopy(self.config))
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| self._init(self.config, self.env_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| self.workers = self._make_workers(env_creator, self._policy,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| return WorkerSet(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| self._local_worker = self._make_worker(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| worker = cls(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| self._build_policy_map(policy_dict, policy_config)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| policy_map[name] = cls(obs_space, act_space, merged_conf)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| DynamicTFPolicy.__init__(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| self._initialize_loss()
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| postprocessed_batch = self.postprocess_trajectory(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| return postprocess_fn(self, sample_batch, other_agent_batches,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 86, in call
|| return session_or_none.run(symbolic_out[0], feed_dict)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 957, in run
|| result = self._run(None, fetches, feed_dict, options_ptr,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1180, in _run
|| results = self._do_run(handle, final_targets, final_fetches,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1358, in _do_run
|| return self._do_call(_run_fn, feeds, fetches, targets, options,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1384, in _do_call
|| raise type(e)(node_def, op, message)
|| tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| [[default_policy/strided_slice_1/_3]]
|| 0 successful operations.
|| 0 derived errors ignored.
||
|| Original stack trace for 'default_policy/functional_1_1/conv_value_1/Relu':
|| File "solution.py", line 127, in <module>
|| main()
|| File "solution.py", line 123, in main
|| wrap_direct(node=node, protocol=protocol)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/interface.py", line 24, in wrap_direct
|| run_loop(node, protocol, args)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 243, in run_loop
|| loop(node_name, fi, fo, node, protocol, tin, tout, config=config, fi_desc=fin, fo_desc=fout)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 29, in init
|| self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| File "/submission/model.py", line 55, in __init__
|| self.model = PPOTrainer(config=config["rllib_config"])
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| Trainer.__init__(self, config, env, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| super().__init__(config, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| self._setup(copy.deepcopy(self.config))
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| self._init(self.config, self.env_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| self.workers = self._make_workers(env_creator, self._policy,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| return WorkerSet(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| self._local_worker = self._make_worker(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| worker = cls(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| self._build_policy_map(policy_dict, policy_config)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| policy_map[name] = cls(obs_space, act_space, merged_conf)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| DynamicTFPolicy.__init__(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| self._initialize_loss()
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| postprocessed_batch = self.postprocess_trajectory(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| return postprocess_fn(self, sample_batch, other_agent_batches,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 84, in call
|| symbolic_out[0] = fn(*placeholders)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 235, in value
|| model_out, _ = self.model({
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/modelv2.py", line 150, in __call__
|| res = self.forward(restored, state or [], seq_lens)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py", line 103, in forward
|| model_out, self._value_out = self.base_model(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| outputs = call_fn(cast_inputs, *args, **kwargs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 385, in call
|| return self._run_internal_graph(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 508, in _run_internal_graph
|| outputs = node.layer(*args, **kwargs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| outputs = call_fn(cast_inputs, *args, **kwargs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/layers/convolutional.py", line 269, in call
|| return self.activation(outputs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 10435, in relu
|| _, _, _op, _outputs = _op_def_library._apply_op_helper(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/op_def_library.py", line 742, in _apply_op_helper
|| op = g._create_op_internal(op_type_name, inputs, dtypes=None,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 3477, in _create_op_internal
|| ret = Operation(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1949, in __init__
|| self._traceback = tf_stack.extract_stack()
||
||
|| The above exception was the direct cause of the following exception:
||
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 339, in loop
|| raise Exception(msg) from e
|| Exception: Exception while calling the node's init() function.
||
|| | Traceback (most recent call last):
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
|| | return fn(*args)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
|| | return self._call_tf_sessionrun(options, feed_dict, fetch_list,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
|| | return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
|| | tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| | (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| | (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| | [[default_policy/strided_slice_1/_3]]
|| | 0 successful operations.
|| | 0 derived errors ignored.
|| |
|| | During handling of the above exception, another exception occurred:
|| |
|| | Traceback (most recent call last):
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| | call_if_fun_exists(node, "init", context=context_data)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| | f(**kwargs)
|| | File "solution.py", line 29, in init
|| | self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| | File "/submission/model.py", line 55, in __init__
|| | self.model = PPOTrainer(config=config["rllib_config"])
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| | Trainer.__init__(self, config, env, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| | super().__init__(config, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| | self._setup(copy.deepcopy(self.config))
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| | self._init(self.config, self.env_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| | self.workers = self._make_workers(env_creator, self._policy,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| | return WorkerSet(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| | self._local_worker = self._make_worker(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| | worker = cls(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| | self._build_policy_map(policy_dict, policy_config)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| | policy_map[name] = cls(obs_space, act_space, merged_conf)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| | DynamicTFPolicy.__init__(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| | self._initialize_loss()
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| | postprocessed_batch = self.postprocess_trajectory(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| | return postprocess_fn(self, sample_batch, other_agent_batches,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| | last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 86, in call
|| | return session_or_none.run(symbolic_out[0], feed_dict)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 957, in run
|| | result = self._run(None, fetches, feed_dict, options_ptr,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1180, in _run
|| | results = self._do_run(handle, final_targets, final_fetches,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1358, in _do_run
|| | return self._do_call(_run_fn, feeds, fetches, targets, options,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1384, in _do_call
|| | raise type(e)(node_def, op, message)
|| | tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| | (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| | (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| | [[default_policy/strided_slice_1/_3]]
|| | 0 successful operations.
|| | 0 derived errors ignored.
|| |
|| | Original stack trace for 'default_policy/functional_1_1/conv_value_1/Relu':
|| | File "solution.py", line 127, in <module>
|| | main()
|| | File "solution.py", line 123, in main
|| | wrap_direct(node=node, protocol=protocol)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/interface.py", line 24, in wrap_direct
|| | run_loop(node, protocol, args)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 243, in run_loop
|| | loop(node_name, fi, fo, node, protocol, tin, tout, config=config, fi_desc=fin, fo_desc=fout)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| | call_if_fun_exists(node, "init", context=context_data)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| | f(**kwargs)
|| | File "solution.py", line 29, in init
|| | self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| | File "/submission/model.py", line 55, in __init__
|| | self.model = PPOTrainer(config=config["rllib_config"])
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| | Trainer.__init__(self, config, env, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| | super().__init__(config, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| | self._setup(copy.deepcopy(self.config))
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| | self._init(self.config, self.env_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| | self.workers = self._make_workers(env_creator, self._policy,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| | return WorkerSet(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| | self._local_worker = self._make_worker(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| | worker = cls(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| | self._build_policy_map(policy_dict, policy_config)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| | policy_map[name] = cls(obs_space, act_space, merged_conf)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| | DynamicTFPolicy.__init__(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| | self._initialize_loss()
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| | postprocessed_batch = self.postprocess_trajectory(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| | return postprocess_fn(self, sample_batch, other_agent_batches,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| | last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 84, in call
|| | symbolic_out[0] = fn(*placeholders)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 235, in value
|| | model_out, _ = self.model({
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/modelv2.py", line 150, in __call__
|| | res = self.forward(restored, state or [], seq_lens)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py", line 103, in forward
|| | model_out, self._value_out = self.base_model(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| | outputs = call_fn(cast_inputs, *args, **kwargs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 385, in call
|| | return self._run_internal_graph(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 508, in _run_internal_graph
|| | outputs = node.layer(*args, **kwargs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| | outputs = call_fn(cast_inputs, *args, **kwargs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/layers/convolutional.py", line 269, in call
|| | return self.activation(outputs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 10435, in relu
|| | _, _, _op, _outputs = _op_def_library._apply_op_helper(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/op_def_library.py", line 742, in _apply_op_helper
|| | op = g._create_op_internal(op_type_name, inputs, dtypes=None,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 3477, in _create_op_internal
|| | ret = Operation(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1949, in __init__
|| | self._traceback = tf_stack.extract_stack()
|| |
|| |
||
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 249, in main
raise InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Getting agent protocol
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
InvalidSubmission:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
robot_ci.write_topic_and_expect_zero("seed", config.seed)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego2" aborted with the following error:
error in ego2 |Unexpected error:
|
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
|| return fn(*args)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
|| return self._call_tf_sessionrun(options, feed_dict, fetch_list,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
|| return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
|| tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| [[default_policy/strided_slice_1/_3]]
|| 0 successful operations.
|| 0 derived errors ignored.
||
|| During handling of the above exception, another exception occurred:
||
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 29, in init
|| self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| File "/submission/model.py", line 55, in __init__
|| self.model = PPOTrainer(config=config["rllib_config"])
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| Trainer.__init__(self, config, env, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| super().__init__(config, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| self._setup(copy.deepcopy(self.config))
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| self._init(self.config, self.env_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| self.workers = self._make_workers(env_creator, self._policy,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| return WorkerSet(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| self._local_worker = self._make_worker(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| worker = cls(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| self._build_policy_map(policy_dict, policy_config)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| policy_map[name] = cls(obs_space, act_space, merged_conf)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| DynamicTFPolicy.__init__(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| self._initialize_loss()
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| postprocessed_batch = self.postprocess_trajectory(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| return postprocess_fn(self, sample_batch, other_agent_batches,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 86, in call
|| return session_or_none.run(symbolic_out[0], feed_dict)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 957, in run
|| result = self._run(None, fetches, feed_dict, options_ptr,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1180, in _run
|| results = self._do_run(handle, final_targets, final_fetches,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1358, in _do_run
|| return self._do_call(_run_fn, feeds, fetches, targets, options,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1384, in _do_call
|| raise type(e)(node_def, op, message)
|| tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| [[default_policy/strided_slice_1/_3]]
|| 0 successful operations.
|| 0 derived errors ignored.
||
|| Original stack trace for 'default_policy/functional_1_1/conv_value_1/Relu':
|| File "solution.py", line 127, in <module>
|| main()
|| File "solution.py", line 123, in main
|| wrap_direct(node=node, protocol=protocol)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/interface.py", line 24, in wrap_direct
|| run_loop(node, protocol, args)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 243, in run_loop
|| loop(node_name, fi, fo, node, protocol, tin, tout, config=config, fi_desc=fin, fo_desc=fout)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 29, in init
|| self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| File "/submission/model.py", line 55, in __init__
|| self.model = PPOTrainer(config=config["rllib_config"])
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| Trainer.__init__(self, config, env, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| super().__init__(config, logger_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| self._setup(copy.deepcopy(self.config))
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| self._init(self.config, self.env_creator)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| self.workers = self._make_workers(env_creator, self._policy,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| return WorkerSet(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| self._local_worker = self._make_worker(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| worker = cls(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| self._build_policy_map(policy_dict, policy_config)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| policy_map[name] = cls(obs_space, act_space, merged_conf)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| DynamicTFPolicy.__init__(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| self._initialize_loss()
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| postprocessed_batch = self.postprocess_trajectory(
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| return postprocess_fn(self, sample_batch, other_agent_batches,
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 84, in call
|| symbolic_out[0] = fn(*placeholders)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 235, in value
|| model_out, _ = self.model({
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/modelv2.py", line 150, in __call__
|| res = self.forward(restored, state or [], seq_lens)
|| File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py", line 103, in forward
|| model_out, self._value_out = self.base_model(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| outputs = call_fn(cast_inputs, *args, **kwargs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 385, in call
|| return self._run_internal_graph(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 508, in _run_internal_graph
|| outputs = node.layer(*args, **kwargs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| outputs = call_fn(cast_inputs, *args, **kwargs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/layers/convolutional.py", line 269, in call
|| return self.activation(outputs)
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 10435, in relu
|| _, _, _op, _outputs = _op_def_library._apply_op_helper(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/op_def_library.py", line 742, in _apply_op_helper
|| op = g._create_op_internal(op_type_name, inputs, dtypes=None,
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 3477, in _create_op_internal
|| ret = Operation(
|| File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1949, in __init__
|| self._traceback = tf_stack.extract_stack()
||
||
|| The above exception was the direct cause of the following exception:
||
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 339, in loop
|| raise Exception(msg) from e
|| Exception: Exception while calling the node's init() function.
||
|| | Traceback (most recent call last):
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
|| | return fn(*args)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
|| | return self._call_tf_sessionrun(options, feed_dict, fetch_list,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
|| | return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
|| | tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| | (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| | (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[{{node default_policy/functional_1_1/conv_value_1/Relu}}]]
|| | [[default_policy/strided_slice_1/_3]]
|| | 0 successful operations.
|| | 0 derived errors ignored.
|| |
|| | During handling of the above exception, another exception occurred:
|| |
|| | Traceback (most recent call last):
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| | call_if_fun_exists(node, "init", context=context_data)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| | f(**kwargs)
|| | File "solution.py", line 29, in init
|| | self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| | File "/submission/model.py", line 55, in __init__
|| | self.model = PPOTrainer(config=config["rllib_config"])
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| | Trainer.__init__(self, config, env, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| | super().__init__(config, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| | self._setup(copy.deepcopy(self.config))
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| | self._init(self.config, self.env_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| | self.workers = self._make_workers(env_creator, self._policy,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| | return WorkerSet(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| | self._local_worker = self._make_worker(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| | worker = cls(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| | self._build_policy_map(policy_dict, policy_config)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| | policy_map[name] = cls(obs_space, act_space, merged_conf)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| | DynamicTFPolicy.__init__(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| | self._initialize_loss()
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| | postprocessed_batch = self.postprocess_trajectory(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| | return postprocess_fn(self, sample_batch, other_agent_batches,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| | last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 86, in call
|| | return session_or_none.run(symbolic_out[0], feed_dict)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 957, in run
|| | result = self._run(None, fetches, feed_dict, options_ptr,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1180, in _run
|| | results = self._do_run(handle, final_targets, final_fetches,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1358, in _do_run
|| | return self._do_call(_run_fn, feeds, fetches, targets, options,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1384, in _do_call
|| | raise type(e)(node_def, op, message)
|| | tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
|| | (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| | (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
|| | [[node default_policy/functional_1_1/conv_value_1/Relu (defined at /usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py:103) ]]
|| | [[default_policy/strided_slice_1/_3]]
|| | 0 successful operations.
|| | 0 derived errors ignored.
|| |
|| | Original stack trace for 'default_policy/functional_1_1/conv_value_1/Relu':
|| | File "solution.py", line 127, in <module>
|| | main()
|| | File "solution.py", line 123, in main
|| | wrap_direct(node=node, protocol=protocol)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/interface.py", line 24, in wrap_direct
|| | run_loop(node, protocol, args)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 243, in run_loop
|| | loop(node_name, fi, fo, node, protocol, tin, tout, config=config, fi_desc=fin, fo_desc=fout)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| | call_if_fun_exists(node, "init", context=context_data)
|| | File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| | f(**kwargs)
|| | File "solution.py", line 29, in init
|| | self.model = RLlibModel(SEED,experiment_idx=0,checkpoint_idx=0,logger=context)
|| | File "/submission/model.py", line 55, in __init__
|| | self.model = PPOTrainer(config=config["rllib_config"])
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
|| | Trainer.__init__(self, config, env, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 455, in __init__
|| | super().__init__(config, logger_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/tune/trainable.py", line 174, in __init__
|| | self._setup(copy.deepcopy(self.config))
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 596, in _setup
|| | self._init(self.config, self.env_creator)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer_template.py", line 115, in _init
|| | self.workers = self._make_workers(env_creator, self._policy,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 662, in _make_workers
|| | return WorkerSet(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 61, in __init__
|| | self._local_worker = self._make_worker(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/worker_set.py", line 237, in _make_worker
|| | worker = cls(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 360, in __init__
|| | self._build_policy_map(policy_dict, policy_config)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 842, in _build_policy_map
|| | policy_map[name] = cls(obs_space, act_space, merged_conf)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 129, in __init__
|| | DynamicTFPolicy.__init__(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 237, in __init__
|| | self._initialize_loss()
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/dynamic_tf_policy.py", line 324, in _initialize_loss
|| | postprocessed_batch = self.postprocess_trajectory(
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy_template.py", line 155, in postprocess_trajectory
|| | return postprocess_fn(self, sample_batch, other_agent_batches,
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 182, in postprocess_ppo_gae
|| | last_r = policy._value(sample_batch[SampleBatch.NEXT_OBS][-1],
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_ops.py", line 84, in call
|| | symbolic_out[0] = fn(*placeholders)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/ppo/ppo_tf_policy.py", line 235, in value
|| | model_out, _ = self.model({
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/modelv2.py", line 150, in __call__
|| | res = self.forward(restored, state or [], seq_lens)
|| | File "/usr/local/lib/python3.8/dist-packages/ray/rllib/models/tf/visionnet_v2.py", line 103, in forward
|| | model_out, self._value_out = self.base_model(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| | outputs = call_fn(cast_inputs, *args, **kwargs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 385, in call
|| | return self._run_internal_graph(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/functional.py", line 508, in _run_internal_graph
|| | outputs = node.layer(*args, **kwargs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 776, in __call__
|| | outputs = call_fn(cast_inputs, *args, **kwargs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/layers/convolutional.py", line 269, in call
|| | return self.activation(outputs)
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 10435, in relu
|| | _, _, _op, _outputs = _op_def_library._apply_op_helper(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/op_def_library.py", line 742, in _apply_op_helper
|| | op = g._create_op_internal(op_type_name, inputs, dtypes=None,
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 3477, in _create_op_internal
|| | ret = Operation(
|| | File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1949, in __init__
|| | self._traceback = tf_stack.extract_stack()
|| |
|| |
||
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 249, in main
raise InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Getting agent protocol
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
InvalidSubmission:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 190, in main
raise InvalidSubmission(msg)
duckietown_challenges.exceptions.InvalidSubmission: Timeout during connection to ego0: <SignalTimeout in state: 2>
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
InvalidSubmission:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 190, in main
raise InvalidSubmission(msg)
duckietown_challenges.exceptions.InvalidSubmission: Timeout during connection to ego0: <SignalTimeout in state: 2>
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
InvalidSubmission:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 190, in main
raise InvalidSubmission(msg)
duckietown_challenges.exceptions.InvalidSubmission: Timeout during connection to ego0: <SignalTimeout in state: 2>
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
InvalidSubmission:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 563, in run_episode
r: MsgReceived = await loop.run_in_executor(executor, f)
File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 135, in write_topic_and_expect
ob: MsgReceived = self.read_one(expect_topic=expect, timeout=timeout)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 218, in read_one
msgs = read_reply(self.fpout, timeout=timeout, waiting_for=waiting_for, nickname=self.nickname,)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 304, in read_reply
others = read_until_over(fpout, timeout=timeout, nickname=nickname)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 331, in read_until_over
raise RemoteNodeAborted(m)
zuper_nodes.structures.RemoteNodeAborted: External node "ego0" aborted:
error in ego0 |Unexpected error:
|
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 355, in loop
|| handle_message_node(parsed, receiver0, context0)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 531, in handle_message_node
|| call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 90, in on_received_get_commands
|| pwm_left, pwm_right = self.compute_action(self.current_image)
|| File "solution.py", line 84, in compute_action
|| action = self.model.predict(observation)
|| File "/submission/model.py", line 71, in predict
|| observation = wrapper.observation(observation)
|| File "/submission/duckietown_utils/wrappers/SegmentObsWrapper.py", line 56, in observation
|| _, pred = torch.max(self.model.forward(obs_prep), 1)
|| File "/submission/duckietown_utils/wrappers/my_models/tiramisu.py", line 145, in forward
|| x = self.featureExtractor(x)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
|| result = self.forward(*input, **kwargs)
|| File "/submission/duckietown_utils/wrappers/my_models/tiramisu.py", line 102, in forward
|| out = self.denseBlocksUp[i](out)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
|| result = self.forward(*input, **kwargs)
|| File "/submission/duckietown_utils/wrappers/my_models/layers.py", line 38, in forward
|| out = layer(x)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
|| result = self.forward(*input, **kwargs)
|| File "/submission/duckietown_utils/wrappers/my_models/layers.py", line 15, in forward
|| return super().forward(x)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/container.py", line 117, in forward
|| input = module(input)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
|| result = self.forward(*input, **kwargs)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/batchnorm.py", line 131, in forward
|| return F.batch_norm(
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/functional.py", line 2056, in batch_norm
|| return torch.batch_norm(
|| RuntimeError: CUDA out of memory. Tried to allocate 14.00 MiB (GPU 0; 10.76 GiB total capacity; 280.34 MiB already allocated; 21.12 MiB free; 340.00 MiB reserved in total by PyTorch)
||
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 286, in main
length_s = await run_episode(
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 567, in run_episode
raise dc.InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Trouble with communication to the agent.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
InvalidEnvironment:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
robot_ci.write_topic_and_expect_zero("seed", config.seed)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego2" aborted with the following error:
error in ego2 |Unexpected error:
|
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 29, in init
|| self.model = RLlibModel(0, experiment_idx=0, checkpoint_idx=0, logger=context)
|| File "/submission/model.py", line 42, in __init__
|| dummy_env = wrap_env(config["env_config"], extra_config={
|| File "/submission/duckietown_utils/env.py", line 46, in wrap_env
|| env = SegmentObsWrapper(env, model=extra_config['model'])
|| File "/submission/duckietown_utils/wrappers/SegmentObsWrapper.py", line 43, in __init__
|| self.model.cuda()
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in cuda
|| return self._apply(lambda t: t.cuda(device))
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
|| module._apply(fn)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
|| module._apply(fn)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 381, in _apply
|| param_applied = fn(param)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in <lambda>
|| return self._apply(lambda t: t.cuda(device))
|| RuntimeError: CUDA error: out of memory
||
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
robot_ci.write_topic_and_expect_zero("seed", config.seed)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego2" aborted with the following error:
error in ego2 |Unexpected error:
|
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 29, in init
|| self.model = RLlibModel(0, experiment_idx=0, checkpoint_idx=0, logger=context)
|| File "/submission/model.py", line 42, in __init__
|| dummy_env = wrap_env(config["env_config"], extra_config={
|| File "/submission/duckietown_utils/env.py", line 46, in wrap_env
|| env = SegmentObsWrapper(env, model=extra_config['model'])
|| File "/submission/duckietown_utils/wrappers/SegmentObsWrapper.py", line 43, in __init__
|| self.model.cuda()
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in cuda
|| return self._apply(lambda t: t.cuda(device))
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
|| module._apply(fn)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
|| module._apply(fn)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 381, in _apply
|| param_applied = fn(param)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in <lambda>
|| return self._apply(lambda t: t.cuda(device))
|| RuntimeError: CUDA error: out of memory
||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
InvalidEnvironment:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
robot_ci.write_topic_and_expect_zero("seed", config.seed)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego3" aborted with the following error:
error in ego3 |Unexpected error:
|
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 29, in init
|| self.model = RLlibModel(0, experiment_idx=0, checkpoint_idx=0, logger=context)
|| File "/submission/model.py", line 42, in __init__
|| dummy_env = wrap_env(config["env_config"], extra_config={
|| File "/submission/duckietown_utils/env.py", line 46, in wrap_env
|| env = SegmentObsWrapper(env, model=extra_config['model'])
|| File "/submission/duckietown_utils/wrappers/SegmentObsWrapper.py", line 43, in __init__
|| self.model.cuda()
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in cuda
|| return self._apply(lambda t: t.cuda(device))
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
|| module._apply(fn)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
|| module._apply(fn)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 381, in _apply
|| param_applied = fn(param)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in <lambda>
|| return self._apply(lambda t: t.cuda(device))
|| RuntimeError: CUDA error: out of memory
||
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
robot_ci.write_topic_and_expect_zero("seed", config.seed)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego3" aborted with the following error:
error in ego3 |Unexpected error:
|
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 29, in init
|| self.model = RLlibModel(0, experiment_idx=0, checkpoint_idx=0, logger=context)
|| File "/submission/model.py", line 42, in __init__
|| dummy_env = wrap_env(config["env_config"], extra_config={
|| File "/submission/duckietown_utils/env.py", line 46, in wrap_env
|| env = SegmentObsWrapper(env, model=extra_config['model'])
|| File "/submission/duckietown_utils/wrappers/SegmentObsWrapper.py", line 43, in __init__
|| self.model.cuda()
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in cuda
|| return self._apply(lambda t: t.cuda(device))
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
|| module._apply(fn)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
|| module._apply(fn)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 381, in _apply
|| param_applied = fn(param)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in <lambda>
|| return self._apply(lambda t: t.cuda(device))
|| RuntimeError: CUDA error: out of memory
||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
InvalidEnvironment:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
robot_ci.write_topic_and_expect_zero("seed", config.seed)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego2" aborted with the following error:
error in ego2 |Unexpected error:
|
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 29, in init
|| self.model = RLlibModel(0, experiment_idx=0, checkpoint_idx=0, logger=context)
|| File "/submission/model.py", line 42, in __init__
|| dummy_env = wrap_env(config["env_config"], extra_config={
|| File "/submission/duckietown_utils/env.py", line 46, in wrap_env
|| env = SegmentObsWrapper(env, model=extra_config['model'])
|| File "/submission/duckietown_utils/wrappers/SegmentObsWrapper.py", line 43, in __init__
|| self.model.cuda()
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in cuda
|| return self._apply(lambda t: t.cuda(device))
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
|| module._apply(fn)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
|| module._apply(fn)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 381, in _apply
|| param_applied = fn(param)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in <lambda>
|| return self._apply(lambda t: t.cuda(device))
|| RuntimeError: CUDA error: out of memory
||
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
robot_ci.write_topic_and_expect_zero("seed", config.seed)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego2" aborted with the following error:
error in ego2 |Unexpected error:
|
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "solution.py", line 29, in init
|| self.model = RLlibModel(0, experiment_idx=0, checkpoint_idx=0, logger=context)
|| File "/submission/model.py", line 42, in __init__
|| dummy_env = wrap_env(config["env_config"], extra_config={
|| File "/submission/duckietown_utils/env.py", line 46, in wrap_env
|| env = SegmentObsWrapper(env, model=extra_config['model'])
|| File "/submission/duckietown_utils/wrappers/SegmentObsWrapper.py", line 43, in __init__
|| self.model.cuda()
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in cuda
|| return self._apply(lambda t: t.cuda(device))
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
|| module._apply(fn)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
|| module._apply(fn)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 381, in _apply
|| param_applied = fn(param)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in <lambda>
|| return self._apply(lambda t: t.cuda(device))
|| RuntimeError: CUDA error: out of memory
||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
InvalidSubmission:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 190, in main
raise InvalidSubmission(msg)
duckietown_challenges.exceptions.InvalidSubmission: Timeout during connection to ego0: <SignalTimeout in state: 2>
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
InvalidEnvironment:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
robot_ci.write_topic_and_expect_zero("seed", config.seed)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego3" aborted with the following error:
error in ego3 |Unexpected error:
|
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "submission.py", line 60, in init
|| self.agent = self.create_agent(self.env)
|| File "submission.py", line 162, in create_agent_dagger
|| agent = DaggerAgent.load_from_checkpoint(
|| File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/utilities/device_dtype_mixin.py", line 124, in cuda
|| return super().cuda(device=device)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in cuda
|| return self._apply(lambda t: t.cuda(device))
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
|| module._apply(fn)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
|| module._apply(fn)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
|| module._apply(fn)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 381, in _apply
|| param_applied = fn(param)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in <lambda>
|| return self._apply(lambda t: t.cuda(device))
|| RuntimeError: CUDA error: out of memory
||
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
yield cie
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
wrap(cie)
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
asyncio.run(main(cie, logdir, attempts), debug=True)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
robot_ci.write_topic_and_expect_zero("seed", config.seed)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego3" aborted with the following error:
error in ego3 |Unexpected error:
|
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
|| call_if_fun_exists(node, "init", context=context_data)
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|| f(**kwargs)
|| File "submission.py", line 60, in init
|| self.agent = self.create_agent(self.env)
|| File "submission.py", line 162, in create_agent_dagger
|| agent = DaggerAgent.load_from_checkpoint(
|| File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/utilities/device_dtype_mixin.py", line 124, in cuda
|| return super().cuda(device=device)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in cuda
|| return self._apply(lambda t: t.cuda(device))
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
|| module._apply(fn)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
|| module._apply(fn)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
|| module._apply(fn)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 381, in _apply
|| param_applied = fn(param)
|| File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in <lambda>
|| return self._apply(lambda t: t.cuda(device))
|| RuntimeError: CUDA error: out of memory
||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
YU CHEN
Victor Adolfo Romero Cano 🇨🇴
Lakshwin Shreesha M K
Akshat Marathe
Jason Waddle
Sampsa Ranta
Frank (Chude) Qian 🇨🇦
Vinsensius Edwin
Andrea Censi 🇨🇭
Patricio Cortes Estay
Saeed Momenzadeh
Pablo Martinez
Sai Prasanna Raman
Randy Moore
Patrick Geneva
Marek Dlugosz
Jay Bromley
luca bonello
Mustafa Kaan Calik
Ashley Reid
Jonathan Jennings
Peder Sefland
Anil Kumar Chavali
Ivan Danov
Roger Miranda Colorado
Thomas Steinle
Maria-Elena Algorri
Daniel Stemmermann
Raju Tadisetti
Marin Dantchev
Nicholas Kostelnik
Gavin Kane
Rene Lobsiger
András Kalapos 🇭🇺
Márton Tim 🇭🇺
Andras Beres
Jean-Sébastien Grondin 🇨🇦
Bea Baselines 🐤
Samuel Alexander
Raphael Jean
Kevin Huang