| message | ==> Entrypoint
Netwo [...]==> Entrypoint
Network configured successfully.
INFO: ROBOT_TYPE is externally set to 'duckiebot'.
<== Entrypoint
DEBUG:commons:version: 6.2.4 *
DEBUG:typing:version: 6.2.3
DEBUG:duckietown_world:duckietown-world version 6.2.38 path /usr/local/lib/python3.8/dist-packages
DEBUG:geometry:PyGeometry-z6 version 2.1.4 path /usr/local/lib/python3.8/dist-packages
DEBUG:aido_schemas:aido-protocols version 6.0.59 path /usr/local/lib/python3.8/dist-packages
DEBUG:nodes:version 6.2.13 path /usr/local/lib/python3.8/dist-packages pyparsing 2.4.6
DEBUG:gym-duckietown:gym-duckietown version 6.1.30 path /usr/local/lib/python3.8/dist-packages
DEBUG:ipce:version 6.1.1 path /usr/local/lib/python3.8/dist-packages
DEBUG:nodes_wrapper:checking implementation
DEBUG:nodes_wrapper:checking implementation OK
DEBUG:nodes_wrapper.PytorchRLTemplateAgent:run_loop
fin: /fifos/ego0-in
fout: fifo:/fifos/ego0-out
DEBUG:nodes_wrapper:Fifo /fifos/ego0-out created. I will block until a reader appears.
DEBUG:nodes_wrapper:Fifo reader appeared for /fifos/ego0-out.
DEBUG:nodes_wrapper.PytorchRLTemplateAgent:Starting reading
fi_desc: /fifos/ego0-in
fo_desc: fifo:/fifos/ego0-out
INFO:nodes_wrapper.PytorchRLTemplateAgent.data:6c869ce4d47c:PytorchRLTemplateAgent: torch.cuda.is_available = True AIDO_REQUIRE_GPU = None
INFO:nodes_wrapper.PytorchRLTemplateAgent.data:6c869ce4d47c:PytorchRLTemplateAgent: init()
INFO:nodes_wrapper.PytorchRLTemplateAgent.data:6c869ce4d47c:PytorchRLTemplateAgent: device 0 of 1; name = 'GeForce RTX 2080 Ti'
INFO:aido_schemas:PytorchRLTemplateAgent init
2022-03-16 14:56:26,927 WARNING deprecation.py:38 -- DeprecationWarning: `monitor` has been deprecated. Use `record_env` instead. This will raise an error in the future!
2022-03-16 14:56:26,927 WARNING ppo.py:143 -- `train_batch_size` (128) cannot be achieved with your other settings (num_workers=1 num_envs_per_worker=1 rollout_fragment_length=200)! Auto-adjusting `rollout_fragment_length` to 128.
2022-03-16 14:56:27,236 WARNING services.py:1748 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 67096576 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing '--shm-size=10.24gb' to 'docker run' (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM.
(pid=368) DEBUG:commons:version: 6.2.4 *
(pid=368) DEBUG:typing:version: 6.2.3
(pid=368) DEBUG:duckietown_world:duckietown-world version 6.2.38 path /usr/local/lib/python3.8/dist-packages
(pid=368) DEBUG:geometry:PyGeometry-z6 version 2.1.4 path /usr/local/lib/python3.8/dist-packages
(pid=368) DEBUG:aido_schemas:aido-protocols version 6.0.59 path /usr/local/lib/python3.8/dist-packages
(pid=368) DEBUG:nodes:version 6.2.13 path /usr/local/lib/python3.8/dist-packages pyparsing 2.4.6
(pid=368) DEBUG:gym-duckietown:gym-duckietown version 6.1.30 path /usr/local/lib/python3.8/dist-packages
(pid=368)
(pid=368) WARNING:wrappers.general_wrappers:Dummy Duckietown Gym reset() called!
(pid=368) 2022-03-16 14:56:31,543 WARNING deprecation.py:38 -- DeprecationWarning: `SampleBatch['is_training']` has been deprecated. Use `SampleBatch.is_training` instead. This will raise an error in the future!
2022-03-16 14:56:33,348 WARNING deprecation.py:38 -- DeprecationWarning: `SampleBatch['is_training']` has been deprecated. Use `SampleBatch.is_training` instead. This will raise an error in the future!
2022-03-16 14:56:33,366 WARNING trainer_template.py:185 -- `execution_plan` functions should accept `trainer`, `workers`, and `config` as args!
2022-03-16 14:56:33,367 WARNING deprecation.py:38 -- DeprecationWarning: `monitor` has been deprecated. Use `record_env` instead. This will raise an error in the future!
2022-03-16 14:56:33,367 WARNING deprecation.py:38 -- DeprecationWarning: `simple_optimizer` has been deprecated. This will raise an error in the future!
WARNING:wrappers.general_wrappers:Dummy Duckietown Gym reset() called!
/usr/local/lib/python3.8/dist-packages/gym/wrappers/monitor.py:86: UserWarning: WARN: Trying to monitor an environment which has no 'spec' set. This usually means you did not create it via 'gym.make', and is recommended only for advanced users.
logger.warn(
2022-03-16 14:56:33,409 WARNING util.py:57 -- Install gputil for GPU system monitoring.
2022-03-16 14:56:33,436 INFO trainable.py:416 -- Restored on 172.17.0.3 from checkpoint: ./tf_models/checkpoint-391
2022-03-16 14:56:33,436 INFO trainable.py:424 -- Current state after restoring: {'_iteration': 391, '_timesteps_total': 0, '_time_total': 23444.33237552643, '_episodes_total': 2069}
INFO:aido_schemas:PytorchRLTemplateAgent init complete
INFO:nodes_wrapper.PytorchRLTemplateAgent.data:6c869ce4d47c:PytorchRLTemplateAgent: Starting episode "episode".
WARNING:nodes_wrapper:Input channel not ready after 9.6 seconds. Will re-try.
> Expecting control message or one of: {ExpectInputReceived(channel='episode_start'), ExpectInputReceived(channel='seed')}
I will warn again in 4.5 seconds.
WARNING:nodes_wrapper:Input channel not ready after 14.1 seconds. Will re-try.
> Expecting control message or one of: {ExpectInputReceived(channel='episode_start'), ExpectInputReceived(channel='seed')}
I will warn again in 6.8 seconds.
WARNING:nodes_wrapper:Input channel not ready after 20.9 seconds. Will re-try.
> Expecting control message or one of: {ExpectInputReceived(channel='episode_start'), ExpectInputReceived(channel='seed')}
I will warn again in 10.1 seconds.
WARNING:nodes_wrapper:Input channel not ready after 31.0 seconds. Will re-try.
> Expecting control message or one of: {ExpectInputReceived(channel='episode_start'), ExpectInputReceived(channel='seed')}
I will warn again in 15.2 seconds.
WARNING:nodes_wrapper:Input channel not ready after 46.2 seconds. Will re-try.
> Expecting control message or one of: {ExpectInputReceived(channel='episode_start'), ExpectInputReceived(channel='seed')}
I will warn again in 22.8 seconds.
*** SIGTERM received at time=1647442655 on cpu 4 ***
PC: @ 0x7f30fe62312b (unknown) select
@ 0x7f30fe551210 1316764208 (unknown)
@ 0x4 (unknown) (unknown)
@ 0x915620 (unknown) (unknown)
[2022-03-16 14:57:35,665 E 1 1] logging.cc:313: *** SIGTERM received at time=1647442655 on cpu 4 ***
[2022-03-16 14:57:35,665 E 1 1] logging.cc:313: PC: @ 0x7f30fe62312b (unknown) select
[2022-03-16 14:57:35,668 E 1 1] logging.cc:313: @ 0x7f30fe551210 1316764208 (unknown)
[2022-03-16 14:57:35,672 E 1 1] logging.cc:313: @ 0x4 (unknown) (unknown)
[2022-03-16 14:57:35,677 E 1 1] logging.cc:313: @ 0x915620 (unknown) (unknown)
ERROR:nodes_wrapper.PytorchRLTemplateAgent:Error in node PytorchRLTemplateAgent
ET: InternalProblem
tb: |Traceback (most recent call last):
| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 356, in loop
| for parsed in inputs(fi, waiting_for=waiting_for):
| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/reading.py", line 21, in inputs
| readyr, readyw, readyx = select.select([f], [], [f], intermediate_timeout)
| File "/usr/local/lib/python3.8/dist-packages/ray/worker.py", line 1027, in sigterm_handler
| sys.exit(signum)
|SystemExit: 15
|
|The above exception was the direct cause of the following exception:
|
|Traceback (most recent call last):
| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 287, in run_loop
| loop(my_logger, node_name, fi, fo, node, protocol, tin, tout, config=config, fi_desc=fin,
| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 480, in loop
| raise InternalProblem(msg) from e # XXX
|zuper_nodes.structures.InternalProblem: Unexpected error:
|
|| Traceback (most recent call last):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 356, in loop
|| for parsed in inputs(fi, waiting_for=waiting_for):
|| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/reading.py", line 21, in inputs
|| readyr, readyw, readyx = select.select([f], [], [f], intermediate_timeout)
|| File "/usr/local/lib/python3.8/dist-packages/ray/worker.py", line 1027, in sigterm_handler
|| sys.exit(signum)
|| SystemExit: 15
||
|
{'audio': ('xaudio2', 'directsound', 'openal', 'pulse', 'silent'), 'debug_font': False, 'debug_gl': True, 'debug_gl_trace': False, 'debug_gl_trace_args': False, 'debug_graphics_batch': False, 'debug_lib': False, 'debug_media': False, 'debug_texture': False, 'debug_trace': False, 'debug_trace_args': False, 'debug_trace_depth': 1, 'debug_trace_flush': True, 'debug_win32': False, 'debug_x11': False, 'graphics_vbo': True, 'shadow_window': True, 'vsync': None, 'xsync': True, 'xlib_fullscreen_override_redirect': False, 'darwin_cocoa': True, 'search_local_libs': True, 'advanced_font_features': False, 'headless': False, 'headless_device': 0}
{'callbacks': <ray.rllib.agents.callbacks.MultiCallbacks object at 0x7f304497dca0>,
'env': 'Duckietown',
'env_config': {'accepted_start_angle_deg': 4,
'action_delay_ratio': 0.0,
'action_type': 'heading',
'aido_wrapper': False,
'camera_rand': False,
'crop_image_top': True,
'distortion': True,
'domain_rand': False,
'dynamics_rand': False,
'episode_max_steps': 10,
'eval': True,
'experiment_name': 'Debug',
'frame_repeating': 0.0,
'frame_skip': 1,
'frame_stacking': True,
'frame_stacking_depth': 3,
'grayscale_image': False,
'mode': 'debug',
'motion_blur': False,
'obstacles': {'duckie': {'density': 0.5, 'static': True},
'duckiebot': {'density': 0, 'static': False}},
'resized_input_shape': '(84, 84)',
'reward_function': 'posangle',
'seed': 0,
'simulation_framerate': 30,
'spawn_forward_obstacle': False,
'spawn_obstacles': False,
'top_crop_divider': 3,
'training_map': 'loop_empty',
'wandb': {'project': 'duckietown-rllib'}},
'evaluation_config': {'monitor': True},
'evaluation_interval': 25,
'evaluation_num_episodes': 10,
'framework': 'torch',
'gamma': 0.99,
'lr': 0.0001,
'monitor': False,
'num_gpus': 1,
'num_workers': 1,
'seed': 1234,
'train_batch_size': 128}
(RolloutWorker pid=368) {'audio': ('xaudio2', 'directsound', 'openal', 'pulse', 'silent'), 'debug_font': False, 'debug_gl': True, 'debug_gl_trace': False, 'debug_gl_trace_args': False, 'debug_graphics_batch': False, 'debug_lib': False, 'debug_media': False, 'debug_texture': False, 'debug_trace': False, 'debug_trace_args': False, 'debug_trace_depth': 1, 'debug_trace_flush': True, 'debug_win32': False, 'debug_x11': False, 'graphics_vbo': True, 'shadow_window': True, 'vsync': None, 'xsync': True, 'xlib_fullscreen_override_redirect': False, 'darwin_cocoa': True, 'search_local_libs': True, 'advanced_font_features': False, 'headless': False, 'headless_device': 0}
Setting the path for recording to /root/ray_results/PPO_Duckietown_2022-03-16_14-56-261tnr8g_b/
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 356, in loop
for parsed in inputs(fi, waiting_for=waiting_for):
File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/reading.py", line 21, in inputs
readyr, readyw, readyx = select.select([f], [], [f], intermediate_timeout)
File "/usr/local/lib/python3.8/dist-packages/ray/worker.py", line 1027, in sigterm_handler
sys.exit(signum)
SystemExit: 15
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 287, in run_loop
loop(my_logger, node_name, fi, fo, node, protocol, tin, tout, config=config, fi_desc=fin,
File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 480, in loop
raise InternalProblem(msg) from e # XXX
zuper_nodes.structures.InternalProblem: Unexpected error:
| Traceback (most recent call last):
| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 356, in loop
| for parsed in inputs(fi, waiting_for=waiting_for):
| File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/reading.py", line 21, in inputs
| readyr, readyw, readyx = select.select([f], [], [f], intermediate_timeout)
| File "/usr/local/lib/python3.8/dist-packages/ray/worker.py", line 1027, in sigterm_handler
| sys.exit(signum)
| SystemExit: 15
|
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "solution.py", line 136, in <module>
main()
File "solution.py", line 132, in main
wrap_direct(node=node, protocol=protocol)
File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/interface.py", line 25, in wrap_direct
run_loop(node, protocol, args)
File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 301, in run_loop
raise Exception(msg) from e
Exception: Error in node PytorchRLTemplateAgent
|
ETU-JBR Team