frame #17: at::_ops::to_dtype_layout::call(at::Tensor const&, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, bool, bool, std::optional<c10::MemoryFormat>) + 0x200 (0x7477414f71c0 in /opt/comfyui/venv043/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #18: <unknown function> + 0x453dfb (0x747754853dfb in /opt/comfyui/venv043/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #19: <unknown function> + 0x4b5d47 (0x7477548b5d47 in /opt/comfyui/venv043/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #20: /opt/comfyui/venv043/bin/python() [0x551418]
frame #21: PyObject_Vectorcall + 0x35 (0x5499b5 in /opt/comfyui/venv043/bin/python)
frame #22: _PyEval_EvalFrameDefault + 0xadf (0x5d710f in /opt/comfyui/venv043/bin/python)
frame #23: /opt/comfyui/venv043/bin/python() [0x54caed]
frame #24: /opt/comfyui/venv043/bin/python() [0x549d87]
frame #25: PyObject_CallFunctionObjArgs + 0xa2 (0x549b02 in /opt/comfyui/venv043/bin/python)
frame #26: <unknown function> + 0xe5898b (0x74775525898b in /opt/comfyui/venv043/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #27: <unknown function> + 0x78abeb (0x747754b8abeb in /opt/comfyui/venv043/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #28: <unknown function> + 0x1b6c27d (0x74774076c27d in /opt/comfyui/venv043/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #29: <unknown function> + 0x1b6ab51 (0x74774076ab51 in /opt/comfyui/venv043/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #30: <unknown function> + 0x2957855 (0x747741557855 in /opt/comfyui/venv043/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #31: at::_ops::to_dtype_layout::call(at::Tensor const&, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, bool, bool, std::optional<c10::MemoryFormat>) + 0x39d (0x7477414f735d in /opt/comfyui/venv043/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #32: <unknown function> + 0x453dfb (0x747754853dfb in /opt/comfyui/venv043/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #33: <unknown function> + 0x4b5d47 (0x7477548b5d47 in /opt/comfyui/venv043/lib/python3.12/site-packages/torch/lib/libtorch_python.so)
frame #34: /opt/comfyui/venv043/bin/python() [0x551418]
frame #35: PyObject_Vectorcall + 0x35 (0x5499b5 in /opt/comfyui/venv043/bin/python)
frame #36: _PyEval_EvalFrameDefault + 0xadf (0x5d710f in /opt/comfyui/venv043/bin/python)
frame #37: /opt/comfyui/venv043/bin/python() [0x54cbb4]
frame #38: PyObject_Call + 0x115 (0x54b1d5 in /opt/comfyui/venv043/bin/python)
frame #39: _PyEval_EvalFrameDefault + 0x4cbe (0x5db2ee in /opt/comfyui/venv043/bin/python)
frame #40: /opt/comfyui/venv043/bin/python() [0x66c0b9]
frame #41: <unknown function> + 0x9d44 (0x7477582abd44 in /usr/lib/python3.12/lib-dynload/_asyncio.cpython-312-x86_64-linux-gnu.so)
frame #42: <unknown function> + 0x9c56 (0x7477582abc56 in /usr/lib/python3.12/lib-dynload/_asyncio.cpython-312-x86_64-linux-gnu.so)
frame #43: /opt/comfyui/venv043/bin/python() [0x582082]
frame #44: /opt/comfyui/venv043/bin/python() [0x6a4b13]
frame #45: /opt/comfyui/venv043/bin/python() [0x581fed]
frame #46: _PyEval_EvalFrameDefault + 0x4cbe (0x5db2ee in /opt/comfyui/venv043/bin/python)
frame #47: /opt/comfyui/venv043/bin/python() [0x54cb52]
frame #48: /opt/comfyui/venv043/bin/python() [0x6f899c]
frame #49: /opt/comfyui/venv043/bin/python() [0x6b93bc]
frame #50: <unknown function> + 0x9cb84 (0x74775989cb84 in /lib/x86_64-linux-gnu/libc.so.6)
frame #51: <unknown function> + 0x129ecc (0x747759929ecc in /lib/x86_64-linux-gnu/libc.so.6)
 (Triggered internally at /__w/pytorch/pytorch/c10/hip/impl/HIPGuardImpl.h:70.)
  new_qdata = qt._qdata.to(device=target_device)
[ERROR] !!! Exception during processing !!! CUDA error: an illegal memory access was encountered
Search for `hipErrorIllegalAddress' in https://rocm.docs.amd.com/projects/HIP/en/latest/index.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Device-side assertion tracking was not enabled by user.
[ERROR] Traceback (most recent call last):
  File "/opt/comfyui/ComfyUI-043/comfy/model_patcher.py", line 1300, in partially_load
    self.load(device_to, lowvram_model_memory=current_used + extra_memory, force_patch_weights=force_patch_weights, full_load=full_load)
  File "/opt/comfyui/ComfyUI-043/comfy/model_patcher.py", line 1106, in load
    x[2].to(device_to)
  File "/opt/comfyui/venv043/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1388, in to
    return self._apply(convert)
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/comfyui/ComfyUI-043/comfy/ops.py", line 1610, in _apply
    return _quantized_apply(self, fn, recurse)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/comfyui/ComfyUI-043/comfy/ops.py", line 1243, in _quantized_apply
    p = fn(param)
        ^^^^^^^^^
  File "/opt/comfyui/venv043/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1374, in convert
    return t.to(
           ^^^^^
  File "/opt/comfyui/venv043/lib/python3.12/site-packages/comfy_kitchen/tensor/base.py", line 387, in __torch_dispatch__
    return handler(qt, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/comfyui/venv043/lib/python3.12/site-packages/comfy_kitchen/tensor/base.py", line 460, in _handle_to
    new_qdata = qt._qdata.to(device=target_device)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
Search for `hipErrorIllegalAddress' in https://rocm.docs.amd.com/projects/HIP/en/latest/index.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Device-side assertion tracking was not enabled by user.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/comfyui/ComfyUI-043/execution.py", line 560, in execute
    output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/comfyui/ComfyUI-043/execution.py", line 353, in get_output_data
    return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/comfyui/ComfyUI-043/execution.py", line 327, in _async_map_node_over_list
    await process_inputs(input_dict, i)
  File "/opt/comfyui/ComfyUI-043/execution.py", line 315, in process_inputs
    result = f(**inputs)
             ^^^^^^^^^^^
  File "/opt/comfyui/ComfyUI-043/comfy_api/internal/__init__.py", line 149, in wrapped_func
    return method(locked_class, **inputs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/comfyui/ComfyUI-043/comfy_api/latest/_io.py", line 2219, in EXECUTE_NORMALIZED
    to_return = cls.execute(*args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/comfyui/ComfyUI-043/comfy_extras/nodes_minimax_h3.py", line 361, in execute
    cond = clip.encode_from_tokens_scheduled(tokens)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/comfyui/ComfyUI-043/comfy/sd.py", line 349, in encode_from_tokens_scheduled
    pooled_dict = self.encode_from_tokens(tokens, return_pooled=return_pooled, return_dict=True)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/comfyui/ComfyUI-043/comfy/sd.py", line 415, in encode_from_tokens
    self.load_model(tokens)
  File "/opt/comfyui/ComfyUI-043/comfy/sd.py", line 473, in load_model
    model_management.load_models_gpu([self.patcher], memory_required=memory_used)
  File "/opt/comfyui/ComfyUI-043/comfy/model_management.py", line 1034, in load_models_gpu
    loaded_model.model_load(lowvram_model_memory, force_patch_weights=force_patch_weights)
  File "/opt/comfyui/ComfyUI-043/comfy/model_management.py", line 820, in model_load
    self.model_use_more_vram(use_more_vram, force_patch_weights=force_patch_weights)
  File "/opt/comfyui/ComfyUI-043/comfy/model_management.py", line 848, in model_use_more_vram
    return self.model.partially_load(self.device, extra_memory, force_patch_weights=force_patch_weights)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/comfyui/ComfyUI-043/comfy/model_patcher.py", line 1302, in partially_load
    self.detach()
  File "/opt/comfyui/ComfyUI-043/comfy/model_patcher.py", line 1322, in detach
    self.unpatch_model(self.offload_device, unpatch_weights=unpatch_all)
  File "/opt/comfyui/ComfyUI-043/comfy/model_patcher.py", line 1179, in unpatch_model
    self.model.to(device_to)
  File "/opt/comfyui/venv043/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1388, in to
    return self._apply(convert)
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/comfyui/venv043/lib/python3.12/site-packages/torch/nn/modules/module.py", line 933, in _apply
    module._apply(fn)
  File "/opt/comfyui/venv043/lib/python3.12/site-packages/torch/nn/modules/module.py", line 933, in _apply
    module._apply(fn)
  File "/opt/comfyui/venv043/lib/python3.12/site-packages/torch/nn/modules/module.py", line 933, in _apply
    module._apply(fn)
  [Previous line repeated 1 more time]
  File "/opt/comfyui/venv043/lib/python3.12/site-packages/torch/nn/modules/module.py", line 966, in _apply
    param_applied = fn(param)
                    ^^^^^^^^^
  File "/opt/comfyui/venv043/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1374, in convert
    return t.to(
           ^^^^^
  File "/opt/comfyui/venv043/lib/python3.12/site-packages/comfy_kitchen/tensor/base.py", line 387, in __torch_dispatch__
    return handler(qt, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/comfyui/venv043/lib/python3.12/site-packages/comfy_kitchen/tensor/base.py", line 460, in _handle_to
    new_qdata = qt._qdata.to(device=target_device)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
Search for `hipErrorIllegalAddress' in https://rocm.docs.amd.com/projects/HIP/en/latest/index.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Device-side assertion tracking was not enabled by user.

Exception in thread Thread-1 (prompt_worker):
Traceback (most recent call last):
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/comfyui/ComfyUI-043/main.py", line 384, in prompt_worker
    e.execute(item[2], prompt_id, extra_data, item[4])
  File "/opt/comfyui/ComfyUI-043/execution.py", line 743, in execute
    asyncio.run(self.execute_async(prompt, prompt_id, extra_data, execute_outputs))
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/opt/comfyui/ComfyUI-043/execution.py", line 853, in execute_async
    comfy.model_management.unload_all_models()
  File "/opt/comfyui/ComfyUI-043/comfy/model_management.py", line 2173, in unload_all_models
    free_memory(1e30, device)
  File "/opt/comfyui/ComfyUI-043/comfy/model_management.py", line 934, in free_memory
    mem_free_total, mem_free_torch = get_free_memory(device, torch_free_too=True)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/comfyui/ComfyUI-043/comfy/model_management.py", line 1858, in get_free_memory
    mem_free_cuda, _ = torch.cuda.mem_get_info(dev)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/comfyui/venv043/lib/python3.12/site-packages/torch/cuda/memory.py", line 863, in mem_get_info
    return torch.cuda.cudart().cudaMemGetInfo(device)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
Search for `hipErrorIllegalAddress' in https://rocm.docs.amd.com/projects/HIP/en/latest/index.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Device-side assertion tracking was not enabled by user.
[ERROR] Error handling request from 127.0.0.1
Traceback (most recent call last):
