Skip to content

windows系统下加载模型错误 #2004

Open
@Lyn0222

Description

@Lyn0222

在windows系统下,通过mindnlp库加载模型的时候会发生报错:

import winfcntlock as fcntl # pylint: disable=import-error
ModuleNotFoundError: No module named 'winfcntlock'

经检查,修改utils/download.py里的import winfcntlock as fcntl为from . import winfcntlock as fcntl可以解决该问题。但是,继续运行之后仍有报错:

Traceback (most recent call last):
  File "E:\git\biogpt\infer_mindspore.py", line 8, in <module>
    tokenizer = BioGptTokenizer.from_pretrained(model_name)
  File "D:\install\anaconda\envs\biogpt\lib\site-packages\mindnlp\transformers\tokenization_utils_base.py", line 2115, in from_pretrained
    resolved_config_file = cached_file(
  File "D:\install\anaconda\envs\biogpt\lib\site-packages\mindnlp\utils\download.py", line 527, in cached_file
    resolved_file = download(
  File "D:\install\anaconda\envs\biogpt\lib\site-packages\mindnlp\utils\download.py", line 655, in download
    raise exp
  File "D:\install\anaconda\envs\biogpt\lib\site-packages\mindnlp\utils\download.py", line 651, in download
    pointer_path = threads_exclusive_http_get(url, storage_folder, download_file_name=relative_filename, proxies=proxies, headers=headers)
  File "D:\install\anaconda\envs\biogpt\lib\site-packages\mindnlp\utils\download.py", line 162, in threads_exclusive_http_get
    fcntl.flock(fd, fcntl.LOCK_UN)
AttributeError: module 'mindnlp.utils.winfcntlock' has no attribute 'flock'. Did you mean: 'lock'?

这是因为,download.py中引用winfcntlock.py中的函数名称和实际名称对不上,有两处不对:①download.py中引用的是fcntl.flock(),但是winfcntlock.py中实际名称为lock;②download.py中引用的是fcntl.LOCK_UN,但是winfcntlock.py中没有这个变量声明;

个人参考网上资料,找到了一个解决方案:修改winfcntlock.py中的lock函数名称为flock,并添加变量LOCK_UN = 0x08 。
修改后,为了解决报错:

Traceback (most recent call last):
  File "D:\install\anaconda\envs\biogpt\lib\site-packages\mindnlp\utils\download.py", line 160, in threads_exclusive_http_get
    raise exp
  File "D:\install\anaconda\envs\biogpt\lib\site-packages\mindnlp\utils\download.py", line 156, in threads_exclusive_http_get
    fcntl.flock(fd, fcntl.LOCK_EX)
  File "D:\install\anaconda\envs\biogpt\lib\site-packages\winfcntl\winfcntl.py", line 180, in flock
    handle = msvcrt.get_osfhandle(fd)
TypeError: '_io.TextIOWrapper' object cannot be interpreted as an integer

修改了winfcntlock.py中的hfile = win32file._get_osfhandle(file.fileno())为fd = hfile = win32file._get_osfhandle(file),不然对int变量取file.fileno()会报错。
修改后,下载模型能够正常进行,但是文件夹里会有很多.lock临时文件:

Image

修改download.py文件,在下载完成后删除.lock临时文件:

lock_file.close()
try:
    os.remove(lock_file_path)
except Exception as delete_exp:
    logging.error(f"Failed to delete lock file: {delete_exp}")
Image

现在在windows系统上可以正常运行了,对于download.py文件没有做很多修改,所以linux的运行应该不受影响。

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions