无需搭建环境，零门槛带你体验 Open-Sora 文生视频应用

import osimport moxing as mox​if not os.path.exists('Open-Sora'):    mox.file.copy_parallel('obs://modelbox-course/open-sora_1.1/Open-Sora', 'Open-Sora')    if not os.path.exists('/home/ma-user/.cache/huggingface'):    mox.file.copy_parallel('obs://modelbox-course/huggingface', '/home/ma-user/.cache/huggingface')    if not os.path.exists('Open-Sora/opensora/models/sd-vae-ft-ema'):    mox.file.copy_parallel('obs://modelbox-course/sd-vae-ft-ema', 'Open-Sora/opensora/models/sd-vae-ft-ema')​if not os.path.exists('Open-Sora/opensora/models/text_encoder/t5-v1_1-xxl'):    mox.file.copy_parallel('obs://modelbox-course/t5-v1_1-xxl', 'Open-Sora/opensora/models/text_encoder/t5-v1_1-xxl')    if not os.path.exists('/home/ma-user/work/t5.py'):    mox.file.copy_parallel('obs://modelbox-course/open-sora_1.1/t5.py', '/home/ma-user/work/t5.py')    if not os.path.exists('Open-Sora/opus-mt-zh-en'):    mox.file.copy_parallel('obs://modelarts-labs-bj4-v2/course/ModelBox/opus-mt-zh-en', 'Open-Sora/opus-mt-zh-en')    if not os.path.exists('/home/ma-user/work/frpc_linux_amd64'):    mox.file.copy_parallel('obs://modelarts-labs-bj4-v2/course/ModelBox/frpc_linux_amd64', '/home/ma-user/work/frpc_linux_amd64')

INFO:root:Using MoXing-v2.1.6.879ab2f4-879ab2f4​INFO:root:List OBS time cost: 0.02 seconds.​INFO:root:Copy parallel total time cost: 41.71 seconds.​INFO:root:List OBS time cost: 0.14 seconds.​INFO:root:Copy parallel total time cost: 2.91 seconds.

!/home/ma-user/anaconda3/bin/conda clean -i!/home/ma-user/anaconda3/bin/conda create -n python-3.10.10 python=3.10.10 -y --override-channels --channel https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main!/home/ma-user/anaconda3/envs/python-3.10.10/bin/pip install ipykernel

/home/ma-user/anaconda3/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.15) or chardet (3.0.4) doesn't match a supported version!  RequestsDependencyWarning)/home/ma-user/anaconda3/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.15) or chardet (3.0.4) doesn't match a supported version!  RequestsDependencyWarning)Collecting package metadata (current_repodata.json): doneSolving environment: failed with repodata from current_repodata.json, will retry with next repodata source.Collecting package metadata (repodata.json): doneSolving environment: done​## Package Plan ##​  environment location: /home/ma-user/anaconda3/envs/python-3.10.10​  added / updated specs:    - python=3.10.10

The following packages will be downloaded:​    package                    |            build    ---------------------------|-----------------    _libgcc_mutex-0.1          |             main           3 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    _openmp_mutex-5.1          |            1_gnu          21 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    bzip2-1.0.8                |       h5eee18b_6         262 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    ca-certificates-2024.3.11  |       h06a4308_0         127 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    ld_impl_linux-64-2.38      |       h1181459_1         654 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    libffi-3.4.4               |       h6a678d5_1         141 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    libgcc-ng-11.2.0           |       h1234567_1         5.3 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    libgomp-11.2.0             |       h1234567_1         474 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    libstdcxx-ng-11.2.0        |       h1234567_1         4.7 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    libuuid-1.41.5             |       h5eee18b_0          27 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    ncurses-6.4                |       h6a678d5_0         914 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    openssl-1.1.1w             |       h7f8727e_0         3.7 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    pip-24.0                   |  py310h06a4308_0         2.7 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    python-3.10.10             |       h7a1cb2a_2        26.9 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    readline-8.2               |       h5eee18b_0         357 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    setuptools-69.5.1          |  py310h06a4308_0        1012 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    sqlite-3.45.3              |       h5eee18b_0         1.2 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    tk-8.6.14                  |       h39e8969_0         3.4 MB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    tzdata-2024a               |       h04d1e81_0         116 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    wheel-0.43.0               |  py310h06a4308_0         110 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    xz-5.4.6                   |       h5eee18b_1         643 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    zlib-1.2.13                |       h5eee18b_1         111 KB  https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main    ------------------------------------------------------------                                           Total:        52.8 MB​The following NEW packages will be INSTALLED:​  _libgcc_mutex      anaconda/pkgs/main/linux-64::_libgcc_mutex-0.1-main  _openmp_mutex      anaconda/pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu  bzip2              anaconda/pkgs/main/linux-64::bzip2-1.0.8-h5eee18b_6  ca-certificates    anaconda/pkgs/main/linux-64::ca-certificates-2024.3.11-h06a4308_0  ld_impl_linux-64   anaconda/pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1  libffi             anaconda/pkgs/main/linux-64::libffi-3.4.4-h6a678d5_1  libgcc-ng          anaconda/pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1  libgomp            anaconda/pkgs/main/linux-64::libgomp-11.2.0-h1234567_1  libstdcxx-ng       anaconda/pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1  libuuid            anaconda/pkgs/main/linux-64::libuuid-1.41.5-h5eee18b_0  ncurses            anaconda/pkgs/main/linux-64::ncurses-6.4-h6a678d5_0  openssl            anaconda/pkgs/main/linux-64::openssl-1.1.1w-h7f8727e_0  pip                anaconda/pkgs/main/linux-64::pip-24.0-py310h06a4308_0  python             anaconda/pkgs/main/linux-64::python-3.10.10-h7a1cb2a_2  readline           anaconda/pkgs/main/linux-64::readline-8.2-h5eee18b_0  setuptools         anaconda/pkgs/main/linux-64::setuptools-69.5.1-py310h06a4308_0  sqlite             anaconda/pkgs/main/linux-64::sqlite-3.45.3-h5eee18b_0  tk                 anaconda/pkgs/main/linux-64::tk-8.6.14-h39e8969_0  tzdata             anaconda/pkgs/main/noarch::tzdata-2024a-h04d1e81_0  wheel              anaconda/pkgs/main/linux-64::wheel-0.43.0-py310h06a4308_0  xz                 anaconda/pkgs/main/linux-64::xz-5.4.6-h5eee18b_1  zlib               anaconda/pkgs/main/linux-64::zlib-1.2.13-h5eee18b_1

Downloading and Extracting Packageslibffi-3.4.4         | 141 KB    | ##################################### | 100% _openmp_mutex-5.1    | 21 KB     | ##################################### | 100% xz-5.4.6             | 643 KB    | ##################################### | 100% tzdata-2024a         | 116 KB    | ##################################### | 100% _libgcc_mutex-0.1    | 3 KB      | ##################################### | 100% zlib-1.2.13          | 111 KB    | ##################################### | 100% bzip2-1.0.8          | 262 KB    | ##################################### | 100% libuuid-1.41.5       | 27 KB     | ##################################### | 100% ca-certificates-2024 | 127 KB    | ##################################### | 100% libstdcxx-ng-11.2.0  | 4.7 MB    | ##################################### | 100% ncurses-6.4          | 914 KB    | ##################################### | 100% openssl-1.1.1w       | 3.7 MB    | ##################################### | 100% wheel-0.43.0         | 110 KB    | ##################################### | 100% python-3.10.10       | 26.9 MB   | ##################################### | 100% pip-24.0             | 2.7 MB    | ##################################### | 100% readline-8.2         | 357 KB    | ##################################### | 100% tk-8.6.14            | 3.4 MB    | ##################################### | 100% setuptools-69.5.1    | 1012 KB   | ##################################### | 100% libgcc-ng-11.2.0     | 5.3 MB    | ##################################### | 100% ld_impl_linux-64-2.3 | 654 KB    | ##################################### | 100% libgomp-11.2.0       | 474 KB    | ##################################### | 100% sqlite-3.45.3        | 1.2 MB    | ##################################### | 100% Preparing transaction: doneVerifying transaction: doneExecuting transaction: done## To activate this environment, use##     $ conda activate python-3.10.10## To deactivate an active environment, use##     $ conda deactivate​Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simpleCollecting ipykernel  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/53/9d/40d5207db523363d9b5698f33778c18b0d591e3fdb6e0116b894b2a2491c/ipykernel-6.29.4-py3-none-any.whl (117 kB)•[2K     •[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━•[0m •[32m117.1/117.1 kB•[0m •[31m10.6 MB/s•[0m eta •[36m0:00:00•[0m......​  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/80/03/6ea8b1b2a5ab40a7a60dc464d3daa7aa546e0a74d74a9f8ff551ea7905db/executing-2.0.1-py2.py3-none-any.whl (24 kB)Collecting asttokens>=2.1.0 (from stack-data->ipython>=7.23.1->ipykernel)  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/45/86/4736ac618d82a20d87d2f92ae19441ebc7ac9e7a581d7e58bbe79233b24a/asttokens-2.4.1-py2.py3-none-any.whl (27 kB)Collecting pure-eval (from stack-data->ipython>=7.23.1->ipykernel)  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/2b/27/77f9d5684e6bce929f5cfe18d6cfbe5133013c06cb2fbf5933670e60761d/pure_eval-0.2.2-py3-none-any.whl (11 kB)Installing collected packages: wcwidth, pure-eval, ptyprocess, typing-extensions, traitlets, tornado, six, pyzmq, pygments, psutil, prompt-toolkit, platformdirs, pexpect, parso, packaging, nest-asyncio, executing, exceptiongroup, decorator, debugpy, python-dateutil, matplotlib-inline, jupyter-core, jedi, comm, asttokens, stack-data, jupyter-client, ipython, ipykernelSuccessfully installed asttokens-2.4.1 comm-0.2.2 debugpy-1.8.1 decorator-5.1.1 exceptiongroup-1.2.1 executing-2.0.1 ipykernel-6.29.4 ipython-8.25.0 jedi-0.19.1 jupyter-client-8.6.2 jupyter-core-5.7.2 matplotlib-inline-0.1.7 nest-asyncio-1.6.0 packaging-24.0 parso-0.8.4 pexpect-4.9.0 platformdirs-4.2.2 prompt-toolkit-3.0.46 psutil-5.9.8 ptyprocess-0.7.0 pure-eval-0.2.2 pygments-2.18.0 python-dateutil-2.9.0.post0 pyzmq-26.0.3 six-1.16.0 stack-data-0.6.3 tornado-6.4 traitlets-5.14.3 typing-extensions-4.12.1 wcwidth-0.2.13

import jsonimport os​data = {   "display_name": "python-3.10.10",   "env": {      "PATH": "/home/ma-user/anaconda3/envs/python-3.10.10/bin:/home/ma-user/anaconda3/envs/python-3.7.10/bin:/modelarts/authoring/notebook-conda/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/ma-user/modelarts/ma-cli/bin:/home/ma-user/modelarts/ma-cli/bin:/home/ma-user/anaconda3/envs/PyTorch-1.8/bin"   },   "language": "python",   "argv": [      "/home/ma-user/anaconda3/envs/python-3.10.10/bin/python",      "-m",      "ipykernel",      "-f",      "{connection_file}"   ]}​if not os.path.exists("/home/ma-user/anaconda3/share/jupyter/kernels/python-3.10.10/"):    os.mkdir("/home/ma-user/anaconda3/share/jupyter/kernels/python-3.10.10/")​with open('/home/ma-user/anaconda3/share/jupyter/kernels/python-3.10.10/kernel.json', 'w') as f:    json.dump(data, f, indent=4)

conda env list

/home/ma-user/anaconda3/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.15) or chardet (3.0.4) doesn't match a supported version!  RequestsDependencyWarning)# conda environments:#base                  *  /home/ma-user/anaconda3python-3.10.10           /home/ma-user/anaconda3/envs/python-3.10.10python-3.7.10            /home/ma-user/anaconda3/envs/python-3.7.10

Note: you may need to restart the kernel to use updated packages.

!python -V

Python 3.10.10

!nvidia-smi

Wed Jun  5 16:22:37 2024       +-----------------------------------------------------------------------------+| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     ||-------------------------------+----------------------+----------------------+| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC || Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. ||                               |                      |               MIG M. ||===============================+======================+======================||   0  Tesla V100-PCIE...  On   | 00000000:00:0D.0 Off |                    0 || N/A   28C    P0    25W / 250W |      0MiB / 32510MiB |      0%      Default ||                               |                      |                  N/A |+-------------------------------+----------------------+----------------------+                                                                               +-----------------------------------------------------------------------------+| Processes:                                                                  ||  GPU   GI   CI        PID   Type   Process name                  GPU Memory ||        ID   ID                                                   Usage      ||=============================================================================||  No running processes found                                                 |+-----------------------------------------------------------------------------+

!pip install --upgrade pip!pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 xformers==0.0.22

Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simpleRequirement already satisfied: pip in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (24.0)Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simpleCollecting torch==2.0.1  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/8c/4d/17e07377c9c3d1a0c4eb3fde1c7c16b5a0ce6133ddbabc08ceef6b7f2645/torch-2.0.1-cp310-cp310-manylinux1_x86_64.whl (619.9 MB)•[2K     •[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━•[0m •[32m619.9/619.9 MB•[0m •[31m8.2 MB/s•[0m eta •[36m0:00:00•[0m00:01•[0m00:01•[0m•[?25hCollecting torchvision==0.15.2  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/87/0f/88f023bf6176d9af0f85feedf4be129f9cf2748801c4d9c690739a10c100/torchvision-0.15.2-cp310-cp310-manylinux1_x86_64.whl (6.0 MB)•[2K     •[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━•[0m •[32m6.0/6.0 MB•[0m •[31m109.5 MB/s•[0m eta •[36m0:00:00•[0ma •[36m0:00:01•[0m•[?25hCollecting torchaudio==2.0.2  Downloading   •[?25hCollecting certifi>=2017.4.17 (from requests->torchvision==0.15.2)  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/5b/11/1e78951465b4a225519b8c3ad29769c49e0d8d157a070f681d5b6d64737f/certifi-2024.6.2-py3-none-any.whl (164 kB)•[2K     •[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━•[0m •[32m164.4/164.4 kB•[0m •[31m23.1 MB/s•[0m eta •[36m0:00:00•[0m•[?25hCollecting mpmath<1.4.0,>=1.1.0 (from sympy->torch==2.0.1)  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl (536 kB)•[2K     •[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━•[0m •[32m536.2/536.2 kB•[0m •[31m32.8 MB/s•[0m eta •[36m0:00:00•[0m•[?25hInstalling collected packages: mpmath, lit, urllib3, sympy, pillow, nvidia-nvtx-cu11, nvidia-nccl-cu11, nvidia-cusparse-cu11, nvidia-curand-cu11, nvidia-cufft-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-cupti-cu11, nvidia-cublas-cu11, numpy, networkx, MarkupSafe, idna, filelock, cmake, charset-normalizer, certifi, requests, nvidia-cusolver-cu11, nvidia-cudnn-cu11, jinja2, triton, torch, xformers, torchvision, torchaudioSuccessfully installed MarkupSafe-2.1.5 certifi-2024.6.2 charset-normalizer-3.3.2 cmake-3.29.3 filelock-3.14.0 idna-3.7 jinja2-3.1.4 lit-18.1.6 mpmath-1.3.0 networkx-3.3 numpy-1.26.4 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-cupti-cu11-11.7.101 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 nvidia-cufft-cu11-10.9.0.58 nvidia-curand-cu11-10.2.10.91 nvidia-cusolver-cu11-11.4.0.1 nvidia-cusparse-cu11-11.7.4.91 nvidia-nccl-cu11-2.14.3 nvidia-nvtx-cu11-11.7.91 pillow-10.3.0 requests-2.32.3 sympy-1.12.1 torch-2.0.1 torchaudio-2.0.2 torchvision-0.15.2 triton-2.0.0 urllib3-2.2.1 xformers-0.0.22

%cd Open-Sora

/home/ma-user/work/ma_share/open-spra_1/Open-Sora

/home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/IPython/core/magics/osm.py:417: UserWarning: This is now an optional IPython functionality, setting dhist requires you to install the `pickleshare` library.  self.shell.db['dhist'] = compress_dhist(dhist)[-100:]

'/home/ma-user/work/ma_share/open-spra_1/Open-Sora'

!pip install colossalai==0.3.6 accelerate==0.29.2 diffusers==0.27.2 ftfy==6.2.0 gdown==5.1.0 mmengine==0.10.3 pre-commit==3.7.0 pyav==12.0.5 tensorboard==2.16.2 timm==0.9.16 transformers==4.39.3 wandb==0.16.6

Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simpleCollecting colossalai==0.3.6  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/05/ed/57e80620ea8e35c3aa63a3207720b1890700fd12eea38b6592e9833e5c1b/colossalai-0.3.6.tar.gz (1.1 MB)•[2K     •[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━•[0m •[32m1.1/1.1 MB•[0m •[31m36.5 MB/s•[0m eta •[36m0:00:00•[0m•[?25h  Preparing metadata (setup.py) ... •[?25ldone•[?25hCollecting accelerate==0.29.2  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/1b/e8/2fc7af3fa77ddac89a9c9b390d2d31d1db0612247ba2274009946959604e/accelerate-0.29.2-py3-none-any.whl (297 kB)•[2K     •[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━•[0m •[32m297.4/297.4 kB•[0m •[31m14.5 MB/s•[0m eta •[36m0:00:00•[0m•[?25hCollecting diffusers==0.27.2  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/75/c5/3b84fd731dd93c549a0c25657e4ce5a957aeccd32d60dba2958cd3cdac23/diffusers-0.27.2-py3-none-any.whl (2.0 MB)​

!pip install .

Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simpleProcessing /home/ma-user/work/ma_share/open-spra_1/Open-Sora  Preparing metadata (setup.py) ... •[?25ldone•[?25hRequirement already satisfied: colossalai in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from opensora==1.1.0) (0.3.6)Requirement already satisfied: accelerate in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from opensora==1.1.0) (0.29.2)Requirement already satisfied: diffusers in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from opensora==1.1.0) (0.27.2)Requirement already satisfied: ftfy in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from opensora==1.1.0) (6.2.0)Requirement already satisfied: gdown in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from opensora==1.1.0) (5.1.0)Requirement already satisfied: mmengine in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from opensora==1.1.0) (0.10.3)Collecting pandas (from opensora==1.1.0)  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/89/1b/12521efcbc6058e2673583bb096c2b5046a9df39bd73eca392c1efed24e5/pandas-2.2.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.0 MB)•[2K     •[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━•[0m •[32m13.0/13.0 MB•[0m •[31m60.4 MB/s•[0m eta •[36m0:00:00•[0m00:01•[0m00:01•[0m•[?25hRequirement already satisfied: pre-commit in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from opensora==1.1.0) (3.7.0)Collecting pyarrow (from opensora==1.1.0)  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/91/83/57572c088ec185582f04b607d545a4a6ef7599c0a3c1e60d397743b0d609/pyarrow-16.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (40.9 MB)•[2K     •[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━•[0m •[32m40.9/40.9 MB•[0m •[31m36.9 MB/s•[0m eta •[36m0:00:00•[0m00:01•[0m00:01•[0m•[?25hCollecting av (from opensora==1.1.0)  Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/0a/11/2b501d0a4de22826217a0b909e832f52fb5d503df50f424f3e31023a7bcc/av-12.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.3 MB)•[2K     •[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━•[0m •[32m34.3/34.3 MB•[0m •[31m96.1 MB/s•[0m eta •[36m0:00:00•[0m00:01•[0m00:01•[0m•[?25hRequirement already satisfied: tensorboard in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from opensora==1.1.0) (2.16.2)Requirement already satisfied: timm in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from opensora==1.1.0) (0.9.16)Requirement already satisfied: tqdm in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from opensora==1.1.0) (4.66.4)Requirement already satisfied: transformers in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from opensora==1.1.0) (4.39.3)Requirement already satisfied: wandb in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from opensora==1.1.0) (0.16.6)Collecting rotary_embedding_torch (from opensora==1.1.0)  Downloading Building wheels for collected packages: opensora, pandarallel  Building wheel for opensora (setup.py) ... •[?25ldone•[?25h  Created wheel for opensora: filename=opensora-1.1.0-py3-none-any.whl size=195249 sha256=86c66de7ded305b2e4fb07992d0147c0408086cc31cdc31d97bcea44d8f69596  Stored in directory: /home/ma-user/.cache/pip/wheels/ae/34/85/7f84dd36f2e448d8d4455272d3358f557d0a570011d1701074  Building wheel for pandarallel (setup.py) ... •[?25ldone•[?25h  Created wheel for pandarallel: filename=pandarallel-1.6.5-py3-none-any.whl size=16673 sha256=b97386c92d34443f19cc88ea717c6cca143ef2b8f1f1ac79f4645c37d230bafc  Stored in directory: /home/ma-user/.cache/pip/wheels/f6/dd/25/a1c3775e721641ff67c71b3652e901e7e52611c6c3091784c9Successfully built opensora pandarallelInstalling collected packages: pytz, tzdata, pyarrow, dill, beartype, av, pandas, pandarallel, rotary_embedding_torch, opensoraSuccessfully installed av-12.1.0 beartype-0.18.5 dill-0.3.8 opensora-1.1.0 pandarallel-1.6.5 pandas-2.2.2 pyarrow-16.1.0 pytz-2024.1 rotary_embedding_torch-0.6.2 tzdata-2024.1

!pip install spaces gradio MoviePy -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn!cp /home/ma-user/work/frpc_linux_amd64 /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/gradio/frpc_linux_amd64_v0.2!chmod +x /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/gradio/frpc_linux_amd64_v0.2

Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simpleCollecting spaces  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/b2/3c/6205090507ea96e6e56d0deda8d0fc4c507026ef3772e55b637a5d0b7c61/spaces-0.28.3-py3-none-any.whl (18 kB)Collecting gradio  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d1/37/f49320600cdf1fa856cc605a2e20e9debd34b5425b53f49abdb2ea463716/gradio-4.32.2-py3-none-any.whl (12.3 MB)•[2K     •[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━•[0m •[32m12.3/12.3 MB•[0m •[31m5.2 MB/s•[0m eta •[36m0:00:00•[0m00:01•[0m00:01•[0m​      Successfully uninstalled decorator-5.1.1•[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.fabric 3.2.2 requires decorator>=5, but you have decorator 4.4.2 which is incompatible.•[0m•[31m•[0mSuccessfully installed MoviePy-1.0.3 aiofiles-23.2.1 altair-5.3.0 anyio-4.4.0 decorator-4.4.2 dnspython-2.6.1 email_validator-2.1.1 fastapi-0.111.0 fastapi-cli-0.0.4 ffmpy-0.3.2 gradio-4.32.2 gradio-client-0.17.0 h11-0.14.0 httpcore-1.0.5 httptools-0.6.1 httpx-0.27.0 imageio-2.34.1 imageio_ffmpeg-0.5.1 importlib-resources-6.4.0 orjson-3.10.3 proglog-0.1.10 pydub-0.25.1 python-dotenv-1.0.1 python-multipart-0.0.9 ruff-0.4.7 semantic-version-2.10.0 shellingham-1.5.4 sniffio-1.3.1 spaces-0.28.3 starlette-0.37.2 tomlkit-0.12.0 toolz-0.12.1 typer-0.12.3 ujson-5.10.0 uvicorn-0.30.1 uvloop-0.19.0 watchfiles-0.22.0 websockets-11.0.3

%%writefile configs/opensora-v1-1/inference/sample.pynum_frames = 16frame_interval = 3fps = 24image_size = (240, 426)multi_resolution = "STDiT2"​# Define modelmodel = dict(    type="STDiT2-XL/2",    from_pretrained="hpcai-tech/OpenSora-STDiT-v2-stage3",    input_sq_size=512,  # 使用huggingface上下载好的模型权重    qk_norm=True,    enable_flash_attn=True,    enable_layernorm_kernel=True,)vae = dict(    type="VideoAutoencoderKL",    from_pretrained="./opensora/models/sd-vae-ft-ema",    cache_dir=None,  # 修改为从当前目录加载    micro_batch_size=4,)text_encoder = dict(    type="t5",    from_pretrained="./opensora/models/text_encoder/t5-v1_1-xxl",    cache_dir=None,  # 修改为从当前目录加载    model_max_length=200,)scheduler = dict(    type="iddpm",    num_sampling_steps=100,    cfg_scale=7.0,    cfg_channel=3,  # or None)dtype = "fp16"​# Conditionprompt_path = "./assets/texts/t2v_samples.txt"prompt = None  # prompt has higher priority than prompt_path​# Othersbatch_size = 1seed = 42save_dir = "./samples/samples/"

Overwriting configs/opensora-v1-1/inference/sample.py

import os​os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'

!cp /home/ma-user/work/t5.py /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/opensora/models/text_encoder/t5.py

# text to video!python scripts/inference.py configs/opensora-v1-1/inference/sample.py --prompt "A fashion girl walking on the streets of Tokyo" --num-frames 32 --image-size 240 426

/home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/colossalai/shardformer/layer/normalization.py:45: UserWarning: Please install apex from source (https://github.com/NVIDIA/apex) to use the fused layernorm kernel  warnings.warn("Please install apex from source (https://github.com/NVIDIA/apex) to use the fused layernorm kernel")Config (path: configs/opensora-v1-1/inference/sample.py): {'num_frames': 32, 'frame_interval': 3, 'fps': 24, 'image_size': [240, 426], 'multi_resolution': 'STDiT2', 'model': {'type': 'STDiT2-XL/2', 'from_pretrained': 'hpcai-tech/OpenSora-STDiT-v2-stage3', 'input_sq_size': 512, 'qk_norm': True, 'enable_flash_attn': True, 'enable_layernorm_kernel': True}, 'vae': {'type': 'VideoAutoencoderKL', 'from_pretrained': './opensora/models/sd-vae-ft-ema', 'cache_dir': None, 'micro_batch_size': 4}, 'text_encoder': {'type': 't5', 'from_pretrained': './opensora/models/text_encoder/t5-v1_1-xxl', 'cache_dir': None, 'model_max_length': 200}, 'scheduler': {'type': 'iddpm', 'num_sampling_steps': 100, 'cfg_scale': 7.0, 'cfg_channel': 3}, 'dtype': 'fp16', 'prompt_path': './assets/texts/t2v_samples.txt', 'prompt': ['A fashion girl walking on the streets of Tokyo'], 'batch_size': 1, 'seed': 42, 'save_dir': './samples/samples/', 'config': 'configs/opensora-v1-1/inference/sample.py', 'prompt_as_path': False, 'reference_path': None, 'loop': 1, 'sample_name': None, 'num_sample': 1}Loading checkpoint shards:   0%|                          | 0/2 [00:00<?, ?it/s]/home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()  return self.fget.__get__(instance, owner)()Loading checkpoint shards: 100%|██████████████████| 2/2 [00:35<00:00, 17.87s/it]/home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.  warnings.warn(100%|█████████████████████████████████████████| 100/100 [02:11<00:00,  1.32s/it]Prompt: A fashion girl walking on the streets of TokyoSaved to ./samples/samples/sample_0.mp4

import osimport randomfrom moviepy.editor import *from IPython.display import Image​# 视频存放目录video_root = 'samples/samples'# 列出所有文件videos = os.listdir(video_root)# 随机抽取视频video = random.sample(videos, 1)[0]# 视频输入路径video_path = os.path.join(video_root, video)# 加载原始视频clip = VideoFileClip(video_path)# 保存为GIF文件clip.write_gif("output_animation.gif", fps=10)# 显示生成结果Image(open('output_animation.gif','rb').read())

MoviePy - Building file output_animation.gif with imageio.

%%writefile configs/opensora-v1-1/inference/sample-ref.pynum_frames = 16frame_interval = 3fps = 24image_size = (240, 426)multi_resolution = "STDiT2"​# Conditionprompt_path = Noneprompt = [    "A car driving on the ocean.",    "In an ornate, historical hall, a massive tidal wave peaks and begins to crash. Two surfers, seizing the moment, skillfully navigate the face of the wave.",]​loop = 2condition_frame_length = 4# (#   loop id, [the loop index of the condition image or video]#   reference id, [the index of the condition image or video in the reference_path]#   reference start, [the start frame of the condition image or video]#   target start, [the location to insert]#   length, [the number of frames to insert]#   edit_ratio [the edit rate of the condition image or video]# )# See https://github.com/hpcaitech/Open-Sora/blob/main/docs/config.md#advanced-inference-config for more details# See https://github.com/hpcaitech/Open-Sora/blob/main/docs/commands.md#inference-with-open-sora-11 for more examplesmask_strategy = [    "0,0,0,0,8,0.3",    None,    "0",]reference_path = [    "https://cdn.openai.com/tmp/s/interp/d0.mp4",    None,    "assets/images/condition/wave.png",]​# Define modelmodel = dict(    type="STDiT2-XL/2",    from_pretrained="hpcai-tech/OpenSora-STDiT-v2-stage3",    input_sq_size=512,  # 使用huggingface上下载好的模型权重    qk_norm=True,    enable_flash_attn=True,    enable_layernorm_kernel=True,)vae = dict(    type="VideoAutoencoderKL",    from_pretrained="./opensora/models/sd-vae-ft-ema",    cache_dir=None,  # 修改为从当前目录加载    micro_batch_size=4,)text_encoder = dict(    type="t5",    from_pretrained="./opensora/models/text_encoder/t5-v1_1-xxl",    cache_dir=None,  # 修改为从当前目录加载    model_max_length=200,)scheduler = dict(    type="iddpm",    num_sampling_steps=100,    cfg_scale=7.0,    cfg_channel=3,  # or None)dtype = "fp16"​# Othersbatch_size = 1seed = 42save_dir = "./samples/samples/"

Overwriting configs/opensora-v1-1/inference/sample-ref.py

%%writefile gradio/app-ref.pyimport argparseimport importlibimport osimport subprocessimport sysimport reimport jsonimport mathimport spacesimport torchimport gradio as grfrom tempfile import NamedTemporaryFileimport datetimefrom transformers import pipeline​zh2en = pipeline("translation", model="./opus-mt-zh-en")​MODEL_TYPES = ["v1.1-stage2", "v1.1-stage3"]CONFIG_MAP = {    "v1.1-stage2": "configs/opensora-v1-1/inference/sample-ref.py",    "v1.1-stage3": "configs/opensora-v1-1/inference/sample-ref.py",}HF_STDIT_MAP = {    "v1.1-stage2": "hpcai-tech/OpenSora-STDiT-v2-stage2",    "v1.1-stage3": "hpcai-tech/OpenSora-STDiT-v2-stage3",}RESOLUTION_MAP = {    "144p": {        "16:9": (256, 144),         "9:16": (144, 256),        "4:3": (221, 165),        "3:4": (165, 221),        "1:1": (192, 192),    },    "240p": {        "16:9": (426, 240),         "9:16": (240, 426),        "4:3": (370, 278),        "3:4": (278, 370),        "1:1": (320, 320),    },    "360p": {        "16:9": (640, 360),         "9:16": (360, 640),        "4:3": (554, 416),        "3:4": (416, 554),        "1:1": (480, 480),    },    "480p": {        "16:9": (854, 480),         "9:16": (480, 854),        "4:3": (740, 555),        "3:4": (555, 740),        "1:1": (640, 640),    },    "720p": {        "16:9": (1280, 720),         "9:16": (720, 1280),        "4:3": (1108, 832),        "3:4": (832, 1110),        "1:1": (960, 960),    },}​​# ============================# Utils# ============================def collect_references_batch(reference_paths, vae, image_size):    from opensora.datasets.utils import read_from_path​    refs_x = []    for reference_path in reference_paths:        if reference_path is None:            refs_x.append([])            continue        ref_path = reference_path.split(";")        ref = []        for r_path in ref_path:            r = read_from_path(r_path, image_size, transform_name="resize_crop")            r_x = vae.encode(r.unsqueeze(0).to(vae.device, vae.dtype))            r_x = r_x.squeeze(0)            ref.append(r_x)        refs_x.append(ref)    # refs_x: [batch, ref_num, C, T, H, W]    return refs_x​​def process_mask_strategy(mask_strategy):    mask_batch = []    mask_strategy = mask_strategy.split(";")    for mask in mask_strategy:        mask_group = mask.split(",")        assert len(mask_group) >= 1 and len(mask_group) <= 6, f"Invalid mask strategy: {mask}"        if len(mask_group) == 1:            mask_group.extend(["0", "0", "0", "1", "0"])        elif len(mask_group) == 2:            mask_group.extend(["0", "0", "1", "0"])        elif len(mask_group) == 3:            mask_group.extend(["0", "1", "0"])        elif len(mask_group) == 4:            mask_group.extend(["1", "0"])        elif len(mask_group) == 5:            mask_group.append("0")        mask_batch.append(mask_group)    return mask_batch​​def apply_mask_strategy(z, refs_x, mask_strategys, loop_i):    masks = []    for i, mask_strategy in enumerate(mask_strategys):        mask = torch.ones(z.shape[2], dtype=torch.float, device=z.device)        if mask_strategy is None:            masks.append(mask)            continue        mask_strategy = process_mask_strategy(mask_strategy)        for mst in mask_strategy:            loop_id, m_id, m_ref_start, m_target_start, m_length, edit_ratio = mst            loop_id = int(loop_id)            if loop_id != loop_i:                continue            m_id = int(m_id)            m_ref_start = int(m_ref_start)            m_length = int(m_length)            m_target_start = int(m_target_start)            edit_ratio = float(edit_ratio)            ref = refs_x[i][m_id]  # [C, T, H, W]            if m_ref_start < 0:                m_ref_start = ref.shape[1] + m_ref_start            if m_target_start < 0:                # z: [B, C, T, H, W]                m_target_start = z.shape[2] + m_target_start            z[i, :, m_target_start : m_target_start + m_length] = ref[:, m_ref_start : m_ref_start + m_length]            mask[m_target_start : m_target_start + m_length] = edit_ratio        masks.append(mask)    masks = torch.stack(masks)    return masks​​def process_prompts(prompts, num_loop):    from opensora.models.text_encoder.t5 import text_preprocessing​    ret_prompts = []    for prompt in prompts:        if prompt.startswith("|0|"):            prompt_list = prompt.split("|")[1:]            text_list = []            for i in range(0, len(prompt_list), 2):                start_loop = int(prompt_list[i])                text = prompt_list[i + 1]                text = text_preprocessing(text)                end_loop = int(prompt_list[i + 2]) if i + 2 < len(prompt_list) else num_loop                text_list.extend([text] * (end_loop - start_loop))            assert len(text_list) == num_loop, f"Prompt loop mismatch: {len(text_list)} != {num_loop}"            ret_prompts.append(text_list)        else:            prompt = text_preprocessing(prompt)            ret_prompts.append([prompt] * num_loop)    return ret_prompts​​def extract_json_from_prompts(prompts):    additional_infos = []    ret_prompts = []    for prompt in prompts:        parts = re.split(r"(?=[{\[])", prompt)        assert len(parts) <= 2, f"Invalid prompt: {prompt}"        ret_prompts.append(parts[0])        if len(parts) == 1:            additional_infos.append({})        else:            additional_infos.append(json.loads(parts[1]))    return ret_prompts, additional_infos​​# ============================# Model-related# ============================def read_config(config_path):    """    Read the configuration file.    """    from mmengine.config import Config​    return Config.fromfile(config_path)​​def build_models(model_type, config, enable_optimization=False):    """    Build the models for the given model type and configuration.    """    # build vae    from opensora.registry import MODELS, build_module​    vae = build_module(config.vae, MODELS).cuda()​    # build text encoder    text_encoder = build_module(config.text_encoder, MODELS)  # T5 must be fp32    text_encoder.t5.model = text_encoder.t5.model.cuda()​    # build stdit    # we load model from HuggingFace directly so that we don't need to    # handle model download logic in HuggingFace Space    from opensora.models.stdit.stdit2 import STDiT2​    stdit = STDiT2.from_pretrained(        HF_STDIT_MAP[model_type],        enable_flash_attn=enable_optimization,        trust_remote_code=True,    ).cuda()​    # build scheduler    from opensora.registry import SCHEDULERS​    scheduler = build_module(config.scheduler, SCHEDULERS)​    # hack for classifier-free guidance    text_encoder.y_embedder = stdit.y_embedder​    # move modelst to device    vae = vae.to(torch.float16).eval()    text_encoder.t5.model = text_encoder.t5.model.eval()  # t5 must be in fp32    stdit = stdit.to(torch.float16).eval()​    # clear cuda    torch.cuda.empty_cache()    return vae, text_encoder, stdit, scheduler​​def parse_args():    parser = argparse.ArgumentParser()    parser.add_argument(        "--model-type",        default="v1.1-stage3",        choices=MODEL_TYPES,        help=f"The type of model to run for the Gradio App, can only be {MODEL_TYPES}",    )    parser.add_argument("--output", default="./outputs", type=str, help="The path to the output folder")    parser.add_argument("--port", default=None, type=int, help="The port to run the Gradio App on.")    parser.add_argument("--host", default=None, type=str, help="The host to run the Gradio App on.")    parser.add_argument("--share", action="store_true", help="Whether to share this gradio demo.")    parser.add_argument(        "--enable-optimization",        action="store_true",        help="Whether to enable optimization such as flash attention and fused layernorm",    )    return parser.parse_args()​​# ============================# Main Gradio Script# ============================# as `run_inference` needs to be wrapped by `spaces.GPU` and the input can only be the prompt text# so we can't pass the models to `run_inference` as arguments.# instead, we need to define them globally so that we can access these models inside `run_inference`​# read configargs = parse_args()config = read_config(CONFIG_MAP[args.model_type])​# make outputs diros.makedirs(args.output, exist_ok=True)​# disable torch jit as it can cause failure in gradio SDK# gradio sdk uses torch with cuda 11.3torch.jit._state.disable()​# import after installationfrom opensora.datasets import IMG_FPS, save_samplefrom opensora.utils.misc import to_torch_dtype​# some global variablesdtype = to_torch_dtype(config.dtype)device = torch.device("cuda")​# build modelvae, text_encoder, stdit, scheduler = build_models(args.model_type, config, enable_optimization=args.enable_optimization)​​def run_inference(mode, prompt_text, resolution, aspect_ratio, length, reference_image, seed, sampling_steps, cfg_scale):    torch.manual_seed(seed)    with torch.inference_mode():        # ======================        # 1. Preparation        # ======================        # parse the inputs        resolution = RESOLUTION_MAP[resolution][aspect_ratio]​        # gather args from config        num_frames = config.num_frames        frame_interval = config.frame_interval        fps = config.fps        condition_frame_length = config.condition_frame_length​        # compute number of loops        if mode == "Text2Image":            num_frames = 1            num_loop = 1        else:            num_seconds = int(length.rstrip('s'))            if num_seconds <= 16:                num_frames = num_seconds * fps // frame_interval                num_loop = 1            else:                config.num_frames = 16                total_number_of_frames = num_seconds * fps / frame_interval                num_loop = math.ceil((total_number_of_frames - condition_frame_length) / (num_frames - condition_frame_length))​        # prepare model args        if config.num_frames == 1:            fps = IMG_FPS​        model_args = dict()        height_tensor = torch.tensor([resolution[0]], device=device, dtype=dtype)        width_tensor = torch.tensor([resolution[1]], device=device, dtype=dtype)        num_frames_tensor = torch.tensor([num_frames], device=device, dtype=dtype)        ar_tensor = torch.tensor([resolution[0] / resolution[1]], device=device, dtype=dtype)        fps_tensor = torch.tensor([fps], device=device, dtype=dtype)        model_args["height"] = height_tensor        model_args["width"] = width_tensor        model_args["num_frames"] = num_frames_tensor        model_args["ar"] = ar_tensor        model_args["fps"] = fps_tensor​        # compute latent size        input_size = (num_frames, *resolution)        latent_size = vae.get_latent_size(input_size)​        # process prompt        prompt = zh2en(prompt_text)[0].get("translation_text")        prompt_raw = [prompt]        print(prompt_raw)        prompt_raw, _ = extract_json_from_prompts(prompt_raw)        prompt_loops = process_prompts(prompt_raw, num_loop)        video_clips = []​        # prepare mask strategy        if mode == "Text2Image":            mask_strategy = [None]        elif mode == "Text2Video":            if reference_image is not None:                mask_strategy = ['0']            else:                mask_strategy = [None]        else:            raise ValueError(f"Invalid mode: {mode}")​        # =========================        # 2. Load reference images        # =========================        if mode == "Text2Image":            refs_x = collect_references_batch([None], vae, resolution)        elif mode == "Text2Video":            if reference_image is not None:                # save image to disk                from PIL import Image                im = Image.fromarray(reference_image)​                with NamedTemporaryFile(suffix=".jpg") as temp_file:                    im.save(temp_file.name)                    refs_x = collect_references_batch([temp_file.name], vae, resolution)            else:                refs_x = collect_references_batch([None], vae, resolution)        else:            raise ValueError(f"Invalid mode: {mode}")​        # 4.3. long video generation        for loop_i in range(num_loop):            # 4.4 sample in hidden space            batch_prompts = [prompt[loop_i] for prompt in prompt_loops]            z = torch.randn(len(batch_prompts), vae.out_channels, *latent_size, device=device, dtype=dtype)​            # 4.5. apply mask strategy            masks = None​            # if cfg.reference_path is not None:            if loop_i > 0:                ref_x = vae.encode(video_clips[-1])                for j, refs in enumerate(refs_x):                    if refs is None:                        refs_x[j] = [ref_x[j]]                    else:                        refs.append(ref_x[j])                    if mask_strategy[j] is None:                        mask_strategy[j] = ""                    else:                        mask_strategy[j] += ";"                    mask_strategy[                        j                    ] += f"{loop_i},{len(refs)-1},-{condition_frame_length},0,{condition_frame_length}"​            masks = apply_mask_strategy(z, refs_x, mask_strategy, loop_i)​            # 4.6. diffusion sampling            # hack to update num_sampling_steps and cfg_scale            scheduler_kwargs = config.scheduler.copy()            scheduler_kwargs.pop('type')            scheduler_kwargs['num_sampling_steps'] = sampling_steps            scheduler_kwargs['cfg_scale'] = cfg_scale​            scheduler.__init__(                **scheduler_kwargs            )            samples = scheduler.sample(                stdit,                text_encoder,                z=z,                prompts=batch_prompts,                device=device,                additional_args=model_args,                mask=masks,  # scheduler must support mask            )            samples = vae.decode(samples.to(dtype))            video_clips.append(samples)​            # 4.7. save video            if loop_i == num_loop - 1:                video_clips_list = [                    video_clips[0][0]] + [video_clips[i][0][:, config.condition_frame_length :]                     for i in range(1, num_loop)                ]                video = torch.cat(video_clips_list, dim=1)                current_datetime = datetime.datetime.now()                timestamp = current_datetime.timestamp()                save_path = os.path.join(args.output, f"output_{timestamp}")                saved_path = save_sample(video, save_path=save_path, fps=config.fps // config.frame_interval)                return saved_path​@spaces.GPU(duration=200)def run_image_inference(prompt_text, resolution, aspect_ratio, length, reference_image, seed, sampling_steps, cfg_scale):    return run_inference("Text2Image", prompt_text, resolution, aspect_ratio, length, reference_image, seed, sampling_steps, cfg_scale)​@spaces.GPU(duration=200)def run_video_inference(prompt_text, resolution, aspect_ratio, length, reference_image, seed, sampling_steps, cfg_scale):    return run_inference("Text2Video", prompt_text, resolution, aspect_ratio, length, reference_image, seed, sampling_steps, cfg_scale)​​def main():    # create demo    with gr.Blocks() as demo:        with gr.Row():            with gr.Column():                gr.HTML("""<h1 align="center">Open-Sora 1.1</h1>""")​        with gr.Row():            with gr.Column():                prompt_text = gr.Textbox(                    label="Prompt",                    placeholder="请输入中文提示词",                    lines=4,                )                resolution = gr.Radio(                     choices=["144p", "240p", "360p", "480p", "720p"],                     value="240p",                    label="Resolution",                 )                aspect_ratio = gr.Radio(                     choices=["9:16", "16:9", "3:4", "4:3", "1:1"],                     value="9:16",                    label="Aspect Ratio (H:W)",                 )                length = gr.Radio(                    choices=["2s", "4s", "8s", "16s"],                     value="2s",                    label="Video Length (only effective for video generation)",                     info="8s may fail as Hugging Face ZeroGPU has the limitation of max 200 seconds inference time."                )​                with gr.Row():                    seed = gr.Slider(                        value=1024,                        minimum=1,                        maximum=2048,                        step=1,                        label="Seed"                    )​                    sampling_steps = gr.Slider(                        value=100,                        minimum=1,                        maximum=200,                        step=1,                        label="Sampling steps"                    )                    cfg_scale = gr.Slider(                        value=7.0,                        minimum=0.0,                        maximum=10.0,                        step=0.1,                        label="CFG Scale"                    )                                reference_image = gr.Image(                    label="Reference Image (Optional)",                )                        with gr.Column():                output_video = gr.Video(                    label="Output Video",                    height="100%"                )​        with gr.Row():             image_gen_button = gr.Button("Generate image")             video_gen_button = gr.Button("Generate video")        ​        image_gen_button.click(             fn=run_image_inference,              inputs=[prompt_text, resolution, aspect_ratio, length, reference_image, seed, sampling_steps, cfg_scale],              outputs=reference_image             )        video_gen_button.click(             fn=run_video_inference,              inputs=[prompt_text, resolution, aspect_ratio, length, reference_image, seed, sampling_steps, cfg_scale],              outputs=output_video             )​    # launch    demo.launch(share=True, inbrowser=True)    ​if __name__ == "__main__":    main()

Writing gradio/app-ref.py

!python gradio/app-ref.py

/home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()  return self.fget.__get__(instance, owner)()/home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/transformers/models/marian/tokenization_marian.py:197: UserWarning: Recommended: pip install sacremoses.  warnings.warn("Recommended: pip install sacremoses.")/home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/colossalai/shardformer/layer/normalization.py:45: UserWarning: Please install apex from source (https://github.com/NVIDIA/apex) to use the fused layernorm kernel  warnings.warn("Please install apex from source (https://github.com/NVIDIA/apex) to use the fused layernorm kernel")Loading checkpoint shards: 100%|██████████████████| 2/2 [00:32<00:00, 16.15s/it]/home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.  warnings.warn(Running on local URL:  http://127.0.0.1:7860Running on public URL: https://64147712240bbb3753.gradio.live​This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)