本文分享自华为云社区《重磅！【支持中文】stable-diffusion-3安装部署教程-SD3 来了》，作者：码上开花_Lancer。
正如承诺的那样，Stability AI 在 6 月 12 日正式开源了 Stable Diffusion 3（Medium 版本）！不愧是 AI 生图领域的“开源英雄”。最近一段时间，正当所有人都在为 OpenAI 发布 Sora 狂欢时，Stability AI 更是推出了 Stable Diffusion 3 的技术报告。这两项技术不约而同都采用了 Diffusion Transformer 的架构设计。
值得注意的是，Stable Diffusion 3 的强大性能其实并不仅限于 Diffusion Transformer 在架构上所带来的增益，其在提示词、图像质量、文字拼写方面的能力都得到了极大的提升。那么究竟是什么让 Stable Diffusion 3 如此强大？今天我们就从 Stable Diffusion 3 的技术报告中解读 stable diffusion 3 强大背后的技术原理。
接下来就讲讲，怎么在本地部署最新的 Stable Diffusion 3，大致分为以下几步（开始操作前，请确保你有“畅通”的网络）：
一、前期准备
1.登录华为云官方账号：
点击右上角“控制台”，搜索栏输入“ModelArts”
点击“开发环境”-“notebook”,“创建”：
进入创建 notebook，名称“notebook-LangChain”,选择 GPU 规格，“GPU: 1*T4(16GB)|CPU: 8 核 32GB”，点击“立即创建”，磁盘规格选择“50G”,点击“创建”
点击返回“任务中心”，点击 notebook 进入
以上步骤是从 ModelArts 上自己创建 notebook，也可以直接点击案例进入体验--stable-diffusion-3重磅来袭。
二、下载模型
Stable Diffusion 3 Medium 是一种多模态扩散转换器 (MMDiT) 文本到图像模型，其特点是在图像质量、排版、复杂提示理解和资源效率方面大大提高了性能。有关更多技术细节，请参阅研究报告。
🔹 本案例需使用 Pytorch-2.0.1 GPU-V100 及以上规格运行
🔹 点击 Run in ModelArts，将会进入到 ModelArts CodeLab 中，这时需要你登录华为云账号，如果没有账号，则需要注册一个，且要进行实名认证，参考《如何创建华为云账号并且实名认证》 即可完成账号注册和实名认证。 登录之后，等待片刻，即可进入到 CodeLab 的运行环境
🔹 出现 Out Of Memory ，请检查是否为您的参数配置过高导致，修改参数配置，重启 kernel 或更换更高规格资源进行规避❗❗❗
首先切换 kernrl
1. 下载代码和模型
import osimport moxing as mox
if not os.path.exists('opus-mt-zh-en'): mox.file.copy_parallel('obs://modelarts-labs-bj4-v2/course/ModelBox/opus-mt-zh-en', 'opus-mt-zh-en')
if not os.path.exists('stable-diffusion-3-medium-diffusers'): mox.file.copy_parallel('obs://modelbox-course/stable-diffusion-3-medium-diffusers','stable-diffusion-3-medium-diffusers') if not os.path.exists('/home/ma-user/work/frpc_linux_amd64'): mox.file.copy_parallel('obs://modelarts-labs-bj4-v2/course/ModelBox/frpc_linux_amd64', '/home/ma-user/work/frpc_linux_amd64')
复制代码
INFO:root:Using MoXing-v2.1.0.5d9c87c8-5d9c87c8
INFO:root:Using OBS-Python-SDK-3.20.9.1
复制代码
import osimport moxing as moxfrom PIL import Image,ImageDraw,ImageFont,ImageFilter
# 导入海报需要的素材if not os.path.exists("/home/ma-user/work/Style"): mox.file.copy_parallel('obs://modelarts-labs-bj4-v2/case_zoo/StableDiffusion/Style/AI_paint.jpg',"/home/ma-user/work/Style/AI_paint.jpg") mox.file.copy_parallel('obs://modelarts-labs-bj4-v2/case_zoo/StableDiffusion/Style/方正兰亭准黑_GBK.ttf',"/home/ma-user/work/Style/方正兰亭准黑_GBK.ttf") if os.path.exists("/home/ma-user/work/material"): print('Download success') else: raise Exception('Download Failed')else: print("Project already exists")
复制代码
2. 配置运行环境
本案例依赖 Python-3.9.15 及以上环境，因此我们首先创建虚拟环境：
!/home/ma-user/anaconda3/bin/conda clean -i!/home/ma-user/anaconda3/bin/conda create -n python-3.9.15 python=3.9.15 -y --override-channels --channel https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main!/home/ma-user/anaconda3/envs/python-3.9.15/bin/pip install ipykernel
复制代码
/home/ma-user/anaconda3/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported versi
RequestsDependencyWarning)
/home/ma-user/anaconda3/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m808.2/808.2 kB[0m [31m11.1 MB/s[0m eta [36m0:00:00[0m00:01[0m
[?25hCollecting jupyter-client>=6.1.12 (from ipykernel)
Successfully installed asttokens-2.4.1 comm-0.2.2 debugpy-1.8.2 decorator-5.1.1 exceptiongroup-1.2.1 executing-2.0.1 importlib-metadata-8.0.0 ipykernel-6.29.5 ipython-8.18.1 jedi-0.19.1 jupyter-client-8.6.2 jupyter-core-5.7.2 matplotlib-inline-0.1.7 nest-asyncio-1.6.0 packaging-24.1 parso-0.8.4 pexpect-4.9.0 platformdirs-4.2.2 prompt-toolkit-3.0.47 psutil-6.0.0 ptyprocess-0.7.0 pure-eval-0.2.2 pygments-2.18.0 python-dateutil-2.9.0.post0 pyzmq-26.0.3 six-1.16.0 stack-data-0.6.3 tornado-6.4.1 traitlets-5.14.3 typing-extensions-4.12.2 wcwidth-0.2.13 zipp-3.19.2
复制代码
import jsonimport os
data = { "display_name": "python-3.9.15", "env": { "PATH": "/home/ma-user/anaconda3/envs/python-3.9.15/bin:/home/ma-user/anaconda3/envs/python-3.7.10/bin:/modelarts/authoring/notebook-conda/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/ma-user/modelarts/ma-cli/bin:/home/ma-user/modelarts/ma-cli/bin:/home/ma-user/anaconda3/envs/PyTorch-1.8/bin" }, "language": "python", "argv": [ "/home/ma-user/anaconda3/envs/python-3.9.15/bin/python", "-m", "ipykernel", "-f", "{connection_file}" ]}
if not os.path.exists("/home/ma-user/anaconda3/share/jupyter/kernels/python-3.9.15/"): os.mkdir("/home/ma-user/anaconda3/share/jupyter/kernels/python-3.9.15/")
with open('/home/ma-user/anaconda3/share/jupyter/kernels/python-3.9.15/kernel.json', 'w') as f: json.dump(data, f, indent=4)
复制代码
创建完成后，稍等片刻，或刷新页面，点击右上角 kernel 选择 python-3.9.15
查看 Python 版本
查看 GPU 型号，至少需要 32GB 显存
Wed Jul 10 23:52:26 2024 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Tesla V100-PCIE... On | 00000000:00:0D.0 Off | 0 | | N/A 30C P0 25W / 250W | 0MiB / 32510MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+
复制代码
安装 SD3 依赖包
!pip install --upgrade pip!pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 !pip install diffusers transformers sentencepiece accelerate protobuf gradio spaces!cp /home/ma-user/work/frpc_linux_amd64 /home/ma-user/anaconda3/envs/python-3.9.15/lib/python3.9/site-packages/gradio/frpc_linux_amd64_v0.2!chmod +x /home/ma-user/anaconda3/envs/python-3.9.15/lib/python3.9/site-packages/gradio/frpc_linux_amd64_v0.2
复制代码
Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple
Requirement already satisfied: pip in /home/ma-user/anaconda3/envs/python-3.9.15/lib/python3.9/site-packages (24.0)
Collecting pip
Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/e7/54/0c1c068542cee73d8863336e974fc881e608d0170f3af15d0c0f28644531/pip-24.1.2-py3-none-any.whl (1.8 MB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m1.8/1.8 MB[0m [31m28.5 MB/s[0m eta [36m0:00:00[0m00:01[0m
[?25hInstalling collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 24.0
Uninstalling pip-24.0:
Successfully uninstalled pip-24.0
Successfully installed pip-24.1.2Successfully installed accelerate-0.32.1 aiofiles-23.2.1 altair-5.3.0 annotated-types-0.7.0 anyio-4.4.0 attrs-23.2.0 click-8.1.7 contourpy-1.2.1 cycler-0.12.1 diffusers-0.29.2 dnspython-2.6.1 email_validator-2.2.0 fastapi-0.111.0 fastapi-cli-0.0.4 ffmpy-0.3.2 fonttools-4.53.1 fsspec-2024.6.1 gradio-4.37.2 gradio-client-1.0.2 h11-0.14.0 httpcore-1.0.5 httptools-0.6.1 httpx-0.27.0 huggingface-hub-0.23.4 importlib-resources-6.4.0 jsonschema-4.23.0 jsonschema-specifications-2023.12.1 kiwisolver-1.4.5 markdown-it-py-3.0.0 matplotlib-3.9.1 mdurl-0.1.2 numpy-1.26.4 orjson-3.10.6 pandas-2.2.2 protobuf-5.27.2 psutil-5.9.8 pydantic-2.8.2 pydantic-core-2.20.1 pydub-0.25.1 pyparsing-3.1.2 python-dotenv-1.0.1 python-multipart-0.0.9 pytz-2024.1 pyyaml-6.0.1 referencing-0.35.1 regex-2024.5.15 rich-13.7.1 rpds-py-0.19.0 ruff-0.5.1 safetensors-0.4.3 semantic-version-2.10.0 sentencepiece-0.2.0 shellingham-1.5.4 sniffio-1.3.1 spaces-0.28.3 starlette-0.37.2 tokenizers-0.19.1 tomlkit-0.12.0 toolz-0.12.1 tqdm-4.66.4 transformers-4.42.3 typer-0.12.3 tzdata-2024.1 ujson-5.10.0 uvicorn-0.30.1 uvloop-0.19.0 watchfiles-0.22.0 websockets-11.0.3
复制代码
3. 生成单张图像
#@title 填写英文提示词 import torchfrom diffusers import StableDiffusion3Pipeline
# 清理 GPU 缓存torch.cuda.empty_cache()
# 确保使用半精度浮点数torch_dtype = torch.float16
# 尝试减少推理步骤num_inference_steps = 20
# 调整引导比例guidance_scale = 5.0
# 定义 Promptprompt = "cinematic photo of a red apple on a table in a classroom, on the blackboard are the words go big or go home written in chalk" #@param {type:"string"}
# 加载模型并将其移动到 GPUpipe = StableDiffusion3Pipeline.from_pretrained("stable-diffusion-3-medium-diffusers", torch_dtype=torch_dtype).to("cuda")
# 根据提供的 Prompt 生成图像image = pipe(prompt=prompt, num_inference_steps=num_inference_steps, guidance_scale=guidance_scale).images[0]
# 定义保存图像的路径save_path = '/home/ma-user/work/your_generated_image.png'
# 保存图像到指定路径image.save(save_path)
# 如果需要在本地查看图像，可以使用 show 方法image.show()prompt = "cinematic photo of a red apple on a table in a classroom, on the blackboard are the words go big or go home written in chalk" #@param {type:"string"}
复制代码
/home/ma-user/anaconda3/envs/python-3.9.15/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm Loading pipeline components...: 33%|███▎ | 3/9 [00:00<00:00, 7.87it/s]You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers Loading pipeline components...: 44%|████▍ | 4/9 [00:00<00:00, 5.87it/s] Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s][A Loading checkpoint shards: 50%|█████ | 1/2 [00:00<00:00, 3.92it/s][A Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 3.95it/s][A Loading pipeline components...: 100%|██████████| 9/9 [00:02<00:00, 3.06it/s] 100%|██████████| 20/20 [00:08<00:00, 2.27it/s]
复制代码
注意：
出现 Out Of Memory ，尝试重启 kernel 再次运行❗❗❗
4.填写作品名称和作者姓名
#@title 填写作品名称和作者姓名 from PIL import Image, ImageDraw, ImageFont, ImageFilter
def gen_poster(img, txt1, txt2, path, zt): # 定义字体和颜色 font1 = ImageFont.truetype(zt, 30) font2 = ImageFont.truetype(zt, 25) # 创建一个可以在图像上绘制的 Draw 对象 img_draw = ImageDraw.Draw(img) # 在图像上绘制文本 img_draw.text((180, 860), txt1, font=font1, fill='#961900') img_draw.text((130, 903), txt2, font=font2, fill='#252b3a') # 保存图像 img.save(path)
# 定义模板图像路径和字体路径template_img = "/home/ma-user/work/Style/AI_paint.jpg"zt = r"/home/ma-user/work/Style/方正兰亭准黑_GBK.ttf"
# 打开模板图像temp_image = Image.open(template_img).convert("RGBA")
# 打开生成的图像image_path = "/home/ma-user/work/your_generated_image.png" # 替换为你生成的图像路径image = Image.open(image_path)
# 计算新的大小以适应模板图像的宽度，同时保持图片的原始比例width_ratio = temp_image.width / image.widthnew_height = int(image.height * width_ratio)new_size = (temp_image.width, new_height)
# 调整生成的图像大小，使用 LANCZOS 重采样算法image = image.resize(new_size, Image.Resampling.LANCZOS)
# 粘贴调整大小后的图像到模板上# 假设图像粘贴的起始点是 (40, 266)temp_image.paste(image, (40, 266))
# 定义作品名称和作者姓名title_char = "苹果" #@param {type:"string"}author_char = "ModelArts" #@param {type:"string"}
# 定义保存海报的路径savepath = '/home/ma-user/work/AI_paint_output.png' # 确保路径正确，并且有写权限
# 调用函数生成海报gen_poster(temp_image, title_char, author_char, savepath, zt)
# 使用 Image.open 来打开并显示生成的海报Image.open(savepath).show()
复制代码
5. 运行 Gradio 应用
with gr.Blocks(css=css) as demo: gr.HTML("""<h1 align="center">Stable Diffusion 3</h1>""") with gr.Column(elem_id="col-container"): with gr.Row(): prompt = gr.Text( label="提示词", show_label=False, max_lines=1, placeholder="请输入中文提示词", container=False, ) run_button = gr.Button("生成", scale=0) result = gr.Image(label="Result", show_label=False) with gr.Accordion("更多参数", open=False): negative_prompt = gr.Text( label="负面提示词", max_lines=1, placeholder="请输入负面提示词", ) seed = gr.Slider( label="Seed", minimum=0, maximum=MAX_SEED, step=1, value=0, ) randomize_seed = gr.Checkbox(label="随机种子", value=True) with gr.Row(): width = gr.Slider( label="宽", minimum=256, maximum=MAX_IMAGE_SIZE, step=64, value=1024, ) height = gr.Slider( label="高", minimum=256, maximum=MAX_IMAGE_SIZE, step=64, value=1024, ) with gr.Row(): guidance_scale = gr.Slider( label="Guidance scale", minimum=0.0, maximum=10.0, step=0.1, value=5.0, ) num_inference_steps = gr.Slider( label="迭代步数", minimum=1, maximum=50, step=1, value=28, ) gr.on( triggers=[run_button.click, prompt.submit, negative_prompt.submit], fn = infer, inputs = [prompt, negative_prompt, seed, randomize_seed, width, height, guidance_scale, num_inference_steps], outputs = [result, seed] ) demo.launch(share=True)
复制代码
运行 Gradio 应用，运行成功后点击 Running on public URL 后的网页链接即可体验！
Loading pipeline components...: 56%|███████▏ | 5/9 [00:02<00:01, 2.28it/s]You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers
Loading pipeline components...: 67%|████████▋ | 6/9 [00:02<00:01, 2.61it/s]
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s][A
Loading checkpoint shards: 50%|█████████ | 1/2 [00:00<00:00, 3.54it/s][A
Loading checkpoint shards: 100%|██████████████████| 2/2 [00:00<00:00, 3.53it/s][A
Loading pipeline components...: 100%|█████████████| 9/9 [00:03<00:00, 2.83it/s]
/home/ma-user/anaconda3/envs/python-3.9.15/lib/python3.9/site-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
return self.fget.__get__(instance, owner)()
/home/ma-user/anaconda3/envs/python-3.9.15/lib/python3.9/site-packages/transformers/models/marian/tokenization_marian.py:175: UserWarning: Recommended: pip install sacremoses.
warnings.warn("Recommended: pip install sacremoses.")
Hardware accelerator e.g. GPU is available in the environment, but no `device` argument is passed to the `Pipeline` object. Model will be on CPU.
Running on local URL: http://127.0.0.1:7860
Running on public URL: https://9c48446865ca38cc99.gradio.live
复制代码
This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)
一幅画的是一位宇航员骑着一只穿着芭蕾舞裙的猪，手里拿着一把粉红色的伞，猪旁边的地上是一只戴着大礼帽的知更鸟，角落里写着“稳定扩散”的字样。
复制代码
A picture of an astronaut riding on a pig in a ballet dress with a pink umbrella next to a big hat on the ground, with the word “stable spread” in the corner.
出现 Out Of Memory ，尝试重启 kernel 再次运行❗❗❗
浏览器打开 local URL: http://127.0.0.1:7860 地址
运行界面：
三、其他案例展示
Prompt: cinematic photo of a red apple on a table in a classroom, on the blackboard are the words "go big or go home" written in chalk
提示:教室里的桌子上有一个红苹果的电影照片，黑板上用粉笔写着“要么做大，要么回家”
Prompt: a painting of an astronaut riding a pig wearing a tutu holding a pink umbrella, on the ground next to the pig is a robin bird wearing a top hat, in the corner are the words "stable diffusion"
提示:一幅画的是一位宇航员骑着一只穿着芭蕾舞裙的猪，手里拿着一把粉红色的伞，猪旁边的地上是一只戴着大礼帽的知更鸟，角落里写着“稳定扩散”的字样。
Prompt: Three transparent glass bottles on a wooden table. The one on the left has red liquid and the number 1. The one in the middle has blue liquid and the number 2. The one on the right has green liquid and the number 3.
提示:三个透明玻璃瓶放在木桌上。左边的是红色液体和数字 1。中间有蓝色液体和数字 2。右边的是绿色液体和数字 3。
参考
点击关注，第一时间了解华为云新鲜技术~
评论