OA0

OA0 是一个探索 AI 的社区

现在注册

已注册用户请登录

🚀 Janus-Series: 统一的多模态理解与生成模型

📥 模型下载 | ⚡ 快速开始 | 📜 许可证 | 📖 引用
🤗 在线体验（Janus-Pro-7B, Janus, JanusFlow）

新闻

2025.01.27: Janus-Pro 发布，它是 Janus 的进阶版本，在多模态理解与视觉生成方面均有显著提升。详见论文

2024.11.13: JanusFlow 发布，一种融合了修正流（Rectified Flow）的新型统一图像生成模型。详见论文、在线体验及使用指南。

2024.10.23: 用于复现论文中多模态理解结果的评估代码已集成到 VLMEvalKit 中。请参考此链接。

2024.10.20: (1) 修复了 tokenizer_config.json 中的一个错误。旧版本导致无分类器指导（CFG）功能异常，图像生成质量较差。(2) 发布 Gradio 演示（在线演示和本地）。

1. 简介

Janus-Pro: 通过数据和模型规模扩展实现统一多模态理解与生成

Janus-Pro 是之前工作 Janus 的进阶版本。具体来说，Janus-Pro 包含了 (1) 优化的训练策略，(2) 扩大的训练数据，以及 (3) 更大的模型规模。通过这些改进，Janus-Pro 在多模态理解与文本到图像的指令遵循能力方面均取得了显著进步，同时增强了文本到图像生成的稳定性。

Janus: 通过解耦视觉编码实现统一多模态理解与生成

Janus 是一种新颖的自回归框架，能够统一多模态的理解与生成。它通过将视觉编码解耦为独立的路径来解决先前方法的局限性，同时仍然使用单一、统一的 Transformer 架构进行处理。这种解耦不仅减轻了视觉编码器在理解和生成角色之间的冲突，也增强了框架的灵活性。Janus 超越了先前的统一模型，其性能可媲美甚至超过特定任务的模型。其简单性、高灵活性和有效性使其成为下一代统一多模态模型的强有力候选方案。

JanusFlow: 协调自回归模型与修正流，实现统一多模态理解与生成

JanusFlow 引入了一种极简架构，将自回归语言模型与修正流（生成建模中最先进的方法之一）相结合。我们的关键发现表明，可以在大型语言模型框架内直接训练修正流，无需复杂的架构修改。广泛的实验表明，JanusFlow 在其各自领域的性能可与专门的模型相媲美甚至更优，同时在标准基准测试上显著优于现有的统一方法。这项工作代表了向更高效、更通用的视觉-语言模型迈进了一步。

2. 模型下载

我们将 Janus 系列模型公开发布，以支持学术界和商业界更广泛、更多样化的研究。
请注意，此模型的使用需遵守许可证部分列出的条款。在这些条款下，允许商业用途。

Huggingface

模型	序列长度	下载
Janus-1.3B	4096	🤗 Hugging Face
JanusFlow-1.3B	4096	🤗 Hugging Face
Janus-Pro-1B	4096	🤗 Hugging Face
Janus-Pro-7B	4096	🤗 Hugging Face

3. 快速开始

Janus-Pro

### 安装在 `Python >= 3.8` 环境下，运行以下命令安装所需依赖：

pip install -e .

### 简单推理示例 #### 多模态理解


import torch
from transformers import AutoModelForCausalLM
from janus.models import MultiModalityCausalLM, VLChatProcessor
from janus.utils.io import load_pil_images

# 指定模型路径
model_path = "deepseek-ai/Janus-Pro-7B"
vl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)
tokenizer = vl_chat_processor.tokenizer

vl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained(
    model_path, trust_remote_code=True
)
vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()

conversation = [
    {
        "role": "<|User|>",
        "content": f"<image_placeholder>\n{question}",
        "images": [image],
    },
    {"role": "<|Assistant|>", "content": ""},
]

# 加载图像并准备输入
pil_images = load_pil_images(conversation)
prepare_inputs = vl_chat_processor(
    conversations=conversation, images=pil_images, force_batchify=True
).to(vl_gpt.device)

# # 运行图像编码器获取图像嵌入
inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)

# # 运行模型获取响应
outputs = vl_gpt.language_model.generate(
    inputs_embeds=inputs_embeds,
    attention_mask=prepare_inputs.attention_mask,
    pad_token_id=tokenizer.eos_token_id,
    bos_token_id=tokenizer.bos_token_id,
    eos_token_id=tokenizer.eos_token_id,
    max_new_tokens=512,
    do_sample=False,
    use_cache=True,
)

answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True)
print(f"{prepare_inputs['sft_format'][0]}", answer)

#### 文本到图像生成

import os
import PIL.Image
import torch
import numpy as np
from transformers import AutoModelForCausalLM
from janus.models import MultiModalityCausalLM, VLChatProcessor


# 指定模型路径
model_path = "deepseek-ai/Janus-Pro-7B"
vl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)
tokenizer = vl_chat_processor.tokenizer

vl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained(
    model_path, trust_remote_code=True
)
vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()

conversation = [
    {
        "role": "<|User|>",
        "content": "一位来自喀布尔的惊艳公主，身着红白传统服饰，蓝眼睛，棕色头发",
    },
    {"role": "<|Assistant|>", "content": ""},
]

sft_format = vl_chat_processor.apply_sft_template_for_multi_turn_prompts(
    conversations=conversation,
    sft_format=vl_chat_processor.sft_format,
    system_prompt="",
)
prompt = sft_format + vl_chat_processor.image_start_tag


@torch.inference_mode()
def generate(
    mmgpt: MultiModalityCausalLM,
    vl_chat_processor: VLChatProcessor,
    prompt: str,
    temperature: float = 1,
    parallel_size: int = 16,
    cfg_weight: float = 5,
    image_token_num_per_image: int = 576,
    img_size: int = 384,
    patch_size: int = 16,
):
    input_ids = vl_chat_processor.tokenizer.encode(prompt)
    input_ids = torch.LongTensor(input_ids)

    tokens = torch.zeros((parallel_size*2, len(input_ids)), dtype=torch.int).cuda()
    for i in range(parallel_size*2):
        tokens[i, :] = input_ids
        if i % 2 != 0:
            tokens[i, 1:-1] = vl_chat_processor.pad_id

    inputs_embeds = mmgpt.language_model.get_input_embeddings()(tokens)

    generated_tokens = torch.zeros((parallel_size, image_token_num_per_image), dtype=torch.int).cuda()

    for i in range(image_token_num_per_image):
        outputs = mmgpt.language_model.model(inputs_embeds=inputs_embeds, use_cache=True, past_key_values=outputs.past_key_values if i != 0 else None)
        hidden_states = outputs.last_hidden_state

        logits = mmgpt.gen_head(hidden_states[:, -1, :])
        logit_cond = logits[0::2, :]
        logit_uncond = logits[1::2, :]

        logits = logit_uncond + cfg_weight * (logit_cond-logit_uncond)
        probs = torch.softmax(logits / temperature, dim=-1)

        next_token = torch.multinomial(probs, num_samples=1)
        generated_tokens[:, i] = next_token.squeeze(dim=-1)

        next_token = torch.cat([next_token.unsqueeze(dim=1), next_token.unsqueeze(dim=1)], dim=1).view(-1)
        img_embeds = mmgpt.prepare_gen_img_embeds(next_token)
        inputs_embeds = img_embeds.unsqueeze(dim=1)


    dec = mmgpt.gen_vision_model.decode_code(generated_tokens.to(dtype=torch.int), shape=[parallel_size, 8, img_size//patch_size, img_size//patch_size])
    dec = dec.to(torch.float32).cpu().numpy().transpose(0, 2, 3, 1)

    dec = np.clip((dec + 1) / 2 * 255, 0, 255)

    visual_img = np.zeros((parallel_size, img_size, img_size, 3), dtype=np.uint8)
    visual_img[:, :, :] = dec

    os.makedirs('generated_samples', exist_ok=True)
    for i in range(parallel_size):
        save_path = os.path.join('generated_samples', "img_{}.jpg".format(i))
        PIL.Image.fromarray(visual_img[i]).save(save_path)


generate(
    vl_gpt,
    vl_chat_processor,
    prompt,
)

### Gradio 演示我们已在 [Huggingface](https://huggingface.co/spaces/deepseek-ai/Janus-Pro-7B) 部署了在线演示。如需本地运行 Gradio 演示，请执行以下命令：

pip install -e .[gradio]

python demo/app_januspro.py

祝使用愉快！

Janus

### 安装在 `Python >= 3.8` 环境下，运行以下命令安装所需依赖：

pip install -e .

### 简单推理示例 #### 多模态理解


import torch
from transformers import AutoModelForCausalLM
from janus.models import MultiModalityCausalLM, VLChatProcessor
from janus.utils.io import load_pil_images

# 指定模型路径
model_path = "deepseek-ai/Janus-1.3B"
vl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)
tokenizer = vl_chat_processor.tokenizer

vl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained(
    model_path, trust_remote_code=True
)
vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()

conversation = [
    {
        "role": "User",
        "content": "<image_placeholder>\n将公式转换为 LaTeX 代码。",
        "images": ["images/equation.png"],
    },
    {"role": "Assistant", "content": ""},
]

# 加载图像并准备输入
pil_images = load_pil_images(conversation)
prepare_inputs = vl_chat_processor(
    conversations=conversation, images=pil_images, force_batchify=True
).to(vl_gpt.device)

# # 运行图像编码器获取图像嵌入
inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)

# # 运行模型获取响应
outputs = vl_gpt.language_model.generate(
    inputs_embeds=inputs_embeds,
    attention_mask=prepare_inputs.attention_mask,
    pad_token_id=tokenizer.eos_token_id,
    bos_token_id=tokenizer.bos_token_id,
    eos_token_id=tokenizer.eos_token_id,
    max_new_tokens=512,
    do_sample=False,
    use_cache=True,
)

answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True)
print(f"{prepare_inputs['sft_format'][0]}", answer)

#### 文本到图像生成

import os
import PIL.Image
import torch
import numpy as np
from transformers import AutoModelForCausalLM
from janus.models import MultiModalityCausalLM, VLChatProcessor


# 指定模型路径
model_path = "deepseek-ai/Janus-1.3B"
vl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)
tokenizer = vl_chat_processor.tokenizer

vl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained(
    model_path, trust_remote_code=True
)
vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()

conversation = [
    {
        "role": "User",
        "content": "一位来自喀布尔的惊艳公主，身着红白传统服饰，蓝眼睛，棕色头发",
    },
    {"role": "Assistant", "content": ""},
]

sft_format = vl_chat_processor.apply_sft_template_for_multi_turn_prompts(
    conversations=conversation,
    sft_format=vl_chat_processor.sft_format,
    system_prompt="",
)
prompt = sft_format + vl_chat_processor.image_start_tag


@torch.inference_mode()
def generate(
    mmgpt: MultiModalityCausalLM,
    vl_chat_processor: VLChatProcessor,
    prompt: str,
    temperature: float = 1,
    parallel_size: int = 16,
    cfg_weight: float = 5,
    image_token_num_per_image: int = 576,
    img_size: int = 384,
    patch_size: int = 16,
):
    input_ids = vl_chat_processor.tokenizer.encode(prompt)
    input_ids = torch.LongTensor(input_ids)

    tokens = torch.zeros((parallel_size*2, len(input_ids)), dtype=torch.int).cuda()
    for i in range(parallel_size*2):
        tokens[i, :] = input_ids
        if i % 2 != 0:
            tokens[i, 1:-1] = vl_chat_processor.pad_id

    inputs_embeds = mmgpt.language_model.get_input_embeddings()(tokens)

    generated_tokens = torch.zeros((parallel_size, image_token_num_per_image), dtype=torch.int).cuda()

    for i in range(image_token_num_per_image):
        outputs = mmgpt.language_model.model(inputs_embeds=inputs_embeds, use_cache=True, past_key_values=outputs.past_key_values if i != 0 else None)
        hidden_states = outputs.last_hidden_state

        logits = mmgpt.gen_head(hidden_states[:, -1, :])
        logit_cond = logits[0::2, :]
        logit_uncond = logits[1::2, :]

        logits = logit_uncond + cfg_weight * (logit_cond-logit_uncond)
        probs = torch.softmax(logits / temperature, dim=-1)

        next_token = torch.multinomial(probs, num_samples=1)
        generated_tokens[:, i] = next_token.squeeze(dim=-1)

        next_token = torch.cat([next_token.unsqueeze(dim=1), next_token.unsqueeze(dim=1)], dim=1).view(-1)
        img_embeds = mmgpt.prepare_gen_img_embeds(next_token)
        inputs_embeds = img_embeds.unsqueeze(dim=1)


    dec = mmgpt.gen_vision_model.decode_code(generated_tokens.to(dtype=torch.int), shape=[parallel_size, 8, img_size//patch_size, img_size//patch_size])
    dec = dec.to(torch.float32).cpu().numpy().transpose(0, 2, 3, 1)

    dec = np.clip((dec + 1) / 2 * 255, 0, 255)

    visual_img = np.zeros((parallel_size, img_size, img_size, 3), dtype=np.uint8)
    visual_img[:, :, :] = dec

    os.makedirs('generated_samples', exist_ok=True)
    for i in range(parallel_size):
        save_path = os.path.join('generated_samples', "img_{}.jpg".format(i))
        PIL.Image.fromarray(visual_img[i]).save(save_path)


generate(
    vl_gpt,
    vl_chat_processor,
    prompt,
)

### Gradio 演示我们已在 [Huggingface](https://huggingface.co/spaces/deepseek-ai/Janus-1.3B) 部署了在线演示。如需本地运行 Gradio 演示，请执行以下命令：

pip install -e .[gradio]

python demo/app.py

祝使用愉快！ ### FastAPI 演示轻松运行 FastAPI 服务器，搭建一个运行与 Gradio 相同功能的 API 服务器。启动 FastAPI 服务器，运行以下命令：

python demo/fastapi_app.py

要测试服务器，可以打开另一个终端并运行：

python demo/fastapi_client.py

JanusFlow

### 安装在 `Python >= 3.8` 环境下，运行以下命令安装所需依赖：

pip install -e .
pip install diffusers[torch]

### 🤗 Huggingface 在线演示请查阅 [此链接](https://huggingface.co/spaces/deepseek-ai/JanusFlow-1.3B) 上的演示。 ### 简单推理示例 #### 多模态理解


import torch
from janus.janusflow.models import MultiModalityCausalLM, VLChatProcessor
from janus.utils.io import load_pil_images

# 指定模型路径
model_path = "deepseek-ai/JanusFlow-1.3B"
vl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)
tokenizer = vl_chat_processor.tokenizer

vl_gpt = MultiModalityCausalLM.from_pretrained(
    model_path, trust_remote_code=True
)
vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()

conversation = [
    {
        "role": "User",
        "content": "<image_placeholder>\n将公式转换为 LaTeX 代码。",
        "images": ["images/equation.png"],
    },
    {"role": "Assistant", "content": ""},
]

# 加载图像并准备输入
pil_images = load_pil_images(conversation)
prepare_inputs = vl_chat_processor(
    conversations=conversation, images=pil_images, force_batchify=True
).to(vl_gpt.device)

# # 运行图像编码器获取图像嵌入
inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)

# # 运行模型获取响应
outputs = vl_gpt.language_model.generate(
    inputs_embeds=inputs_embeds,
    attention_mask=prepare_inputs.attention_mask,
    pad_token_id=tokenizer.eos_token_id,
    bos_token_id=tokenizer.bos_token_id,
    eos_token_id=tokenizer.eos_token_id,
    max_new_tokens=512,
    do_sample=False,
    use_cache=True,
)

answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True)
print(f"{prepare_inputs['sft_format'][0]}", answer)

#### 文本到图像生成

import os
import PIL.Image
import torch
import numpy as np
from janus.janusflow.models import MultiModalityCausalLM, VLChatProcessor
import torchvision


# 指定模型路径
model_path = "deepseek-ai/JanusFlow-1.3B"
vl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)
tokenizer = vl_chat_processor.tokenizer

vl_gpt = MultiModalityCausalLM.from_pretrained(
    model_path, trust_remote_code=True
)
vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()

from diffusers.models import AutoencoderKL
# 请使用 bfloat16 数据类型，此 VAE 不支持 fp16
vae = AutoencoderKL.from_pretrained("stabilityai/sdxl-vae")
vae = vae.to(torch.bfloat16).cuda().eval()

conversation = [
    {
        "role": "User",
        "content": "一位来自喀布尔的惊艳公主，身着红白传统服饰，蓝眼睛，棕色头发",
    },
    {"role": "Assistant", "content": ""},
]

sft_format = vl_chat_processor.apply_sft_template_for_multi_turn_prompts(
    conversations=conversation,
    sft_format=vl_chat_processor.sft_format,
    system_prompt="",
)
prompt = sft_format + vl_chat_processor.image_gen_tag


@torch.inference_mode()
def generate(
    mmgpt: MultiModalityCausalLM,
    vl_chat_processor: VLChatProcessor,
    prompt: str,
    cfg_weight: float = 5.0,
    num_inference_steps: int = 30,
    batchsize: int = 5
):
    input_ids = vl_chat_processor.tokenizer.encode(prompt)
    input_ids = torch.LongTensor(input_ids)

    tokens = torch.stack([input_ids] * 2 * batchsize).cuda()
    tokens[batchsize:, 1:] = vl_chat_processor.pad_id
    inputs_embeds = vl_gpt.language_model.get_input_embeddings()(tokens)

    # 我们移除最后的 <bog> 标记，稍后用 t_emb 替换
    inputs_embeds = inputs_embeds[:, :-1, :] 

    # 使用修正流 ODE 生成
    # 步骤 1：使用 vision_gen_enc 编码
    z = torch.randn((batchsize, 4, 48, 48), dtype=torch.bfloat16).cuda()

    dt = 1.0 / num_inference_steps
    dt = torch.zeros_like(z).cuda().to(torch.bfloat16) + dt

    # 步骤 2：运行 ODE
    attention_mask = torch.ones((2*batchsize, inputs_embeds.shape[1]+577)).to(vl_gpt.device)
    attention_mask[batchsize:, 1:inputs_embeds.shape[1]] = 0
    attention_mask = attention_mask.int()
    for step in range(num_inference_steps):
        # 准备 LLM 输入
        z_input = torch.cat([z, z], dim=0) # 用于 CFG
        t = step / num_inference_steps * 1000.
        t = torch.tensor([t] * z_input.shape[0]).to(dt)
        z_enc = vl_gpt.vision_gen_enc_model(z_input, t)
        z_emb, t_emb, hs = z_enc[0], z_enc[1], z_enc[2]
        z_emb = z_emb.view(z_emb.shape[0], z_emb.shape[1], -1).permute(0, 2, 1)
        z_emb = vl_gpt.vision_gen_enc_aligner(z_emb)
        llm_emb = torch.cat([inputs_embeds, t_emb.unsqueeze(1), z_emb], dim=1)

        # 输入到 LLM
        # 我们为 CFG 应用注意力掩码：1 表示未屏蔽的标记，0 表示被屏蔽的标记。
        if step == 0:
            outputs = vl_gpt.language_model.model(inputs_embeds=llm_emb, 
                                             use_cache=True, 
                                             attention_mask=attention_mask,
                                             past_key_values=None)
            past_key_values = []
            for kv_cache in past_key_values:
                k, v = kv_cache[0], kv_cache[1]
                past_key_values.append((k[:, :, :inputs_embeds.shape[1], :], v[:, :, :inputs_embeds.shape[1], :]))
            past_key_values = tuple(past_key_values)
        else:
            outputs = vl_gpt.language_model.model(inputs_embeds=llm_emb, 
                                             use_cache=True, 
                                             attention_mask=attention_mask,
                                             past_key_values=past_key_values)
        hidden_states = outputs.last_hidden_state

        # 将 hidden_states 转换回 v
        hidden_states = vl_gpt.vision_gen_dec_aligner(vl_gpt.vision_gen_dec_aligner_norm(hidden_states[:, -576:, :]))
        hidden_states = hidden_states.reshape(z_emb.shape[0], 24, 24, 768).permute(0, 3, 1, 2)
        v = vl_gpt.vision_gen_dec_model(hidden_states, hs, t_emb)
        v_cond, v_uncond = torch.chunk(v, 2)
        v = cfg_weight * v_cond - (cfg_weight-1.) * v_uncond
        z = z + dt * v

    # 步骤 3：使用 vision_gen_dec 和 sdxl vae 解码
    decoded_image = vae.decode(z / vae.config.scaling_factor).sample

    os.makedirs('generated_samples', exist_ok=True)
    save_path = os.path.join('generated_samples', "img.jpg")
    torchvision.utils.save_image(decoded_image.clip_(-1.0, 1.0)*0.5+0.5, save_path)

generate(
    vl_gpt,
    vl_chat_processor,
    prompt,
    cfg_weight=2.0,
    num_inference_steps=30,
    batchsize=5
)

### Gradio 演示如需本地运行 Gradio 演示，请执行以下命令：

pip install -e .[gradio]

python demo/app_janusflow.py

祝使用愉快！

4. 许可证

本代码仓库采用 MIT 许可证。Janus 模型的使用受 DeepSeek 模型许可证的约束。

5. 引用

@article{chen2025janus,
  title={Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling},
  author={Chen, Xiaokang and Wu, Zhiyu and Liu, Xingchao and Pan, Zizheng and Liu, Wen and Xie, Zhenda and Yu, Xingkai and Ruan, Chong},
  journal={arXiv preprint arXiv:2501.17811},
  year={2025}
}

@article{wu2024janus,
  title={Janus: Decoupling visual encoding for unified multimodal understanding and generation},
  author={Wu, Chengyue and Chen, Xiaokang and Wu, Zhiyu and Ma, Yiyang and Liu, Xingchao and Pan, Zizheng and Liu, Wen and Xie, Zhenda and Yu, Xingkai and Ruan, Chong and others},
  journal={arXiv preprint arXiv:2410.13848},
  year={2024}
}

@misc{ma2024janusflow,
      title={JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation}, 
      author={Yiyang Ma and Xingchao Liu and Xiaokang Chen and Wen Liu and Chengyue Wu and Zhiyu Wu and Zizheng Pan and Zhenda Xie and Haowei Zhang and Xingkai yu and Liang Zhao and Yisong Wang and Jiaying Liu and Chong Ruan},
      journal={arXiv preprint arXiv:2411.07975},
      year={2024}
}

6. 联系方式

如有任何问题，请提交 Issue 或发送邮件至 service@deepseek.com。

项目地址：https://github.com/deepseek-ai/Janus

32 次点击 ∙ 0 人收藏

登录后收藏

0 条回复

Janus — 轻量级多模态理解与生成研究项目

🚀 Janus-Series: 统一的多模态理解与生成模型

新闻

1. 简介

2. 模型下载

Huggingface

3. 快速开始

Janus-Pro

Janus

JanusFlow

4. 许可证

5. 引用

6. 联系方式