trustworthy-ai

Illegally using fine-tuned diffusion models to forge human portraits has been a major threat to trustworthy AI. While most existing work focuses on detection of the AI-forged contents, our recent work instead aims to mitigate such illegal domain adaptation by applying safeguards on diffusion models. Being different from model unlearning techniques that cannot prevent the illegal domain knowledge from being relearned with custom or public data, our approach, namely FreezeGuard, suggests that the model publisher selectively freezes tensors in pre-trained models that are critical to illegal model adaptations while minimizing the impact on other legal adaptations. Experiments in multiple text-to-image applications domains show that our method providing 37% stronger mitigation power while incurring less than 5% impact on legal model adaptations.

Cloaking techniques such as character deformation, homophonic substitution, abbreviation and code-mixing with pinyin or emojis let hate speech on Chinese social networks evade text-only detectors. MMBERT is a BERT-based multimodal framework that integrates textual, speech and visual modalities through a Mixture-of-Experts architecture, with modality-specific experts, a shared self-attention mechanism and router-based expert allocation. A progressive three-stage training paradigm keeps MoE training stable, and MMBERT significantly surpasses fine-tuned BERT-based encoders, fine-tuned LLMs and in-context learning with LLMs on several Chinese hate speech datasets.