Uni-ISP: Unifying the Learning of ISPs from Multiple Cameras
Lingen Li, Mingde Yao, Xingyu Meng, Muquan Yu, Tianfan Xue, and Jinwei Gu
The Chinese University of Hong Kong
We propose Uni-ISP, a model that unifies the learning of inverse and forward ISP behaviors of multiple cameras simultaneously. By leveraging the shared characteristics across various camera ISPs, our method can achieve higher performance in inverse and forward ISP (A) compared to previously learned ISP methods tailored for only one camera separately. Meanwhile, the device-aware property of the Uni-ISP enables new cross-camera ISP applications for a learned ISP model, including photographic appearance transfer (B and C), inter/extrapolation (D), and zero-shot image forensics (E and F).
Method
The model design of Uni-ISP. Uni-ISP contains two modules, the inverse ISP module and the forward ISP module. Both two modules share the same structure. For visual simplicity, we draw the inverse ISP moduleas a thumbnail, whose inner structure is the same as the forward ISP module. The device-aware embeddings are optimizable parameters and will be selected to interact with the bottleneck features via the DEIM during the training or inference.
New Dataset: FiveCam
The preview of 3 scenes in our new dataset (left) and our capture devices (right). Each scene includes synchronized sRGB-Raw pairs of five smartphone cameras: Apple iPhone 14 Pro Max, Google Pixel 6 Pro, Huawei P40, Samsung Galaxy S20, and Xiaomi Mi 12. The raw images are visualized as XYZ images here, which can be converted back to raw without loss.
Applications
1. Inverse and Forward ISP
Other Results - HDR Rendering:
Based on the inverse ISP ability learned by our model, we can convert a given sRGB back to the linear color space, and sythesize exposure stack to render HDR images.
Sample Scene
Sample Scene
Click on a tag below to switch among LDR input, HDR rendering results of each method, and the ground truth.
Selected Item: Input (LDR)
2. Photographic Appearance Transfer
This transfer ability can be extended to video without explicit temporal design. Please refer to the video section below.
3. Photographic Appearance Interpolation
4. Zero-shot Image Forensics
Uni-ISP has bonus ability in zero-shot image forensics, including source camera identification and image splice detection, without any trainig on forensics datasets. Please refer to the paper for more details. Here we show an example of image splice detection. A fake object is integrated into each real photo using Poisson image editing, with darker regions in the result indicating suspected modifications.
Video
Click on the button to view the short introduction video of Uni-ISP. It might take a while to load this video.
Citation
😊 Consider citing this paper if you found our work/dataset useful in your research.
@article{li2024uni,
title={Uni-ISP: Unifying the Learning of ISPs from Multiple Cameras},
author={Li, Lingen and Yao, Mingde and Meng, Xingyu and Yu, Muquan and Xue, Tianfan and Gu, Jinwei},
journal={arXiv preprint arXiv:2406.01003},
year={2024}
}