HaineiFRDM Frame Restoration Training Pipeline
Figure 1 : Training pipeline of our proposed HaineiFRDM. The model is input degraded patched frames and extracts each frame features with Preprocess Module. Then the frame features input into ControlNet, in which we use Global-Prompt-Fusion Module and Global-Frame-Fusion Module to help the model have a global-frame awareness and use freuqency mdoule to maintain origin frame textures. Lastly, the learned ControlNet residuals is input into Unet to produce restored frame features z 0 ^ \hat{z_{0}} and we map z 0 ^ \hat{z_{0}} into RGB space and use defect loss to highlight defects region in restored frames.
论文上下文
Paper title: HaineiFRDM: Explore Diffusion to Restore Defects in Fast-Movement Films Abstract: Existing open-source film restoration methods show limited performance compared to commercial methods due to training with low-quality synthetic data and employing noisy optical flows. In addition, high-resolution films have not been explored by the open-source methods.We propose HaineiFRDM(Film Restoration Diffusion Model), a film restoration framework, to explore diffusion model's powerful content-understanding ability to help human expert better restore indistinguishable film defects.Specifically, we employ a patch-wise training and testing strategy to make restoring high-resolution films on one 24GB-VRAMR GPU possible and design a position-aware Global Prompt and Frame Fusion Modules.Also, we introduce a global-local frequency module to reconstruct consistent textures among different patches. Besides, we firstly restore a low-resolution result and use it as global residual to mitigate blocky artifacts caused by patching process.Furthermore, we construct a film restoration dataset that contains restored real-degraded films and realistic synthetic data.Comprehensive experimental results conclusively demonstrate the superiority of our model in defect restoration ability over existing open-source methods. Code and the dataset will be released. Passages referencing this figure: 2 Related Works Figure 1 : Training pipeline of our proposed HaineiFRDM. 1 Patch-based Training Framework As depicted in Fig. 1 , our model’s training framework is mainly composed of VAE, UNet, Restoration-Guidance Network, Global-Frame-Fusion Module and Texture-Reconstruction Module. To improve high-frequency detail’s consistency, we refer to SFHformer [ jiang2024fast ] ’s idea to restore texture in frequency domain and propose a frequency-based texture-reconstruction module, depicted in Fig. 1 (b).
