Two Papers Accepted to IEEE ACCESS

Title: Lightweight Optical Flow Estimation using 1D Matching

Authors: Wonyong Seo, Woonsung Park and Munchurl Kim

Abstract:

Recent advancements in optical flow estimation have led to notable performance gains, driven by the adoption of transformer architectures, enhanced data augmentation, self-supervised learning techniques, the use of multiple video frames and iterative refinement of estimate optical flows. Nonetheless, these cutting-edge methods encounter substantial challenges with surge in computational complexity and memory demands. In response, we introduce a lightweight optical flow method, called MaxFlow, to address the trade-off between computational complexity and prediction performance. By leveraging MaxViT, we design a network with a global receptive field at reduced complexity, and proposed 1D matching to alleviate the computational complexity from (HxW)^2 to HxW(H+W), wher H and W denotes height and width of input image. Consequently, our method achieves the lowest computational complexity compared to both state of the arts (SOTA) and other lightweight optical flow estimation methods, while still achieving competitive results with the SOTA techniques. We performed extensive experiments to show the effectiveness of our method, achieving about 5 to 6 times reductions in computation complexity while maintaining the prediction accuracy with only degradation of 16\% in term of end point error(EPE) at Sintel test clean sequences with respect to RAFT method.

Title: ProNeRF: Learning Efficient Projection-Aware Ray Sampling for Fine-Grained Implicit Neural Radiance Fields

Authors: Juan Luis Gonzalez Bello*, Minh-Quan Viet Bui*, and Munchurl Kim (* Equal Contribution)

Abstract:

Recent advances in neural rendering have shown that although computationally expensive and slow for training, implicit compact models can accurately learn a scene's geometries and view-dependent appearances from multiple views. To maintain such a small memory footprint but achieve faster inference times, recent works have adopted `sampler' networks that adaptively sample a small subset of points along each ray in the implicit neural radiance fields NeRF, effectively reducing the number of network forward passes to render a ray color. Although these methods achieve up to a 10X reduction in rendering time, they still suffer from considerable quality degradation compared to vanilla NeRF. In contrast, we propose a new projection-aware neural radiance field model, referred to as ProNeRF, which provides an optimal trade-off between the memory footprint (similar to NeRF), speed (faster than HyperReel), and quality (better than K-Planes). ProNeRF is equipped with a novel projection-aware sampling (PAS) network together with a new training strategy for ray exploration and exploitation, allowing for efficient fine-grained particle sampling. Our exploration and exploitation training strategy allows ProNeRF to learn the color and density distributions of full scenes, while also learning efficient ray sampling focused on the highest-density regions. ProNeRF yields state-of-the-art metrics, being 15 to 23X faster with 0.65dB higher PSNR than the vanilla NeRF and showing 0.95dB higher PSNR performance compared to the best published sampler-based method, HyperReel. We provide extensive experimental results that support the effectiveness of our method on the widely adopted forward-facing and 360 datasets, LLFF and Blender, respectively. Additionally, we present real-world applications of ProNeRF on hand-held captured scenes}. Our project page is publicly available at https://kaist-viclab.github.io/pronerf-site.