Image and Video

chengjun_1120.png

Chengjun Liu
Associate Professor
chengjun.liu@njit.edu

Research Areas:Computer vision, pattern and face recognition, video processing

Making Intelligent Transportation Systems Smarter

The New Jersey Department of Transportation designated more than 400 closed-circuit video cameras statewide for incident monitoring, traffic congestion control and public safety operations. Video streams from these cameras feed to a back-end system. There, video analytics software is used to perform target detection and incident monitoring applications. We are actively working on incorporating wireless sensor networks, hierarchical edge-computing and computer vision to mitigate the challenging problems in various illumination and weather conditions in order to achieve fast and automated video based traffic monitoring.

Video Analytics Pilot Studies and Testing of Technologies

We propose a new modular approach for statistical modeling of traffic incidents and model selection in order to improve state-of-the-art traffic detection and monitoring. We investigated and developed automated video analytics systems to replace human operators for traffic incident detection and to monitor the cameras installed along the major New Jersey highways. We test the proposed technologies and benchmark their performance.

Frank_Shih.png

Frank Shih
Professor
shih@njit.edu

Research Areas: Image processing, artificial intelligence, computer vision, digital watermarking, digital forensics, robot sensing, neural networks

Deep Learning for Pneumonia Classification and Segmentation on Medical Images

Automatic identification of pneumonia on medical images has attracted intensive study. In this project, we develop a novel joint-task architecture that can learn pneumonia classification and segmentation simultaneously. Two modules, including an image preprocessing module and an attention module, are developed to improve both classification and segmentation accuracies. Experimental results performed on the massive dataset of the Radiology Society of North America have confirmed its superiority over other existing methods.

Deep Morphological Neural Networks and Applications

Given a target image, determining suitable morphological operations and structuring elements is a cumbersome and time-consuming task. In this project, we propose new morphological neural networks, which includes a nonlinear feature extraction layer to learn the structuring element correctly and an adaptive layer to automatically select appropriate morphological operations. We also use them for their classification applications, including hand-written digits, geometric shapes, traffic signs and brain tumors. Experimental results show higher computational efficiency and higher accuracy when compared with existing convolutional neural network models.

Adaptive Data Transformation in Contrastive Learning

Although contrastive learning has been playing a critical role in pattern recognition, how to optimize positive pairs through data transformation is still not well developed up to now. In this project, we propose a novel Adaptive Data Transformation, named ADTrans, to identify an optimal sequence of data transformations, which enables generating highquality positive pairs adaptively during contrastive training. Extensive experiments on benchmark datasets have shown that ADTrans can improve the performance of representation learning on downstream tasks significantly, including image classification, instance segmentation and object detection.

Image Captioning with Contextual Feature Exploitation

Existing methods for image captioning lack the incorporation of sufficient contextual information in the generated captions. In this project, we present ContExCap, a novel and effective approach that exploits contextual features from an image to enhance its caption with more contextual information. Moreover, we introduce a new module, named ObjectContext Attention, to capture the interaction and relationship between objects and contextual features. To utilize the low-level features, we also incorporate feature fusion of every second encoding layers with spatial shift operation that enables the features to align with neighbors.

Grace.png

Guiling “Grace” Wang
distinguished Professor
gwang@njit.edu

Research Areas:  Deep Learning, AI in Finance, AI in Transportation, Large Language Model (LLM)

Reverse Pass-through VR and Full head Avatars

We introduce a novel, state-of-the-art method for reverse pass-through VR and one-shot full-head avatar generation. Our approach enables real-time reconstruction of complete facial images from partial VR inputs, delivering a seamless reverse pass-through experience. Using deep learning and 3DMM models, we generate photorealistic, one-shot full-head avatars from just a single image. The system functions in an end-to-end manner, ensuring an immersive VR experience. To support further research, we developed a new dataset to simulate complex VR conditions, including diverse occlusions and lighting variations. Our framework surpasses existing state-of-the-art methods in both visual quality and realism

HierGAN: Combined RGB and Depth inpainting

We present Hierarchical Inpainting GAN (HierGAN), a real-time, novel approach for RGBD inpainting designed to overcome the limitations of current methods in Diminished Reality (DR) applications. HierGAN employs a hierarchical GAN architecture that leverages RGB, depth, edge and segmentation label images to effectively use RGB and depth in painting. By incorporating segmentation label images as auxiliary inputs, HierGAN significantly enhances inpainting performance, achieving superior results over existing methods with an end-to-end, fully optimized workflow.