I am a postdoctoral researcher in the Anima AI + Science Lab and the Computing + Mathematical Sciences Department at California Institute of Technology. Prior to joining Caltech, I completed my Ph.D. at Vision Science and Berkeley AI Research (BAIR), UC Berkeley, where I worked with Stella Yu on fundamental machine learning and computer vision models and Meng Lin on applying them to healthcare. My research lies at the intersection of machine learning, computer vision and AI for healthcare. I am currently on the job market.

New! Pose-Aware Self-Supervised Learning accepted to ECCV 2024 with an oral presentation!
INSIGHT accepted to MICCAI 2024!
Deep Multimodal Fusion won the best paper at ML4H 2023!

Research Interests

My research lies at the intersection of machine learning, computer vision and AI for healthcare. I develop machine learning (ML) and computer vision (CV) methods to address data insufficiency and model interpretability challenges. My techniques leverage physics simulations, reduce human supervision by learning from imbalanced data, and enhance interpretability through multi-level structure-aware representations. My goal is to incorporate domain knowledge into data-driven and interpretable models that not only perform well but also facilitate scientific discovery and provide insights for scientists and stakeholders. Notably, we developed the first method that can reconstruct lung anatomy with the commercial ultrasound device. More details can be found in my research statement.

Publications

See the full list in Google Scholar.

Pose-Aware Self-Supervised Learning with Viewpoint Trajectory Regularization
 Jiayun Wang, Yubei Chen, Stella X. Yu.
ECCV 2024. Oral (2.3%). [paper] [project]



Human Reposing and Virtual Try-On from Multi-View Images
 Jiayun Wang, Amin Kheradmand, Himanshu Arora.
WACV 2024. [paper]



Compact and Optimal Deep Learning with Recurrent Parameter Generators
 Jiayun Wang, Yubei Chen, Stella X. Yu, Brian Cheung, Yann LeCunn.
WACV 2023. [project] [video] [slides] [paper] [code] [blog] [media coverage (Chinese)]



Predicting demographics from meibography using deep learning
Jiayun Wang, Andrew D. Graham, Stella X. Yu, Meng C. Lin.
Nature - Scientific Reports 2022. [paper] [slides] [media coverage]



Open Long-Tailed Recognition in A Dynamic World
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, Stella X. Yu.
IEEE TPAMI 2022. [paper][slides] [code]



Unsupervised Scene Sketch to Photo Synthesis
 Jiayun Wang, Sangryul Jeon, Stella X. Yu, Xi Zhang, Himanshu Arora, Yu Lou.
ECCV 2022. [project] [slides] [paper] [code] [media coverage]



3D Shape Reconstruction from Free-Hand Sketches
 Jiayun Wang, Jierui Lin, Qian Yu, Runtao Liu, Yubei Chen, Stella X. Yu.
AIM @ ECCV 2022. [paper] [slides] [code]



Quantifying Meibomian Gland Morphology Using Artificial Intelligence
Jiayun Wang, Shixuan Li, Thao N. Yeh, Rudrasis Chakraborty, Andrew D Graham, Stella X. Yu, Meng C. Lin.
Optometry and Vision Science 2021. [paper] [code] [demo]



Spatial Transformer for 3D Point Clouds
 Jiayun Wang, Rudrasis Chakraborty, Stella X. Yu.
IEEE TPAMI 2021 [project] [video] [poster] [paper] [code]

We propose to learn different non-rigid transformations of the input point cloud for different local neighborhoods at each layer.

Orthogonal Convolutional Neural Networks
 Jiayun Wang, Yubei Chen, Rudrasis Chakraborty, Stella X. Yu.
CVPR 2020. [project] [video] [slides] [paper] [code] [blog (Chinese)]

Orthogonal convolutional neural networks is a light-cost regularizer which reduces the feature redundancy and improves network performance and robustness under attack.

Sur-Real: Frechet Mean and Distance Transform for Complex-Valued Deep Learning
 Rudrasis Chakraborty, Jiayun Wang, Stella X. Yu.
PBVS, CVPR 2019. Best Paper. [paper] [poster] [code]

This work develops a novel deep learning architecture for naturally complex-valued data, with improved results and only 10% of the parameters as the baseline model.

A Deep Learning Approach for Meibomian Gland Atrophy Evaluation in Meibography Images
Jiayun Wang, Thao N. Yeh, Rudrasis Chakraborty, Stella X. Yu, Meng C. Lin.
Translational Vision Science and Technology 2019. [paper] [code]

This work develops a deep learning approach to digitally segmenting meibomian gland atrophy area and computing percent atrophy in meibography images.

Large-scale Long-Tailed Recognition in an Open World
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, Stella X. Yu.
CVPR 2019. Oral. [project] [paper] [code] [blog]

This work formally defines the problem of open long-tailed recognition (OLTR) as optimizing for the overall accuracy of a naturally-distributed dataset with the presence of open classes.

Insights and Approaches Using Deep Learning to Classify Wildlife
 Zhongqi Miao, Kaitlyn M Gaynor, Jiayun Wang, Ziwei Liu, Oliver Muellerklein, Mohammad S. Norouzzadeh, Alex McInturff, Rauri C. K. Bowie, Ran Nathon, Stella X. Yu, Wayne M. Getz.
Nature - Scientific Reports 2019. [Paper]

This work aims to interpret the concepts behind the convolutional neural networks (CNNs) in classifying wildlife.

Deep Ranking Model by Large Adaptive Margin Learning for Person Re-identification
Jiayun Wang, Sanping Zhou, Jinjun Wang, Qiqi Hou.
Pattern Recognition 2018. [paper] [arxiv] [code]

This work designs a dynamically adaptive loss function to overcome the drawbacks of conventional loss functions for person re-identification.

Successive Embedding and Classification Loss for Aerial Image Classification
Jiayun Wang, Patrick Virtue, Stella X. Yu.
arXiv 2017. [paper] [code]

This work aims to address the overfitting problem in remote sensing image classification.

Point to Set Similarity Based Deep Feature Learning for Person Re-identification
Sanping Zhou, Jinjun Wang, Jiayun Wang, Yihong Gong, Nanning Zheng.
CVPR 2017. [paper] [code]

This work presents a novel metric learning method based on P2S similarity comparison for person re-identification.

Teaching

At Berkeley, I taught the following classes as a graduate student instructor:
      CS182/282, Deep Learning, Spring 2023
      CS189/289, Machine Learning, Fall 2020
      VS205, Visual Perception, Fall 2019, 2018

Talks

I gave talks at seminars of Apple, Amazon, Bosch, Oxyopia, etc. You could watch some of the conferencepresentations of my works on YouTube:
      Towards Real-World Structure-Aware Representation Learning, 2023
      Compact and Optimal Deep Learning with Recurrent Parameter Generators, 2022
      Spatial Transformer for 3D Point Clouds, 2021
      Orthogonal Convolutional Neural Networks, 2020