Xingyu Lin

I am a research scientist at OpenAI working on general purpose robots.

I received my Ph.D. in Robotics from Carnegie Mellon University, advised by David Held and my B.S. in computer science from Peking University. I have also spent two years as a postdoc the Berkeley Artificial Intelligence Research (BAIR) Lab working with Pieter Abbeel.

Google Scholar / Github / Twitter / CV (Mar. 2024) / xingyulin2016 [at] gmail (dot) com


Research

My goal is to build general-purpose robots that can assist humans in unstructured environments. I work on the following themes:

  • Learning structured world models with spatial and temporal abstractions.
  • Learning from pre-trained multimodal foundation modelto enable knowledge transfer from Internet-scale data.
  • Building a diverse and reproducible robotics benchmark.

News

08/2024
I started a new job at OpenAI in a new robotics team.
08/2024
We are organizing the Workshop on Whole-body Control and Bimanual Manipulation at CoRL 2024.
05/2024
Four papers accepted at RSS 2024 and IROS 2024.
07/2023
Invited talk at RSS 2023 workshop on Interdisciplinary Exploration of Generalizable Manipulation Policy Learning
06/2023
I am co-organizing the RSS 2023 Pioneer Workshop.
03/2023
Two papers accepted at ICRA 2023 and RSS 2023

08/2022
I have successfully defended my PhD thesis! A big thanks to my thesis committee: David Held, Abhinav Gupta, Deepak Pathak and Ken Goldberg.
05/2022
I am selected as one of the RSS Pioneers of 2022.
02/2022
Invited talks at UC Berkeley, MIT and Stanford.
01/2022
One paper accepted at ICLR 2022! [OpenReview]
10/2021
Proposed my PhD thesis. [Link]
09/2021
One paper on cloth manipulation accepted at CoRL 2021! [OpenReview]
07/2021
Invited talk at RSS 2021 workshop on Deformable Object Simulation. (Co-present with David Held) [Link]
Show More


Publications

(show selected / show all by date)
GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators
Philipp Wu, Yide Shentu, Zhongke Yi, Xingyu Lin, Pieter Abbeel
IROS 2024
Project Page / Paper / Code / Hardware / Abstract / Bibtex
Oral presentation at CoRL 2023 workshop [Link]
Learning Generalizable Tool-use Skills through Trajectory Generation
Carl Qi*, Yilin Wu*, Lifan Yu, Haoyue Liu, Bowen Jiang, Xingyu Lin**, David Held**
IROS 2024
Project Page / Paper / Code (Coming soon) / Abstract / Bibtex
HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation
Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, Pieter Abbeel
RSS 2024
Project Page / Paper / Code / Abstract / Bibtex
Any-point Trajectory Modeling for Policy Learning
Chuan Wen*, Xingyu Lin*, John So*, Kai Chen, Qi Dou, Yang Gao, Pieter Abbeel
RSS 2024
Project Page / Paper / Code / Abstract / Bibtex
SpawnNet: Learning Generalizable Visuomotor Skills from Pre-trained Networks
Xingyu Lin*, John So*, Sashwat Mahalingam, Fangchen Liu, Pieter Abbeel
ICRA 2024
Project Page / Paper / Code / Abstract / Bibtex
Oral presentation at CoRL 2023 workshop [Link]
RoboNinja: Learning an Adaptive Cutting Policy for Multi-Material Objects
Zhenjia Xu, Zhou Xian, Xingyu Lin, Cheng Chi, Zhiao Huang, Chuang Gan, Shuran Song
RSS 2023
Project Page / Paper / Code / Abstract / Bibtex
Self-supervised Cloth Reconstruction via Action-conditioned Cloth Tracking
Zixuan Huang, Xingyu Lin, David Held
ICRA 2023
Project Page / Paper / Abstract / Bibtex
Planning with Spatial-Temporal Abstraction from Point Clouds for Deformable Object Manipulation
Xingyu Lin*, Carl Qi*, Yunchu Zhang, Zhiao Huang, Katerina Fragkiadaki, Yunzhu Li, Chuang Gan, David Held
CoRL 2022
Project Page / Paper / Code / Abstract / Bibtex
Learning Closed-loop Dough Manipulation using a Differentiable Reset Module
Carl Qi, Xingyu Lin, David Held
Robotics and Automation Letters (RA-L) with presentation at IROS 2022
Project Page / Paper / Abstract / Bibtex
Covered by [CMU SCS News] [Washington Post]
Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation
Zixuan Huang, Xingyu Lin, David Held
RSS 2022
Project Page / Paper / Abstract / Bibtex
DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools
Xingyu Lin, Zhiao Huang, Yunzhu Li, Joshua B. Tenenbaum, David Held, Chuang Gan
ICLR 2022
Project Page / Paper / Code / Abstract / Bibtex
Covered by [MIT News] [TechCrunch] [COSMOS] [TechTalks] [AZO Robotics]
Transparent Liquid Segmentation for Robotic Pouring
Gautham Narasimhan, Kai Zhang, Ben Eisner, Xingyu Lin, David Held
ICRA 2022
Project Page / Paper / Code / Abstract / Bibtex
Covered by [CMU SCS News (Pour me a Glass)]
Learning Visible Connectivity Dynamics for Cloth Smoothing
Xingyu Lin*, Yufei Wang*, Zixuan Huang, David Held
CoRL 2021
Project Page / Paper / Code / Video / Abstract / Bibtex
Abridged in ICRA 2021 workshop on Representing and Manipulating Deformable Object [Link]
SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation
Xingyu Lin, Yufei Wang, Jake Olkin, David Held
CoRL 2020
Project Page / Paper / Code / Bibtex
Visual Self-Supervised Reinforcement Learning with Object Reasoning
Yufei Wang*, Gautham Narayan*, Xingyu Lin, Brian Okorn, David Held
CoRL 2020
Project Page/ Paper / Code / Bibtex
Adaptive Auxiliary Task Weighting for Reinforcement Learning
Xingyu Lin*, Harjatin Baweja*, George Kantor, David Held
NeurIPS 2019
Paper / Code / Poster / Slides / Bibtex
Abridged in ICML 2019 workshop on Multi-Task and Lifelong Reinforcement Learning [Link]
Adaptive Variance for Changing Sparse-Reward Environments
Xingyu Lin, Pengsheng Guo, Carlos Florensa, David Held
ICRA 2019
Paper / Project Page / Video / Bibtex
Reinforcement Learning without Ground-Truth State
Xingyu Lin, Harjatin Singh Baweja, David Held
ICML 2019, Workshop on Multi-Task and Lifelong Reinforcement Learning [Link]
Paper / Video / Bibtex
Time‐varying light motion in single convergence
Xingyu Lin, Mingxuan Chai, Sheng Li, Guoping Wang
CASA 2018
Paper / Code / Video
Transfer of View-manifold Learning to Similarity Perception
Xingyu Lin, Hao Wang, Zhihao Li, Yimeng Zhang, Alan Yuille, Tai Sing Lee
ICLR 2017
Paper / Bibtex

Service

  • Conference Reviewer: NeurIPS, ICML, ICLR, RSS, CoRL, ICRA, IROS
  • Journal Reviewer: IEEE RA-L, AURO, T-RO