Jeonghyun Byun

Safety-critical bilateral teleoperation for omnidirectional aerial manipulation using force-sensorless haptic feedback

2026-08-12T00:00:00+09:00

📃 Abstract: This paper presents a safety-critical bilateral teleoperation framework for omnidirectional aerial manipulators that integrates visual and force-sensorless haptic wrench feedback. Unlike existing approaches that either rely on onboard force/torque sensors or use model-dependent wrench estimates, which may become unreliable under model uncertainties or induce unintended feedback during free-flight, our method implements a hierarchical safety filter based on control barrier functions to avoid such limitations. The safety filter, being the key contribution, explicitly accounts for tracking errors arising from physical interaction between the aerial manipulator and its surroundings while enforcing thrust limits, a factor overlooked despite its critical importance for flight safety. This safety filter adjusts the command from the operator to ensure safe and stable aerial manipulation and evade motor saturation. The adjustment made by the filter is mapped to haptic feedback, which is intuitive to the operator and conveys information on physical interaction and impending motor saturation. By actual experiments with a hexarotor-based omnidirectional aerial manipulator, we demonstrate that the proposed method avoids haptic feedback during free-flight, provides directionally consistent feedback under physical interaction, and can be operated for diverse manipulative tasks. Moreover, an ablation study further shows that the saturation filter improves interaction stability by explicitly preventing motor saturation and informing the operator of corrective actions.

✊ Motivation: Addressing key limitations in bilateral teleoperation for omnidirectional aerial physical interaction (APhI):
❗ Existing haptic-feedback approaches often require onboard force/torque (FT) sensors or model-dependent wrench estimation, which can become unreliable under model uncertainties and generate unintended feedback even during free-flight
❗ Motor thrust limits are critical for flight safety during physical interaction, yet they are rarely considered in existing bilateral teleoperation frameworks

💡 Main contributions:

✔️ Force-sensorless bilateral teleoperation framework for omnidirectional aerial manipulation, integrating both visual and haptic feedback without requiring FT sensors on either the aerial manipulator or the master device
✔️ Two-stage hierarchical CBF-based safety filter that modifies the operator’s command according to tracking errors caused by physical interaction and motor thrust constraints
✔️ Interaction filter that limits position and attitude tracking errors and converts the resulting command adjustment into intuitive, directionally consistent haptic feedback, without explicitly estimating the interaction wrench
✔️ Saturation filter that explicitly enforces individual motor thrust limits and prevents motor saturation by minimally modifying the desired motion command
✔️ Filter corrections mapped directly to haptic wrench feedback, allowing the operator to perceive both physical interaction and impending motor saturation together with the direction of the required corrective action
✔️ No unintended haptic feedback during free-flight, unlike a momentum-based wrench-estimation baseline that exhibits estimation drift and consequently generates undesired motion commands
✔️ Real-world validation on a hexarotor-based omnidirectional aerial manipulator through cart pulling, plug pulling, push-and-slide, and pick-and-place experiments, demonstrating directionally coherent feedback and applicability to both static and dynamic physical interaction tasks
✔️ Ablation study validating the saturation filter, showing that explicitly preventing motor saturation suppresses destabilizing motion and simultaneously provides haptic cues that guide the operator away from unsafe commands

Bibtex

@INPROCEEDINGS{kim2026safety,
  author={Kim, Yubin and Lee, Jinwoo and Lee, Yongjun and Kim, H. Jin and Byun, Jeonghyun},
  booktitle={2026 IEEE/RSJ International Conference on Intelligent Robots & Systems (IROS)}, 
  title={Safety-critical bilateral teleoperation for omnidirectional aerial manipulation using force-sensorless haptic feedback}, 
  year={2026},
  volume={},
  number={},
  pages={},
  keywords={},
  doi={}}

[MATLAB] (Korean) Ubuntu 20.04에서 MATLAB 설치 후 실행 시 폰트가 깨지는 문제 해결

2026-01-26T00:00:00+09:00

우리에게 익숙한 운영체제인 Windows가 아닌 Ubuntu상에서 MATLAB을 설치 및 실행시킬 경우, Ubuntu가 영어로 설치되지 않은 경우 아래와 같은 폰트 깨짐 현상이 발생할 수 있습니다.

(출처: https://kr.mathworks.com/matlabcentral/answers/2132191-)

이런 현상이 발생할 경우, 아래와 같이 간단한 방법으로 이 문제를 해결할 수 있습니다.

Procedure

Step 1: 나눔 폰트 설치

매트랩 창이 아닌 ubuntu의 terminal 창에 아래와 같이 입력 후 enter를 눌러 나눔 폰트를 시스템 상에 설치한다.

sudo apt-get install fonts-nanum fonts-nanum-coding fonts-nanum-extra

이 과정을 거치면, 아래 사진과 같이 폰트 깨짐이 발생하지 않음을 알 수 있다.

Reference

[1] https://kr.mathworks.com/matlabcentral/answers/2132191-

[IROS 2025 Workshop] Safety-critical aerial physical interaction

2025-10-24T00:00:00+09:00

On October 24th, I am invited as a speaker in the workshop titled “Advancements in Aerial Physical Interaction”.

My talk was titled “Safety-Critical Aerial Physical Interaction.” You can download the presentation slides here.

Also, you can check out the picture of my presentation below:

Safety-critical control for aerial physical interaction in uncertain environment

2025-02-05T00:00:00+09:00

📃 Abstract: Aerial manipulation for safe physical interaction with their environments is gaining significant momentum in robotics research. In this paper, we present a disturbance-observer-based safety-critical control for a fully actuated aerial manipulator interacting with both static and dynamic structures. Our approach centers on a safety filter that dynamically adjusts the desired trajectory of the vehicle’s pose, accounting for the aerial manipulator’s dynamics, the disturbance observer’s structure, and motor thrust limits. We provide rigorous proof that the proposed safety filter ensures the forward invariance of the safety set—representing motor thrust limits—even in the presence of disturbance estimation errors. To demonstrate the superiority of our method over existing control strategies for aerial physical interaction, we perform comparative experiments involving complex tasks, such as pushing against a static structure and pulling a plug firmly attached to an electric socket. Furthermore, to highlight its repeatability in scenarios with sudden dynamic changes, we perform repeated tests of pushing a movable cart and extracting a plug from a socket. These experiments confirm that our method not only outperforms existing methods but also excels in handling tasks with rapid dynamic variations.

✊ Motivation: Consideration on two important factors of the APhI (Aerial Physical Interaction) shown below:
❗ External disturbances generated by the physical interaction
❗ Vehicle’s actuation limit such as motor thrust limit

💡 Main contributions:

✔️ No need for interaction wrench measurement or estimation
✔️ Precise motion-tracking performance achieved through a DOB-based control structure
✔️ Rigorous guarantee of motor thrust limits acquired by a CBF-based safety filter
✔️ Prevention of excessive oscillations in the desired pose and twist of the aerial manipulator
✔️ Real-world validation demonstrated through pushing and pulling experiments with both static and dynamic structures

Bibtex

@INPROCEEDINGS{byun2025safety,
  author={Byun, Jeonghyun and Kim, Yeonjun, Lee, Dongjae and Kim, H. Jin},
  booktitle={2025 International Conference on Robotics and Automation (ICRA)}, 
  title={Safety-critical control for aerial physical interaction in uncertain environment}, 
  year={2025},
  volume={},
  number={},
  pages={7526--7532},
  keywords={},
  doi={}}

Haptic-based bilateral teleoperation of aerial manipulator for extracting wedged object with compensation of human reaction time

2024-08-26T00:00:00+09:00

📃 Abstract: Bilateral teleoperation of an aerial manipulator facilitates the execution of industrial missions thanks to the combination of the aerial platform’s maneuverability and the ability to conduct complex tasks with human supervision. Heretofore, research on such operations has focused on flying without any physical interaction or exerting a pushing force on a contact surface that does not involve abrupt changes in the interaction force. In this paper, we propose a human reaction time compensating haptic-based bilateral teleoperation strategy for an aerial manipulator extracting a wedged object from a static structure (i.e., plug-pulling), which incurs an abrupt decrease in the interaction force and causes additional difficulty for an aerial platform. A haptic device composed of a 4-degree-of-freedom robotic arm and a gripper is made for the teleoperation of aerial wedged object-extracting tasks, and a haptic-based teleoperation method to execute the aerial manipulator by the haptic device is introduced. We detect the extraction of the object by the estimation of the external force exerted on the aerial manipulator and generate reference trajectories for both the aerial manipulator and the haptic device after the extraction. As an example of the extraction of a wedged object, we conduct comparative plug-pulling experiments with a quadrotor-based aerial manipulator. The results validate that the proposed bilateral teleoperation method reduces the overshoot in the aerial manipulator’s position and ensures fast recovery to its initial position after extracting the wedged object.

✊ Motivation:

❗ “Human-in-the-loop” control of an aerial manipulator (AM) to utilize humans’ decision-making ability while conducting complex tasks
❗ Transmission of sensory information gathered by the AM to the human operator to aid the decision-making process
❗ A few situations in which a human operator cannot quickly pilot the UAM due to the limitation of the humans’ reaction time

💡 Main contributions:

✔️ First attempt to conduct the haptic-based bilateral teleoperation of a UAM extracting a wedged object from a static structure
✔️ Newly produce a haptic device emulating the movement of the UAM grabbing and extracting an object wedged
✔️ Design an admittance controller that makes the haptic device compliant with the external torque exerted by a human’s hand w/o FT sensor to avoid destabilization or an excessive
✔️ Design algorithm for the detection of the object extraction overshoot in the position of the UAM after extracting the wedged object
✔️ Conduct actual aerial plug-pulling experiment via teleoperation

Bibtex

@INPROCEEDINGS{10557012,
  author={Byun, Jeonghyun and Eom, Dohyun and Kim, H. Jin},
  booktitle={2024 International Conference on Unmanned Aircraft Systems (ICUAS)}, 
  title={Haptic-Based Bilateral Teleoperation of Aerial Manipulator for Extracting Wedged Object with Compensation of Human Reaction Time}, 
  year={2024},
  volume={},
  number={},
  pages={624-630},
  keywords={Visualization;Heuristic algorithms;Force;Tactile sensors;Predictive models;Manipulators;Haptic interfaces},
  doi={10.1109/ICUAS60882.2024.10557012}}

Image-based time-varying contact force control of aerial manipulator using robust impedance filter

2024-08-24T00:00:00+09:00

📃 Abstract: The use of aerial manipulators for safe and efficient physical interaction with their surrounding environments has been gaining attention within the aerial robotics research community. In this letter, we present an image-based time-varying force tracking controller for an aerial manipulator conducting forceful interaction with a static surface. To this end, we first extract visual features from the surface using a monocular camera and calculate image feature vectors for the rotational and translational movements of the camera. Then, the RISATE (Robust Integral of SATuration Error)-based impedance filter continuously updates the desired values of the image features based on the previously designated force profile. Our stability analysis verifies that the error between the desired and actual contact force is uniformly ultimately bounded in an arbitrarily small bound with the proposed strategy even with the switching between free-flight and contact modes. Moreover, through time-varying force-tracking experiments with a quadrotor-based aerial manipulator, we validate the reproducibility and improved force-tracking performance of the proposed method.

✊ Motivation:

❗ Remote manipulation in hard-to-reach areas such as walls or windows installed on tall structures and high ceilings
❗ Regulation of the contact force exerted on the aerial manipulator’s end-effector
❗ Estimation of the surface pose using onboard sensors due to a lack of prior information on the surface in real applications

💡 Main contributions:

Bibtex

@article{byun2024image,
  title={Image-Based Time-Varying Contact Force Control of Aerial Manipulator using Robust Impedance Filter},
  author={Byun, Jeonghyun and Kim, Junha and Eom, Dohyun and Lee, Dongjae and Kim, Changhyeon and Kim, H Jin},
  journal={IEEE Robotics and Automation Letters},
  year={2024},
  publisher={IEEE}
}

[ROS1] Install CasADi and use it in ROS1 package (c++)

2024-08-22T00:00:00+09:00

CasADi is an open-source tool for nonlinear optimization and algorithmic differentiation.

In this post, I will present you how to install CasADi and use it in ROS1 package with c++ nodes. Also, since a type of quadratic programming solvers, qpOASES, is not automatically installed by just installing CasADi, I will also present you how to install qpOASES and use it with CasADi.

Operating System: Ubuntu 20.04, 64 bits
ROS version: noetic

Procedure

Step 1: Install development tools and libraries

Before installing qpOASES and CasADi, install the necessary development tools and libraries.

sudo apt-get update
sudo apt-get install -y build-essential cmake g++ python3-dev git

Step 2: Install CasADi from source

At first, download the CasADi repository from github as followss:

git clone https://github.com/casadi/casadi.git
cd casadi

Then, make a build folder and build CasADi at that folder.

mkdir build
cd build
cmake ..
make -j4

Finally, install CasADi.

sudo make install

Step 3: Add the following command lines on your package’s CMakeLists.txt

To add the dependency on CasADi in your ROS package, add the following command lines on their proper locations (They depend on your originally made CMakeLists.txt).

  (omission)
  ...
  set(CASADI_INCLUDE_DIRS /usr/local/include/casadi)
  set(CASADI_LIBRARIES /usr/local/lib/libcasadi.so)
  ...
  (syncopation)
  ...
  include_directories( ${CASADI_INCLUDE_DIRS} )
  ...
  (syncopation)
  ...
  target_link_libraries( ${catkin_LIBRARIES} ... ${CASADI_LIBRARIES})
  ...
  (omission)

Then, add the casadi header fiie on your c++-coded node.

(Option) qpOASES addition process

To use the qpOASES, go back to your default installation location and execute the following commands on your terminal. (In my case, it was “Documents” folder.)

git clone https://github.com/coin-or/qpOASES.git
cd qpOASES
mkdir build
cd build
cmake ..
make -j4
sudo make install

Then, go back to your casadi build folder (/casadi/build, in my case, ~/Documents/casadi/build/), then execute the following commands:

cmake .. -DWITH_QPOASES=ON -DWITH_LAPACK=ON
make -j4
sudo make install

Reference

https://web.casadi.org/
https://github.com/coin-or/qpOASES

Autonomous aerial perching and unperching using omnidirectional tiltrotor and switching controller

2024-05-02T00:00:00+09:00

👥 Authors: Dongjae Lee, Sunwoo Hwang, Jeonghyun Byun, Seung Jae Lee and H. Jin Kim

📃 Abstract: Aerial unperching of multirotors has received little attention as opposed to perching that has been investigated to elongate operation time. This study presents a new aerial robot capable of both perching and unperching autonomously on/from a ferromagnetic surface during flight, and a switching controller to avoid rotor saturation and mitigate overshoot during transition between free-flight and perching. To enable stable perching and unperching maneuvers on/from a vertical surface, a lightweight (~ 1 kg), fully actuated tiltrotor that can hover at 90 [deg] pitch angle is first developed. We design a perching/unperching module composed of a single servomotor and a magnet, which is then mounted on the tiltrotor. A switching controller including exclusive control modes for transitions between free-flight and perching is proposed. Lastly, we propose a simple yet effective strategy to ensure robust perching in the presence of measurement and control errors and avoid collisions with the perching site immediately after unperching. We validate the proposed framework in experiments where the tiltrotor successfully performs perching and unperching on/from a vertical surface during flight. We further show effectiveness of the proposed transition mode in the switching controller by ablation studies where large overshoot and even collision with a perching site occur. To the best of the authors’ knowledge, this work presents the first autonomous aerial unperching framework using a fully actuated tiltrotor.

✊ Motivation:

❗ Autonomous percing AND unperching capabilities can save both the energy and need for human intervention.

💡 Main contributions:

✔️ Omnidirectional tilrotor with unperching mechanism to perform both perching and unperching on a vertical ferromagnetic wall
✔️ Switching controller and motion strategy to avoid rotor saturation during perching and overshoot during unperching
✔️ Experimental validation of aerial perching and unperching on/from a vertical surface

Bibtex

@inproceedings{lee2024autonomous,
  title={Autonomous aerial perching and unperching using omnidirectional tiltrotor and switching controller},
  author={Lee, Dongjae and Hwang, Sunwoo and Byun, Jeonghyun and Lee, Seung Jae and Kim, H Jin},
  booktitle={2024 IEEE International Conference on Robotics and Automation (ICRA)},
  pages={1590--1596},
  year={2024},
  organization={IEEE}
}

Safe receding horizon motion planning with infinitesimal update interval

2024-05-01T00:00:00+09:00

📃 Abstract: Safety verification in motion planning is known to be computationally burdensome, despite its importance in robotics. In this paper, we investigate the behavior of safe receding horizon motion planners when the update interval becomes infinitesimal. By requiring the trajectory parameters to evolve continuously in time, the trajectory optimization problem is reformulated into a time-derivative form, whose decision variables are their rate of change. This results in a quadratic programming problem which directly provides safe input, and can be regarded as a real-time safety filter. The input expressivity is also enhanced by leveraging the differentiable structure of the parameter space. The proposed safety filter is experimentally validated using a wheeled ground robot in obstacle-cluttered environments. The result shows that the safety filter is capable of generating safe inputs in real-time, while addressing hundreds of constraints simultaneously.

✊ Motivation:

❗ Complex trajectory planning in a receding horizon manner still remains nonlinear and nonconvex in general, making online safety verification computationally burdensome.
❗ Since a failure to find a feasible trajectory within the trajectory update time interval may put the robotic system at a safety risk, blindly choosing a short τplan in the current form is not practicable.

💡 Main contributions:

✔️ We examine the asymptotic behavior of safe receding horizon motion planning (RHP) problems when the update interval becomes infinitesimal.
✔️ With some modifications to the optimal RHP problem, infinitesimal update interval yields a safety filter based on quadratic programming (QP).
✔️ The resulting safety filter is safe, recursively feasible, and can be solved in real-time.
✔️ Prevention of excessive oscillations in the desired pose and twist of the aerial manipulator.
✔️ We demonstrate its performance through real-world experiments.

Bibtex

@inproceedings{jang2024safe,
  title={Safe receding horizon motion planning with infinitesimal update interval},
  author={Jang, Inkyu and Hwang, Sunwoo and Byun, Jeonghyun and Kim, H Jin},
  booktitle={2024 IEEE International Conference on Robotics and Automation (ICRA)},
  pages={14244--14250},
  year={2024},
  organization={IEEE}
}

[PERSONAL RESEARCH] 3D Animation of the aerial plug-pulling task

2023-07-05T00:00:00+09:00

Aerial manipulation is one of the rising topics which can simultaneously leverage versatility of the robotic manipulator and maneuverability of the unmanned aerial vehicle (UAV).

Currently, I am conducting the research on the aerial plug-pulling task which entails two different dynamical models, a free-flight model and a plug-pulling model. Since my goal is to design an optimal controller which satisfactorily controls the given plug-pulling scenario, to validate my proposed assumptions and theorems, there needs an elaborate simulator. As a result I made a MATLAB simulation which visualizes the aerial manipulator holding onto the plug which is attached to the socket. The attached video is a 3D animation which shows the performance of my control law which is designed based on the plug-pulling situation.

Source code of a MATLAB simulation based on [2021, ICCAS, Byun] is uploaded on my private Github repository.