publications | João Bimbo

2022

Trajectory Control – An Effective Strategy for Controlling Multi-DOF Upper Limb Prosthetic Devices

Yuri Gloumakov, Joao Bimbo, and Aaron M Dollar

IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022

Abs DOI PDF

Despite great innovations in upper-extremity prosthetic hardware in recent decades, controlling a multiple joint upper limb prosthesis such as an elbow/wrist/hand system is still an open clinical challenge, in large part due to an insufficient number of control inputs available to users. While simultaneous control is in its early stages, the common control approach is sequential control, in which joints and grasps are driven one at a time. In this paper, we introduce and evaluate a concept we call trajectory control, that builds upon this approach, in which motions of the wrist, elbow, and shoulder DOFs (and subsets of them) are coupled into predefined sets of coordinated trajectories; to be selected by the user and driven with a single input variable. These trajectories were designed based on an earlier motion study of activities of daily life obtained from human demonstrations. We experimentally evaluate the efficacy of our approach through a human subjects study in which tasks are performed in a virtual environment. The results show that as device complexity increased (i.e. greater number of DOFs corresponding to more proximal amputations), participants were able to complete tasks faster with trajectory control while exhibiting similar levels of body compensation when compared to sequential and simultaneous control. Additionally, participants found trajectory control to be more intuitive and displayed more natural movement.
Force-based Simultaneous Mapping and Object Reconstruction for Robotic Manipulation

Joao Bimbo, Andrew S Morgan, and Aaron M Dollar

IEEE Robotics and Automation Letters, 2022

Abs DOI PDF

This letter presents an algorithm to simultaneously reconstruct the geometry of an unknown object and its environment via physical interactions. Applications involving highly cluttered or occluded workspaces prevent the effective use of vision. To address some of the challenges that arise, we propose an approach that instead utilizes force and torque measurements at the robot end-effector to solve for possible contact locations and probabilistically determine the occupancy likelihood on a 3D map. Our procedure constructs two occupancy maps: one fixed that represents the environment and another map that moves with the robot end-effector and reconstructs the grasped object shape, where each map informs the probability updates on the other. The algorithm is applied and tested on two scenarios: retrieving a tangled object from a scene and reconstructing the geometry of an object. We compare the results against a configuration space planner and a reinforcement learning algorithm, with our method requiring fewer collisions with the environment to extract the object.

2021

ViTac: Integrating Vision and Touch for Multimodal and Cross-Modal Perception [Editorial]

S Luo, NF Lepora, U Martinez-Hernandez, and 2 more authors

Frontiers in Robotics and AI, 2021
The Vicarios Virtual Reality Interface for Remote Robotic Teleoperation

Abdeldjallil Naceri, Dario Mazzanti, Joao Bimbo, and 5 more authors

Journal of Intelligent & Robotic Systems, 2021

Abs PDF

Intuitive interaction is the cornerstone of accurate and effective performance in remote robotic teleoperation. It requires high-fidelity in control actions as well as perception (vision, haptic, and other sensory feedback) of the remote environment. This paper presents Vicarios, a Virtual Reality (VR) based interface with the aim of facilitating intuitive real-time remote teleoperation, while utilizing the inherent benefits of VR, including immersive visualization, freedom of user viewpoint selection, and fluidity of interaction through natural action interfaces. Vicarios aims to enhance the situational awareness, using the concept of viewpoint-independent mapping between the operator and the remote scene, thereby giving the operator better control in the perception-action loop. The article describes the overall system of Vicarios, with its software, hardware, and communication framework. A comparative user study quantifies the impact of the interface and its features, including immersion and instantaneous user viewpoint changes, termed “teleporting”, on users’ performance. The results show that users’ performance with the VR-based interface was either similar to or better than the baseline condition of traditional stereo video feedback, approving the realistic nature of the Vicarios interface. Furthermore, including the teleporting feature in VR significantly improved participants’ performance and their appreciation for it, which was evident in the post-questionnaire results. Vicarios capitalizes on the intuitiveness and flexibility of VR to improve accuracy in remote teleoperation.

2020

Quasi-static analysis of planar sliding using friction patches

M Mahdi Ghazaei Ardakani, Joao Bimbo, and Domenico Prattichizzo

The International Journal of Robotics Research, 2020

Abs PDF

Flat objects lying on a surface are hard to grasp, but could be manipulated by sliding along the surface in a non-prehensile manner. This strategy is commonly employed by humans as pre-manipulation, for example to bring a cell phone to the edge of a table to pick it up. To endow robots with a similar capability, we introduce a mathematical model of planar sliding by means of a soft finger. The model reveals various aspects of interaction through frictional contacts, which can be used for planning and control. Specifically, using a quasi-static analysis we are able to derive a hybrid dynamical system to predict the motion of the object and the interaction forces. The conditions for which the object sticks to the friction patch, pivots, or completely slides against it are obtained. It is possible to find fixed points of the system and the path taken by the object to reach such configurations. Theoretical as well as comprehensive experimental results are presented.
Hand closure model for planning top grasps with soft robotic hands

Maria Pozzi, Sara Marullo, Gionata Salvietti, and 3 more authors

The International Journal of Robotics Research, 2020

Abs PDF

Automating the act of grasping is one of the most compelling challenges in robotics. In recent times, a major trend has gained the attention of the robotic grasping community: soft manipulation. Along with the design of intrinsically soft robotic hands, it is important to devise grasp planning strategies that can take into account the hand characteristics, but are general enough to be applied to different robotic systems. In this article, we investigate how to perform top grasps with soft hands according to a model-based approach, using both power and precision grasps. The so-called closure signature (CS) is used to model closure motions of soft hands by associating to them a preferred grasping direction. This direction can be aligned to a suitable direction over the object to achieve successful top grasps. The CS-alignment is here combined with a recently developed AI-driven grasp planner for rigid grippers that is adjusted and used to retrieve an estimate of the optimal grasp to be performed on the object. The resulting grasp planner is tested with multiple experimental trials with two different robotic hands. A wide set of objects with different shapes was grasped successfully
Multi-axis force sensor [Patent]

Yohan Noh, Joao Bimbo, Kawal Rhode, and 4 more authors

Feb 2020

US Patent App. 16/340,020

Abs PDF

Aspects described herein provide structures for a force sensor, and force sensors using such structures, which are compact and easy to manufacture, for example by 3D printing. In particular the structures comprise a pair of stacked ring sensor elements, the ring sensor elements in turn being formed by upper and lower ring elements joined together at points around the circumference thereof by resiliently mounted connection bars. The connection bars may extend in the same plane as the rings, in which case sensitivity to torque about the axis of the rings is much reduced, such that a five-axis sensor is effectively obtained, or the connection bars may extend obliquely between the upper and lower rings of each sensor element, such that they have a directional component in the direction of the axis of the rings (the rings of each element being co-axially stacked). In this second case application of a torque about the ring axis causes the oblique connection bars to either increase or decrease their directional component in the axial direction, thus providing sensitivity to torque about the axis, and providing a compact six axis sensor.
Trajectory Control For a Myoelectric Prosthetic Wrist

Yuri Gloumakov, Joao Bimbo, and Aaron M Dollar

In MEC Symposium Conference, Feb 2020
Trajectory Control for 3 Degree-of-Freedom Wrist Prosthesis in Virtual Reality: A Pilot Study

Yuri Gloumakov, Joao Bimbo, and Aaron M Dollar

In 8th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob), Feb 2020
Maintaining stable grasps during highly dynamic robot trajectories

Giandomenico Martucci, Joao Bimbo, Domenico Prattichizzo, and 1 more author

In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Feb 2020

Abs PDF

One of the key advantages of robots is the high speeds at which they can operate. In industrial settings, increased velocities can lead to higher throughputs and improved efficiency. Some manipulation tasks might require the robot to perform highly dynamic operations such as shaking, or swinging while grasping an object. These fast movements may produce high accelerations and thus give rise to inertial forces that can cause a grasped object to slip. In this paper a method is proposed to determine the inertial forces that arise on a grasped object during a trajectory, find the instances at which the object might slip, and avoid these slippages by changing the trajectory, namely the orientation of the object. To exemplify the usage of this approach, two grasping tasks are realised: a prehensile and a non-prehensile grasp, and strategies to successfully perform these tasks without changing the overall duration of the trajectory are defined and evaluated.
Soft Hand-Environment Interaction in Grasping Tasks

Maria Pozzi, Chiara Gaudeni, Joao Bimbo, and 7 more authors

In Shaping quality metric of a grasp with the manipulation task: Grasping as a sub action of object manipulation, Workshop at ICRA 2020, Feb 2020

2019

Collision detection and isolation on a robot using joint torque sensing

Joao Bimbo, Claudio Pacchierotti, Nikos Tsagarakis, and 1 more author

In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Feb 2019
Exploiting Robot Hand Compliance and Environmental Constraints for Edge Grasps

Joao Bimbo, Enrico Turco, Mahdi Ghazaei Ardakani, and 5 more authors

Frontiers in Robotics and AI, Feb 2019

Abs DOI PDF

This paper presents a method to grasp objects that cannot be picked directly from a table, using a soft, underactuated hand. These grasps are achieved by dragging the object to the edge of a table, and grasping it from the protruding part, performing so-called slide-to-edge grasps. This type of approach, which uses the environment to facilitate the grasp, is named Environmental Constraint Exploitation (ECE), and has been shown to improve the robustness of grasps while reducing the planning effort. The paper proposes two strategies, namely Continuous Slide and Grasp and Pivot and Re-Grasp, that are designed to deal with different objects. In the first strategy, the hand is positioned over the object and assumed to stick to it during the sliding until the edge, where the fingers wrap around the object and pick it up. In the second strategy, instead, the sliding motion is performed using pivoting, and thus the object is allowed to rotate with respect to the hand that drags it toward the edge. Then, as soon as the object reaches the desired position, the hand detaches from the object and moves to grasp the object from the side. In both strategies, the hand positioning for grasping the object is implemented using a recently proposed functional model for soft hands, the closure signature, whereas the sliding motion on the table is executed by using a hybrid force-velocity controller. We conducted 320 grasping trials with 16 different objects using a soft hand attached to a collaborative robot arm. Experiments showed that the Continuous Slide and Grasp is more suitable for small objects (e.g., a credit card), whereas the Pivot and Re-Grasp performs better with larger objects (e.g., a big book). The gathered data were used to train a classifier that selects the most suitable strategy to use, according to the object size and weight. Implementing ECE strategies with soft hands is a first step toward their use in real-world scenarios, where the environment should be seen more as a help than as a hindrance.
Towards a Virtual Reality Interface for Remote Robotic Teleoperation

Abdeldjallil Naceri, Dario Mazzanti, Joao Bimbo, and 4 more authors

In 2019 19th International Conference on Advanced Robotics (ICAR), Feb 2019

DOI

2018

Design and Evaluation of a Wearable Skin Stretch Device for Haptic Guidance

F. Chinello, C. Pacchierotti, Joao Bimbo, and 2 more authors

IEEE Robotics and Automation Letters, Jan 2018

Abs DOI PDF

We present a wearable skin stretch device for the forearm. It is composed of four cylindrical end effectors, evenly distributed around the user’s forearm. They can generate independent skin stretch stimuli at the palmar, dorsal, ulnar, and radial sides of the arm. When the four end effectors rotate in the same direction, the wearable device provides cutaneous stimuli about a desired pronation/supination of the forearm. On the other hand, when two opposite end effectors rotate in different directions, the cutaneous device provides cutaneous stimuli about a desired translation of the forearm. To evaluate the effectiveness of our device in providing navigation information , we carried out two experiments of haptic navigation. In the first one, subjects were asked to translate and rotate the forearm toward a target position and orientation, respectively. In the second experiment, subjects were asked to control a 6-DoF robotic manipulator to grasp and lift a target object. Haptic feedback provided by our wearable device improved the performance of both experiments with respect to providing no haptic feedback. Moreover, it showed similar performance with respect to sensory substitution via visual feedback, without overloading the visual channel.
The Closure Signature: A Functional Approach to Model Underactuated Compliant Robotic Hands

M. Pozzi, G. Salvietti, Joao Bimbo, and 2 more authors

IEEE Robotics and Automation Letters, Jul 2018

DOI
Real-Time Vision-Based Stiffness Mapping

Angela Faragasso, Joao Bimbo, Agostino Stilli, and 3 more authors

Sensors (Basel, Switzerland), Jul 2018
Transparency-optimal passivity layer design for time-domain control of multi-DoF haptic-enabled teleoperation

Olmo Alonso Moreno Franco, Joao Bimbo, Claudio Pacchierotti, and 3 more authors

In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Jul 2018
Transparency-oriented passivity control design for haptic-enabled teleoperation systems with multiple degrees of freedom

Gianni Bianchini, Joao Bimbo, Claudio Pacchierotti, and 2 more authors

In IEEE Conference on Decision and Control (CDC), Jul 2018
Disposable Stiffness Sensor for Endoscopic Examination

Angela Faragasso, Joao Bimbo, Atsushi Yamashita, and 1 more author

In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Jul 2018

2017

Teleoperation in cluttered environments using wearable haptic feedback

Joao Bimbo, Claudio Pacchierotti, Marco Aggravi, and 2 more authors

In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Jul 2017
Particle-filter-based estimation of the location and force of robot collisions using torque measurements

Joao Bimbo, Claudio Pacchierotti, Nikos Tsagarakis, and 1 more author

In Revisiting Contact - Turning a problem into a solution Workshop at RSS’17 [Poster], Jul 2017
Optimizing Damping Factors in a 3DoF Passive Two-layer Approach for Bilateral Telemanipulation

Olmo A Moreno, Joao Bimbo, Claudio Pacchierotti, and 2 more authors

In Work-in-Progress session at World Haptics ’17 [Poster], Jul 2017
Robotic tactile perception of object properties: A review

Shan Luo, Joao Bimbo, Ravinder Dahiya, and 1 more author

Mechatronics, Jul 2017

2016

In-hand Object Pose Estimation using Covariance-Based Tactile To Geometry Matching

Joao Bimbo, Shan Luo, Kaspar Althoefer, and 1 more author

IEEE Robotics and Automation Letters, Jul 2016
Stable Grip Control on Soft Objects With Time-Varying Stiffness

Thrishantha Nanayakkara, Allen Jiang, Marı́a del Rocı́o Armas Fernández, and 3 more authors

IEEE Transactions on Robotics, Jul 2016
A New Miniaturised Multi-Axis Force/Torque Sensors Based on Optoelectronic Technology and Simply-Supported Beam

Yohan Noh, Joao Bimbo, Agostino Stilli, and 5 more authors

In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Jul 2016
Multi-Axis Force/Torque Sensor Based on Simply-Supported Beam and Optoelectronics

Yohan Noh, Joao Bimbo, Sina Sareh, and 7 more authors

Sensors, Jul 2016

2015

Multi-axis Stiffness Sensing Device for Medical Palpation

A Faragasso, A Stilli, Joao Bimbo, and 2 more authors

In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Jul 2015
Finger contact sensing and the application in dexterous hand manipulation

Hongbin Liu, Kien Cuong Nguyen, Véronique Perdereau, and 5 more authors

Autonomous Robots, Jul 2015
Global estimation of an object’s pose using tactile sensing

Joao Bimbo, Petar Kormushev, Kaspar Althoefer, and 1 more author

Advanced Robotics, Jul 2015
Object Pose Estimation Using Tactile to Geometric Covariance Matching

Joao Bimbo, Kaspar Althoefer, and Hongbin Liu

In IROS Late Breaking Results Session [Poster], Jul 2015
Soft Fingers for Robotic Grasping

Joao Bimbo and Hongbin Liu

In Perceptions on Soft-based Contact workshop at IEEE CASE 2015 [Invited Talk], Jul 2015

2014

Control a contact sensing finger for surface haptic exploration

Junghwan Back, Joao Bimbo, Yohan Noh, and 3 more authors

In IEEE Int. Conf. on Robotics and Automation (ICRA), Jul 2014
Novel uniaxial force sensor based on visual information for minimally invasive surgery

A Faragasso, Joao Bimbo, Y Noh, and 6 more authors

In IEEE Int. Conf. on Robotics and Automation (ICRA), Jul 2014
Endoscopic add-on stiffness probe for real-time soft surface characterisation in MIS

A Faragasso, A Stilli, Joao Bimbo, and 6 more authors

In 36th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (EMBC), Jul 2014

2013

Combining touch and vision for the estimation of an object’s pose during manipulation

Joao Bimbo, Lakmal D Seneviratne, Kaspar Althoefer, and 1 more author

In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Jul 2013
Fusing Visual and Tactile Sensing for Manipulation of Unknown Objects

Joao Bimbo, Silvia Rodriguez-Jimenez, Hongbin Liu, and 4 more authors

In Mobile Manipulation Workshop on Interactive Perception at ICRA 2013 [Poster], Jul 2013

2012

Intelligent Fingertip Sensing for Contact Information Identification

Hongbin Liu, Xiaojing Song, Joao Bimbo, and 2 more authors

Advances in Reconfigurable Mechanisms and Robots I, Jul 2012
Tactile image based contact shape recognition using neural network

Hongbin Liu, Juan Greco, Xiaojing Song, and 3 more authors

In IEEE Conf. on Multisensor Fusion and Integration for Intelligent Systems (MFI), Jul 2012
Object pose estimation and tracking by fusing visual and tactile information

Joao Bimbo, Silvia Rodriguez-Jimenez, Hongbin Liu, and 5 more authors

In IEEE Conf. on Multisensor Fusion and Integration for Intelligent Systems (MFI), Jul 2012
Object surface classificaiton based on friction properties for intelligent robotic hands

Xiaojing Song, Hongbin Liu, Joao Bimbo, and 2 more authors

In World Automation Congress (WAC), 2012, Jul 2012
Surface material recognition through haptic exploration using an intelligent contact sensing finger

Hongbin Liu, Xiaojing Song, Joao Bimbo, and 2 more authors

In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Jul 2012
A novel dynamic slip prediction and compensation approach based on haptic surface exploration

Xiaojing Song, Hongbin Liu, Joao Bimbo, and 2 more authors

In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Jul 2012
Adaptive grip control on an uncertain object

Allen Jiang, Joao Bimbo, Simon Goulder, and 5 more authors

In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Jul 2012
Active Perception of Objects for Robot Grasping

Joao Bimbo, Hongbin Liu, Lakmal D Senerivatne, and 2 more authors

In Closing the Action-Perception Loop Workshop at IROS 2012 [Presentation], Jul 2012
Managing Coordinate frames with ROS

Joao Bimbo

In Handling ROS Introductory tutorial to ROS and its use for robot in-hand manipulation Workshop at IROS [Workshop], Jul 2012
Finger Surface Following Control through Intrinsic Contact Sensing

Junghwan Back, Joao Bimbo, Mark Addison, and 6 more authors

In Autonomous Grasping and Manipulation: An Open Challenge at ICRA [Poster], Jul 2012