Journal Papers

2022

1. 

   Trajectory Control – An Effective Strategy for Controlling Multi-DOF Upper Limb Prosthetic Devices

   Yuri Gloumakov,  Joao Bimbo, and Aaron M Dollar

   IEEE Transactions on Neural Systems and Rehabilitation Engineering 2022

   Abs PDF

   Despite great innovations in upper-extremity prosthetic hardware in recent decades, controlling a multiple joint upper limb prosthesis such as an elbow/wrist/hand system is still an open clinical challenge, in large part due to an insufficient number of control inputs available to users. While simultaneous control is in its early stages, the common control approach is sequential control, in which joints and grasps are driven one at a time. In this paper, we introduce and evaluate a concept we call trajectory control, that builds upon this approach, in which motions of the wrist, elbow, and shoulder DOFs (and subsets of them) are coupled into predefined sets of coordinated trajectories; to be selected by the user and driven with a single input variable. These trajectories were designed based on an earlier motion study of activities of daily life obtained from human demonstrations. We experimentally evaluate the efficacy of our approach through a human subjects study in which tasks are performed in a virtual environment. The results show that as device complexity increased (i.e. greater number of DOFs corresponding to more proximal amputations), participants were able to complete tasks faster with trajectory control while exhibiting similar levels of body compensation when compared to sequential and simultaneous control. Additionally, participants found trajectory control to be more intuitive and displayed more natural movement.

2. 

   Force-based Simultaneous Mapping and Object Reconstruction for Robotic Manipulation

   Joao Bimbo, Andrew S Morgan, and Aaron M Dollar

   IEEE Robotics and Automation Letters 2022

   Abs PDF

   This letter presents an algorithm to simultaneously reconstruct the geometry of an unknown object and its environment via physical interactions. Applications involving highly cluttered or occluded workspaces prevent the effective use of vision. To address some of the challenges that arise, we propose an approach that instead utilizes force and torque measurements at the robot end-effector to solve for possible contact locations and probabilistically determine the occupancy likelihood on a 3D map. Our procedure constructs two occupancy maps: one fixed that represents the environment and another map that moves with the robot end-effector and reconstructs the grasped object shape, where each map informs the probability updates on the other. The algorithm is applied and tested on two scenarios: retrieving a tangled object from a scene and reconstructing the geometry of an object. We compare the results against a configuration space planner and a reinforcement learning algorithm, with our method requiring fewer collisions with the environment to extract the object.

2021

1. 

   The Vicarios Virtual Reality Interface for Remote Robotic Teleoperation

   Abdeldjallil Naceri, Dario Mazzanti,  Joao Bimbo, and 5 more authors

   Journal of Intelligent & Robotic Systems 2021

   Abs PDF

   Intuitive interaction is the cornerstone of accurate and effective performance in remote robotic teleoperation. It requires high-fidelity in control actions as well as perception (vision, haptic, and other sensory feedback) of the remote environment. This paper presents Vicarios, a Virtual Reality (VR) based interface with the aim of facilitating intuitive real-time remote teleoperation, while utilizing the inherent benefits of VR, including immersive visualization, freedom of user viewpoint selection, and fluidity of interaction through natural action interfaces. Vicarios aims to enhance the situational awareness, using the concept of viewpoint-independent mapping between the operator and the remote scene, thereby giving the operator better control in the perception-action loop. The article describes the overall system of Vicarios, with its software, hardware, and communication framework. A comparative user study quantifies the impact of the interface and its features, including immersion and instantaneous user viewpoint changes, termed “teleporting”, on users’ performance. The results show that users’ performance with the VR-based interface was either similar to or better than the baseline condition of traditional stereo video feedback, approving the realistic nature of the Vicarios interface. Furthermore, including the teleporting feature in VR significantly improved participants’ performance and their appreciation for it, which was evident in the post-questionnaire results. Vicarios capitalizes on the intuitiveness and flexibility of VR to improve accuracy in remote teleoperation.

2020

1. 

   Quasi-static analysis of planar sliding using friction patches

   M Mahdi Ghazaei Ardakani,  Joao Bimbo, and Domenico Prattichizzo

   The International Journal of Robotics Research 2020

   Abs PDF

   Flat objects lying on a surface are hard to grasp, but could be manipulated by sliding along the surface in a non-prehensile manner. This strategy is commonly employed by humans as pre-manipulation, for example to bring a cell phone to the edge of a table to pick it up. To endow robots with a similar capability, we introduce a mathematical model of planar sliding by means of a soft finger. The model reveals various aspects of interaction through frictional contacts, which can be used for planning and control. Specifically, using a quasi-static analysis we are able to derive a hybrid dynamical system to predict the motion of the object and the interaction forces. The conditions for which the object sticks to the friction patch, pivots, or completely slides against it are obtained. It is possible to find fixed points of the system and the path taken by the object to reach such configurations. Theoretical as well as comprehensive experimental results are presented.

2. 

   Hand closure model for planning top grasps with soft robotic hands

   Maria Pozzi, Sara Marullo, Gionata Salvietti, and 3 more authors

   The International Journal of Robotics Research 2020

   Abs PDF

   Automating the act of grasping is one of the most compelling challenges in robotics. In recent times, a major trend has gained the attention of the robotic grasping community: soft manipulation. Along with the design of intrinsically soft robotic hands, it is important to devise grasp planning strategies that can take into account the hand characteristics, but are general enough to be applied to different robotic systems. In this article, we investigate how to perform top grasps with soft hands according to a model-based approach, using both power and precision grasps. The so-called closure signature (CS) is used to model closure motions of soft hands by associating to them a preferred grasping direction. This direction can be aligned to a suitable direction over the object to achieve successful top grasps. The CS-alignment is here combined with a recently developed AI-driven grasp planner for rigid grippers that is adjusted and used to retrieve an estimate of the optimal grasp to be performed on the object. The resulting grasp planner is tested with multiple experimental trials with two different robotic hands. A wide set of objects with different shapes was grasped successfully

2019

1. 

   Exploiting Robot Hand Compliance and Environmental Constraints for Edge Grasps

   Joao Bimbo, Enrico Turco, Mahdi Ghazaei Ardakani, and 5 more authors

   Frontiers in Robotics and AI 2019

   Abs PDF

   This paper presents a method to grasp objects that cannot be picked directly from a table, using a soft, underactuated hand. These grasps are achieved by dragging the object to the edge of a table, and grasping it from the protruding part, performing so-called slide-to-edge grasps. This type of approach, which uses the environment to facilitate the grasp, is named Environmental Constraint Exploitation (ECE), and has been shown to improve the robustness of grasps while reducing the planning effort. The paper proposes two strategies, namely Continuous Slide and Grasp and Pivot and Re-Grasp, that are designed to deal with different objects. In the first strategy, the hand is positioned over the object and assumed to stick to it during the sliding until the edge, where the fingers wrap around the object and pick it up. In the second strategy, instead, the sliding motion is performed using pivoting, and thus the object is allowed to rotate with respect to the hand that drags it toward the edge. Then, as soon as the object reaches the desired position, the hand detaches from the object and moves to grasp the object from the side. In both strategies, the hand positioning for grasping the object is implemented using a recently proposed functional model for soft hands, the closure signature, whereas the sliding motion on the table is executed by using a hybrid force-velocity controller. We conducted 320 grasping trials with 16 different objects using a soft hand attached to a collaborative robot arm. Experiments showed that the Continuous Slide and Grasp is more suitable for small objects (e.g., a credit card), whereas the Pivot and Re-Grasp performs better with larger objects (e.g., a big book). The gathered data were used to train a classifier that selects the most suitable strategy to use, according to the object size and weight. Implementing ECE strategies with soft hands is a first step toward their use in real-world scenarios, where the environment should be seen more as a help than as a hindrance.

2018

1. 

   Design and Evaluation of a Wearable Skin Stretch Device for Haptic Guidance

   F. Chinello, C. Pacchierotti,  Joao Bimbo, and 2 more authors

   IEEE Robotics and Automation Letters Jan 2018

   Abs PDF

   We present a wearable skin stretch device for the forearm. It is composed of four cylindrical end effectors, evenly distributed around the user’s forearm. They can generate independent skin stretch stimuli at the palmar, dorsal, ulnar, and radial sides of the arm. When the four end effectors rotate in the same direction, the wearable device provides cutaneous stimuli about a desired pronation/supination of the forearm. On the other hand, when two opposite end effectors rotate in different directions, the cutaneous device provides cutaneous stimuli about a desired translation of the forearm. To evaluate the effectiveness of our device in providing navigation information , we carried out two experiments of haptic navigation. In the first one, subjects were asked to translate and rotate the forearm toward a target position and orientation, respectively. In the second experiment, subjects were asked to control a 6-DoF robotic manipulator to grasp and lift a target object. Haptic feedback provided by our wearable device improved the performance of both experiments with respect to providing no haptic feedback. Moreover, it showed similar performance with respect to sensory substitution via visual feedback, without overloading the visual channel.

2. 

   The Closure Signature: A Functional Approach to Model Underactuated Compliant Robotic Hands

   M. Pozzi, G. Salvietti,  Joao Bimbo, and 2 more authors

   IEEE Robotics and Automation Letters Jul 2018
3. 

   Real-Time Vision-Based Stiffness Mapping

   Angela Faragasso,  Joao Bimbo, Agostino Stilli, and 3 more authors

   Sensors (Basel, Switzerland) Jul 2018

2017

1. 

   Robotic tactile perception of object properties: A review

   Shan Luo,  Joao Bimbo, Ravinder Dahiya, and 1 more author

   Mechatronics Jul 2017

2016

1. 

   In-hand Object Pose Estimation using Covariance-Based Tactile To Geometry Matching

   Joao Bimbo, Shan Luo, Kaspar Althoefer, and 1 more author

   IEEE Robotics and Automation Letters Jul 2016
2. 

   Stable Grip Control on Soft Objects With Time-Varying Stiffness

   Thrishantha Nanayakkara, Allen Jiang, Marı́a del Rocı́o Armas Fernández, and 3 more authors

   IEEE Transactions on Robotics Jul 2016
3. 

   Multi-Axis Force/Torque Sensor Based on Simply-Supported Beam and Optoelectronics

   Yohan Noh,  Joao Bimbo, Sina Sareh, and 7 more authors

   Sensors Jul 2016

2015

1. 

   Global estimation of an object’s pose using tactile sensing

   Joao Bimbo, Petar Kormushev, Kaspar Althoefer, and 1 more author

   Advanced Robotics Jul 2015

Conference Papers

2020

1. 

   Trajectory Control For a Myoelectric Prosthetic Wrist

   Yuri Gloumakov,  Joao Bimbo, and Aaron M Dollar

   In MEC Symposium Conference 2020
2. 

   Trajectory Control for 3 Degree-of-Freedom Wrist Prosthesis in Virtual Reality: A Pilot Study

   Yuri Gloumakov,  Joao Bimbo, and Aaron M Dollar

   In 8th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob) 2020
3. 

   Maintaining stable grasps during highly dynamic robot trajectories

   Giandomenico Martucci,  Joao Bimbo, Domenico Prattichizzo, and 1 more author

   In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) 2020

   Abs PDF

   One of the key advantages of robots is the high speeds at which they can operate. In industrial settings, increased velocities can lead to higher throughputs and improved efficiency. Some manipulation tasks might require the robot to perform highly dynamic operations such as shaking, or swinging while grasping an object. These fast movements may produce high accelerations and thus give rise to inertial forces that can cause a grasped object to slip. In this paper a method is proposed to determine the inertial forces that arise on a grasped object during a trajectory, find the instances at which the object might slip, and avoid these slippages by changing the trajectory, namely the orientation of the object. To exemplify the usage of this approach, two grasping tasks are realised: a prehensile and a non-prehensile grasp, and strategies to successfully perform these tasks without changing the overall duration of the trajectory are defined and evaluated.

2019

1. 

   Collision detection and isolation on a robot using joint torque sensing

   Joao Bimbo, Claudio Pacchierotti, Nikos Tsagarakis, and 1 more author

   In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) 2019
2. 

   Towards a Virtual Reality Interface for Remote Robotic Teleoperation

   Abdeldjallil Naceri, Dario Mazzanti,  Joao Bimbo, and 4 more authors

   In 2019 19th International Conference on Advanced Robotics (ICAR) 2019

2018

1. 

   Transparency-optimal passivity layer design for time-domain control of multi-DoF haptic-enabled teleoperation

   Olmo Alonso Moreno Franco,  Joao Bimbo, Claudio Pacchierotti, and 3 more authors

   In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) 2018
2. 

   Transparency-oriented passivity control design for haptic-enabled teleoperation systems with multiple degrees of freedom

   Gianni Bianchini,  Joao Bimbo, Claudio Pacchierotti, and 2 more authors

   In IEEE Conference on Decision and Control (CDC) 2018
3. 

   Disposable Stiffness Sensor for Endoscopic Examination

   Angela Faragasso,  Joao Bimbo, Atsushi Yamashita, and 1 more author

   In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 2018

2017

1. 

   Teleoperation in cluttered environments using wearable haptic feedback

   Joao Bimbo, Claudio Pacchierotti, Marco Aggravi, and 2 more authors

   In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) 2017

2016

1. 

   A New Miniaturised Multi-Axis Force/Torque Sensors Based on Optoelectronic Technology and Simply-Supported Beam

   Yohan Noh,  Joao Bimbo, Agostino Stilli, and 5 more authors

   In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) 2016

2015

1. 

   Multi-axis Stiffness Sensing Device for Medical Palpation

   A Faragasso, A Stilli,  Joao Bimbo, and 2 more authors

   In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) 2015
2. 

   Finger contact sensing and the application in dexterous hand manipulation

   Hongbin Liu, Kien Cuong Nguyen, Véronique Perdereau, and 5 more authors

   Autonomous Robots 2015

2014

1. 

   Control a contact sensing finger for surface haptic exploration

   Junghwan Back,  Joao Bimbo, Yohan Noh, and 3 more authors

   In IEEE Int. Conf. on Robotics and Automation (ICRA) 2014
2. 

   Novel uniaxial force sensor based on visual information for minimally invasive surgery

   A Faragasso,  Joao Bimbo, Y Noh, and 6 more authors

   In IEEE Int. Conf. on Robotics and Automation (ICRA) 2014
3. 

   Endoscopic add-on stiffness probe for real-time soft surface characterisation in MIS

   A Faragasso, A Stilli,  Joao Bimbo, and 6 more authors

   In 36th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society (EMBC) 2014

2013

1. 

   Combining touch and vision for the estimation of an object’s pose during manipulation

   Joao Bimbo, Lakmal D Seneviratne, Kaspar Althoefer, and 1 more author

   In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) 2013

2012

1. 

   Tactile image based contact shape recognition using neural network

   Hongbin Liu, Juan Greco, Xiaojing Song, and 3 more authors

   In IEEE Conf. on Multisensor Fusion and Integration for Intelligent Systems (MFI) 2012
2. 

   Object pose estimation and tracking by fusing visual and tactile information

   Joao Bimbo, Silvia Rodriguez-Jimenez, Hongbin Liu, and 5 more authors

   In IEEE Conf. on Multisensor Fusion and Integration for Intelligent Systems (MFI) 2012
3. 

   Surface material recognition through haptic exploration using an intelligent contact sensing finger

   Hongbin Liu, Xiaojing Song,  Joao Bimbo, and 2 more authors

   In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) 2012
4. 

   A novel dynamic slip prediction and compensation approach based on haptic surface exploration

   Xiaojing Song, Hongbin Liu,  Joao Bimbo, and 2 more authors

   In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) 2012
5. 

   Adaptive grip control on an uncertain object

   Allen Jiang,  Joao Bimbo, Simon Goulder, and 5 more authors

   In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) 2012

Workshops and others

2021

1. 

   ViTac: Integrating Vision and Touch for Multimodal and Cross-Modal Perception [Editorial]

   S Luo, NF Lepora, U Martinez-Hernandez, and 2 more authors

   Frontiers in Robotics and AI 2021

2020

1. 

   Multi-axis force sensor [Patent]

   Yohan Noh,  Joao Bimbo, Kawal Rhode, and 4 more authors

   Feb 2020

   Abs PDF

   Aspects described herein provide structures for a force sensor, and force sensors using such structures, which are compact and easy to manufacture, for example by 3D printing. In particular the structures comprise a pair of stacked ring sensor elements, the ring sensor elements in turn being formed by upper and lower ring elements joined together at points around the circumference thereof by resiliently mounted connection bars. The connection bars may extend in the same plane as the rings, in which case sensitivity to torque about the axis of the rings is much reduced, such that a five-axis sensor is effectively obtained, or the connection bars may extend obliquely between the upper and lower rings of each sensor element, such that they have a directional component in the direction of the axis of the rings (the rings of each element being co-axially stacked). In this second case application of a torque about the ring axis causes the oblique connection bars to either increase or decrease their directional component in the axial direction, thus providing sensitivity to torque about the axis, and providing a compact six axis sensor.

2. Soft Hand-Environment Interaction in Grasping Tasks

   Maria Pozzi, Chiara Gaudeni,  Joao Bimbo, and 7 more authors

   In Shaping quality metric of a grasp with the manipulation task: Grasping as a sub action of object manipulation, Workshop at ICRA 2020 Feb 2020

2017

1. 

   Particle-filter-based estimation of the location and force of robot collisions using torque measurements

   Joao Bimbo, Claudio Pacchierotti, Nikos Tsagarakis, and 1 more author

   In Revisiting Contact - Turning a problem into a solution Workshop at RSS’17 [Poster] Feb 2017
2. Optimizing Damping Factors in a 3DoF Passive Two-layer Approach for Bilateral Telemanipulation

   Olmo A Moreno,  Joao Bimbo, Claudio Pacchierotti, and 2 more authors

   In Work-in-Progress session at World Haptics ’17 [Poster] Feb 2017

2015

1. Object Pose Estimation Using Tactile to Geometric Covariance Matching

   Joao Bimbo, Kaspar Althoefer, and Hongbin Liu

   In IROS Late Breaking Results Session [Poster] Feb 2015
2. Soft Fingers for Robotic Grasping

   Joao Bimbo, and Hongbin Liu

   In Perceptions on Soft-based Contact workshop at IEEE CASE 2015 [Invited Talk] Feb 2015

2013

1. Fusing Visual and Tactile Sensing for Manipulation of Unknown Objects

   Joao Bimbo, Silvia Rodriguez-Jimenez, Hongbin Liu, and 4 more authors

   In Mobile Manipulation Workshop on Interactive Perception at ICRA 2013 [Poster] Feb 2013

2012

1. 

   Object surface classificaiton based on friction properties for intelligent robotic hands

   Xiaojing Song, Hongbin Liu,  Joao Bimbo, and 2 more authors

   In World Automation Congress (WAC), 2012 Feb 2012
2. Active Perception of Objects for Robot Grasping

   Joao Bimbo, Hongbin Liu, Lakmal D Senerivatne, and 2 more authors

   In Closing the Action-Perception Loop Workshop at IROS 2012 [Presentation] Feb 2012
3. Managing Coordinate frames with ROS

   Joao Bimbo

   In Handling ROS Introductory tutorial to ROS and its use for robot in-hand manipulation Workshop at IROS [Workshop] Feb 2012
4. Finger Surface Following Control through Intrinsic Contact Sensing

   Junghwan Back,  Joao Bimbo, Mark Addison, and 6 more authors

   In Autonomous Grasping and Manipulation: An Open Challenge at ICRA [Poster] Feb 2012