Work performed during first year

With reference to Annex I of the Grant Agreement and the timing of the six research and development work packages into which the project is articulated, the work performed during the first year has dealt with the following challenges:

  • observation of dual-arm/hand manipulation activities performed by humans, where the observation is done at different levels of granularity to deduce goals and strategies at different levels of abstraction;
  • representation of scene dynamics to interpret the observation of human demonstration and provide support for reasoning methods, leading to task models including different levels of abstraction from basic actions to high level manipulation activities;
  • task dependent decision for manipulation activities, with decomposition into atomic actions, integration of context information into the grasp process, and adaptive planning in a human aware context;
  • development of suitable techniques for real-time estimation of contact/environment compliance, as well as algorithms for finger control, control of single-hand and dual-arm manipulation, and manipulation of objects commonly held by humans and robots;
  • definition of the kinematic structure of the hand and the fingers, along with the design of new types of sensors (position, force, torque, tactile) and their integration within the hand;
  • specification of suitable benchmarks to evaluate performance of dexterous single-hand and dual-arm manipulation tasks, to be classified into three categories: component level, system level and application level.

Results achieved so far

In the following the main results achieved so far are described, whereas a more detailed account with reference to the various tasks of the project is deferred to the reports of the individual work packages.


WP1 ― Observation and Learning from Human

  • Novel hand kinematic model with 25 dof’s, incorporating knowledge about interdependencies among joints of one finger and among different fingers during different manipulation tasks
  • Magnetic Resonance Imaging (MRI) processing pipeline for experimental validation

  • Sensor fusion algorithm aimed at estimating the joint variables, using a switching Extended Kalman Filter in which the prediction model is based on the human hand kinematic model


WP2 Scene, Objects and Dexterous Manipulation Representation

  • Representation of scene dynamics to interpret the observation of human demonstration and provide support for reasoning methods
  • Taxonomy for bimanual grasping and manipulation


WP3 Artificial Cognitive System for Dual-Arm/Hand Manipulation

  • Execution reasoning based on a suitable concept for decomposition of manipulation activities into executable atomic actions and clustering of knowledge for decision methods
  • Adaptive planning based on the integration of context information into grasp and path planning process and setup of real time planning in human aware context


WP4 Dual-Arm/Hand Control

  • Finger control strategies accounting for compliant joints and compliant/frictional tendons, under-actuation and joint couplings
  • Strategies for the execution of grasps and re-grasps of unknown objects with the use of visual information
  • Coordinated control strategies for bimanual manupulation
  • Impedance control strategies for safe manipulation of objects commonly held by humans and robots


WP5 Towards the Next Generation of Robotic Hands

  • Design of joint angular displacement sensor and integration in new finger kinematic design
  • Design of joint torque sensor to measure tendon tension at motor side
  • Design of strain sensor to measure tendon tension at joint side
  • New solutions for design of articulated fingers


WP6 Benchmarking and Experiments

  • Task-driven benchmarks for evaluation at component level of a hand design, the control algorithms, and sensor fusion algorithms

  • Task-driven benchmarks for evaluation at system level of cognitive performance with real robots and hands

Expected results and their potential impact

The achievement of the research objectives proposed within DEXMART will have an important impact toward the achievement of robust and versatile behaviour of artificial systems in open-ended environments providing intelligent response in unforeseen situations, and enhancing human-machine interaction.


The key innovations to bringing about this impact through the research carried out within the DEXMART project are:

  • development of original approaches to interpretation, learning, and modelling, from the observation of human manipulation at different levels of abstraction;
  • development of original approaches to task planning, coordination and execution so as to confer to the robotic system self-adapting capabilities and reactivity to changing environment and unexpected situations, also in the case of humans cooperating with it;
  • design of effective control strategies for a dual-hand/arm robot manipulator that can be easily parameterised so as to preserve smoothness during the transitions at the contact with objects;
  • design and development of new actuators, as well as new mechanical structures and materials, able to overcome the limitations of current manipulation devices;
  • development of meaningful benchmarks for dual-hand manipulation.


To sum up, the DEXMART project has the ambition to fill the gap between the use of robots in industrial environments and the use of future robots in everyday human and unstructured environments, contributing to reinforce European competitiveness in all those domains of personal and service robotics where dexterous and autonomous dual-hand manipulation capabilities are required.

Double strike in one day 16.12.2013

What an incredible coincidence: two Ex-DEXMARTians were awarded on the very same day!

Best scientific computer science PhD thesis of 2012 01.08.2013

Prize for Sven Schmidt-Rohr of Karlsruhe University, Germany