VIDEO-CONDITIONED MULTI-TASK IMITATION LEARNING FOR ROBOTIC SYSTEMS: ENHANCING ROBUSTNESS THROUGH OBJECT-CENTRIC REASONING

THIS THESIS EXPLORES ADVANCEMENTS IN ROBOTIC CONTROL FOR COMPLEX, MULTI-TASK ENVIRONMENTS, FOCUSING ON A TECHNIQUE CALLED LEARNING FROM DEMONSTRATION (LFD). THE MAIN CHALLENGE ADDRESSED IS ENABLING ROBOTS TO AUTONOMOUSLY PERFORM A WIDE RANGE OF TASKS BASED ON VIDEO DEMONSTRATIONS. IN THIS APPROACH, THE ROBOT RECEIVES A DEMONSTRATION PERFORMED BY ANOTHER AGENT IN A DIFFERENT SETTING AND MUST INTERPRET THE INPUT VIDEO TO COMPLETE THE TASK IN ITS OWN ENVIRONMENT, WHICH MAY HAVE DIFFERENT CONFIGURATIONS. THE CONTROL POLICY IS TRAINED TO HANDLE VARIOUS VARIATIONS OF A SPECIFIC TASK (E.G., PICKING AN OBJECT FROM DIFFERENT POSSIBLE LOCATIONS) OR EVEN ENTIRELY DIFFERENT TASKS (E.G., A SINGLE CONTROL POLICY THAT SOLVES BOTH PICKING AND PLACING TASKS AS WELL AS ASSEMBLY TASKS). THE GOAL IS TO GENERALIZE NOT ONLY WITH RESPECT TO THE OBJECTS BEING MANIPULATED AND THE INITIAL CONDITIONS BUT ALSO WITH RESPECT TO THE TASKS THEMSELVES. IN THIS CONTEXT, A KEY ISSUE IDENTIFIED IN THE CURRENT STATE-OF-THE-ART METHODS IS RELATED TO TARGET MISINDENTIFICATION, WHERE THE LEARNED CONTROL POLICY GENERATES VALID TRAJECTORIES, ENABLING THE ROBOT TO REACH, PICK, AND PLACE OBJECTS, BUT FREQUENTLY MANIPULATES THE WRONG OBJECT. TO SOLVE THIS PROBLEM, TWO MAIN CONSIDERATIONS WERE MADE: (1) ARCHITECTURES PROPOSED IN THE CURRENT LITERATURE ARE PREDOMINANTLY END-TO-END, TRANSLATING HIGH-DIMENSIONAL INPUTS, SUCH AS IMAGES, INTO CORRESPONDING LOW-DIMENSIONAL ACTIONS. AS A RESULT, THE MODEL MUST LEARN AN IMPLICIT REPRESENTATION THAT ENCODES BOTH THE TASK OBJECTIVE AND THE CURRENT STATE OF THE ENVIRONMENT, INCLUDING THE LOCATION OF THE TARGET OBJECT. (2) THE LEARNING PROCEDURE OPTIMIZES AN ACTION-CENTRIC METRIC, MEANING THAT IT IS NOT DIRECTLY LINKED TO TASK SUCCESS BUT INSTEAD FOCUSES ON MIMICKING THE EXPERT'S ACTIONS ON AVERAGE. THIS ACTION-FOCUSED OPTIMIZATION CAN LEAD TO POOR ENCODING OF CRITICAL INFORMATION, SUCH AS OBJECT POSITIONS. TO ADDRESS THESE LIMITATIONS, THE THESIS PROPOSES A MODULAR SYSTEM THAT DIVIDES MANIPULATION TASKS INTO TWO SUBCOMPONENTS: (1) A COGNITIVE TASK FOCUSED ON IDENTIFYING TARGET LOCATIONS, AND (2) A CONTROL TASK THAT GENERATES THE REQUIRED ACTIONS. TO SOLVE THE COGNITIVE TASK, A CONDITIONED OBJECT DETECTOR (COD) HAS BEEN DEVELOPED. THIS MODULE, GIVEN THE VIDEO DEMONSTRATION AND THE CURRENT AGENT OBSERVATION AS INPUT, PREDICTS THE CATEGORY-AGNOSTIC BOUNDING BOX RELATED TO THE TARGET OBJECT AND THE FINAL PLACING LOCATION. THIS LOW-LEVEL POSITIONAL INFORMATION IS THEN PROVIDED TO THE CONTROL MODULE, WHICH PREDICTS THE ACTIONS TO PERFORM. TO SOLVE THE CONTROL TASK THE OBJECT CONDITIONED CONTROL POLICY (OCCP) HAS BEEN PROPOSED, WHICH FOCUSES ON SOLVING THE CONTROL PROBLEM USING LOW-LEVEL POSITIONAL INFORMATION THAT CAN BE EASILY MAPPED INTO THE CORRESPONDING ACTIONS. THE FINAL SYSTEM HAS BEEN EXTENSIVELY TESTED IN SIMULATION ENVIRONMENTS, THEN IT WAS ALSO VALIDATED ON A REAL-WORLD ROBOTIC PLATFORM. THE PROPOSED METHODS SHOWED PROMISING PERFORMANCE, WITH SIGNIFICANT IMPROVEMENTS OVER BASELINE METHODS THAT LACK OBJECT-ORIENTED REASONING. THIS MODULAR, OBJECT-ORIENTED APPROACH PROVED EFFECTIVE FOR LFD TASKS, ENHANCING BOTH INTERPRETABILITY AND RELIABILITY, AS THE IDENTIFIED OBJECT LOCATIONS (BOUNDING BOXES) CLARIFY THE ROBOT’S INTENDED MOVEMENTS FOR THE END USER. IN REAL-WORLD TESTS, THE PROPOSED METHOD DEMONSTRATED ROBUSTNESS EVEN WITH LIMITED, NOISY DATA COLLECTED THROUGH TELEOPERATION, UNDERSCORING THE FEASIBILITY OF USING OBJECT-FOCUSED APPROACHES IN REAL ENVIRONMENTS. THESE RESULTS CONFIRM THAT INCORPORATING OBJECT-AWARENESS CAN SUPPORT RELIABLE AND ADAPTABLE ROBOTIC SYSTEMS, EVEN UNDER CHALLENGING CONDITIONS WITH SPARSE DATA.