[Documentation] [TitleIndex] [WordIndex



This package implements the logic which connects different functionalities such as (passive) scene recognition, object pose prediction and Next-Best-View estimation to Active Scene Recognition. In particular, it is the interface for executing different 3D-object-search procedures one of which Active Scene Recognition is. Active Scene Recognition is equivalent to the "object search", which consists of the "direct search" and "indirect search" modes. The search will be executed by using different packages of the asr framework.


The state_machine controlls the flow of the 3D-object-search. There are different approches (modes) implemented to realise the 3D-object-search. The common goal is to search for objects in the search space and to conclued which scenes they represent. The procedure is called the active_scene_recognition. To conclude which scenes the found objects represent a scene model is needed. Checkout AsrResourcesForActiveSceneRecognition to look up how scenes are recorded and trained.

During the 3D-object-search different run time information will be generated and saved under the path "~/log/" according to the start time. The information are log-files, a rosbag file and some other. So it is possible to reproduce and comprehend the executed searches.

There are two abort criteria for the 3D-object-search. One is if all objects of all scenes are detected and found. The second depends on a subset of the complete patterns (found scenes) and their confidence. This can be useful if scenes overlap and only one has to be recognized and the object sets do not fit. The subset can be specified over a custom python script specified for a concrete scene set. An example:

#!/usr/bin/env python

from asr_world_model.msg import CompletePattern

def evaluateCompletePatterns(completePatterns): for completePattern in completePatterns: if ((completePattern.patternName = "Gedeck_Ende" or completePattern.patternName = "Gedeck_Schrank_2" or completePattern.patternName = "Spuelkorb_3" or completePattern.patternName = "gedeck") and completePattern.confidence >= 0.85): return True

return False

To start active_scene_recognition in indirect search mode, you have to change the param/params.yaml. There are multiple configs which you can change:

The basic idea behind the indirect search is to use prior knowledge to reduce the number of poses and time to search for objects. The prior knowledge consists of scenes which represent objects and their relative relations between each other. Starting from a found object hypothesis will be generated (in the SCENE_RECOGNITION and OBJECT_POSE_PREDICTION) were the other objects of the scenes could be located. The hypothesis will be used to generate a NBV (in NBV_SET_POINT_CLOUD and NBV_CALCULATION) to search for the missing objects. If no objects are found the rest of the hypothesis will be used for generating new NBVs. If one is found new predictions will be made for the missing objects and the cylce starts again.

To start the indirect search an object has to be in front of the robot so that it is possible to predict poses of the missing objects. Because of that the search starts in INDIRECT_SEARCH_INIT with an OBJECT_DETECTION. To overcome this restriction the indirect search can be combined with the direct search to the object search.




To start active_scene_recognition in direct search mode, you have to change the param/params.yaml. There are multiple configs which you can change:

The basic idea behind the direct search is to search for objects without the use of prior knowledge. The views to take (GET_GOAL_CAMERA_POSE) will be handled by the asr_direct_search_manager. There are different approches to generate the views. All share the common goal to cover the search space with the views. So that sooner or later the objects will be found.

However, without the use of prior knowledge the 3D-object-search can take a while if each view covers only a small amout of the search space. To overcome this issue the direct search can use prior knowledge to prioritise some views over some other. At the current time the knowlege is based on the object_poses_from_demonstrations_given. As long as views are left the OBJECT_LOCATIONS_BASED_SEARCH will be executed. After that the SCENE_EXPLORATION starts which just takes the views in a given order.

The use of the prior knowledge will be encapsulated by the asr_direct_search_manager. For the state_machine the OBJECT_LOCATIONS_BASED_SEARCH and SCENE_EXPLORATION works the same way. So only the state_machine of the OBJECT_LOCATIONS_BASED_SEARCH will be shown.

The CHECK_SEARCH_FINISHED wil be used to handle an early end of the search, because of StopAtfirst or !SearchAllobject .



To start active_scene_recognition in object search mode, you have to change the param/params.yaml. There are multiple configs which you can change:

The basic idea behind the object search is to combine the indirect and direct search. The main weak spot of the indirect search is that it needs a found object to generated predictions for the other missing objects. Because of that, the direct search will be executed beforehand until an object has been found.

It is possible that the indirect search is not able to produce new predictions of hypothesis for the missing objects, because the robot has searched at each possible spot were hypothesis have been predicted. This could happen if there are multiple scenes which do not share the same objects. If this happens the direct search will be executed again.

The object search is designed in a way that the direct and indirect search do search at similiar spots after the same objects.


To start active_scene_recognition in random search mode, you have to change the param/params.yaml. There are multiple configs which you can change:

The basic idea behind the random search is, that the robot will move to a random position with a random orientation and tilt, in each iteration step. A simplified iteration looks like



In this mode, the asr_next_best_view will not be used. The key difference to the others modes is GENERATE_RANDOM_POSE. It generates a pose and validates if the pose is allowed.

The generated position of the robot will be limited by the size of the room. The orientation is generated between 0° and 360°. The tilt of the PTU config will be in its possible range and pan will be constant 0, because pan is almost redundant to the orientation of the robot.

The validation starts by checking if the generated position is approachable by the robot (uses the asr_robot_model_services).

If !AllowSameViewInRandomSearch is false it will be checked if the generated pose is approximately equal to one of the earlier poses. Two robot states are approx_equal if the difference of the position, orientation and PTU config are under a specific threshold. The threshold of the position is the radius of the robot's volume. For orientation, it is the half of the frustum's width. And for the tilt the half of the frustum's height.

In the progress of the random search, the generation will take longer time, because there are more old robot_states to compare with and the possibility that two states are approx_equal increases.

To start active_scene_recognition in cropbox search mode, you have to change the param/params.yaml. There are multiple configs which you can change:

The basic idea behind the cropbox search is to define cropboxes where the robot should search for objects. The cropboxes can be tables and shelfs. If the searched objects are in the cropboxes, they should be found sooner or later.

The state automat works similar to the one of the indirect_search. The key difference is that the point_cloud will be set only once in the beginning in CROPBOX_SEARCH_INIT. The CROPBOX_SEARCH is the same as the RELATION_BASED_SEARCH of the indirect_search.


The mOmegaUtility and mOmegaBase, params of asr_next_best_view, can be adjusted with dynamic reconfigure so that asr_next_best_view will take views with more normals.

The generation of the point cloud takes place in two steps. At first, the points will be generated in the area of the whole room. The step between two points on the ground is the half of the frustums width and in the height the half of the frustums height.

The second part takes place in the NBV::setPointCloud. The point cloud, received from the state_automat, will be filtered to the cropboxes. Additionally, normals will be added to the point cloud points, which are defined per cropbox. The mesh of the points will be spheres (also defined in the asr_next_best_view).

Before you can run the active_scene_recognition you have to define the cropboxes. Each cropbox is defined by a min_x, min_y, min_z and max_x, max_y, max_z. To get more flexibility you can define multiple normals per cropbox. The normals will be added to all points which are in the cropbox. The cropboxes can overlap.


Cropbox Record

To start active_scene_recognition in cropbox record mode, you have to change the param/params.yaml.

This mode works similar to cropbox search. The key difference is that in this mode no object will be searched and scenes recognized. Instead the Views of the asr_next_best_view will be recorded and saved in a XML file. The path for the XML file will be configured in the launch file(s) (param name= !CropBoxRecordingPath).

The recorded poses can be retraced later with the asr_direct_search_manager (mode 2: recording_manager). This mode is useful when a cropbox for the whole room will be used, because the asr_next_best_view needs much time. Also the generated poses can be filtered and sorted later on.

A tutorial can be found under AsrDirectSearchManagerTutorialRecord.

To start active_scene_recognition in grid init record search mode, you have to change the param/params.yaml.

This mode works similar to direct search. The key difference is that in this mode no object will be searched and scenes recognized. Instead the Views of the asr_direct_search_manager will be recorded and saved in a XML file. The path for the XML file will be configured in the launch file(s) (param name= initializedGridFilePath).

The recorded poses can be retraced later with the asr_direct_search_manager (mode 1: grid_manager). This mode is needed to initialize the grid (with asr_direct_search_manager: mode 3 grid_initialisation). Before the initialisation the grid is saved as an xml file containing the grid points (robot poses). However, we need the corresponding camera poses to use the reorder function based on prior knowledge. Instead of doing the manual transforming between the robot and camera poses, we save the transforming from TF.

This mode is part of the tutorial AsrDirectSearchManagerTutorialGrid.


Create "log" directory under home -> ~/log/

Needed packages













Start system

The system can be started through the start-scripts in the package asr_resources_for_active_scene_recognition.

To start them manually most of the packages of the active_scene_recognition have to be started before.

After the start the initialObjects can be entered. They can be used in the indirect search so save time in the first search step. They will not be used in the other modes.


In simulation the object_detection and moving of the robot will be simulated.

roslaunch asr_state_machine scene_exploration_sim.launch


roslaunch asr_state_machine scene_exploration.launch

ROS Nodes

Subscribed Topics

Published Topics


Launch file


!IntermediateObject _XXX.xml:_ A file containing the _intermediate_objects, which can be used in the direct_search mode. If !SearchAllobject==false only the contained objects will be searched and not all. The XXX will be replaced with the current database name. So the name has not to be changed manually after a database change !CompletePatterns_XXX.py: A optional file which can be used to have a different abort criterion, beside found_all_objects. The abort criterion is found_all_required_scenes. The python snippet should contain a method which gets all CompletePatterns and returns true or false depending on the CompletePatterns. The XXX works analog here, too. For an example see chapter functionality

!CropBoxRecordingPath: The path to a file to store the recorded poses by the Cropbox Record mode

initializedGridFilePath: The path to a file to store the recorded poses by the Grid init record search mode

yaml file


mode: The mode of the 3D-object-search. 1 for direct search only, 2 for indirect search only, 3 for object search, 4 for random search, 5 for cropbox search, 6 for cropbox record, 7 for grid init record search !UseFakeMoveBase: Use FakeMoveBase if robot should be set on target pos, instead of moving to it. This can be useful in simulation to save time and ignore errors in navigation !SearchAllobject: Only for direct search -> search for all objects or only intermediate objects specified in !IntermediateObject _XXX.xml !StopAtfirst: Only for direct search -> stop search after first object is found

enableSmachViewer: If you want to use smach viewer you can set this to true. However, it will cost a lot of CPU usage, despite if smach viewer runs

percentageTooManyDeactivatedNormals: The percentage to determine when too many normals are deactivated during SET_NBV_POINT_CLOUD. -> Abort if numberOfDeletedNormals / numberOfActiveNormals <= percentageTooManyDeactivatedNormals min_utility_for_moving: The minimum utility value of a NBV estimate that is required to consider moving to it. If utility (not rating!) is below, return no_nbv_found filter_viewports: Filter viewports of NBV which are approx equale to already visited viewports !AllowSameViewInRandomSearch: Only for random search -> if false robot will not move to approx equale robot states again

Placeholder(!ObjectType /Identifier) !InCropboxSearch: Only for cropbox search/record -> the Type/Identifier of the points in the point cloud. Only for information purpose !RoomHeight: Only for cropbox search/record -> the height of the room. Will be used for generating the point cloud !SavePointClouds: Used to save pointclouds in the log folder so they can be debugged in nbv later

recoverLostHypothesis: If pointcloud hypothesis should be kept when they are removed by percentageTooManyDeactivatedNormals or min_utility_for_moving

Needed Services

Needed Actions


Tutorial to start indirect search: AsrStateMachineIndirectSearchTutorial

Tutorial to start direct search: AsrStateMachineDirectSearchTutorial

To start the object search do all steps like described in AsrStateMachineIndirectSearchTutorial and AsrStateMachineDirectSearchTutorial. After that the mode has to be changed to 3 for object search

Tutorial to start random search: AsrStateMachineRandomSearchTutorial

Tutorial to start cropbox search: AsrStateMachineCropboxSearchTutorial

Cropbox record is part of AsrDirectSearchManagerTutorialRecord

Grid init record is part of AsrDirectSearchManagerTutorialGrid

2024-07-20 12:40