Hello COMPARErs! There has been quite a lot of activity in the robot manipulation ecosystem. In addition to the many assets that continue to be add to our repositories and new competitions underway, we are very excited to report that a paper reviewing work-in-progress in the COMPARE Ecosystem to develop standards and guidelines to improve the modularity of open-source components in robot grasping and manipulation pipelines received the Best Paper Award at The 3rd AAAI Fall Symposium on Unifying Representations for Robot Application Development (UR-RAD)! Check out the paper and presentation slides. Thank you to the organizers for recognizing our work! [LinkedIn post] We’ve got quite a few modular components and pipelines under development that we are eager to share with the community soon. Thanks for staying engaged with COMPARE and please continue to share your work and questions to keep the ecosystem going. Cheers! 
Huajing Zhao from UMass Lowell (right) with fellow Best Paper Award nominee from Oregon State University |
Here’s what you may have missed and what’s coming up soon:Several new benchmarking assets have been added including new task artifacts, tools, testbeds, leaderboards, and object datasets, with new competitions announced, and NeurIPS and WRS 2025 happening next month!💬 = recent discussions in the  🤖 = recent additions to  |
🤖 Physical Objects and Artifacts: New entry (11 total): NIST Assembly Task Board (ATB) M1: Benchmarking task for robot systems to insert the given pegs into their corresponding holes on the task board in a pre-specified order, featuring various geometries and clearance levels. 
🤖 Tools and Testbeds: New entries (14 total): ClutterGen: ClutterGen is a physically compliant simulation scene generator capable of producing highly diverse, cluttered, and stable scenes for robot learning. Generating such scenes is challenging as each object must adhere to physical laws like gravity and collision. 
ManipBench: ManipBench is a novel benchmark to evaluate the low-level robot manipulation reasoning capabilities of VLMs across various dimensions, including how well they understand object-object interactions and deformable object manipulation. 
ManipulationNet: ManipulationNet is a community-driven global infrastructure that enables benchmarking robot manipulation research at scale with any robot at anytime and anywhere, providing both hardware and open-source software to host standardized benchmarking tasks, allowing participants to submit solutions from their own locations at anytime in a distributed manner, and then get their performance evaluated in a centralized manner to shine on global leaderboards. 
🤖 Leaderboards: ManipulationNet: ManipulationNet organizes benchmarking tasks in two tracks: 1) Physical Skills Track that evaluates robot abilities in physical interaction-rich tasks; and 2) Embodied Reasoning Track that challenges robot capabilities in grounding language, vision, and language + vision prompts into effective manipulation actions. 
RoboChallenge: A real-world robotics testing and evaluation platform where researchers and developers can validate and compare their robot policies in a unified environment—spanning from fundamental tasks to complex real-world scenarios. 
|
🤖 Competitions: 2025 BEHAVIOR Challenge: BEHAVIOR (Benchmark for Everyday Household Activities in Virtual, Interactive, and Realistic environments) is a large-scale embodied AI benchmark with 1,000 defined household tasks grounded in real human needs. These tasks introduce long-horizon mobile manipulation challenges in realistic settings, bridging the gap between current research and real-world, human-centric applications. The 1st BEHAVIOR Challenge will take place at NeurIPS 2025. 
AI for Industry Challenge: Brought to you by Intrinsic and Open Robotics, the AI for Industry Challenge targets a high-value bottleneck in modern manufacturing: electronics assembly. Train an AI model using open-source simulators (e.g., Isaac Sim, MuJoCo, Gazebo), leveraging ROS for communication. Finalists will deploy their models from simulation to a physical workcell which will be hosted at Intrinsic’s HQ. The top five solutions and teams win a share of the $180,000 prize pool. 
What’s coming up soon?
NeurIPS 2025: The 39th Annual Conference on Neural Information Processing Systems, December 2-7, 2025, San Diego, California, USA 
WRS 2025: World Robot Summit, Manufacturing Robotics Challenge, December 12-14, 2025, Aichi, Japan |
