Are you curious about how PoDIUM ensures the reliability and impact of its CCAM (Cooperative, Connected, and Automated Mobility) solutions? Meet Vicomtech, our partner leading the development of PoDIUM’s evaluation methodology. This robust framework is key to assessing the technical performance, user acceptance, and socioeconomic benefits of the project, providing the foundation for successful deployment in real-world scenarios.
Vicomtech is an applied research centre for Interactive Computer Graphics and Multimedia in San Sebastian (Spain). It is a non-profit association, founded in 2001 as a joint venture by the Fraunhofer INI- GraphicsNet Foundation and the EiTB Broadcasting Group. Vicomtech provides society with technology by transfer of primary research to industry, which is achieved through collaborative R&D projects such as PoDIUM. Vicomtech’s expertise lies in computer vision, computer graphics and interaction; technologies which it applies in multiple sectors.
An overview of PoDIUM’s evaluation methodology
To ensure that PoDIUM’s outcomes are reliable, reproducible, and comparable across different scenarios and environments, Vicomtech has developed a robust evaluation methodology. This comprehensive approach assesses the technical performance, user acceptance, and socioeconomic impact of the project’s CCAM solutions. The evaluation methodology builds on methodologies from previous projects like ICT4CART and 5G-IANA, incorporating lessons learned to tailor it specifically to PoDIUM’s goals.
The methodology is structured around a phased approach inspired by the FESTA V-model, which includes three main phases:
- Preparing: This involves defining the pilot’s functions, use cases, research questions, and KPIs (Key Performance Indicators), and setting up the study design, measures, and participant recruitment.
- Using: This phase focuses on the actual execution of the pilot, where data is collected based on the predefined test cases.
- Analysing: The collected data is then stored, processed, and analysed to answer research questions. This phase also includes impact assessments and deployment scenario analyses.
Central to this methodology are test cases organised into six test suites, each linked to specific PoDIUM platform functions or use cases. The results are evaluated against target KPIs, and alternative approaches, such as simulations or network stress tests, are used when direct measurement isn’t feasible.
Given the complexity of CCAM solutions and the diverse conditions under which they must operate, a structured evaluation approach allows us to rigorously test and validate the technology’s performance, identify potential issues, and quantify the benefits and impacts. This methodological rigour is essential for gaining the trust of stakeholders, including policymakers, industry partners, and end-users, and for providing actionable insights that can guide the successful deployment of these solutions in real-world settings.
Integration with PoDIUM’s Work Packages
The evaluation methodology is intricately linked with multiple tasks and Work Packages (WPs) within the project. The foundation for this methodology was established in WP2, which defined the use cases and KPIs essential for both technical and user acceptance evaluations. WP4 focuses on data management, including defining common data formats and logging standards, ensuring that all data collected during trials are consistent and comparable. The core of the evaluation process is handled in WP5, where the PoDIUM platform, Use Cases (UCs) and user acceptance evaluations are planned, executed, and analysed.
Next steps
The next steps involve the practical application of the developed evaluation methodology. This includes the execution of planned test cases across various test suites, focusing on both technical performance and user acceptance. Pre-tests will be conducted to validate the experimental setups, followed by comprehensive data collection. The collected data will then be analysed to refine the evaluation model, with the results documented and integrated into recommendations for deploying CCAM solutions.
You can find more details on the PoDIUM evaluation methodology in Deliverable D5.1 which will be available on our website soon.
Vicomtech and PoDIUM
Vicomtech is involved in various important areas of PoDIUM. In addition to leading WP5 on ‘Use Cases Evaluation and Demonstration’, Vicomtech also leads Task 2.5 ‘Availability and Cooperation Enablers Definition and Evaluation Data Specifications’. In this task, Vicomtech defines critical Key Performance Indicators (KPIs) for assessing the PoDIUM platform’s architecture efficiency and the success of system performance during trial demonstrations. This task also involves developing user acceptance and social inclusion KPIs, standardizing data formats, and ensuring compliance with data protection regulations. The outputs from this task are essential for guiding the work in WP3, WP4, and WP5. Beyond these leadership roles, Vicomtech contributes to technical and strategic aspects across various work packages, driving key elements of the project to ensure its overall success.
PoDIUM aligns with Vicomtech’s mission to advance cutting-edge technologies in mobility and telecommunications. Participating in PoDIUM allows Vicomtech to apply its expertise in real-world scenarios and contribute to developing next-generation CCAM solutions. Additionally, the project represents an opportunity to collaborate with other industry leaders and research institutions, fostering innovation and driving progress in critical areas such as 5G, AI, and automated mobility. The project also helps Vicomtech showcase its capabilities in technical evaluation and data management, which are essential to modern technological advancements.