14

參加人數

議題提供單位

Pervasive Artificial Intelligence Research (PAIR) Labs, National Chiao Tung University (NCTU), Taiwan

The Pervasive AI Research (PAIR) Labs, a group of national research labs funded by the Ministry of Science and Technology, Taiwan, is commissioned to achieve academic excellence, nurture local AI talents, build international linkage, and develop pragmatic approaches in the areas of applied AI technologies toward services, products, workflows, and supply chains innovation and optimization. PAIR is constituted of 18 distinguished research institutes in Taiwan to conduct research in various of applied AI areas.
Website: https://pairlabs.ai/

2021/06/03 Award Announcement (updated)

Dear Participants, The ACM ICMR 2021 Grand Challenge: PAIR competition awardees are: Champion: "as798792" (國立台南大學) 1st Runner-up: "Deep Learner" (北京交通大學) 3rd-place: "UCBH" (國立成功大學電通所) Best accuracy award: "as798792" (國立台南大學) Best bicycle detection award: "as798792" (國立台南大學) Best scooter detection award: "abcda" (國立臺灣科技大學) Congratulations!

2021/05/11 Award Announcement

Dear Participants, The ACM ICMR 2021 Grand Challenge: PAIR competition awardees are: Champion: "as798792" (國立台南大學) 1st Runner-up: "Deep Learner" (北京交通大學) 3rd-place: "UCBH" (國立成功大學電通所) Special Awards Best accuracy award: "as798792" (國立台南大學) Best bicycle detection award: "as798792" (國立台南大學) Best scooter detection award: "abcda" (國立臺灣師範大學資工系) Congratulations!

2021/04/12 The final leaderboard has been published

Dear ICMR2021 Grand Challenge PAIR Competition Participants, Congratulations on your good performance in the ICMR2021 Grand Challenge PAIR Competition! This is an invitation to submit a paper elaborating your innovative method. Besides the source-code, a successful submission of paper is required for winning the final Challenge Prizes. We encourage all the participants to submit their paper for winning some additional prizes. At the same time, ICMR proceedings will accept innovative papers more than the quota of prizes. Please submit the manuscript by April 18, 2021. Please submit your contest papers to the Special Session -- Embedded Deep Learning Object Detection Model Compression Competition for Traffic in Asian Countries through the ICMR2021 paper submission website below: https://easychair.org/account/signin?l=6RpWzFEVnC5Vv9Vr5QD6It# If you have any further questions, please feel free to contact Yu-Shu, Ni through the email: a321andy6314@gmail.com Thank you very much! Congratulations to you again! Prof. Jiun-In Guo, Co-organizer of the ICMR2021 GC PAIR Competition

2021/03/22 Deadline Extension

Dear Finalists, The competition host has decided to extend the Final Competition End Date to 3/26/2021 4:00 PM UTC+08:00 to encourage the finalists to build better and upload models for the MTK AI platform. Please be aware that this extension is FINAL, no further extension will be made. ------ 親愛的參賽者, 為了讓所有進入決賽的隊伍能夠有機會完成更完善的模型設計並在 AI運算平台得到更快的速度,主辦單位決定將決賽模型上傳期限延後至110年3月26日下午4:00,此為最後期限。請所有參與決賽隊伍繼續努力,完成最佳設計。

2021/03/10 Notes for Final Competition

Dear Finalists, Congratulations to you for advancing into the Final Competition! Please pay attention to the following information. 1. Your submission file will only be evaluated for one (1) time per team per day. 2. The upload page will show “Calculating...” status after you made a submission. The evaluation and scoring process will start at a later time. Score for the submission will be automatically shown on the upload page once the competition host completed running your submission on the MediaTek Dimensity 1000. 3. The daily submission is due for 16:00 (UTC+08:00). You can cancel your submission and upload a new file no later than 16:00 (UTC+08:00). 4. !!!ATTENTION!!! If you cancel your submission later than 16:00 (UTC+08:00), your submission for the day won’t get scored. Final Competition descriptions and rules: http://buckets.aidea-web.tw/ICMR%202021_grand_challenge_final_competition.pdf ------ 親愛的參賽者, 恭喜各位以優異成績晉級 Final Competition,以下是本階段運作的機制,請大家多留意,謝謝! 1. 每天每隊只會計算一次分數。 2. 檔案上傳後,畫面會顯示計算中,上傳的檔案不會即時算分,需等主辦單位收件後,於 MediaTek Dimensity 1000 執行算分,待執行完畢並計分完成,才會更新該筆成績。 3. 每天下午4點(UTC+08:00)收件,如參賽者於當天4點(UTC+08:00)之前想要更新檔案,可以按取消再重新上傳新的檔案。 4. !!!請注意!!!若是在4點(UTC+08:00)之後按了取消,則當天的上傳不會計分。 Final Competition 說明下載: http://buckets.aidea-web.tw/ICMR%202021_grand_challenge_final_competition.pdf

簡介


Object detection in the computer vision area has been extensively studied and making tremendous progress in recent years using deep learning methods. However, due to the heavy computation required in most deep learning-based algorithms, it is hard to run these models on embedded systems, which have limited computing capabilities. In addition, the existing open datasets for object detection applied in ADAS applications usually include pedestrian, vehicles, cyclists, and motorcycle riders in western countries, which is not quite similar to the crowded Asian countries like Taiwan with lots of motorcycle riders speeding on city roads, such that the object detection models training by using the existing open datasets cannot be applied in detecting moving objects in Asian countries like Taiwan.

In this competition, we encourage the participants to design object detection models that can be applied in Taiwan’s traffic with lots of fast speeding motorcycles running on city roads along with vehicles and pedestrians. The developed models not only fit for embedded systems but also achieve high accuracy at the same time.

This competition is divided into two stages: qualification and final competition.

  • Qualification competition: all participants submit their answers online. A score is calculated. The top 15 teams would be qualified to enter the final round of the competition.

  • Final competition: the final score will be evaluated on MediaTek Dimensity 1000 Series platform for the final score.

The goal is to design a lightweight deep learning model suitable for constrained embedded system design to deal with traffic in Asian countries like Taiwan. We focus on detection accuracy, model size, computational complexity, performance optimization and the deployment on MediaTek’s Dimensity 1000 platform.

With MediaTek’s Dimensity 1000 platform and its heterogeneous computing capabilities such as CPUs, GPUs and APUs (AI processing units) embedded into the system-on-chip products, developers are provided the high performance and power efficiency for building the AI features and applications. Developers can target these specific processing units within the system-on-chip or, they can also let MediaTek NeuroPoint SDK intelligently handle the processing allocation for them.

Given the test image dataset, participants are asked to detect objects belonging to the following four classes {pedestrian, vehicle, scooter, bicycle} in each image, including class, bounding box, and confidence.

獎項

According to the points of each team in the final evaluation, we select the highest three teams for regular awards.

  1. Champion: $USD 1,500

  2. 1st Runner-up: $USD 1,000

  3. 2nd Runner-up: $USD 750

Special Awards

  1. Best accuracy award – award for the highest mAP in the final competition: $USD 200;

  2. Best bicycle detection award – award for the highest AP of bicycle recognition in the final competition: $USD 200;

  3. Best scooter detection award – award for the highest AP of scooter recognition in the final competition: $USD 200;

All the award winners must agree to submit contest paper and attend the ACM ICMR2021 Grand Challenge PAIR Competition Special Session to present their work.

活動時間

DateActivity
1/4/2021Qualification Competition Start Date
1/4/2021
Date to Release Public Testing Data
3/1/2021Date to Release Private Testing Data for Qualification
3/8/2021 12:00 PM UTCQualification Competition End Date
3/9/2021 12:00 AM UTCFinalist Announcement
3/9/2021Final Competition Start Date
3/15/2021Date to Release Private Testing Data for Final
3/26/2021 08:00 PM UTC+08:00Final Competition End Date
4/12/2021 12:00 PM UTCAward Announcement

評估標準

  • Qualification Competition

    The grading rule is based on MSCOCO object detection rule.

    • The mean Average Precision (mAP) is used to evaluate the result.

    • Intersection over union (IoU) threshold is set at 0.5.

    The resulting average precision (AP) of each class will be calculated and the mean AP (mAP) over all classes is evaluated as the key metric.

    Besides, during the qualification competition period, each team has to submit a team composition document, including team name, leader, team members, affiliation, and contact information, etc.


  • Final Competition

    • Mandatory Criteria
      • Accuracy of final submission cannot be 5% lower (include) than their submitted model of qualification.

      • The summation of Preprocessing & Postprocessing time of final submission cannot be 50% slower (include) than the inference time of the main model. (evaluated on the host machine)

    • [Host] Accuracy(mAP) – 40%

      The team with the highest accuracy will get the full score (40%) and the team with the lowest one will get zero. The rest teams will get scores directly proportional to the mAP difference. 

    • [Host] Model size (number of parameters * bit width used in storing the parameters) – 15%

      The team with the smallest model will get the full score (15%) and the team with the largest one will get zero. The rest teams will get scores directly proportional to the model size difference.

    • [Host] Model Computational Complexity(GOPs/frame) – 15%

      The team with the smallest GOP number per frame will get the full score (15%) and the team with the largest one will get zero. The rest teams will get scores directly proportional to the GOP value difference.

    • [Device] Speed on MediaTek Dimensity 1000 Series Platform – 30%

      The team with a single model (w/o Preprocessing & Postprocessing) to complete the detection task in the shortest time will get the full score (30%) and the team that takes the longest time will get zero score. The rest teams will get scores directly proportional to the execution time difference.

    • The evaluation procedure will be toward the overall process from reading the private testing dataset in final to completing submission.csv file, including parsing image list, loading images, and any other overhead to conduct the detection through the testing dataset.

Reference
[1] Average Precision (AP):
https://en.wikipedia.org/wiki/Evaluation_measures_%28information_retrieval%29#Average_precision

[2] intersection over union (IoU):
https://en.wikipedia.org/wiki/Jaccard_index

[3] COCO API:
https://github.com/cocodataset/cocoapi


Coordinator Contacts

Ted Kuo, tkuo@nctu.edu.tw

Jenq-Neng Hwang, hwang@uw.edu

Jiun-In Guo, jiguo@nctu.edu.tw

Marvin Chen, marvin.chen@mediatek.com

Hsien-Kai Kuo, hsienkai.kuo@mediatek.com

Chia-Chi Tsai, chia-chi.tsai@mediatek.com

規則

  • Team mergers are not allowed in this competition.
  • Each team can consist of a maximum of 6 team members.
  • The task is open to the public. Individuals, institutions of higher education, research institutes, enterprises, or other organizations can all sign up for the task.
  • A leaderboard will be set up and make available publicly.
  • Multiple submissions are allowed before the deadline and the last one will be used to enter the final qualification consideration.
  • The upload date/time will be used as the tiebreaker.
  • Privately sharing code or data outside of teams is not permitted. Its okay to share code if made available to all participants on the forums.
  • Personnel of IVSLAB team are not allowed to participate in the task.
  • A common honor code should be observed. Any violation will be disqualified.