Rubric-Constrained Figure Skating Scoring

Figure skating automatic scoring is the task of estimating the competition score of a performance video. The technical element score (TES) aggregates the technical quality (grade of execution) and difficulty (base value) scores for each element. Most prior work, adapted from short-term action quality assessment, entangle difficulty and quality and compute TES for the entire video, reducing interpretability for athletes. This is mainly due to no element segmentation and difficulty annotations in existing datasets. Motivated by increasing interpretability, we propose a novel method that implicitly segments a video to produce element-level representation and uses adherence with a natural language rubric to score each element, without needing additional annotations. We compute element-level representations using learnable element queries in a transformer and propose implicit segmentation regularization to encourage element queries to attend to elements rather than background transitions between elements (most of video). Additionally, we use the element list (sequence of elements) to isolate difficulty, just like judges who receive the routine list in advance, so we can focus on the more critical problem of how well elements are done. These components significantly improve interpretability, scoring precision, and ranking capability.