Evaluating Representations by the Complexity of Learning Low-loss Predictors

Published in In submission, 2020

[arxiv], [blog]

We consider the problem of evaluating representations of data for use in solving a downstream task. We propose to measure representation quality by how efficiently we can learn a “sufficiently good” predictor on top of the representation.

Our methods require the user to specifiy what qualifies as a “sufficiently good” performance. This allows our evaluation measures to not depend on the evaluation set size, and prevents premature judgements when comparing different representations with insufficient data.