Knowledge distillation has recently been successfully applied to neural machine translation. It basically allows for building shrunk networks while the resulting systems retain most of the quality of the original model. Despite that many authors report on the benefits of knowledge distillation, few works discuss the actual reasons why it works, especially in the context of neural MT. In this paper, we conduct several experiments aiming at understanding why and how distillation impacts accuracy on an English-German translation task. We show that translation complexity is actually reduced when building a distilled\/synthesized bi-text when compared to the reference bi-text. We further remove noisy data from synthesized translations and merge filtered synthesized data together with original reference, thus achieving additional accuracy gains.<\/p>","protected":false},"excerpt":{"rendered":"
Knowledge distillation has recently been successfully applied to neural machine translation. It basically allows for building shrunk networks while the resulting systems retain most of the quality of the original model. Despite that many authors report on the benefits of knowledge distillation, few works discuss the actual reasons why it works, especially in the context … \uacc4\uc18d\ub428<\/a><\/p>","protected":false},"featured_media":0,"template":"","class_list":["post-3284","paper","type-paper","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/www-staging.systransoft.com\/kr\/wp-json\/wp\/v2\/paper\/3284","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www-staging.systransoft.com\/kr\/wp-json\/wp\/v2\/paper"}],"about":[{"href":"https:\/\/www-staging.systransoft.com\/kr\/wp-json\/wp\/v2\/types\/paper"}],"wp:attachment":[{"href":"https:\/\/www-staging.systransoft.com\/kr\/wp-json\/wp\/v2\/media?parent=3284"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}