Published Paper Details:

Deep Neural Networks (DNNs) have shown strong performance on complex learning tasks, especially when large labeled datasets are available. However, they struggle with mapping input sequences directly to output sequences. In this paper, the authors propose a general, end-to-end method for sequence learning that relies on minimal assumptions about the structure of sequences. Their approach uses a multi-layer Long Short-Term Memory (LSTM) network to encode an input sequence into a fixed-size vector, followed by another deep LSTM network that decodes this vector into an output sequence. A key result of their study is that this LSTM-based model achieved a BLEU score of 34.8 on the full test set for English-to-French translation using the WMT'14 dataset, despite being penalized for generating out-of-vocabulary words. Notably, the model performed well even on long sentences. For comparison, a phrase-based Statistical Machine Translation (SMT) system scored 33.3 on the same dataset. Further improvement was observed when the LSTM was used to rerank the top 1000 translation hypotheses from the SMT system, raising the BLEU score to 36.5--close to the best result reported at the time. The LSTM also learned meaningful representations of phrases and sentences that were sensitive to word order and robust to changes like converting between active and passive voice. Interestingly, performance improved significantly when the order of words in the source sentences was reversed (while keeping target sentences in normal order). This created more short-term dependencies between source and target sequences, which helped make the training process more effective.