Character Level Text Generation using an Encoder Decoder Language Model with Bahdanau Attention
Colab Notebook: https://colab.research.google.com/dri...
Text Generation Playlist: • Text Generation in Deep Learning with...
TensorFlow Input Pipeline Playlist: • TensorFlow Data Pipeline: How to Desi...
Encoder-Decoder Playlist: • Seq2Seq Learning Tutorials
All About LSTM playlist: • All About LSTM
Character Level Text Generation with an Encoder Decoder Model
This tutorial is the sixth part of the "Text Generation in Deep Learning with Tensorflow & Keras" series.
In this series, we have been covering all the topics related to Text Generation with sample implementations in Python, Tensorflow & Keras.
In this tutorial, we will focus on how to build a Language Model using Encoder Decoder approach with Bahdanau Attention mechanism for Character Level Text Generation.
First, we will download a sample corpus (text file).
After opening the file, we will apply the TensorFlow input pipeline that we have developed in Part B to prepare the training dataset by preprocessing and splitting the text into input character sequence (X) and output character (y).
Then, we will design an Encoder Decoder Model with Bahdanau Attention as the Language Model. We will train this model using the train set.
Later on, we will apply several sampling methods that we have implemented in Part D to generate text and observe the effect of these sampling methods on the generated text.
Thus, in the end, we will have a trained Encoder Decoder based Language Model for character-level text generation with three sampling methods.
If you would like to learn more about Deep Learning with practical coding examples, please subscribe to Murat Karakaya Akademi YouTube Channel or follow my blog on Medium. Do not forget to turn on notifications so that you will be notified when new parts are uploaded.
You can access this Colab Notebook using the link given in the video description below.
If you are ready, let's get started!
Text Generation in Deep Learning with Tensorflow & Keras Series:
Part A: Fundamentals
Part B: Tensorflow Data Pipeline for Character Level Text Generation
Part C: Tensorflow Data Pipeline for Word Level Text Generation
Part D: Sampling in Text Generation
Part E: Recurrent Neural Network (LSTM) Model for Character Level Text Generation
Part F: Encoder-Decoder Model for Character Level Text Generation
Part G: Transformer Model for Character Level Text Generation
Part H: Recurrent Neural Network (LSTM) Model for Word Level Text Generation
Part I: Encoder-Decoder Model for Word Level Text Generation
Part J: Transformer Model for Word Level Text Generation
You can watch all these parts on Murat Karakaya Akademi channel on YouTube in ENGLISH or TURKISH
I assume that you have already watched all previous parts.
Please ensure that you have reviewed the previous parts in order to utilize this part better.
References
What is a Data Pipeline?
tf.data: Build TensorFlow input pipelines
Text classification from scratch
Working with Keras preprocessing layers
Character-level text generation with LSTM
Toward Controlled Generation of Text
Attention Is All You Need
Talk to Transformer
What is the difference between word-based and char-based text generation RNNs?
The survey: Text generation models in deep learning
Generative Adversarial Networks for Text Generation
FGGAN: Feature-Guiding Generative Adversarial Networks for Text Generation
How to sample from language models
How to generate text: using different decoding methods for language generation with Transformers
Hierarchical Neural Story Generation
How to sample from language models
Text generation with LSTM
A guide to language model sampling in AllenNLP
Generating text from language model
How to Implement a Beam Search Decoder for Natural Language Processing
Controllable Neural Text Generation
What is a Character Level Text Generation?
A Language Model can be trained to generate text character-by-character. In this case, each of the input and output tokens is a character. Moreover, Language Model outputs a conditional probability distribution over the character set.
PREPARE SAMPLING METHODS
In Text Generation, sampling means randomly picking the next token according to the generated conditional probability distribution.
That is, after generating the conditional probability distribution over the set of tokens (vocabulary) for the given input sequence, we need to carefully decide how to select the next token (sample) from this distribution.
There are several methods for sampling in text generation (see here and here):
Greedy Search (Maximization)
Temperature Sampling
Top-K Sampling
Top-P Sampling (Nucleus sampling)
Beam Search
In this tutorial, we will code Greedy Search, Temperature Sampling, and Top-K Sampling.
Watch video Character Level Text Generation using an Encoder Decoder Language Model with Bahdanau Attention online without registration, duration hours minute second in high quality. This video was added by user Murat Karakaya Akademi 01 January 1970, don't forget to share it with your friends and acquaintances, it has been viewed on our site 1,864 once and liked it 31 people.