Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upTutorial: NumPy deep learning on MNIST from scratch #33
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
This surely was many hours of effort -- thank you. A few clarifications in the opening lines might help readers:
Isn't it the entire NN that's written without external libraries, not just parsing and loading?
Why would I, the reader, want to do that?
This sentence parses ambiguously and the reference of "this" is unclear. Maybe not worth fixing -- leave it out.
Trask might appreciate a link to the original work, especially if it's his new book, rather than simply to his home page.
|
|
Hi @8bitmp3 , thank you for this tutorial, I really think this will be a nice addition and it is quite an effort to implement this without any pre-packaged library. From the text it is clear that you are in your element, too! I have a few comments, mainly in the interest of making the document a bit simpler. I think some of the functions can be simplified or rewritten, and this way we can use less code-comments and more text. I believe this can make the whole document more fluid and easier to follow. Feel free to adapt or try different things if you think it doesn't fit your own narrative. I'm focusing first on the start and some of the functions there, but I'll continue trying to bring ideas for the rest. In particular, I think "Neural network building blocks with NumPy" and "Model architecture summary" run a bit long - I think your explanations are spot on, but I'm worried that we'll lose track of the code at this point. From an educational point of view, I usually create variables or import libraries close to where they will actually be used, instead of in the beginning of the file. So you'll see that I did that for the first sections. Then, since we are using a notebook, we might as well make use of literate programming ideas; namely, instead of using code comments, using actual text as much as possible. This made me rethink some of your functions and realize that, for the purpose of this document, they are really not necessary since those operations will only be performed once. So I tried rewriting the initial download/save/retrieve data without using functions at all. Let me know what you think and if you believe strongly those should be functions we can work on that. Also, some of the text might need re-working now, I just took some of the code comments and adapted but you may have better ideas for that :) Keep in mind that my suggestions are not meant to be the best solution - just another way of doing things. Here's my take on the document (not that I only touched the first section, and removed the need for Overall, I'm pretty happy that we'll see this kind of content on the repo - so thanks again and let me know if there's anything I can help you with! |
|
Thanks @bjnath for the awesome feedback.
Changed the words around and refactored for clarity:
Agreed. Although it's not uncommon to use a deep learning library to add a popular dataset in a couple of lines of code, I've removed this anyway to avoid repetition.
Definitely. I also added a link to the repo and a reference to Trask's book - it's a must read. I'm basing my third project on Grus' work, with his permission (a small automatic differentiation library with NumPy), so will mention him there instead of this project. |
Definitely, thanks so much for the help @melissawm
Good point! I want to merge some of the stuff with the "compose section" (with the code). Simpler is better. Thanks so much!
Yup, got it.
I think this is awesome. I mostly try to use the second person, but other than that Thank you |
|
Hi @8bitmp3 ! This is looking really good - I have a few comments but overall I think you did an awesome job. This is a tough subject and I do have some background in machine learning though - but I suspect this will also be the case for readers of this tutorial, so I think your links and references are right on target. I don't know if there's more simplification possible in the code - the main training loop is just as it should be. We could take some of the code-comments out but I'd like to see some other opinions on that. The only thing you might consider is building a function for the test/training loss+accurate prediction calculation. But I'm not sure that would be helpful for readers... For now my main comments will be on some typos and a few suggestions but nothing major. Thanks again for this! |
| @@ -0,0 +1,1017 @@ | |||
| { | |||
melissawm
Oct 26, 2020
Member
I would add to that a suggestion about conda, since it's become the standard for most scientific computing applications in Python.
Reply via ReviewNB
I would add to that a suggestion about conda, since it's become the standard for most scientific computing applications in Python.
Reply via ReviewNB
8bitmp3
Oct 28, 2020
Author
Done.
This tutorial can be run locally in an isolated environment, such as
[Virtualenv](https://virtualenv.pypa.io/en/stable/)
+ and [conda](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)
...
Done.
This tutorial can be run locally in an isolated environment, such as
[Virtualenv](https://virtualenv.pypa.io/en/stable/)
+ and [conda](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)
...
melissawm
Nov 4, 2020
Member
nit: I'd say virtualenv or conda, but that's just the mathematician in me talking :)
nit: I'd say virtualenv or conda, but that's just the mathematician in me talking :)
| @@ -0,0 +1,1017 @@ | |||
| { | |||
melissawm
Oct 26, 2020
Member
- I think you meant matrix multiplication function
np.dot (instead of *module*) - In the activation function description I think you meant "inputs and outputs" instead of "inputs and inputs" :)
- "In this example, you will use a method called dropout (dilution that " (I think there is a formatting error here with the parenthesis?)
Reply via ReviewNB
- I think you meant matrix multiplication function
np.dot(instead of *module*) - In the activation function description I think you meant "inputs and outputs" instead of "inputs and inputs" :)
- "In this example, you will use a method called dropout (dilution that " (I think there is a formatting error here with the parenthesis?)
Reply via ReviewNB
8bitmp3
Oct 28, 2020
Author
- Good catch :) Also changed "
np.random.random() function" and "np.random.randint() function".
- Nice one. Changed to "inputs and outputs"
- There's a weird rendering issue in some notebook viewers - maybe because the URL ends with a bracket
[...](https://en.wikipedia.org/wiki/Dilution_(neural_networks)). Changed it to:
- ... dropout ([dilution](https://en.wikipedia.org/wiki/Dilution_(neural_networks))) ...
+ dropout — [dilution](https://en.wikipedia.org/wiki/Dilution_(neural_networks)) — ...
- Good catch :) Also changed "
np.random.random()function" and "np.random.randint()function". - Nice one. Changed to "inputs and outputs"
- There's a weird rendering issue in some notebook viewers - maybe because the URL ends with a bracket
[...](https://en.wikipedia.org/wiki/Dilution_(neural_networks)). Changed it to:
- ... dropout ([dilution](https://en.wikipedia.org/wiki/Dilution_(neural_networks))) ...
+ dropout — [dilution](https://en.wikipedia.org/wiki/Dilution_(neural_networks)) — ...
melissawm
Nov 4, 2020
Member
The regularization link seems to be missing a parenthesis at the end. Thanks for the other fixes :)
The regularization link seems to be missing a parenthesis at the end. Thanks for the other fixes :)
| @@ -0,0 +1,1017 @@ | |||
| { | |||
melissawm
Oct 26, 2020
Member
- Why do you test every 10 epochs?
- Is it necessary to use
[i, i+1] for each item, or could [i] do the same job?
Reply via ReviewNB
- Why do you test every 10 epochs?
- Is it necessary to use
[i, i+1]for each item, or could[i]do the same job?
Reply via ReviewNB
8bitmp3
Oct 29, 2020
Author
- I think in the original gist, you test (evaluate) the model every 10 epochs because it should have more "knowledge" to be tested after the 10th training iteration. I've tried testing at every epoch and noticed the results were OK in this simple example. It's reflected in the new code.
[i:i+1] is necessary here (doesn't seem to work without it)
- I think in the original gist, you test (evaluate) the model every 10 epochs because it should have more "knowledge" to be tested after the 10th training iteration. I've tried testing at every epoch and noticed the results were OK in this simple example. It's reflected in the new code.
[i:i+1]is necessary here (doesn't seem to work without it)
melissawm
Nov 4, 2020
•
Member
- Why do you print
"Epoch" + str(j+10)? I think there is no need to add 10 to this result - am I missing something?
- Ok I got the reason for the indexing but this code could be simplified as follows:
# To store training and test set losses and accurate predictions
# for visualization.
store_training_loss = []
store_training_accurate_pred = []
store_test_loss = []
store_test_accurate_pred = []
# This is a training loop.
# Run the learning experiment for a defined number of epochs (iterations).
for j in range(epochs):
# Set the initial loss/error and the number of accurate predictions to zero.
training_loss = 0.0
training_accurate_predictions = 0
# For all images in the training set, perform a forward pass
# and backpropagation and adjust the weights accordingly.
for i in range(len(training_images)):
# Forward propagation/forward pass:
# 1. The input layer:
# Initialize the training image data as inputs.
layer_0 = training_images[i]
# 2. The hidden layer:
# Take in the training image data into the middle layer by
# matrix-multiplying it by randomly initialized weights.
layer_1 = np.dot(layer_0, weights_1)
# 3. Pass the hidden layer's output through the ReLU activation function.
layer_1 = relu(layer_1)
# 4. Define the dropout function for regularization.
dropout_mask = np.random.randint(0, high=2, size=layer_1.shape)
# 5. Apply dropout to the hidden layer's output.
layer_1 *= dropout_mask * 2
# 6. The output layer:
# Ingest the output of the middle layer into the the final layer
# by matrix-multiplying it by randomly initialized weights.
# Produce a 10-dimension vector with 10 scores.
layer_2 = np.dot(layer_1, weights_2)
# Backpropagation/backward pass:
# 1. Measure the training error (loss function) between the actual
# image labels (the truth) and the prediction by the model.
training_loss += np.sum((training_labels[i] - layer_2) ** 2)
# 2. Increment the accurate prediction count.
training_accurate_predictions += int(np.argmax(layer_2) == np.argmax(training_labels[i]))
# 3. Differentiate the loss function/error.
layer_2_delta = (training_labels[i] - layer_2)
# 4. Propagate the gradients of the loss function back through the hidden layer.
layer_1_delta = np.dot(weights_2, layer_2_delta) * relu2deriv(layer_1)
# 5. Apply the dropout to the gradients.
layer_1_delta *= dropout_mask
# 6. Update the weights for the middle and input layers
# by multiplying them by the learning rate and the gradients.
weights_1 += learning_rate * np.outer(layer_0, layer_1_delta)
weights_2 += learning_rate * np.outer(layer_1, layer_2_delta)
# Store training set losses and accurate predictions.
store_training_loss.append(training_loss)
store_training_accurate_pred.append(training_accurate_predictions)
# Evaluate on the test set:
# 1. Set the initial error and the number of accurate predictions to zero.
test_loss = 0.0
test_accurate_predictions = 0
# 2. Start testing the model by evaluating on the test image dataset.
for i in range(len(test_images)):
# 1. Pass the test images through the input layer.
layer_0 = test_images[i]
# 2. Compute the weighted sum of the test image inputs in and
# pass the hidden layer's output through ReLU.
layer_1 = relu(np.dot(layer_0, weights_1))
# 3. Compute the weighted sum of the hidden layer's inputs.
# Produce a 10-dimensional vector with 10 scores.
layer_2 = np.dot(layer_1, weights_2)
# 4. Measure the error between the actual label (truth) and prediction values.
test_loss += np.sum((test_labels[i] - layer_2) ** 2)
# 5. Increment the accurate prediction count.
test_accurate_predictions += int(np.argmax(layer_2) == np.argmax(test_labels[i]))
# Store test set losses and accurate predictions.
store_test_loss.append(test_loss)
store_test_accurate_pred.append(test_accurate_predictions)
# 3. Display the error and accuracy metrics in the output.
print("\n" + \
"Epoch: " + str(j) + \
" Training set error:" + str(training_loss/ float(len(training_images)))[0:5] +\
" Training set accuracy:" + str(training_accurate_predictions/ float(len(training_images))) +\
" Test set error:" + str(test_loss/ float(len(test_images)))[0:5] +\
" Test set accuracy:" + str(test_accurate_predictions/ float(len(test_images))))
Let me know if this makes sense to you. There was nothing wrong with the original code - I just think it is easier to avoid transposing if you can, and using 1d arrays instead of 2d.
- Why do you print
"Epoch" + str(j+10)? I think there is no need to add 10 to this result - am I missing something? - Ok I got the reason for the indexing but this code could be simplified as follows:
# To store training and test set losses and accurate predictions
# for visualization.
store_training_loss = []
store_training_accurate_pred = []
store_test_loss = []
store_test_accurate_pred = []
# This is a training loop.
# Run the learning experiment for a defined number of epochs (iterations).
for j in range(epochs):
# Set the initial loss/error and the number of accurate predictions to zero.
training_loss = 0.0
training_accurate_predictions = 0
# For all images in the training set, perform a forward pass
# and backpropagation and adjust the weights accordingly.
for i in range(len(training_images)):
# Forward propagation/forward pass:
# 1. The input layer:
# Initialize the training image data as inputs.
layer_0 = training_images[i]
# 2. The hidden layer:
# Take in the training image data into the middle layer by
# matrix-multiplying it by randomly initialized weights.
layer_1 = np.dot(layer_0, weights_1)
# 3. Pass the hidden layer's output through the ReLU activation function.
layer_1 = relu(layer_1)
# 4. Define the dropout function for regularization.
dropout_mask = np.random.randint(0, high=2, size=layer_1.shape)
# 5. Apply dropout to the hidden layer's output.
layer_1 *= dropout_mask * 2
# 6. The output layer:
# Ingest the output of the middle layer into the the final layer
# by matrix-multiplying it by randomly initialized weights.
# Produce a 10-dimension vector with 10 scores.
layer_2 = np.dot(layer_1, weights_2)
# Backpropagation/backward pass:
# 1. Measure the training error (loss function) between the actual
# image labels (the truth) and the prediction by the model.
training_loss += np.sum((training_labels[i] - layer_2) ** 2)
# 2. Increment the accurate prediction count.
training_accurate_predictions += int(np.argmax(layer_2) == np.argmax(training_labels[i]))
# 3. Differentiate the loss function/error.
layer_2_delta = (training_labels[i] - layer_2)
# 4. Propagate the gradients of the loss function back through the hidden layer.
layer_1_delta = np.dot(weights_2, layer_2_delta) * relu2deriv(layer_1)
# 5. Apply the dropout to the gradients.
layer_1_delta *= dropout_mask
# 6. Update the weights for the middle and input layers
# by multiplying them by the learning rate and the gradients.
weights_1 += learning_rate * np.outer(layer_0, layer_1_delta)
weights_2 += learning_rate * np.outer(layer_1, layer_2_delta)
# Store training set losses and accurate predictions.
store_training_loss.append(training_loss)
store_training_accurate_pred.append(training_accurate_predictions)
# Evaluate on the test set:
# 1. Set the initial error and the number of accurate predictions to zero.
test_loss = 0.0
test_accurate_predictions = 0
# 2. Start testing the model by evaluating on the test image dataset.
for i in range(len(test_images)):
# 1. Pass the test images through the input layer.
layer_0 = test_images[i]
# 2. Compute the weighted sum of the test image inputs in and
# pass the hidden layer's output through ReLU.
layer_1 = relu(np.dot(layer_0, weights_1))
# 3. Compute the weighted sum of the hidden layer's inputs.
# Produce a 10-dimensional vector with 10 scores.
layer_2 = np.dot(layer_1, weights_2)
# 4. Measure the error between the actual label (truth) and prediction values.
test_loss += np.sum((test_labels[i] - layer_2) ** 2)
# 5. Increment the accurate prediction count.
test_accurate_predictions += int(np.argmax(layer_2) == np.argmax(test_labels[i]))
# Store test set losses and accurate predictions.
store_test_loss.append(test_loss)
store_test_accurate_pred.append(test_accurate_predictions)
# 3. Display the error and accuracy metrics in the output.
print("\n" + \
"Epoch: " + str(j) + \
" Training set error:" + str(training_loss/ float(len(training_images)))[0:5] +\
" Training set accuracy:" + str(training_accurate_predictions/ float(len(training_images))) +\
" Test set error:" + str(test_loss/ float(len(test_images)))[0:5] +\
" Test set accuracy:" + str(test_accurate_predictions/ float(len(test_images))))Let me know if this makes sense to you. There was nothing wrong with the original code - I just think it is easier to avoid transposing if you can, and using 1d arrays instead of 2d.
| @@ -0,0 +1,1017 @@ | |||
| { | |||
melissawm
Oct 26, 2020
Member
It would be really nice to close with a plot showing that your NN can identify an image that was not in the training set (I know the prints also show this, but just for a visual element).
Reply via ReviewNB
It would be really nice to close with a plot showing that your NN can identify an image that was not in the training set (I know the prints also show this, but just for a visual element).
Reply via ReviewNB
8bitmp3
Oct 26, 2020
•
Author
Great idea @melissawm
A matplotlib visualization of the error rate falling and accuracy improving (hopefully) over time would also be super — and it's a common practice. Working on it!
Great idea @melissawm
A matplotlib visualization of the error rate falling and accuracy improving (hopefully) over time would also be super — and it's a common practice. Working on it!
8bitmp3
Oct 29, 2020
•
Author
I'm not 100% sure how to do "inference" with a trained model to classify images, since I don't think we have the Saved Model or checkpoint saving capability in the simple example. But we can do the following:
TODO:
- 1. We can add graphs with
matplotlib for:
- The error rate (loss) for training and test sets (y) per epochs (x) (it should be gradually going down).
- The accuracy rate for training and test sets (y) per epochs (x) (it should be gradually going up).
- Need to add a few lines to save the loss and accuracy (prediction) data at each epoch by appending them separate lists (for storage).
- 2. Add a simple image to help new users:
- Showing a loop: inputs as vectors -> forward pass through layers -> output (prediction) + loss calculation -> backpropagation with gradient descent -> weight update.
Other improvement suggestions:
- Try Fashion MNIST - a similar dataset but more fun.
I'm not 100% sure how to do "inference" with a trained model to classify images, since I don't think we have the Saved Model or checkpoint saving capability in the simple example. But we can do the following:
TODO:
- 1. We can add graphs with
matplotlibfor:- The error rate (loss) for training and test sets (y) per epochs (x) (it should be gradually going down).
- The accuracy rate for training and test sets (y) per epochs (x) (it should be gradually going up).
- Need to add a few lines to save the loss and accuracy (prediction) data at each epoch by appending them separate lists (for storage).
- 2. Add a simple image to help new users:
- Showing a loop: inputs as vectors -> forward pass through layers -> output (prediction) + loss calculation -> backpropagation with gradient descent -> weight update.
Other improvement suggestions:
- Try Fashion MNIST - a similar dataset but more fun.
| @@ -0,0 +1,1067 @@ | |||
| { | |||
melissawm
Nov 4, 2020
Member
| @@ -0,0 +1,1067 @@ | |||
| { | |||
| @@ -0,0 +1,1067 @@ | |||
| { | |||
melissawm
Nov 4, 2020
Member
The plots are great, I think that closes the tutorial nicely!
Tip: you can add plt.tight_layout() to improve on the spacing between the subplots - at least locally I had the title and x-axis label of the subplots superimposed without using tight_layout.
Reply via ReviewNB
The plots are great, I think that closes the tutorial nicely!
Tip: you can add plt.tight_layout() to improve on the spacing between the subplots - at least locally I had the title and x-axis label of the subplots superimposed without using tight_layout.
Reply via ReviewNB
If you're explicit about what this section produces, readers can read just the first lines and skip down to the next heading. You could reword this as:
|
We're not really saving to disk here -- that implies we're writing files. How about
|
This needs to be a little clearer about how the data in the files is formatted so we understand what this step is doing. |
Comments would make it a little clearer: |
Not clear what "standard notation" is referring to. How about:
|
If you were really making sure, you'd code a test, not a print statement. And you should say what the user should expect to see. How about
|
'Optional' not needed. This step follows another optional step, looking at the shapes, so it can read:
|
I apologize if I've misread the Python, but have we actually created 4 files? It looks like we have a dictionary of 4 arrays. |
In general, tutorials should not include side comments in the main text. |
|
|
The opening of "Preprocess the data" is wordy and doesn't explain much. I'd replace the entire opening, up to the first heading, with just this:
This also makes it easy to read the first lines and then skip to the next section, where the real NN stuff begins. |
This and most of the other "Note" sections should be integrated into the main text. Saying "note" isn't necessary; the typography of the note sections is incongruous; and Google style calls this kind of usage a placeholder and discourages it. |
I can see the justification for saying "floating points" ("Convert the sheds to bungalows"), but I think in this context it's more idiomatic to say "floating point" without the s ("Convert the tickets to business class."). Others can weigh in. |
Suggested:
|
Suggested:
|
It's unclear which part of the preceding 120-word description "this type" refers to. Supervised learning means learning from labeled examples. If you want to explain what supervised learning is, say that. My suggestion, though, is to drop the sentence. |
I'd leave backpropagation out of the first line; you say later that it trains using forward propagation and backpropagation.
|
It's unnecessary and confusing to introduce this detail here. I suggest leaving it out. |
Suggested:
And drop the reference in Prerequisites. |
Hi @melissawm @mattip @bjnath @rgommers🙌 GSoD
All feedback is welcome!
Thank you everyone🙏
cc @iamtrask