Transformer and BERT in mlpack

Mon, August 10

Week 9 and 10 (27 July - 09 August)

Hello Everyone! Here is the update for week 9 and 10.

1. The Lookup layer now works for batch size greater than 1. The fix lookup layer pull request got merged.

2. The BLEU Score pull request also got merged.

3. The Multihead Attention layer is also completed and we will get it merged in next 2-3 days. This was a hard nut to crack. We tried different APIs, corrected lots of silly mistakes. There were memory errors showing in this PR which didn't looked related. I think probably we should separate out some module tests from ann_layer_test.cpp as it is significantly large file.

4. I also worked on Positional Encoding this week. This PR is also complete. There seems to be memory error which I think will be fixed within next 2-3 tries (atleast something will work 😁).

5. The long pending pull request to add getter and setter methods for weight of some ann layers also got approved.

6. I also tried the Transformer Encoder layer and it works locally. Once the multihead attention layer gets merged I will add this one also.

For this week I aim to:

1. Complete the Transformer Encoder and Decoder layer both and get all the builds green.

2. Get the Positional Encoding layer merged.

I think that's it for this week. I hope we will get some other nice additions when we will meet next week. Also, I have started to read books again as using computer for all purposes (learning, coding, playing, watching whatever) had ruined the habbit of reading 😁. I don't know why but to me there is a difference between reading something on laptop and the same thing on paper. I started with "Reality is Not What it Seems". Let's see how it goes :)

See you next time. Be Safe! Be Healthy!

Back