Learning Notes

c++ notes

1. Why should use member initialization list?

For POD class members, it makes no difference.

For class members which are classes, then it avoids an unnecessary call to a default constructor.

Furthermore, if a class doesn’t have a default constructor, or you have a const member variable, you must use an initializer list:

floating point number

浮点数最新标准为IEEE 754-2019

浮点数格式如下：

S(sign)	E (biased exponent)	T (trailing significand field)
1 bit	w bits	t bits, t = p -1

具有如下关系：

\[\begin{aligned} e & = E - bias \\ e_{max} & = bias = 2^{w-1} - 1 \\ e_{min} & = 1 - e_{max} \end{aligned}\]

关于biased E的说明:

1. normal number: [1 , \(2^w - 2\)]， 值为 \((-1)^s \times 2^{E-bias} \times (1+ 2^{1-p} \times T)\)
2. 0, 当T=0表示 \(\pm 0\); 当T!=0 表示 subnormal number, 值为 \((-1)^s \times 2^{e_{min}} \times (0+ 2^{1-p} \times T)\)
3. \(2^w − 1\) (二进制全部为1), 当T=0, 表示 \(\pm \infty\); 当T != 0, 表示 NaN.

ieee 规定的16, 32, 64, 128比特的浮点数格式列表

参数	binary16	binary32	binary64	binary128
指数位数	5	8	11	15
emax/bias	15	127	1023	16383
小数位数	10	23	52	112

cuda related

how to install nvidia driver in linux?

Problem: NVIDIA kernel module ‘nvidia-drm’ already be loaded in kernel

Solution (in root):

1. goto true tty3, pressing Ctrl+Alt+F2

2. disable the graphical target:

   systemctl isolate multi-user.target
   

3. unload the Nvidia drivers:

   modprobe -r nvidia-drm
   

4. install cuda driver

5. start the GUI again:

   systemctl start graphical.target
   

6. confirm your driver version:

   nvidia-smi
   

download nvidia driver

website: https://www.nvidia.com/Download/Find.aspx

Backpropagation的推导

约定

(1)\[z^{l+1}_j =\sum_k w^l_{jk} a^l_k + b^l_j, \quad a^l_j=\sigma(z^l_j) \]

其中，\(z^l_j\) 表示未激活前第 \(l\) 层、第 \(j\) 个神经元的值，\(w^l_{jk}\) 为连接第 \(l\) 层第 \(j\) 个神经元和第 \(l+1\) 层第k个神经元的权重， \(a^l_k\) 表示激活后的第 \(l\) 层第 \(k\) 个神经元的值， \(b^l_j\) 为偏移量bias，\(\sigma\) 为激活函数。

注意和书籍 http://neuralnetworksanddeeplearning.com/chap2.html 中公式(23)的约定由区别，我们把weight和bias和神经元的值放到同一层中。

公式 (1) 写成矩阵形式为：

\[z^{l+1}=w^l a^l + b^l, \quad a^l=\sigma(z^l) \]

公式推导

我们约定C为损失函数（loss function），并记：

\[\delta^l = \frac{\partial C}{\partial z^l} \]

约定 Hadamard product 或者elementwise相乘为（重复指标不求和）：

\[u\odot v = u_i * v_i \]

根据公式 (1) 可以直接得出对偏移量 \(b\) 的偏导数（梯度）：

\[\frac{\partial C}{\partial b^l_j} = \sum_i \frac{\partial C}{\partial z^{l+1}_i} \frac{\partial z^{l+1}_i}{\partial b^l_j} = \frac{\partial C}{z^{l+1}_j} = \delta^{l+1}_j \]

上式写成矩阵形式为：

\[\frac{\partial C}{\partial b^l} = \delta^{l+1} \]

对权重 \(w\) 的求导为：

\[\frac{\partial C}{\partial w^l_{jk}} = \sum_i \frac{\partial C}{\partial z^{l+1}_i} \frac{\partial z^{l+1}_i}{\partial w^l_{jk}} = \frac{\partial C}{\partial z^{l+1}_j} a^l_k = \delta^{l+1}_j a^l_k \]

上式写成矩阵形式为：

\[\frac{\partial C}{\partial w^l} = \delta^{l+1} (a^l)^T \]

\(l\) 层 \(\delta^l\) 和 \(l+1\) 层的 \(\delta^{l+1}\) 的关系为：

\[\frac{\partial C}{\partial z^l_j} = \sum_{i,k} \frac{\partial C}{\partial z^{l+1}_i} \frac{\partial z^{l+1}_i}{\partial a^l_k} \frac{\partial a^l_k}{\partial z^l_j} = \sum_i \delta^{l+1}_i w^l_{ij} \sigma^{'}(z^l_j) \]

上式写成矩阵形式为：

\[\delta^l = (w^l)^T \delta^{l+1}\odot\sigma^{'}(z^l) \]

可以看出：

\[\nabla_a C = (w^l)^T \delta^{l+1} \]

BP算法总结

BP算法可以概括为以下四个关系式：

\[\begin{aligned} \delta^l &= \frac{\partial C}{\partial z^l} = \nabla_z C \\ \frac{\partial C}{\partial w^l} &= \delta^{l+1} (a^l)^T \\ \frac{\partial C}{\partial b^l} &= \delta^{l+1} \\ \delta^l &= (w^l)^T \delta^{l+1}\odot\sigma^{'}(z^l) \end{aligned} \]

可以看出，可以从 \(\delta^{l+1}\) 的推导出对第 \(l\) 层的权重和偏移量的偏导，以及第 \(l\) 层的未激活前的神经元的偏导。

convolution arithmetic

reference:

• https://github.com/vdumoulin/conv_arithmetic
• https://arxiv.org/abs/1603.07285

1. convolution

Set input data size \(i\), convolution kernel size \(k\), stride size \(s\), and zero padding size \(p\). Then the output size \(o\) is:

(1)\[o = \left\lfloor{\frac{i + 2p - k}{s}}\right\rfloor + 1 \,. \]

The floor function \({\lfloor}\,{\rfloor}\) can found at https://en.wikipedia.org/wiki/Floor_and_ceiling_functions.

2. pooling

According to (1), pooling output size is:

(2)\[o = \left\lfloor{\frac{i-k}{s}}\right\rfloor + 1 \,. \]

3. tansposed convolution

explanation:The convolution operation can be rewritten to matrix multiplication.

4. dilated convolution

The dilation “rate” is controlled by an additional hyperparameter \(d\). A kernel of size k dilated by a factor d has an effective size:

\[\hat{k} = k + (k-1)(d-1) \,. \]

Combined with (1) the output size is:

(3)\[o = \left\lfloor{\frac{i + 2p - k - (k-1)(d-1)}{s}}\right\rfloor + 1 \,. \]

tensorrt 学习笔记

1. nvinfer1:Dims 表示的是CHW的各个纬度，而不是NCHW！
2. ModelImporter 类实现了nvonnxparser::IParser
3. onnx parser 中增加对插件的支持，需要修改 builtin_op_importers.cpp
4. 增加插件后，需要在文件 InferPlugin.cpp 注册插件

cmake notes

1. cmake command line option for x64 architecture:

cmake -A x64 ..

git tricks

How to delete all commit history in github?

1. Checkout

git checkout --orphan latest_branch

2. Add all files

git add -A

3. Commit the changes

git commit -am "commit message"

4. Delete the branch

git branch -D master

5. Rename the current branch to master

git branch -m master

6. Finally, force update your repository

git push -f origin master

linux notes

linux installer shell maker

https://github.com/megastep/makeself

How to disable gui when ubuntu desktop version booting

1. modify grub:

sudo vim /etc/default/grub

change

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

to

GRUB_CMDLINE_LINUX_DEFAULT="text"

update grub

sudo update-grub

2. disable lightdm service

sudo systemctl disable lightdm.service

3. If you want to start desktop

sudo servie lightdm start

4. To enable lightdm service, systemd has bug! ref: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1595454 solution(root user or use sudo):

systemctl enable lightdm
ln -s /lib/systemd/system/lightdm.service /etc/systemd/system/display-manager.service

How to disable guest session

sudo vim /usr/share/lightdm/lightdm.conf.d/50-ubuntu.conf

add:

allow-guest=false

ubuntu 18.04 disable xorg using nvidia card!

ref: https://askubuntu.com/questions/1061551/how-to-configure-igpu-for-xserver-and-nvidia-gpu-for-cuda-work

1. create file /etc/X11/xorg.conf with the following content:

Section "Device"
  Identifier      "intel"
  Driver          "intel"
  BusId           "PCI:0:2:0"
EndSection

Section "Screen"
    Identifier      "intel"
    Device          "intel"
EndSection

fix time difference in Ubuntu & Windows Dual Boot

Ubuntu use the hardware clock (RTC, real time clock) in universal time (UTC) by default while Windows use the clock in local time.

easy solution in ubuntu

$ sudo timedatectl set-local-rtc 1

reStructuredText notes

headings

• # H1, with overline, for parts
• * H2, with overline, for chapters
• = H3, for sections
• - H4, for subsections
• ^ H5, for subsubsections
• ” H6, for paragraphs

Indices and tables

• Index
• Module Index