Note that the finite-depth linear wave theory was derived as part of the problem class, so you can examine the video there for more details. Firstly, we look to linearise the two surface boundary conditions,
\begin{gather*}
\pd{\phi}{y}= \pd{\eta}{t} + \pd{\phi}{x}\pd{\eta}{x},\\
\pd{\phi}{t} + \frac{1}{2}|\nabla \phi|^2 + g\eta = 0,
\end{gather*}
which are imposed on \(y = \eta(x)\text{.}\) The quadratic terms can be ignored by assuming that the fluid is linearised about a small disturbance. Hence \(\eta\) is assumed to be small. Similarly, the velocity terms are expanded about \(y = 0\text{.}\) For example,
\begin{equation*}
\pd{\phi}{x}\biggr\rvert_{y = \eta} = \pd{\phi}{x}\biggr\rvert_{y = 0} + \pd{^2\phi}{x^2}\biggr\rvert_{y = 0}\eta + \ldots.
\end{equation*}
Thus we can approximate this quantity to leading order by
\begin{equation*}
\pd{\phi}{x}\biggr\rvert_{y = \eta} \sim \pd{\phi}{x}\biggr\rvert_{y = 0}.
\end{equation*}
Therefore, both kinematic and dynamic conditions are simplified in this manner, yielding
\begin{gather*}
\pd{\phi}{y}= \pd{\eta}{t},\\
\pd{\phi}{t} + g\eta = 0,
\end{gather*}
now imposed on \(y = 0\text{.}\)
We substitute now the ansatzes
\begin{equation*}
\eta = A\cos(kx - \omega t), \qquad \phi = f(y)\sin(kx - \omega t),
\end{equation*}
into the governing equations.
Written in terms of \(f\text{,}\) we must now solve the following system:
\begin{gather*}
f'' + k^2 f = 0, \\
f'(-h) = 0, \\
f'(0) = (\omega)A, \\
-(\omega) f(0) + gA = 0.
\end{gather*}
Now instead of writing the solution for \(f\) in terms of the two exponentials \(\e^{\pm ky}\text{,}\) it is better to write the general solution of the second-order ODE as
\begin{equation*}
f(y) = B \text{cosh}[k(y - y_0)],
\end{equation*}
where \(y_0\) is constant. We can then see that if \(y_0 = -h\text{,}\) then the boundary condition \(f'(-h) = 0\) is satisfied.
The last two boundary conditions, evaluated on \(y = 0\) yield
\begin{equation}
\begin{pmatrix}
\omega & -k \sinh (kH) \\
g & -(\omega)\cosh(kh)
\end{pmatrix}
\begin{pmatrix}
A \\ B
\end{pmatrix}
= 0.\tag{5.5.1}
\end{equation}
For there to be non-trivial solutions, we thus need
\begin{equation*}
(\omega)^2 = gk \tanh(kh),
\end{equation*}
and therefore there are two possible wave speeds, namely
\begin{equation*}
c = \mp \left(\frac{g}{k}\tanh(kh)\right)^{1/2},
\end{equation*}
and both are simply negatives of one another; so represent left and rightwards propagating waves of the same speed.