Hoeffding inequality formular‘s left side is about something bad happending. You don‘t want this bad thing to happen, so that you can use a upper bound to constraint it. The right side of the Hoeffding inequality is the restriction. As you can see, it is either/both the larger the sample number (N) you have, or the smaller tolerance (epslon) you set that can make the upperbound smaller.
on the other hand, if your hyphothesis set size M which is large (say infinity), the upper bound of the Hoeffding inequality needs to multiply this M (according to some math), then the upper bound becomes infinity. So we need to abstract quantity from this infinity to make it finite.
the feasibility of learning is split into two questions:
can we make sure that E_out(g) is close enought to E_in(g), where E_in(g) is the hypothesis g‘s in-sample error, E_out(g) is hypothesis g‘s out-sample error. --> Hoeffding inequality answers this.
can we make E_in(g) small enough ? --> depends on the complexity of H -- the number (M) of the hypothesis in the hypothesis set H, and the complexity of the target function f -- learning a non-linear target function is more prone to make E_in(g) bigger.
a model corresponds to a hypothesis set (H), a H contains a set of hypothesis (h), you choose one h and it is not called g, which (you believe) is approximate to the target function f.
how you pick g depends on the algorithm, hypothesis set and data you use, take perceptron for example, a g is picked util all the points are classified. There are multiple hypothesis that classify points correct, so how do you pick up the g?
