One design pattern for the creation of an FIR lowpass with transition band of width $\Delta\omega$ is the so called Raised Cosine (RC) filter. The name follows from the frequency response which is in the shape of a raised cosine. The frequency response $H_{\mathrm{RC,LP}}(\omega)$ is defined as follows $$ H_{\mathrm{RC,LP}}(\omega)=\begin{cases} 1 & |\omega|\leq\omega_{\mathrm{d}}\\ \frac{1}{2}\left(1+\cos\left(\frac{\pi}{\Delta\omega}\cdot\left(|\omega|-\omega_{\mathrm{d}}\right)\right)\right) & \omega_{\mathrm{d}}\leq|\omega|\leq\omega_{\mathrm{s}}\\ 0 & \omega_{\mathrm{s}}\leq|\omega| \end{cases} $$ This representation needs to be transformed to the standard form (cf. [3]) $$ H_{\mathrm{RC,LP}}\left(\omega\right)=\begin{cases} 1 & |\omega|\leq\pi\cdot\frac{1-\alpha}{T}\\ \frac{1}{2}\left(1+\cos\left(\frac{T}{2\alpha}\cdot\left(|\omega|-\pi\cdot\frac{1-\alpha}{T}\right)\right)\right) & \pi\cdot\frac{1-\alpha}{T}\leq|\omega|\leq\pi\cdot\frac{1+\alpha}{T}\\ 0 & \pi\cdot\frac{1+\alpha}{T}\leq|\omega| \end{cases} $$ By comparing these two equations we find the values for the parameters $T$ and $\alpha$ $$ T=\frac{2\pi}{\omega_{\mathrm{s}}+\omega_{\mathrm{d}}}=\frac{\pi}{\omega_{\mathrm{c}}} $$ $$ \alpha=\frac{\omega_{\mathrm{s}}-\omega_{\mathrm{d}}}{\omega_{\mathrm{s}}+\omega_{\mathrm{d}}}=\frac{\Delta\omega}{\omega_{\mathrm{s}}+\omega_{\mathrm{d}}}=\frac{\Delta\omega}{2\pi}\cdot T=\frac{\frac{\Delta\omega}{2}}{\omega_{\mathrm{c}}} $$ Where $\omega_{\mathrm{c}}$ is defined as follows $$ \omega_{\mathrm{c}}=\frac{\omega_{\mathrm{s}}+\omega_{\mathrm{d}}}{2} $$ The invers fourier transformation of the frequency response parametrized by $T$ and $\alpha$ is given in [3] as follows $$ h_{\mathrm{RC,LP}}(t)=\begin{cases} \frac{1}{T}\cdot\operatorname{si}\left(\frac{\pi}{2\alpha}\right)\cdot\frac{\pi}{4} & |t|=\frac{T}{2\alpha}\\ \frac{1}{T}\cdot\operatorname{si}\left(\pi\frac{t}{T}\right)\cdot\frac{\cos\left(\pi\alpha\frac{t}{T}\right)}{1-\left(2\alpha\frac{t}{T}\right)^{2}} & otherwise \end{cases} $$ Insertion of $T$ and $\alpha$ results in the following equation parametrized by $\Delta\omega$ and $\omega_{\mathrm{c}}$ $$ h_{\mathrm{RC,LP}}(t)=\begin{cases} \frac{\omega_{\mathrm{c}}}{\pi}\cdot\operatorname{si}\left(\omega_{\mathrm{c}}\frac{\pi}{\Delta\omega}\right)\cdot\frac{\pi}{4} & |t|=\frac{\pi}{\Delta\omega}\\ \frac{\omega_{\mathrm{c}}}{\pi}\cdot\operatorname{si}\left(\omega_{\mathrm{c}}t\right)\cdot\frac{\cos\left(\frac{\Delta\omega}{2}t\right)}{1-\left(\frac{\Delta\omega}{\pi}t\right)^{2}} & otherwise \end{cases} $$ After designing the lowpass impulse response, we need to shift it to the desired frequency range to obtain the notch filter $$ h_{\mathrm{RC}}(t)=h_{\mathrm{RC,LP}}(t)\cdot e^{\mathrm{j}\omega_{\mathrm{off}}t} $$ As a final step the impulse response is sampled at $L$ steps to obtain the discrete FIR filter impulse response $$ h_{\mathrm{RC,FIR}}[n]=h_{\mathrm{RC}}\left(n-\frac{L-1}{2}\right)\quad n=0,1,2,\ldots,L-1 $$
Least Squares designOne approach to design an optimal filter is the least squares method as presented in [9] which seeks to minimize the squared error between the designed amplitude response $A(\omega)$ and the desired amplitude response $A_{\mathrm{d}}(\omega)$ $$ \min\intop_{0}^{\pi}W(\omega)\cdot\left|A_{\mathrm{d}}(\omega)-A(\omega)\right|{}^{2}\mathrm{d}\omega $$ The weight function $W(\omega)$ is choosen so that only the frequencies of the stopband and the passband are included in the optimization, excluding the frequencies from the transitionband $$ W(\omega)=\begin{cases} 1 & |\omega|\leq\omega_{\mathrm{d}}\\ 0 & \omega_{\mathrm{d}}<|\omega|<\omega_{\mathrm{s}}\\ 1 & \omega_{\mathrm{s}}\leq|\omega|\leq\pi \end{cases} $$ The impulse response of the least squares filter is then computed by the in Matlab integrated function $\mathtt{firls()}$, for further details see [9] or [10] $$ h_{\mathrm{LS,FIR,LP}}[n]=\mathtt{firls(n,f,a)} $$ where
Another approach to design filters with "don't care" transition bands it the equiripple filterdesign [11]. In this approach one seeks to minimize the maximum difference between the desired amplitude response $A_{\mathrm{d}}(\omega)$ and the designed response $A(\omega)$: $$ \min\left[\max_{\omega\in\left[0,\pi\right)}\left|W(\omega)\cdot\left(A_{d}(\omega)-A(\omega)\right)\right|\right] $$ The impulse response is computed similar to the least squares design, for details see [11] and [12] $$ h_{\mathrm{PM,FIR,LP}}\left[n\right]=\mathtt{firpm(n,f,a)} $$ where