‘So I think I need to apply a bandpass filter in freq domain.’
Don’t. That is extremely difficult, since the Fourier transform is symmetrical and it would be necessary to do that on the ‘positive’ as well as the ‘negative’ (complex-conjugate) parts, and so is generally not worth the effort.
Assuming that ‘x’ is your time-domain signal, do this instead:
fs = 500e6;
fn = fs/2;
f1 = 10e3, f2 = 100e3;
wn = [f1 f2]/fn;
[z,p,k] = butter(20,wn,'bandpass');
[sos,g] = zp2sos(z,p,k);
set(subplot(2,1,1), 'XLim',[0 200E3])
set(subplot(2,1,2), 'XLim',[0 200E3])
x_filtered = filtfilt(sos,g,x);
The transfer-function implementation of that filter is demonstrably unstable. The second-order-section implementation is stable and will likely do what you want it to.
I suggest using a different filter design, perhaps an elliptic filter. See ellipord and ellip as well as zp2sos.