It's exactly like the title says, when I set my monitor (which does in fact support 165Hz and does so perfectly on X11) to 165Hz the screen starts flashing. More specifically when I change focus from one window to another, or when I press any key on my keyboard, the screen just turns off for about 5-10 seconds. I genuinely don't know what's going on. I am using OpenRC for my init system so that may have something to do with it, but I didn't see anything about it on the Gentoo wiki. I am also using an AMD GPU, which should be working better than an NVIDIA GPU, but maybe i have something configured wrong? I am also using the Zen kernel, and have vrr disabled (as far as I can tell, according to hyprctl vrr is false on my monitors, and I have my config set with vrr = 0. I used vrrtest, but I honestly couldn't make heads or tales of it, but it seems like the bars are stuttering but not tearing). Any help is appreciated!
Hm not sure if this is the same as what I experienced, but I just bought a new 4k 120hz tv and when i tried to go to that resolution (in hyprland) I had the exact same thing (going to black, specifically when hitting a key or moving the mouse around between windows).
The fix was to turn off "game mode" on my tv so I assumed it was something specific to my tv, but this does sound pretty much exactly like what was happening with my tv.
Obviously I am not offering any solution, since I would assume your monitor doesn't have a "game mode" but anyway, I experienced the same thing using hyprland.
Reporting that everything works well is not much of a help but I don't have any problems running HP Omen 24" over HDMI on Hyprland.
monitor=DP-1,1920x1080@165,0x0,1
monitor=eDP-1,disable
Same thing with XLibre and .xinitrc
xrandr --dpi 96
xrandr --output HDMI-1-1 --mode 1920x1080 --rate 165 --primary
xrandr --output eDP --off
Zero problems in both cases.
Again, not sure if your issue is the same as mine. But I was able to circumvent the blacking out by switching down from 120hz to 100hz. With my TV in "Game Mode" it functions fine at 4k 100hz, but at 4k 120hz I have your blacking out issue.