Zooming causes constant intersection checks in React Three Fiber even with only onPointerDown event handler added

[snhwv]

I came across an issue where comments suggested using BVH(#320), but it doesn't solve my underlying problem. When zooming the view, React Three Fiber keeps performing intersection checks with the scene, even though I only set an onPointerDown event handler. Am I using it incorrectly? Is there any configuration or setting that can help avoid this issue?
here is my example:
https://codesandbox.io/p/sandbox/view-skissor-2-forked-r36tcc