It will almost certainly get too hot (eventually) if you remove the fan.
You might try to use a PC fan speed adjuster on it, or just a small resistor. Search google for what size resistors people use to slow the fan down and make it quieter - I've never done it that way. Or just experiment!
You will only really know by trying it, but I bet that removing it completely is a bad idea. You might get away with slowing the fan down.
A final idea: buy a quieter fan, but I guess you want to do it more cheaply.
Edit to add: I am assuming the fan is powered from the 12V - check that!!! There is a possibility it might run from the mains on an old PSU. If so, don't mess with it.