Compact 433 MHz PCB help me choose an antenna

I have designed a small transmitter (and receiver) that will use a Microchip MRF49XA transceiver chip. I have layed out both boards but on the transmitter board (remote control) I am having trouble selecting an antenna. The board is very compact as you can see in the picture below. I was planning on using a piece of wire for antenna but realized it would have to be like 17 cm which I figured wouldn't work since it would have to loop around the box in order to fit. On that board also there will be a CR2032 battery on the bottom side (you can see the silkscreen if you look in the center).



I looked into helical antennas such as this one: https://www.mouser.com/ds/2/238/ant-xxx-hexx-3485.pdf
but they need to have a good bit of clearance from the ground plane (0.5 in. for that one). I will however probably use that antenna on the receiver board (because I have much more room to play with). Anyways just looking for some tips or suggestions about which direction to go. The transceiver chip can operate in the 868 MHz and 915 MHz bands also which is another option if it would help with a smaller antenna. I'm pretty new on the wireless stuff but trying to learn more so any tips would be appreciated.

Could someone also explain in what case a ground plane needs to be right near the antenna and in which case not. I thought monopoles used the ground plane as the other half or reference and to reflect the waves. How come a helical needs to be farther away?

If you're interested more about what the project is for exactly see my blog:
ltkenbo.blogspot.com

I wouldn't worry about the antenna just yet. I see a bit of a problem with your layout in the RF section. First of all all the RF traces should be 50 Ohms. Second of all the placement of the inductors is not ideal - I'd put L2 symmetrically between RFN and RFP, similarly L3 and arrange all caps so that both RFN and RFP are as short as possible, as straight as possible and symmetrical.
Then there is the ground plane in the bottom layer. I cannot see exactly but I think that the trace from pin 10 (reset) cuts right through the ground plane. If this is the case then all the vias in the SE corner are connected to a partially discontinued ground plane, which defeats their objective.
The last thing is that thicker trace between C6 and A1. Is this a 50Ohm trace? it looks about 100mils wide. That means you use standard 1.6mm (62mils) FR4. That also means that all other RF traces are much higher impedance. I'd estimate they are about 20mils wide, which means they are approximately 110 Ohms. This will have a negative effect on the balun performance and decrease sensitivity and output power.
I'd use a thinner board, possibly 0.8mm (50Ohm trace would be about 50mils).

And yes if you can use higher frequencies.

Here is the bottom of the board:



Ok I can rearrange the balun circuit, I was just trying to arrange them to keep the traces between them as short as possible.

The last thing is that thicker trace between C6 and A1. Is this a 50Ohm trace? it looks about 100mils wide. That means you use standard 1.6mm (62mils) FR4. That also means that all other RF traces are much higher impedance. I'd estimate they are about 20mils wide, which means they are approximately 110 Ohms. This will have a negative effect on the balun performance and decrease sensitivity and output power.

Click to expand...

You are correct the wider trace was supposed to be the feed line (73 mil) which yes you are correct I was planning on using 63 mil FR4. This should be around 50 ohms. The traces within the balun circuit are actually 8 mils which would be even higher impedance I see. I layed out another RF circuit once but used a balun package instead of discrete components so I am new to this. So I can leave the differential output at < 50 ohms right? I was under the impression that the 50 ohm impedance was taken at the end of the balun circuit (this was just an assumption though). But everything else after the differential output has to be a 50 ohm trace? < 63 mil board isn't really an option because I planned on using sparkfun's batchpcb prototype pcb service.

I cannot see exactly but I think that the trace from pin 10 (reset) cuts right through the ground plane. If this is the case then all the vias in the SE corner are connected to a partially discontinued ground plane, which defeats their objective.

Click to expand...

So I should just remove those vias? They do have another path from the left side since the bottom side is relatively clear of traces.

The vias should stay. it is not as bad in the bottom layer as I thought. But if you can run reset line in the top layer, then do it.

73mil with 63mil board gives you 65Ohm! (Epsilon r around 4.5).

You need to rearrange the balun and keep the non-50Ohm rf traces length to absolute minimum. It will not be ideal but will acceptable.

Careful with copper floods in the top layer. If you cannot properly ground them then remove them.

But what should the width of the traces between the balun components be? Hmm maybe I used the the wrong epsillon r.

Because it is a lumped element design there should be no traces between balun components. The next best thing would be 50Ohms as short as possible.
Since you cannot physically have 50ohm traces make them as short as possible.

One more thing - the DC power line to L1 should have much higher impedance than the RF trace it injects DC to. Again you cannot possibly make it thinner than 7 mil but at least make it as thin as current carrying capacity and PCB technology permits.

Ok awesome, did not know that about the power line. I'm working on those fixes right now. Any way you could answer some of my questions about the antenna also?

How can I found out more about this stuff without asking people? Any good books that are not crammed purely intense mathematical examples? I am an electrical engineering major about to graduate and have had internships but most involving MCUs and digital stuff, not much RF technology (which is what I want to learn more about).

Edit:

Here is what I did, I tried to follow your advice and make the order of components similar to the schematic itself. I jammed them really close together so I included 2 pics, one normal and one showing only the top layer: