Welcome to our site! EDAboard.com is an international Electronic Discussion Forum focused on EDA software, circuits, schematics, books, theory, papers, asic, pld, 8051, DSP, Network, RF, Analog Design, PCB, Service Manuals... and a whole lot more! To participate you need to register. Registration is free. Click here to register now.
Network security is an enormous topic. Can you narrow down the scope of your project for us? Did your teacher give you any guidance on what your project should demonstrate?
How about a packet sniffer? If you are more interested in the defense side of things, you can see if you can make your program detect possible recon probes, such as a fin and ACK packet from an address that never properly set up a TCP connection. Could also see if you can detect ICMP traffic, port scans, etc. To do this properly is very complex(see Snort), but can be simplified down to something manageable. An issue is whether or not your lan is routed or switched, if it is the latter you may need to arp poison or run this on a machine acting like a firewall or gateway. Generally, promiscuous mode is useless in a switched network, unless of course you only care about monitoring one node.
A program that tries to probe and penetrate firewalls, AKA firewalking.
A TCP or UDP based traceroute program? This is very simple, and part of a firewalking program.
Write a program that sets up a man in the middle attack, there is at least one Java program out there that can do this: Paros.
These require access to raw packets, something that is not supported officially in Java, but there are implementations that utilize the libpcap and libnet libraries.
Write a Java based fuzzer(automated vulnerability discovery) against a single network protocol: FTP, HTTP, etc. This might require raw packets.
It is not really security related, but you could use raw packets and implement TCP/IP.
Getting back to the java.net package, you could take the Datagram classes and make them more reliable(AKA implement TCP over a UDP implementation.