Mysterious incoming net packets keep waking up systems

I have three offices where there's a server and a number of stations
and all are set to go to standby, but to be able to be awakened if I
use RDP to connect for remote admin. But in the one office the systems
wake up a few seconds after going to sleep. I've narrowed it down to
something coming in from the internet, apparently with packets capable
of waking the systems up. In the network card settings there's a
selection for it can be woken by the "magic" frame or the "wake up"
frame. I forget which, but one type is apparently what RDP triggers.
If I turn off that kind of wake up the system stays in standby but
then I can't wake it remotely.

I have a nice router and switch which can probably tell me more than I
know how to use, but there is a small, steady stream of unicast
packets coming in from the internet and they are apparently the
culprit. I don't know how normal this is, if it's a remnant of someone
having had a virus, if it's a remnant of some file sharing software
(which I take off). I would appreciate any insight as to how to deal
with this.

Thanks,
Tom