The link to yours isn't working for me.
I'm pretty much in the same place - my girlfriend's son is three years old and decently responsible with "important" things that can get broken. Every now and then he likes to bang on stuff just for fun, so it's not a great idea to let him sit at a computer unattended for long periods of time. With that being said, he LOVES Microsoft paint. Just drawing circles keeps him occupied for hours... which, at three years old, can be a blessing.
Anyway, I recently came across an older Athlon Thunderbird 950MHz system with 512MB of DDR and integrated graphics that runs Windows Xp perfectly fine. It's really too old to even give away, so I'm going to set it up for him to have a blast on. The best part is, it cost abosolutely nothing.
If yours is out of the key-banging-for-fun stage, I'd probably hit Dell or similar companies for one of the $400 specials. The $349 model I'm looking at now comes with a Sempron 3600+, 160GB hard drive, and 1GB of memory, but no monitor - that's a pretty good computer for the money. The next step up ($549) comes with a Pentium dual-core (Core 2 Duo based, I believe), 1GB of memory, 160GB hard drive, and a 17" monitor. Both computers are pretty cheap, and actually quite fast considering the price.