I wouldn't think you'd have anywhere near an enjoyable experience. I recall back when BD/HDDVD were coming out I had an Athlon x2 4600 and it could not handle disks without hardware acceleration and could not handle 1080p MKVs at all due to no HA for MKV at the time. That CPU was roughly twice as powerful as a P4 2.6ghz.
That's just on the playback side of things, there is many other considerations to take into account. Win Vista/7/8/8.1 chug on a single core, I would not install a modern OS on that, WinXP doesn't have much support these days and will be officially dropped on April 9th, I would not use XP too much of a risk.
That leaves a linux based setup like XBMC running on openelec. I have no idea how it performs compared to the Windows versions but I found XBMC's interface was a little chuggy on my recently replaced Athlon II 250 with 4gb of ram and a Radeon HD6570, I would think the interface would not be smooth on older hardware.
In your position I would scrap that system and start fresh, either with a new PC build or one of the cheap media streamer devices. Both would give you a better experience than a P4 system.