You don't mention what OS your machine is using, but you might have a problem similar to one I had.
We run a retail store using POS software originally written as a DOS application that ran well under DOS and Windows 3 and Win 98. We updated to the latest version of that in time for Y2K, but still that supplier's last version based on the old DOS code base. (They had already re-written a version for Windows.) It still runs fine under Win XP, but it took a couple of tricks. The root of the problem is that the software, like lots written in that environment, writes directly to the serial port to print out a sales receipt. Under Win XP and later, that is not possible - Windows completely virtualizes the ports, and no software can write directly to them. Since you are using the serial port with very old software, you may NOT be able to use that with a current OS. But if it was working anyway, and especially if your OS is Win 98 or earlier, you are OK as long as you keep that. (I will note, however, that people have had a challenge trying to install and run Win98 (even 98SE) on a modern computer, because sometimes you can't find drivers for today's motherboard devices.)
By the way, I suspect your system uses the serial port bi-directionally for two functions. One is communication with the shop machines. The other, I suspect, is communication with the "dongle" - much software of that era used a unique hardware dongle on a serial port as a copy protection security measure.
In my case the solution was given us by Tech Support at the supplier. The key step is to use the NET USE commands in a small DOS batch file (that also starts up the app software) to re-direct traffic for the printer to the serial port. But the trick here is that you can't re-direct traffic intended for a serial port to a serial port. So I "installed" a fake identical printer to a Parallel port in Windows, then configured the app software to tell it that its printer was on that parallel port now. The NET USE commands then redirect that parallel port to the serial port, and it works. This communication is both ways - the software receives error signals, etc back from the printer. This technique also can be used to allow old software to print to a USB port.
The second part, for us, was to deal with the fact that the computer is not on a network, so how can Network Redirection work? There are two ways. One is to buy and install a cheap router that does not even connect to an outside network through its WAN port. But you DO connect one computer LAN port to one LAN port of the router so that, to the computer, it thinks it is on a network and you can install the networking software. The other technique is to use a small software network stack emulator that is among the many files on your Windows Install CD. It was originally designed to simulate the existence of a network for testing purposes on the computer, even when it actually has nothing connected to its LAN ports. I creates a virtual network stack and allows NET USE and similar network tools to work. If you need details of this, post here and I'll dig out the exact file name needed and the command lines required in WIN.INI (I think).