Hi All!
I've been lurking around the Caching discussions for months now and have finally moved country and put together a new system and need your advice guys!
So... here's my use case and what I have in my system and what I want to achieve.
---
Overview Of system
What I'm trying to achieve
In a theoretical sense, I'm trying to maximize my read/writes for the large data I have to process. Typically my work means loading several files totalling about 200-1000mb, doing some processing to them, and then writing them back to the HDD as a single file at about 500-2000mb.
What I think i need is a RAM disk/cache (not sure which is best here) that is anywhere from 5gb-20gb and then use that as a write-back cache. I don't care too much if I lose power during the RAM writing the data over to permanent storage, I'll just redo that file. Possibly an NVME cache drive would also help.
---
What I used to have, which worked well but had limitations, was a RAID0 array of 12TB (4x4). This would give me a solid 500mb/s speeds. I then had that "mirror" itself to a NAS (I don't have a 10GbE NAS yet) and the NAS would take care of the data backup/safety as it was RAID5. With this workflow it was like having a local cache of the NAS files stored locally in an unsafe array (RAID0) but with 5x speeds (500mb/s).
I've tried StoreMI but it's limited to 2GB or 4GB with a license and doesn't seem to use the RamCache as a write back (only read cache from the looks).
Options I'm considering and need advice on:
I really appreciate the advice you can all give in advance! I know this is a really long post but I hope I have described my situation well.
@BonzaiDuck is likely the perfect person to answer this but I appreciate if he's too busy at the moment!
I've been lurking around the Caching discussions for months now and have finally moved country and put together a new system and need your advice guys!
So... here's my use case and what I have in my system and what I want to achieve.
---
Overview Of system
- AMD 3900x
- Gigabyte Aorus Master motherboard
- 64gb of RAM in the system - most of the time only need 30-40gb of it, so there's room for RAM cache/disk
- 1tb 960 Evo (system drive)
- 4x 2TB conventional drives
What I'm trying to achieve
In a theoretical sense, I'm trying to maximize my read/writes for the large data I have to process. Typically my work means loading several files totalling about 200-1000mb, doing some processing to them, and then writing them back to the HDD as a single file at about 500-2000mb.
What I think i need is a RAM disk/cache (not sure which is best here) that is anywhere from 5gb-20gb and then use that as a write-back cache. I don't care too much if I lose power during the RAM writing the data over to permanent storage, I'll just redo that file. Possibly an NVME cache drive would also help.
---
What I used to have, which worked well but had limitations, was a RAID0 array of 12TB (4x4). This would give me a solid 500mb/s speeds. I then had that "mirror" itself to a NAS (I don't have a 10GbE NAS yet) and the NAS would take care of the data backup/safety as it was RAID5. With this workflow it was like having a local cache of the NAS files stored locally in an unsafe array (RAID0) but with 5x speeds (500mb/s).
I've tried StoreMI but it's limited to 2GB or 4GB with a license and doesn't seem to use the RamCache as a write back (only read cache from the looks).
Options I'm considering and need advice on:
- Continue with RAID0 HDD array as described
- RAID0 array + Nvme SSD Cache (is this possible?)
- Use PrimoCache or similar and setup as write-back
- Upgrade the NAS with a 10GbE model and forget about cache's etc
- Honestly this upgrade is going happen anyway as I need more than a 4bay NAS now.
I really appreciate the advice you can all give in advance! I know this is a really long post but I hope I have described my situation well.
@BonzaiDuck is likely the perfect person to answer this but I appreciate if he's too busy at the moment!