@Hermetian I'm not very familiar with Mathematica, but looking at your code, I don't see anything very Mathematica-specific. Nothing about math function manipulation or integral approximations or anything like that. It just looks like string manipulations with regular expressions.For a product similar to Mathematica, look at Maple license cost. Are they overcharging too?
For people who need that kind of products, they're worth every penny spent.
Mathematics is the language of measurement. College calculus is a small part of its realm.looking at your code, I don't see anything very Mathematica-specific. Nothing about math function manipulation.
Thank you for the suggestion. I'm familiar with Python and many other languages, beginning with octal and Algol. Python is inherently slow and poorly supported in terms of efficient, error-free libraries. It does support a functional programming model but it doesn't compile any better than sequential style. The best compiler I've had experience with is Cray Fortran. Wolfram is a close second, and comes in first in terms of functionality.i really think you should look into doing this in Python instead.
record Result(Matcher regexMatcher, String pattern){};
public Stream<Result> evalRegex(Stream<String> dnaSequence, String regex) {
var pattern = Pattern.compile(regex);
return dnaSeqence.parallel()
.map((dna)->new Result(pattern.matcher(dna), dna))
.filter((r)->r.regexMatcher().matches())
}I agree. The point in monitoring it is two-fold: checking that usage is scaling linearly with number of parallel processes, and estimating the number of processes that would induce swapping.While "% CPU load" shows whether threads are busy or are sleeping, it does unfortunately not show how the CPU time is spent.
In this application I am using regex to generate thousands of candidate matches which are then assayed by mathematical criteria. When written in a functional programming model, Mathematica produces an efficient computation.Are you using any of the mathematics capabilities of Mathematica at this point or mainly regex processing?
If speed were the only issue I would program in Assembler. It harkens back to the days when we wrote graphics in raw postscript!Here's an example of how you can do parallel processing of a regex against a bunch of strings
You could split this into 2 applications, 1 which generates the candidate matches and a second that runs the mathematical criteria. That might give you the best of both worlds.which are then assayed by mathematical criteria.
I'd STRONGLY encourage you not to do this. Modern compilers and JITs are really good. You are not only unlikely to beat them in terms of performance, you'll likely make things slower.If speed were the only issue I would program in Assembler. It harkens back to the days when we wrote graphics in raw postscript!
In that example, yes - although 20M bases.If I understand the application correctly, you're searching the 200m-ish bases of chromosome 4 for partial matches to a 22-character string?
Yes. Now if these are required to be the same length as the query string (e g. 22 characters) then this can be quickly done with regular discrete correlation, esp. since it is the foundation of string searches in modern processors. However, here the "matches" are allowed to vary in length. For example, a couple of the errors might be due to missing characters in the sequence, or likewise due to a few inserted characters.The trick would seem to be "partial" matching.
Yes and no.Does Java allow addressing more than 4GB these days
On inspection, the dataset was the wrong size 😳Also notice the hiccup that occurs at 900kB.
Edit 2: The M1 max massively outperforms the i7 in the multi-kernel benchmarking with the M1 max scoring 13.257 and the i7 scoring 6.296 when both are run with 8 kernels.
It's 13k for a 8P+2E cores vs 6k for an i7 with 4 cores, so I'm not sure the advantage is that large.
That's another option for you. How much does your i7 score in the multi-kernel benchmark?
Things may have improved considerably since then and especially by the time he decides to get a Mac, he may be able to snag an M4 device.It's 13k for a 8P+2E cores vs 6k for an i7 with 4 cores, so I'm not sure the advantage is that large.
Yes it should be quite competitive... but quite expensive too!Things may have improved considerably since then and especially by the time he decides to get a Mac, he may be able to snag an M4 device.
Oh I didn't know you resurrected your Mac, nice 😀I was kinda taken aback by my M1 MBA last night scoring just 7 seconds on battery power in overclock.net Excel benchmark sheet using Librecalc Apple Silicon version. Ryzen 9950X scores about 11 seconds first try and roughly 9 seconds on the second try. Will try again tonight to see what the 2nd run time is on the M1.
Thanks to your charging tipOh I didn't know you resurrected your Mac, nice 😀
It's an i9-10900KFThat's another option for you. How much does your i7 score in the multi-kernel benchmark?
I'm almost sure that your time could be cut by close to half if you invest in this build: https://pcpartpicker.com/list/VvqwMVActual runtime on "standard" problem: 4.7 days for queries of 8 markers on an Apple genome library.
Estimated runtime of "wannabe" problem of 1200 markers on 12 Tea genomes: 36.6 weeks. Still a bit too long, but getting closer. 🙂
While running the tests, did you do any system profiling? Was it effectively keeping all cores busy? How busy was RAM? Did you notice anything that appeared to be a limiting factor?@Nothingness
I have finished the benchmark tests of the improved algorithm on my memory-upgraded system. Here are some takeaways.
Overall improvement in runtimes: from weeks to days.
Overall paradigm shift: prior to memory upgrade, the most efficient approach was to parallelize the innermost loop. After the memory upgrade, the most efficient approach is to parallelize the outermost loop (previously not possible). Implementation of the built-in SequenceAlignment function provided further speed-up.
Actual runtime on "standard" problem: 4.7 days for queries of 8 markers on an Apple genome library.
Estimated runtime of "wannabe" problem of 1200 markers on 12 Tea genomes: 36.6 weeks. Still a bit too long, but getting closer. 🙂
Yes.While running the tests, did you do any system profiling? Was it effectively keeping all cores busy? How busy was RAM? Did you notice anything that appeared to be a limiting factor?