Here's the results of timing tests over a large range of problem sizes.
Note how poorly my current parallelization approach is for target (the strings to be searched) sizes less than about 70kB.
Also notice the hiccup that occurs at 900kB. It appears to be hardware related.
In that example, yes - although 20M bases.
Yes. Now if these are required to be the same length as the query string (e g. 22 characters) then this can be quickly done with regular discrete correlation, esp. since it is the foundation of string searches in modern processors. However, here the...
In this application I am using regex to generate thousands of candidate matches which are then assayed by mathematical criteria. When written in a functional programming model, Mathematica produces an efficient computation.
At present I am focused on where to "slice" the data for parallel...
I agree. The point in monitoring it is two-fold: checking that usage is scaling linearly with number of parallel processes, and estimating the number of processes that would induce swapping.
Here what I'm noticing the processes are scaling fine, but each could be computing a lot more data.
Mathematics is the language of measurement. College calculus is a small part of its realm.
The algorithm I've implemented here is Variable Width Discrete Correlation. It is an application of signal processing to noisy data. In this case the signals are cell chromosomes.
Thank you for the...
@Ken g6
I have an extended Mathematica license permitting up to 12 parallel kernels. The next step up is out of my budget. Thus a 12 core system is about all I'll consider.
Here's some additional data from the test that produced the chart above. At 12 kernels I'm approaching the maximum...
@StefanR5R
Thank you for the ideas and specs. 🙂
The combinatorics of this computation are:
(# of targets)x(# of queries)x(# of regex per query).
My current design parallelizes across the #regex.
I'm checking to see if this is still the best choice. In fact, I might need a hybrid algorithm that...
@yottabit
Thank you for the details. 🙂
4.8 GHz is tempting.
The # of cores and channels is also interesting, esp. since my current Mathematica license limits me to 12 parallel processes.
I'm not sure if my liquid cooler would fit the form factor of these "builders", so a current system might...
I'm just about done rewriting the code. Previously it was a two program operation, first the regex scans with some basic filtering, then post processing with narrow filtering. I've rewritten the first part in such a way that the second is unnecessary. This week I also wrote an API plus some...
My perspective is that I've thanked folks here repeatedly for their input. I've also disagreed with some people and the discussion that followed was helpful.
That's standard procedure for a factory-authorized warranty installation.
But I didn't know about that BiOS setting - thanks for the pointer!
Also, someone here recommend turning on XMP so that's already completed.
@igor_kavinski
S. Wolfram was never a genius, nor is E. Musk. Both of them though hired brilliant people to start and sustain their enterprise. I've assisted some true geniuses over my career, so I do know the difference. Their capabilities are startling.
@Markfw
You are getting carried away with superlatives. Here is what I actually said:
Two things are very clear about my situation, which I've previously stated in this discussion thread: (1) I will have to upgrade my hardware to tackle a larger dataset which is on the back burner for now (2)...
You missed this in prior discussion:
On CPU with N cores, I will launch N-1 processes - each running a copy of my computation on a different segment of the data. This results in 70% to 80% CPU utilization. I also use advanced Bios settings, including overclock.
Please check prior posts for...
That's 20% of the cost of a warrantied system complete with liquid cooling. Then I'd need to purchase a CPU-locked Wolfram license to use it. Overall, a very poor use of my resources.
The speed-up has two components. 1/3 of it (in runtime) is due to the built-in SequenceAlignment function. 2/3 of it is due to the way I incorporated it. In particular, I eliminated all control structures and expressed the computation as a composite function (math terminology). This permits the...
For an example of functional programming in Wolfram, see "vettedCandidates" computation in the "FOURTH UPDATE" posted on this page: https://community.wolfram.com/groups/-/m/t/3269106
Absolutely! 🙂
SequenceAlignment is used to compare the actual primer (String of 16 or so letters) with each of the candidates returned by the regular expression search. So I don't believe it will be affected that way.
Yes, I've previously observed the bottleneck caused by independent kernels...
At the moment I'm driving a i7-1165G7 laptop with 4 cores, 16 GB RAM, and 12 MB cache. The non-parallel version of my code runs in turbo mode at about 4.6 GHz. When performing a fixed-length regex search on a String of length ~14 M bytes, I see 5 interrupts of about 0.015 seconds -- which I...
There are N-body problems in astronomy for which a well-crafted algorithm on a hypercube is the most efficient known solution. And yet, some of the most challenging problems in petroleum hydrology fail there but do very well on systems like the Frontier. And so on. Finally, I'll mention...
I came to this forum because I previously depended on AnandTech news briefs for up to date industry information. They closed shop last month.
I imagine the regulars on this forum are accustom to looky-loos asking for advice on a new system, and thus the responses I've received pointing at...
There many different supercomputer architectures. Speed (e.g. petaflops) is a poor measure of any unless you have a very large embarrassingly parallel vector computation.
No you can't, because you are lacking a Wolfram license, which would have to be the extended version for extra parallel kernels.
Two things are very clear about my situation, which I've previously stated in this discussion thread: (1) I will have to upgrade my hardware to tackle a larger...
... if the load being tested is a standard application. But I'm not running a database, engaged in AI, or trying to profit from bit-coin mining. I am spawning parallel computations on cores of the same processor.
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.