A Google search will provide suitable answers.
after https://support.wolfram.com/39353?src=mathematica under Central Processing Unit.The kernel uses highly optimized multithreaded libraries such as Intel MKL and IPP, which are tuned for optimal performance and take advantage of advanced CPU features when available. This is important for vectorized machine arithmetic and numerical linear algebra (BLAS, …) routines, which are fundamental building blocks for many computational tasks.
I don't know off-hand. There are memory chip stats for my processor published somewhere.Efficient in what sense? If you mean one with lowest latency, I would be curious to see the CPU-Z memory timings tab.
You need not write any code. A number of "DNA analysis" benchmarks are available. Additionally, Mathematica has a benchmark suite which just might run under the trial version.If you could provide a simple example of one whole calculation, I think I could whip up a quick C program because this seemingly looks like text processing more than anything else, unless some indexed lookups are involved in there too?
Ah. That's great! Need to look at it then.Additionally, Mathematica has a benchmark suite which just might run under the trial version.
Thanks the details. So your problem is mostly around regexp matching. I should read more about bioinformatics before thinking more about the issues at hand. Is there some freely available resources that I could read to get a better grasp at what you're doing?The post-processing run time is less than 20 minutes, sometimes less than 3. It involves the assimilation of hundreds of files, vetting the contents, and then writing a single output file. I have a high-end SSD, plus I'm using a binary file format so the I/O portion is under a second.
Yes, going from a high-level language such as Wolfram Language and its tuned libraries would be an enormous task that would require a lot of expertise in many algorithmic fields.Of course one could write this code in a number of other languages for compilation to a binary executable. It would require construction or acquisition of compiler-compatible libraries of set-theoretic functions; i.e. a lot of software management overhead.
There's no writable data shared during the parallel phase, right? And many regexps apply to the same genome data?The regular expressions are not complex. In fact I initially tried using a single, complex regex per marker but rapidly found out that the recursion limit will be exceeded on chromosome size strings. Consequently I replaced the complex regex with many individual cases. Soon after that I realized that parallelization would be necessary.
Real men do it with sed. And they have fabs!This actually happens when the users are too focused on doing their job. One of the funniest examples was a bunch of guys in the Telex dept of my organization who used to open huge text files in Notepad, then do a Replace on them and sit back and watch as Notepad replaced the occurrences of the text string one by one from top to bottom. Some had old PCs so it would take anywhere from 30 seconds to a minute or more for the process to finish. And they did this multiple times a day. When I saw them doing this, I told them to open the text file in Wordpad and do the replace. It finished in the blink of an eye. Now of course, I wouldn't have known that if I hadn't noticed the speed difference myself. That's why when trying to solve problems, I evaluate multiple possibilities because settling on one or the first solution may lead to wasting tons of time in the long run.
Bioinformatics is a very general term, I believe you are asking about genomics. The facts in genomics have significantly changed in the past 5 years with the introduction of the PacBio Sequel series of long read sequencers. The situation is akin to the Michelson–Morley experiment in physics.Is there some freely available resources that I could read to get a better grasp at what you're doing?
There's no writable data shared during the parallel phase, right? And many regexps apply to the same genome data?
I've long been interested in Wolfram Language.
Last year I made a thorough study of how run time scales with problem sizes, parallelism sizes 0 to 14, and parallelization approaches. I plan to repeat it after the memory update in a few weeks. Thank you for the pointer to PerfMon.... Start with a smaller example dataset on a single core and time a run. Add a core and do it again.
Once you mentioned that, I immediately thought of GPUs and sure enough: https://www.nvidia.com/en-us/case-studies/long-read-sequencing/PacBio Sequel series of long read sequencers.
Some very good thoughts!Just some thoughts ...
And speaking of the bit encoding, here's an article about just that https://medium.com/analytics-vidhya/bioinformatics-2-bit-encoding-for-dna-sequences-9b93636e90e2
Here's an example regex string from my codes
CAA.{17,17}AG
// Example usage
const dnaString = "AACAAAAAAAAAAAAAAAAAAAAGG";
const bitEncodedDNA = encodeDNA(dnaString);
const decodedDNA = decodeDNA(bitEncodedDNA);
console.log('Decoded DNA Sequence:', decodedDNA);
const patternPrefix = 'CAA';
const patternSuffix = 'AG';
const wildcardLength = 17; // 17 nucleotides 34 bits
const matches = searchDNA(bitEncodedDNA, patternPrefix, patternSuffix, wildcardLength);
console.log("Matches found at nucleotide positions:", matches);
const wildcards = extractWildcards(bitEncodedDNA, matches, patternPrefix.length, wildcardLength);Decoded DNA Sequence: AACAAAAAAAAAAAAAAAAAAAAGGAAA
Matches found at nucleotide positions: [ 2 ]
Wildcard sequences found: [ 'AAAAAAAAAAAAAAAAA' ](* Define the bit patterns for each nucleotide *)
nucleotideToBit = <|"A" -> "00", "C" -> "01", "G" -> "10", "T" -> "11"|>;
bitToNucleotide = <|"00" -> "A", "01" -> "C", "10" -> "G", "11" -> "T"|>;
(* Function to encode a DNA sequence into a ByteArray *)
encodeDNA[sequence_String] := Module[{bitString, paddedBitString, byteList},
bitString = StringJoin[sequence /. nucleotideToBit];
(* Pad the bitString to make it a multiple of 8 bits (if necessary) *)
paddedBitString = StringPadRight[bitString, Ceiling[StringLength[bitString]/8]*8, "0"];
(* Convert the padded bit string to a list of bytes *)
byteList = FromDigits[#, 2] & /@ StringPartition[paddedBitString, 8];
(* Return the ByteArray *)
ByteArray[byteList]
]
(* Function to decode a ByteArray back into a DNA sequence *)
decodeDNA[byteArray_ByteArray] := Module[{bitString, nucleotideList, decodedSequence},
bitString = StringJoin[IntegerString[#, 2, 8] & /@ Normal[byteArray]];
nucleotideList = StringPartition[bitString, 2];
decodedSequence = StringJoin[nucleotideList /. bitToNucleotide];
decodedSequence
]
(* Function to convert a DNA string pattern to its bitwise integer representation *)
dnaPatternToBits[pattern_String] := StringJoin[pattern /. nucleotideToBit]
(* Function to match bits in the ByteArray with the pattern *)
matchBits[byteArray_ByteArray, startBit_, patternBits_String] := Module[
{bitString, bitSegment},
bitString = StringJoin[IntegerString[#, 2, 8] & /@ Normal[byteArray]];
bitSegment = StringTake[bitString, {startBit + 1, startBit + StringLength[patternBits]}];
bitSegment === patternBits
]
(* Function to search for the pattern using optimized bitwise comparison *)
searchDNA[byteArray_ByteArray, patternPrefix_String, patternSuffix_String, wildcardNucleotides_] := Module[
{bitLength, prefixBits, suffixBits, prefixLength, suffixLength, wildcardLength, matches = {}, i},
bitLength = Length[byteArray] * 8;
prefixBits = dnaPatternToBits[patternPrefix];
suffixBits = dnaPatternToBits[patternSuffix];
prefixLength = StringLength[prefixBits];
suffixLength = StringLength[suffixBits];
wildcardLength = wildcardNucleotides * 2; (* Convert nucleotides to bits *)
For[i = 0, i <= bitLength - (prefixLength + wildcardLength + suffixLength), i += 2,
If[
matchBits[byteArray, i, prefixBits] &&
matchBits[byteArray, i + prefixLength + wildcardLength, suffixBits],
AppendTo[matches, i/2] (* Convert bit position to nucleotide position *)
]
];
matches
]
(* Function to extract the actual wildcard strings found between the prefix and suffix *)
extractWildcards[byteArray_ByteArray, matches_List, prefixLength_Integer, wildcardNucleotides_Integer] := Module[
{wildcards = {}, wildcardLength, i, match, bitString, wildcardBits, nucleotideList, wildcardDNA},
wildcardLength = wildcardNucleotides * 2; (* Convert nucleotides to bits *)
For[i = 1, i <= Length[matches], i++,
match = matches[[i]];
bitString = StringJoin[IntegerString[#, 2, 8] & /@ Normal[byteArray]];
wildcardBits = StringTake[bitString, {match * 2 + prefixLength + 1, match * 2 + prefixLength + wildcardLength}];
(* Convert the bit string back to a DNA string *)
nucleotideList = StringPartition[wildcardBits, 2];
wildcardDNA = StringJoin[nucleotideList /. bitToNucleotide];
AppendTo[wildcards, wildcardDNA];
];
wildcards
]
(* Example usage *)
dnaString = "AACAAAAAAAAAAAAAAAAAAAAGG";
bitEncodedDNA = encodeDNA[dnaString];
decodedDNA = decodeDNA[bitEncodedDNA];
Print["Decoded DNA Sequence: ", decodedDNA];
patternPrefix = "CAA";
patternSuffix = "AG";
wildcardNucleotides = 17;
matches = searchDNA[bitEncodedDNA, patternPrefix, patternSuffix, wildcardNucleotides];
Print["Matches found at nucleotide positions: ", matches];
wildcards = extractWildcards[bitEncodedDNA, matches, StringLength[patternPrefix] * 2, wildcardNucleotides];
Print["Wildcard sequences found: ", wildcards];I would literally go nuts from boredom without a computing device (tablets/phones don't count).(I'm on holiday without a computer to play with)
Well, looks like I'm behind on where regex compilers are today.I'm not sure if it is still true, but 20 years ago compilers for vectorized CPUs performed String searches by vectorized discrete correlation.