How Large Is the Java Standard Library Comparatively

[chrstrbrts]

Hello,

Java is the only language that I know. I've got the syntax down well enough, but I'm intimidated by the standard Java library; it's huge. I realize that it's futile to even attempt to memorize all of the packages, classes, etc.

Is this typical of high level languages? Do Python and the others have massive libraries like Java?

Where does Java rank size wise?

Thanks.

 

[Markbnj]

Yes, it's typical these days, and it's a good thing. Those huge libraries and frameworks represent a ton of code you don't have to write before you can get professional results on a project.

The Java standard library is probably roughly comparable to the .NET framework on the Microsoft stack. Python is a bit smaller in terms of the "standard library" but there are many other packages available for it, and so the overall size is probably in the same ballpark.

You don't need to memorize standard libraries and frameworks, but you need to know how they are put together, which pieces offer which classes, and you need to be good at accessing reference materials to figure out the specifics when you need to implement something. Nobody keeps it all in their heads anymore.

 

[chrstrbrts]

I'm getting depressed. I feel so far removed from the hardware.........

 

[Cogman]

If you wanted to be close to the hardware, then Java wasn't a good choice for a language. If you really want to get mired in what is close to the machine, then you should dump java and start working with C or C++.

We talk about languages in terms of "levels" How far away is the language from the hardware. In general, it is considered to be a good thing that a language is "High level" because being concerned with how the machine does things is a sure fire way to start down a nasty maintenance path. Java and C# are pretty high level languages, getting depressed because they are far removed from what the actual machine is doing is a bit silly. They are removed from the machine because they were specifically designed to be removed from the machine.

 

[chrstrbrts]

If you wanted to be close to the hardware, then Java wasn't a good choice for a language.

Tell me about it.

If you really want to get mired in what is close to the machine, then you should dump java and start working with C or C++.

As upset as I am at being so far removed from the metal, the idea of stopping something 70% of the way through upsets me even more.

I have the syntax for Java down (it's not that hard), but the size of the standard library is getting to me.

Also, the compiler offers far more than just the javac and java commands; I know practically nothing of what it really offers.

I'm sticking with Java until I have the lang, util, io, net, awt, swing, and javafx packages down pat. It'll probably take me 2 or 3 more months before I move on. But, when I do, I'm definitely moving on to C.

....getting depressed because they are far removed from what the actual machine is doing is a bit silly.

I'm a little dramatic at times. I just like the idea of being as close to the metal as possible.

 

[Markbnj]

As upset as I am at being so far removed from the metal, the idea of stopping something 70% of the way through upsets me even more.

It doesn't make sense to be upset about this, and that's not to say it isn't an interesting topic. Based on your posts here what you should be learning is how programs work, how they are structured, etc. Java will teach you that. While there is no longer much reason for a working developer to get "close to the metal" (in most jobs) I am not one of those who thinks it's a waste of time, either. Anything you learn about how the machine works will stand you in good stead later. But there's no reason to disdain Java at this point in the learning curve.

 

[Merad]

If you want to be "close to the metal" these days your options are pretty much embedded systems, OS/kernel development, or drivers. If you really think you want to do that, buy an Arduino, download Atmel Studio and the appropriate datasheets, and have at it.

Edit: Also, for the most part, it doesn't matter what kind of programming you do, you'll be using a large standard library, a large number of in-house or 3rd party libraries, or some combination of the two. It's the nature of the beast.

 

[brianmanahan]

I'm getting depressed. I feel so far removed from the hardware.........

eh, hardware sucks. abstraction is nice.

 

[beginner99]

In general, it is considered to be a good thing that a language is "High level" because being concerned with how the machine does things is a sure fire way to start down a nasty maintenance path.

Exactly. And thinking about being far away from hardware has a smell of premature optimization. Java is often just as fast as C/C++ (while using more memory). For very critical stuff you can always use native code together with Java.

 

[LevelSea]

eh, hardware sucks. abstraction is nice.

Embedded 4 lyfe yo

 

[Cogman]

Exactly. And thinking about being far away from hardware has a smell of premature optimization. Java is often just as fast as C/C++ (while using more memory). For very critical stuff you can always use native code together with Java.

We have stuff that is pretty performance critical at my job and we use only java. I haven't found a case yet where performance was so bad that we simply needed to drop down into native calls. In every performance case I've dealt with, the solution has been "use a better algorithm or compute less".

But yeah, java is a memory hog, however every year that becomes less and less of a problem. Memory is getting cheap.

 

[Cogman]

It doesn't make sense to be upset about this, and that's not to say it isn't an interesting topic. Based on your posts here what you should be learning is how programs work, how they are structured, etc. Java will teach you that. While there is no longer much reason for a working developer to get "close to the metal" (in most jobs) I am not one of those who thinks it's a waste of time, either. Anything you learn about how the machine works will stand you in good stead later. But there's no reason to disdain Java at this point in the learning curve.

Certainly, I agree. I think it is worthwhile for every programmer to have some basic computer architecture understandings. But that knowledge isn't the most important thing for a programmer to know, it is just useful to have. The most useful thing to understand IMO is what a CPU cache is and roughly how it works. Beyond that, most other hardware thingies don't really matter all that much.

 

[wolf_squad]

learning the library just for the sake of it seems like a waste to me. learning how to do what you want to do using the library or finding the best library for your needs and getting up to speed quickly, now that's a different matter.

 

[Cogman]

learning the library just for the sake of it seems like a waste to me. learning how to do what you want to do using the library or finding the best library for your needs and getting up to speed quickly, now that's a different matter.

This I agree with as well. A good portion of my job is looking at various libraries for a problem and evaluating the pros/cons of using them.

 

[sourceninja]

Yes, it's typical these days, and it's a good thing. Those huge libraries and frameworks represent a ton of code you don't have to write before you can get professional results on a project.

The Java standard library is probably roughly comparable to the .NET framework on the Microsoft stack. Python is a bit smaller in terms of the "standard library" but there are many other packages available for it, and so the overall size is probably in the same ballpark.

You don't need to memorize standard libraries and frameworks, but you need to know how they are put together, which pieces offer which classes, and you need to be good at accessing reference materials to figure out the specifics when you need to implement something. Nobody keeps it all in their heads anymore.

Someone needs to tell some hiring managers I've dealt with that.

 

[IHateMyJob2004]

I'd say that jvisualvm.exe (in the jdk bin folder) is something you should look at. Once you play with it you will learn to love it. Helps with deadlocks in multithreaded apps.

 

[Markbnj]

Someone needs to tell some hiring managers I've dealt with that.

I hear you. That's just a lack of knowledge and creativity on their part, imo.

 

[chrstrbrts]

learning the library just for the sake of it seems like a waste to me.

I know that you are much more experienced than me, but I disagree with you.

In Java at least there are some packages / classes that are just so crucial, you should know them in and out.

The .util package, the main package .lang, .io, .net these packages contain pre-written code that covers very basic aspects of what computers do. It seems that you should know them well.

 

[Cogman]

I know that you are much more experienced than me, but I disagree with you.

In Java at least there are some packages / classes that are just so crucial, you should know them in and out.

The .util package, the main package .lang, .io, .net these packages contain pre-written code that covers very basic aspects of what computers do. It seems that you should know them well.

There is also a lot of crap in the utils, lang, io, and net packages that nobody should learn or use. For example, Vector, Enumeration, Date.

You are much better off learning algorithms and data structures devoid of language and then looking for implementations of the data structure or algorithm in the language of your choice.

It is like Guava for java. There is a ton of useful things and features in Guava, but nobody needs to memorize it all. Rather, it is more useful to know the sorts of things provided and then look deeper into them as need arrises. Spending a lot of time just learning everything a package can do is a waste of time because most packages do way more than you'll ever need.

 

[Spungo]

I'm getting depressed. I feel so far removed from the hardware.

Dealing with hardware seriously sucks. It takes 10x as long to get anything done and there's about a 99% chance your code will suck. It's the difference between driving a car and trying to build your own car. A car made by me would suck. Libraries written by me would suck. Some List.Sort() written by me would be extremely slow, and I wouldn't even be sure that it works properly. It's better if someone else does that and we all agree to use that library.
You should try to prove me wrong. Try working with a bare bones language. If you like it, that's fantastic. If you find that it sucks, you'll have a better appreciation of the hand holding Java does, and that's a good outcome as well.

The most useful thing to understand IMO is what a CPU cache is and roughly how it works.

You'll learn it when it becomes a problem.
What kind of problems does one run into if they don't understand CPU cache?

 

[Cogman]

You'll learn it when it becomes a problem.
What kind of problems does one run into if they don't understand CPU cache?

It is useful when discussing the performance characteristics of various algorithms. For example, In java, you'll often see people recommend that LinkedList should be used everywhere. Why do they recommend this? Because they (mistakenly) believe that the O(1) insertion time is ideal (after all, O(1) is really fast, right?) On top of that, insertion into an ArrayList will possibly mean that everything gets copied, so, why would you use that? All of the other performance characteristics are roughly the same right?

Well, wrong, ArrayLists are almost always faster than LinkedLists, even in cases of end insertion and middle insertion. In fact, it takes a pretty big list of elements to get to the point where LinkedList performance overtakes ArrayList performance (and that number grows every day. Right now on our local systems it is in the millions). These differences are almost completely due to the nature of CPU caching.

But it goes deeper than that, (especially in java). The classic example is this, you want to store an array of Points (with an x, y, and z component) what is the fastest way to do this?

Well, currently, it is this ugly beast

int[] xs; int[] ys; int[] zs;

and not this

class Point{ int x; int y; int z };  Point[] points

Why is that? Because in java all objects introduce a second level of indirection which can result in the memory being accessed in unpredictable ways (which kills cache performance).

This problem, in particular, is the whole reason Java 9 (or 10?) is thinking about adding value types. And interestingly enough, some of the java garbage collectors do locality optimizations and memory defragmentation just for this reason (though, it is somewhat lacking).

So where would you need this? Well, you'll know when you need it if you are profiling down and trying to improve the performance somewhere. But just knowing about it might change the way you structure some code to be more cache friendly. Cache friendly stuff isn't often that much more complex than cache hating code.

 

[Spungo]

It is useful when discussing the performance characteristics of various algorithms. For example, In java, you'll often see people recommend that LinkedList should be used everywhere. Why do they recommend this? Because they (mistakenly) believe that the O(1) insertion time is ideal (after all, O(1) is really fast, right?) On top of that, insertion into an ArrayList will possibly mean that everything gets copied, so, why would you use that? All of the other performance characteristics are roughly the same right?

Well, wrong, ArrayLists are almost always faster than LinkedLists, even in cases of end insertion and middle insertion. In fact, it takes a pretty big list of elements to get to the point where LinkedList performance overtakes ArrayList performance (and that number grows every day. Right now on our local systems it is in the millions). These differences are almost completely due to the nature of CPU caching.

Interesting. This is probably similar to the reason I never change words that I've typed incorrectly. If I type a word like "kangaroo" but it comes out "kanagroo", it actually takes less time to erase the entire word and type it new rather than figure out how the letters need to shuffle around to become correct. This is also sometimes true when typing a sentence. It's often faster to erase an entire sentence and type it out again rather than remove a word here, add a word there, move a word, make a word plural, etc. It intuitively seems like it would be faster to only fix the mistakes, but it's not.

 

[Sureshot324]

Memorizing the entire util, io, lang, and net packages is not a good use of your time. The best way to learn is to do a project and learn the standard libraries as you need them.

Coding close to the metal doesn't make sense these days for most projects. If your goal is to get a job, most software development jobs involve CRUD apps (create, read, update, delete from database with a UI, usually a web app). There are already standard libraries that handle all the hardware stuff, so you'd be reinventing the wheel writing the native hardware code yourself.

If you really want to code close to the hardware, than as others have said get into embedded development, or possibly game development, though even that is getting more abstracted.

 

[Cogman]

Interesting. This is probably similar to the reason I never change words that I've typed incorrectly. If I type a word like "kangaroo" but it comes out "kanagroo", it actually takes less time to erase the entire word and type it new rather than figure out how the letters need to shuffle around to become correct. This is also sometimes true when typing a sentence. It's often faster to erase an entire sentence and type it out again rather than remove a word here, add a word there, move a word, make a word plural, etc. It intuitively seems like it would be faster to only fix the mistakes, but it's not.

It boils down to the fact that there is a high amount of latency between the CPU and everything that isn't the CPU. To get any piece of information, the CPU has to wait for thousands of cycles. Because of this high latency and the fact that very often memory is accessed in a localized fashion (you usually aren't jumping around everywhere, for example, you often access the stack which is generally all within the same location) when the CPU requests a bit of memory it doesn't just take the specific address asked for, but rather it loads up all of the memory in the same locality.

So, in example, when you have an array foo and you access foo[0]. If foo[0] isn't already cached, then it will be loaded, but not just foo[0], foo[1], foo[2], and foo[3] will also be loaded and if you access foo[1] and then foo[2] the CPU will preemptively start to load up foo[4], foo[5], and foo[6].

It should be noted that the logic behind the preemptive loading in CPU's is really smart. Smart enough to detect patterns like loading foo[4], foo[3], and then foo[2], or foo[0], foo[2], foo[4]. I believe it even does well with binary search patterns. The CPU engineers have invested a lot of time trying to make this thing as smart as possible so that as long as you are using some common memory access pattern, you will get pretty decent performance.

Unfortunately, the likes of java make that pretty hard to do well. With every object being a reference to potentially random memory locations, the CPU struggles to find meaningful patterns to save you from the large memory load waits (which, again, is why the JVM garbage collectors practice moving object references into the same memory locality and why value types are being added).

Bjarne Stroustrup has a pretty good explanation of this.

 

[Merad]

What kind of problems does one run into if they don't understand CPU cache?

Cache loads data in chunks, the size of which depends on the line size of the cache. For Core i7, the line size is 64 bytes. So assuming that we're dealing with 4 byte integers, in an ideal pattern of sequential data access, one L1 cache miss should be followed by 15 L1 hits. However if your memory accesses jump around so that accesses are more than 64 bytes apart, you could at worst have an L1 miss for every single access. Actually, it could get even worse if the data you're working on is too large to fit entirely into L2 cache, and you end up causing the L2 to thrash as well.

For reference, the access times to the data hierarchy on Core i7 are (approximately, this is from memory):
L1 - 4 cycles
L2 - 10 cycles
L3 - 40 cycles
Main memory - 100+ cycles

tl;dr - Two pieces of code can do exactly the same thing, but one might be hundreds or even thousands of times slower depending on their data access patterns.

It boils down to the fact that there is a high amount of latency between the CPU and everything that isn't the CPU. To get any piece of information, the CPU has to wait for thousands of cycles.

It's not quite that slow. IIRC even with a TLB miss main memory access is like 150-200 cycles. The only time you should have latencies in the range of thousands of cycles is on a page fault.