Latvians are not as concerned as Ukrainians primarily because the Latvian ecomony is better than the Russian economy which is better than the Ukrainian economy.
Just from talking with someone I know who's parents immigrated over from Estonia, and still has a lot of family back there, saying that in these countries ethnic Russians are actually treated as second-class citizens, denied social & economic opportunities open to the rest of the population. It's not so cut & dry thinking only corrupt rabble-rousers are the cause of the succession movements. When combined in the Southern nations like Ukraine or Moldova, the economies are so much weaker that the conflicts intensify.
They're still worried about Russian aggression but they don't have the same level of internal problems that enable Russian aggression.
The grand moral is, treat others fairly and fewer conflicts arise.
Well, what keeps them going is NATO and the EU, but that truly brings them no real confidence (memory of Roosevelt abandoning them to Stalin is actually still fresh in their minds).
As I mentioned earlier, my SO is Latvian and all of her family still lives there. I visited in 2008 and hope to go back in ~2 months (Assuming Putin isn't pushing his tanks in).
Even so, the situation in Latvia is a bit different than Estonia. Latvia hasn't charged ahead economically like Estonia or Lithuania over the last several years. And endemic Russian population is quite different--Estonia and Lithuania kicked most of them out in 1991, fired anyone with government of public positions and made everyone re-apply for their jobs; only Latvia offered non-military families to stay. In most "large" towns, particularly Riga and Liepaja, there are still tiny pockets of old soviet neighborhoods. You visit these little ghettos, they haven't been updated since 1957. If you were to chat with these Russians, they still believe the USSR exists and that they are being held in prison by evil Latvians--these are the old guard. Their children tend to be more reasonable, because they sure as shit don't want to go back to Russia. Most consider themselves Russian, but only really think of Latvia as home, and so tend to be more amenable towards learning and using the Latvian language.
Also, the "mistreatment" of Russians within Latvia amounts to nothing more than being forced to learn Latvian, and not allowing dual citizenship. Imagine Spanish speaking immigrants, legal or illegal, demanding Spanish be recognized as an official language in the US, and the illegal ones demanding US passports. Now, imagine that those people were originally moved in as part of an occupying nation, to replace half of the US citizens, including your family and neighbors, perhaps yourself, who were sent off to a labor camp for several decades, or simply murdered.
I would consider such claims of "native citizenship" to be spurious, at best. This is, in fact, what Putin is doing as part of grand Russian/Soviet strategy in defense of "native Russian populations" wrg to Georgian territories, and now Crimea. As long as we forget that forced population replacement is part of that strategy, then I guess annexation seems simple on the surface.