Unless you are leaning toward vegetarianism for your health only, then it should matter to you that you still eat fish. The almost magical abundance of the oceans is being turned into a desert because we humans multiply like rats and eat everything, just as rats do. And like rats, we foul our nests and destroy our food sources.
Eating any animal is the problem, not just eating beef and pork and chicken. Eating the by-products of animals is the problem too. Even the free-range egg industry kills the male chicks, usually by grinding them up alive. The milk and cheese industry takes bawling baby calves away from their crying mothers and kills some (the lucky ones) and the rest are put into tiny veal crates to soften their little bones and drain their little muscles of nourishment until they are pale and tender for the witless monsters who lick their lips over their Veal Parmigiana in posh restaurants.