The tea leaves inside the tea bags are compostable.
However, most tea bags are not compostable (yet). If a tea bag has any form of plastic in its material, it is not compostable and only its contents should be put for council or home composting, not the tea bag itself, otherwise it could release microplastics into the compost. The plastic could be polypropylene or nylon in the bag, or the bag could be sealed using glue which is also plastic.
Some manufacturers have come up with biodegradable materials for tea bags. Some of these materials (like cotton) are compostable at high temperatures in industrial facilities, but may take months to years to break down in the home compost. Some (like Nature-flex) are home compostable within weeks. Some (like Soilon, polylactic acid) are recommended for only industrial facilities as the temperature in the home compost will not be high enough to break them down. It is useful to note that UK councils currently have almost no facilities to decompose these materials, so it is worth questioning your council how they distinguish between & deal with different materials of tea bags. See here for more details.
This link provides some insights into which companies have tea bags without plastic.
After doing this research, I have switched to loose leaf tea.. :)