When you are speaking to anyone, keep in mind that they do to you what you do to other speakers: evaluate the speaker's ability to tell you things you can trust. What we do is evaluate the credibility of speakers/sources constantly. We listen to some with great interest and ignore others. In fact, we fear recalling too much of what some sources tell us because we don't want to be embarrassed by inaccurate or outright false information.
Be careful what you say when talking. Avoid anything that will damage your credibility. Self deprecation can be a useful tool, but don't let it affect the content of what you are saying. All listeners want to learn everything they can quickly and easily , especially in the areas of their personal interest. Then, they want to feel comfortable passing on that information to others. That way they appear to be brighter and more intelligent than they really are. If the information is faulty or incorrect, they appear to be gullible or less intelligent then they really are.
The solution to the problem is two fold: 1) develop habits of regular and constant data gathering and 2) do nothing to cause your listeners to question your credibility. That way when you speak in any size group, folk will listen, because they will learn something important to them quickly and easily. You are providing a major benefit to them through your ability to learn, organize and delivery information that they need/want and will use.