The first cause that makes it difficult for you to speak English is that the environment does not support you to speak English frequently. The environment here means the people around you.
Few people may think that you just want to show off when you speak English for daily conversation, while others may just finger out mistakes in your conversation. This response can make you lose your self-confidence to improve your speaking skills.