The only way to improve is speaking English as much as you can, there's really no way around it.
Find a buddy you can have conversations with; the preferred option, of course, is picking someone who already speaks English well (if you know a native speaker who could help you out, that would be awesome) but also someone who is about at your level and is willing to learn is fine.
If this is not possible for any reasons, you can always practise it by yourself. Who doesn't have conversations by themselves? Just change the language. Stand in front of a mirror and start speaking about a topic you know something about, be it what you did today, or mathematics and physics. Don't be shy; if you are self-conscious and don't want anyone to listen, you can always do it while taking a shower.
Nowadays, though, finding a partner is easy: just do it online. There are several sites such as
italki which you may find suitable to your needs.
The key thing to remember is, you are going to make mistakes. Just don't care. You say that when you manage to speak your speech is broken and rough, but the important thing is to be understood. Give it a few months and it will be much better.