Got a crazyflie and assembled it (i think the right way

Now i have bought a xbox360 controller and trying out some flight fun.
The problem i have is to get a stable flight (or this is intended and i need to practice more).
From what i understand it should be possible to flight the crazyflie and hold it stable in the air (straight up).
When i thrust my crazyflie goes up, but into a direction (cannot get it to stable flight, straight up).
So what can be the problem (center of mass?, propellor balancing) ?
Or is it related to the controller..?
Any ideas welcome!
Best regards,
Martijn