The goal of this work is to develop a deep-learning-based 3D hand pose tracking system, which can efficiently and robustly detect the hand from the raw depth image before estimating the 3D hand pose. It mainly includes three parts, as the hand detector, the hand verifier and the pose estimator. We evaluated our system on the tracking task of Hands In the Million (HIM2017) challenge benchmark and placed second. In addition, we also applied our modified tracking system on the object-interactive task of HIM2017 challenge benchmark and placed first.