In cognitive radio network (CRN), TCP end to end throughput is one of the key issues to measure its performance. However, most of existing research efforts devoted to TCP performance improvement have two weaknesses as follows. First, most of them only consider the underlying parameters to optimize the physical performance, but the TCP performance is neglected. Second, they are largely formulated as a Markov decision process (MDP), which requires a complete knowledge of network and cannot be directly applied to CRNs. To solve the above problems, a Q-BMDP algorithm is proposed in this paper. Each user in CRN combines modulation type and transmitting power at the physical layer, access channels at the media access control layer and TCP congestion control factor to maximize the TCP throughput. Due to the existence of perception error of environment, this issue is formulated as a partial observable Markov decision process (POMDP) which is then converted to belief state MDP, with Q-value iteration to find the approximately optimal strategy. Simulation and analysis results show that the proposed algorithm can be approximately converged to optimal strategy under a maximum error limit, and can effectively improve TCP throughput in a dynamic wireless network under the premise of the limited power consumption.