Compressed sensing is a revolutionary signal processing technique, which allows the signals of interest to be acquired at a sub-Nyquist rate, meanwhile still permitting the signals from highly incomplete measurements to be reconstructed perfectly. As is well known, the construction of sensing matrix is one of the key technologies to promote compressed sensing from theory to application. Because the Toeplitz sensing matrix can support fast algorithm and corresponds to discrete convolution operation, it has essential research significance. However, the conventional random Toeplitz sensing matrix, due to the uncertainty of its elements, is subject to many limitations in practical applications, such as high memory consumption and difficulty of hardware implementation. To avoid these limitations, we propose a bipolar Toeplitz block-based chaotic sensing matrix (Bi-TpCM) by combining the intrinsic advantages of Toeplitz matrix and bipolar chaotic sequence. Firstly, the generation of bipolar chaotic sequence is introduced and its statistical characteristics are analyzed, showing that the generated bipolar chaotic sequence is an independent and identically distributed Rademacher sequence, which makes it possible to construct the sensing matrix. Secondly, the proposed Bi-TpCM is constructed, and it is proved that Bi-TpCM has almost optimal theoretical guarantees in terms of the coherence, and also satisfies the restricted isometry condition. Finally, the measurement performances on one-dimensional signals and images by using the proposed Bi-TpCM are investigated and compared with those of its counterparts, including random matrix, random Toeplitz matrix, real-valued chaotic matrix, and chaotic circulant sensing matrix. The results show that Bi-TpCM not only has better performance for these testing signals, but also possesses considerable advantages in terms of the memory cost, computational complexity, and hardware realization. In particular, the proposed Bi-TpCM is extremely suitable for the compressed sensing measurement of linear time-invariant (LTI) systems with multiple inputs and single output, such as the joint parameter and time-delay estimation for finite impulse response. Moreover, the construction framework of the proposed Bi-TpCM can be extended to different chaotic systems, such as Logistic or Cat chaotic systems, and it is also possible for the proposed Bi-TpCM to derive the Hankel blocks, additional stacking of blocks, partial circulant blocks sensing matrices. With these block-based sensing architectures, we can more easily implement compressed sensing for various compressed measurement problems of LTI systems.