X-ray communication, which was first introduced by Keith Gendreau in 2007, is potential to compete with conventional communication methods, such as microware and laser communication, against space surroundings. Researchers have spent much time and effort on the mission making the initial idea into reality in recent years. Eventually, the X-ray communication demonstration system based on the grid-controlled X-ray source and single-photon detection technique can deliver both audio and video information in a 6-meter vacuum tunnel, and the bit-error-rate performance of the communication system is analyzed. But it is difficult to implement applications in industries. The point is to find a way to reduce the signal divergence geometrical attenuation and increase the distance of the communication which can be regarded as an important foundation of future deep-space X-ray communication applications. Therefore, it is urgent to study the X-ray communication system. By using a nested X-ray focusing optics as transmitting and receiving antennas of the communication system, the signal gain and the distance of X-ray communication can be greatly improved. Specifically, the nested X-ray focusing optics is similar to the Wolter type I telescope, which is widely used in the field of X-ray astronomy. The difference between them is that the Wolter type I optics is originally proposed based on a paraboloid mirror and a hyperboloid mirror, but X-ray focusing optics, the simplified Wolter type I optics, provides a single reflection by a conical approximation mirror, and it is more suitable for X-ray communication. In this paper, aiming at the future demand of X-ray communication, the optimization and analysis of the nested X-ray focusing optics are carried out, and the recurrence relations between the layers of mirrors are derived. Reasonable initial structural parameters and structure of the optics are designed. In addition, the theoretical effective collection area is calculated. Feasibility of using the X-ray focusing optics as transmitting and receiving antennas is analyzed, and the theory and structural design of the X-ray focusing optical are discussed. Signal divergence of transmitting antenna, effective area of receiving antenna, the focal spot size, and the signal gain properties are preliminary studied. The results show that the signal divergence is about 3 mrad, and the transmit gain is 23 dB; the effective area of receiving antenna is 5700 mm2 at 1.5 keV. Moreover, the focal spot diameter and the receive gain are 4.5 mm and 25 dB, respectively, and the total gain of this communication system can reach up to 48 dB.