The enriched neodymium-150 (Nd-150) isotope plays a critical role in nuclear industry and basic scientific research. The Nd isotope separation could be conducted by Atomic Vapor Laser Isotope Separation (AVLIS), where the target isotope is selectively ionized through the λ₁ = 596 nm → λ₂ = 579 nm → λ₃ = 640 nm photoionization scheme, and non-target isotopes remain neutral due to the frequency-detuned excitation; subsequently an external electric field is applied to extract the ions in the laser-produced plasma. The Nd-150 abundance in the product cannot meet the requirement of the application, attributed to the nearly negligible isotope shift of the λ₂ = 579 nm transition, which causes the excess ionization of non-target isotopes. A new high-selectivity photoionization scheme is desirable to address this limitation, and its expected parameter values could be determined through numerical calculations prior to the time-consuming atomic spectroscopy experiment. In this study, a three-step selective photoionization model is established based on the density matrix theory, with the consideration of the hyperfine structures and magnetic sublevels. This model allows flexible adjustments of atomic parameters (e.g., branching ratios, isotope shifts, hyperfine constants) and laser parameters (e.g., frequency, power density, bandwidth, polarization), while the ionization probabilities of magnetic sublevel transitions could be quantitatively predicted. For the existing scheme, the branching ratios are determined by the comparison between literature data and numerical results, and the Nd-150 abundances under different laser bandwidths are evaluated. Further, we numerically explore an alternative scheme under the assumption that the first transition remains unchanged and the second transition has a more significant isotope shift and a less branching ratio, and the Nd-150 abundances under different combinations of isotope shifts, hyperfine structures, and laser bandwidths are evaluated with all the natural Nd isotopes included. From the numerical results, a scheme with the angular momentum of the second excited state J₃ = 6, the isotope shift between Nd-148 and Nd-150 IS_23,148 ≥ 300 MHz, and a lower reduced dipole matrix element of the second transition reaching approximately 30% of that of λ₂ = 579 nm, could produce the high-abundance Nd-150 (>95%, equivalent to that of the electromagnetic separation method) under the bandwidth b₁₂ ≤ 0.5 GHz, b₂₃ ≤ 1.0 GHz, and parallel linear-polarized lasers. Higher abundance, superior to the electromagnetic separation method, could be achieved with the narrower-bandwidth lasers. The expected high-abundance Nd-150 could be attributed to the combined effects of multi-factors: the larger isotope shift between Nd-150 and Nd-148 compared with that between other isotope pairs, the unsignificant hyperfine splitting of odd isotopes, as well as the match between narrow-bandwidth lasers and Nd I spectroscopic parameters. These parameter values could serve as helpful benchmarks for experimental parameter selection in the forthcoming high-precision spectroscopy experiments.