Method of numerical solving optimization problem of envelope shape search of spectral-effective AM/PSK modulated signals under condition of energy and peak factor limitations is considered. Estimation of RMS deviation of envelop shape forms of signals under condition of function approximation with finite Fourier series are given. If is shown, that min and max values of peak factor exist in case of fixed number of Fourier series terms. This values can be reached at fixed speed of out-of-band emission recession.