The radiation from a rectangular waveguide with an infinite impedance flange into the homogeneous isotropic magnetodielectric is rigorously studied by using the variation method of the multimode approximation. The fields in the waveguide are expanded in terms of the waveguide modes and in the half-space - as discontinuous integrals on the plane wave. Continuity of the tangential aperture fields yields integral equation for considered problem. The integral equation caused matrix equations for the magnitudes of mode waves. Numerical results are presented for the reflection coefficient. To verify the validity of presented method, the results are compared with other methods.