Wave-induced instability of untrenched pipeline on sandy seabed is a 'wave-soil-pipeline' coupling dynamic problem. To explore the mechanism of the pipeline instability, the hydrodynamic loading with U-shaped oscillatory flow tunnel is adopted, which is quite different from the previous experiment system. Based on dimensional analysis, the critical conditions for pipeline instability are investigated by altering pipeline submerged weight, diameter, soil parameters, etc. Based on the experimental results, different linear relationships between Froude number (Fr) and non-dimensional pipeline weight (G) are obtained for two constraint conditions. Moreover, the effects of loading history on the pipeline stability are also studied. Unlike previous experiments, sand scouring during the process of pipe's losing stability is detected in the present experiments. In addition, the experiment results are compared with the previous experiments, based on Wake II model for the calculation of wave-induced forces upon pipeline. It shows that the results of two kinds of experiments are comparable, but the present experiments provide better physical insight of the wave-soil-pipeline coupling effects.