Facing capital outflows and a shrinking trade surplus, China is opening the door wider to foreign institutional investors—but their surging government-bond purchases create risks for both them and Beijing, which has long kept a tight hold on this $2 trillion market.