The crisis in many state-pension funds is by now familiar: Unfunded obligations are at $1 trillion or more, according to the Pew Charitable Trusts. But there is a lesser-known aspect of the problem. Many boards of government pension funds, often composed of politicians and political appointees, have paid more than necessary to manage the $2.5 trillion of assets in these funds.

Consider the Illinois State Board of Investment. Over the past three years, it has paid hedge-fund managers more than $180 million in fees. Yet, even excluding compensation, the performance generated by these managers was worse than that of a balanced index fund. Such funds invest broadly in equities and bonds that simply track the market’s performance, while the current system adopts complexity at high costs.

Illinois isn’t alone. Thirty-three state pension systems spent $6 billion on asset-management fees in 2014, according to a 2015 study by the Maryland Public Policy Institute. The 10 states that spent the most on fees—including New Jersey, Maryland, South Carolina and Missouri—achieved a rate of return no higher than the states that spent the least. Over a five-year period, according to the study, hedge-fund managers employed by pension systems significantly underperformed both the S&P 500 and the Vanguard Balanced U.S. Fund, which invests in stocks and bonds.

Sometimes the problem isn’t high fees but the opportunity for political mischief. Federico Buenrostro Jr., the former chief executive of Calpers, California’s public-pension fund, admitted in 2014 to accepting bribes, a felony. Prosecutors said he collected money, casino chips and other gifts from a middleman who connected Wall Street investment firms with the fund.

In New York, State Comptroller Alan Hevesi accepted $1 million in gifts from a money manager in return for steering him $250 million in state-pension-fund money to invest. Mr. Hevesi spent more than a year in prison after pleading guilty to corruption charges in 2011.

The Illinois State Board of Investment, which oversees $16 billion of pension assets, is moving to maximize the value it gets from fees paid to investment firms, while minimizing the potential for political meddling. A majority of the board was replaced last year by Gov. Bruce Rauner, and several of the new trustees have deep investment experience. (I was appointed to the board by the governor and subsequently elected chairman.)

In addition to slashing our hedge-fund portfolio by 70%, our board has taken several significant actions, including terminating its long-standing investment consultant and replacing about 40% of our high-cost, underperforming investment managers with index-based portfolios. Once the board’s actions are fully implemented, about 70% of the money previously invested in stocks and bonds will be in funds that employ indexing, with a typical fee of less than 0.1% of assets.

By moving to index funds, a pension system may miss out on investment managers who beat the market. But it is extremely difficult to identify these managers in advance. No more than 23% of active managers in any segment of the U.S. equities market outperformed the S&P 500 over the past five years, according to the S&P Dow Jones SPIVA U.S. Scorecard. Moreover, research by Vanguard shows that a manager’s ability to outperform the market in the past doesn’t predict whether he will do so in the future.

Some investment firms deal in specialized assets that require higher fees. Firms that manage real-estate holdings or invest in private companies, for instance, sometimes create unique assets that generate returns significantly above an index fund. Illinois’s investment board is working hard to understand whether these types of firms add value for the fees they charge.

The changes adopted by the board have generated controversy. On cue, supporters of the old system engineered legislative hearings into the terminations of some former investment managers. They also complained through Chicago media that the new trustees were playing politics. But the entire board came together to support the changes because they are easily understood and will save taxpayers hundreds of millions of dollars in the coming years. The state’s portfolio is now much simpler and more transparent. The board’s general consultant, Meketa Investment Group, projects that the new model will provide higher returns and lower risks.

Investing is hard enough outside the political spotlight. Hitting the 7.7% return, which is now the average target for state and local pension funds, is a herculean task. Eliminating unnecessary costs and limiting the incentive of political interference through use of index funds is a step in the right direction.

Mr. Levine is the chairman of the Illinois State Board of Investment.