I don't think he considered it a "close" but rather a passing of the torch in the Civil Rights Movement to young leaders like Martin Luther King.

Rustin had been involved in an earlier planned March on Washington, which never occurred. In 1941, Rustin, along with C.L. Dellums, A. Philip Randolph, and A.J. Muste, started organizing a giant March on Washington to press their demands to prohibit discrimination in the defense industry and desegregate the armed forces. They backed down only when FDR signed Executive Orders prohibiting segregation and employment discrimination under federal contracts in the defense industry and establishing a “Fair Employment Practice Committee” to enforce this ban on discrimination.

Rustin was again involved in organizing the 1963 March on Washington, again with Randolph and Muste. It was King, though, who captured the hearts of the crowd. Thus, the torch was passed to a new generation of activists.

As you note, this was not the end of Rustin's activism. He just moved on to a different role. Thank you for shining a light on an important, but not well-known, leader of the Civil Rights Movement.