Restriction varieties and geometric branching rules

This paper develops a new method for studying the cohomology of orthogonal flag varieties. Restriction varieties are subvarieties of orthogonal flag varieties defined by rank conditions with respect to (not necessarily isotropic) flags. They interpolate between Schubert varieties in orthogonal flag varieties and the restrictions of general Schubert varieties in ordinary flag varieties. We give a positive, geometric rule for calculating their cohomology classes, obtaining a branching rule for Schubert calculus for the inclusion of the orthogonal flag varieties in Type-A flag varieties. Our rule, in addition to being an essential step in finding a Littlewood-Richardson rule, has applications to computing the moment polytopes of the inclusion of SO(n) in SU(n), the asymptotic of the restrictions of representations of SL(n) to SO(n) and the classes of the moduli spaces of rank two vector bundles with fixed odd determinant on hyperelliptic curves. Furthermore, for odd orthogonal flag varieties, we obtain an algorithm for expressing a Schubert cycle in terms of restrictions of Schubert cycles of Type-A flag varieties, thereby giving a geometric (though not positive) algorithm for multiplying any two Schubert cycles.