Permutation pattern matching (PPM) is the problem of determining whether a permutation π is contained in another permutation τ, i.e. τ has a subsequence that is order-isomorphic to π. We study the complexity of PPM. In particular, we consider the variants of PPM where π itself does not contain a fixed permutation σ. We first investigate the incidence graphs avoiding a fixed permutation σ, and show that the treewidth of these graphs can be almost linear for almost all σ. This indicates that the restricted variants may be as hard as unrestricted PPM. For the counting variant #PPM, we show that, indeed, almost all restricted variants require exponential time, unless the exponential time hypothesis fails. Finally, we consider two on-line versions of PPM where π is fixed, and give upper and lower bounds on their space requirements depending on π.