You can let anything go wild. The main problem you run into with wisteria is that it sends out so many suckers from the base of the plant, which, if not removed, will get to be about 20 feet long in a couple of months. (At least that's what it seems like.) Mowing around the wisteria would probably prevent this.
If there is nothing else around for the wisteria to climb on, it will will climb the walnut in a general kind of way. But it will also probably produce many branches that go all different directions and flop over onto the ground, and then new growth will grow on the flopped over branches. You would probably wind up pruning a lot of the lower growth.