The Navy's Bureau of Aeronautics sponsored a Joint Fighter Conference that was held at NAS Patuxent River, Maryland on October 16-23, 1944. Representatives were invited from the Army, Navy, Marine Corps, Royal Air Force, Royal Navy, and Royal Canadian Air Force as well as from NACA and the manufacturers of engines/equipment and fighter aircraft like Grumman.
A large fleet of fighters was assembled for the pilots to evaluate:
General Motors FM-2
Grumman F6F-5, F6F-5N, F7F-1, XF8F-1
Vought F4U-1C, F4U-1D, XF4U-4
Goodyear FG-1*, FG-1A, XF2G-1
Lockheed P-38J, P-38L-5
Republic P-47D, P-47M
North American P-51D
Bell YP-59A, P-63
North American P-51D-5
Fairey Firefly Mk 4
Supermarine Seafire L2C, Seafire 3
Mitsubishi "Zeke 52"
More than one of some types was available on the line.
The manufacturers were eager to participate and put their best foot forward. Some of the airplanes were still in flight test or fresh off the production line. Here, an evaluation pilot poses in one such F4U. (The cut line in the Navy photo names him as Royal Navy Lt. J.P.M. Reid but as Bing Chandler points out in his comment below, facial hair rules this out; I agree that he resembles Commander Paul Ramsey.)
In addition, a Hoover Horizon in an NH (a single-engine, high wing, instrument trainer built by the Howard Aircraft Company) was available for flights. It combined the directional gyro and artificial horizon on one instrument along with a depiction of clouds above the horizon and trees below. Anti-G suits were provided for flights in airplanes with that capability. The F6F-5 was equipped with a lead-computing sight and an SNJ was assigned to be the target.
Each pilot (there were about 60 in all including Charles Lindberg) listed the airplanes that he wanted to fly and whether in daylight or at night. Daily flight schedules were prepared accordingly. What the pilot did during his flight was entirely up to him. The only obligation was to get the airplane back on time and fill out a two-page post-flight questionnaire based on what was looked at.
The only caution with respect to the operating area was to not bounce any airplanes in the Armament Test range south of the Patuxent River since "If pilots should jump an airplane there they might not be seen by the people operating turrets and the possibility exists of running into some complication along that line."
For purposes of identification for this evaluation, each airplane was marked with a number on the mid fuselage behind or below the cockpit beginning with an FM-2 that was 1. (The large number on some cowls in the photo above was the last three digits of the Bureau Number, often applied when an airplane wasn't assigned to an operational squadron.) The call sign was simply "Blue" and the evaluation number.
Here, Commander Paul H. Ramsey, Director of Test at NAS Patuxent River and chairman of the conference is climbing into Blue 24, a P-47 (the M according to the photo caption but a D according to the serial number). Note the gloves in hand, dress shoes, knee board strapped to thigh, and ear phones around the neck in lieu of a helmet. The life vest was recommended because there was a lot of water in the area.
Checkout, even by the standards of the day, could only be described as perfunctory. At the opening session, each contractor was given three to four minutes to describe their aircraft's "special qualities and restrictions, and general information relative to their use, speed, etc." Knee-board cards were provided that listed general data, the engine limits, and in some cases, airspeed limits and other cautions/restrictions. Each airplane was assigned a check-out pilot who stood on the wing and provided advice and counsel to the evaluation pilot until he had started the engine.
Here, McDonnell test pilot Woodward Burke is being given a checkout in the Zeke by Lt. C.C. Andrews, NATC project officer, Tactical Test, on the left with C.L. Sharp, a Chance Vought test pilot, on the left wing.
Report of Joint Fighter Conference: NAS Patuxent River, MD 16-23 Oct. 1944 was published by Shiffer Military History in 1998, with technical editing by Francis H. Dean. It is a compilation of documents and transcripts from the conference. It includes a memo to the BuAer Chief summarizing the results, the transcribed (including errors) remarks of the daily briefings and wide-ranging discussions, a summary of the opinions (qualitative and quantitative) for each type, etc. The Report is a fascinating time capsule of the status of U.S. fighter airplane capability, issues, and development as of October 1944. My favorite is the statement by a test pilot from Hamilton Standard who was answering a question about counter-rotating props: "Commander, after just flying the (YP-59A), I am out of business and I don't think there is any reason why we should talk about dual rotation. If anybody needs a good tug boat captain or something, that is about all I can talk about. I just finished the test in the squirt job and I have about two pages of comments. First I am going to hand in my resignation."
5 September 2016: Ted Dettman created a summary of the rankings (1 is best) from the data in the book:
http://tailspintopics.blogspot.com/2011/06/seahorse.html) but by that time, jet fighters were clearly the future so they bought what was arguably a jet-powered P-51 from North American instead.
A possibly apocryphal (but definitely plausible) anecdote that might explain two Navy airplanes being in the top three was that early on in the production of the F4U and the F6F, the Navy gave Grumman an F4U to evaluate and Vought, an F6F, suggesting more of the better airplane would be bought and maybe the one they were building wasn't it. Certainly the Navy had a practice of contracting for the development of two different airplanes for the same fighter mission, thereby incentivizing its contractors to do their best.