Irving Picard's 'clawback' suit is seeking $3.54 billion withdrawn by the Connecticut-based fund manager before Mr Madoff's fraud unraveled last December.
Mr Picard contests that since 1995, the Fairfield funds invested approximately $4.5 billion with Bernard L Madoff Investment Securities (BLMIS).
The lawsuit states that the defendants either knew or should have known that BLMIS was engaged in a fraud when they began receiving "unrealistically high and consistent annual returns" from 1996 onwards.
In the six years prior to Mr Madoff's arrest, the hedge funds withdrew $3.2 billion from BLMIS accounts, including $1.2 billion that was taken out in the final three months before the broker's fraud was uncovered.
Fairfield Greenwich is already facing civil charges in Massachusetts linked to the Madoff fraud. The firm is contesting allegations that it misled investors about its due diligence on BLMIS investments.
Last week, Mr Picard said he had so far received 8,848 claims for losses from former Madoff clients.