The maximum size of a Prism Package (.pwc) is 4.3 GB. The maximum size of a Prism SIF (self-installing files, .exe) is 1 GB, which is an operating system size limit for executables.

Does your installation allow you to only install parts of the application? If so, you may be able to install part of it, for example all the registry settings and the core files (up to the 4 GB size limit). You could then take another baseline picture and run the installation again to modify it and more components of the application. In this way you could potentially capture the full installation within several pacakges that are all within the size limit.