Setting mem config is changed in the new WLS 10.3.1. and in the embedded WLS of JDevelper 126.96.36.199.1.0 (actually this is the same thing)
In the previous ver it was setted like this.
Now the only thing you need to do is to the add the following line in setDomainEnv.cmd file right after the comments in the begging:
The file is located in a different folder that the previous release of JDev:
set USER_MEM_ARGS=-Xms64m -Xmx1024m -XX:MaxPermSize=512m
(of course the above is one line!!)
C:\Documents and Settings\spido\Application Data\JDeveloper\system188.8.131.52.33.54.07\DefaultDomain\bin
that is windows user app data folder and of course instead of spido put your username.