Setting mem config is changed in the new WLS 10.3.1. and in the embedded WLS of JDevelper 11.1.1.1.1.0 (actually this is the same thing)
In the previous ver it was setted like this.
Now the only thing you need to do is to the add the following line in setDomainEnv.cmd file right after the comments in the begging:
set USER_MEM_ARGS=-Xms64m -Xmx1024m -XX:MaxPermSize=512m
(of course the above is one line!!)
C:\Documents and Settings\spido\Application Data\JDeveloper\system11.1.1.1.33.54.07\DefaultDomain\bin
that is windows user app data folder and of course instead of spido put your username.
Enjoy