Fixing the problem

Steps followed

1. Check what Visual studio Matlab was compiled with. Browsed to C:\Program Files\MATLAB\R2011a\bin\win64, I see boost dlls with vc90 in their names, so I am guessing Visual studio 2008 AKA VC9 was used. Just to be absolutely sure, I used Dependency Walker and opened libmex.dll in it. It shows VC9 runtimes (msvc*90.dll) in dependency as well.
2. Get OpenCV compiled with same Visual Studio Version. I already have  this located at C:\OpenCV2.3\build\x64\vc9.
3. Configured Matlab mex compiler to use VC9. Done.
4. Compile test.cpp with mex and run it. 

• Debug mode first: mex -g test.cpp -IC:\OpenCV2.3\build\include\  -lopencv_core230d  -LC:\OpenCV2.3\build\x64\vc9\lib
• The compiling/linking is is ok.
• Tried running the “test”.

• ??? Unexpected Standard exception from MEX file.
  What() is:c:\Users\vp\work\ocv\opencv\modules\core\src\convert.cpp:265: error: (-215) mv && n > 0

• Debug mode with SECURE_SCL=1: Leads to crash of Matlab!
• Release mode:  mex test.cpp -IC:\OpenCV2.3\build\include\  -lopencv_core230  -LC:\OpenCV2.3\build\x64\vc9\lib
• Leads to crash of Matlab!
  
• Release mode with SECURE_SCL=1: mex test.cpp -IC:\OpenCV2.3\build\include\  -lopencv_core230  -LC:\OpenCV2.3\build\x64\vc9\lib -v COMPFLAGS=”/c /Zp8 /GR /W3 /EHs /D_CRT_SECURE_NO_DEPRECATE /D_SCL_SECURE_NO_DEPRECATE /DMATLAB_MEX_FILE /nologo /MD”
• No crash. Now to more thorough testing!