I have compiled standard applications on an ARM microcontroller before (like PolarSSL). You'll likely need a POSIX compliant environment providing stubs for syslog, fread, fwrite, fopen, etc and support for malloc, realloc and free. Implementing malloc, free, realloc is not difficult although the biggest challenge is to prevent memory fragmentation. Maybe you need threads/multitasking as well. The easiest way to get multitasking is using an RTOS (realtime OS). Booting will be quick. You have to analyse what OpenCV needs in terms of memory management and threads first. If some processing task takes a long time it will help to have multitasking in order to keep the user and/or communication interface going.
In my last OpenCV project I created some simple C++ classes around the coordinate lists used by OpenCV so the memory gets freed automagically when the list goes out of scope. AFAIK there is also a standard C++ inteface to OpenCV but the documentation for that is even more scarse than the documentation on OpenCV itself.