You can't do a separate browser window that's not full-size.
Javascript windows are possible, if you just need to notify/query the user.
For more complex stuff, you can play with divs, spans, etc. with css.
Check http://meyerweb.com/eric/css/edge/ for some examples.