Feature discussion - full retina / high res support for openFrameworks

[ofTheo]

For 0.10.0 I would love to get real retina / HiDPI support working in the OF core.

Currently the state of things is that if you set High Resolution Capable via a plist setting you get a window which has a frame buffer with twice as many pixels as the window size. This means that in HiRes mode a 1024x768 window is half the size on your screen as in regular mode.

Currently we're converting all OF window size / position and screen size coordinates by this scale, but that starts to get very confusing when you start doing multi window stuff with a mixture of retina and non retina screens.

The current approach makes it really hard to make an app support both retina and non retina screens as all the scaling and adjustment is left up to the user.

A different approach would be to keep the window size the same for both retina and non retina but setup the viewports and matrixes to make the retina window look the same as the non retina window, with 2x the width and height when it comes to the frame buffer.

If we could make this work, an app that was loaded as retina vs a non retina app would have exactly the same user code, but the retina one would be rendering at a higher pixel density.

I have most of this working in a separate branch - but fonts get quite tricky as you would need to load them in with 2x the DPI and then still display them at the scale and spacing of the non retina dpi.

Anyway, before I get too heavy into this would love any thoughts from people on the matter :)

pinging @braitsch @Xrave @danzeeeman @openframeworks/core

[arturoc]

i'd say that having some setting like point or something that is resolution independent would work best than setting things up with double the pixels asked for. if we can somehow get the dpi from the screen the app is running on then we would could also specify font size in points as is done in css for example.

in a couple of years all screens will be high resolution and if we just double the pixels for some screens we'll end up with a system that is very specific to a transition period for apple hardware and in which the pixels we specify we'll be half of the real number on every screen

[ofTheo]

Totally agree! The getPixelCoordScale function in ofAppGLFWWindow returns the ratio of the frame buffer to window size. In the current case its 2 for HiDPI and 1 for non HiDPI - but it should be pretty future proof if we use that number ( it could be 3x or 4x on other systems ).

In the current branch I have ( which I can push ) I was able to get pretty good parity by setting the viewport bounds in ofMatrixStack to be multiplied by this scale and doing an ofScale at the end of ofSetupScreenPerspective / Ortho ( we could do this part better ). It seemed to work for most things - but the big exception is fonts, which you would want to load at 2x ( or whatever getPixelCoordScale() returns ) the DPI and then draw as if they were 1x. I started on that yesterday, but it will take a little bit of time to iron out.

I'm sure there are some other edge cases and we'd need to see how it affects shaders etc.

From a quick look it seems that cinder takes this approach:
https://github.com/cinder/Cinder/blob/663a60bb099b7594aa88a1597ec283da50986592/src/cinder/app/Window.cpp#L113

Here they mention they do the same thing with fonts.
https://github.com/cinder/Cinder/blob/663a60bb099b7594aa88a1597ec283da50986592/samples/RetinaSample/src/RetinaSampleApp.cpp#L85

[arturoc]

that feels like a big hack to me :) it seems like a quick way to get old applications to work with high density screens but not like a definitive solution.

we could have something like that to make old apps work with new screens as some kind of helper function or even and addon to set the matrices and viewport with some settings that make and old application with fixed resolution look ok on a new screen but i don't think we should be doing that as the default in the windowing system

by reporting half the resolution and then scaling by 2 you are actually loosing that resolution in some calculations, for example if you want to position some text in the center of a box of 735 you'll end up in 367.5 but if you draw in .5 you'll end up with anti aliasing problems so you usually cast to int to 367. but if the real resolution of the screen is double of that you shouldn't have need to do that because the real size of the box was 1470 and the center 735 so you are unnecessarily moving things by 2 physical pixels.

also this is also not an osx only problem where the resolutions in "retina" screens are perfect multiples of the old screens.. in mobile we have the same problem, android is specially difficult because there's so many resolutions out there so it's also probably a model to look into since it's probably the api that covers more cases.

glfw has methods to get the physical size of the monitor in mm: http://www.glfw.org/docs/latest/group__monitor.html#ga7d8bffc6c55539286a6bd20d32a8d7ea and from that you can get the speicific dpi of the scren, from their docs too:

const double dpi = widthPx / (widthMM / 25.4);

the window classes should report the dpi and once we have the dpi we can have some kind of measure that is resolution independent that we can use to create windows of a certain size, create gui elements that are not tiny in certain screens or set the dpi of the fonts to the real dpi of the screen so they always have the same physical size if we want but i think the pixels should stay as pixels and the matrices shouldn't be doing any scaling bydefault.

[ofTheo]

adjusting based on dpi - seems like a much more thorough solution.
I was definitely looking at a a simpler approach, to start with, but I didn't realize we could query DPI directly from GLFW.

I'm curious how would this work at the render level?
Would people have to supply units that are DPI adjusted or would we do the adjustments for them internally?

[arturoc]

i'm not completely clear on the details but i guess we could have some mode for the renderers in which you work in points instead of pixels or even some way to pass units instead of a unitless value to some functions.

we should take a look at what are the corresponding apis in ios, osx, android and try to figure out what's a good balance between the easiest/most flexible.

[arturoc]

in any case,

Would people have to supply units that are DPI adjusted or would we do the adjustments for them internally? 

i guess if we have some resolution independent unit then no, what seems awkward is to fix the resolution independent unit to what was pixels in most screens up until 2014 :)

[ofTheo]

Just trying to wrap my head around this a bit :)

So essentially all drawing commands like:
ofDrawCircle(100, 100, 50); would now be in points?

And as a result a circle of 50 points would be the same size on any screen?
ofGetWidth() / ofGetScreenWidth() would then be points I assume.

Any call which works with pixels ofPixels etc would still be in pixels?

Would we run into AA issues by working in points as:
ofDrawRect(0, 0, 50, 50); The 50 points might not correspond by a non floating point ratio to pixels?

[ofTheo]

I should note currently on iOS if you work in points you get layouts that look different between devices. Thats because the screen resolutions are different and iOS aims to have UI elements be physically the same size on different screens ( for usability of buttons sliders etc ). This means though that an app can have quite a different look / layout depending on the number of screen pixels and the dpi.
This type of approach could be useful in some situations in OF ( and I can imagine on mobile this is more more important ), but I don't think this should be the default behavior on Desktop apps.

I think whichever approach we go with the default should be that an app laid out on a non Retina screen looks the same on a Retina one ( unless a user actively wants the two to look different ).

[braitsch]

Hi, I would discourage using points as the unit of measure as 1) they're traditionally a measurement associated with typography in print 2) I think conceptually most people just think in terms of pixels.

Perhaps consider how web browsers handle this issue. On any given display JavaScript returns the pixel dimensions irrespective of the screens dpi and provides an api for querying the screen resolution that the developer can use to load in the appropriate image assets.

For example on my 15" rMBP which has a screen resolution 2880x1800 JavaScript returns the following:

console.log(window.innerWidth) // 1200 -> width of my browser
console.log(screen.width) //  1440  -> width of my screen
console.log(window.devicePixelRatio) // 2 -> indicates retina

I would encourage having calls to ofGetWidth() et al return the lowest common denominator e.g 1x the actual screen dpi and let the user create multiple sets of bitmap assets specific to each dpi they want to support just as they do for the web.

[elliotwoods]

To note:

• 2x scaling is a special case. Most people I know on OSX use 'more space', i.e. <2x scaling. Also on Windows you rarely choose the 200% option, the default is often 150% for 4k screens. Maybe we can presume 2x for current generation iPhones and iPads only, but it seems pointless to refer to '2x' as a usual case when it's actually decreasingly common.
• Should we ditch the term 'DPI'? It seems to add to the confusion in this issue.

Other situations where this can become confusing:

• Allocating anything from ofGetWidth() is a scaled parameter
• ofFbo (is it scaled inside the fbo?)
• ofViewport
• Mouse coordinates
• Window placement (often this becomes quite buggy with scaling, with glfw using a mix of pixels and points)
• Moving windows between monitors with different scaling settings (to act properly, the window should adopt the scaling setting of whichever screen it is currently on, so when you move a window it can cause a change).

I'm for keeping the oF API in pixels because it keeps things consistent (and less likely to break projects). I presume the main reason for using 'points' is when you're are targeting multiple devices whilst being strict about on-screen layout (e.g. mobile apps rather than installation works).

I presume whatever we do will:

• Have the scaling behaviour defined in ofWindowSettings
• Will mean we have to lazy load fonts so they can see which screen they're drawing on (or perhaps just check the scaling setting of all active windows when they are loaded).
• Need a new ofDrawBitmapString font

I propose we:

• Keep all positioning and scaling code in pixels
• Give an option to interpret font sizes as points
• Provide useful functions to transfer points<->pixels (including transforms)
• Enable ofDrawBitmapString to be able to draw nicely at multiple scales (and be able to auto-select this scale based on current screen scaling)
• Provide options for 'scaling whole app' for backwards compatibility, it'll generally look quite rough but I can see that running old projects is perhaps the most common scenario for scaling (i.e. if pixels are even smaller in 5 years time than they are now, you might still not want to faff around now with points vs pixels at every step just so it won't look tiny or resampled in 5 years time)