From f57675c42e8254e633d6a4a9ba7ff8370ecf51b7 Mon Sep 17 00:00:00 2001 From: Bastiaan Olij Date: Sun, 26 Feb 2023 16:35:05 +1100 Subject: [PATCH] Add documentation about OpenXR passthrough --- tutorials/xr/img/xr_export_passthrough.webp | Bin 0 -> 7124 bytes tutorials/xr/index.rst | 1 + tutorials/xr/openxr_passthrough.rst | 83 ++++++++++++++++++++ 3 files changed, 84 insertions(+) create mode 100644 tutorials/xr/img/xr_export_passthrough.webp create mode 100644 tutorials/xr/openxr_passthrough.rst diff --git a/tutorials/xr/img/xr_export_passthrough.webp b/tutorials/xr/img/xr_export_passthrough.webp new file mode 100644 index 0000000000000000000000000000000000000000..2ee7c838ec18f9481c1967598760f2b9d346861a GIT binary patch literal 7124 zcmV;_8!O~eNk&G@8vp=TMM6+kP&gpK8vp=sm;jvtD(e9m0X~sNolB*oqavYl8o;m; z32AQOYw%{SE6!*S`*RY%uoXEzN&eT`7x*vBpYA@9_*V7zx4xy=+v@L8-;@2V{kQfH zH~-`RfboankI!DfzRBi>fOIs0Ywt^E)F@A4n#e{TJ({+0f(|6ltL zfRE@O+xR#N+_bzwDPqKM8uk z(PktsqsqyE3q#lXuuIkki!mX69#%95^(&4DZa8g&poXGm-Q#-`zvsQ1kh3``H#^N7 z%9E@@Z@#j(O}xfiJIYv!^c4}ex$gqx%q*xxP0r{i+xNy+A=!C51o^{*zW6_%!}H>D z_OszeWA#y~12C;6y}ZKOVB4!@(GvBT{;5l55fdk=JkzTUAMrE{brmt`op5rv?~SI)&XJ_fAxfu5c5Es8n?om7Whd9hC1?a+evB(Qqbs{|a~T;m&LX^Uu3AesSa5=%-@yh*eQi-z#Ycx&B^o#lF3X zv}k~J^gzU50U11ir{mP(x(7z!7sHzCH;YEJSg3gW+T@UW|KyZHB)2%bYyvC|a~Aj! z;YdxulrYcMgkFIgw!kANkPq^Ri1uvt-(=~Opk8k43fwUSJs+OY3W_cEVyz)gvL#)_ zAhC3IJeC`ERm_S|M1)#>07Un^2-s{0VHldP&+Oo!G>%d?KkdZcO4|U8hAkBAu1)N9 z_#n|!`Gp@CYyvWQ0I^5s4yvxktnv^(oI$adi{6a_k4vH?OjR^HDVoLLl{|o`#9(@3 zyR^*;@XGxK+z%QoTjRkhcGxR`1B( zoGp1-_Iv+0KUS~*YjAHd267Dt{8X?n-%8pyiPDI)a*JKukC>?6P=Nr9od!PE)5(H*I@?L4KFsucA0sY!m!p zQ(Ad#^TvSON#ubbMUvmW`~PV)VU{6Y^@l$jLJ)R=ZABWxlmU+R&|i@zcjiw?)8Z3^ z@xbAxdBm&Fp7uE{KW2=SJQp0M*Bp>8IUUjb*z+2`YKvnB<%k5HKvcd~*aT$q0;Te{ zzQoy8&0_!n{>#)=Yf(`86^2~5Z4nxP@b8WjmtmNw(hIb?)092tjdKX?NR%R8SN?z$ z|NkbPC!IkLFe>bkBwqAZy9;vg)MG=2sqeii!?~4A9=-d6SR3Qt@B!tHaQJ^cN$_>M z$yaKw)m^H)Rd+?IQo^V`4SG7S&*zoP^>%aH*n0(4td7$x4Rd`0!kZE|f{a5pIeAdf zDJ=v*!rIay(G~f63ogYHA(E{nZ32kbW!QBL{}O6|zGv?$Q$&N z^hy!%Ag3LaI?<;qizrLM(y1*Z-n%7as|&4_x1d333dh8Exnxk$r_!??F`z+oQWOEQ0!z{FDSt+THF8BxOh)fP2V~=5_MYwnj z?v(@+Y!9-l?SfYqGlxgI73w(kKTpp55z& zGlH+Y@HdT%w=4?!C5+ZO#JU&eIeL+b$cPM0iLai=G<~DfQgiPoA{zX%*t!)IU zY@4?1svHaR^_j|gV_@i?f3-MlWR!lgB5S%9ioGa)f6BO}WMfjvgO7OBh6LAsbY`l1 z)yUg)o0L>LwaE$_eTK|be_0_0{2udGE<{hPu6HLP+mZd*(ch^Aoj<2`}+s{B&`lB+x~X|+s?CESlv_?8e%LO$%oXRMiWa-2;q z8CW9co4rX1=MQvo&-?1EE*Gx`#E75u2xOzb*8|hxt zF6?_LMpy@N-^H9#UphwKmRab~%wFewACFzw!qHW`{jQz1Av$BTmBLEGaGp5kf*FIV z!@qU=RG{&JPP!w{E^N0`QRi8P?#yC-S|VGL@ax>QgC1<=OBQCV3k&;qgtVmuZzNXs z`>JH^u>Ls#j7|vG9+B8;S3Eh~QggANpGgbWKKK8ap_K78^=w{-P~Dd5SayU7-EisO z4-(=pSfHT}hpfU6AQ-H)9xzIYTbUPNjv3a1PAdVMiu+XcIJz2yV8^zqFe$t?aI4-r z0asBG>++_}6<(JAE6enDntFrqjxueP?|UQRBk^-w7{_{1Zvm3RNYKfLjjuvB>h0df z^23}LIjK4+BJNxZLBr~&8y~wgPuhafO z2d&Aw?DWybbpW`EZ%}OReD4%RT3mmIQ}2EhpDGe15LmS05g?GKOn_( zIb7Q_@a<+fAP~xY-HPJhI9mJGDgX}|z$jN}b&ma9iG#Kq>xKoUGZ`?fPt7WoM#4RN!mR0OU_CYi z(1v<0XSgl2Wt33y0d(=BU8H+@ZeD|(Rfn$B_6XwG8v5vTjr&m&J<*6BqOpffonXV; zc~+FJ0YAB$|5_ZRw=gVnbT6@CdsY7ump{+v*6c0lW}|It)R3`Q{7z^`t6n$qs=CM{ zLeq@$@>uKAwn6n-w1((U^(800nF2}Q&LVhRxTdrhiG0*yf|u_1z)L+Zh!_Egj76A= zZRd~=L<$3p{CPDI{$d-(m0w=?WGE>C*%Z0ythZ=)+dw8WXJikFTXnmZI*P8Zxz^o) zAtm$vX&51ur#XUr&hwf{&$}7%`4`0eU-_$*K*45-DTbD9k*Gvi3scL@BP*H0DkRP* zyFgww{-i=E+?A3QWOV@A5b*5{tLr_2f1p(GOc3(ww{AExs5sQY$QcLe)jocJN$uvgVFLbp8 zr-=6wnRw*lURNbKwS;s|asHD0BEoFrR6$&xm&z7;n;OiE>KV*=!PETNQ#&7hi41zj zL5B_k=Jv7!8~R+zhQUL%Bh|w+j)#5`TUjKv^DYHjJoyt;)OP#z{JVrMySUhfhf|?X zC^sO|rl<_*vKbbAECy>&zxsnuz1SiJRqaqhPCf;BsuHE^t#m_BGLGb}8K=}rUTjz$ z1zdcX|Bnnfa2GpV?ynms&6D4gY6~06oFDZC>0Ci|Pzfrh!;&GV}J4n{TUoATGmW zdER`u7bpO^wnTV3u7rI5#%;cdL5P-lBh$km?(?qAm>_B3I>$x6FsO0jXI}QlWBqPV zSIyOM0%N4kZ_7wtTPT1|y_oP?XXw3Bee|v!<1;BKxx*KYs?YJwK0jbZ6&}C|F7>99 zbj&7080#cl2f-`_0zO<_n=A(nv{^jPDkp{O2Qg;AV)p_Jz_9Hu%b=C@N*1jjT6>?S z;21XTG~9GiTAcdAi7sNFz^OE>Ln;$Mvdj~n+F?z7iX^3l(M0AZxShS1p0c)Qcv z5DkKa*r}o&BlJLkVqCn@x|#p@Wjr#S(uN>-pa6q>`uq!}Etlb5JU*O37_L3V&9B`_ z*>v?%rn_A8C}Dl??OkxYNFYe3LM%dMtd1&%nB_2V{DTw zZ?lRrxSnmFM68}lc}({u!3ZRwmf4hVTDV^8QFhyLpZ@E?fHrYuC;y^U{;iktDo1_D zG0{~=x&#jOkr2MOSzL#1IHKY-_Or6WMqn>+}Hl9p4i6J@uop?2tZYS#0Wzi<4WS$KfQE`cqMJqhLzLK zeJgdY;D<(z0-01jW(0FRN1-eC(g)~It@Mv25QBVaI@%OT|7KYXmaKdhP{zd}X|*N+ zPvP{1TC=PcfHQv+?RR+T5YTGb338u)ImrhTDTouu9k@fY{$j#7V~fSD7&_8CKym;8 zEalRkmj^m^5xW61aG@Ww+r0+($ruN@Kq(#t4m3T;G4)mE)3lgy?8;-m0RnsBLW-d8 zp$-&4F%i2dsfg9CA~ZzDm?-V+(OY>cJ?~N-D6I;rZ{dHnVI%VT&H_8Dqa<`L?5r=e zYzM`lC(I=Gm^QXjQFhFh}~2rgacz5ItcHA z2GCo!!zr#C4A5Y?f!aqPYrd+l3XX+7xxMf~(~`wYoA3>KAv)<7&(-zD+Bu&g@xCcL zGz9?XK0E!(A5}7*H>ObvkEcq!7h-mWu>M3NQ>qS`?sH^#_5OL`wh}%%{TA`BbAGCE z0BN3h@dG&=T!S9aIK~U2opJuk^{Y!NTK~xe(EV}o-^GiC*#3kfI;cqN7bB~!@Iaae z&V_>`r&b+rf^_Iv4}?a)d;gN~3U-fM3C;YfOrzls363s`Lw(u4)vL85@$%R%(~}8* zt=_8tKQYB!=!L41T{_RZ)&X_GTmT)<;Zxew$OE?JURn~P^uti-t{q>daFq(Xn~SL`rC|ReG-?wzBI&sG~208UJ~T+5@QdUsgI)QRN{g9x^T_ z@c?A@MP+3zdmN!fke&ma_l^GBsI)|<=Szq;$&5P;ch1&*=$tc=lnsyuHKmfqDdB%o z(Mf|LwRP@bxp9u3ezMd$`IQ;LMpXeq1+)4xQ4iCpM&_n7Qs#3z1Wl$g{|Y1IZ(K{Y z2$b;P9EE|hf0GC+Z|1nxo7IpN2oC0B`UT+(Nw)X2rILeQvgJIwi*(~%ql8)pHEKMw7D79H1kH`o z4ktaXWW>xNe#lk^ndgrXGm*u?T-gHTiFD;%f69jMWiaxZI|DsTZ%hJ>2O+fM##peP z9hmgw=af4m9vTy;6MZ#h;IR$ku(3WP@t`saSrZ<{;A(-i*KfCJtU?4b1jl~-Wn7`= z)JNYxjP?bF%Gw${+ul5>t$^%?dgd!+k81Zl`fPH@-6Mq(3p?|fIx^!*blm0e1`yTR zMVQBod-dl(lb!NZ$!rj;2DfpumA@uaM_&}dP@u@kP@7uVgogv2)|m{nQ1;t0xxzZk zVWM&>&96nj6HA1sARGtujiEu3xQGK4SECQF4K|)GZ5^}l;_=tWQ_nHYsL1{#DqIm9 zNgw_3uwb$~qkg`VdQEXS7lQhL4UnXyan;rst6GZL z2Yrh$D+d^UqDtGbamX_(hCESIUU@#yje~+A-bk3g zkmgO2j-N!p{RxVKq!`VoXH~&3bR<}rDmf}ghDI-`w9$FYH(*UDE$Za_BmMkZSW<5DM zk5m$+^dr}nyhY0O;=XRsbqcqgQLeJO@vY43WFbXwy; z@GRBkr60TnwnCA;snrLO4{Jo`hiT5mA?OCL^fvF>3LUn=^L068+jPz6K{V%4P zz>J)ev7H>dw2_4_B^fc|mJNX|(9vSx^QS7j0RL_(oC|&qqXd{0C^WJyp6*CtcqsTq z9#o3kOVTYAH!=JxC4$1-K>!hOkY=JX|K#3t5l+Y6(!3M7b!We+r8t0Sd}mKGH-3Rk zW=WX1`Z8_xYq4+}q;}cB_&FZ%RFx3;?qhr9+3WYQvr?imBR~OVI zXbG3gePbniNx?}hwZ!tzf8YnYZ8C-Dz2KUjg~}4C39o+F;})vosRz|xlzyw-mla_` zCP;^8bwMmjgT(Dhg!1)kbEcA0U2*cE_d^Ml*h({iblO=wa&t3S_ijI-ajoZ`RR4DB zCt9-~CmYd;KY6oyv-`&LX7ybX?ec=dVGg8ouJvki{kFloX|S=}4}~Ha28Glrz4G+Q z1t0bQ zN$_pDdB$*F+lyJ`-9e&n`S^K5`lsi+%mx(3v_`pro`@y>{l{yWp}iY`H`mBSR-=}{ zRbZ7@MIE9$)>FLhySW?#^gxu8`~1I6Orcj?SkOEI)9qN6>dD6C(S_X<7(-40S8cP< z6n-4>v2T$FZs0b-Xn_}aF070W>@Z(eb*QfR{^k6sF4i?vg4?}t+ zy$Mkc>fu!8F_78^VjSK<2$REh3xzWM^@Ic4k0+k}eP*ZO79ms;K&g#C+^@xw!Vvl2 zkqX#$uNUo7iI>cbT|xZXKQor#L3UBPsOARx+meTq{eS_qp0l3TUjDOQh;m#|%Qkv1 z9{ixKLymQ#p8huz6#Hb+-O~J&!7avColo(u{@hmy_@Ua<3z(AYbL7bB+)?vbbSTbG zU)p(TyO?-NL;^X78^d_bwuU)+kn-S;Rn5*FZx33^ML|*{Sb#jA(E)i&*#TX|j~Bo1 zUN@d%vjCsziSTO0E|QF4DlM?SyMDFRCy3&zkSyl3Z%VT!;4^GlgM=>6&s6;zI)E)1 zZXLD;kc)WJoBjbw$8FwS=TeUDtz|_66_1CM00000rz$0;*km`~O^#VG2h@ffn}X>B K?+^j5fB*n++U2JJ literal 0 HcmV?d00001 diff --git a/tutorials/xr/index.rst b/tutorials/xr/index.rst index 91dc601f9..a5e4ddc9b 100644 --- a/tutorials/xr/index.rst +++ b/tutorials/xr/index.rst @@ -23,6 +23,7 @@ Advanced topics xr_action_map xr_room_scale openxr_hand_tracking + openxr_passthrough .. note: diff --git a/tutorials/xr/openxr_passthrough.rst b/tutorials/xr/openxr_passthrough.rst new file mode 100644 index 000000000..775db2473 --- /dev/null +++ b/tutorials/xr/openxr_passthrough.rst @@ -0,0 +1,83 @@ +.. _doc_openxr_passthrough: + +The OpenXR passthrough +====================== + +Passthrough is a technique where camera images are used to present the environment of the user as the background. +This turns a VR headset into an AR headset, often referred to as Mixed Reality or MR. + +.. note:: + + As passthrough is relatively new there isn't a singular way this is implemented across platforms. + There may be additions in the future so this is a work in progress. + +Passthrough extension +--------------------- + +OpenXR has a vendor extension for passthrough submitted by Meta. +Currently this extension is only supported on Quest but may be adopted by other headsets in the future. + +:ref:`XRInterface ` has entry points for passthrough so different interfaces can implement this feature. +For :ref:`OpenXRInterface ` the meta passthrough extension is implemented here. + +In code you can call ``is_passthrough_supported`` to check if this extension is available. +If so you can simply enable passthrough by calling ``start_passthrough``. +You can call ``stop_passthrough`` to disable passthrough. + +This will automatically set the main viewports ``transparent_bg`` property to true. +It will also result in the camera image being displayed as the background. +This will result in the background settings in the environment being ignored and alpha being applied. + +.. note:: + + For privacy reasons **no access** is given to the camera image. + +.. warning:: + + After passthrough is enabled it is possible to change settings that will break passthrough. + Be sure not to change the ``transparent_bg`` setting or the environment blend mode. + This will result in the camera image no longer being visible but you still incur the overhead. + + Always use ``stop_passthrough`` if you wish to turn off passthrough. + +Finally, for using passthrough on the Quest you must set the following export property: + +.. image:: img/xr_export_passthrough.webp + +Passthrough through AR +---------------------- + +Some of the headsets recently adding OpenXR support have taken a different approach. +They simply mimic being an AR device. The Lynx R1 is such a device but others may be doing the same. + +The following thus applies to both passthrough devices that mimic AR, and actual AR devices. + +If ``is_passthrough_supported`` returns false the next step is to call ``get_supported_environment_blend_modes``. +This will return a list of supported blend modes for submitting the main render image to OpenXR. + +We need to check if ``XR_ENV_BLEND_MODE_ALPHA_BLEND`` is present in this list. +If so we can tell OpenXR to expect an image that can be alpha blended with a background. +To do this, we simply call ``set_environment_blend_mode(XR_ENV_BLEND_MODE_ALPHA_BLEND)``. + +We must also set ``transparent_bg`` to true to ensure we submit the right image. + +Putting it together +------------------- + +Putting the above together we can use the following code as a base: + +.. code-block:: gdscript + + func enable_passthrough() -> bool: + var xr_interface : XRInterface = XRServer.primary_interface + if xr_interface and xr_interface.is_passthrough_supported(): + return xr_interface.start_passthrough() + else: + var modes = xr_interface.get_supported_environment_blend_modes() + if XR_ENV_BLEND_MODE_ALPHA_BLEND in modes: + xr_interface.set_environment_blend_mode(XR_ENV_BLEND_MODE_ALPHA_BLEND) + return true + else: + return false + +