From 107088fb39c4e71c76a9ec72ecebe2aba2e382cc Mon Sep 17 00:00:00 2001 From: Andrey Khomyakov Date: Fri, 17 Jul 2026 11:47:40 -0700 Subject: [PATCH 1/2] Update NVLink user docs --- images/nvlink-add-user.png | Bin 0 -> 47091 bytes netris-nvlink-integration.rst | 517 +++++++++++++++++++++++++--------- 2 files changed, 380 insertions(+), 137 deletions(-) create mode 100644 images/nvlink-add-user.png diff --git a/images/nvlink-add-user.png b/images/nvlink-add-user.png new file mode 100644 index 0000000000000000000000000000000000000000..82c4e08d64531fa605f8beccde308c06e51b468e GIT binary patch literal 47091 zcmd?Rbx>W)vo;JHhd>|%5}csH6WrZBxI^%byE_2_1b24}?rtHtyGw$*L-22rb8-&% z{_d^%zN+t!x9UyRhLyEv)|%;_>3;g@9s*^hMUWA(5g{NTki|p=D;iY| zB1icHf*4`O_t)@)q0}WSK}1ASnxi?NeM}+sX+Fas1?vh3EfNO%RM29YsVu=g`{=mb zbh(j|kaU_9WjoAuIn>PX@EjrtTb)GQ7Xw1{m0?iFZIki)a%;u~TQ5jTFKA<4-@#tg z_x$`m5W1E3hZhI5Ub(k4!3Am$_Ya;-LbPfIfd$cKN2pPl5&R&Rcsc=@B;HWPXZ*GXq zXx0!v!EX}orKYy{1Y2=@BNGjU|50##_PupN>|T9PP*sYVL$!;+HG#RQmQWzNYv_(A zY}HfUB~jl2Sx-OwO84R2_x?9V7p^-&yjV!+ui%C&yf{*L2HgTZ*Qjmj& z4QS^%WR=Lxtw*TI5!?*ed?b~=Q=ihieKS&RB9g4Jh>Y)zH>hzcffF;yrXJcX*FQh~ zg*IB5O!`@kdY6WY(~35LA~=}zV@31PCl4*#-Rr)gH(aZ9uCL1UZ5gi6?kj9{hR=u5 z={HOazxTMTo;0~{*bo_9WlGY`s%2K8s&dl}%3u!nAjS<23es*aETGO}<(5J58)Cil z#w&xn_d-dAv=Lx%W=oJy(kVe*4}$5vmBfZ}26JLd^ptDxR#w;8L+k7;4rv6lT;?tl zX9uMP9}__75QIUq40S@AKvby{4GUS*8WO^ICBn3ABZdklGocH}-047|^*|Y7!eQya%nQLpSpryX~XZ%PMdA z72G=L5^tOpl-Ce)@1ALSSs$TD@}qw8eT#}A7hp#y(1Tmyw-wH@1_i_SQIeNC z6jbcb%-<RD}^kN zS;`Fe9;yQ0?-n>)q^)>nDh6rRB-xm3=3tLs^C~ zMX5lQtgxjVFPlv{_ZF5am&)P;XTFn=wID*Ce)d|{cSTCsxl%hV@aLe+9r*+W6no-x zh;t%)6|6$Dq5(PS541|$MGJ+TC7Ic}r7vhSdgvf@Ng>YfN zV(QwU8g@(>PS9q2*t&&FEXz$WjNhOfyi4RtL|2jPFmTOfXCZPxMclj=E;cOyup7 zZ^vyrZinG)y{4vDF(1)CY75s9Wm&^HpxC`lZNNCS>Zmo#nER$7D_vW@A>K4$G@ zJr%|kud5WQpH&RhnpL`i4RkP!su#b`?-W-1+9#ev?n2^_CP5^*C7~ou*BI8|&YR57 zJG|u3ZnSsUcPQN(+;_fk-8*EH#lvJXv)Quw^>b_Ot%I2Zu06e#xYbwdojj^>fi3l| zxRm!pT0^_ZGHjl9)^@#iY&$1;g?^@Aclo9x{JM~wk)BiI+Kk)kHSXI+HHI; zxoo+VahY(XI_`;~YgvDpY}uLzu2s<^<^J^p@5BE69W@PRz+?~hb1Os@bgMVvxUcb5Fn*ErP>bSD zl*4PiEHm!T3k8HJi+_3j4)N4#c#s;EBc6iKLUe6nWp zD-9V7EXU~0E~<4d2^$GYDQqzZiKPh6zRQ)k({5-HZwf#BFy^Ril+FSNg#3uM#p7Z)ltY7|I_wQ4H7YFwT^H}~ctH1Uge2&56HlJ8kUHiRUf0%iX`A1FTJXZ}%G-d2Y z49dPrRaW&EB_)Lmhs~1|<-Yy?%LAf=$h}AxVh-M@DX0g9c@akv4|_`gx;hwb%hK%L z>P`&9!7pXwE@N*p-ojZYnj!Y56us#A)?vlH~vs%o8th2N9OP%4UBXTH#;46j{qY&a6;yht@5lc?(6SrS}Tb*kvKL~<6Cz=?Yoo{7^d0e~nx&pKV zwcTBt*F&z8u4^wW8_D>|@LMH3S*}{Pe6Iui!_LAM(=vIuk_PU-+-Mwko=J31x0OC` z&GEb;(+!>-mxJ(o#!7+UHGK{ttBY1_1qJal5=)cId}N@ujnVtzJHwpo9O}BT_52WAxQz72I!S(qCF9R_@+uKzHn>w{CkE!o1N+w{sNtos>rE(;bGeUpaY z^Ywo40ZB|BM#nR|{~ExrN(L%ohEh@xl)yDS1Y`gh0vfo21U}fn2Y|Yf{t$4$?-#&F zFdg*gTPU1#s6W>r$H#_z@&aOFz;Agy8v_FiTVqSRhOu)1Geb^+6;$k0q$Jt(EX`>@ z>Raj>&^no0J$8ZMbYcfC%?<265;~ciS=h2WaS{D)!46zM-liiW{N2RPl#573N|sQ- z(#C+0nf4v+J0fmGLPA1L8+}7|IYHsSy958@A~LqKvtp;Cb98j1b!4Emv@xQ4&&I|^ z_l};9o}LD1L1XJ|VfWFA#=@5PuZR43j-Y|9o({&y z=`(P_)Mdf6MP+53uVyAcH=||`1clYr2^@Le2ZRk``p6M zcvFP?^65C>nSbFt8771`A{yaLKB^K*(9>~1 z^XsI;U9$5jSBcS^C-cXk2~UVo+@a{JkP?GCRx2ylpFNB;Q+xZZJm*moDx5Bg zk>P&eud!GpJ3g+?ah)#dx&Te}*N#~r^)&6I3 zdJ|&2o;g%NW0+9$MU=XzbhZbD9B8sP#?2y2|BO>sV+Q7G4Ci#oQC(VPIj>|gk$+H5 zHQAl!L5oh-YN^gU#BEw;wp1ad<+6UxKTF%s?B~FG{c${&=A_q1`}4u_;e@>Y7R;;o zRm+&XG}rd#`cNw8y}*Z_0Rkhj8wd0CpAj9TT*LzSt|vSdV_&jH6e%{6OBt9iq$S`ojqxOUA=Ptv96pW?bf965MY{Zn}>3; zx>tX4A3#w5{4N*^?(~;uYV6sWUA^)5cR0pwSLf3w1E+NLwPMm@5y&_!y{D^MXp1ej z0VoIdO?>~Z>-79K-f=21U&K7WEZOvrQ!y-fH*-z6B4e})fq}(27Afo(Z{1zaUz)ei zG#iB>Q`=<>u`-!oPuonbFsyy;r>mS#vE9&d-XE-8^cGXCDd)gEl`wL0`B@%7O2$5C z@e!L?e+-Ub&UQLqYY`R;hs8Xyb;&-!Zf|7Cf}u^&PLIGtfjj9U2bs6Q;Xq%Fn;XMi zXlpX7u}H01_^0VKb1@;f!sgthK$)%`#c&!kvP6>|BPt4Ay~+6aMPP6U6Ac!z8I@mZ zEqIlz!ekKJHQmg%B=FFauvo1o{O0~%`LZ2{IiHTM`MRMnF*Bg=QH>d(dJyPC@bSTrTmSD>*Q8a{ORI}~!cUd&Akg z$fDqh*YL2{WNXed*BA(>b0;!`iI_|$%G&)8vfKS($@b?+<}Mq|uKHr+YAx`Pee zu>(_RY#1S@uzxJ%3X(T46XQaEGIKtshmHcjzIT^-Y5HqN84d}d86(-n&@fRB^)_;< zHP$ufXV~cI^2XNj30M+Qr<_Rpu=YnQZ|d*py4Bzi55hk-C~UuHG#(`sh-%TYITkX^ zEq-~mlVrJ9fkCA#Ha(QG?<&*!Rk6_Um63s=vl`8;NFG)pCWgWEZXMF#Y7v{QP`AYv zEAsvZlUqI()cDsJ^EYG2%V{SOcN8uojN?0U@0h6H3FoZXEIqMevI=dB_I*{x#x`oK z$?tK6YI!1t*Abb|_}oqhbS(G3Wi3V^k~NE^U5OBW>dena*UkYADdTYKkv|3*RVa8BL?TR%lZs?fEg={ zv_G~tta77A-caY-lfY9ld1bw8x1`!Te6gUdRJ1_?V`PQq9}mE*3bWuX+}^%|4;vD9 z?j%%8rOZN^@>9tuhW&D{a;?f63h_>Z(Jh{9y2)82wSF|)n)pVwo9m@yzZ8{sR9{6% z<$iCI$z{R*yrZa=MJgU~#wy9#CI>@{{?E1Ndtc;TH1?TuG?VRc1Ee?A3zeisn>sr? z6K7L%XN>}l`vKfKJ%1nrhw^ex5{l7&+$-B4c*9{a>9C1h+w&%cD{m->Np}miUUwj8 zgJM;+LT{QD>{i7w!_Lqc9cAP`$eBz*7vH{OD&a(^n-O&i`R^|B&I0*?LO{Ol!6?{t z&sFMuLS}R4!$Kpr42d!_#kv`{LMn%}`$Za>>MdG~xi{YTFk)_Y>nMmD|`kGJFq%(dVSPiEF12tNS;>5`!9^cJb4+C(hr zL7gLzw|`|NGV1{QVlLPyr4`sGN&84hb z7U`TqE6n7rZqVIJizyQWzH|iSBqX?h-D*2Q=h6bt?>dDQ8wyjTV5_K8aOMyjEWNq8 zIIO$^kL5=oe+sf_T(;{Uv!(%) zW=cjQT->L(L26z)#OZ}$15ZrHN-F?%@6}n>p8ea>+Y6I$27YUY`1A?3{Z$Il?*A}_ z%qH-!ZzN2IfoBFaT=#JPvuzw1kB*nN&*#mq4Z5)k~aW?oTYL#3AG&B$j&E1RJD`ufPoNTQ54 zL~q`_d4VR4d$_46eBmQY*sT$OeYTnKPzZPxF%=aR)YwR(M+4$b+J^T*XYy73xx6<3 zMiCeygv}(s2nq_$cy-8Bir1jRt9jqvy7dkX#ZFW)j+)9rp}csMwU;FzdAHr+Szkdy zDk_Pu-JRkJ3NO8d!zU;@;Uyu>2S+eN*P-=_Osvrd{wVO(?Umy?&y8cvfIRg+iA;SvrI(ia_5|8=H4eJ6l2bVVG^15wNf4<*D-6 z?X-WrcxiC+j^dP;;`bE)EauPAR5LU6rf~}{-I^87{jsam%dVC_O-Ez>-89dqV6g25 zAHd!N2^_<)I_+xGeHg4D?Aslvp?@d=uhPdC{h}PdcUaOG4>~>0OY6Plxz+K^^|(b~ zalTg)R_mHM)1B%+`UU7VK}# z>8($0#!Af9Ra-rpsZ|>!b@lWNz8`%YYtdA14D%?wnLH~s*gkBPP%}xa0!O{wOmPh* ziMKT!yF0&~uQ89x>Gy~;98B!n`tYh3h>?2f$mq&9z$0;?Eh=g?#vf~^wL8~8q^WRN z@p{hISx0>r>=td6mx=iN8UE6vi7%Z9+T+f=-0a=7z@{`F7k*3Y702Fabz!$XwcsBk zPAQNsr^bAi^f0~bqOMkJLh9Lc3=5tp^WN9Zf%GP84kX~Qj;vzv{)VB~)swNN@4T~Z z^j)=8ZFROd1jg*H+M$2_os!`h0{buo50)u60GP6~> zhWjLj4SsQ%jL9~%f&lWUse7z1KC z>mOY*GsD9P2f^c#>CGmSCRexp zbc@~U)h;1TXMLt8n;Z{6m!_H{S^g|&UKn_VKF#u_w!GY2@oCx=(sFyM`kyE7Z4PFQ z#ME3GXvXr=+-ZGbQNqzW{KdOb_N91jn)hZam6Z~cNXh*I0(yNP8kQ|KsZA%p=6re2 za;e#97qMH?EIc-NjDpLmqbwpMln@&@S`(8YrJI>-dy#N5pV{dLTS-n<6H6tfRYjU9 zbEswb@XR*(AxewOwdZOxg}Cu@&)PA7k|qophcTQ?3R~jt`h2f2N<<6ot@O%DhJfLh zVYh_)K}WJSJW9N|+87z02maVn6|mzd?`pAQw?qKS{#s(8RGZsdRebjo3Y<^1Of>e_ zmr1g|Mg|uTsiCP z%d^>}XcB-qq2KQ>UFZxB42@0UaGYBacxA$i_Tt4)tx`*118&@FR2oX%aI=vLnM{5q z%|ph1E4ERm?Ri?fH`}PJRHUK$c4QUriDxdkG@>3dI5@bnKeJG1e*bxr<(EoZ zo4#m~%XTMUUMv_JsokYT#|6U%$SzTqQ! zu$fP(Sh%HeJec{gLryXSp$=t-T*Z_!RvY%EWvl06uCq@J2!(2<;~fgW%9CrIsUR3W zR8=uvE;m)u{9_8eYJZzN@swF~0dZ-5(@)rsd{04p<8tc8$inQ|1@2COQxInLKkjJKg~V)r*OCt+M0%k;X0H z$r1(WF|hvbM8T>XXNEG@gQHx4Urj4%0Q#Gvk79~^22w991ebm|>yziKwtaNhU$ zW}@SU|3M$#L8x!*2^1)dCUD~UpkRBR8^Y%0^v4Er_gF5sxSw1!9wxM`)H~pQ!Y7(; z^4x4bHr&YDzxeFzd=+V)D)FII%cpsC^!JERDa;Ro) z*Hs!0c*WMh$@l2BOE$r-iK_d$lbO!8&cDh%=fRfGlMdwOYMTnSm$JcU`5#Vpa4h4e zLPba@`$$Vf51DmWkb#jOp^TXKyG zxNwD$Cqp{q%Ym?0E|{V)2ND^kr8jX22Mu3ir5 z>}QHB=ovbW6>yv{fbbrkNNlqIQsHH_=bD@9F>Wn^_Y zkxK;s{VpSy9%#Q)B38WlQ-gz>LYfxW`VJLs?d~r>-ggLF4`B`mOSWU0*1o+y6_2cX z3nsuc|Je1h@_=@uu%d4lRy`?%ECHs(z*h(RMyT6qODcw2uOp!-qe}09nMH@fa6kbd zd_$3bNzBKc!$SuFIfC?wmsf%i-d_o24=fZEAp_}&R;VD&BMOaj^J!?oNFP<9kh%Or zqs9pSfl9Jipml6QEi9}x+&`Y`QShko037pHwOqsgv!nnl*ZPWZ=IDl!$oEN0=v?5n zI#W2m@#5c`r|l3xnYS%LJ%mpHo;L+=8Cv|gaLLoa(-i=Ct))ctui-)LLgqEj#v}-k z&~9|;)0tXPB)?JL@?${Kg3`l>3NI)uEUu)agc|*ch!FJ`$c1KyN$j`7IBEv)9HWb16F(v=`=8SULE*P`uc|JaKcKx zq@a76@VGeggoK0(@&H=Jdrj3f+P5%3f}j&){Y6|qX-0-y9AR6Z!dHp6sn^n?szIH|3^_xpx6HcoBr1)qT;`vC5eOo)e2Gh z;mQ*)qGJV9hxzMDiMBY9obsk=BTV3PCx!pMkM>c_LYftSz4l9DoD~xwtgH4L04Em%j&C+K*77 z5WVcPf`rDR931L$@`Jv2158-2UcF*8$Rd1{TRISILz@gnShl|C>FbMBWBMxb z7(e)HPd-a1FagaaX7F`;3-l@$Z$-ZXq;Vtx3H$lRpgO#O7kwlNM~x9wJk?|Q4)N2c zPcx<;O2z()Ngh{=n>O4Up(r%J6bmJA6No5GM;{}K#|0(g)kysRa{$3~Ud)hJZ(HV5 zI`oAQCeGOrEala#eKp??Ri9n{8yR5e;?8_HUi|V1`l5QQH6>CQ7$AIR~ zA}0p{a=$d#phSKO5TGA+G@wOp>v=EqqRZPmcaGJ4{ z^zOr4jaYPx`}NEoKD(W$mh(RH)hP{&n55(Y!mAAX-Kof^Bw~41pC`JP+7)&M z11T)p%$7?;Qc27)9(k!mi;ecx^AFeaM5^UBIaOk2=IV0aI*8YNPKOy}QU;@jTjR;L zu>uT6+8}L)RE-+KT8$319v#GNjeQcM@eJ<}&n0f9XQnHc<Eu_q={M(Y(vpY^LsmT~SwW?)x|`A|y=O=<3}mXU^{jT&`xkTAHuQ@HLg3>s?D=KUYI9& zx7=N!*NxKXE^hLb^L|bKg(3c&Lk@yf6~^Y(EA2>N!mgY=NKeJd``y;|J3`+V*Y)bv z3#Tn_PqF4tpH^dC_4xP~8y+Ei|EIvH_4}KH7%PT%O*-smYGgxPn`T(tX8D>$?d@L@ z7!20GDgrq+$3|*8I%SKy@xdA@r8}4N^>Xsm=*nSSY5hS*h67HZ9@I&0#B4_W>#q)%8=FE2RAQfM6f z&|R(U;g%629aiw=0oTyHO{rJds37G zN8%!n#@MkmI729%A$@t4@ll zl`B@O-h+a)r_>-%;WiF$`gKBObGdx~rk-r(HEVin%C*DT+VgCo(~TiR1CRZw>fmJa z-1bRY^~nL1P#`|#2zTRs@G#(p#d%VtAUg@oPfuSN8j@I&KDN6jeRHvoj>lNyTS`T> zzfj+wTv;P1jE@y;z0w9*pjx>hw)uvJMyd)&C58PM@FVAal(3r!5Z>9@Bw$%OzN&e` zM`ulyu1&vR^`JC;Y%a|Do6z%85HIs)0JYNs4u6kH@(LvO$RC;nHxYU9mLVzpiVx9& z%XWL>7ztMZhOwg(N#f~_OHT$6$U3yX;~#_jXnKzxNy1sboTr^-r2&%A22}BsOoVp= zsK}$eZ*osNzs&>uFB{}69;66w8o2-f9hQ--Xs!Wp0SPF71*m0UebGcDi{jnqreeTW z(OUwqfCX97ODRA35-66q)d0pgO!avW#@~xOhntIw_#M3jEgHNH`) z!lwYgV5&Y++BgY*fh$^?k)5zxK#Eq6ZrG?u7i|$10Lg#wb7#bsq*od3xV0vjhGSz%!Z4k$V zg|b1bIFNV3)!5$tmPc$oGh(i7pcg<#zF^35!mc;G<#Wpx80_&hK8TU!1h~oV;c2R^6OOhf7IBcYPRqN-83Ro2K`d2|?ZZ4Mn~^ ztd?FnJ)=adp=O;|`fh4wR)oy>b9aL6;HdTXItkz>6__n&i4L1sw1!yk7=fagS?l?7 z2yb5W8Nicd$HT?hA54tuO|>Rn`3^irO3o|V_|vXxPAhGES2{Zn`iDy-1#z;s!;k?iQ;Gr@}YAi^BL}YQ9+cx@F%a(m<=e-QL$Kn_a z2Yi$-zc&Fd;iF%IM|#eYG35jBD)ZOxbbrYv9N$UK0?hE)_M`+X3N9f!nN+N~daYP6 zCJjJlmA;JYH*h~!pR#OCc|)aC5?xov3SKTLeul&T-65{8hm|^NDNVPgb*By|L;m_A zg6(ue>@by!q{IcNa^bx-pi!^wYeXmOjh7IX zFde7Q_wLSE6Bw%!LqBEQc==e3!pU=BP~TsL+LZv2_UX?yjwV})(IIAI>s*R|$*C$R zSO)~UdD)sAXGvqn8(kY`gnp1x(9BKcJ zmgg!>U`TKbRcW0D3c*l#VAN9FBi$EvuqdS*fvZ~WQoT(dro8jlkWA=b!sE} zg~OPq6~8nq84jnA2QVb{ePv(iP?gLa$ z&d|feh7T6?fl`r2{55yya5zh>?S5B$+v4V@z@iNTieI#7vEL7PAf)3Mw=84vd{9cV zWdSd!NHcYFxR3CPNl8T+X*qqM#b@Jhxwr7&n<*dHx&i7Xvak9BbcB*QwFrjgkEB!&>?7uIYC& zBgSfFEYSt+qKfBh=JmD)`73j!)^?~$%4o0w^o@Q8R53xv#8&ZCTi{@0Q#MQP!XiL= zqOm${g#(!k?Oe~(lrOmN71*mRsnR^|{iwCe#|%F}h14$L(c!UIF%&x}_p}T2o@_=fP(|)ztH=tT+W|g^n*5H29IxOmska=%69x_&MD>83W zp}$S<=uz#qKUWnYKfv^`{c3#e9*!Ua76`DmJ`fw6Csv>2Abexot}B8gaM$4P__kMX zCpqPr>b~R#F7GX&+n(MWZ;7-tbfbIBa2(W~%S7t6Bs+aulYlCUJjw5-E&y~B^+z{W zf^AeZIG)TbbS}QK?}YI2qQBlHrZKTFNn?M!cH2b`$tBBm^^elc)XR3iaTpQ_EK!nj zI9LGl1OP!SW=Kq-QWE!8<$l|wP;gxQRM61DzOyecA>vwM+m8bPc@|?VO714X!eQ&1 zBDMb}h;!FvTJNSOa_vp)JHMc*STwwFIG7o&(bziA0nUJs?K87pXA;kwzM>*GU$E16 z3;0b9FX2h2_t z2lAv^U;|gAO_!)!s=cq7yBwF=l~#zBuwZzY+G>SMBryLGf3->tp3SUZ>>8&-e%q3K zUyZC)r6@d}mbCbmWzi+z*02u?4PVO){~VVzW_bKc7lEAEp~ z8Fo?5QMdjblIC*jNCy0lt3BwqLq?m5mL|aQlh(+{L=#K;xmr`BuFrx_d(7*59aJE1 z!gQAVIhsB(7aDLaJ~F;ta%5=Cud`F|kV#-@V7X=}T<$p98;H`P+-WyGT|W=u?EHaF2eS1zJRf+dj- z3Jt9>m>kBD3eQ><?L33K8Gcil$KongDB(X_5(0*?k<-`DMkv}zqGNEF z6$0{Z%CQREn@#{W{>x(>KZfl=JWVtAQvf5NU>pS;BD5zGyQwy12@}D`HHGyexT;a? zHvar6x#!~yCS)u6TWp;G+`A-cjybsu&D=4D&?+_Kjhlq`&qG3Mp1Dwdaj-v$W_(w7 zY3bO13xg!ZTQ>8A@z#dldO5kAqy&;$Y;=goOLdms7)rA2ax5xm*+KHS?{u&o<81|k z3`t4x{m*8!1 zjZjv)kUWE7KiI(9Qe{m_Z4QkiZ@p7BB`%KcTDw72TLWK$D$SN!>u=Pb?(`VesSmQm zUn3ijW-rp8&G(*=HDirE9Pft2k?l)Y+sz*4hTW7~D6ezD-4L(HfLv^gR==F#T(uiE z7?KaZhxFn;{V5nFeOFYBz|N}fSrv1%bVCrHHaIcDu9cYp{y1{FZJ)CkP-QwjX3)R> z?Q}S*rw5*)&LuV3q=t(1KnyO~l6f&X)nyOuQfC*aw5x9gjVR&lZ)C3uC4`SeXDV<5%u(TALN>D<&Ns&W=GT!^Vp3?d-A{{~Obs1u)C7nm zQPUaP!Gv0BW4w3gwM*rzxOVDyMcosS%hC;V182-v9bs$a6m8O-;4rDb;LU(rDCfP$%F2Mt{#f zIWU&f?iWfvdSa4(I!0C6+md~E0Z*k=OwTam?zS*mrJjhA`|xqf>@p4msA4v5g&w6f zB3&=9I>B?K$NamLyv-W4ZZey1+Vaueej&IUuc``$eOGGq?*1yQ6=O~cF7lORZTzg6 z-O|Iz2x@_~otKHDZ6)f*YwgAXnfBF1>-jXKgFaViketpyPS(26>DrBFJSO};{<@7& zAHm>6l?zmHfo(U6gW5u)B*eWj#q)#l5*eDox{w7!yN&ICuD!_1yK3 zKjGvhWlwyQnSR;TA2*n;htw{|Jm*d}=?>7aTB^@vxf+T`-c-qH-dQ;~9T-p z;pES|oy1HVhea-H+%Uc2BC$27sW>e+H7Jl`qb9fA=j!X+)X zK2B@?7cCBr`d0NZb}T$A6_VhFHATxm0kPki`nn(-9PlrJMlFx+)9XUHMyxTU3;k`~ z_A|2=efs7!KM)M$Y38djS}N&kRj&*9emOgP#vxS#i{6=YXBecp@3a0TZDiGyI|YI8 zY{O%@-zEGMRP#oFnR z0OX>%$LP6hFL}xF7q|Owy-9kePYB_UjMwlAK=p@kj*Trn5YnX!dVkYCjH#obdK&se zcvk?dfAnGy&C|Op&;a=^4G}E$^k(`V5EB~I>PkF`gi%=`0Y(75O;`Tu&9{xfYiEmR zp0M zJejMHGC;UE?Z!qvVa-l-kKDuNp(>dtU^ID0fHeW_jv?RHhx#=4AZ*}eivk^l zHZt`LnH>dJoYoKt^LL?|LM{ZH{23vlwoH!F;$pMyJ;{LKXFVkDh&8>-K zf02qtjWMpUv`G$|G5>JM5~#~Qx!^eLrA3dPct4LvFB3RrMDYcmNeNns%~i~ zFEh{%HCs$0c9pjwp&*^cER)uVp9n#UBdhMn0lU@TvtYaFM8gBD##^G57*EoN<7rn; zSd5%%>X*I!VxC_@LaM!mo4^TgB@*ukYw7j6E^TH>vvSm@OJ1O((`;j5owP@Xt+ftw zlj|Km=L$EG*rLT4QdKs;wApeNHsv#;#RF=wA_59L9AWVgV;535+4D$6;7FF>BM$la zCEC8zZ1KqZ_w;$_O^e1}I z##)yP4$QL(6{8+A9NN-pv;io1747~tZdE|li^Z;4KBU5-78hv;H2Li8?2ztXc*zGi zOl*~UTd`ecHY0!1dW`*vdO4N?OVxR-)*$-3Fq#!~ZyH1=kQkstjOOBh`y(63xZA9> zwezbB3VsT%VRaIpV=w!-DvZVRF*4a>ju5<%A0c^T(s0Y_8k#&;8+LXuE6<&L&2#l% zIE2yT7s^`J^|;*0Z_vq`>7)`FG0jy0I}o8*s4StRl$$q>p_W^WflXp(yj(J3bBBtt z9vLkB%#m)n$)NJFg6qb%oyAvDNANe#2+!?pFkc~#%!nAP<{P=yBg6qvNJzMg;?s&Cs8Sxa?H zSoAv|3;z}pFRk3ob;{bgM2kjJZ6YLEcHF6a$u3QNh?U->V^;POY}RG9#rjtzW!rnm z(%Zsco9Y(yt612$9ABMZVWEBGF|ExpO7m=QwgAfWYDfiMF@ndwH*=D)3(vhWHXpH& z;&42u+7;mEHw*6SQ!E6MJ+qmbUomO4h=6J?hePct7N)P$Wl4qx27y%c6*G(cgAX6< z7W^$le#BwyWp(FMsAHl)#qWQSPn&dMD5fBoe`bM2v$}~}wwRZ^Z%42gr_+J< zX~89ATyeaQ$v za2jU~yu6Ci1p97v`h_D7B}$<65$1Csjm z4|y<;`AGKQzsxf{$h&HwX;nM}s*NF(+2+y|oft<7NhC4>vDWyc%5v2F*CThRzyw(; znNmJEfM0EKcfIOSg%raQSrcl!>mQmmlD$|EOMA0O&2jIEn3FQ>(A#iJf3-fntluXh zFGMhv2i0}%v`Q`+?P2Ioi7d>R;&{OJ0uPU7nDep;IMyaqwJrb`&+{F9MIow=m|Iyo{H7^f_9nN8yNFz&3{fZ>x~v&y8wp1xn6Jc&p+7SCInyof+$5MJG5fa`xZ z9`u96o#-mfvPG;*CYd^n>ppu?+V!MH%#6mf`Gyw9igo_v_n}Xq!eWk#lA-Zv#WGLf z$HzDHj#7zNKMRiHQH+`oBz~{Qx74VS5ee&8+JoQbuR6(7v{)D!1*@o{a}^l;iN8HN zp#`TqieXL1Mm8IkImd!$jdj!*#9fp=bGzz2ctb_Xvb1wUmu0V3lNOvR(G8l5MFo2KCeO6h_q->Fkjrfy28Q3sn6CX z8ehgdTLa)2;Yv(wNS`kCbz95N)YT7Dw_9K2&vz!N#y7B6Sox$A9CU~uT_;eTVf=!C zmTDlXBX~$O2S%*uu)_B{7Yxpvkn-hUsm7O>E81P(@ZJ8}ci%wl_U7Hf5hsn#^iHE^ zotZz$Sw`LCFKY5%|2{{7#vXs7{C5Bk={*!eoMZ4U!bk{1MPpBWQR(u{ZwP=($J^g^ zOh*0NC*^+wm4C<>D|C)l2!0k*-B>|F)qOhB)pLY8UQH>-&dgY;kl_ATEpj)2dvGnk zz4$$HN?KWBc7^Ne(8#XpZw9;fxU;6Ig%Qy<`mtZxtv|}K&@MP6ne?%SP~OM!AEWW6 zGXh9>?32px0~4|aFpTkp>G>6tGVN%O1p-w!E|3udPfzr=9RqN>JE;uXJL<&|qvD1b z-scoxdujlXX#J<&BkpJ`1X^gVujwc%@@~$MzX1+lz*|XO%V>t91Fmxp_)@X)o}xJp znrJX;Z+x7g$&L{PNP4cLjr{x2oH77lZi;|{28k2W(aF#=Fzm~q0I9r5yaZ$v@1rs+YoYq@>o#J>Z@p!1c*GKxh(c@a=1&XTR&#*}z{v_=H z>+`Ds$UFV#5Whh+L#z=1gl`XUrW(lc-b6)3t<6UxzWEH4^#5KjgjDfD4i}gX;SqElF=7)t zakt0Y@YGb=9~(&%Gc&IPKYuQR3?ML!qJD!q0Rql>N2a9E6cm6nr}Z^7G-TucAM)NZ ztg5a58kUd{0TlrOC8VTG~G&89&FmF{junoV~;bK^PZ<~;BH zJlFrj`{lh}KJa0)*?XRtr(0Q#-ySmc-(A}?le`f)d z&ODM?|9^_ZLeZX+vR!v58*Yug|Dr;68;mUJ12D4bote*Wj%)-NS%DE0ha28pQa`Y; z^vq?UxpU*r*Ax@4N|6?MAODr7_1378S>2A0utPjB2b8;Axx%$O_-dO$ z*xIganxOMR2QR(mkArn?+|A(P!=*#vGgf0sgXOvyXUyc}FP6HKsQa2(Wr>f+Tojnd zS|+-;v{VvObvg#K2CJm-o4bez{(WBn*7o7cT+n#i-<*-+Kb`v#9vS%?i@m(OLigA2 zqs61eE=oprS;i)VOkh1Qup2Rx8EU>1UFyA(7#tkTEK+q{T?xoiE>g2?+#LJM!hnty zEdjcQfsgp50mHk_e*o9(z%^1GG#!KNWkDLeMwxAGZ8c*~ju7YZS8?Z;53QP@#P#fy z)YM&FB~6ULUEr9fn&HFO=IJfaRcEul=npK?4lW<3>8O}qz-sL_Cn*7K6%G_d?LZBp zJ2h%Lm@a}*yR=64ZIOG>MzZU)LqDZUPJkg)TqQG+&A#+5%M=d{HMJQ3Wj*WbpBnwpu{q`QD9S1hA+hLu$jPfAcIXENqZ!NBnsj z^%;Q86k~zDkNAutYd;4^s?6E9c6d3nT^OIZE@89ZIvMJ5d)WBXb*UD=EjZ=5I&8d5 zkxDx8!_gQK?EIV;u2`BmGfrimEi~8n459lT81{ln+#a|lx{v*MkJyuE_qXiwF%9@n zHXw2ZlvBTTTE7Re3-qF(ViQNgcuA5< z3e>DN(2d^9$?)`+&a9Zg=*ygB+6aNzrvWkaHp*Q@K#<$zdZo$8xy8!`T(XN(TYBv| zRGb5m#+@04ll{)?6mGWo{l&xU0X{yRc7h=AYP=gPS`XWl=FoCWn%;?V`tZ3HeQP44ekMn$ zAev>VTnct}x+Hq3jm>XzDDc+nb|b;qsK@>6EMY|;vH2}qaRqR6mX7bCt?I|-;zw3- zyoYTOmXay_gAEQ9zu4(Mo`hob#eha_)zo>VMOxcfaEyHRqUg{mCeep=DD1HDGGDdA zXWHSc$o5Nvt5$_=5m*618P1f49+U%P%E_7bRy1Bu!nzU8o5ij(#q9QeGW^NDQz>=1 zN?qT7dbB`TDR3=;0QkcD77Hjld}9B&9?>ui+SJQc=E5 zf%4|TB&|ZTE8m1JB0e>nu`T_*T!(#)4jP(Y{{G>O+4ahjlKIkSyL9?ipZ{1Iq6tk{ z&8Ed=1x!N&Q&=S~PmY>vj|9RvWa1ywsY_c2I=yWmCZDvX8*=89@bUvs%oyII%K@1=Ia4B4M#KauKMVUf6x-rQHXp7UPEI49EiW!g))6FbX7al@ zN@e(9CWTNoVG*&kRYvj})YC`Rq^ag1%?D8F^ZHaw>u4oKiD*?FP*ePF|7wl%`jfXe zKDVzdx$|h8SI_H)ABd69R9=T=Y~TM+l|;A#v#V4-`Ta<6iK(4-^Sj14;_3>Q6vJe( zw3)I8YoiPY{gdhG9KR=pUsT%go%jZ^oWnS|(tuK7BYEi%G$J+c&x55iMqDZPR zieHNOPn&3J%!hh?am&R07YiX36x?<3ky-VjD9DYDhowWX&B-eDa+$ZOUimgC1%B;* zEc#h?CCS-zDpGbwlr!>s*)yqn@#hA&Dvb_|c%)v(VXwVw1qSnWUdVcM@VEZ+;y1OD zr*R&?;;o$}q)h@>_BbnzCoV5grH_g`cGt}Vq}P88EMa5+_;CMb!Fo2o7mX=WyXMc# z(kspEG{)zKtCHE(i)gcOFqUu>rpq#dONf=&9WEut#@fd&_Y5{^p+UIuP|01x`e}07Iuw$SC1oJAbY9^=1|TVUb-oa z7FV{=33b3r`5eZvOzBUjJC^QWIsmXvYZ#@6pJ}Dy_ihyD|LX_J z^<%75XaV^DU!9blWcMCCPzY|E04w|khA(WZrt-mXel;=iR?GK}+A@4ZZ{(-sn1$v+b&nX2vG`dG(AlD6F4k8H76|MH+(K-D%l#7% zB=c_Ir|sHX+84(SciOLs)W3fHAKuvF(^uer_r`$TgX;lXR7)c7+;}XcV0=2`Mg0J< z-hH5tC(RT0zh4TctxJ`}z5~MC?=}#w`O(FsDnH6VdW{T%Jgx2uaaz0JuY}(!D z!a)>6w(Pda6>>|C>*r%+l>PekYw9&L+%CIi_maQ(mMqC*L_pR((0dv*9{UIPidjSk zDv;je{0sM*l5FIRWuiR(^Q4%A+tEi;Do{TQDl1374he_?LI}hl(mX}~`>}9=Jj3G- znmiBHTb4m!0FdJVN9|Q8OcM;Q5z&YdGGSB8av|Eod;MmC|IOd<*?smmuqgd+9QFUy zIE%*q82#^V>-9S2pT*qGsXmWpZ@M-G_tll@_{Jf;%yhkQe1K21p6K;26K}33;vyFN zev1TF^+6`!HdznBD^^r0%%D?-#Ut2V!uAP6r@J^k=ha0y{euH4GCuY)*^ND5c zdogs9X3Yn3aAzwzCT0}KacPHuG!wP>X^R<>d|sAYHxBEyh==8EA`G`nRgj867Q%b% zK3uHX1&av{CFMB(iCz9-f~BLYs{+9fYAfw{g#;8$LFlp?gCKAMt~ZH`4t-7(3D?IF zFgxdVAG>C6Ojk*r?ypfIp<;ys=UW%FXev`A0MD{#L}k+DXk&&B&Tqom#%!09s0r6< z>(yZiQElfiU_zO6USXee<#_>IgEa8qB;dj&V7K;jsjEN{a%}2is(%efkrQThZfvH5 zZNvgpRHE6=tV-$9#5PXUF3d2gj_&SAdHf)2yqy~;@)jZ`TBNc2F)*-Zx;8p`BVsOV zFsMUA!&-eO1Yf8NR%X30AHA~PBLR%6SjR@K)#6Yee|5t0(o&~S5RvZrCy7BSW_ib> z4Vs~*c*YRk!y4A-KS;RV*A!}2%Ql@H^3=O5{glS3ER06Sp_aOGy)dyW((Q`r`i-5< znG-WMXD=0gxD^2xhymd`eH@>DF}+$tP5+6OD~xRwE-!YQIYsiNSpP=yZ zx+s(9C6k1MjZPf($$$6YuNj+Hb#SX3P(U)Os>Wp{>lUn{DD_zv(Q>|sLP45L2g=L7 zMmLgeMbPXY z0@Y`l4#YO0Wqz_J*Fde-GU^;ga>dGkxEh|EOvV-hea<=MFe&!K@}YFt{rk5Ch>!Rg zH^=wu2n`JGU|>|8IGlG}jqecq%8V5q!X_omAt;#!{(khA$Bj-0k>8|(QHZ3X#hHex z0&alVTZp>&;Clz4GDH%xvL8ddpV?0WePdjnh0ca`P1S;N2Y|8|RrDs44HLo3ejz2A z7S%Y8C0<{WDjj>&(3wfBHNIVwwgsk^M@iLPSs6^x2p4B@%8`#Nl&hbq4>Br(Rc^!_ z9s_&HS(Xz=-K!{>;G!E9g7Yb8W@UU_5Vbr_724f&!s)hRcV(MNWHC|61Pftd*OLPd z$n5)1d{RfnGbS;4DR}giDOQ%2-M4leva8bcYm-~kAo7c`#c!OBs3s6*lDKUue+=yq zOxrp;bLS-3cr1^`tTIn2Rq~BU^H2+j#0SD)z|%%XfXH zf-@23@1)uE-n-#q)864%nb^jjp#N0xA{wA$;|$|*j(L8{As`6c>+cW)W<_$FZq}v0 zt6XHgZfi#q)~4BLVcy=rLV?*(!$}!IDzBNfCx(uUU5-v`8^+d#w9`8Mh-p{2wc0G` z6rW7;jx7L0S?fWMK{$1zw!eA_Pnz+6TrkM*6Hk!yUIs%_gz|~~|M zt_@)@fE9mN1i@Dr_C-WWuju4CktOe^5-HIL7K)}mxV<8(^m#lH4iTP-kdmsiOVZ#J z+|&zv!3I$5JoQ04;y)h+u*x%lRRXId<8I8vZ6tsR1#*$g|IoVE-z@kXuDjPMQlPp~ z(awQ{pcR|egC4m_5O8WA5~wF4t}FC^tKS}inGUor(GKB!|2Lnbn3$B~_jeL9QW@Ul zXloHq;o^3K6}H@)uE_0I0@o<+fk#6FFDNLL0$kR4cf!@O9(hpw?}#j)2Ttj?fJB}9 z4K`BOF!-+%*v^9^vdEN(8!Y9$1~|}Cua3N_a8H55n*Y)JKAr7vBt8b@8yt}3OZz_v zg#)tC5z%Y_$lBq6ter0Qmr^hskO3!=)Efgo7t4o)`m~iJ9w@SHw6t?7jbN52~DmMnc31oNKg_JISl;e2_`^z?KzR$8P)E7DhiaQ8s}$N5*5 z%<83&t{(y{w;z#e#Z4yC8Qk7q^1%yAEz-J!Cx}Q!Mt1mL!1M!;S3^Pm;8EIyKd-j$ z@+1@q9P*d@^PBBq^EboqkQ?BnvXU&;-?GT-3H?8|8T`dRZa$XQZtZF9Zs6kV<4^72 z&6UtUrHN16gJ8UpC*Z$vE$5x!s7XTN>~~Wc_QN}jtm7-gWnOm)p9@Y_o2zU-xZGZ# zhlgvNucsVM>Rdn#!dQ*QU#*Q4P=iVZ^k-X9zTaof1!B%VJpP;5zP?8q;G%=pj&FQ+ zGc*9Cl#)PLSU&VlJM5lIE&TRyo1s2<@F1~AcI^N-4l|8odN*AJPWG+1mN$Uv_dXGk zF;dVmx6Smr!wowWh^kLcc!yuGvu@HhfWdo0}Tm<+gN=1Xwyx|~tL!fky zm<~dxD7!kOLe(fBQe1mVYgHJ3m{&%bqz|afEX^rsIl#KA<<@Zqr7Z7{p99&yY0J~? zf7$FBK(Ol#vX6N{Z>L~YyXy$z78<}rNaI#_5)vK}K@NI-p1)(u9>r(7uxTD{%`pLx zszzi0Cv@QMNliOov6?6~jBpWXM<#;QF9E#lvJdZF2?z!l6SvPrZf`65$^u<5EU6fCbk&Kaz=(J#N& z&6^{_$bBTLi#0dc{~_5r!m|beKvJ#Su;ZGl{9vF$f03tCCnEClC9%pNd$tI?kpf%= znL=Kt;faYGv)S~^13~J*(yGH)KfI|j7kG9&=7k-A!YWG_aOQq9>GGC`CA^uN`xg_Y zNAV96Hg=oa?;NBR4Av>p*@;$-WPrgg??R@Ehf`Y87sohLl4(VFGj+C6@oYvl)l6Lp z<%wb;Lxrl4qn}2^Jyzue2C}6;^?u$2N?P{;FT`Bx=5};zC%sluPOdt@XoQ!Azg0CQ zC5GulOaLD3Hkj@JQ7zrpk7I|?#qIulUAuQUMNxK8cI(?jNzEAelA{r3vLBSxdT93 zrLms#Rio2cc>@iMSC-xcE1AEE)VfVei8*xrW+7j#Y{A>A^YKh!17?fXGsAco* z5y8>gy5%3zN2>6e}jn*zX5ZzWZT&AgE*t%IRQ{Uf)lQy)vZJto0joC)A-EQj&%x?GH{{F8)v^05n z<@$ZZXBt?*zbFCJv+EI^2RR0`FkX@BWeDQzy?*^p*4!ASfmY&;Klrsc25(~l>lv`Z)VDq4*oM?xHql`NQCnE+Jm9R z$BKF*P7LH709&AZlyOS=jYr=r4^YN%{kqa_oKJ2iculGHkmjaW5EoQ`d5@aMHV^(mT80=I|z^W(ui=? z68eA*z+52EAV%dG-BAle1ibs_x zGtlD=^J+loc{RTFU}GO(r23N=fb@xBIA$0gk@`t1C5RLRnrsQH>@O%-DTb$4UXRAX z2{%r}C6>sBb6%nikb89Bg7f*cNRR(vDp3^n4wua~+6P_T2|@DiZkSx^ecl5}2}G%D zW{YiQ>^~CBf0(d_X^pA$`^X|`XlTd{?oh`6mMjt6pDIF*MjQV0b*!MWt*rNx`W24@ zwGYFu<#De|ZWGV|ZU^z`-yr@-SKx}xCVv{z8&8%0{yzI&(S8-?+t*U0XCYr^8=V$? zK~$H&Qo+VM?n+~(oD}v~YokT#$*tu~;KN=~B7^^k%GJB0I&h7K2!uAY26`V0SlqUD zx7XDocWhc&S+Vreja^=z7C01w*Ue9#kewR%>g}YYO$vF>=ilV%B`?dzNu=P{>1snw zrc%i3u@)sA^^*BqD(POYjiE(+zgF*kihEMRV)d;S6n>9iw>xd@GO<~@K(L&jpWgy; za%9F3R{X&!#aO2}aIY?{_IT^_3|RTbbB)PStbVkVnXQthPS}4GFS|cY?Zkh0ecNsa zyI;C}IPs6#yh<^ds8do8mbH|eYdWM5V0c3+6?4G|?;?Vz%H1p2463uz;>njC zYNr<{pKc+1o%JJ4r9Z;MXVkb*FPh97F)|(Dwy1>KOsPrDU_3mU4WM-T>`spkA^WiP z>FTV(qL`yY25mZ#g5&7O_3qI$WOlD*maSgz+Z zqwg1~q};KYPODl~1&m)JouT{U5}&b+2_IU*sloGm+XcEh!2M+TW4L}%&aFX+#Sx_D z1X8Csitg_2Egp@XL%%y#&pMx;EVC>`#j%-fw|*7QQp}NMaou>sb(%{~WV2SH-xCg; z@EU;siWY@w170X}7`QV`zgQl}4m=z87T4xWs$(P)V3y_BLJniPUJa@7=49pazF z=lKAp%JORNYKbes^+lE&r&jWge`Y2{f_b)&Beb?GmEmbg^GP&FZyrSp=#==wgE!s&9zsMXAt6CEA8TgEX}9}fj%QQ-tg}1L*0A$rc`7;x#g4x{NGZF_ z`CD|kC@P^NA*N*d)6UEn!i5g`r$WiGt?&s%b-ZGVMXKIhRbUEwbis*ZQNzB?~zo^vl{A{h~3~*eFb;hpCT*Ya(O-D!Zf;i?} z#mdXe`Ra==exf=YRE_CQL1BX(`z?HXff7q0ZKQp+TemUau)4p2I@i7w5$RZw&-ndm zA^2eSTg78MJSv>;ArY8F?2;NLiGh5-ED$k6urNTW5nRXbZzl4+OJsD_T$1&K>sLPt z#sx@wnt(Y5s;~M!SuA(e7fITf`u0hpFE}h)MEgV11a))flH$-o?0%-@>RcS&A-8=Y zI8{9ygXk@ueWI%?{*%c4)fx5Tx?OzGW7l{1UFH@5feIl7Ed%=@%H0(ve3`iS_1YH~ zm#Dl`)$yr<=EwLmpi-R>t}3h8)WdpT+Rp)IY| zRF&Oq`Zr|GP>!FWOQkSWQI=ammiN;B6@q;Vn_CAqm$G$alAlfC0&yg_wuU7V379!TT4r;X>}~YU~K2Q6Xkw| z$hFj!b~^yC%VLN!PtmAH|INm15*>!L)B))bVsX(zOXQs+8S`~beN0-(Lm5dnsCJ@B zJ8jH{XB`7+Q}98ruP%J`YKqY8iK&leWh#`*N2Fw?m0W2Y^R}x*=qpKay>gRgsbDm{ zD(%(9Ttv7D$~^Gl6s1=j8{`+W?|CPDX_d$j3t@44RmW)#vq$esaKvMc<=hQPrj05ecH&5YDFQSq7%;xpU=z_aPjFw$UR$ zQYqnA>6Z+C?y>z|kUv3Xkq${>rVcYpwMm%H=@VTQLt{@_FP?YPoM#!D`DPr7OKbLb z_+-uJ+3xj&*tvL!U9$B`e~W`te+f5I)MW`fhuzkdpN(?9&HC6FXUt)v&1J4big>+q zR2(XIlJhV0>}02egFWecRm10G*0WAqeJt?W@9u`!=%_b-H=16Pu~ml7`jcVVC>|yz zD+<=5w`w&PT>|$)HWMFFt=o=n5HAjs^U}Z6E20VSq@8Q=Bfb_}g6H_AK{?X1Am|(E z#{#pmOa&Eet6t}oa_fqHyvFLZq9NSfOUk-x49TvCTpiuh@8?t!#8Q=Gg=|3|JU-g% zIhVIQtrp!|7Rj&d=5d7dt-Ph7&X%NDunm zU;H>^P_3h1LBjNwk}JM4Py4R5bXMo6_~OT?(>1{OrH! z#&ldIJF4mRB$F>L8LomzlgN~-jY00S^6Sjj)Kz2?4BwOT+`>X>l>PNFS~XVZ>L;w4 zN$XuS882!+WQ)d*2GtGC9m)g%y(q_4etl6jmuSefzKeloehjCr5u5ogW}p-M06%H@!bg{z{Sb74Qp1s@0zw)m@Axz~+iZjFl72nw?T= z%i#H`bIr|d3Hm|~Tg#8vzpwY6)1KJ6YDzR%!M$S2Z2#J^>SQ zK3g!tWihSgMYY)~W;ELfuVCN#GQ_Z&g2ya<37LZb$kab-(r9oz8YgM7)6m<{-zowX zK92?0@%kkMZ4-(sD6=t&gJmAC9t2YBt5)O2?)XXQy{(^;@bYPO66@+L0#ItI$uM_~ z%TQqNUUQ4))1IMLbk;Sp_iTu0tQmq4RVJ)Daa_2FMY8VELCSU#&tB2tpjAc(Ve_B2 z^Gmank)_R|3t5*(czPmotp^S*Qb5AQt+Ik9aKjCW_pAB(x_{M}t4+UwkSnR3J%==r zPg^JDy9J6_MTK3r+5e{0TtTeMF$h`^BD-hM;u9>OQECz93~_?~UdBIe5an(>J1*0K zQBZRiJiX2sNi7B|G)ys`S6j|VzKg^1G)&Z>`b;LlPHJK~TWylwi0o-IYq>`V<(LzF4uVT@t?XWPe)v zV|w+qva*Ddl6T{TA^Au;AsTX^pn>P#*XcO9ZUng?L%xy_ zB-0qN#dvxmmkTN>MLm4@P~2IguyNq?b47D7l&KE4X{AF;5ZvC$mr+nqh(E$UG@ixy zG66pi62$5ICxY*A5m9MhX=u>zl$BA9`|@wwju#j7awb|y@I71A|2t4yuu-aVEQzYPu@6CDgwT7ZPo&5@S@gIu&~ z^a}56bZZ&FT zq&EnWenFO$lv*^us?i2G3wZBGqAx4@1Y#Wamb->|%pzh}R`f~+zHi*==+29TIVdS; zD4X_?5aH4q!dfZ#xHMR}U$Q#Iagu?M~6Kzf@cC?DiaLn%+EFzv>)kn=0l_ON(9sKA?fFbc5>^U6AGDG*R zmU|*}yHwgyNn9yFR7b`7K7@DN*>>jKP1jZ^s&HD3(HJ_CG+JF)E@_Le)InR(sJ$#s zFiokzP4Q`>{2I9Rhag{zy@H}=X5-LoQR`FL(mz-b{^F)WiZ?c6Hk7R_T` zuf%P$X5GJRlJ87N>+YIy(d)F;1(K1U5LC$zQSzC zT^yK|Un}FK?jSG(HrpG9~$XrVc+1gdDUaflL>>#i$Lylv8K+>D>=fGgGcdg|ad z1XG0&;+e(i4*jlWz0g_22$hD~kK9+&R^8Qwd_6&mc_6#)?n$#06SEt|14<4Km1~i8 zrTyoZ@uObo3qLwO{vckUSpvQ}vORqIP)#l_cu->;(Cs`h?$qQVxlq<_u8n8YFNT$X ztSW6M*^3vM5?!3BZf>g8W=XX)+KhyS@SW7Mqtcs~HD)mcVlS5Bh$<7VtRt+~qc9EB$S%%#zas}K{X{{*#FegFX8!GvxHGoXD0P$#$hCcDp7K`YvZN zd=nC;%)6O0y35eB1Ca@5xn-JpjeEhH{Xxvi*+lo%MH_1eYs%*#Ii7FOXN?9;rX$Lw z9r{UR4pf4gl^;GE2e@K7o4XRUlqOB%*d{b+Fs@^8sWyt5IlNe7 zJIr<+G|cI|y_`hN(?d@Vk$Ib$XUo6kww-41*0Og(zoVpwI6oyk=?e+s|nMlOR&9y7mRB0i2>)C;jN682o%D2)@SPFCUGB~T2Nl@ zP(7bAGUGis2jmddn6^_;Ag6T=@9ucB?!tGO!$+9q&{$=ht;o3AZ}!U*N~b|5z9cU6 z1!Kiga4*MfB>&#@Uite)yIEiR=qjxr38b*EH2C6UjGDabss@Azn8cSm^R0)=bpn?X zdfm=FwgSVqRMjK!ViV5jdbYNY%f#A$Qn|;zSEJ#-_$Af1ELzVzV===9t$NDuYCd1= z(W)DxMUT^6T|xNiAwGWBNOO(G_gJP|R7sZQCL9RfuGtL{wiA*N4`X3rVXwJvS3alU zz$yG#BZo?+dRHMq;nX6GmDp@wVwdjoAew4{y2AB%I^Ct=^X@f=dMc52ZylU0tM@hg z;-4cig2f|a+w0s0U7k#7EH3c^z~+93Riev>o%}~zo7#ZsRP_|mg-$Skr|td0DD$QR zsC43UtzK3nF}4HdxxmZWyRw=G7e;+OGWPbdK=H3`Y)v+3uFlkC;m@a6-=Wfc(tKW? zGLi5U_8^^Owo3nCAgM~bXqLiwjPYXiZJi3}QBFEXCzy6Deb;J+jSXj6>8sXnqSkL* zgrMM&#DByEGpO#az=zlw>qXp{uBz;5DH(j+<*DM??7l{5puu1Qw0W9Y$#!ejG~p*q zvcwIfIjQP5&Tg#JQ_BuI{|F>DoijShS8q@g(re%zvi0Cv%*27&n!+aDB4RQuD=qny zGtxXxRS0;G%!v@7+ThLCgbkSwjE@q^u;1EQ!(yB%QL*aZzt!5&xZM2$PWEW^PR^-opRCTsyoSsG zf~DUXHWj&ES@6?t>Bw!YSd(2{b5!|Y$p|(RtzPj>7$Zyv#Y-+F!5Gz>`nyjD+fYOGTdS-QygHKNc(zCmp-PjNAFK7 zu?2qLlbGh%cKr3}<8EASZPQ+{lgT~iZdMn$M;yhrc-a`^j5KTOr5LMXd70%^5(UTs*cBDftK&)EVIY&bLw0Q}&z zG+BWTU1Fw3x~=-48F73=Nk)ZSsV2dT|CMydYeHLvF3 zG{YIhKJ&c6ywG@#VwJgS%fXmnA1vO3zB@=kLmjd>=$#OFZD-Oi%l5T735H$JRyk6qjgyxwoOOwnD*!e${l= zalRgFm%)z@;wwd7{2=4nN~riU7t8osUz&RaBBd`(O6}wF)xpm{n>6>PR1 z#u*AaF5SsS1HZm0=$e)G#ahQt?=B~Re7%y=vzX{n6Ed)gC-kgDy}4Jhjm3ridy}4X zUnkO=evciav;`@n9I*Y%xCg9i%&A$9S$A;xu{|}6qj2U9t9M+n0WPk9OGTCrW1^nh z@}nmizmi4^FIx6eX}1pOs|($GS+a1j8V-7$(y?)eI_@SrE^!U{SG7kefAVWb8_+j5 zFn$NC9F;an_4*-}@R4LT&kNlTzRw9Y(Y6*K5HG=oO9LI*XKNXHou*0ltvvnxt1ZS* zw==8kFI?HrSSE}>>djF_0h`In_?P>snV^%}j(BiIQux^@%f+~=xmnh? z-sKF7=ld@m$f2;lBZ%%&K+0p3>jK{0-;QE9m|jtAHCd(x3defGB#t85y!|bN!y*e1BBLJWuF*Bc>LUYnTxpZXZKwIYI4#wppfAS zH%5P^qFk9=-pGXPniNhqsgr3(%>k5$Vq*GuHBfz0S2|@ZO#*e+IH0$g*zN0K$=J@b zi*9$VtxP08o9Ln#tzVQ2X5P2ZWTy+HX9;f z6C%DJD_a$NJg~xHUu4Iep>BG!(*=sg!`bBntd2%Dy`uu>IJYmHN6!@_Ip z6-LiuFQMjYkxwX^98_iq3LLXU|Dm->GcyYtR4=dcW+2w7_GJXe`ACf}m>Q;%Xi2qV z`fh{z@&lbGnwh1V>r-gyGum{dq zROZh}VD%+)!ehi6u*VVlP-6@^7((0RU>f$r^PUn{LlEFMUckrry~Aq^2?uCYGM#QC zq1v)Kk@p|oX6fm+>p3A9*jttUgqmQM;#keMf2eK_)qyFOsG6#IU{|eI@|d~p1yUx3 zPdhbx@u}zpPlc&aVsJ!v;^A1cQrXDdi3uYZ7|e5dqhqU20w|d;Y#w@s6a9nh9ItIn zT4sOVE>>#qc!N4H$`PuJ^khplIZ?;C(WtBS67hld514sUuSu5T7_syqXwUs>c_DPj$r|Gc((83~F37+X0hx$kcDr^8GU4E3p~I%5Xh5p8K^3yuZ|) z|E)3!d%=vdadj|{*xGor@Lpo(LZcq8HMDjozoqrlCG#Mh3152MX*)D24_F*ITw}c? zNeF^O9l_iHA0+^nA1NA%R(^lJh3JaGs?I^%EG3D8R{0*GPQL0_z>5JjZ^2ZXj6udq zjP4Edx$b0BeWN|^+?lsOLY^2?3k@h+T+4z4H9_lpExz>HUDtoHq@$;NUgkAKs@ z2FHJo^DRP?Ki69ptpD(jU|stiaJ{UepIvTnV}kSUK!Sl48;ErC#Z+d%m<7qdCqX78 zOYI8>`3&yn9!t*OQBuIRb?#LF(F#g|&#L^yy=-EI*pw_dw?r%9eGiQkfj6_JMK2Sd?0oyE{L!w7K-3zz$E6whQN>Su64V>iw7Z}S zIQxT+(tqh1&w*KfJ0QgChFVH6l^>kr>4!h*5r31quIR^n0?SL~kK>w*lcoFO>FE5c zkCh%_iGuzQ{-Xy8jQG}%$(OSN!_4G6P?HhF{-Mbk(T~}PMtlb8xlBhcEcJYUT8egp z%?=+h4y4^@#EmimX%$exZ^^WI%zB~QlHf7}`Pj20! z&60R2^coC=2r_8yF{-xT$qqPpWH{ z-4oF@XyTZ{&O1_8PiTJZwftVbUo)uvTBmJip(1ffe zQW{1$Ba4@O&LMZ=3b}F%zjvSf2&=ijqL|XjgcrK>qPBARr6*?J$lJ6c^w)983Uts& zHrjrRwPGO|oFVp@tTU_0nt#3;N`rOmPCdR@Hf)G(t} zm7|udaa8LJu!m(=0PLfqbM^Sc`itbKlCIoH{&MRh(G$yMyYNT%shd~b!ULJtFB|qL zCzQkK*?Qy=?$b%I$zSmt?I??!~Sf>K7O zqqL(!?f%wdRh=6fq2L-EZjl!pZiJQC6_;}y1~nqrcLn?&oCuM$7tft$`A~jWYJ;&k z`6{!peafVwoqeWscDDRg^4xaS`5^M63`NH-H@D;0OR&=xGdHtVm0icBYD?Mb)q|F>3@l=F6cqwd1n zSncuSsx}|74gpaO+Q~`gVjVYC#ZnoqvyaJl2_}2=@`~_7f=_GF@4nX+UK3PJu2&sh zGOsAY`S^3dvh~Z*;)<*n-`SQw1R{sx#9AMIe zY{htn@fzLLuJSAjv2K?r8A5su_9JcElbXtRmhwn55z0%tlJ7qxN<)fpWoh&2BR0wr zmy4`FOO%5hs?#$FP|FJ%`kVNw&wiF?af&6ggG45t(Zmfl1=zz6a|m9OHa1vM!z+ij zTsei}+#=MDqZRJ%M2PC6x}C-$eW8kOwsrf&Y83}2>pl#--`%vLGWxh=QHY@&Z)`K} zdY@%iAM~nw`G>niCXpE(6CCT*g1hI|~c0`l3|ijVBEFA#3%=nthU`3ErXS|l8rv!}CFvz@csd^}=$K;0au|ueI}0)II*oFwzg&$=6x5l| zZ7D_}bS%!b*pdq@KA|~~42rZLO5bTIPw_lUfwDUn9=X#0ReuDx(OK2IzE?7>xKt}F)&17&Noiwq=Cbp<1rFOU zM-DR~rS)!5oo|;|Iy{Iyv=-yc8L*RCms18VSUH4(mTD4eV37K8|M!9hlhktCbJgk! zy`yK#nSZU1UMs_lqpjIDoJMs1v_IkO6_bY^274t@WHt+qiNPIg^oYn@}~)zo2u@I;_b+_{pVF*c4OK7G>mupSOt4OypW8xcXSL=bwiE(RW-w zcb&oLfGyyD2WjRY)id?RzRm7-QU2Ac^RoW!DrX>M=*X35`W1f~RqU}yiq2xo)*Pfl z$wk*ISRmeG{)%;I<#Lj@Z((Pk*ZW~<^* z5puJls?o-3hb28XMR}+}b^dp#KW7$AL2;>98|0S)Rjy3C$piVj>y+tCqO0_B<3v<1Q-V*I9V0@Y>*IP@XZHfdSx~mIEL|bK&<4x)2Fj zAJ6irDVs6GuuTyXV~_OlwwYVRi0|!NAO=Ctd4y*ERJm;x(o|1OH@dDJ(ETiRN5K|5 z3}MFJfz0H0dtJio?M{>7N1dL)*Pu(;eH?5tWaIyl5L2rBZ$r;(_qSei!03}pp4x(p zAo2uH`7f~Z1>p`b-*zJ`wQb%20br)Z0961!O2Ew*`>g>QlhK!uz4_uiC2$FOw>-Up z7SLFe0KOYi4aY&WL5|-eAhMC|{ij@~dc##tling-K7)mL#fUio1mfPnq2ROsz!gr$ zkAB^B)7L2mK0L@9I32Q8SN;K zh3qMu`aZ;AMaw4c;_ll2j+j10Mn-Mzz@Rs>HFI};}$t@e*eo2?ZG&&0`?~_;a&U}pgjWUYPy2* zW(2Ayd_($NCV{9MZ~Vds?0ZKh9e+O?!3Fn7g>O&3)+M)i1LIl%4A#o0ZVBJ*`Y-51 znhO_gpYAF;{QU~}+cenE&!R6n@y5S`(en&SZ90E_6G#8Qet4wQbA?h}os=9Ug1dL_ zWYsy!c};lx_=ulX=X&`dl+!^?Ml$UmX1sl0ka?{MNhNLJfO z0Sz#J+n2(=7=KY=wF~w#u$S-)xG+SLx;gLqvyso&M5Cf2=nS7k5Ir%-O+=w|lpZaf zX1^_=``7qfk5pC|W9uQD<^NfRq829`TNjEd9^MijcE%9&x z*5@3j_E;ZkJ@{{@8BFuJoR3b-au0)uGLwO@u7o(_$G;8nuseU02ZSNyxu(>$Se1eJAVfuY?%Y4Bm)j_qkDiG=;S%Du-FOs>n-11~@Og?{CMYhU z>d<;$7>QH6m@Z;{uG{o&$SKFgUf@fva=X_TJn*UFDVv&E`P!@zhDAe58$LYzTKx)W z`y@3q5~Uj#*g-}U7B>#Cc4`qYGYFcbXAfddKdf*Zx0^S zG1AjZTQ}$D)7EPTC3O=~hfJ;~9i^qEJs{p3%d#&2h3laQ9^hX(x0EOP{F0A>*d!}8 zRnP~>T6{wYbh=pVW_eYi&fd=3V6*r}bq=sm`Wz}YVf$9&$UK~L0=Jl2iMuufQO4#R z8R`^$diBg+$@;KCoE!gkXrxNy*u+(AMsPTDhS-Rzz(SOY?)JP;cw!_yXf|h`pT?+6 zxUie|Miy(<`^~0uh&k#m8Si%Fq;zEAYd>4~9$fUX6X;xCJ05LPM#%DsG$pf1@n39M z)n1IM8*OBmTVDjK=#Q#y6+U9IFFijTYqp+cKQoq>F9Ul+waq>@ntv&rT88+?&Oli2 z6m-GJdBw|h+Wwhs^q)vEwHSA8ZC?^l){#*iH?m3GBHxoPBqxL;3D!-s|VjaoyKB z=RV(a&Ug7NLXD+e`Nth!#SxZ$(5+uM06eFV!9jGZrsbk**#*}PZ4bho{?2LaqVvl< zzehnAq;~001sFeOfq z1Ti)ial-~yhQc?w9-GPHw`rJ0*loH-tE?O56+)i7cy_dcPB{0dPrrX!)s#%Fh*S>acvwvhw`6e!oE>hUIV3k#1+X?pdAurZ+ko15cWnK{!4uZSG;aq0 z(@hb_xvRB$&uLtBFGN5x_qwC@XrV_4tB_0#`(!SEq3}@?Pw&~PIR~w@Ycn%5=%lK3 z2T$Qs!W0{L2)>R`M%$*gxQe91U z%h|P>I<=Pt1j+&pos&h;){+mVmHJ~Aj3l=|FeEH6rudBeEIRg9CwuRVyKc_+8~Hg` z8`=rK4zv}UAE9$J>4_f7iQ1ick@IFOwTwnJhx)z7^+~UObw~UvAWB37_qEXr2er#A z87;Q4MH73o`p9wxN_@dYr8d$5V_|d~h{LC&MqKU0b7!oDolDT3Jq@*UD}7a^?@S1R zW5ND(BdJf2RDL!T;fu`26Wk2^A?F_qj+&)7{LJ9sh%>e&=9Wf&85jD+6;rB<(v7z) z5GNNH?;D9G$xUIlDP>(_EHRe2nnNxjxiVCa0C z@-47yH*s51KHaaLlXItdzAxIUTI4JvBX)=aG9lr*0>Fgc0pS--$GYj&7XveWpo)y%c(Rx(l^Z zJZ1GPUg$Ez$ME%X{>49;61GS#>iBu_nt09#HH}xr(&;nXs&_IheX>QZUibEh0k9d; zM5xR@8k(8LUjI5x-gx4wP1p5nf^)!}evM%*2G7#Y@7jEIv)O zMMHwOkKEk7ukPN>V{b@CCMPpBV`X@h`tQCFU=V(sg>4}&h|Dcbpv%DLY40*HjWxzq zudfiDTUkk+HnO)JAv8b>l~p? zMlYK8ziAFt=hxAfY;v>dPX*3Ztx=k?_L<+L2i?qEJ+<~55%E)h_Bg@xH9FvwU;@Pj z{e&y$oihAK56qL_X-e!%!~;n|mmRw-bQvU-evfJFq+a2b{z$?NzzBFn+~3#SURM`a z0g5!W;sBYrBvxbm4-FeCLcqnH<%k)lUYKG*!= z#y*pb*Vbbw{(ZoD_ADlDX#`9QeqP1va8k?p%e0Q{=7Oy788!xevAD*j)4QdO%R2($ z{=3Vj=+4e^LDq3WO8;6|?%ar(iEqZT)AW$&JE^Itaobd%zK|4GDH;IhEt*o9O|Wqm z)hkN#UM!{FhdL&V%4`bZT)+gAJJxIq3N3i^sBu7>0n(O!;Ag!5wM7sMQt!+H%{ELstNV@`a3@^SAJUi|d^npI749Q?DCI>x{{*CZ>{sPF=Xx8HH zu7GJA;{E%-4Qj2+ZFoeN5yKPLA&UMLH3WMU%fJuK5;&6c({KYkuceu-?5S+a#SLZL zS2zpZ2ouG6*;*jAXtZJ=8KkF&Af=;cdF<4fY~yu-9*kLgPrgxgr%uHy@SW3Kt{TsMvFvd&T_a45$?sR@ z@fegqhWPTde}4Y(J-sb^VrF@jta_I5aG^?Wd=^x+F|i$oD_T=f(SEAk8o2QLo*p?$ zvO6)J;dcEMs^HuG!(XOQTI_Jatp^HjeQ)%krSMPS5LDy#nw^J^@sMPns%I*4Xz34M z8v<76@G<^#djF?(5B@c2NkCsyZuU@Mu>(m)4w%}Bu3R1aL(xI^Q0Ix0uG2!K_@P4l z22f}?^MN@v_s}5nQzoeP6RW35LWn;bXL58Tj?8Lv*21q}!2c?Nl;mWlkoTdbbRR$o zf5T@rN9y;|BVzW8&DCy8L~e@^dzzk8ET)*6R484l6X z)8lUD=AO;c<>TWcD>ORrI{Ip$s+pN7+)26lt8#Xc0ffQJe1)P(guwRiKFrZeT84%^K*8kb zJr~dC@0cEZ=I=Bp*W)TnE>wq?3YFE4Jg(HW2sjmMj%NDxo~2Sax_p!IOjnWV95yGY zKfldju7)W_ms~gWnT`{FSem2v&E`Y?GY*y9s&$9My9$K&%L6hk-$T#eD466wR7Lpw z*+x1l7w}?dKREt+=g{IxQjhuYiGxC9n)F!h)5vZ9S`5!*8i|SGjRG%4Ln9{m2KJB_4g@d&NAK*(9jzamRzM~BiRg+=5lA9}=?#S}#69M6f zfw|FNmnnDt%t(oQqFOCMMsoF=%UN)@oeI?O7SF8W&}C@3hjX20UZDmf-* zvSdX(fbjbX#555kxQ3i4^Ye2D9H zSqp)9we=@3y?HM`{Z<}_cFweF8{kcznLb0kHX4hPzIF0SZAQU$nOiT;BiK=)xepxr zlkccKbeW3J5qWJN?GtOjr+PKo>xH8_O&G>jT|cno=L`4<*-^ftM^ilKL~LJn_p$j- zD`<-65@dJp)$Q(_mz>S!oLM_S>3Yn~w&fMpPD%>7F5U%gfjRX&r3_=QMxg&OHn=u2 zr8%;^!-XgAZ$CL@>gT%|BjWbO+I3rfR$fT*CA|Meafi>I zy5@Mk-DF%EERgCHy^LYx<*aZAvve>0)Sesr_%Hu#TkE~@sVdQON*PstzZJgRbN~E7 zm6QoZ#_e{f*|d@}=3>=W*$c53&VtUi7+XR80GC5o^g91+m9>wei%XxMJsofqV^)_m zQ@kiOx{}v2?s*W%lRd)8vEyE+>`RO*re-#=jlGtYfmW~Bz2)NK(GGe~jLQX^t$V}D zH*W?wE(7zqtBc~Q78WprBHQBKnQn0m6O;74b)S{ySRg&dgQ+p>m&FDxj2-tzD&a5l zy=;$Ae>HBWUE&0bSGe`G^%g_a^lF3$8k#yc;_GrEZ+bjb3O5rH_&Fm2Ltuo;SlVj@ z)m$AOX$e-I5Yo=id5#nS;kOd0#r`||ipW9FxUb6FysN!wBL0rWS(WSl z-|V~}R@h=J0#2S1O%m;1QZ1Ua{rabM4`vBc86J`OIOO)b{&m0>s;cr{OEfdJ3Jj~5 zfSElzeVz|!Jw%xTceer8?qLZbk-tkm!h9`RcpX0V6pU%YpAj@}-D1BUh=^Ch)4-Rb5gFN4v!3M9qJ}iB7Dk*E#=ZqkxViZ;WKBHG|}JTg_!j4bVo0Y zGUhb?9OvzXM;G$1^p^Cp6rQlU3BCm1YhE%n_P*=4rwH}L@?_-&WDen9+RdQ`bGdpy zm-bYr=)cw<%BlnCxL#(xh>naf^p>5R3;LGMbaSjt>jc2q?tAUwmt{8hqT=E%>6`k+ ztHBn|B?il&~U&O}weg;EZhHWfYTv|1)YjgrmTj zQtt=>;#OCiGPxz4u^d6fF5a%l9sU6o!;_Jgda0APfm=UDWJRyJsEoahUV z_Tvv_L%sFbQeR6!g!A0z0=`T?r$zvbSL_SxP2Gt^26$1DNj*Uo!=KIO5IsfUVTEsT zB9Gk&<-DJU=AV2WG-_WE^n}MZE}%N9Zz$DLmNig(bSOI_mG%334lpx+fmv+8B%jXh zKSv-p1ahe9$9p8Xe-LWfXl zc^j)ejSC)Hh6E4QO8%o+ZYw>i0+N1oBuP8#N6+;MS#PiQ9fylWx3T=MNl6;(m7-9k zK@GD+G8+>;#Q&V6`Bs|Vxc`bVA@#w!R@I&-=ZO!7SBrO+MD+Y)O{NO}>i6H9TW}uo1d!m)r{%VI(q55)}A;Hsm?qdFrvz zajAE5va_jFf~eM`*MZ*rI)fk(&)Z~im@whOUmm%H@19_{uUIPZtRF(AIS+G5dh2RR zZWoJxjyROv l(@{B*`VIe2PY-@T`9Yk4%7FDv))DZhd`m;2;HG)Ne*pl<$npRH literal 0 HcmV?d00001 diff --git a/netris-nvlink-integration.rst b/netris-nvlink-integration.rst index 8eb04df0..e0d50b72 100644 --- a/netris-nvlink-integration.rst +++ b/netris-nvlink-integration.rst @@ -8,23 +8,23 @@ NVIDIA NMX-C (NVLink) Integration Plugin for Netris Controller Overview ======== -The Netris-NMX plugin provides seamless integration between Netris Controller and `NVIDIA NMX-C (NMX-Controller) `_ for AI infrastructures with NVIDIA NVLink Multi-Node (NVL72/NVL144) fabrics present. This integration allows infrastructure operators to define compute multi-tenancy in a single place through Netris, significantly simplifying management across all network types. +The Netris NVLink plugin provides an integration between your Netris Controller and `NVIDIA NMX Controller (NMX-C) `_ for AI infrastructures that include NVIDIA NVLink Multi-Node NVL72 fabrics. Netris already acts as your Ethernet fabric manager and, with this NVLink integration, allows you to continue using Netris as your one source of truth for all your data center networking intent, including your East-West, North-South, Out-of-Band management, and scale-up (NVL72) networking fabrics. See the "How an AI network differs from a traditional network" section of :doc:`Introduction to Netris ` for more details about different network fabrics in a typical AI data center. Key Benefits -------------- -- **Unified Management Interface**: Define tenant isolation by simply listing servers in a :doc:`Server Cluster ` object -- **Automated Provisioning**: Automatically configure NVLink partitions on NVL72/NVL144 Multi-Node fabrics to align with tenant boundaries configured on other fabrics such as East-West (via Ethernet or :doc:`InfiniBand `) and North-South Ethernet. -- **Simplified Operations**: Eliminate the need to manage SwitchPorts, VLANs, VRFs on Ethernet, GUIDs, PKeys, SHARP groups on :doc:`InfiniBand `, and NVLink partitions and GPU UIDs separately. +- **Unified Management Interface**: Define tenant isolation intent by simply listing servers in a :doc:`Server Cluster ` object. Netris implements your declared intent in all appropriate Netris-managed fabrics, including Ethernet, InfiniBand, DPUs, and NVL72. +- **Automated Provisioning**: Automatically configure NVLink partitions on NVL72 Multi-Node fabrics to align with tenant boundaries configured on other fabrics such as East-West (via Ethernet or :doc:`InfiniBand `) and North-South Ethernet. No additional actions are required to be taken outside of Netris to partition the NVL72 domain, making Netris your one-stop shop for declaring your multi-tenancy intent across all Netris-managed networking fabrics. +- **Simplified Operations**: The :doc:`Server Cluster ` feature eliminates the need to manage NVL72 partitions and GPU UIDs separately, as well as switch ports, VLANs, VRFs on Ethernet, and GUIDs, PKeys, SHARP groups on :doc:`InfiniBand `. Architecture ============= -The Netris-NMX plugin acts as the integration layer between Netris Controller and NVIDIA NMX-Controller: +The Netris NVLink plugin acts as the integration layer between Netris Controller and NVIDIA NMX controllers (NMX-C): -1. **Netris Controller**: Orchestrates the Ethernet switches and provides the primary user interface. -2. **NVIDIA NMX-C**: Manages the NVLink Multi-Node fabric switches and provides specialized NVLink functionality. -3. **Netris-NMX Plugin**: Synchronizes configurations between both systems +1. **Netris Controller**: Orchestrates the Ethernet switches and provides the primary user interface for all your network automation, abstraction, and multi-tenancy actions. +2. **NVIDIA NMX Controller (NMX-C)**: Manages the NVLink Multi-Node fabric switches and provides specialized NVLink functionality, such as hardware lifecycle management. +3. **Netris NVLink Plugin**: Synchronizes multi-tenancy configurations between both systems. .. image:: images/NVIDIA_NVLink-Integration.svg :align: center @@ -32,36 +32,35 @@ The Netris-NMX plugin acts as the integration layer between Netris Controller an .. raw:: html -
+

Figure: Netris NVLink Integration architecture

-.. tip:: - NVIDIA NetQ (NMX-M) is not required by Netris, but it is recommended for granular Network + GPU telemetry +.. tip:: + NVIDIA NetQ (NMX-M) is not required by Netris, but it is recommended for granular Network + GPU telemetry. When you define a :doc:`Server Cluster ` in Netris, the plugin automatically: -- Discovers GPU UIDs for each server using the preloaded GPU ledger. -- Creates and manages appropriate NVL partitions in NMX-C -- Assigns appropriate GPU UIDs to appropriate NVL partitions +- Discovers GPU UIDs for each server using the preloaded GPU ledger (see the :ref:`nvlink_loading_gpu_inventory` section below for more details). +- Creates and manages appropriate NVL partitions in NMX-C. +- Assigns appropriate GPU UIDs to appropriate NVL partitions. -The Netris-NMX plugin runs continuously and validates that the operator's intent is correctly applied every 20 seconds by default. If the NVLink partition doesn't match the intent declared in the Netris Controller, the Netris-NMX plugin will enforce the intent in the NMX-Controller. +The Netris NVLink plugin runs continuously and, by default, validates that the operator's intent is correctly applied every 10 seconds. If the NVLink partition doesn't match the intent declared in the Netris Controller, the Netris NVLink plugin will enforce the intent in the appropriate NMX controller (NMX-C). -The NMX-Controller remains the source of truth for the state of the GPU assignments to NVLink partitions. The Netris Controller is the source of truth for the operator's intent, and the Netris-NMX plugin will continuously enforce the operator's intent as expressed through the :doc:`Server Cluster ` object in the Netris Controller. +The NMX controller (NMX-C) remains the source of truth for GPU assignment to NVLink partitions (see the :ref:`nvlink_verification` section below). The Netris Controller is the source of truth for the operator's intent, and the Netris NVLink plugin will continuously enforce the operator's intent as expressed through the :doc:`Server Cluster ` object in the Netris Controller. -When more than one NMX-Controller is defined in the configuration, Netris will automatically discover which NMX-Controller creates the NVLink partition and will only create the partitions in the appropriate NMX-Controllers. +NVL72 is a rack-scale system, and a typical data center deployment will have multiple NMX controllers (NMX-Cs) — one per rack (see `NVIDIA materials `_ for more information on NVL72). Netris will automatically discover which NMX controllers (NMX-Cs) must create the NVL72 partition and will only create partitions in the appropriate NMX controllers. High-Level Workflow ----------------------- -1. Preload GPU mapping -2. Configure NVLink agent -3. Agent discovers the current state from NMX-C(s) -4. Netris determines the desired partition based on the server cluster template and server cluster membership. -5. Agent reconciles every 20s -6. Debug via script +1. Install and configure the Netris NVLink plugin. +2. Preload GPU mapping. +3. The Netris NVLink plugin automatically discovers the current state from defined NMX-C(s). +4. Netris determines the desired partition based on the Server Cluster Template and :doc:`Server Cluster ` membership. +5. The Netris NVLink plugin reconciles every 10 seconds. -.. tip:: Server Cluster is the only supported way for Netris to manage NVLink partitions. +.. tip:: :doc:`Server Cluster ` is the only supported way for Netris to manage NVLink partitions. -.. tip:: Netris does not perform any additional NVLink management tasks, like NVLink switch life cycle management, etc., other than creating, modifying, and destroying NVLink partitions and adding or removing GPU UIDs to those partitions. NVIDIA NMX-C is the NVLink fabric manager. +.. tip:: Netris creates, modifies, and destroys NVL72 partitions, and adds or removes GPU UIDs from them. All other NVLink related activities, such as NVLink switch life cycle management, are performed by NMX-C and/or other relevant NVIDIA solutions. NVIDIA NMX-C is the NVLink fabric manager. Version Compatibility ===================== @@ -73,79 +72,300 @@ Version Compatibility - Minimum Version * - Netris Controller - 4.6.0+ - * - Netris-NMX Plugin + * - Netris NVLink Plugin - Bundled with Netris Controller * - NVIDIA NMX-C - Consult your NVIDIA representative for supported versions -.. note:: The Netris-NMX plugin communicates with NMX-C using its gRPC API with TLS client certificate authentication. Ensure your NMX-C version supports gRPC API access. +.. tip:: The Netris NVLink plugin communicates with NMX-C using its gRPC API with TLS client certificate authentication. Ensure your NMX-C version supports gRPC API access. Prerequisites ============== -Before installing the Netris-NMX plugin, ensure: +Before installing the Netris NVLink plugin, ensure there is: -1. A functioning Netris Controller environment -2. A properly configured NVIDIA NMX-C installation -3. Network connectivity between the Netris Controller or a dedicated server running the Netris-NMX plugin and the NVIDIA NMX-Controller -4. Admin credentials for the Netris Controller -5. A GPU UID inventory file (CSV) mapping each server hostname to its GPU UIDs, ready to be loaded via `netris-nvl-loader` -6. mTLS client certificate, private key, and root CA for authenticating to the NMX-Controller(s) +1. A functioning Netris Controller environment. See :doc:`Netris Controller Installation ` documentation for more details. +2. Network connectivity between the Netris Controller and the NVIDIA NMX controller (NMX-C). +3. Credentials for the Netris Controller. +4. A GPU UID inventory file (CSV) mapping each server hostname to its GPU UIDs, ready to be loaded via ``nvlink-loader`` (see the :ref:`nvlink_loading_gpu_inventory` section below for CSV format details). +5. mTLS client certificate(s), private key(s), and the root CA certificate for authenticating NMX controllers (NMX-Cs). -Installation -================ +Netris NVLink Plugin Installation +================================== -Installing the plugin ------------------------- +Installing the Netris NVLink plugin +-------------------------------------- + +The Netris NVLink plugin ships with the Netris Controller, but requires the additional steps outlined below to initialize. In the rest of the steps below, the guide assumes you unpacked the tarball into ``~/netris-controller-ha/`` (see :doc:`Installing HA Netris Controller in Air-Gapped Environments ` for more details). + +.. tip:: Perform all steps in this section from the primary Netris Controller node only; they do not need to be repeated on standby nodes. + +Step 1: Configuring the NMX-C plugin credentials +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Provide the controller credentials that the Netris NVLink plugin will use to communicate with the Netris Controller by editing and applying the ``netris-controller-ha/manifests/netris-controller/nvlink/secret.yaml`` file. + +.. code-block:: yaml + + ~$ vi ./netris-controller-ha/manifests/netris-controller/nvlink/secret.yaml + apiVersion: v1 + kind: Secret + metadata: + name: netris-controller-nvlink-agent-envs + namespace: netris-controller + type: Opaque + stringData: + NETRIS_CONTROLLER_ADDR: http://netris-controller-ha-web-service-backend.netris-controller + NETRIS_CONTROLLER_LOGIN: "netris" + NETRIS_CONTROLLER_PASSWORD: "newNet0ps" + NETRIS_SITE_NAME: "Site" + # NETRIS_VERIFY_SSL: "true" + +In the ``secret.yaml`` file you should only need to update the values of the ``NETRIS_CONTROLLER_LOGIN``, ``NETRIS_CONTROLLER_PASSWORD``, and ``NETRIS_SITE_NAME`` variables. The value of ``NETRIS_SITE_NAME`` must match the Site name defined in the Netris controller. See :doc:`Netris Site ` for more details on creating a Site in Netris. + +.. warning:: Do not modify the value of the ``NETRIS_CONTROLLER_ADDR`` variable. + +.. tip:: + The username and password supplied in the ``secret.yaml`` must have "Permit All" selected as a value of the "Permission Group" field and "All Tenants" added and Edit selected when adding a Netris user. + + .. image:: images/nvlink-add-user.png + :align: center + :class: with-shadow + + .. raw:: html + +

Figure: Adding a Netris user with the Permission Group set to Permit All and All Tenants selected

-The Netris-NMX plugin can be installed on the same server as the Netris Controller or on a dedicated server, depending on customer requirements and network topology. +.. tip:: + If you update the credentials, please repeat Step 1 of this installation guide and restart the deployment. -If a dedicated server option is chosen, ensure that the server running the Netris-NMX plugin can reach all the NVIDIA NMX-Controllers in the environment. This dedicated server must also be able to reach the Netris Controller. All communication between the Netris Controller, NVIDIA NMX-C, and the Netris-NMX plugin is initiated from the server running the Netris-NMX plugin. + .. code-block:: bash -1. Download the Kubernetes deployment YAML file: + kubectl rollout restart deployment/netris-controller-nvlink-agent -n netris-controller + +Apply the updated ``secret.yaml`` .. code-block:: bash - wget https://get.netris.io/netris-controller-nmx.yaml + ~$ kubectl apply -f ./netris-controller-ha/manifests/netris-controller/nvlink/secret.yaml + +Step 2: Configuring NMX Controller connection +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -2. Edit the YAML file to update the secret values based on your environment: +Edit the ``./netris-controller-ha/manifests/netris-controller/nvlink/config.yaml`` file and provide the IP addresses of all the NMX Controllers (NMX-Cs) in scope, as well as the NMX-C PKI certificate bundle, i.e., the file names for the Signed Certificate, Private Key, and the Root CA Certificate. .. code-block:: yaml - nmx-config: - verify-ssl: true - cert-file: /home/ubuntu/netris-nvlink-agent/client.crt - key-file: /home/ubuntu/netris-nvlink-agent/client.key - root-ca: /home/ubuntu/netris-nvlink-agent/rootCA.crt - common-name: nmxc-01.acme.com - nmx-c: - nmxc_01: - addresses: - - nmxc-01.acme.com:8601 + ~$ vi ./netris-controller-ha/manifests/netris-controller/nvlink/config.yaml + apiVersion: v1 + kind: ConfigMap + metadata: + name: netris-controller-nvlink-agent-config + namespace: netris-controller + data: + config: |- + nmx-config: + verify-ssl: true + cert-file: /etc/netris-nvlink-agent/tls/client.crt + key-file: /etc/netris-nvlink-agent/tls/client.key + root-ca: /etc/netris-nvlink-agent/tls/rootCA.crt + common-name: nmxc.example.com + nmx-c: + nmxc_001: + addresses: + - 192.0.2.1:8601 + nmxc_002: + addresses: + - 192.0.2.2:8601 + nmxc_003: + addresses: + - 192.0.2.3:8601 + nmxc_004: + addresses: + - 192.0.2.4:8601 + nmxc_005: + addresses: + - 192.0.2.5:8601 + nmxc_006: + addresses: + - 192.0.2.6:8601 + +.. tip:: The ``/etc/netris-nvlink-agent/tls/`` path is the path to the file inside the container, and not on the host system. It should match the value of the ``mountPath`` key in the ``deploy.yaml`` file. + +Depending on your NMX-C configuration, you may provide one shared PKI certificate bundle for all NMX Controllers, as shown above, or specify a unique certificate bundle per NMX-C as shown below. + +.. code-block:: yaml + + apiVersion: v1 + kind: ConfigMap + metadata: + name: netris-controller-nvlink-agent-config + namespace: netris-controller + data: + config: |- + nmx-config: + verify-ssl: true + root-ca: /etc/netris-nvlink-agent/tls/rootCA.crt + nmx-c: + nmxc_001: + cert-file: /etc/netris-nvlink-agent/tls/nmx01/client.crt + key-file: /etc/netris-nvlink-agent/tls/nmx01/client.key + common-name: nmxc001.example.com + addresses: + - 192.0.2.1:8601 + nmxc_002: + cert-file: /etc/netris-nvlink-agent/tls/nmx02/client.crt + key-file: /etc/netris-nvlink-agent/tls/nmx02/client.key + common-name: nmxc002.example.com + addresses: + - 192.0.2.2:8601 + nmxc_003: + cert-file: /etc/netris-nvlink-agent/tls/nmx03/client.crt + key-file: /etc/netris-nvlink-agent/tls/nmx03/client.key + common-name: nmxc003.example.com + addresses: + - 192.0.2.3:8601 + +.. dropdown:: Netris NVLink Plugin Configuration Parameters + + The following configuration options are available in the Netris NVLink plugin YAML configuration file: + + - **nmx-config** - top-level mapping for the plugin configuration + - **verify-ssl** - key to signal the plugin whether to use TLS authentication when accessing the NMX controller (NMX-C) + - **cert-file** - absolute path to the client certificate + - **key-file** - absolute path to the private key of the client certificate + - **root-ca** - absolute path to the root CA certificate file. + - **common-name** - must match the value of the CN field of the certificate presented by the NMX controller (NMX-C). + - **nmx-c** - contains a mapping describing each NMX controller (NMX-C) you'd like Netris to create NVLink partitions in. It must contain at least one key with a value of a list of hostnames and port numbers + + - **addresses** - IP address and port number of each NMX-C endpoint. + + Each NMX-C must be presented through a separate key. + +Apply the configuration to your Kubernetes cluster: + +.. code-block:: bash + + ~$ kubectl apply -f ./netris-controller-ha/manifests/netris-controller/nvlink/config.yaml + +Load the PKI certificate bundle into the K8S secrets (the example command below assumes the relevant PKI files are located in current directory): + +.. code-block:: bash + + ~$ kubectl -n netris-controller create secret generic netris-controller-nvlink-agent-tls \ + --from-file=client.key=./client.key \ + --from-file=client.crt=./client.crt \ + --from-file=rootCA.crt=./rootCA.crt + +If your deployment requires a unique client certificate per NMX-C, you will need to create multiple K8S secrets as shown below: + +.. code-block:: bash + + ~$ kubectl -n netris-controller create secret generic netris-controller-nvlink-agent-tls-nmx01 \ + --from-file=client.key=./nmx01/client.key \ + --from-file=client.crt=./nmx01/client.crt \ + --from-file=rootCA.crt=./rootCA.crt + ~$ kubectl -n netris-controller create secret generic netris-controller-nvlink-agent-tls-nmx02 \ + --from-file=client.key=./nmx02/client.key \ + --from-file=client.crt=./nmx02/client.crt \ + --from-file=rootCA.crt=./rootCA.crt + ~$ kubectl -n netris-controller create secret generic netris-controller-nvlink-agent-tls-nmx03 \ + --from-file=client.key=./nmx03/client.key \ + --from-file=client.crt=./nmx03/client.crt \ + --from-file=rootCA.crt=./rootCA.crt +Step 3: Applying the K8S deployment +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -3. Apply the configuration to your Kubernetes cluster: +If you are using a unique certificate bundle per NMX-C, you will need to edit ``./netris-controller-ha/manifests/netris-controller/nvlink/deploy.yaml`` before using it to include all the volume mounts and secrets you have created earlier. Otherwise, no changes to ``deploy.yaml`` are necessary. + +The example below shows the per-NMX-C-certificate variant of the ``deploy.yaml`` file. + +.. code-block:: yaml + + apiVersion: apps/v1 + kind: Deployment + metadata: + name: netris-controller-nvlink-agent + namespace: netris-controller + spec: + replicas: 1 + selector: + matchLabels: + app: netris-controller-nvlink-agent + template: + metadata: + labels: + app: netris-controller-nvlink-agent + spec: + containers: + - name: nvlink-agent + image: netrisai/bare-metal-nvlink-agent:4.6.1-001 + command: ["/app/servicebin", "-c", "/app/config.yaml"] + envFrom: + - secretRef: + name: netris-controller-nvlink-agent-envs + env: + - name: NETRIS_TIMEOUT + value: "10" + - name: RECONCILE_INTERVAL + value: "10" + volumeMounts: + - name: nvlink-tls-nmxc01 + mountPath: /etc/netris-nvlink-agent/tls/nmxc01 + readOnly: true + - name: nvlink-tls-nmxc02 + mountPath: /etc/netris-nvlink-agent/tls/nmxc02 + readOnly: true + - name: nvlink-tls-nmxc03 + mountPath: /etc/netris-nvlink-agent/tls/nmxc03 + readOnly: true + - name: nvlink-config + mountPath: /app/config.yaml + subPath: config + readOnly: true + volumes: + - name: nvlink-tls-nmxc01 + secret: + secretName: netris-controller-nvlink-agent-tls-nmx01 + defaultMode: 0400 + - name: nvlink-tls-nmxc02 + secret: + secretName: netris-controller-nvlink-agent-tls-nmx02 + defaultMode: 0400 + - name: nvlink-tls-nmxc03 + secret: + secretName: netris-controller-nvlink-agent-tls-nmx03 + defaultMode: 0400 + - name: nvlink-config + configMap: + name: netris-controller-nvlink-agent-config + +Apply the deployment .. code-block:: bash - kubectl apply -f netris-controller-nmx.yaml + ~$ kubectl apply -f ./netris-controller-ha/manifests/netris-controller/nvlink/deploy.yaml + +.. _nvlink_loading_gpu_inventory: -Loading GPU inventory ----------------------------- +Loading GPU Inventory +------------------------ -Netris must have the mapping between the GPU UIDs and the servers those GPUs are installed in preloaded before automatic NVL partition management can start. +You must preload the mapping between the GPU UIDs and the servers in which those GPUs are installed before the automatic NVL partition management can start. Here is an example of the GPU UID inventory file: .. code-block:: bash + hostname,gpuUid hgx-pod00-su0-h00,875835130816197840 hgx-pod00-su0-h00,961186615343340613 hgx-pod00-su0-h00,796824814706104730 hgx-pod00-su0-h00,684212070855729123 hgx-pod00-su0-h01,718625720642846212 hgx-pod00-su0-h01,788578661925003442 - hgx-pod00-su0-h01,910329703472956766 + hgx-pod00-su0-h01,910329783472956766 hgx-pod00-su0-h01,814561743235261831 hgx-pod00-su0-h02,996615732638596030 hgx-pod00-su0-h02,884228998345288014 @@ -155,21 +375,36 @@ Here is an example of the GPU UID inventory file: hgx-pod00-su0-h03,825286183620844317 hgx-pod00-su0-h03,784007583961668668 hgx-pod00-su0-h03,713366763878128965 - … -Execute ``netris-nvl-loader`` script to import the GPU UID inventory into the Netris Controller +In this file, the ``hostname`` refers to the server's object name in :doc:`Netris Inventory `. + +Step 1: Install the nvlink-loader +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``nvlink-loader`` binary ships as part of the Netris Controller distribution package. Copy the ``nvlink-loader.bin`` binary into the Netris controller's local ``/usr/local/bin`` and make it executable. + +.. code-block:: bash + + ~$ sudo cp ./netris-controller-ha/files/k3s/nvlink-loader.bin /usr/local/bin/nvlink-loader && sudo chmod +x /usr/local/bin/nvlink-loader + +Step 2: Load the GPU inventory into the Netris Controller +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You only need to load the inventory on the primary Netris controller node. Do not repeat this action on standby controller nodes. + +Execute ``nvlink-loader`` script to import the GPU UID inventory into the Netris Controller .. code-block:: bash - ./netris-nvl-loader --csv-file gpu-mapping.csv --netris-url "https://controller.acme.com" --username "admin" --password "passw0rd" + ~$ nvlink-loader --csv-file gpu-mapping.csv --netris-url "https://controller.acme.com" --username "admin" --password "passw0rd" -Where +Where .. code-block:: bash --csv-file specifies the comma-separated value (CSV) file with GPU UID inventory - --netris-url "” specifies the Netris Controller URL + --netris-url "" specifies the Netris Controller URL --username "" specifies the Netris administrator username @@ -177,19 +412,22 @@ Where Upon successful import, you should see output similar to the one below -.. code-block:: bash +.. code-block:: text INFO [0000] Found 72 GPU mappings in CSV file INFO [0000] Logging in to Netris… INFO [0000] Successfully logged in to Netris INFO [0000] Fetching inventory from Netris (import mode)... INFO [0001] Found 38 server inventory items - INFO [0002] Successfully updated server ‘hgx-pod00-su0-h00’ with 4 GPU mappings - INFO [0003] Successfully updated server ‘hgx-pod00-su0-h01’ with 4 GPU mappings - INFO [0004] Successfully updated server ‘hgx-pod00-su0-h02’ with 4 GPU mappings - … + INFO [0002] Successfully updated server 'hgx-pod00-su0-h00' with 4 GPU mappings + INFO [0003] Successfully updated server 'hgx-pod00-su0-h01' with 4 GPU mappings + INFO [0004] Successfully updated server 'hgx-pod00-su0-h02' with 4 GPU mappings + + +Step 3: Verify the imported GPU inventory +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -You can further confirm successful import by examining the appropriate server objects in the Netris controller. In the Custom JSON field of each GPU server in scope, you should see a JSON object similar to the following +You can also confirm successful import by examining the appropriate server objects in the Netris controller. In the Custom JSON field of each relevant server object in Netris :doc:`Inventory `, you should see a JSON object similar to the following .. image:: images/NVL-Server-GPU-inventory.png :align: center @@ -197,71 +435,76 @@ You can further confirm successful import by examining the appropriate server ob .. raw:: html -
- -.. warning:: Netris does not enforce the completeness of the GPU inventory or whether the mapping is correct. Please ensure that you validate your inventory file content before loading the inventory into the Netris Controller. - -.. tip:: In the most basic sense, when you create a Server Cluster using a Server Cluster Template that references NVLink integration, as shown in the Server Cluster documentation, Netris will look up GPU UIDs in the Netris server inventory for each server in the Server Cluster and create one NVLink partition per NMX-C named after the Server Cluster, including the Server Cluster ID, and assign the appropriate GPUs to that partition. - -.. warning:: NVIDIA does not support NVLink partitions spanning more than one NVL domain. +

Figure: GPU UID mapping shown in the server's Custom JSON field

-Netris-NMX Plugin Configuration Parameters ------------------------------------------- +Using the Netris NVLink plugin +================================ -The following configuration options are available in the Netris-NMX plugin YAML configuration file: +After successfully configuring the Netris NVLink plugin and loading the GPU inventory, as shown above, you can use the :doc:`Server Cluster ` functionality to create, update, and delete NVL72 NVLink partitions. -- **nmx-config** - top-level mapping for the plugin configuration -- **verify-ssl** - key to signal the plugin whether to use TLS authentication when accessing the NMX-Controller -- **cert-file** - absolute path to the client certificate -- **key-file** - absolute path to the private key of the client certificate -- **root-ca** - absolute path to the root CA certificate file. -- **common-name** - must match the value of the CN field of the certificate presented by the NMX-Controller. -- **nmx-c** - contains a mapping describing each NMX-Controller you’d like Netris to create NVLink partitions in. It must contain at least one key with a value of a list of hostnames and port numbers - - - **addresses** - is a list of hostnames and port numbers of each NMX-C node in an NMX-C HA cluster. +.. tip:: You must update your Server Cluster Template to include NVLink integration. See the :doc:`Server Cluster documentation ` for more details. -Each NMX-C must be presented through a separate key. The ``addresses`` key is intended to contain a list of every node in a single NMX-C HA instance. +.. tip:: In the most basic sense, when you create a Server Cluster using a Server Cluster Template with NVLink integration, as shown in the :doc:`Server Cluster documentation `, Netris will look up the relevant GPU UIDs in the Netris :doc:`Inventory ` for each server object included in the given Server Cluster and create one NVL72 partition per NMX-C named after the Server Cluster, including the Server Cluster ID. Then Netris will assign the appropriate GPU UIDs to that partition. -In the deployment where each NMX-Controller requires a separate client authentication certificate, the relevant keys may be included in the mapping for that specific NMX-Controller, like so: +.. warning:: Netris does not enforce the completeness of the GPU inventory or the correctness of the server-to-GPU ID mapping. Please verify the content of your inventory CSV file before loading it into the Netris Controller. -.. code-block:: yaml +.. warning:: NVIDIA does not support NVL72 partitions spanning more than one NVL domain. If your Server Cluster spans multiple NVL72 domains, each NVL72 domain will have a partition created within it with GPU UIDs from that domain only. Please reach out to Netris support or your NVIDIA representative with questions about operating multiple NVL72 domains. - nmx-config: - verify-ssl: true - cert-file: /home/ubuntu/netris-nvlink-agent/client.crt - key-file: /home/ubuntu/netris-nvlink-agent/client.key - root-ca: /home/ubuntu/netris-nvlink-agent/rootCA.crt - common-name: nmxc-x.acme.com - nmx-c: - nmxc_01: - cert-file: /home/ubuntu/netris-nvlink-agent/client01.crt - key-file: /home/ubuntu/netris-nvlink-agent/client01.key - root-ca: /home/ubuntu/netris-nvlink-agent/rootCA.crt - common-name: nmxc-01.acme.com - addresses: - - nmxc-01.acme.com:8601 - nmxc_02: - addresses: - - nmxc-02.acme.com:8601 - nmxc_03: - addresses: - - nmxc-03.acme.com:8681 - - nmxc-03.acme.com:8682 - -After successfully installing and configuring the Netris-NMX agent, you can use :doc:`Server Cluster ` to create NVLink partitions via the creation of Server Clusters. You must update your Server Cluster Template to include NVLink integration. See the :doc:`Server Cluster documentation ` for more details. +.. _nvlink_verification: Verification -============ - -Netris ships ``nmx-get-partititions.sh`` script, which helps the operator to verify proper operation of the Netris-NMX plugin. +============== -Below are a few examples of running the verification script. +When you need to directly validate NVL72 partitions and their membership, Netris recommends using a script to simplify access to NMX-C. -The output shows that only a default partition is present in the NMX-Controller and no GPU UIDs are assigned to it. +Here is an example of that script. -.. code:: bash +.. code-block:: bash - > ./nmx-get-partitions.sh + #!/bin/bash + + # NMX-C Partition Info Script + # Usage: ./nmx-get-partitions.sh [gateway_id] + + # Configuration + GATEWAY_ID="${1:-gateway_id}" + NMX_HOST="nmxc-01.acme.com:8601" + CERT_PATH="/home/ubuntu/netris-nvlink-agent/client.crt" + KEY_PATH="/home/ubuntu/netris-nvlink-agent/client.key" + CACERT_PATH="/home/ubuntu/netris-nvlink-agent/rootCA.crt" + + echo "=== NMX-C Partition Information ===" + echo "Gateway ID: $GATEWAY_ID" + echo "Host: $NMX_HOST" + echo "=====================================" + echo + + # Get partition info from NMX-C + grpcurl -d "{\"gatewayId\": \"$GATEWAY_ID\"}" \ + -cert "$CERT_PATH" \ + -key "$KEY_PATH" \ + -cacert "$CACERT_PATH" \ + "$NMX_HOST" \ + nmx_c.NMX_Controller.GetPartitionInfoList | \ + jq -r ' + .partitionInfoList[] | + "Partition ID: \(.partitionId.partitionId)", + "Name: \(.name // "N/A")", + "Number of GPUs: \(.numGpus // "N/A")", + "GPU UIDs:", + (if .gpuUidList then (.gpuUidList[] | " - \(.)") else " - None" end), + "Health: \(.health // "N/A")", + "Type: \(.partitionType // "N/A")", + "----------------------------------------" + ' + +Below are a few examples of running this verification script. + +The following output shows that only a default partition is present in the NMX controller (NMX-C) and no GPU UIDs are assigned to it. + +.. code-block:: text + + ~$ ./nmx-get-partitions.sh === NMX-C Partition Information === Gateway ID: gateway_id Host: nmxc-01.acme.com:8601 @@ -270,18 +513,18 @@ The output shows that only a default partition is present in the NMX-Controller Partition ID: 32766 Name: Default Partition Number of GPUs: N/A - GPU UIDS: + GPU UIDs: - None - Health: NAX_PARTITION_HEALTH_HEALTHY + Health: NMX_PARTITION_HEALTH_HEALTHY Type: NMX_PARTITION_TYPE_GPUUID_BASED -The customer may choose to configure the NVLink domain with a default partition (see `NVIDIA NVLink Multi-Node Documentation `_ for more details). Netris is fully compatible with this scenario and will remove the GPU UIDs from the default NVLink partition when those GPU UIDs are scheduled to be assigned to a new tenant partition. +The customer may choose to configure each NVL72 domain with a default partition (see `NVIDIA NVLink Multi-Node Documentation `_ for more details). Netris is fully compatible with this scenario and will remove the GPU UIDs from the default NVL72 partition prior to assigning them to a new tenant partition. -The output below shows a new NVLink partition created with 8 GPUs after a server cluster was instantiated containing servers with those GPU UIDs. Note that the partition name contains the Server Cluster ID (192 in this example), which may be helpful during troubleshooting. Netris will always include the Server Cluster ID in the NVLink partition name. +The output below shows a new NVL72 partition created with 8 GPUs after a Server Cluster was instantiated containing servers with those GPU UIDs. Note that the partition name contains the Server Cluster ID (192 in this example), which may be helpful during troubleshooting. Netris will always include the Server Cluster ID in the NVL72 partition name. -.. code:: bash +.. code-block:: text - > ./nmx-get-partitions.sh + ~$ ./nmx-get-partitions.sh === NMX-C Partition Information === Gateway ID: gateway_id Host: nmxc-01.acme.com:8601 @@ -290,31 +533,31 @@ The output below shows a new NVLink partition created with 8 GPUs after a server Partition ID: 32766 Name: Default Partition Number of GPUs: N/A - GPU UIDS: + GPU UIDs: - None - Health: NMX_PARTITION HEALTH_ HEALTHY - Туре: NMX_PARTITION_TYPE_GPUUID_BASED - —----------------------------------- - Partition 1D: 8593 + Health: NMX_PARTITION_HEALTH_HEALTHY + Type: NMX_PARTITION_TYPE_GPUUID_BASED + ----------------------------------- + Partition ID: 8593 Name: netris-cluster-192 Number of GPUs: 8 GPU UIDs: - 875835130816197840 - 961186615343340613 - - 796824814706184730 + - 796824814706104730 - 684212070855729123 - 718625720642846212 - 788578661925003442 - 910329783472956766 - 814561743235261831 - Health: NMX_PARTITION HEALTH_ HEALTHY + Health: NMX_PARTITION_HEALTH_HEALTHY Type: N/A - —----------------------------------- + ----------------------------------- Maintenance and Deprovisioning =============================== -If you need to perform maintenance on one or more GPU servers that are part of an NVLink partition, Netris recommends that you remove those servers from the Server Cluster before performing this maintenance. Doing so will remove the relevant GPU UIDs from the tenant's NVLink partition. +If you need to perform maintenance on one or more GPU servers that are part of an NVL72 partition, Netris recommends that you remove those servers from the Server Cluster before performing this maintenance. Doing so will remove the relevant GPU UIDs from the tenant's NVL72 partition. .. warning:: Removing a server from a Server Cluster will also remove this server from every and all V-Nets and VPCs that this server was a member of as a result of being a member of a Server Cluster. Netris will not remove this server from any V-Nets where the switch ports connected to this server were assigned to this V-Net manually or using :ref:`labels `. From 179eab2087cd4b3fbf9530fceb460547530ac17a Mon Sep 17 00:00:00 2001 From: Andrey Khomyakov Date: Fri, 17 Jul 2026 12:13:48 -0700 Subject: [PATCH 2/2] Minor nvlink doc fixes --- netris-nvlink-integration.rst | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/netris-nvlink-integration.rst b/netris-nvlink-integration.rst index e0d50b72..431182cd 100644 --- a/netris-nvlink-integration.rst +++ b/netris-nvlink-integration.rst @@ -5,6 +5,10 @@ NVIDIA NMX-C (NVLink) Integration Plugin for Netris Controller ################################################################ +.. contents:: Table of Contents + :local: + :depth: 4 + Overview ======== @@ -117,7 +121,7 @@ Provide the controller credentials that the Netris NVLink plugin will use to com stringData: NETRIS_CONTROLLER_ADDR: http://netris-controller-ha-web-service-backend.netris-controller NETRIS_CONTROLLER_LOGIN: "netris" - NETRIS_CONTROLLER_PASSWORD: "newNet0ps" + NETRIS_CONTROLLER_PASSWORD: "Password!" NETRIS_SITE_NAME: "Site" # NETRIS_VERIFY_SSL: "true" @@ -125,8 +129,14 @@ In the ``secret.yaml`` file you should only need to update the values of the ``N .. warning:: Do not modify the value of the ``NETRIS_CONTROLLER_ADDR`` variable. +Apply the updated ``secret.yaml`` + +.. code-block:: bash + + ~$ kubectl apply -f ./netris-controller-ha/manifests/netris-controller/nvlink/secret.yaml + .. tip:: - The username and password supplied in the ``secret.yaml`` must have "Permit All" selected as a value of the "Permission Group" field and "All Tenants" added and Edit selected when adding a Netris user. + The username and password supplied in the ``secret.yaml`` must have "Permit All" selected as the value of the "Permission Group" field, "All Tenants" added, and Edit selected when adding a Netris user. .. image:: images/nvlink-add-user.png :align: center @@ -143,12 +153,6 @@ In the ``secret.yaml`` file you should only need to update the values of the ``N kubectl rollout restart deployment/netris-controller-nvlink-agent -n netris-controller -Apply the updated ``secret.yaml`` - -.. code-block:: bash - - ~$ kubectl apply -f ./netris-controller-ha/manifests/netris-controller/nvlink/secret.yaml - Step 2: Configuring NMX Controller connection ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -376,7 +380,7 @@ Here is an example of the GPU UID inventory file: hgx-pod00-su0-h03,784007583961668668 hgx-pod00-su0-h03,713366763878128965 -In this file, the ``hostname`` refers to the server's object name in :doc:`Netris Inventory `. +.. tip:: In this file, the ``hostname`` refers to the server's object name in :doc:`Netris Inventory `. Step 1: Install the nvlink-loader ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -390,7 +394,7 @@ The ``nvlink-loader`` binary ships as part of the Netris Controller distribution Step 2: Load the GPU inventory into the Netris Controller ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -You only need to load the inventory on the primary Netris controller node. Do not repeat this action on standby controller nodes. +.. warning:: You only need to load the inventory on the primary Netris controller node. Do not repeat this action on standby controller nodes. Execute ``nvlink-loader`` script to import the GPU UID inventory into the Netris Controller