Térfogatvizualizáció Szirmay-Kalos László

Térfogati modellek (skalár mezők) 3D tér pontjaiban egy skalár érték v(x,y,z) v(x,y,z) z y x Skalár: hőmérséklet, sűrűség nyomás, potenciál, … Származás: Euler-i szimuláció, Rekonstrukció (tomográfia) tárolás: 3D textúra vagy „voxel tömb”

Térfogati modell megjelenítése Megjelenítés fényszóró anyag (participating media) analógiáját felhasználva (belsejébe belelátunk) Adott szintfelület kiemelése (külsőt lehámozzuk) Kép szintézis Transzfer függvény grad v Hesse v Optikai paraméterek Kép Sűrűség + deriváltak

Fényszóró közeg ds A=1 ·ds = P(ütközés) Hatáskeresztmetszet, alias kioltási tényező ·ds = P(ütközés) A=1 Albedo a: a nem-elnyelődés valószínűsége feltéve, hogy az ütközés bekövetkezett Fekete test: albedo = 0

Sugársűrűség változása ds Sugársűrűség változása s L(s+ds) L(s) L(s+ds)= L(s) – L(s)·(s)·ds + Le(s)·ds + (s)·a(s)·ds·f(‘,)Li(‘)d‘ // Kiszóródás+abszorbció // Emisszió // Beszóródás dL(s)/ds = –L(s)·(s) +Le(s)+Linscatter(s) Megoldás fényelnyelő közegre (emisszió és beszóródás nincs): L(s)= L(0)·exp(–s (s)ds)

Szóródás Klein-Nishina: Rayleigh: Compton formula: E’ E x Szóródott foton Érkező foton φ  z E ütközés y Rayleigh: Compton formula:

Sugár masírozás (ray marching) Megoldás: dL(s)/ds = –L(s)·(s) +Le(s)+Linscatter(s) L(s + s) L(s) L(0) s L(s+s) = L(s) – L(s)·(s)·s + (Le(s) + Linscatter(s))·s (s) C(s) opacitás hozzájárulás L(s+s) = (1 – (s)) · L(s) + C(s) átlátszóság

Back-to-front ray marching L(s+s) = (1 – (s)) · L(s) + C(s) C(s), (s)) s=0 L = L0; for(s = 0; s < T; s += s) { L = (1 – (s)) · L + C(s); }

Front-to-back ray marching L*(s) (s) L*(ss)=L*(s)+(1 (s)) · C(s) (ss)=((s)) · ((s)) L*   0; for( s  T; s >0 ; s -= s ) { L* + (1 ) · C(s); (1 ) · ((s)); if (break; }

Voxel szín és opacitás: Transfer func: (C,)=T(v függv)·s Röntgen: (C,)=T(v(x,y,z))·s opacitás = v(x,y,z)·s L(0) = I, egyébként C(s)=0 Klasszikus árnyalási modellek opacitás: v osztályozása C = árnyalási modell (diffúz + Phong) normál = grad v opacitás *= | grad v | Magasabb rendű derivált (görbület) Transzlucens anyagok (subsurface scattering) grad v

(C,) = T(v)·s = pow(v/vmax, aexp)·s C = HLS((v/vmax+rot)·360, 0.5, 1)·s

Klasszikus BRDF modellek First hit ray casting: Diffúz+Phong árnyalás v két osztályba: kék-átlátszó, sárga-átlátszatlan Csont : 1, másé 0 Csont =1, másé=0 Hús =1, másé=0

Illusztratív vizualizáció

Illusztratív vizualizáció

Transzlucens megjelenítés Felező vektor

Transzlucens megjelenítés

Szintvonal, szintfelület

Marching cubes v(x,y,z) > szint v(x,y,z) < szint 6 2 5 1 3 7 4 4 0; Eset: 000011102 =14 14 2; 0-1; 0-3; 2-6; 0-3; 3-7; 2-6 14 255 0;

Szintérték = 110 Masírozó kockák Szintérték = 60

First hit (isosurface) ray casting v(x,y,z) > szint normal = grad v v(x,y,z) < szint

GPU first hit ray-casting right up lookat v(x,y,z) eye q p entry exit p = lookat + X right + Y up X, Y in [-1,1] Egység kocka 3D textúrával

CPU first hit ray-casting For each pixel Find pixel center p raydir = normalize(p – eye); Find exit and entry for(t = entry; t < exit; t+=dt) { q = eye + raydir * t; if (volume[q] > isovalue) break; } normal vector estimation; illumination Full screen quad Interpolation from the corners central differences

GPU Isosurface ray-casting volume eye, isolevel, material/light properties Ray casting CPU program Pixel shader Vertex shader Rasterization Interpolation Vertices of the window quad hpos=fullscreen textcoords ray/window Volume

CPU program - OpenGL display void Display( ) { // PASS: non uniform parameters glBegin( GL_QUADS ); Vector p = lookat - Right + Up; glTexCoord3f(p.x, p.y. p.z); glVertex3f(-1, 1, 0); p = lookat - Right - Up; glTexCoord3f(p.x, p.y. p.z); glVertex3f(-1, -1, 0); p = lookat + Right - Up; glTexCoord3f(p.x, p.y. p.z); glVertex3f(1, -1, 0); p = lookat + Right + Up; glTexCoord3f(p.x, p.y. p.z); glVertex3f(1, 1, 0); glEnd(); } Camera window as texture coordinates Full screen quad

Ray casting: vertex shader void VertexShader( in float4 hPosIn : POSITION, in float3 wPosIn : TEXCOORD0, out float4 hPosOut : POSITION, out float3 wPosOut : TEXCOORD0 ) { hPosOut = hPosIn; wPosOut = wPosIn; }

Ray casting: fragment shader void FragmentShader( in float3 p : TEXCOORD0, // point on window uniform float3 eye, uniform sampler3D volume, // voxels uniform float isolevel, uniform float3 lightdir, lightint, kd  out float3 color : COLOR ) { float3 raydir = normalize(p – eye); float3 t0 = (float3(0,0,0)-eye)/raydir; float3 t1 = (float3(1,1,1)-eye)/raydir; float3 ti = min(t0, t1); float3 to = max(t0, t1); float entry = max(max(ti.x, ti.y), ti.z); float exit = min(min(to.x, to.y), to.z); color = float(0, 0, 0); bool found = (exit <= entry); cont’d… t1.y t1.x ti.yto.y 1 t0.x ti.xto.x t0.y 1

Ray casting fragment shader cont’d if ( !found ) { float3 q, normal; float dt = (exit – entry) / STEPS; for(t = entry; t < exit; t += dt) { q = eye + raydir * t; if (tex3D(volume, q).r > isolevel) found = true; } if ( found ) { normal.x = tex3d(volume, q + float3(1/RES,0,0)) – tex3d(volume, q - float3(1/RES,0,0)); normal.y = tex3d(volume, q + float3(0,1/RES,0)) – tex3d(volume, q - float3(0,1/RES,0)); normal.z = tex3d(volume, q + float3(0,0,1/RES)) – tex3d(volume, q - float3(0,0,1/RES)); normal = normalize( normal ); color = lightint * kd * max(dot(lightdir, normal), 0); q

Tomográfia L(s)= L(0)·exp(–(s)ds) L(s)= Le(s)ds Abszorpciós Emissziós LOR L(s)= L(0)·exp(–(s)ds) (s)ds = – log(L(s)/L(0)) L(s)= Le(s)ds

X-ray Computed Tomography Forgatás Detektorok X-ray forrás (CRT) sugarak

Mediso NanoPET/CT One particular equipment we work with is the NanoPet/CT of Mediso, developed for pharmaceutical research and therefore scans mice and rats. The gamma photon detector structure is shown here, which can measure several hundred million lines, which are distributed not only in 2D slices but forming a complex 3D structure. From the measurements, we should reconstruct the density of the radiotracer material.

Bigger boys need big toys: AnyScan PET/CT

Tomográfiás rekonstrukció Having made the measurements, we have the integrals of the scalar field along many lines. The question is how scalar fields can be reconstructed from its line integrals. A very simple approach would be backprojection, which takes a line and distributes the measured integral along it uniformly. As only the integral of the values is available, we cannot do more with a single line. Repeating the same step for all lines, the reconstructed value will be higher where many lines of large integrals meet, so we get a rough approximation of the original scalar field. Clearly, it is blurred and distorted approximation since a point source will be backprojected as a line and the total activity after backprojection will be larger than expected as line cross each other.

Szűrt visszavetítés (FBP=Filtered backprojection) Súlyfüggvény: w(x,y)=w(r)  1/r i(x,y)=(x,y) Mérés + Visszavetítés R kör: rd w(x,y) dxdy=2Rw(r) rdrd= =2 Rw(r)r dr  R dr r Korrekció Fourier térben: o(x,y)=i(x,y)  w(x,y) A standard and old method to improve this naive approach is filtered back projection, which examines what happens with a point source if it is measured and then back projected, and corrects the back projected data accordingly. Let us consider a point source, so the measured scalar field is modeled as a Dirac delta. After backprojection, lines crossing the original point source will have non-zero activity and the reconstructed activity distribution is denoted by function w. This depends just on the distance from the point source due to symmetry, and its integral on a circle of radius R must be proportional to R since the contribution of each constant activity line segment grows linearly with the length. From this, we can easily prove that impulse response of the naive backprojection is a function that decreases proportionally with the distance. Based on the superposition principle, the measurement and back projection of an arbitrary input signal can be expressed as the convolution of th input signal and the impulse response. In Fourier domain, convolution becomes multiplication and the Fourier transform, so the blurring can be compensated by dividing the Fourier transform of the backprojected signal with the transfer function of the system, which is the Fourier transform the impulse response. This requires filtering with a so called ramp filter, which is proportional to the absolute value of the frequency vector. If the signal to noise ratio is not high enough, we get nothing but a mess of noise after reconstruction. This method is fast if multi-dimensional fast Fourier transformation is applied, but it has a significant problem. Ramp filter is a high-pass filter, so not only the blurring is eliminated but high frequency noise is also amplified. FxFy o[x,y] = FxFy i[x,y]FxFyw[x,y] Rámpa szűrő 1/| | FxFy i[x,y] = FxFy o[x,y]  | |

Zaj! 1 2 Becsapódások véletlen események! Emissziós tomográfia: Poisson eloszlás 1 Noise can be significant especially in emission tomography, due to the random nature of the measurement process and the ignored physical effects. For example, when a positron-electron pair is annihilated, the direction of the generated photon pair will be random due to quantum effects. The number of hits in a LOR is also random and follows Poisson distribution. It is tempting that we believe in the theorem of large numbers and state that event frequency is close to the expected values, but it is a very bad idea. In practical PET measurements, at least when we do not wish to kill the patient, the number of hits in a detector is small, so we are very far from the confortable zone offered by the theorem of large numbers. 2 Nagy számok törvénye?

Algebrai visszavetítés Lin egyenlet (V<D): d = A  v, d = [d1,..., dD] v = [v1,..., vV] Moore féle pseudo-inverz: AT  d = ATA v v = (ATA)-1AT  d = A+ d v1 v2 (D) (D×V) (V) d4 v3 v4 d3 d2 (V×D) (D) (V×V) d1 = A11v1 +A13v3

Expectation Maximization Véletlen mérési folyamat Aktivitás: v eredmény: d Mi a v ha tudjuk d-t? Amelyik éppen a mérési eredményt maximálja P(d|v) Likelihood maximalizálás: logP Előny: a mérés statisztikai modelljét is figyelembe veszi If we have a statistical model of the measurement process, that is we know the distribution of the hit numbers, we can make it even better since this information can be built into the reconstruction and can replace the blind Euclidean distance. The concept is called maximum likelihood estimation in statistics. Based on the observed or measured values, we search for an activity field that would produce this set of measurements with the highest probability. So again, we end up to solve an optimization problem but not the probability needs to be maximized. The typical trick is to maximize the logarithm of this probability which would not modify the optimum since the logarithm is a monotonous function but turns products to sums which makes our life much easier when derivatives are computed during a gradient search.

PET: iteratív séma Poisson és pozitivitási kényszer LOR L : annak valószínűsége, hogy a V voxelben bekövetkezett bomlást az L LOR detektálja voxel V

Iteratív séma Skalármező Projekció Összevetés a mért értékekkel korrekciója

Beütések száma a detektorokban Inverz problémák Projekció (Tomográf) Aktivitás eloszlás Beütések száma a detektorokban Ill-posed: nincs megoldás vagy nem egyértelmű. Nem ismerjük n-et (zaj). Közelítő megoldásokból melyik (zaj hatása)? Plusz információ bevitele: regularizáció

Megoldási sémák Algebrai Valószínűségi 2-es norma Kullback-Leibler divergencia y Gauss eloszlású y Poisson eloszlású

Overfitting

Regularizáció Likelihood Regularizációs paraméter Zajos megoldások büntetése R(x) „rossz” megoldásoknál nagy, „jó” megoldásoknál kicsi (tökéletes megoldást ne büntesd, de nem ismerjük), konvex függvény. Teljes variáció (TV):

PET Line Of Response (LOR) P P e+ N e- Additionally, noise can also come from the physical phenomena that are ignored in the simple backprojection operation. In reality, when a positron is born, it is not annihilated instantly but it may take an excursion in the material before it meets and electron. This is called positron range, which depends on the isothope and on the material as well. When the photon pair is generated, the two photons are not exactly parallel if the original impulse of the electron and positron was not exactly zero. The photons may get absorbed or scattered in the measured object, so they do not necessarily follow a straight path. Photons may also be scattered in the detector grid, so a different detector will absorb it not the one where it arrived. Finally, the absorption detection may also make random errors. At the end, the LOR reported by the system may be very far and may not go through the point where the positron was born. Line Of Response (LOR)

Nincs szóródás 324×315×315 voxels

Positron range Nélküle Vele

Szóródás a testben Nélküle Vele

Szóródás a detektorban … 324×315×315 resolution + scattering in detectors 3382 x 845 res <0.1 mm voxels With