Note
Go to the end to download the full example code.
CI test: sink state stays synchronized across MPI ranks through dump/reload#
Creates a tiny SPH setup with a few sinks and gas particles, checks sink sync, evolves, dumps, reloads into a fresh context, and checks again.
SPH setup: generating particles ...
SPH setup: Nstep = 0 ( 0.0e+00 ) Ntotal = 0 ( 0.0e+00 rank min = 0.0e+00 max = 0.0e+00) rate = 0.000000e+00 N.s^-1
SPH setup: Nstep = 70 ( 7.0e+01 ) Ntotal = 70 ( 7.0e+01 rank min = 8.6e+04 max = 7.0e+01) rate = 7.000000e+01 N.s^-1
SPH setup: Nstep = 106 ( 1.1e+02 ) Ntotal = 176 ( 1.8e+02 rank min = 1.3e+05 max = 1.8e+02) rate = 1.760000e+02 N.s^-1
SPH setup: Nstep = 118 ( 1.2e+02 ) Ntotal = 294 ( 2.9e+02 rank min = 8.5e+04 max = 2.9e+02) rate = 2.940000e+02 N.s^-1
SPH setup: Nstep = 109 ( 1.1e+02 ) Ntotal = 403 ( 4.0e+02 rank min = 4.3e+05 max = 4.0e+02) rate = 4.030000e+02 N.s^-1
SPH setup: Nstep = 110 ( 1.1e+02 ) Ntotal = 513 ( 5.1e+02 rank min = 1.4e+05 max = 5.1e+02) rate = 5.130000e+02 N.s^-1
SPH setup: Nstep = 106 ( 1.1e+02 ) Ntotal = 619 ( 6.2e+02 rank min = 4.7e+05 max = 6.2e+02) rate = 6.190000e+02 N.s^-1
SPH setup: Nstep = 118 ( 1.2e+02 ) Ntotal = 737 ( 7.4e+02 rank min = 4.8e+05 max = 7.4e+02) rate = 7.370000e+02 N.s^-1
SPH setup: Nstep = 110 ( 1.1e+02 ) Ntotal = 847 ( 8.5e+02 rank min = 1.1e+05 max = 8.5e+02) rate = 8.470000e+02 N.s^-1
SPH setup: Nstep = 109 ( 1.1e+02 ) Ntotal = 956 ( 9.6e+02 rank min = 5.2e+05 max = 9.6e+02) rate = 9.560000e+02 N.s^-1
SPH setup: Nstep = 115 ( 1.2e+02 ) Ntotal = 1071 ( 1.1e+03 rank min = 4.2e+05 max = 1.1e+03) rate = 1.071000e+03 N.s^-1
SPH setup: Nstep = 121 ( 1.2e+02 ) Ntotal = 1192 ( 1.2e+03 rank min = 4.2e+05 max = 1.2e+03) rate = 1.192000e+03 N.s^-1
SPH setup: Nstep = 118 ( 1.2e+02 ) Ntotal = 1310 ( 1.3e+03 rank min = 1.3e+05 max = 1.3e+03) rate = 1.310000e+03 N.s^-1
SPH setup: Nstep = 119 ( 1.2e+02 ) Ntotal = 1429 ( 1.4e+03 rank min = 5.9e+05 max = 1.4e+03) rate = 1.429000e+03 N.s^-1
SPH setup: Nstep = 121 ( 1.2e+02 ) Ntotal = 1550 ( 1.6e+03 rank min = 6.1e+05 max = 1.6e+03) rate = 1.550000e+03 N.s^-1
SPH setup: Nstep = 121 ( 1.2e+02 ) Ntotal = 1671 ( 1.7e+03 rank min = 6.1e+05 max = 1.7e+03) rate = 1.671000e+03 N.s^-1
SPH setup: Nstep = 118 ( 1.2e+02 ) Ntotal = 1789 ( 1.8e+03 rank min = 5.9e+05 max = 1.8e+03) rate = 1.789000e+03 N.s^-1
SPH setup: Nstep = 119 ( 1.2e+02 ) Ntotal = 1908 ( 1.9e+03 rank min = 6.5e+05 max = 1.9e+03) rate = 1.908000e+03 N.s^-1
SPH setup: Nstep = 122 ( 1.2e+02 ) Ntotal = 2030 ( 2.0e+03 rank min = 1.1e+05 max = 2.0e+03) rate = 2.030000e+03 N.s^-1
SPH setup: Nstep = 120 ( 1.2e+02 ) Ntotal = 2150 ( 2.2e+03 rank min = 7.9e+05 max = 2.2e+03) rate = 2.150000e+03 N.s^-1
SPH setup: Nstep = 118 ( 1.2e+02 ) Ntotal = 2268 ( 2.3e+03 rank min = 6.4e+05 max = 2.3e+03) rate = 2.268000e+03 N.s^-1
SPH setup: Nstep = 63 ( 6.3e+01 ) Ntotal = 2331 ( 2.3e+03 rank min = 3.2e+05 max = 2.3e+03) rate = 2.331000e+03 N.s^-1
SPH setup: the generation step took : 0.014421269 s
SPH setup: final particle count = 2331 beginning injection ...
SPH setup: injected 2000 / 2331 => 85.8% | ranks with patchs = 1 / 1 -> local loop <-
SPH setup: injected 2331 / 2331 => 100.0% | ranks with patchs = 1 / 1 <- global loop -> (msg count : 0)
SPH setup: the injection step took : 0.035048542 s
SPH setup: the setup took : 0.055025241 s
Sinks are in sync !
---------------- t = 0, dt = 0 ----------------
Warning: High interface/patch volume ratio. [InterfaceGen][rank=0]
This can lead to high mpi overhead, try to increase the patch split crit
Warning: smoothing length is not converged, rerunning the iterator ... [Smoothinglength][rank=0]
largest h = 0.09799885899543734 unconverged cnt = 2273
Warning: High interface/patch volume ratio. [InterfaceGen][rank=0]
This can lead to high mpi overhead, try to increase the patch split crit
Warning: smoothing length is not converged, rerunning the iterator ... [Smoothinglength][rank=0]
largest h = 0.10779874489498108 unconverged cnt = 826
Warning: High interface/patch volume ratio. [InterfaceGen][rank=0]
This can lead to high mpi overhead, try to increase the patch split crit
Warning: smoothing length is not converged, rerunning the iterator ... [Smoothinglength][rank=0]
largest h = 0.1185786193844792 unconverged cnt = 55
Warning: High interface/patch volume ratio. [InterfaceGen][rank=0]
This can lead to high mpi overhead, try to increase the patch split crit
Warning: smoothing length is not converged, rerunning the iterator ... [Smoothinglength][rank=0]
largest h = 0.13043648132292712 unconverged cnt = 2
Warning: High interface/patch volume ratio. [InterfaceGen][rank=0]
This can lead to high mpi overhead, try to increase the patch split crit
---------------- t = 0, dt = 6.551773540016009e-06 ----------------
Warning: High interface/patch volume ratio. [InterfaceGen][rank=0]
This can lead to high mpi overhead, try to increase the patch split crit
---------------- t = 6.551773540016009e-06, dt = 0.00022275667803016146 ----------------
Warning: High interface/patch volume ratio. [InterfaceGen][rank=0]
This can lead to high mpi overhead, try to increase the patch split crit
Warning: the corrector tolerance are broken the step will be re rerunned [BasicGasSPH][rank=0]
eps_v = 0.010106726402497119
---------------- t = 0.00022930845157017747, dt = 0.00018328014262681866 ----------------
Warning: High interface/patch volume ratio. [InterfaceGen][rank=0]
This can lead to high mpi overhead, try to increase the patch split crit
---------------- t = 0.00041258859419699616, dt = 0.00034020216026755444 ----------------
Warning: High interface/patch volume ratio. [InterfaceGen][rank=0]
This can lead to high mpi overhead, try to increase the patch split crit
Sinks are in sync !
Sinks are in sync !
---------------- t = 0.0007527907544645507, dt = 0.0004464158410420694 ----------------
Warning: High interface/patch volume ratio. [InterfaceGen][rank=0]
This can lead to high mpi overhead, try to increase the patch split crit
---------------- t = 0.0011992065955066201, dt = 0.0005126549605206108 ----------------
Warning: High interface/patch volume ratio. [InterfaceGen][rank=0]
This can lead to high mpi overhead, try to increase the patch split crit
---------------- t = 0.001711861556027231, dt = 0.0005523343280162558 ----------------
Warning: High interface/patch volume ratio. [InterfaceGen][rank=0]
This can lead to high mpi overhead, try to increase the patch split crit
Warning: the corrector tolerance are broken the step will be re rerunned [BasicGasSPH][rank=0]
eps_v = 0.011840076446588806
---------------- t = 0.002264195884043487, dt = 0.0002864278912784332 ----------------
Warning: High interface/patch volume ratio. [InterfaceGen][rank=0]
This can lead to high mpi overhead, try to increase the patch split crit
Warning: the corrector tolerance are broken the step will be re rerunned [BasicGasSPH][rank=0]
eps_v = 0.0257622846130577
---------------- t = 0.00255062377532192, dt = 0.00019897300678063906 ----------------
Warning: High interface/patch volume ratio. [InterfaceGen][rank=0]
This can lead to high mpi overhead, try to increase the patch split crit
Sinks are in sync !
run_test_sink_synchro: OK
Current sinks:
[{'pos': (0.09877660057755726, -0.0003699820152028823, 0.00029285849440530896), 'velocity': (-0.7909594697171893, -0.428355094200577, 0.21579163220191738), 'sph_acceleration': (-2.7881186060647245, 6.194080856496949, -3.3863025439730663), 'ext_acceleration': (-371.68294829298736, -190.87247862152321, 85.76563352228861), 'mass': 1.0219999999999998, 'angular_momentum': (1.535513224220801e-05, -3.404279116936254e-05, -0.0013917210465080316), 'accretion_radius': 0.15}, {'pos': (-0.19723897477916802, 0.0992597386368148, 5.457513126197259e-05), 'velocity': (1.7840284522444, -0.646223010107699, 0.039967403747977075), 'sph_acceleration': (40.13267477144062, -16.361891265380407, -1.2376352720798192), 'ext_acceleration': (456.65294536581416, -211.7760255223151, 16.187810528070777), 'mass': 0.523, 'angular_momentum': (1.4588956579124883e-08, -1.6035141770323977e-07, -0.0009268830439052644), 'accretion_radius': 0.15}, {'pos': (0.001919965581750201, -0.1466495904953547, 0.048938432104203365), 'velocity': (2.1365106397204268, 3.0220342305086376, -0.9498166179408467), 'sph_acceleration': (4.433172833108282, 16.09220042839756, -6.362794011098256), 'ext_acceleration': (522.3351212189341, 1132.7056833309907, -355.99519394799984), 'mass': 0.27, 'angular_momentum': (4.446934644992772e-05, 0.0006693090866850242, 0.0020383710492193246), 'accretion_radius': 0.15}]
check_ref_dataset: OK
Current sums: [np.float64(65.79475628688994), np.float64(-15.508903903707498), np.float64(-1.0412208430674998), np.float64(22613.455824802637), np.float64(-5.195701678016261), np.float64(15.008486573476972), np.float64(-19336.88835500462), np.float64(-2117.975619213109), np.float64(5826.038830234711), np.float64(245.85176557074223)]
9 import numpy as np
10
11 import shamrock
12
13 DUMP_NAME = "sink_sync_test.sham"
14
15
16 def check_sinks_are_in_sync(ctx, model):
17 s = str(model.get_sinks())
18 # Collective: every rank must call this
19 hist = shamrock.algs.all_string_histogram([s], delimiter="\n", hash_based=False)
20 if len(hist) != 1:
21 raise RuntimeError(f"sinks not in sync across ranks: {hist}")
22 key, count = next(iter(hist.items()))
23 if count != shamrock.sys.world_size():
24 raise RuntimeError(
25 f"expected count={shamrock.sys.world_size()}, got {count} for key={key!r}"
26 )
27 shamrock.sys.mpi_barrier()
28 if shamrock.sys.world_rank() == 0:
29 print("Sinks are in sync !")
30
31
32 si = shamrock.UnitSystem()
33 sicte = shamrock.Constants(si)
34 codeu = shamrock.UnitSystem(
35 unit_time=sicte.year(),
36 unit_length=sicte.au(),
37 unit_mass=sicte.sol_mass(),
38 )
39 ucte = shamrock.Constants(codeu)
40 G = ucte.G()
41
42
43 def build_model_with_sinks():
44 ctx = shamrock.Context()
45 ctx.pdata_layout_new()
46
47 model = shamrock.get_Model_SPH(context=ctx, vector_type="f64_3", sph_kernel="M4")
48
49 cfg = model.gen_default_config()
50 cfg.set_self_gravity_none()
51 cfg.set_artif_viscosity_Constant(alpha_u=1.0, alpha_AV=1.0, beta_AV=2.0)
52 cfg.set_eos_isothermal(1.0)
53 cfg.set_particle_mass(1e-3)
54 cfg.set_boundary_periodic()
55 cfg.set_show_cfl_detail(True)
56 cfg.set_units(codeu)
57 model.set_solver_config(cfg)
58
59 model.set_cfl_cour(0.1)
60 model.set_cfl_force(0.1)
61 model.set_eta_sink(1.0)
62
63 model.init_scheduler(1000, 1)
64
65 # Very coarse HCP cube -> handful of SPH particles
66 dr = 0.05
67 bmin = (-0.6, -0.6, -0.6)
68 bmax = (0.6, 0.6, 0.6)
69 model.resize_simulation_box(bmin, bmax)
70
71 setup = model.get_setup()
72 gen = setup.make_generator_lattice_hcp(dr, bmin, bmax)
73 setup.apply_setup(gen)
74
75 eng = shamrock.algs.gen_seed(42)
76
77 def vel_func(r):
78 return (10.0, 0.0, 0.0)
79
80 model.set_field_value_lambda_f64_3("vxyz", vel_func)
81
82 # A few sinks (must be added after init_scheduler, on all ranks)
83 model.add_sink(1.0, (0.1, 0.0, 0.0), (0.0, 0.05, 0.0), 0.15)
84 model.add_sink(0.5, (-0.2, 0.1, 0.0), (0.0, -0.03, 0.0), 0.15)
85 model.add_sink(0.25, (0.0, -0.15, 0.05), (0.02, 0.0, 0.0), 0.15)
86
87 return ctx, model
88
89
90 def check_ref_dataset(sinks):
91 if shamrock.sys.world_rank() == 0:
92 print("Current sinks:")
93 print(sinks)
94
95 ref_dataset = [
96 {
97 "pos": (0.09877660057755726, -0.0003699820152028823, 0.00029285849440530886),
98 "velocity": (-0.7909594697171893, -0.42835509420057705, 0.2157916322019174),
99 "sph_acceleration": (-2.7881186060647103, 6.194080856496935, -3.386302543973038),
100 "ext_acceleration": (-371.6829482929875, -190.87247862152324, 85.76563352228864),
101 "mass": 1.0219999999999998,
102 "angular_momentum": (
103 1.5355132242207996e-05,
104 -3.404279116936264e-05,
105 -0.001391721046508032,
106 ),
107 "accretion_radius": 0.15,
108 },
109 {
110 "pos": (-0.19723897477916802, 0.0992597386368148, 5.457513126197254e-05),
111 "velocity": (1.7840284522444, -0.6462230101076988, 0.039967403747977054),
112 "sph_acceleration": (40.132674771440605, -16.361891265380414, -1.2376352720798245),
113 "ext_acceleration": (456.6529453658143, -211.77602552231514, 16.18781052807078),
114 "mass": 0.523,
115 "angular_momentum": (
116 1.4588956579124888e-08,
117 -1.6035141770323982e-07,
118 -0.0009268830439052644,
119 ),
120 "accretion_radius": 0.15,
121 },
122 {
123 "pos": (0.0019199655817502001, -0.1466495904953547, 0.048938432104203365),
124 "velocity": (2.1365106397204268, 3.0220342305086376, -0.9498166179408467),
125 "sph_acceleration": (4.433172833108257, 16.09220042839757, -6.362794011098254),
126 "ext_acceleration": (522.3351212189343, 1132.705683330991, -355.99519394799995),
127 "mass": 0.27,
128 "angular_momentum": (
129 4.4469346449927724e-05,
130 0.0006693090866850239,
131 0.0020383710492193246,
132 ),
133 "accretion_radius": 0.15,
134 },
135 ]
136
137 errors = []
138
139 if len(sinks) != len(ref_dataset):
140 errors.append(f"sink count mismatch: got {len(sinks)}, expected {len(ref_dataset)}")
141 else:
142 for i, (got_sink, ref_sink) in enumerate(zip(sinks, ref_dataset)):
143 for key, ref_val in ref_sink.items():
144 got_val = got_sink[key]
145 got_arr = np.asarray(got_val, dtype=float)
146 ref_arr = np.asarray(ref_val, dtype=float)
147 rtol = 1e-14 if key == "sph_acceleration" else 1e-15
148 if not np.all(np.isclose(got_arr, ref_arr, rtol=rtol, atol=1e-18)):
149 abs_diff = np.abs(got_arr - ref_arr)
150 with np.errstate(divide="ignore", invalid="ignore"):
151 rel_diff = np.where(ref_arr != 0, abs_diff / np.abs(ref_arr), abs_diff)
152 errors.append(
153 f"sink[{i}].{key} mismatch:\n"
154 f" got={got_val}\n"
155 f" ref={ref_val}\n"
156 f" max abs diff={np.max(abs_diff)}\n"
157 f" max rel diff={np.max(rel_diff)}"
158 )
159
160 for err in errors:
161 print(err)
162
163 if errors:
164 raise RuntimeError(f"check_ref_dataset failed with {len(errors)} error(s)")
165
166 if shamrock.sys.world_rank() == 0:
167 print("check_ref_dataset: OK")
168
169
170 def main():
171 ctx, model = build_model_with_sinks()
172
173 check_sinks_are_in_sync(ctx, model)
174
175 for _ in range(5):
176 model.timestep()
177 check_sinks_are_in_sync(ctx, model)
178
179 sinks_before_dump = str(model.get_sinks())
180 model.dump(DUMP_NAME)
181
182 del model
183 del ctx
184
185 ctx2 = shamrock.Context()
186 ctx2.pdata_layout_new()
187 model2 = shamrock.get_Model_SPH(context=ctx2, vector_type="f64_3", sph_kernel="M4")
188 model2.load_from_dump(DUMP_NAME)
189
190 sinks_after_reload = str(model2.get_sinks())
191 if sinks_before_dump != sinks_after_reload:
192 raise RuntimeError(
193 "sink content changed across dump/reload:\n"
194 f" before={sinks_before_dump!r}\n"
195 f" after ={sinks_after_reload!r}"
196 )
197
198 check_sinks_are_in_sync(ctx2, model2)
199
200 for _ in range(5):
201 model2.timestep()
202 check_sinks_are_in_sync(ctx2, model2)
203
204 if shamrock.sys.world_rank() == 0:
205 print("run_test_sink_synchro: OK")
206
207 check_ref_dataset(model2.get_sinks())
208
209 dic = ctx2.collect_data()
210
211 if shamrock.sys.world_rank() > 0:
212 return
213
214 assert 2266 == len(dic["xyz"])
215
216 sum_pos = np.sum(dic["xyz"], axis=0)
217 sum_vel = np.sum(dic["vxyz"], axis=0)
218 sum_acc = np.sum(dic["axyz"], axis=0)
219 sum_hpart = np.sum(dic["hpart"], axis=0)
220
221 dat = np.concatenate([sum_pos, sum_vel, sum_acc, np.atleast_1d(sum_hpart)])
222 print("Current sums: ", [dat[i] for i in range(len(dat))])
223
224 ref_sums = [
225 65.79475628688992,
226 -15.508903903707498,
227 -1.0412208430674998,
228 22613.455824802637,
229 -5.195701678016285,
230 15.008486573476965,
231 -19336.888355004605,
232 -2117.975619213131,
233 5826.038830234707,
234 245.85176557074223,
235 ]
236
237 mismatch = False
238 for i in range(len(dat)):
239 if not np.isclose(dat[i], ref_sums[i], rtol=1e-12, atol=1e-18):
240 abs_diff = np.abs(dat[i] - ref_sums[i])
241 rel_diff = abs_diff / np.abs(ref_sums[i])
242 print(f"sum[{i}] mismatch: got {dat[i]}, expected {ref_sums[i]}")
243 print(f" max abs diff={np.max(abs_diff)}")
244 print(f" max rel diff={np.max(rel_diff)}")
245 mismatch = True
246 if mismatch:
247 raise RuntimeError("sums mismatch")
248
249
250 main()
Total running time of the script: (0 minutes 3.491 seconds)
Estimated memory usage: 217 MB