FFS

Environment

• DCS World Dedicated Server 2.9.29.27468 (MT), Linux (Ubuntu 24.04 / kernel 7.0) + WineHQ stable 11.0 (bookworm) + Docker
• DCS Olympus v2.0.5 — the backend (olympus.dll / core.dll / luatools.dll / cpprest_2_10.dll) runs inside the DCS process
• CPU: Core Ultra 7 265K, sim thread already pinned to P-cores (CPU 0-7) — see the previous article
• Mission: permanent free flight (777 units)

Symptom

On launch DCS starts at 117 FPS / 42% process CPU. Even with zero players, the sim thread's CPU keeps climbing by about +25 pt per hour, saturating one core after ~2.5–3 hours and pinning FPS at 28. A restart resets it, then it decays along the same slope. Unrelated to player count, takeoffs, SAMs or mission content.

# Run started 9/4 16:01 (zero players throughout). Process CPU% / FPS
Elapsed   10min   30min   60min   90min  120min  150min  180min  240min
CPU    42     51     59     72     81     98    108    108
FPS   116    107     95     73     64     37     29     29

RSS also grows monotonically, 7.1 GB → 8.5 GB. The previous article's fix (E-core pinning) lowered the baseline but this slope did not change.

When it started

Plotting elapsed-time-vs-process-CPU% per run shows exactly one boundary.

# server1、process CPU% by time since run start (players)
run start            10min    60min   120min   180min   360min
08-31 04:07         38/0    39/0    39/0    40/0    41/0    ← old DCS, no Olympus
09-02 04:19         33/0    34/0    35/0    35/0    38/0
09-03 04:14         43/0    43/0    44/0    44/0    47/0    ← after the DCS 2.9.29 update, no Olympus: flat
09-03 17:23         52/0    69/0    94/0   108/0           ← Olympus enabled on server1: reproduced in every run since
09-04 04:18         45/0    65/0   100/2   108/0
09-04 16:01         44/0    65/0    93/0   108/1

Even after the DCS core update (9/3 01:12), runs without Olympus stay flat for 6 hours — the DCS update is irrelevant. It happens without exception from the first run with Olympus enabled on server1.

Method (measuring inside without breaking things)

"Removing Olympus fixed it, therefore Olympus is at fault" tells you nothing about what is actually happening, so we profiled the saturated process without stopping it. PE modules under Wine have no symbols, so attribution is done at DSO granularity (which DLL) and by real address ranges of loaded PEs.

# 1. Sample the sim thread (main tid) at 499 Hz, aggregate per DLL
perf record -F 499 -t <main_tid> -- sleep 20
perf report --sort dso,sym --stdio

# 2. Get callers via LBR (Last Branch Record). perf can't read PE .pdata,
#    so LBR — which needs no frame pointers / unwind info — is the only option
perf record -F 499 -t <main_tid> --call-graph lbr -- sleep 10

# 3. Map [JIT] anonymous mappings to PE modules:
#    read 'MZ' → PE header → export names from the start of each region in /proc/<pid>/maps
#    (via /proc/<pid>/mem from the same PID namespace)

# 4. Sample registers inside the hot loop to read the requested size directly
perf record -F 997 -t <main_tid> --user-regs=R13,R12 -- sleep 6
perf script -F ip,uregs

Findings

80% of CPU time is one loop in Wine's ntdll.dll

    85.40%  ntdll.dll          ← 77% of it is 5 instructions at ntdll.dll+0x4f500
      9.18%  [JIT]              ← DCS core (World/Scripting/edCore/lua etc., anonymous mappings)
      3.09%  ntdll.so
      0.56%  ucrtbase.dll

Disassembly shows heap_allocate_block walking the free list with mov rbx,[rbx], comparing each block's size against the requested size (r13) — a first-fit scan repeating "too small, move on".

17004f500:  movzx  edx,BYTE PTR [rbx-0x1]     ; block flags
17004f504:  cmp    dl,0x3
17004f50c:  movzx  eax,WORD PTR [rbx-0x8]     ; block size
17004f515:  shl    eax,0x4
17004f518:  cmp    rax,r13                    ; size >= requested ?
17004f51b:  jae    found
17004f521:  mov    rbx,QWORD PTR [rbx]        ; next free block
17004f529:  cmp    rbx,r8                     ; list end ?
17004f52c:  jne    17004f500

The caller is Olympus's JSON construction

World.dll → Scripting.dll → edCore.dll → lua.dll        (DCS executing mission Lua)
  → olympus.dll → core.dll → luatools.dll (recursion ×7)     (Lua table → JSON conversion)
  → cpprest_2_10.dll (+0x8d81a: 71%)                     (json::value construction)
  → ucrtbase.dll!malloc → ntdll!RtlAllocateHeap → heap_allocate_block free-list scan

setUnitsData in OlympusCommand.lua hands the DLL a table of 50 units every 50 ms; the DLL converts it into cpprestsdk JSON objects and throws them away. That is ~1,000 allocations and frees per second.

100% of requests are 1,520 bytes

# 5,160 samples taken inside the hot loop: requested size (r13) and heap handle (r12)
block_size 0x5f0 (1.5 KB): 5160 samples (100.0%)
heap handle: only 0x7ffffe220000 (ucrtbase's CRT heap)

Root Cause (wine-11.0 dlls/ntdll/heap.c)

Wine's heap has an LFH (Low Fragmentation Heap) front-end that auto-enables per size class (bin). The enabling conditions live in bin_try_enable():

if (bin == heap->bins && alloc > 0x10) enable = TRUE;
else if (bin - heap->bins < 0x30 && alloc > 0x800) enable = TRUE;          /* small bins: enable after 2048 cumulative allocs */
else if (bin - heap->bins < 0x30 && alloc - freed > 0x10) enable = TRUE;
else if (alloc - freed > 0x400000 / block_size) enable = TRUE;             /* otherwise: live block count only */

First hypothesis (wrong): 1,520 bytes falls in bin 0x37 (≥ 0x30), and LFH activation requires "more than 2,730 live blocks", so Olympus's allocate→immediate-free pattern would never activate LFH for this size. We tested a relaxed ntdll.dll (patch 1) — the scan persisted and CPU climbed along the same slope (40% at 10 min → 55% at 34 min). Reading struct heap directly from process memory showed bin 0x37 with enabled = 1, and the standard-path counters were not increasing.

Confirmed mechanism: disassembling the direct caller reported by LBR, RtlAllocateHeap+0x725, showed it is not an ordinary allocation but an LFH group allocation (group_allocate, inlined).

17001b5e0:  mov    rax,r12          ; r12 = 0x30 (bin 2 = 48-byte blocks)
17001b5e3:  shl    rax,0x5
17001b5e7:  sub    rax,r12          ; rax = 31 * 0x30  (GROUP_BLOCK_COUNT = 31)
17001b5ea:  lea    r8,[r15+rax*1+0x27]
17001b5f6:  and    r8,0xfffffffffffffff0   ; = 0x5f0 = group block size
  ...
17001b655:  call   heap_allocate_block     ; ← 100% of the scan comes from here

• 0x5f0 = 1,520 bytes is the size of an LFH group: "31 × 48-byte JSON nodes + headers". cpprest creates large numbers of json::value nodes in 48-byte units.
• Where emptied LFH groups go: thread affinity slot (1) → bin shared list (up to 32) → beyond that, group_release() frees them back to the standard heap. Olympus creates and discards thousands of nodes per cycle, so more than 33 groups empty every cycle and the surplus is freed. Next cycle, group_allocate() re-allocates via heap_allocate_block(0x5f0). This happens hundreds to a thousand times per second.
• While LFH is active, requests reaching the standard path are effectively only these group allocations. The remainders produced by first-fit splitting ("fragments below 0x5f0, i.e. 0x500–0x5ef") are never consumed by anyone — requests of that size class get absorbed by LFH and never reach the standard path — so they keep piling up on the same list and are all skipped on every group allocation. Fragment count grows proportionally with uptime, so scan time grows proportionally too.
• Windows' LFH does not return groups to the standard heap at this scale, so it cannot happen on real Windows. It is Wine-specific; from Olympus's perspective the code is "fine on Windows".

Hypotheses checked and ruled out

• Players / SAMs / mission: same slope in runs with zero players.
• DCS 2.9.29 update: post-update runs without Olympus are flat for 6 hours.
• Olympus lists growing (units/weapons/logs): the backend API's /olympus/units is 62 KB, /weapons steady at 4 entries. The Lua-side Olympus.units also removes dead units.
• Experiment forcing LFH via heap manipulation from Lua: holding 4,000 strings of ~1,520 bytes changed nothing. DCS's Lua VM uses its own allocator and does not share the heap with cpprest (lua.dll imports realloc but never uses it).
• Split locks / wineserver / memory pressure: all a few percent or less.

Fix

1. ntdll.dll: stop returning emptied LFH groups to the standard heap (the primary fix)

--- a/dlls/ntdll/heap.c
+++ b/dlls/ntdll/heap.c
@@ heap_release_bin_group()
-    if (RtlQueryDepthSList( &bin->groups ) <= ARRAY_SIZE(affinity_mapping))   /* 32 */
+    if (RtlQueryDepthSList( &bin->groups ) <= 0x1000)
     {
         RtlInterlockedPushEntrySList( &bin->groups, &group->entry );
         return STATUS_SUCCESS;
     }
     return group_release( heap, flags, bin, group );   /* ← never falls through to here anymore */

@@ bin_try_enable()   (auxiliary: LFH-enable all bins by alloc frequency. Ineffective on its own)
-    else if (bin - heap->bins < 0x30 && alloc > 0x800) enable = TRUE;
+    else if (alloc > 0x800) enable = TRUE;

Keep up to 0x1000 empty groups per bin (for bin 2 that caps at 1,520 bytes × 4,096 = 6 MB). Groups no longer return to the standard heap, so the group_allocate() scan never happens and no fragments are created. Applied to the wine-11.0 source, rebuilt only the PE ntdll.dll with mingw (configure 7 s, make 4 s) and swapped out /opt/wine-stable/lib/wine/x86_64-windows/ntdll.dll. The unix-side ntdll.so stays stock (same source version). Swap by copying into the same directory and then mv -f (rename), so running processes' old mappings aren't broken.

Pre-verification: in a throwaway container using the same image as production, ran 20,000 cycles of an allocate→free load per cycle (48 B ×4,000 + 24 B ×2,000 + 1,512 B ×50 + small blocks). wineboot and execution both fine; stock 230 µs/cycle → patched 205 µs/cycle (−11%). The synthetic load could not reproduce stock's time-proportional degradation (production fragmentation requires multi-threading + hundreds of thousands of cycles), so the mechanism was confirmed with perf on production.

2. Interim mitigation (when the patch isn't usable)

Changing setUnitsData's period in OlympusCommand.lua from return time + 0.05 to 0.25 cuts allocations to 1/5 and stretches time-to-saturation to ~12 hours (the map's all-units update cycle goes 0.8 s → 4 s). Fragmentation itself still accumulates.

Status / Verification

• 2026-09-04 23:12 JST: rebooted server1 with patch 1 only (relaxed LFH activation rules) → CPU 40% → 55% at 34 min, scan still present. Ineffective. This verification revealed the caller is the group allocation.
• 2026-09-04 23:51 JST: applied patches 1+2, rebooted with Olympus period still 0.05 s. Perf 4 minutes in: ntdll.dll at 9.6% of the sim thread (40% at the same point in the previous run, 85% when saturated), no heap_allocate_block scan detected; DCS core code 56%. Main CPU 26% (37–40% at the same point in the previous run).
• 2026-09-05 01:47 JST update: 2 hours after the fix, completely flat with zero players. Previously it reached 100% at the 120-minute mark.
• 2026-09-05 02:46 JST update: 175 minutes in (0 players), CPU 38.2% / 117.7 FPS. The 3-hour criterion passes.RSS 6.8 → 8.3 GB (estimated mission-side growth, within the historical 11.4 GB seen inside a 12-h restart cycle). Behaviour during manned hours continues to be recorded.

# 4 minutes after reboot, sim-thread breakdown (perf, per DSO)
              previous run (patch 1 only)   this run (patches 1+2)
ntdll.dll              40.8%              9.6%   ← heap_allocate_block gone
DCS core [JIT]         38.5%             55.6%
ntdll.so (syscall)     11.1%             20.0%

• Report filed upstream to Wine: "a workload that creates and discards thousands of same-sized allocations per cycle returns emptied LFH groups to the standard heap beyond 32, and the re-allocation first-fit scan grows linearly over time from accumulated split fragments". Reproduction data (perf/LBR/registers/disassembly) is in this article.
• Also informing the Olympus upstream, including that it does not happen on Windows. Reducing throwaway JSON objects would make it lighter under Wine too.

Environment

• DCS World Dedicated Server 2.9.29.27468 (MT build, --server --norender)
• Linux (Ubuntu 24.04 / kernel 7.0) + Wine 11.0 stable (WineHQ bookworm package) + Docker
• CPU: Intel Core Ultra 7 265K — 8 P-cores (CPU 0-7, up to 5.5 GHz) + 12 E-cores (CPU 8-19, up to 4.6 GHz), no SMT
• Process management: DCSServerBot 3.0.4 (Linux native, launching Wine's DCS_server.exe via sudo -u <user> wine)
• Mission: permanent free flight (777 units / 473 groups, of which 309 client slots, 87 SAM sites · 20 EWR)

Symptom

When a player takes off, the sim thread saturates one core (process CPU 105–118%) and server FPS drops from the usual 110 to 5–30. It recovers immediately once everyone lands.

The player-count dependence is non-linear: some days 16 players get 52 FPS, other days one player gets 20 FPS. On 8/30, 4–5 players sustained 6.6 FPS for over an hour. RAM / I/O / swap idle — pure CPU-bound.

Note: DCS 2.9.x dedicated servers are already the multithreaded (MT) build, but the simulation core itself runs on one thread. MT offloads mostly the rendering side, and a --norender server benefits little. This is the kind of problem that "throwing more cores at it" does not solve.

Method

• Stratified analysis of 45 days of DCSServerBot Monitoring's per-minute serverstats (FPS / process CPU / player count), grouped by mission and player count
• JOIN with statistics (who flew what, when) to correlate low-FPS windows with airframes and pilots
• Sampled the sim thread's allowed CPUs (Cpus_allowed_list from /proc/<pid>/task/*/status) and the running core (ps -o psr) 60 times at 1 s intervals
• Read through DCSServerBot's core/process/processmanager.py and core/process/linux/cpu.py

Finding 1: the sim thread was pinned to E-cores

DCSServerBot 3.0's EXPERIMENTAL auto_affinity (ProcessManager, added 2026-01) automatically assigns 1–2 physical cores of CPU affinity to the executable it launches. The Linux implementation classifies topology by sysfs physical_package_id (= 0 for every core) and always treats Efficiency Class as 0, so it cannot distinguish P-cores from E-cores.

Affinity is set on the Wine launcher process and inherited by DCS's main thread (= the sim thread) through exec. DCS's worker threads reset their own mask to all cores, so only the sim thread stays trapped on 1–2 cores.

# Measured that day (empty, mission running)
Cpus_allowed_list (main thread) : 6,8        ← 1 P-core + 1 E-core
running core, 60 s sampling      : cpu8 = 55× / cpu6 = 5×   ← 92% of the time on an E-core
other 66 threads                 : 0-19 (unrestricted)

The post-launch rebalancing (the "cooperative" pass every 2 s) only reaches the launcher process (the DCS process runs under a different UID and sched_setaffinity returns EPERM), so the cores drawn at launch stay pinned for the entire 12-hour run. Days that draw P-cores are fast; days that draw E-cores are slow with the same player count. This explains the "performance differs by day" observation.

Note it ran despite auto_affinity never being enabled in nodes.yaml. Estimated cause: ProcessManager is a singleton and initialised with its default (auto_affinity=True) before config is passed in (upstream behaviour).

Finding 2: A/B test of the SAM radar-control script

The mission-bundled SAM_RadarOparationLogic (a Lua that scans all groups every 5 s and toggles SAM radar emission based on EWR detection) was a suspect, so from 8/20 we ran a version (B) with the script replaced by a no-op stub for comparison. Under B, all SAM radars are permanently ON.

# server1、45 days, RUNNING only
mission                state       avg FPS   5%tile   FPS<50    FPS<25    proc CPU
A: original (Lua on)    empty       110.3     103.3    0.0 %     0.0 %     36.9 %
A: original (Lua on)    flying      100.2      57.5    3.1 %     0.8 %     53.3 %
B: stub (all ON)        empty        94.2      67.2    0.7 %     0.3 %     39.1 %
B: stub (all ON)        flying       76.7      31.7   12.1 %     2.4 %     54.5 %

# avg FPS by player count (A / B)
1p: 106 / 85   2p: 101 / 83   4p: 97 / 74   5p: 91 / 65   8p: 72 / 40

Conclusion: the cost is not the Lua but the engine-side radar simulation once radars are ON. The script lowers load by keeping all radars OFF while nobody is airborne and turning on only ones with contacts in detection range; removing it makes things worse. The fundamental cause is the density itself — 87 SAM sites / 20 EWR — and it gets heavier as players fly into detection envelopes (consistent with low FPS mostly on low-altitude A-10 / Su-25T / attacker flights).

Not the cause (suspects cleared)

• Wine sync overhead (wineserver): wineserver's CPU time is ~5–8% of the sim thread's. ntsync (kernel sync primitives on Linux 6.14+) is supported by Wine 11.0, but WineHQ's bookworm package is built without it, so using it requires rebuilding Wine. Expected gain was small, so skipped this time.
• Other server instances on the same host: no difference in server1 flying FPS whether other instances were running or stopped (90.6 vs 89.2).
• Memory / swap: PSI (memory / io) during low-FPS windows is 0.
• Process uptime: a separate phenomenon where unmanned FPS decays 106 → 96 over 12 h exists (see the [Investigation] article below), but the collapse to 5–30 FPS here happens right after startup too.

Fix

1. Explicitly pin DCS's sim thread to P-cores (nodes.yaml)

When affinity is set explicitly per instance, DCSServerBot applies it verbatim and takes the instance out of auto_affinity management. On the Core Ultra 7 265K, P-cores are CPU 0-7.

# nodes.yaml (under each node's instances.<name>)
      affinity: 0,1,2,3,4,5,6,7

Applying it requires restarting the bot container (= restarting DCS). After restart, check every DCS process shows Cpus_allowed_list: 0-7 right after boot; dcs.log's Created boot pool: n:8 also changes to 8 cores (previously n:20).

To apply to an already-running process, run as the same UID as DCS and target all threads (even root gets EPERM with a different UID):

sudo -u <dcs-user> taskset -a -cp 0-7 <pid of DCS_server.exe>

We did not adopt the cgroup cpuset approach (docker's --cpuset-cpus): if auto_affinity picked only cores outside the cpuset, sched_setaffinity returns EINVAL and DCS startup can fail with an exception.

2. Revert the mission to the original (SAM radar-control Lua enabled)

B was consistently worse than the original, so we reverted to the original. With 0 players online, pointed serverSettings.lua's current / listStartIndex back at the original mission and restarted DCS.

Result

# Immediate comparison on the same mission (B), near-identical load (0–1 players) — E-core → P-core
                      players  FPS      proc CPU
before (pinned to cpu8)   0.8     107.3    56.5 %
after (0-7)                1.0     112.6    49.5 %     ← same work, ~12% less CPU time

# after rebooting with the original mission + P-core pinning (1 player)
                                   116.7    39.1 %    ← on par with A's historical baseline (110 FPS / 37%)

Unmanned FPS sticks to the ceiling (~110–118), so the effect first shows up in process CPU %. The FPS improvement under load will be tracked via future serverstats (single-thread performance difference between E-cores and P-cores is roughly 25–35%).

Status

• 2026-09-04 addendum: the earlier claim in this article's Result that "the post-fix rise of 40–78% was due to more players" was wrong — it was a separate Olympus-origin phenomenon (heap free-list scanning growing proportionally with uptime). See the article above, "FPS collapses to 28 within ~2.5 hours after enabling Olympus". This article's E-core pinning and A/B conclusions are unchanged.
• P-core pinning and the mission revert were applied on 2026-09-03.
• Effect verification: over the next 2–4 weeks we will compare the number of "FPS < 50 sustained for 5+ minutes" episodes per 100 hours against the pre-fix A-side baseline of 3.79 per 100 h.
• As a root-cause measure, revisiting the mission's SAM / EWR density (number of simultaneously emitting radars) is under consideration.
• The fact that DCSServerBot's Linux auto_affinity ignores P/E cores will be reported upstream. If you run DCSServerBot on a Linux host with a hybrid CPU, set affinity explicitly.

Environment

• DCS World Dedicated Server (Linux / Wine, running under Docker)
• Process management via DCSServerBot, scheduled restarts every 12 hours
• Mission: large permanent map (~580 AI units, ~300 client slots)
• Observed player counts: 1–15

Symptom

Players reported "everyone in the server jumps/teleports at the same instant" — a momentary freeze of a few hundred milliseconds, recurring irregularly.

A second symptom accompanied it: the mission gets generally heavier the longer it has been running (a rubber-banding feel). It later turned out both are different faces of the same cause.

Why it is hard to detect

Standard monitoring barely catches it. Why the diagnosis took time:

• Server FPS is a 1-minute average, so a 300 ms stall barely shows up. Stalls happened in windows where average FPS looked healthy.
• The NIC's total byte counters can't distinguish it. Telemetry export and DB traffic share the same interface, so even if game-state delivery stops for everyone, the aggregate keeps flowing.
  Measured: while all clients were silent for 300 ms, the largest interface-level gap was only 25 ms.
• Process-wide CPU usage (1 s averages) also masks a 300 ms full-core burst.

Method

Capturing it required measuring the following three things at the same time.

1. Sample the sim thread at 200 Hz with nanosecond precision

Jiffy-based CPU time (/proc/<pid>/stat) can show false periods from quantisation noise. Use schedstat instead.

# field 1 = cumulative on-CPU time (ns), field 2 = run-queue wait (ns)
cat /proc/<pid>/task/<main_tid>/schedstat

2. Detect per-client send gaps

Aggregates hide it, so extract only UDP leaving the game port with a BPF filter and track silence per destination. Payloads are not read.

# # BPF: IPv4 && UDP && src port == <game port>
# AF_PACKET raw socket + SO_ATTACH_FILTER
# keep last-seen per destination IP; record silences over the threshold
#
# verdict: silences of multiple clients start AND end together  → server side
#          only one client                                      → that client's link

3. Thread state "during" the stall

Grab all threads' states and wchan at the instant a stall is detected. Post-hoc captures can't tell.

# # R  = running       → stuck in computation
# D  = uninterruptible → waiting on I/O
# S + wchan          → what it waits on is visible by name

Findings

Stalls happen server-side, for everyone at once

Example with two clients: starts within 13 ms of each other, recoveries within 1 ms. Not explainable by individual links.

client A  silent  T+0.295 .. T+0.598   (303 ms)
client B  silent  T+0.308 .. T+0.597   (289 ms)

A 9-client sample confirmed all nine went silent simultaneously.

During stalls the sim thread is "computing"

Every captured stall had thread state R (running). Major faults at the same moment: 0. It is a computation stall, not I/O. All 7 captured stalls aligned with a CPU burst within the same second.

Bursts grow with process uptime

Sim-thread burst measurements with the player count held near-constant (0–2):

process uptime  burst length (median)   duty ratio
  +0.3 h            56 ms            0.3 %
  +2.3 h            46 ms            0.9 %
  +4.3 h            46 ms            3.6 %
  +6.3 h            50 ms            6.8 %
  +8.3 h            65 ms            9.1 %
 +11.3 h            90 ms           12.7 %

Burst frequency levels off at ~1.5/s while each burst keeps getting longer — matching the behaviour of a fixed-period job scanning ever-growing data.

The stall rate has a clear threshold

Rate with time-on-connected-players as the denominator:

mission age     exposure        stalls    rate
  2- 4 h         65.8 min        0         0.0 /h
  4- 6 h         92.2 min        0         0.0 /h
  6- 8 h         43.3 min        0         0.0 /h
  8-10 h         96.6 min        3         1.9 /h
 10-12 h         56.5 min        3         3.2 /h

Across 201 player-hours total, no stalls were recorded below 8 hours of uptime.

The FPS decay is another face of the same cause

3-week aggregate (with 3–5 players). Time below 60 FPS:

mission age        avg FPS     time below 60 FPS
 fresh   (< 4h)     114.9       0 min / 459 min
 mid     (4-8h)     102.6       0 min / 595 min
 stale   (8h+)       86.8     137 min / 794 min  (low of 7.2 FPS)

Root Cause Isolation

To isolate whether the accumulation lives in the mission's Lua state or in the DCS process, we reloaded only the mission while keeping the process alive (done with 0 players).

                          PID        duty ratio
mission +4.2h             same       0.0524
mission-reload only       same       0.0642   ← not reset
(ref) full process restart new     0.122 → 0.003   ← reset

Reloading the mission does not clear the accumulation — showing the cause is on the DCS server-process side, not the mission scripts (MIST / CTLD etc.). Optimising scripts won't help.

Only a full process restart resets it, reproduced independently across two scheduled restarts (duty ratio 0.122→0.003 and 0.127→0.001).

Mitigation

We can't touch the process-side accumulation itself, so we handle it by shortening the restart interval.

The requirement is "busy hours must not exceed 8 hours of process uptime". A 12-hour cycle can't keep the busy band under 8 hours, so we moved to an 8-hour cycle (3 restarts/day) and placed the restart times so one lands just before the busy band.

Restart times should be chosen from each community's occupancy distribution; DCSServerBot users can list multiple times in the scheduler plugin's action.times.

Status

The mechanism is confirmed. Mitigation is not yet applied and its effect is unverified. Once applied, we'll track the stall rate during busy hours with the same instrumentation.

Also still unknown:

• What exactly accumulates on the process side has not been identified (only that it is not mission-script-derived is certain)
• Lack of measured stall-length data under high load with 10+ players

Related (separate issue)

During the investigation we observed the kernel swapping out the DCS process due to host memory pressure (165 major faults/s). With vm.swappiness left at its default 60, the server's working set gets evicted before the page cache. On latency-sensitive hosts, set it low.

sudo sysctl -w vm.swappiness=10

However, no causal link to this article's stall phenomenon has been established. Major faults were 0 during the periods stalls were captured — treat it as a separate problem.

Environment

• DCS World Server 2.9.27.24969 (latest as of 2026-06)
• Launched on Linux using Wine + Docker (aterfax/dcs-world-dedicated-server)
• Scope: Wine-based Linux DCS servers in general (incl. DCSSB Option C / Linux-native)
• Affected Wine versions: wine-8.0 (Debian bookworm default); wine-11.0 upstream also unfixed initially

Symptom

The DCS dedicated server (DCS_server.exe) crashes right after launch with no log output. Wine reports:

EXCEPTION_WINE_STUB: 0x80000100

With WINEDEBUG=err+module set:

err:module:import_dll Library MSVCP140_ATOMIC_WAIT.dll (which is needed by DCS_server.exe) not found
err:module:loader_init Importing dlls for DCS_server.exe failed, status c0000135

The server writes no dcs.log and never reaches Listening on port XXXXX.

Root Cause

In DCS 2.9.27.24969 the msvcp140.dll dependency graph was updated so that msvcp140_atomic_wait.dll now statically imports __std_tzdb_get_sys_info and __std_tzdb_delete_sys_info.

These are time-zone DB query functions used by the C++ STL's chrono timestamp formatting. In wine-8.0 and wine-11.0, msvcp140_atomic_wait.dll only has @ stub entries for them, so when DCS calls them at DLL load it raises EXCEPTION_WINE_STUB (0x80000100) and aborts.

Relevant Microsoft STL ABI (the _Sys_info struct):

struct tzdb_sys_info {
    int32_t error;      // offset 0 (0=success)
    // 4 bytes padding
    double begin;       // offset 8  — range start (epoch ms)
    double end;         // offset 16 — range end   (epoch ms)
    int32_t offset;     // offset 24 — UTC offset in ms
    int32_t save;       // offset 28 — DST offset in ms
    char   *abbrev;     // offset 32 — timezone abbreviation
};                      // total: 40 bytes (_CRT_PACKING=8)

Fix

Build a patched msvcp140_atomic_wait.dll from the wine-11.0 sources, with minimal implementations of the two problematic functions, and replace it.

1. Edit the spec file (dlls/msvcp140_atomic_wait/msvcp140_atomic_wait.spec)

Change the two stub entries to real exports:

@ stdcall __std_tzdb_get_sys_info(ptr long double)
@ stdcall __std_tzdb_delete_sys_info(ptr)

Also remove any @ stub entries for __std_atomic_*_indirect / __std_atomic_*_cmpxchg16b if present (another crash source).

2. Add the implementation to main.c

struct tzdb_sys_info {
    enum tzdb_error error;
    double begin;
    double end;
    int32_t offset;
    int32_t save;
    char *abbrev;
};

struct tzdb_sys_info * __stdcall __std_tzdb_get_sys_info(const char *name, size_t len, double sys)
{
    struct tzdb_sys_info *info = calloc(1, sizeof(*info));
    if (!info) return NULL;
    info->error  = TZDB_ERROR_SUCCESS;
    info->begin  = -1e15;
    info->end    =  1e15;
    info->offset = 0;
    info->save   = 0;
    info->abbrev = malloc(4);
    if (info->abbrev) {
        info->abbrev[0] = 'U';
        info->abbrev[1] = 'T';
        info->abbrev[2] = 'C';
        info->abbrev[3] = 0;
    }
    return info;
}

void __stdcall __std_tzdb_delete_sys_info(struct tzdb_sys_info *info)
{
    if (!info) return;
    free(info->abbrev);
    free(info);
}

3. Build and install

# # Inside a Debian bookworm container (winehq-stable installed):
apt-get source winehq-stable   # or fetch the wine-11.0 source tarball
# # After applying the edits above:
./configure --without-x --without-freetype --enable-win64
cd dlls/msvcp140_atomic_wait && make

# # Install
cp .libs/msvcp140_atomic_wait.dll     /opt/wine-stable/lib/wine/x86_64-windows/
cp .libs/msvcp140_atomic_wait.dll.so  /opt/wine-stable/lib/wine/x86_64-unix/

4. Fix the Wine prefix (if migrated from wine-8.0)

A prefix initialised with wine-8.0 needs two extra steps:

# # a) replace the old 128KB native PE stub in system32 with wine-11.0's 2KB fake PE stub
cp /opt/wine-stable/lib/wine/x86_64-windows/msvcp140_atomic_wait.dll \
   "$WINEPREFIX/drive_c/windows/system32/msvcp140_atomic_wait.dll"

# b) # b) fix user.reg — wine-8.0 registers the DLL as "native", which breaks loading
sed -i 's/"[*]msvcp140_atomic_wait"="native"/"*msvcp140_atomic_wait"="builtin"/' \
   "$WINEPREFIX/user.reg"

Known Side Effect

DCS log timestamps become fixed to UTC (the implementation always returns UTC offset=0). Cosmetic only; no gameplay impact.

Tested on

• DCS 2.9.27.24969, wine-stable 11.0.0.0~bookworm-1
• Linux, Docker, DCSSB Option C (Linux-native bot managing Wine DCS)
• Confirmed all 3 server instances reach simulation started, state=ssRunning