I just wonder if somebody could help me with analysing memory.dmp file. Recently we moved around 80 machines to Office 365 and Azure Active Directory. Mostly they are working fine, however 8 of them blue screen every few days [or more often]. There is no pattern in terms of hardware, as we have 5 years old HP ZBook, few HP EliteBook 840 G3, few new XPS [PC, laptops different models]. Analyse of memory.dmp file shows, that all of them are crashing due to DRIVER_IRQL_NOT_LESS_OR_EQUAL and NETIO.SYS which is not very helpful. All machines are rebuilt with Windows 10 Pro image and latest drivers.

Common things to all [crashing and not crashing] machines in this company:

• All are Azure AD joined
• Intune and policies set
• Office 365 and OneDrive installed
• ESET Endpoint Security as antivirus
• Duo Security as 2FA
• Safetica as DLP [only monitoring]
• All laptops are encrypted, however some with BitLocker other with ESET EE

We are trying to resolve this issue for few weeks with no success do I would be very grateful if somebody could help

Thank you

• SysnativeFileCollectionApp.zip 2.8 MB · Views: 7
• PC Info.txt 140.1 KB · Views: 4

philc43

BSOD Forum Moderator, BSOD Academy Instructor, BSOD Kernel Dump Expert

• 2

Hello PiotrIr,

I can see you have been a member for some time but this is your first post! Welcome!

I've looked at your two most recent crash dumps provided in the FileCollection zip and can see reference to a Rivet Networks Killer Network driver which seems to be the cause for the crash.

Please try and look into this driver - it may be getting loaded by mistake as there is also another driver listed in the msinfo kfecosvc KfeCoSvc c:\windows\system32\drivers\rivetnetworks\killer\kfeco10x64.sys

Code:

12: kd> .load pde;!dpx

PDE v11.3 - Copyright 2017 Andrew Richards

Start memory scan : 0xfffff580712b55a8 [$csp]
End memory scan : 0xfffff580712b8000 [Kernel Stack Base]
               rsp : 0xfffff580712b55a8 : 0xfffff80152407769 : nt!KiBugCheckDispatch+0x69
               r11 : 0xfffff580712b56e8 : 0xfffff80152403a69 : nt!KiPageFault+0x469
0xfffff580712b55a8 : 0xfffff80152407769 : nt!KiBugCheckDispatch+0x69
0xfffff580712b55d0 : 0xfffff80157dcd7f4 : NETIO!StreamInvokeCalloutAndNormalizeAction+0x5c
Unable to load image \SystemRoot\system32\DRIVERS\epfwwfp.sys, Win32 error 0n2
*** WARNING: Unable to verify timestamp for epfwwfp.sys
0xfffff580712b56e8 : 0xfffff80152403a69 : nt!KiPageFault+0x469
Unable to load image \SystemRoot\system32\drivers\netmonitor_wfp.sys, Win32 error 0n2
*** WARNING: Unable to verify timestamp for netmonitor_wfp.sys
0xfffff580712b5858 : 0xfffff80157dcd7f4 : NETIO!StreamInvokeCalloutAndNormalizeAction+0x5c
0xfffff580712b58b0 : 0xfffff80157e3a000 : NETIO!WPP_GLOBAL_Control
0xfffff580712b58b8 : 0xfffff80157dcdc52 : NETIO!StreamUnpublishClassifyData+0xda
0xfffff580712b58e8 : 0xfffff80157e1bf08 : NETIO!StreamDataBlock+0x158
0xfffff580712b5948 : 0xfffff80157dcd508 : NETIO!StreamProcessCallout+0x3fc
0xfffff580712b5a78 : 0xfffff80157dccb86 : NETIO!ProcessCallout+0x706
0xfffff580712b5b60 : 0xfffff80157e3a590 : NETIO!handleTableLock
0xfffff580712b5bf8 : 0xfffff80157dc95eb : NETIO!ArbitrateAndEnforce+0x71b
0xfffff580712b5d58 : 0xfffff80157dc818a : NETIO!KfdClassify+0x37a
0xfffff580712b60e8 : 0xfffff801522ec5ca : nt!HalPerformEndOfInterrupt+0x1a
0xfffff580712b6148 : 0xfffff80157dc789b : NETIO!StreamClassify+0x28b
0xfffff580712b62e8 : 0xfffff80157dc74ba : NETIO!StreamCommonInspect+0x282
0xfffff580712b62f8 : 0xfffff8015224caf2 : nt!ExFreeHeapPool+0x362
0xfffff580712b66a8 : 0xfffff80157e3a000 : NETIO!WPP_GLOBAL_Control
0xfffff580712b66c8 : 0xfffff80157dc6dcd : NETIO!WfpStreamInspectReceive+0x18d
0xfffff580712b66d8 : 0xfffff80157e63d24 : NDIS!NdisAllocateNetBufferAndNetBufferList+0x214
0xfffff580712b66e0 : 0xfffff80157dc26d0 : NETIO!NetioCompleteCopyNetBufferListChain
0xfffff580712b6758 : 0xfffff8015931eca0 : tcpip!InetInspectReceive+0x80
0xfffff580712b6768 : 0xfffff8015931ee00 : tcpip!TcpCovetNetBufferList+0xf8
0xfffff580712b6798 : 0xfffff8015931ee39 : tcpip!TcpCovetNetBufferList+0x131
0xfffff580712b6808 : 0xfffff8015931ebb8 : tcpip!TcpInspectReceive+0x9c
0xfffff580712b6988 : 0xfffff80159345ede : tcpip!WfpTlShimInspectRecvTcpDatagram+0x2de
0xfffff580712b6ba8 : 0xfffff80159302a52 : tcpip!IpNlpFastContinueSendDatagrams+0x502
0xfffff580712b6bf8 : 0xfffff801592e77a1 : tcpip!TcpTcbExtractDatagram+0xd1
0xfffff580712b6c80 : 0xfffff801594d1938 : tcpip!AddressFamilyInformation
0xfffff580712b6ce8 : 0xfffff801592e54ee : tcpip!TcpTcbReceive+0x30e
0xfffff580712b6cf8 : 0xfffff801593039e4 : tcpip!IppSendDatagramsCommon+0x4c4
0xfffff580712b6e48 : 0xffffcd0c14f740c0 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b6e50 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b6e78 : 0xfffff801593024a9 : tcpip!IpNlpFastSendDatagram+0x349
0xfffff580712b6e90 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b6ea8 : 0xfffff80157e63ee3 : NDIS!NdisAllocateNetBufferList+0xc3
0xfffff580712b6eb0 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b6ef8 : 0xfffff80159308897 : tcpip!ShimIpPacketInV4+0x93
0xfffff580712b6f38 : 0xfffff801592e470f : tcpip!TcpMatchReceive+0x51f
*** WARNING: Unable to verify timestamp for KfeCo10X64.sys
0xfffff580712b70a0 : 0xfffff80158e07a90 : fwpkclnt!FwppInjectComplete
0xfffff580712b70d8 : 0xfffff801529b1019 : nt!ExFreePool+0x9
0xfffff580712b7148 : 0xfffff801592e0d4e : tcpip!TcpValidateReceive+0xee
0xfffff580712b7158 : 0xfffff801594cec00 : tcpip!Fl6tpProviderState+0x6f0
0xfffff580712b7168 : 0xfffff80157dc5b64 : NETIO!NetioDereferenceNetBufferListChain+0x104
0xfffff580712b71e8 : 0xfffff801592e3798 : tcpip!TcpReceive+0x358
0xfffff580712b71f8 : 0xfffff801594cec00 : tcpip!Fl6tpProviderState+0x6f0
0xfffff580712b7208 : 0xfffff801594c9598 : tcpip!Ipv4Global+0x368
0xfffff580712b7218 : 0xfffff8015942bb01 : tcpip!McTemplateK0qbr0qbr2qqqqqqqqqqqqqqqq_EtwWriteTransfer+0xe9
0xfffff580712b7230 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b72a0 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b72a8 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b72b8 : 0xffffcd0c14f740c0 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b72d0 : 0xfffff801594c9598 : tcpip!Ipv4Global+0x368
0xfffff580712b72d8 : 0xfffff80159349142 : tcpip!TcpNlClientReceiveDatagrams+0x22
0xfffff580712b7318 : 0xfffff8015930a241 : tcpip!IppProcessDeliverList+0xc1
0xfffff580712b7348 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b73a8 : 0xfffff801522083eb : nt!KeQueryCurrentStackInformationEx+0x8b
0xfffff580712b73d0 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b73f8 : 0xfffff80159306b8b : tcpip!IppReceiveHeaderBatch+0x21b
0xfffff580712b7408 : 0xfffff8015235460e : nt!KiExpandKernelStackAndCalloutSwitchStack+0x11e
0xfffff580712b74e0 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b74f8 : 0xfffff8015930610f : tcpip!IppFlcReceivePacketsCore+0x32f
0xfffff580712b7500 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b7548 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b75c8 : 0xfffff801592f2e86 : tcpip!RtlAcquireReadLock+0x42
0xfffff580712b7618 : 0xfffff80159430152 : tcpip!IppInspectInjectReceiveEx+0x172
0xfffff580712b7658 : 0xfffff80158e086a0 : fwpkclnt!FwppInjectionStackCallout
0xfffff580712b7668 : 0xfffff8015942ffd4 : tcpip!IppInspectInjectReceive+0x24
0xfffff580712b76c8 : 0xfffff80158e087b6 : fwpkclnt!FwppInjectionStackCallout+0x116
0xfffff580712b76d0 : 0xfffff80158e086a0 : fwpkclnt!FwppInjectionStackCallout
0xfffff580712b76e8 : 0xfffff8015235460e : nt!KiExpandKernelStackAndCalloutSwitchStack+0x11e
0xfffff580712b7738 : 0xfffff80158e086a0 : fwpkclnt!FwppInjectionStackCallout
0xfffff580712b7758 : 0xfffff80152354488 : nt!KeExpandKernelStackAndCalloutInternal+0x78
0xfffff580712b77c8 : 0xfffff801523543fd : nt!KeExpandKernelStackAndCalloutEx+0x1d
0xfffff580712b77d0 : 0xfffff80158e086a0 : fwpkclnt!FwppInjectionStackCallout
0xfffff580712b77f8 : 0xfffff80158e07fd7 : fwpkclnt!FwppInjectPrologue+0x13b
0xfffff580712b7808 : 0xfffff80158e0a2b4 : fwpkclnt!NetioExpandKernelStackAndCallout+0x58
0xfffff580712b7848 : 0xfffff80158e09ea4 : fwpkclnt!FwpsInjectTransportReceiveAsync0+0x304
0xfffff580712b7b08 : 0xfffff80152317e25 : nt!PspSystemThreadStartup+0x55
0xfffff580712b7b58 : 0xfffff801523fcdd8 : nt!KiStartSystemThread+0x28
0xfffff580712b7b70 : 0xfffff80152317dd0 : nt!PspSystemThreadStartup

Last edited: Dec 14, 2020

• 4

Many thanks for your reply and welcome.

Is any thing in memory.dmp which wold point to something else than NIC driver? I have 8 different machines with different NIC models [Intel as well], on some I've already updated drivers - and this didn't help...

philc43

BSOD Forum Moderator, BSOD Academy Instructor, BSOD Kernel Dump Expert

• 5

Without seeing the crash dumps from all 8 machines it would be difficult to give an opinion. If any of the other 8 do

not

have the Rivet Killer Network device I would be interested in seeing their crash dumps.

• 6

There are tcpip.sys and fwpkclnt.sys, both microsoft drivers. Fwpkclnt.sys could have something to do with security, hence firewall/antivirus. You could try to uninstall eset altogether in this machine to see if it resolves the issue.

Last edited: Dec 14, 2020

• 7

It's not do with the NIC directly, it's related to one of the network filter drivers, which usually end up causing issues since the level of privilege they have within the kernel.

Rich [BB code]:

12: kd> .trap 0xfffff580712b56f0
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000000 rbx=0000000000000000 rcx=fffff580712b59f8
rdx=ffffcd0c332eb7c0 rsi=0000000000000000 rdi=0000000000000000
rip=fffff80157dcd7f4 rsp=fffff580712b5880 rbp=fffff580712b58f9
 r8=ffffcd0c332eb7c0  r9=0000000000000045 r10=ffffcd0c347693f0
r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
NETIO!StreamInvokeCalloutAndNormalizeAction+0x5c:
fffff801`57dcd7f4 488b4808        mov     rcx,qword ptr [rax+8] ds:00000000`00000008=????????????????

The crash was caused because a driver used a pointer which contains a garbage address and hence couldn't be resolved by the page fault handler. I would be suspicous of ESET due to the issues I've seen it cause in the past. It would be worthwhile seeing the other crash dumps though.

• 8

Thank you for all your replies. I will organise other dump file from machine which doesn't have killer NIC - they show the same reason - DRIVER_IRQL_NOT_LESS_OR_EQUAL and NETIO.SYS. We are on latest version of ESET and we are using it on around 500 machines so this is why I didn't suspect this. I'm attaching crash dump from another XPS with killer.

• SysnativeFileCollectionApp.zip 3.7 MB · Views: 3

• 9

This is crash dump from laptop which doesn't have the Killer NIC. I just wander - is i possible to list network filter drivers which may be responsible for this issue?

• SysnativeFileCollectionApp.zip 375.6 KB · Views: 2

• 10

  Thank you for all your replies. I will organise other dump file from machine which doesn't have killer NIC - they show the same reason - DRIVER_IRQL_NOT_LESS_OR_EQUAL and NETIO.SYS. We are on latest version of ESET and we are using it on around 500 machines so this is why I didn't suspect this. I'm attaching crash dump from another XPS with killer.

5 minidumps for this machine. They say BugCheck 116, {DifferentAddressForAllDumps, DifferentAddressForAllDumps, 0, d} Probably caused by : nvlddmkm.sys.

Resolution:

• You may need to install the latest updates for your display driver, so that it properly supports the TDR process.
• Hardware issues that impact the ability of the video card to operate properly, including:
  • Over-clocked components, such as the motherboard
  • Incorrect component compatibility and settings [especially memory configuration and timings]
  • Insufficient system cooling
  • Insufficient system power
  • Defective parts [memory modules, motherboards, etc.]
• Visual effects, or too many programs running in the background may be slowing your PC down so that the video card can not respond as necessary.

Given that the driver is already updated, you can try the NVIDIA Studio Driver [instead of game ready]. If that won't work: - load bios [optimized] defaults - run memtest [see hardware tutorials].

Last edited: Dec 14, 2020

• 11

Hmm, as all 8 laptops are on similar set [O365, Intune, Safetica, ESET, Duo] I was suspecting the same reason for all of them, especially memory dump points to the same - DRIVER_IRQL_NOT_LESS_OR_EQUAL and NETIO.SYS.

philc43

BSOD Forum Moderator, BSOD Academy Instructor, BSOD Kernel Dump Expert

• 12

  This is crash dump from laptop which doesn't have the Killer NIC. I just wander - is i possible to list network filter drivers which may be responsible for this issue?

This set of logfiles for the laptop without the Killer driver does not show any BSOD crash dumps.

What I can see that is common to both laptop type whether Killer or Intel is the driver called: netmonitor_wfp.sys Thu Jul 2 11:14:53 2020

This was implicated in the crash analysis I did for you in my first post. I could not determine where this driver came from, perhaps you will know?

• 13

If you're able could you please check the following directory on one of the problematic machines:

The file will need to be zipped and then uploaded to a file sharing site such as DropBox or OneDrive.

I want try and see if I find what netmonitor_wfp.sys belongs to.

• 15

Thank you! I'm really grateful for your help.

Yes, netmonitor_wfp belongs to Safetca and this may cause BOSD [it has in past]. However is past I got clear pointing in memory.dmp which showed this. I can log a call with their support but somehow need to proof this is their issue. Is it possible to list filters attached to network adapter? Do they have also order which matters? I would like to have something to compare with laptops which actually work...

Dump file from EliteBook below: MEMORY.zip

• 16

Thanks for providing me with the dump file, I'll have a look into it and see what I can find.

• 18

Thank you! I will start from removing safetica first to see if this resolved the issue!

• 19

Rich [BB code]:

ffff868e`b07942f8  fffff804`6b3d4b29 nt!KiBugCheckDispatch+0x69
ffff868e`b0794300  00000000`0000000a
ffff868e`b0794308  00000000`00000008
ffff868e`b0794310  00000000`00000002
ffff868e`b0794318  00000000`00000000
ffff868e`b0794320  fffff804`6f06bdb2 NETIO!StreamInvokeCalloutAndNormalizeAction+0x62
[...]
ffff868e`b07943e8  ffff868e`b0794630
ffff868e`b07943f0  ffff0584`a0635c23
ffff868e`b07943f8  fffff804`73a1b39b epfwwfp+0xb39b
ffff868e`b0794400  ffffcd02`978892b0
ffff868e`b0794408  ffff868e`b0794750
ffff868e`b0794410  ffffcd02`978892b0
ffff868e`b0794418  ffff868e`b0794a70
ffff868e`b0794420  ffff868e`b0794a00
ffff868e`b0794428  ffff868e`b0794728
ffff868e`b0794430  ffffcd02`94ccc4b0
ffff868e`b0794438  fffff804`6b3d0e69 nt!KiPageFault+0x469