Lỗi driver_irql_not_less_or_equal netio.sys năm 2024

I just wonder if somebody could help me with analysing memory.dmp file. Recently we moved around 80 machines to Office 365 and Azure Active Directory. Mostly they are working fine, however 8 of them blue screen every few days (or more often). There is no pattern in terms of hardware, as we have 5 years old HP ZBook, few HP EliteBook 840 G3, few new XPS (PC, laptops different models). Analyse of memory.dmp file shows, that all of them are crashing due to DRIVER_IRQL_NOT_LESS_OR_EQUAL and NETIO.SYS which is not very helpful. All machines are rebuilt with Windows 10 Pro image and latest drivers.

Common things to all (crashing and not crashing) machines in this company:

  • All are Azure AD joined
  • Intune and policies set
  • Office 365 and OneDrive installed
  • ESET Endpoint Security as antivirus
  • Duo Security as 2FA
  • Safetica as DLP (only monitoring)
  • All laptops are encrypted, however some with BitLocker other with ESET EE

We are trying to resolve this issue for few weeks with no success do I would be very grateful if somebody could help

Thank you

  • SysnativeFileCollectionApp.zip 2.8 MB · Views: 7
  • PC Info.txt 140.1 KB · Views: 4

Lỗi driver_irql_not_less_or_equal netio.sys năm 2024

philc43

BSOD Forum Moderator, BSOD Academy Instructor, BSOD Kernel Dump Expert
  • 2

Hello PiotrIr,

I can see you have been a member for some time but this is your first post! Welcome!

I've looked at your two most recent crash dumps provided in the FileCollection zip and can see reference to a Rivet Networks Killer Network driver which seems to be the cause for the crash.

Please try and look into this driver - it may be getting loaded by mistake as there is also another driver listed in the msinfo kfecosvc KfeCoSvc c:\windows\system32\drivers\rivetnetworks\killer\kfeco10x64.sys

Code:

12: kd> .load pde;!dpx

PDE v11.3 - Copyright 2017 Andrew Richards
Start memory scan : 0xfffff580712b55a8 ($csp) End memory scan : 0xfffff580712b8000 (Kernel Stack Base) rsp : 0xfffff580712b55a8 : 0xfffff80152407769 : nt!KiBugCheckDispatch+0x69 r11 : 0xfffff580712b56e8 : 0xfffff80152403a69 : nt!KiPageFault+0x469 0xfffff580712b55a8 : 0xfffff80152407769 : nt!KiBugCheckDispatch+0x69 0xfffff580712b55d0 : 0xfffff80157dcd7f4 : NETIO!StreamInvokeCalloutAndNormalizeAction+0x5c Unable to load image \SystemRoot\system32\DRIVERS\epfwwfp.sys, Win32 error 0n2 *** WARNING: Unable to verify timestamp for epfwwfp.sys 0xfffff580712b56e8 : 0xfffff80152403a69 : nt!KiPageFault+0x469 Unable to load image \SystemRoot\system32\drivers\netmonitor_wfp.sys, Win32 error 0n2 *** WARNING: Unable to verify timestamp for netmonitor_wfp.sys 0xfffff580712b5858 : 0xfffff80157dcd7f4 : NETIO!StreamInvokeCalloutAndNormalizeAction+0x5c 0xfffff580712b58b0 : 0xfffff80157e3a000 : NETIO!WPP_GLOBAL_Control 0xfffff580712b58b8 : 0xfffff80157dcdc52 : NETIO!StreamUnpublishClassifyData+0xda 0xfffff580712b58e8 : 0xfffff80157e1bf08 : NETIO!StreamDataBlock+0x158 0xfffff580712b5948 : 0xfffff80157dcd508 : NETIO!StreamProcessCallout+0x3fc 0xfffff580712b5a78 : 0xfffff80157dccb86 : NETIO!ProcessCallout+0x706 0xfffff580712b5b60 : 0xfffff80157e3a590 : NETIO!handleTableLock 0xfffff580712b5bf8 : 0xfffff80157dc95eb : NETIO!ArbitrateAndEnforce+0x71b 0xfffff580712b5d58 : 0xfffff80157dc818a : NETIO!KfdClassify+0x37a 0xfffff580712b60e8 : 0xfffff801522ec5ca : nt!HalPerformEndOfInterrupt+0x1a 0xfffff580712b6148 : 0xfffff80157dc789b : NETIO!StreamClassify+0x28b 0xfffff580712b62e8 : 0xfffff80157dc74ba : NETIO!StreamCommonInspect+0x282 0xfffff580712b62f8 : 0xfffff8015224caf2 : nt!ExFreeHeapPool+0x362 0xfffff580712b66a8 : 0xfffff80157e3a000 : NETIO!WPP_GLOBAL_Control 0xfffff580712b66c8 : 0xfffff80157dc6dcd : NETIO!WfpStreamInspectReceive+0x18d 0xfffff580712b66d8 : 0xfffff80157e63d24 : NDIS!NdisAllocateNetBufferAndNetBufferList+0x214 0xfffff580712b66e0 : 0xfffff80157dc26d0 : NETIO!NetioCompleteCopyNetBufferListChain 0xfffff580712b6758 : 0xfffff8015931eca0 : tcpip!InetInspectReceive+0x80 0xfffff580712b6768 : 0xfffff8015931ee00 : tcpip!TcpCovetNetBufferList+0xf8 0xfffff580712b6798 : 0xfffff8015931ee39 : tcpip!TcpCovetNetBufferList+0x131 0xfffff580712b6808 : 0xfffff8015931ebb8 : tcpip!TcpInspectReceive+0x9c 0xfffff580712b6988 : 0xfffff80159345ede : tcpip!WfpTlShimInspectRecvTcpDatagram+0x2de 0xfffff580712b6ba8 : 0xfffff80159302a52 : tcpip!IpNlpFastContinueSendDatagrams+0x502 0xfffff580712b6bf8 : 0xfffff801592e77a1 : tcpip!TcpTcbExtractDatagram+0xd1 0xfffff580712b6c80 : 0xfffff801594d1938 : tcpip!AddressFamilyInformation 0xfffff580712b6ce8 : 0xfffff801592e54ee : tcpip!TcpTcbReceive+0x30e 0xfffff580712b6cf8 : 0xfffff801593039e4 : tcpip!IppSendDatagramsCommon+0x4c4 0xfffff580712b6e48 : 0xffffcd0c14f740c0 : 0xfffff801594c9230 : tcpip!Ipv4Global 0xfffff580712b6e50 : 0xfffff801594c9230 : tcpip!Ipv4Global 0xfffff580712b6e78 : 0xfffff801593024a9 : tcpip!IpNlpFastSendDatagram+0x349 0xfffff580712b6e90 : 0xfffff801594c9230 : tcpip!Ipv4Global 0xfffff580712b6ea8 : 0xfffff80157e63ee3 : NDIS!NdisAllocateNetBufferList+0xc3 0xfffff580712b6eb0 : 0xfffff801594c9230 : tcpip!Ipv4Global 0xfffff580712b6ef8 : 0xfffff80159308897 : tcpip!ShimIpPacketInV4+0x93 0xfffff580712b6f38 : 0xfffff801592e470f : tcpip!TcpMatchReceive+0x51f *** WARNING: Unable to verify timestamp for KfeCo10X64.sys 0xfffff580712b70a0 : 0xfffff80158e07a90 : fwpkclnt!FwppInjectComplete 0xfffff580712b70d8 : 0xfffff801529b1019 : nt!ExFreePool+0x9 0xfffff580712b7148 : 0xfffff801592e0d4e : tcpip!TcpValidateReceive+0xee 0xfffff580712b7158 : 0xfffff801594cec00 : tcpip!Fl6tpProviderState+0x6f0 0xfffff580712b7168 : 0xfffff80157dc5b64 : NETIO!NetioDereferenceNetBufferListChain+0x104 0xfffff580712b71e8 : 0xfffff801592e3798 : tcpip!TcpReceive+0x358 0xfffff580712b71f8 : 0xfffff801594cec00 : tcpip!Fl6tpProviderState+0x6f0 0xfffff580712b7208 : 0xfffff801594c9598 : tcpip!Ipv4Global+0x368 0xfffff580712b7218 : 0xfffff8015942bb01 : tcpip!McTemplateK0qbr0qbr2qqqqqqqqqqqqqqqq_EtwWriteTransfer+0xe9 0xfffff580712b7230 : 0xfffff801594c9230 : tcpip!Ipv4Global 0xfffff580712b72a0 : 0xfffff801594c9230 : tcpip!Ipv4Global 0xfffff580712b72a8 : 0xfffff801594c9230 : tcpip!Ipv4Global 0xfffff580712b72b8 : 0xffffcd0c14f740c0 : 0xfffff801594c9230 : tcpip!Ipv4Global 0xfffff580712b72d0 : 0xfffff801594c9598 : tcpip!Ipv4Global+0x368 0xfffff580712b72d8 : 0xfffff80159349142 : tcpip!TcpNlClientReceiveDatagrams+0x22 0xfffff580712b7318 : 0xfffff8015930a241 : tcpip!IppProcessDeliverList+0xc1 0xfffff580712b7348 : 0xfffff801594c9230 : tcpip!Ipv4Global 0xfffff580712b73a8 : 0xfffff801522083eb : nt!KeQueryCurrentStackInformationEx+0x8b 0xfffff580712b73d0 : 0xfffff801594c9230 : tcpip!Ipv4Global 0xfffff580712b73f8 : 0xfffff80159306b8b : tcpip!IppReceiveHeaderBatch+0x21b 0xfffff580712b7408 : 0xfffff8015235460e : nt!KiExpandKernelStackAndCalloutSwitchStack+0x11e 0xfffff580712b74e0 : 0xfffff801594c9230 : tcpip!Ipv4Global 0xfffff580712b74f8 : 0xfffff8015930610f : tcpip!IppFlcReceivePacketsCore+0x32f 0xfffff580712b7500 : 0xfffff801594c9230 : tcpip!Ipv4Global 0xfffff580712b7548 : 0xfffff801594c9230 : tcpip!Ipv4Global 0xfffff580712b75c8 : 0xfffff801592f2e86 : tcpip!RtlAcquireReadLock+0x42 0xfffff580712b7618 : 0xfffff80159430152 : tcpip!IppInspectInjectReceiveEx+0x172 0xfffff580712b7658 : 0xfffff80158e086a0 : fwpkclnt!FwppInjectionStackCallout 0xfffff580712b7668 : 0xfffff8015942ffd4 : tcpip!IppInspectInjectReceive+0x24 0xfffff580712b76c8 : 0xfffff80158e087b6 : fwpkclnt!FwppInjectionStackCallout+0x116 0xfffff580712b76d0 : 0xfffff80158e086a0 : fwpkclnt!FwppInjectionStackCallout 0xfffff580712b76e8 : 0xfffff8015235460e : nt!KiExpandKernelStackAndCalloutSwitchStack+0x11e 0xfffff580712b7738 : 0xfffff80158e086a0 : fwpkclnt!FwppInjectionStackCallout 0xfffff580712b7758 : 0xfffff80152354488 : nt!KeExpandKernelStackAndCalloutInternal+0x78 0xfffff580712b77c8 : 0xfffff801523543fd : nt!KeExpandKernelStackAndCalloutEx+0x1d 0xfffff580712b77d0 : 0xfffff80158e086a0 : fwpkclnt!FwppInjectionStackCallout 0xfffff580712b77f8 : 0xfffff80158e07fd7 : fwpkclnt!FwppInjectPrologue+0x13b 0xfffff580712b7808 : 0xfffff80158e0a2b4 : fwpkclnt!NetioExpandKernelStackAndCallout+0x58 0xfffff580712b7848 : 0xfffff80158e09ea4 : fwpkclnt!FwpsInjectTransportReceiveAsync0+0x304 0xfffff580712b7b08 : 0xfffff80152317e25 : nt!PspSystemThreadStartup+0x55 0xfffff580712b7b58 : 0xfffff801523fcdd8 : nt!KiStartSystemThread+0x28 0xfffff580712b7b70 : 0xfffff80152317dd0 : nt!PspSystemThreadStartup

Last edited: Dec 14, 2020

Lỗi driver_irql_not_less_or_equal netio.sys năm 2024

  • 4

Many thanks for your reply and welcome.

Is any thing in memory.dmp which wold point to something else than NIC driver? I have 8 different machines with different NIC models (Intel as well), on some I've already updated drivers - and this didn't help...

Lỗi driver_irql_not_less_or_equal netio.sys năm 2024

philc43

BSOD Forum Moderator, BSOD Academy Instructor, BSOD Kernel Dump Expert
  • 5

Without seeing the crash dumps from all 8 machines it would be difficult to give an opinion. If any of the other 8 do

not

have the Rivet Killer Network device I would be interested in seeing their crash dumps.

Lỗi driver_irql_not_less_or_equal netio.sys năm 2024

  • 6

There are tcpip.sys and fwpkclnt.sys, both microsoft drivers. Fwpkclnt.sys could have something to do with security, hence firewall/antivirus. You could try to uninstall eset altogether in this machine to see if it resolves the issue.

Last edited: Dec 14, 2020

Lỗi driver_irql_not_less_or_equal netio.sys năm 2024

  • 7

It's not do with the NIC directly, it's related to one of the network filter drivers, which usually end up causing issues since the level of privilege they have within the kernel.

Rich (BB code):

12: kd> .trap 0xfffff580712b56f0
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000000 rbx=0000000000000000 rcx=fffff580712b59f8
rdx=ffffcd0c332eb7c0 rsi=0000000000000000 rdi=0000000000000000
rip=fffff80157dcd7f4 rsp=fffff580712b5880 rbp=fffff580712b58f9
 r8=ffffcd0c332eb7c0  r9=0000000000000045 r10=ffffcd0c347693f0
r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
NETIO!StreamInvokeCalloutAndNormalizeAction+0x5c:
fffff801`57dcd7f4 488b4808        mov     rcx,qword ptr [rax+8] ds:00000000`00000008=????????????????

The crash was caused because a driver used a pointer which contains a garbage address and hence couldn't be resolved by the page fault handler. I would be suspicous of ESET due to the issues I've seen it cause in the past. It would be worthwhile seeing the other crash dumps though.

  • 8

Thank you for all your replies. I will organise other dump file from machine which doesn't have killer NIC - they show the same reason - DRIVER_IRQL_NOT_LESS_OR_EQUAL and NETIO.SYS. We are on latest version of ESET and we are using it on around 500 machines so this is why I didn't suspect this. I'm attaching crash dump from another XPS with killer.

  • SysnativeFileCollectionApp.zip 3.7 MB · Views: 3
  • 9

This is crash dump from laptop which doesn't have the Killer NIC. I just wander - is i possible to list network filter drivers which may be responsible for this issue?

  • SysnativeFileCollectionApp.zip 375.6 KB · Views: 2

Lỗi driver_irql_not_less_or_equal netio.sys năm 2024

  • 10

    Thank you for all your replies. I will organise other dump file from machine which doesn't have killer NIC - they show the same reason - DRIVER_IRQL_NOT_LESS_OR_EQUAL and NETIO.SYS. We are on latest version of ESET and we are using it on around 500 machines so this is why I didn't suspect this. I'm attaching crash dump from another XPS with killer.

5 minidumps for this machine. They say BugCheck 116, {DifferentAddressForAllDumps, DifferentAddressForAllDumps, 0, d} Probably caused by : nvlddmkm.sys.

Resolution:

  • You may need to install the latest updates for your display driver, so that it properly supports the TDR process.
  • Hardware issues that impact the ability of the video card to operate properly, including:
    • Over-clocked components, such as the motherboard
    • Incorrect component compatibility and settings (especially memory configuration and timings)
    • Insufficient system cooling
    • Insufficient system power
    • Defective parts (memory modules, motherboards, etc.)
  • Visual effects, or too many programs running in the background may be slowing your PC down so that the video card can not respond as necessary.

Given that the driver is already updated, you can try the NVIDIA Studio Driver (instead of game ready). If that won't work: - load bios (optimized) defaults - run memtest (see hardware tutorials).

Last edited: Dec 14, 2020

  • 11

Hmm, as all 8 laptops are on similar set (O365, Intune, Safetica, ESET, Duo) I was suspecting the same reason for all of them, especially memory dump points to the same - DRIVER_IRQL_NOT_LESS_OR_EQUAL and NETIO.SYS.

Lỗi driver_irql_not_less_or_equal netio.sys năm 2024

philc43

BSOD Forum Moderator, BSOD Academy Instructor, BSOD Kernel Dump Expert
  • 12

    This is crash dump from laptop which doesn't have the Killer NIC. I just wander - is i possible to list network filter drivers which may be responsible for this issue?

This set of logfiles for the laptop without the Killer driver does not show any BSOD crash dumps.

What I can see that is common to both laptop type whether Killer or Intel is the driver called: netmonitor_wfp.sys Thu Jul 2 11:14:53 2020

This was implicated in the crash analysis I did for you in my first post. I could not determine where this driver came from, perhaps you will know?

Lỗi driver_irql_not_less_or_equal netio.sys năm 2024

  • 13

If you're able could you please check the following directory on one of the problematic machines:

The file will need to be zipped and then uploaded to a file sharing site such as DropBox or OneDrive.

I want try and see if I find what netmonitor_wfp.sys belongs to.

  • 15

Thank you! I'm really grateful for your help.

Yes, netmonitor_wfp belongs to Safetca and this may cause BOSD (it has in past). However is past I got clear pointing in memory.dmp which showed this. I can log a call with their support but somehow need to proof this is their issue. Is it possible to list filters attached to network adapter? Do they have also order which matters? I would like to have something to compare with laptops which actually work...

Dump file from EliteBook below: MEMORY.zip

Lỗi driver_irql_not_less_or_equal netio.sys năm 2024

  • 16

Thanks for providing me with the dump file, I'll have a look into it and see what I can find.

  • 18

Thank you! I will start from removing safetica first to see if this resolved the issue!

Lỗi driver_irql_not_less_or_equal netio.sys năm 2024

  • 19

Rich (BB code):

ffff868e`b07942f8  fffff804`6b3d4b29 nt!KiBugCheckDispatch+0x69
ffff868e`b0794300  00000000`0000000a
ffff868e`b0794308  00000000`00000008
ffff868e`b0794310  00000000`00000002
ffff868e`b0794318  00000000`00000000
ffff868e`b0794320  fffff804`6f06bdb2 NETIO!StreamInvokeCalloutAndNormalizeAction+0x62
[...]
ffff868e`b07943e8  ffff868e`b0794630
ffff868e`b07943f0  ffff0584`a0635c23
ffff868e`b07943f8  fffff804`73a1b39b epfwwfp+0xb39b
ffff868e`b0794400  ffffcd02`978892b0
ffff868e`b0794408  ffff868e`b0794750
ffff868e`b0794410  ffffcd02`978892b0
ffff868e`b0794418  ffff868e`b0794a70
ffff868e`b0794420  ffff868e`b0794a00
ffff868e`b0794428  ffff868e`b0794728
ffff868e`b0794430  ffffcd02`94ccc4b0
ffff868e`b0794438  fffff804`6b3d0e69 nt!KiPageFault+0x469 << Crash because of the illegal page fault
[...]
ffff868e`b07944a8  fffff804`673c4450 netmonitor_wfp+0x14450 << Suspect this driver
[...]
ffff868e`b0794528  fffff804`767a1b41 Netwtw06+0x1b1b41 << Intel Wi-Fi driver
ffff868e`b0794530  ffff868e`b0794ef8
ffff868e`b0794538  fffff804`73a13257 epfwwfp+0x3257 << ESET firewall driver
ffff868e`b0794540  ffff868e`b07946d0
ffff868e`b0794548  fffff804`766cd879 Netwtw06+0xdd879
ffff868e`b0794550  00000000`00000000
ffff868e`b0794558  ffff868e`b0794728
ffff868e`b0794560  00000000`00000000
ffff868e`b0794568  ffff868e`b0794a70
ffff868e`b0794570  fffff804`6f0d7000 NETIO!WPP_GLOBAL_Control
ffff868e`b0794578  fffff804`673b148f netmonitor_wfp+0x148f
[...]
ffff868e`b07945a8  fffff804`6f06bdb2 NETIO!StreamInvokeCalloutAndNormalizeAction+0x62

The reason I'm suspicious of the Safetca driver is because it's the last third-party callout driver which was present at the time of the crash.

Rich (BB code):

3: kd> knL
 # Child-SP          RetAddr           Call Site
00 ffff868e`b07942f8 fffff804`6b3d4b29 nt!KeBugCheckEx
01 ffff868e`b0794300 fffff804`6b3d0e69 nt!KiBugCheckDispatch+0x69
02 ffff868e`b0794440 fffff804`6f06bdb2 nt!KiPageFault+0x469
03 ffff868e`b07945d0 fffff804`6f06b879 NETIO!StreamInvokeCalloutAndNormalizeAction+0x62
04 ffff868e`b07946a0 fffff804`6f06aed9 NETIO!StreamProcessCallout+0x2c9
05 ffff868e`b07947e0 fffff804`6f06d1da NETIO!ProcessCallout+0x5a9
06 ffff868e`b0794960 fffff804`6f069b91 NETIO!ArbitrateAndEnforce+0xc3a
07 ffff868e`b0794af0 fffff804`6f06933a NETIO!KfdClassify+0x5a1
08 ffff868e`b0794f00 fffff804`6f068f63 NETIO!StreamClassify+0x26a
09 ffff868e`b0795090 fffff804`6f068bdc NETIO!StreamCommonInspect+0x293
0a ffff868e`b0795470 fffff804`6f5ed0b8 NETIO!WfpStreamInspectReceive+0x19
[...]

Essentially, what is happening is the NIC receives a TCP packet which is then passed through the network stack and then passed to the filter engine which then executes the filter callout drivers. In this case, it's the ESET firewall and the Safetica monitoring program, which will inspect the network data byte stream and do what they've been written to do.

More information is available here - Windows Filtering Platform Architecture Overview - Windows drivers

Here's a list of the third-party callout filter drivers which are possible culprits:

Rich (BB code):

3: kd> r $t0=ffffcd02`85935000;.for ( r $t1=0; @$t1 < 400; r $t1=@$t1+1 ) {dps @$t0+2*@$ptrsize L2; r $t0=@$t0+50;}
[...]
ffffcd02`8593b0a8  fffff804`736f8d40 epfw+0x8d40 << ESET
[...]
ffffcd02`8593b0f0  fffff804`73a13ce0 epfwwfp+0x3ce0 << ESET Firewall Driver
[...]
ffffcd02`8593b4b0  fffff804`673b1460 netmonitor_wfp+0x1460 << Safetca Driver
[...]

  • 20

Thank you! This is great answer. I've just removed Safetica on 3 machines to see if this resolve issue.