FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

MORTON JR., AL

BMWG,

 

After extensive discussion at previous sessions, a revised draft (08) is available for

another Working Group Last Call.

 

Please review and express your support and/or comments on the bmwg-list by

 

     23:59 UTC on May 17, 2021

 

Thanks to the author/editors for their efforts to bring us to this point, and to everyone

who has participated in previous reviews!

 

see you on the list,

Al

bmwg co-chair and doc shepherd

 

 

 

From: bmwg <[hidden email]> On Behalf Of [hidden email]
Sent: Thursday, April 22, 2021 1:28 PM
To: [hidden email]
Subject: [bmwg] New version of draft-ietf-bmwg-ngfw-performance

 

Folks,

 

The latest draft has been posted and is ready for review. The Diff between version 7 and version 8 can be found here:

 

https://tools.ietf.org/rfcdiff?difftype=--hwdiff&url2=draft-ietf-bmwg-ngfw-performance-08.txt

 

and here:

 

https://tools.ietf.org/rfcdiff?url2=draft-ietf-bmwg-ngfw-performance-08.txt

 

Please review and provide comments. I believe Al Morton is going to call for last call.

 

Brian

 

 


_______________________________________________
bmwg mailing list
[hidden email]
https://www.ietf.org/mailman/listinfo/bmwg
Reply | Threaded
Open this post in threaded view
|

Re: FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

Gábor LENCSE

Dear Al, Authors and BMWG members,

I have reviewed the first 22 pages of the draft.

As far as I could read, I have found the draft reasonable and well written. I have found some minor issues, which I classified as Discuss/Bugs/Nits:

Discuss:
--------

page 13-14

   (Option 2)  DUT/SUT deployment scenario 2 : 0.1-0.2 Mbit/s per IP
               (e.g.  50,000-100,000 IPs per 10Gbit/s throughput)

[...]

   (Option 1)  100 % IPv4, no IPv6

[...]

   Note: The IANA has assigned IP address range for the testing purpose
   as described in Section 8.

However, from the 198.18.0.0/15 range, only 198.18.0.0/16 can be used on the one side, 198.19.0.0/16 is for the other side of the DUT.
A /16 has only 64k number of IP addresses, thus 100,000 IPs can not be ensured for the 10Gbit/s throughput. (And of course, there are even faster links, e.g. 25/40/100Gbit/s)

As a possible resolution, perhaps the usage of the private use 10.0.0.0/8 range could be allowed?

------------------

page 14

   For HTTP traffic emulation, the emulated browser MUST negotiate HTTP
   1.1.  HTTP persistence MAY be enabled depending on the test scenario.
   The browser MAY open multiple TCP connections per Server endpoint IP
   at any time depending on how many sequential transactions are needed
   to be processed.  Within the TCP connection multiple transactions MAY
   be processed if the emulated browser has available connections.  The
   browser SHOULD advertise a User-Agent header.  Headers MUST be sent
   uncompressed.  The browser SHOULD enforce content length validation.


I highly recommend to include HTTP/2 (defined by RFC 7540 in 2015), too.

Rationale:

"The standardization effort was supported by Chrome, Opera, Firefox,[11] Internet Explorer 11, Safari, Amazon Silk, and Edge browsers.[12] Most major browsers had added HTTP/2 support by the end of 2015.[13] About 97% of web browsers used have the capability,[14] while according to W3Techs, as of April 2021, 50% of the top 10 million websites supported HTTP/2.[15]"
Source: https://en.wikipedia.org/wiki/HTTP/2

As I expect that HTTP/2 will dominate withing a few years, I recommend to include HTTP/2 with a SHOULD and include HTTP/1.1 either also with a SHOULD or with a MAY.

What do others think?

------------------

page 16

   The server pool for HTTP SHOULD listen on TCP port 80 and emulate
   HTTP version 1.1 with persistence. 


As for the HTTP version, please see my comments regarding the client side text on page 14.

------------------

page 16

   [...]  The behavior of the
   client is to sweep through the given server IP space, sequentially
   generating a recognizable service by the DUT. 

I am not sure if sequential sweep is the right choice for two reasons:
- I feel that the spirit of RFC 4814 would rather suggest pseudorandom. To surely include all of them, I recommend random permutation.
- According to my experience, iptables has shown significantly different maximum connection establishment rate, when I used pseudorandom port numbers or when I enumerated the port numbers in increasing order. (Details available upon request.)

------------------




Bugs:
-----

Page 9 / Table 3:

DLP ??? -- Previously, in Table 1 (on page 7)  DLP was used for Data Loss Protection

I do not understand this definition:

   +------------------+------------------------------------------------+
   | DLP              | DUT/SUT detects and blocks the transmission    |
   |                  | of Personally Identifiable Information (PII)   |
   |                  | and specific files across the monitored network|
   +------------------+------------------------------------------------+

To me it sounds like preventing the leaking of sensitive information.
If it is so, then I would call DLP as Data *Leak* Protection in Table 1.

------------------

page 13

   o  The IPv4 Type of Service (ToS) byte or IPv6 traffic class should
      be set to '00' or '000000' respectively.

According to RFC 8200, IPv6 Traffic Class field is 8 bits wide.

------------------

page 16

   o  A default gateway is permitted.  The IPv4 ToS byte and IPv6
      traffic class bytes should be set to '00' and '000000'
      respectively.

According to RFC 8200, IPv6 Traffic Class field is 8 bits wide.

------------------

page 18

   Test bed reference pre-tests help to ensure that the maximum desired
   traffic generator aspects such as throughput, transaction per second,
   connection per second, concurrent connection, and latency.

I do not understand this sentence. I feel that something is missing from the end of the sentence (e.g. "are satisfied").


Nits:
-----

page 5

   In some deployment scenarios, the network security devices (Device
   Under Test/System Under Test) are connected to routers and switches
   which will reduce the number of entries in MAC or ARP tables of the
   Device Under Test/System Under Test (DUT/SUT). 

I THINK it is non-defining:

swithes which
-->
swithes, which

------------------

Page 9 / Table 3:

   +------------------+------------------------------------------------+
   | Logging and      | DUT/SUT logs and reports all traffic at the    |
   | Reporting        | flow level across the monitored.               |
   +------------------+------------------------------------------------+

-->

   +------------------+------------------------------------------------+
   | Logging and      | DUT/SUT logs and reports all traffic at the    |
   | Reporting        | flow level across the monitored network.       |
   +------------------+------------------------------------------------+

------------------

   o  All RECOMMENDED security inspection enabled
-->
   o  All RECOMMENDED security inspections enabled

I think here "inspection" is used in a countable sense (various types of inspections), thus it has a plural, please refer to: https://www.wordhippo.com/what-is/the-plural-of/inspection.html


------------------

page 13

   SYN/ACK, ACK).  and it MUST be closed via either a TCP three way
-->
   SYN/ACK, ACK), and it MUST be closed via either a TCP three way

------------------

page 14

   profile,the
-->
   profile, the

------------------

page 17

use a set approximate number of
-->
use a set of approximate number of

(If I understood the sentence correctly.)

------------------

page 17

traffic SHOULD ramp from zero
-->
traffic SHOULD ramp up from zero

(perhaps)

------------------

page 19

   2.  Summary of test bed software and Hardware details
-->
   2.  Summary of test bed software and hardware details

(I felt it inconsistent to capitalize "software" and "hardware" differently.)

 

page 22

HTTP Transaction Latency (Section 7.8)
-->
HTTPS Transaction Latency (Section 7.8)


That's all for today. (Now I had only so much time.)

If you find my comments useful, I would be happy to read the rest of the document, however, I cannot promise to keep the below deadline due to my other workload.

Best regards,

Gábor

On 5/3/2021 7:36 PM, MORTON JR., AL wrote:

BMWG,

 

After extensive discussion at previous sessions, a revised draft (08) is available for

another Working Group Last Call.

 

Please review and express your support and/or comments on the bmwg-list by

 

     23:59 UTC on May 17, 2021

 

Thanks to the author/editors for their efforts to bring us to this point, and to everyone

who has participated in previous reviews!

 

see you on the list,

Al

bmwg co-chair and doc shepherd

 

 

 

From: bmwg [hidden email] On Behalf Of [hidden email]
Sent: Thursday, April 22, 2021 1:28 PM
To: [hidden email]
Subject: [bmwg] New version of draft-ietf-bmwg-ngfw-performance

 

Folks,

 

The latest draft has been posted and is ready for review. The Diff between version 7 and version 8 can be found here:

 

https://tools.ietf.org/rfcdiff?difftype=--hwdiff&url2=draft-ietf-bmwg-ngfw-performance-08.txt

 

and here:

 

https://tools.ietf.org/rfcdiff?url2=draft-ietf-bmwg-ngfw-performance-08.txt

 

Please review and provide comments. I believe Al Morton is going to call for last call.

 

Brian

 

 


_______________________________________________
bmwg mailing list
[hidden email]
https://www.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[hidden email]
https://www.ietf.org/mailman/listinfo/bmwg
Reply | Threaded
Open this post in threaded view
|

Re: FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

Brian Monkman

Hi Gabor,

 

Thank you very much for your comments. On initial review they all look reasonable to us. How long do you think it will take you to complete your review of the rest of the document?

 

Brian

 

From: bmwg <[hidden email]> On Behalf Of Gabor LENCSE
Sent: Monday, May 10, 2021 11:09 AM
To: [hidden email]
Subject: Re: [bmwg] FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

 

Dear Al, Authors and BMWG members,

I have reviewed the first 22 pages of the draft.

As far as I could read, I have found the draft reasonable and well written. I have found some minor issues, which I classified as Discuss/Bugs/Nits:

Discuss:
--------

page 13-14

   (Option 2)  DUT/SUT deployment scenario 2 : 0.1-0.2 Mbit/s per IP
               (e.g.  50,000-100,000 IPs per 10Gbit/s throughput)

[...]

   (Option 1)  100 % IPv4, no IPv6

[...]

   Note: The IANA has assigned IP address range for the testing purpose
   as described in Section 8.

However, from the 198.18.0.0/15 range, only 198.18.0.0/16 can be used on the one side, 198.19.0.0/16 is for the other side of the DUT.
A /16 has only 64k number of IP addresses, thus 100,000 IPs can not be ensured for the 10Gbit/s throughput. (And of course, there are even faster links, e.g. 25/40/100Gbit/s)

As a possible resolution, perhaps the usage of the private use 10.0.0.0/8 range could be allowed?

------------------

page 14

   For HTTP traffic emulation, the emulated browser MUST negotiate HTTP
   1.1.  HTTP persistence MAY be enabled depending on the test scenario.
   The browser MAY open multiple TCP connections per Server endpoint IP
   at any time depending on how many sequential transactions are needed
   to be processed.  Within the TCP connection multiple transactions MAY
   be processed if the emulated browser has available connections.  The
   browser SHOULD advertise a User-Agent header.  Headers MUST be sent
   uncompressed.  The browser SHOULD enforce content length validation.


I highly recommend to include HTTP/2 (defined by RFC 7540 in 2015), too.

Rationale:

"The standardization effort was supported by Chrome, Opera, Firefox,[11] Internet Explorer 11, Safari, Amazon Silk, and Edge browsers.[12] Most major browsers had added HTTP/2 support by the end of 2015.[13] About 97% of web browsers used have the capability,[14] while according to W3Techs, as of April 2021, 50% of the top 10 million websites supported HTTP/2.[15]"
Source: https://en.wikipedia.org/wiki/HTTP/2

As I expect that HTTP/2 will dominate withing a few years, I recommend to include HTTP/2 with a SHOULD and include HTTP/1.1 either also with a SHOULD or with a MAY.

What do others think?

------------------

page 16

   The server pool for HTTP SHOULD listen on TCP port 80 and emulate
   HTTP version 1.1 with persistence. 


As for the HTTP version, please see my comments regarding the client side text on page 14.

------------------

page 16

   [...]  The behavior of the
   client is to sweep through the given server IP space, sequentially
   generating a recognizable service by the DUT. 

I am not sure if sequential sweep is the right choice for two reasons:
- I feel that the spirit of RFC 4814 would rather suggest pseudorandom. To surely include all of them, I recommend random permutation.
- According to my experience, iptables has shown significantly different maximum connection establishment rate, when I used pseudorandom port numbers or when I enumerated the port numbers in increasing order. (Details available upon request.)

------------------




Bugs:
-----

Page 9 / Table 3:

DLP ??? -- Previously, in Table 1 (on page 7)  DLP was used for Data Loss Protection

I do not understand this definition:

   +------------------+------------------------------------------------+
   | DLP              | DUT/SUT detects and blocks the transmission    |
   |                  | of Personally Identifiable Information (PII)   |
   |                  | and specific files across the monitored network|
   +------------------+------------------------------------------------+

To me it sounds like preventing the leaking of sensitive information.
If it is so, then I would call DLP as Data *Leak* Protection in Table 1.

------------------

page 13

   o  The IPv4 Type of Service (ToS) byte or IPv6 traffic class should
      be set to '00' or '000000' respectively.

According to RFC 8200, IPv6 Traffic Class field is 8 bits wide.

------------------

page 16

   o  A default gateway is permitted.  The IPv4 ToS byte and IPv6
      traffic class bytes should be set to '00' and '000000'
      respectively.

According to RFC 8200, IPv6 Traffic Class field is 8 bits wide.

------------------

page 18

   Test bed reference pre-tests help to ensure that the maximum desired
   traffic generator aspects such as throughput, transaction per second,
   connection per second, concurrent connection, and latency.

I do not understand this sentence. I feel that something is missing from the end of the sentence (e.g. "are satisfied").

Nits:
-----

page 5

   In some deployment scenarios, the network security devices (Device
   Under Test/System Under Test) are connected to routers and switches
   which will reduce the number of entries in MAC or ARP tables of the
   Device Under Test/System Under Test (DUT/SUT). 

I THINK it is non-defining:

swithes which
-->
swithes, which

------------------

Page 9 / Table 3:

   +------------------+------------------------------------------------+
   | Logging and      | DUT/SUT logs and reports all traffic at the    |
   | Reporting        | flow level across the monitored.               |
   +------------------+------------------------------------------------+

-->

   +------------------+------------------------------------------------+
   | Logging and      | DUT/SUT logs and reports all traffic at the    |
   | Reporting        | flow level across the monitored network.       |
   +------------------+------------------------------------------------+

------------------

   o  All RECOMMENDED security inspection enabled
-->
   o  All RECOMMENDED security inspections enabled

I think here "inspection" is used in a countable sense (various types of inspections), thus it has a plural, please refer to: https://www.wordhippo.com/what-is/the-plural-of/inspection.html


------------------

page 13

   SYN/ACK, ACK).  and it MUST be closed via either a TCP three way
-->
   SYN/ACK, ACK), and it MUST be closed via either a TCP three way

------------------

page 14

   profile,the
-->
   profile, the

------------------

page 17

use a set approximate number of
-->
use a set of approximate number of

(If I understood the sentence correctly.)

------------------

page 17

traffic SHOULD ramp from zero
-->
traffic SHOULD ramp up from zero

(perhaps)

------------------

page 19

   2.  Summary of test bed software and Hardware details
-->
   2.  Summary of test bed software and hardware details

(I felt it inconsistent to capitalize "software" and "hardware" differently.)

 

page 22

HTTP Transaction Latency (Section 7.8)
-->
HTTPS Transaction Latency (Section 7.8)

 

That's all for today. (Now I had only so much time.)

 

If you find my comments useful, I would be happy to read the rest of the document, however, I cannot promise to keep the below deadline due to my other workload.

 

Best regards,

 

Gábor

 

On 5/3/2021 7:36 PM, MORTON JR., AL wrote:

BMWG,

 

After extensive discussion at previous sessions, a revised draft (08) is available for

another Working Group Last Call.

 

Please review and express your support and/or comments on the bmwg-list by

 

                23:59 UTC on May 17, 2021

 

Thanks to the author/editors for their efforts to bring us to this point, and to everyone

who has participated in previous reviews!

 

see you on the list,

Al

bmwg co-chair and doc shepherd

 

 

 

From: bmwg [hidden email] On Behalf Of [hidden email]
Sent: Thursday, April 22, 2021 1:28 PM
To: [hidden email]
Subject: [bmwg] New version of draft-ietf-bmwg-ngfw-performance

 

Folks,

 

The latest draft has been posted and is ready for review. The Diff between version 7 and version 8 can be found here:

 

https://tools.ietf.org/rfcdiff?difftype=--hwdiff&url2=draft-ietf-bmwg-ngfw-performance-08.txt

 

and here:

 

https://tools.ietf.org/rfcdiff?url2=draft-ietf-bmwg-ngfw-performance-08.txt

 

Please review and provide comments. I believe Al Morton is going to call for last call.

 

Brian

 

 



_______________________________________________
bmwg mailing list
[hidden email]
https://www.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[hidden email]
https://www.ietf.org/mailman/listinfo/bmwg
Reply | Threaded
Open this post in threaded view
|

Re: FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

Gábor LENCSE

Hi Brian,

Perhaps I can do another chunk later this week, and I plan to complete it next week.

Best regards,

Gábor

On 5/10/2021 6:45 PM, [hidden email] wrote:

Hi Gabor,

 

Thank you very much for your comments. On initial review they all look reasonable to us. How long do you think it will take you to complete your review of the rest of the document?

 

Brian



_______________________________________________
bmwg mailing list
[hidden email]
https://www.ietf.org/mailman/listinfo/bmwg
Reply | Threaded
Open this post in threaded view
|

Re: FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

Brian Monkman

Thanks Gabor.

 

Al, could you extend the deadline for WGLC to May 21st from May 17th?

 

Brian

 

From: Gabor LENCSE <[hidden email]>
Sent: Monday, May 10, 2021 12:58 PM
To: [hidden email]
Cc: 'Bala Balarajah' <[hidden email]>; 'Bala Balarajah' <[hidden email]>; [hidden email]; 'MORTON, ALFRED C (AL)' <[hidden email]>; 'Sarah Banks' <[hidden email]>
Subject: Re: [bmwg] FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

 

Hi Brian,

Perhaps I can do another chunk later this week, and I plan to complete it next week.

Best regards,

Gábor

On 5/10/2021 6:45 PM, [hidden email] wrote:

Hi Gabor,

 

Thank you very much for your comments. On initial review they all look reasonable to us. How long do you think it will take you to complete your review of the rest of the document?

 

Brian

 


_______________________________________________
bmwg mailing list
[hidden email]
https://www.ietf.org/mailman/listinfo/bmwg
Reply | Threaded
Open this post in threaded view
|

Re: FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

MORTON JR., AL

of course Brian and Gábor, I can use the extra time myself.

 

 

BMWG,

We will extend the deadline for WGLC to May 21, as requested.

Al

bmwg co-chair

 

From: [hidden email] <[hidden email]>
Sent: Monday, May 10, 2021 1:02 PM
To: 'Gabor LENCSE' <[hidden email]>
Cc: 'Bala Balarajah' <[hidden email]>; 'Bala Balarajah' <[hidden email]>; [hidden email]; 'MORTON, ALFRED C (AL)' <[hidden email]>; 'Sarah Banks' <[hidden email]>
Subject: RE: [bmwg] FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

 

Thanks Gabor.

 

Al, could you extend the deadline for WGLC to May 21st from May 17th?

 

Brian

 

From: Gabor LENCSE <[hidden email]>
Sent: Monday, May 10, 2021 12:58 PM
To: [hidden email]
Cc: 'Bala Balarajah' <[hidden email]>; 'Bala Balarajah' <[hidden email]>; [hidden email]; 'MORTON, ALFRED C (AL)' <[hidden email]>; 'Sarah Banks' <[hidden email]>
Subject: Re: [bmwg] FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

 

Hi Brian,

Perhaps I can do another chunk later this week, and I plan to complete it next week.

Best regards,

Gábor

On 5/10/2021 6:45 PM, [hidden email] wrote:

Hi Gabor,

 

Thank you very much for your comments. On initial review they all look reasonable to us. How long do you think it will take you to complete your review of the rest of the document?

 

Brian

 


_______________________________________________
bmwg mailing list
[hidden email]
https://www.ietf.org/mailman/listinfo/bmwg
Reply | Threaded
Open this post in threaded view
|

Second part -- Re: FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

Gábor LENCSE

Dear Al, Authors and BMWG Members,

Al, thank you very much for extending the deadline. However, today I managed to finish the review of the draft.

I think this draft is matured enough. I could find only minor formal issues in the text about the benchmarking procedures. (Although I kept the categories of Discuss/Bugs/Nits, I did not find any significant new issue. My entries in the "Discuss" category are either reiterations from my previous review, or rather editorial ones.)

Discuss:
--------

page 25
7.1.4.2.

   [...] The test equipment SHOULD start to measure and record
   all specified KPIs and the frequency of measurements SHOULD be less
   than 2 seconds.


What does the "frequency of measurements" mean?
I would expect that the KPIs are mausered continously.
Perhaps it is the "frequency of results produced", that is, the "time window for collecting the results and calculating their average" should be less than 2 seconds?

------------------

page 26

   The client SHOULD negotiate HTTP 1.1 and close the connection with
   FIN immediately after completion of one transaction.  In each test
   iteration, client MUST send GET command requesting a fixed HTTP
   response object size.

As I mentioned before (I mean: in my previous review), I recommend the inclusion of HTTP/2.

------------------

page 27

   c.  During the sustain phase, traffic should be forwarded at a
       constant rate.

 
- How is "constant rate" defined?
  E.g. the maximum deviation SHOULD be less than 10%, as required at "d."?
- Why "should" is not capitalized here?

------------------

page 28

   The test equipment SHOULD start to measure and record all specified
   KPIs and the frequency of measurements SHOULD be less than 2 seconds.
   Continue the test until all traffic profile phases are completed.


The same comment, as for the "frequency of measurements" on page 25.

------------------

page 30

   b.  Traffic should be forwarded constantly.

The same comments as regarding the "constant rate" on page 27.
From now on, I will not report it again...

------------------

pages 28-32

Unlike in other cases, HTTP version was not mentioned at all in Section 7.3.
However, I think it is as important here, as in other cases.

------------------

page 31

   [...] The test
   equipment SHOULD start to measure and record all specified KPIs and
   the frequency of measurements SHOULD be less than 2 seconds.


The same comment, as for the "frequency of measurements" on page 25.

From now on, I will not report it again...

------------------

page 32

   For consistency both the single and multiple transaction test MUST be
   configured with HTTP 1.1.


As I mentioned before, I recommend the inclusion of HTTP/2.
HTTP 1.1 is mentioned again it the next two paragraphs and later many times. I will not report it again...


------------------------------------------------------


Bugs:
-----

page 30

                          Table 4: Mixed Objects


There is a "Table 4" on page 11.
(Should be renumbered. Please take care for their citations in the text, too!)

------------------

page 31

   The measured KPIs during the sustain phase MUST meet the test results
   validation criteria "a" defined in Section 7.3.3.3.

Why only "a" counts?
There are "b" and "c" too.

------------------

page 34

   results validation criteria a, b, c, d, e and f defined in
   Section 7.4.3.3.


Criterion "f" is undefined, as the last one is "e".

------------------

page 37

   During the sustain phase, the DUT/SUT SHOULD reach the "Initial
   concurrent TCP connections".  The measured KPIs during the sustain
   phase MUST meet the test results validation criteria "a" and "b"
   defined in Section 7.5.3.3.


Why only "a" and "b" counts?
There is "c" too.

------------------

page 44

   The measured KPIs during the sustain phase MUST meet the test results
   validation criteria "a" defined in Section 7.7.3.3.

Why only "a" counts?
There are "b" and "c" too.

------------------

page 47

   results validation criteria a, b, c, d, e and f defined in
   Section 7.8.3.3.


Criterion "f" is undefined, as the last one is "e".

------------------

page 50

   During the sustain phase, the DUT/SUT SHOULD reach the "Initial
   concurrent TCP connections".  The measured KPIs during the sustain
   phase MUST meet the test results validation criteria "a" and "b"
   defined in Section 7.9.3.3.


Why only "a" and "b" counts?
There is "c" too.


------------------------------------------------------


Nits:
-----

page 27

   "initial connections per second" as defined in the parameters
-->
   "Initial connections per second" as defined in the parameters

------------------

page 41


   iteration can be interrupted and the result for 64 KByte SHOULD NOT
   be reported).
-->
   iteration MAY be interrupted and the result for 64 KByte SHOULD NOT
   be reported).

(It is so in Section 7.2.4.2, thus it would be consistent to do so here, too)

------------------

Page 43:

Table 5 seems to have the same content as Table 4.
If it is so, it is worth including it twice?

------------------

page 44

   "initial inspected throughput" as defined in the parameters
-->
   "Initial inspected throughput" as defined in the parameters

------------------

page 53

   mode and all detected attack traffic MUST be dropped and the session
   Should be reset
-->
   mode and all detected attack traffic MUST be dropped and the session
   SHOULD be reset.

(perhaps)

------------------

   This document attempts to classify the DUT/SUT in four different four
   different categories based on its maximum supported firewall
-->
   This document attempts to classify the DUT/SUT in four different
   categories based on its maximum supported firewall


("four different" was repeated)

------------------

page 56

   Extra Small (XS) - supported throughput less than 1Gbit/s

   Small (S) - supported throughput less than 5Gbit/s

   Medium (M) - supported throughput greater than 5Gbit/s and less than
   10Gbit/s

   Large (L) - supported throughput greater than 10Gbit/s

Mathematicians could ask: and what if the supported throughput is exactly 10Gbit/s? ;-)

That's all.

Best regards,

Gábor


On 5/10/2021 7:34 PM, MORTON JR., AL wrote:

of course Brian and Gábor, I can use the extra time myself.

 

 

BMWG,

We will extend the deadline for WGLC to May 21, as requested.

Al

bmwg co-chair



_______________________________________________
bmwg mailing list
[hidden email]
https://www.ietf.org/mailman/listinfo/bmwg
Reply | Threaded
Open this post in threaded view
|

Re: Second part -- Re: FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

Brian Monkman

Hi Gabor,

 

Apologies for the tardy reply. All of the comments/suggestions/nits you submitted look fine, with one exception. In your original e-mail you said:

 

“page 16

   [...]  The behavior of the
   client is to sweep through the given server IP space, sequentially
   generating a recognizable service by the DUT. 

“I am not sure if sequential sweep is the right choice for two reasons:
- I feel that the spirit of RFC 4814 would rather suggest pseudorandom. To surely include all of them, I recommend random permutation.
- According to my experience, iptables has shown significantly different maximum connection establishment rate, when I used pseudorandom port numbers or when I enumerated the port numbers in increasing order. (Details available upon request.)”

 

We feel that in order to meet the goal of reproducible results that introducing randomness is not something we want to do.  While we do understand your position, we hope you can support our desire to no make this change.

 

Brian

 

From: Gabor LENCSE <[hidden email]>
Sent: Friday, May 14, 2021 12:56 PM
To: MORTON JR., AL <[hidden email]>; [hidden email]
Cc: 'Bala Balarajah' <[hidden email]>; 'Bala Balarajah' <[hidden email]>; [hidden email]; 'MORTON, ALFRED C (AL)' <[hidden email]>; 'Sarah Banks' <[hidden email]>; Gábor LENCSE <[hidden email]>
Subject: Second part -- Re: [bmwg] FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

 

Dear Al, Authors and BMWG Members,

Al, thank you very much for extending the deadline. However, today I managed to finish the review of the draft.

I think this draft is matured enough. I could find only minor formal issues in the text about the benchmarking procedures. (Although I kept the categories of Discuss/Bugs/Nits, I did not find any significant new issue. My entries in the "Discuss" category are either reiterations from my previous review, or rather editorial ones.)

Discuss:
--------

page 25
7.1.4.2.

   [...] The test equipment SHOULD start to measure and record
   all specified KPIs and the frequency of measurements SHOULD be less
   than 2 seconds.


What does the "frequency of measurements" mean?
I would expect that the KPIs are mausered continously.
Perhaps it is the "frequency of results produced", that is, the "time window for collecting the results and calculating their average" should be less than 2 seconds?

------------------

page 26

   The client SHOULD negotiate HTTP 1.1 and close the connection with
   FIN immediately after completion of one transaction.  In each test
   iteration, client MUST send GET command requesting a fixed HTTP
   response object size.

As I mentioned before (I mean: in my previous review), I recommend the inclusion of HTTP/2.

------------------

page 27

   c.  During the sustain phase, traffic should be forwarded at a
       constant rate.

 
- How is "constant rate" defined?
  E.g. the maximum deviation SHOULD be less than 10%, as required at "d."?
- Why "should" is not capitalized here?

------------------

page 28

   The test equipment SHOULD start to measure and record all specified
   KPIs and the frequency of measurements SHOULD be less than 2 seconds.
   Continue the test until all traffic profile phases are completed.


The same comment, as for the "frequency of measurements" on page 25.

------------------

page 30

   b.  Traffic should be forwarded constantly.

The same comments as regarding the "constant rate" on page 27.
From now on, I will not report it again...

------------------

pages 28-32

Unlike in other cases, HTTP version was not mentioned at all in Section 7.3.
However, I think it is as important here, as in other cases.

------------------

page 31

   [...] The test
   equipment SHOULD start to measure and record all specified KPIs and
   the frequency of measurements SHOULD be less than 2 seconds.


The same comment, as for the "frequency of measurements" on page 25.

From now on, I will not report it again...

------------------

page 32

   For consistency both the single and multiple transaction test MUST be
   configured with HTTP 1.1.


As I mentioned before, I recommend the inclusion of HTTP/2.
HTTP 1.1 is mentioned again it the next two paragraphs and later many times. I will not report it again...


------------------------------------------------------


Bugs:
-----

page 30

                          Table 4: Mixed Objects


There is a "Table 4" on page 11.
(Should be renumbered. Please take care for their citations in the text, too!)

------------------

page 31

   The measured KPIs during the sustain phase MUST meet the test results
   validation criteria "a" defined in Section 7.3.3.3.

Why only "a" counts?
There are "b" and "c" too.

------------------

page 34

   results validation criteria a, b, c, d, e and f defined in
   Section 7.4.3.3.


Criterion "f" is undefined, as the last one is "e".

------------------

page 37

   During the sustain phase, the DUT/SUT SHOULD reach the "Initial
   concurrent TCP connections".  The measured KPIs during the sustain
   phase MUST meet the test results validation criteria "a" and "b"
   defined in Section 7.5.3.3.


Why only "a" and "b" counts?
There is "c" too.

------------------

page 44

   The measured KPIs during the sustain phase MUST meet the test results
   validation criteria "a" defined in Section 7.7.3.3.

Why only "a" counts?
There are "b" and "c" too.

------------------

page 47

   results validation criteria a, b, c, d, e and f defined in
   Section 7.8.3.3.


Criterion "f" is undefined, as the last one is "e".

------------------

page 50

   During the sustain phase, the DUT/SUT SHOULD reach the "Initial
   concurrent TCP connections".  The measured KPIs during the sustain
   phase MUST meet the test results validation criteria "a" and "b"
   defined in Section 7.9.3.3.


Why only "a" and "b" counts?
There is "c" too.


------------------------------------------------------


Nits:
-----

page 27

   "initial connections per second" as defined in the parameters
-->
   "Initial connections per second" as defined in the parameters

------------------

page 41


   iteration can be interrupted and the result for 64 KByte SHOULD NOT
   be reported).
-->
   iteration MAY be interrupted and the result for 64 KByte SHOULD NOT
   be reported).

(It is so in Section 7.2.4.2, thus it would be consistent to do so here, too)

------------------

Page 43:

Table 5 seems to have the same content as Table 4.
If it is so, it is worth including it twice?

------------------

page 44

   "initial inspected throughput" as defined in the parameters
-->
   "Initial inspected throughput" as defined in the parameters

------------------

page 53

   mode and all detected attack traffic MUST be dropped and the session
   Should be reset
-->
   mode and all detected attack traffic MUST be dropped and the session
   SHOULD be reset.

(perhaps)

------------------

   This document attempts to classify the DUT/SUT in four different four
   different categories based on its maximum supported firewall
-->
   This document attempts to classify the DUT/SUT in four different
   categories based on its maximum supported firewall


("four different" was repeated)

------------------

page 56

   Extra Small (XS) - supported throughput less than 1Gbit/s

   Small (S) - supported throughput less than 5Gbit/s

   Medium (M) - supported throughput greater than 5Gbit/s and less than
   10Gbit/s

   Large (L) - supported throughput greater than 10Gbit/s

Mathematicians could ask: and what if the supported throughput is exactly 10Gbit/s? ;-)

That's all.

Best regards,

Gábor

 

On 5/10/2021 7:34 PM, MORTON JR., AL wrote:

of course Brian and Gábor, I can use the extra time myself.

 

 

BMWG,

We will extend the deadline for WGLC to May 21, as requested.

Al

bmwg co-chair

 


_______________________________________________
bmwg mailing list
[hidden email]
https://www.ietf.org/mailman/listinfo/bmwg
Reply | Threaded
Open this post in threaded view
|

Sequential vs. random -- Re: FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

Gábor LENCSE
Dear Brian,

I think that you know firewall testing better then me, thus you can make a better decision than me, and I will accept your decision anything it will be.

However, I would like to support you decision by sharing my experience and results with you. Full disclosure: I find the question of sequential vs. random very interesting and important regarding my own research. I hope that our discussion may be beneficial for both of us.


STATELESS TESTING

1. DNS64 and authoritative DNS server benchmarking

I used sequential source port numbers in dns64perf++ (implemented by my BSc student Dániel Bakai) only to support RSS (Receive-Side Scaling). It made me possible to perform authoritative DNS server performance measurements over 3 million queries per second rate. (The details were published in my open access paper: G. Lencse, "Benchmarking Authoritative DNS Servers", IEEE Access, vol. 8. pp. 130224-130238, July 2020. DOI: 10.1109/ACCESS.2020.3009141)

2. SIIT (also called stateless NAT64) benchmarking

In the original version of my DPDK-based RFC 8219 compliant SIIT tester called siitperf, I implemented the possibility of using random destination networks to comply with the requirement of RFC 2544. But I used fixed source and destination port numbers by literally following the Test Frame Format supplied in the appendix of RFC 2544. Then Al Morton has called my attention to RFC 4814. Then I added the feature of pseudorandom port numbers to comply with RFC 4814. Besides pseudorandom port numbers, I also added the possibility of increasing (or decreasing) port numbers. (I have seen this feature also in commercial RFC 2544 testers.)

I have very interesting measurement results with using pseudorandom vs. sequential port numbers in IPv4 router benchmarking. (IPv4 routers were chosen to be able to achieve high throughput.) Please see Table I. in my open access paper: G. Lencse, "Adding RFC 4814 Random Port Feature to Siitperf: Design, Implementation and Performance Estimation", International Journal of Advances in Telecommunications, Electrotechnics, Signals and Systems, vol 9, no 3, pp. 18-26, 2020, DOI: 10.11601/ijates.v9i3.291 available also from: http://www.hit.bme.hu/~lencse/publications/291-1113-1-PB.pdf
If you consider the median throughput rates using:
- random source ports (and fixed destination port): 3,469,891 fps
- increasing source ports (and fixed destination port): 3,485,627 fps
- random destination ports (and fixed source port): 3,400,966 fps
- increasing destination ports (and fixed source port): 3,447,165 fps
Then you can see that there is not much difference, but increasing port numbers result in somewhat higher throughput than random ones. (It is also interesting that varying source port numbers cause higher throughput than varying destination port numbers. I attribute this difference to an asymmetry in the hash function: varying source port numbers can distribute the interrupts more evenly among the CPU cores.)

The fact that all the above result were close to the 3,402,271 fps throughput measured using pseudorandom source and destination port numbers to comply with RFC 4814, is very important for me: using sequential source (or destination, but not both) port numbers can be a computationally cheaper alternative to using pseudorandom source and destination port numbers.


STATEFUL TESTING

This is my very recent research area, and my surprising results made me recommending you to consider pseudorandom port numbers.

When I designed the stateful extension of siitperf, my original idea was to use sequential sweep in the port number range, which I called as "port number enumeration", in the preliminary phase of the stateful testing to load the (source IP, source port, destination IP, destination port) four tuples into both the connection tracking table of the DUT and the state table of the "Responder" subsystem of the Tester.
However, my test results showed huge difference in the maximum connection establishment rate of iptables!
Parameters of the tests:
source port range: 10,000 - 49,999
destination port range: 80 - 179
(Number of all possible combinations: 40,000*100=4,000,000)
nf_conntrack_max: 4,194,304
hashsize: 524,288
Number of preliminary frames: 4,000,000
Maximum connection establishment rate was determined by binary search. The results were shocking:
- with enumerated port numbers: 669,587 fps
- with random port numbers: 1,406,320

I was aware that random port numbers did not fully exhaust the possible port number combinations (some combinations appeared twice or even more times). Therefore, I have repeated the measurement with 40,000,000 preliminary frames using random port numbers. The result was 1,271,023fps, which still about the double of the result with enumerated port numbers.

You can find all the details in my paper currently under review: http://www.hit.bme.hu/~lencse/publications/SFNAT64-tester-for-review.pdf

Regarding the possible root cause of the phenomenon, I have my own explanation, but I do not want influence you.

What do you think of it?

This experience was the reason, why we wrote in our I-D that:
   Important warning: in normal router testing, the port number
   selection algorithm (whether it is pseudo-random or enumerated) does
   not affect final results.  However, our experience with iptables
   shows that if the connection tracking table is filled using port
   number enumeration in increasing order, then the maximum connection
   establishment rate of iptables degrades significantly compared to its
   performance using pseudorandom port numbers [LEN2021].
Source: https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-benchmarking-stateful


APPLICATION TO FIREWALL TESTING

Of course, I am aware that your case may be different in several aspects than yours.

For example, I willfully increased the size of the connection tracking table of iptables from its default value to a value recommended for a high-loaded NAT server: https://ixnfo.com/en/tuning-nf_conntrack.html
In addition to that, I have nearly fully filled the connection tracking table of the DUT!

These parameters likely do not apply to firewalls preforming deep packet inspection. :-)

My aim was to test the abilities of my stateful Tester in a demanding situation. I used NAT44 instead of NAT64 to be able to achieve higher rates. NAT44 is much-much simpler than the tasks performed by your firewalls, thus the lookup of the connections may dominate the execution time in my case, but it may be marginal in your case.

And there may be several other differences, which I do not even think of. This is why wrote that you need to decide, what is suitable for you. The most I can do is to give ideas to consider. :-)

Best regards,

Gábor



18/05/2021 22:28 keltezéssel, [hidden email] írta:

Hi Gabor,

 

Apologies for the tardy reply. All of the comments/suggestions/nits you submitted look fine, with one exception. In your original e-mail you said:

 

“page 16

   [...]  The behavior of the
   client is to sweep through the given server IP space, sequentially
   generating a recognizable service by the DUT. 

“I am not sure if sequential sweep is the right choice for two reasons:
- I feel that the spirit of RFC 4814 would rather suggest pseudorandom. To surely include all of them, I recommend random permutation.
- According to my experience, iptables has shown significantly different maximum connection establishment rate, when I used pseudorandom port numbers or when I enumerated the port numbers in increasing order. (Details available upon request.)”

 

We feel that in order to meet the goal of reproducible results that introducing randomness is not something we want to do.  While we do understand your position, we hope you can support our desire to no make this change.

 

Brian



_______________________________________________
bmwg mailing list
[hidden email]
https://www.ietf.org/mailman/listinfo/bmwg
Reply | Threaded
Open this post in threaded view
|

Re: FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

MORTON JR., AL
In reply to this post by MORTON JR., AL

Thanks to the authors/editor for continuing to chase-down comments and resolving them.

I know very well how much work this is.

 

I’m trying to finish-up my review tonight (as a participant):

 

   o  Inspected Throughput

      The number of bits per second of allowed traffic a network

      security device is able to transmit to the correct destination

      interface(s) in response to a specified offered load.  The

      throughput benchmarking tests defined in Section 7 SHOULD measure

      the average OSI model Layer 2 throughput value.  This document

      recommends presenting the throughput value in Gbit/s rounded to

      two places of precision with a more specific Kbit/s in

      parenthesis.

 

Let me suggest a couple of tweaks on this definition:

 

It took me several minutes to remember that “inspected” invokes the role of security device, for some reason.

 

The OSI model doesn’t have a good reputation here at IETF, and the Internet was not built based on ISO’s model.

 

So I’ll try a little editing:

 

   o  Inspected Throughput

      The number of bits per second of examined and allowed traffic a network

      security device is able to transmit to the correct destination

      interface(s) in response to a specified offered load.  The

      throughput benchmarking tests defined in Section 7 SHOULD measure

      the average Layer 2 throughput value when the DUT is “inspecting” traffic.  This document

      recommends presenting the inspected throughput value in Gbit/s rounded to

      two places of precision with a more specific Kbit/s in

      parenthesis.

 

I also checked the nits, which revealed use of 2 incorrect addresses as examples.

BMWG has it’s own IPv4 and v6 address space assigned. Please use it where appropriate.

I’m not sure where the offending addresses are, sorry!

 

Checking nits according to https://www.ietf.org/id-info/checklist :
  ----------------------------------------------------------------------------
 
  == There are 1 instance of lines with non-RFC6890-compliant IPv4 addresses
     in the document.  If these are example addresses, they should be changed.
 
  == There are 1 instance of lines with non-RFC3849-compliant IPv6 addresses
     in the document.  If these are example addresses, they should be changed.

 

The tables below are from RFC 6890:

                    +----------------------+---------------+
                    | Attribute            | Value         |
                    +----------------------+---------------+
                    | Address Block        | 198.18.0.0/15 |
                    | Name                 | Benchmarking  |
                    | RFC                  | [RFC2544]     |
                    | Allocation Date      | March 1999    |
                    | Termination Date     | N/A           |
                    | Source               | True          |
                    | Destination          | True          |
                    | Forwardable          | True          |
                    | Global               | False         |
                    | Reserved-by-Protocol | False         |
                    +----------------------+---------------+
 
          Table 12: Network Interconnect Device Benchmark Testing

 

and

 

 
                    +----------------------+----------------+
                    | Attribute            | Value          |
                    +----------------------+----------------+
                    | Address Block        | 2001:2::/48    |
                    | Name                 | Benchmarking   |
                    | RFC                  | [RFC5180]      |
                    | Allocation Date      | April 2008     |
                    | Termination Date     | N/A            |
                    | Source               | True           |
                    | Destination          | True           |
                    | Forwardable          | True           |
                    | Global               | False          |
                    | Reserved-by-Protocol | False          |
                    +----------------------+----------------+
 
                          Table 24: Benchmarking

 

 

Also:

 

  Miscellaneous warnings:
  ----------------------------------------------------------------------------
 
  == The document seems to lack the recommended RFC 2119 boilerplate, even if
     it appears to use RFC 2119 keywords. 
 
     (The document does seem to have the reference to RFC 2119 which the
     ID-Checklist requires).
 

The section seems to exist, but may need slight re-wording,

to match RFC 8174, or this version of the nits-checker is out of date

(it doesn’t mention rfc 8174?):

 

2.  Requirements
 
   The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119], [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

 

thanks,

Al

 

 

From: bmwg <[hidden email]> On Behalf Of MORTON JR., AL
Sent: Monday, May 10, 2021 1:34 PM
To: [hidden email]; 'Gabor LENCSE' <[hidden email]>
Cc: 'MORTON, ALFRED C (AL)' <[hidden email]>; [hidden email]; 'Bala Balarajah' <[hidden email]>; 'Bala Balarajah' <[hidden email]>
Subject: Re: [bmwg] FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

 

***Security Advisory: This Message Originated Outside of AT&T ***
Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.

of course Brian and Gábor, I can use the extra time myself.

 

 

BMWG,

We will extend the deadline for WGLC to May 21, as requested.

Al

bmwg co-chair

 

From: [hidden email] <[hidden email]>
Sent: Monday, May 10, 2021 1:02 PM
To: 'Gabor LENCSE' <[hidden email]>
Cc: 'Bala Balarajah' <[hidden email]>; 'Bala Balarajah' <[hidden email]>; [hidden email]; 'MORTON, ALFRED C (AL)' <[hidden email]>; 'Sarah Banks' <[hidden email]>
Subject: RE: [bmwg] FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

 

Thanks Gabor.

 

Al, could you extend the deadline for WGLC to May 21st from May 17th?

 

Brian

 

From: Gabor LENCSE <[hidden email]>
Sent: Monday, May 10, 2021 12:58 PM
To: [hidden email]
Cc: 'Bala Balarajah' <[hidden email]>; 'Bala Balarajah' <[hidden email]>; [hidden email]; 'MORTON, ALFRED C (AL)' <[hidden email]>; 'Sarah Banks' <[hidden email]>
Subject: Re: [bmwg] FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

 

Hi Brian,

Perhaps I can do another chunk later this week, and I plan to complete it next week.

Best regards,

Gábor

On 5/10/2021 6:45 PM, [hidden email] wrote:

Hi Gabor,

 

Thank you very much for your comments. On initial review they all look reasonable to us. How long do you think it will take you to complete your review of the rest of the document?

 

Brian

 


_______________________________________________
bmwg mailing list
[hidden email]
https://www.ietf.org/mailman/listinfo/bmwg
Reply | Threaded
Open this post in threaded view
|

Re: FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

Brian Monkman

Hi Al,

 

Thanks for sending this over. Our comments are embedded below, prefaced by [bpm].

 

Brian

 

From: MORTON JR., AL <[hidden email]>
Sent: Wednesday, May 19, 2021 10:01 PM
To: [hidden email]; 'Gabor LENCSE' <[hidden email]>
Cc: 'MORTON, ALFRED C (AL)' <[hidden email]>; [hidden email]; 'Bala Balarajah' <[hidden email]>; 'Bala Balarajah' <[hidden email]>
Subject: RE: [bmwg] FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

 

Thanks to the authors/editor for continuing to chase-down comments and resolving them.

I know very well how much work this is.

 

I’m trying to finish-up my review tonight (as a participant):

 

   o  Inspected Throughput

      The number of bits per second of allowed traffic a network

      security device is able to transmit to the correct destination

      interface(s) in response to a specified offered load.  The

      throughput benchmarking tests defined in Section 7 SHOULD measure

      the average OSI model Layer 2 throughput value.  This document

      recommends presenting the throughput value in Gbit/s rounded to

      two places of precision with a more specific Kbit/s in

      parenthesis.

 

Let me suggest a couple of tweaks on this definition:

 

It took me several minutes to remember that “inspected” invokes the role of security device, for some reason.

 

The OSI model doesn’t have a good reputation here at IETF, and the Internet was not built based on ISO’s model.

 

So I’ll try a little editing:

 

   o  Inspected Throughput

      The number of bits per second of examined and allowed traffic a network

      security device is able to transmit to the correct destination

      interface(s) in response to a specified offered load.  The

      throughput benchmarking tests defined in Section 7 SHOULD measure

      the average Layer 2 throughput value when the DUT is “inspecting” traffic.  This document

      recommends presenting the inspected throughput value in Gbit/s rounded to

      two places of precision with a more specific Kbit/s in

      parenthesis.

 

[bpm] Good suggestion. We will change the wording.

 

I also checked the nits, which revealed use of 2 incorrect addresses as examples.

BMWG has it’s own IPv4 and v6 address space assigned. Please use it where appropriate.

I’m not sure where the offending addresses are, sorry!

 

Checking nits according to https://www.ietf.org/id-info/checklist :
  ----------------------------------------------------------------------------
 
  == There are 1 instance of lines with non-RFC6890-compliant IPv4 addresses
     in the document.  If these are example addresses, they should be changed.
 
  == There are 1 instance of lines with non-RFC3849-compliant IPv6 addresses
     in the document.  If these are example addresses, they should be changed.

 

The tables below are from RFC 6890:

                    +----------------------+---------------+
                    | Attribute            | Value         |
                    +----------------------+---------------+
                    | Address Block        | 198.18.0.0/15 |
                    | Name                 | Benchmarking  |
                    | RFC                  | [RFC2544]     |
                    | Allocation Date      | March 1999    |
                    | Termination Date     | N/A           |
                    | Source               | True          |
                    | Destination          | True          |
                    | Forwardable          | True          |
                    | Global               | False         |
                    | Reserved-by-Protocol | False         |
                    +----------------------+---------------+
 
          Table 12: Network Interconnect Device Benchmark Testing

 

and

 

 
                    +----------------------+----------------+
                    | Attribute            | Value          |
                    +----------------------+----------------+
                    | Address Block        | 2001:2::/48    |
                    | Name                 | Benchmarking   |
                    | RFC                  | [RFC5180]      |
                    | Allocation Date      | April 2008     |
                    | Termination Date     | N/A            |
                    | Source               | True           |
                    | Destination          | True           |
                    | Forwardable          | True           |
                    | Global               | False          |
                    | Reserved-by-Protocol | False          |
                    +----------------------+----------------+
 
                          Table 24: Benchmarking

 

[bpm] We have looked through the document multiple times and in multiple ways and cannot find anything that might trigger this. However, we did catch a format error in the IPv6 address in section 8 – IANA considerations. So, we are updating that section.

 

 

Also:

 

  Miscellaneous warnings:
  ----------------------------------------------------------------------------
 
  == The document seems to lack the recommended RFC 2119 boilerplate, even if
     it appears to use RFC 2119 keywords. 
 
     (The document does seem to have the reference to RFC 2119 which the
     ID-Checklist requires).
 

The section seems to exist, but may need slight re-wording,

to match RFC 8174, or this version of the nits-checker is out of date

(it doesn’t mention rfc 8174?):

 

2.  Requirements
 
   The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119], [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

 

[bpm] I went and looked at the wording RFC 8174 and it is identical to the wording above.

 

 

thanks,

Al

 

 

From: bmwg <[hidden email]> On Behalf Of MORTON JR., AL
Sent: Monday, May 10, 2021 1:34 PM
To: [hidden email]; 'Gabor LENCSE' <[hidden email]>
Cc: 'MORTON, ALFRED C (AL)' <[hidden email]>; [hidden email]; 'Bala Balarajah' <[hidden email]>; 'Bala Balarajah' <[hidden email]>
Subject: Re: [bmwg] FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

 

***Security Advisory: This Message Originated Outside of AT&T ***
Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.

of course Brian and Gábor, I can use the extra time myself.

 

 

BMWG,

We will extend the deadline for WGLC to May 21, as requested.

Al

bmwg co-chair

 

From: [hidden email] <[hidden email]>
Sent: Monday, May 10, 2021 1:02 PM
To: 'Gabor LENCSE' <[hidden email]>
Cc: 'Bala Balarajah' <[hidden email]>; 'Bala Balarajah' <[hidden email]>; [hidden email]; 'MORTON, ALFRED C (AL)' <[hidden email]>; 'Sarah Banks' <[hidden email]>
Subject: RE: [bmwg] FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

 

Thanks Gabor.

 

Al, could you extend the deadline for WGLC to May 21st from May 17th?

 

Brian

 

From: Gabor LENCSE <[hidden email]>
Sent: Monday, May 10, 2021 12:58 PM
To: [hidden email]
Cc: 'Bala Balarajah' <[hidden email]>; 'Bala Balarajah' <[hidden email]>; [hidden email]; 'MORTON, ALFRED C (AL)' <[hidden email]>; 'Sarah Banks' <[hidden email]>
Subject: Re: [bmwg] FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

 

Hi Brian,

Perhaps I can do another chunk later this week, and I plan to complete it next week.

Best regards,

Gábor

On 5/10/2021 6:45 PM, [hidden email] wrote:

Hi Gabor,

 

Thank you very much for your comments. On initial review they all look reasonable to us. How long do you think it will take you to complete your review of the rest of the document?

 

Brian

 


_______________________________________________
bmwg mailing list
[hidden email]
https://www.ietf.org/mailman/listinfo/bmwg
Reply | Threaded
Open this post in threaded view
|

Re: Sequential vs. random -- Re: FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

Brian Monkman
In reply to this post by Gábor LENCSE

HI Gabor,

 

We will update page 16 of the draft to allow for either sequential or pseudorandom port numbers.

 

Thanks for your detailed explanation.

 

Brian

 

From: Gábor LENCSE <[hidden email]>
Sent: Wednesday, May 19, 2021 3:49 AM
To: [hidden email]; 'MORTON JR., AL' <[hidden email]>
Cc: 'Bala Balarajah' <[hidden email]>; 'Bala Balarajah' <[hidden email]>; [hidden email]; 'MORTON, ALFRED C (AL)' <[hidden email]>; 'Sarah Banks' <[hidden email]>; Dr. Lencse Gábor <[hidden email]>
Subject: Sequential vs. random -- Re: [bmwg] FW: WGLC on New version of draft-ietf-bmwg-ngfw-performance

 

Dear Brian,

I think that you know firewall testing better then me, thus you can make a better decision than me, and I will accept your decision anything it will be.

However, I would like to support you decision by sharing my experience and results with you. Full disclosure: I find the question of sequential vs. random very interesting and important regarding my own research. I hope that our discussion may be beneficial for both of us.


STATELESS TESTING

1. DNS64 and authoritative DNS server benchmarking

I used sequential source port numbers in dns64perf++ (implemented by my BSc student Dániel Bakai) only to support RSS (Receive-Side Scaling). It made me possible to perform authoritative DNS server performance measurements over 3 million queries per second rate. (The details were published in my open access paper: G. Lencse, "Benchmarking Authoritative DNS Servers", IEEE Access, vol. 8. pp. 130224-130238, July 2020. DOI: 10.1109/ACCESS.2020.3009141)

2. SIIT (also called stateless NAT64) benchmarking

In the original version of my DPDK-based RFC 8219 compliant SIIT tester called siitperf, I implemented the possibility of using random destination networks to comply with the requirement of RFC 2544. But I used fixed source and destination port numbers by literally following the Test Frame Format supplied in the appendix of RFC 2544. Then Al Morton has called my attention to RFC 4814. Then I added the feature of pseudorandom port numbers to comply with RFC 4814. Besides pseudorandom port numbers, I also added the possibility of increasing (or decreasing) port numbers. (I have seen this feature also in commercial RFC 2544 testers.)

I have very interesting measurement results with using pseudorandom vs. sequential port numbers in IPv4 router benchmarking. (IPv4 routers were chosen to be able to achieve high throughput.) Please see Table I. in my open access paper: G. Lencse, "Adding RFC 4814 Random Port Feature to Siitperf: Design, Implementation and Performance Estimation", International Journal of Advances in Telecommunications, Electrotechnics, Signals and Systems, vol 9, no 3, pp. 18-26, 2020, DOI: 10.11601/ijates.v9i3.291 available also from: http://www.hit.bme.hu/~lencse/publications/291-1113-1-PB.pdf
If you consider the median throughput rates using:
- random source ports (and fixed destination port): 3,469,891 fps
- increasing source ports (and fixed destination port): 3,485,627 fps
- random destination ports (and fixed source port): 3,400,966 fps
- increasing destination ports (and fixed source port): 3,447,165 fps
Then you can see that there is not much difference, but increasing port numbers result in somewhat higher throughput than random ones. (It is also interesting that varying source port numbers cause higher throughput than varying destination port numbers. I attribute this difference to an asymmetry in the hash function: varying source port numbers can distribute the interrupts more evenly among the CPU cores.)

The fact that all the above result were close to the 3,402,271 fps throughput measured using pseudorandom source and destination port numbers to comply with RFC 4814, is very important for me: using sequential source (or destination, but not both) port numbers can be a computationally cheaper alternative to using pseudorandom source and destination port numbers.


STATEFUL TESTING

This is my very recent research area, and my surprising results made me recommending you to consider pseudorandom port numbers.

When I designed the stateful extension of siitperf, my original idea was to use sequential sweep in the port number range, which I called as "port number enumeration", in the preliminary phase of the stateful testing to load the (source IP, source port, destination IP, destination port) four tuples into both the connection tracking table of the DUT and the state table of the "Responder" subsystem of the Tester.
However, my test results showed huge difference in the maximum connection establishment rate of iptables!
Parameters of the tests:
source port range: 10,000 - 49,999
destination port range: 80 - 179
(Number of all possible combinations: 40,000*100=4,000,000)
nf_conntrack_max: 4,194,304
hashsize: 524,288
Number of preliminary frames: 4,000,000
Maximum connection establishment rate was determined by binary search. The results were shocking:
- with enumerated port numbers: 669,587 fps
- with random port numbers: 1,406,320

I was aware that random port numbers did not fully exhaust the possible port number combinations (some combinations appeared twice or even more times). Therefore, I have repeated the measurement with 40,000,000 preliminary frames using random port numbers. The result was 1,271,023fps, which still about the double of the result with enumerated port numbers.

You can find all the details in my paper currently under review: http://www.hit.bme.hu/~lencse/publications/SFNAT64-tester-for-review.pdf

Regarding the possible root cause of the phenomenon, I have my own explanation, but I do not want influence you.

What do you think of it?

This experience was the reason, why we wrote in our I-D that:

   Important warning: in normal router testing, the port number
   selection algorithm (whether it is pseudo-random or enumerated) does
   not affect final results.  However, our experience with iptables
   shows that if the connection tracking table is filled using port
   number enumeration in increasing order, then the maximum connection
   establishment rate of iptables degrades significantly compared to its
   performance using pseudorandom port numbers [LEN2021].

Source: https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-benchmarking-stateful


APPLICATION TO FIREWALL TESTING

Of course, I am aware that your case may be different in several aspects than yours.

For example, I willfully increased the size of the connection tracking table of iptables from its default value to a value recommended for a high-loaded NAT server: https://ixnfo.com/en/tuning-nf_conntrack.html
In addition to that, I have nearly fully filled the connection tracking table of the DUT!

These parameters likely do not apply to firewalls preforming deep packet inspection. :-)

My aim was to test the abilities of my stateful Tester in a demanding situation. I used NAT44 instead of NAT64 to be able to achieve higher rates. NAT44 is much-much simpler than the tasks performed by your firewalls, thus the lookup of the connections may dominate the execution time in my case, but it may be marginal in your case.

And there may be several other differences, which I do not even think of. This is why wrote that you need to decide, what is suitable for you. The most I can do is to give ideas to consider. :-)

Best regards,

Gábor


18/05/2021 22:28 keltezéssel, [hidden email] írta:

Hi Gabor,

 

Apologies for the tardy reply. All of the comments/suggestions/nits you submitted look fine, with one exception. In your original e-mail you said:

 

“page 16

   [...]  The behavior of the
   client is to sweep through the given server IP space, sequentially
   generating a recognizable service by the DUT. 

“I am not sure if sequential sweep is the right choice for two reasons:
- I feel that the spirit of RFC 4814 would rather suggest pseudorandom. To surely include all of them, I recommend random permutation.
- According to my experience, iptables has shown significantly different maximum connection establishment rate, when I used pseudorandom port numbers or when I enumerated the port numbers in increasing order. (Details available upon request.)”

 

We feel that in order to meet the goal of reproducible results that introducing randomness is not something we want to do.  While we do understand your position, we hope you can support our desire to no make this change.

 

Brian

 


_______________________________________________
bmwg mailing list
[hidden email]
https://www.ietf.org/mailman/listinfo/bmwg
Reply | Threaded
Open this post in threaded view
|

Re: WGLC on New version of draft-ietf-bmwg-ngfw-performance

Sarah Banks-3
In reply to this post by MORTON JR., AL
Hi,
    Sharing feedback with my participant hat on. I do apologize for sending it so late. A few high level comments:

- I still land in the "I don't agree that this doc clearly covers "all next gen security devices" - and they're not clearly defined, either. For the three that are covered, I think there's a lumping of the IPS and IDS that would be better served separated. From my perspective, an IPS *is* a bump in the wire, and does make "blocking" decisions, as called out in this document. An IDS however, does not - it tells you it happened after the fact, and it's a passive device. This draft is really keyed around "active inline" functions - ie/ a bump in the wire - and I think keeping the focus there makes sense, and lumping the IDS in does more harm than good.

In short, I don't support this draft moving forward with the inclusion of IDS - it doesn't make sense to me. One way forward might be to remove the IDS as a device from scope (as a suggestion).


- I've been testing for a long time, but I find this draft ... uncomfortable to approach. I expected, and would have preferred, something that walks me through a specific set of recommended scenarios with specifics on what to configure for repeatable tests that I could compare results with, but there are a lot of moving parts here and a lot of assertions that have to be made for the test beds as a whole where I think the test results could vary wildly when the same topology is handed to Test Person A and Test Person B.

- Most of the feedback below omits the IDS pieces, because so much of it didn't apply.

Thanks,
Sarah



- The draft aims to replace RFC3511, but expands scope past Firewalls, to "next generation security devices". I'm not finding a definition of what a "next generation security device is", nor an exhaustive list of the devices covered in this draft. A list that includes is nice, but IMO not enough to cover what would be benchmarked here - I'd prefer to see a definition and an exhaustive list.
- What is a NGIPS or NGIDS? If there are standardized definitions pointing to them is fine, otherwise, there's a lot of wiggle room here.
- I still have the concern I shared at the last IETF meeting, where here, we're putting active inline security devices in the same category as passive devices. On one hand, I'm not sure I'd lump these three together in the first place; on the other, active inline devices typically include additional functions to allow administrators to control what happens to packets in the case of failure, and I don't see those test cases included here.
- Section 4.1 - it reads as if ANY device in the test setup cannot contribute to network latency or throughput issues, including the DUTs - is that what you intended?
- Option 1: It'd be nice to see a specific, clean, recommended test bed. There are options for multiple emulated routers. As a tester, I expect to see a specific, proscribed test bed that I should configure and test against.
- Follow on: I'm curious as to the choice of emulated routers here. The previous test suggests you avoid routers and switches in the topo, but then there are emulated ones here. I'm curious as to what advantages you think these bring over the real deal, and, why they aren't subject to the same limitations previously described?
- In section 4.1 the text calls out Option 1 as the preferred test bed, which includes L3 routing, but it's not clear why that's needed?
- The difference between Option 1 and Option 2 is the inclusion of additional physical gear in Option 2 - it's not clear why that's needed, or why the tester can't simply directly connect the test equipment to the DUT and remove extraneous devices from potential influence on results?
- Section 4.2, the table for NGFW features - I'm not sure what the difference is between RECOMMENDED and OPTIONAL? (I realize that you might be saying that RECOMMENDED is the "must have enabled" features, where as optional is at your discretion, but would suggest that you make that clear)
- Proscribing a list of features that have to be enabled for the test, or at least more than 1, feels like a strange choice here - I'd have expected tests cases that either test the specific features one at a time, or suggest several combinations, but that ultimately, we'd tell the tester to document WHICH features were enabled, to make the test cases repeatable? This allows the tester to apply a same set of apples to apples configurations to different vendor gear, and omit the 1 feature that doesn't exist on a different NGFW (for example), but hold a baseline that could be tested.
- Table 2: With the assumption that NGIPS/IDS are required to have the features under "recommended", I disagree with this list. For example, some customers break and inspect at the tap/agg layer of the network - in this case, the feed into the NGIDS might be decrypted, and there's no need to enable SSL inspection, for example.
- Table 3: I disagree that an NGIDS IS REQUIRED to decrypt SSL. This behaviour might be suitable for an NGIPS, but the NGIDS is not a bump on the wire, and often isn't decrypting and re-encrypting the traffic.
- Table 3: An NGIDS IMO is still a passive device - it wouldn't be blocking anything, but agree that it might tell you that it happened after the fact.
- Table 3: Anti-evasion definition - define "mitigates".
- Table 3: Web-filtering - not a function of an NGIDS.
- Table 3: DLP: Not applicable for an NGIDS.
- Can you expand on "disposition of all flows of traffic are logged" - what's meant here specifically, and why do they have to be logged? (Logging, particularly under high loads, will impact it's own performance marks, and colours output)
- ACLs wouldn't apply to an IDS because IDS's aren't blocking traffic :)
- It might be helpful to testers to say something like "look, here's one suggested set of ACLs. If you're using them, great, reference that, but otherwise, make note of the ACLs you use, and use the same ones for repeatable testing".
- 4.3.1.1 The doc proscribes specific MSS values for v4/v6  with no discussion around why they're chosen - that color could be useful to the reader.
- 4.3.1.1 - there's a period on the 3rd to last line "(SYN/ACL, ACK). and" that should be changed.
- 4.3.1.1 - As a tester with long time experience with major test equipment manufacturers, I can't possibly begin to guess which ones of them would conform to this - or even if they'd answer these questions. How helpful is this section to the non test houses? I suggest expansion here, ideally with either covering the scope of what you expect to cover, or hopefully which (open source/generally available) test tools or emulators could be considered for use as examples.
- 4.3.1.3 - Do the emulated web browser attributes really apply to testing the NGIPS?
- 4.3.2.3 - Do you expect to also leverage TLS 1.3 as a configuration option here?
- 4.3.4 - I'm surprised to see the requirement that all sessions establish a distinct phase before moving on to the next. You might clarify why this is a requirement, and why staggering them is specifically rejected?
- 5.1 - I like the sentence, but it leaves a world of possibilities open as to how one confirmed that the ancillary switching or routing functions didn't limit the performance, particularly the virtualized components?
- 5.3 - this is a nice assertion but again, how do I reasonably make the assertion?
- 6.1 - I would suggest that the test report include the configuration of ancillary devices on both client/server side as well
- 6.3 - Nothing on drops anywhere?
- 7.1.3.2 - Where are these numbers coming from? How are you determining the "initial inspected throughput"? Maybe I missed that in the document overall, but it's not clear to me where these KPIs are collected? I suggest this be called out.
- 7.1.3.3 - what is a "relevant application traffic mix" profile?
- 7.1.3.4 - where does this monitoring occur?
- 7.1.3.4 - This looks a bit like conformance testing -  Why does item (b) require a specific number/threshold?
- 9: Why is the cipher squite recommendation for a real deployment outside the scope of this document?





_______________________________________________
bmwg mailing list
[hidden email]
https://www.ietf.org/mailman/listinfo/bmwg
Reply | Threaded
Open this post in threaded view
|

Re: WGLC on New version of draft-ietf-bmwg-ngfw-performance

Brian Monkman
Sarah,

Thanks for your comments. Given when it was received with respect to the
WGLC timeframe your suggestions will be considered for the next draft.
(Which is being posted shortly.)

We will get back to you in due course.

Brian

-----Original Message-----
From: Sarah Banks <[hidden email]>
Sent: Thursday, May 20, 2021 4:35 PM
To: ALFRED MORTON <[hidden email]>
Cc: [hidden email]; Gabor LENCSE <[hidden email]>; Bala
Balarajah <[hidden email]>; Bala Balarajah <[hidden email]>;
[hidden email]; MORTON, ALFRED C (AL) <[hidden email]>
Subject: Re: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance

Hi,
    Sharing feedback with my participant hat on. I do apologize for sending
it so late. A few high level comments:

- I still land in the "I don't agree that this doc clearly covers "all next
gen security devices" - and they're not clearly defined, either. For the
three that are covered, I think there's a lumping of the IPS and IDS that
would be better served separated. From my perspective, an IPS *is* a bump in
the wire, and does make "blocking" decisions, as called out in this
document. An IDS however, does not - it tells you it happened after the
fact, and it's a passive device. This draft is really keyed around "active
inline" functions - ie/ a bump in the wire - and I think keeping the focus
there makes sense, and lumping the IDS in does more harm than good.

In short, I don't support this draft moving forward with the inclusion of
IDS - it doesn't make sense to me. One way forward might be to remove the
IDS as a device from scope (as a suggestion).


- I've been testing for a long time, but I find this draft ... uncomfortable
to approach. I expected, and would have preferred, something that walks me
through a specific set of recommended scenarios with specifics on what to
configure for repeatable tests that I could compare results with, but there
are a lot of moving parts here and a lot of assertions that have to be made
for the test beds as a whole where I think the test results could vary
wildly when the same topology is handed to Test Person A and Test Person B.

- Most of the feedback below omits the IDS pieces, because so much of it
didn't apply.

Thanks,
Sarah



- The draft aims to replace RFC3511, but expands scope past Firewalls, to
"next generation security devices". I'm not finding a definition of what a
"next generation security device is", nor an exhaustive list of the devices
covered in this draft. A list that includes is nice, but IMO not enough to
cover what would be benchmarked here - I'd prefer to see a definition and an
exhaustive list.
- What is a NGIPS or NGIDS? If there are standardized definitions pointing
to them is fine, otherwise, there's a lot of wiggle room here.
- I still have the concern I shared at the last IETF meeting, where here,
we're putting active inline security devices in the same category as passive
devices. On one hand, I'm not sure I'd lump these three together in the
first place; on the other, active inline devices typically include
additional functions to allow administrators to control what happens to
packets in the case of failure, and I don't see those test cases included
here.
- Section 4.1 - it reads as if ANY device in the test setup cannot
contribute to network latency or throughput issues, including the DUTs - is
that what you intended?
- Option 1: It'd be nice to see a specific, clean, recommended test bed.
There are options for multiple emulated routers. As a tester, I expect to
see a specific, proscribed test bed that I should configure and test
against.
- Follow on: I'm curious as to the choice of emulated routers here. The
previous test suggests you avoid routers and switches in the topo, but then
there are emulated ones here. I'm curious as to what advantages you think
these bring over the real deal, and, why they aren't subject to the same
limitations previously described?
- In section 4.1 the text calls out Option 1 as the preferred test bed,
which includes L3 routing, but it's not clear why that's needed?
- The difference between Option 1 and Option 2 is the inclusion of
additional physical gear in Option 2 - it's not clear why that's needed, or
why the tester can't simply directly connect the test equipment to the DUT
and remove extraneous devices from potential influence on results?
- Section 4.2, the table for NGFW features - I'm not sure what the
difference is between RECOMMENDED and OPTIONAL? (I realize that you might be
saying that RECOMMENDED is the "must have enabled" features, where as
optional is at your discretion, but would suggest that you make that clear)
- Proscribing a list of features that have to be enabled for the test, or at
least more than 1, feels like a strange choice here - I'd have expected
tests cases that either test the specific features one at a time, or suggest
several combinations, but that ultimately, we'd tell the tester to document
WHICH features were enabled, to make the test cases repeatable? This allows
the tester to apply a same set of apples to apples configurations to
different vendor gear, and omit the 1 feature that doesn't exist on a
different NGFW (for example), but hold a baseline that could be tested.
- Table 2: With the assumption that NGIPS/IDS are required to have the
features under "recommended", I disagree with this list. For example, some
customers break and inspect at the tap/agg layer of the network - in this
case, the feed into the NGIDS might be decrypted, and there's no need to
enable SSL inspection, for example.
- Table 3: I disagree that an NGIDS IS REQUIRED to decrypt SSL. This
behaviour might be suitable for an NGIPS, but the NGIDS is not a bump on the
wire, and often isn't decrypting and re-encrypting the traffic.
- Table 3: An NGIDS IMO is still a passive device - it wouldn't be blocking
anything, but agree that it might tell you that it happened after the fact.
- Table 3: Anti-evasion definition - define "mitigates".
- Table 3: Web-filtering - not a function of an NGIDS.
- Table 3: DLP: Not applicable for an NGIDS.
- Can you expand on "disposition of all flows of traffic are logged" -
what's meant here specifically, and why do they have to be logged? (Logging,
particularly under high loads, will impact it's own performance marks, and
colours output)
- ACLs wouldn't apply to an IDS because IDS's aren't blocking traffic :)
- It might be helpful to testers to say something like "look, here's one
suggested set of ACLs. If you're using them, great, reference that, but
otherwise, make note of the ACLs you use, and use the same ones for
repeatable testing".
- 4.3.1.1 The doc proscribes specific MSS values for v4/v6  with no
discussion around why they're chosen - that color could be useful to the
reader.
- 4.3.1.1 - there's a period on the 3rd to last line "(SYN/ACL, ACK). and"
that should be changed.
- 4.3.1.1 - As a tester with long time experience with major test equipment
manufacturers, I can't possibly begin to guess which ones of them would
conform to this - or even if they'd answer these questions. How helpful is
this section to the non test houses? I suggest expansion here, ideally with
either covering the scope of what you expect to cover, or hopefully which
(open source/generally available) test tools or emulators could be
considered for use as examples.
- 4.3.1.3 - Do the emulated web browser attributes really apply to testing
the NGIPS?
- 4.3.2.3 - Do you expect to also leverage TLS 1.3 as a configuration option
here?
- 4.3.4 - I'm surprised to see the requirement that all sessions establish a
distinct phase before moving on to the next. You might clarify why this is a
requirement, and why staggering them is specifically rejected?
- 5.1 - I like the sentence, but it leaves a world of possibilities open as
to how one confirmed that the ancillary switching or routing functions
didn't limit the performance, particularly the virtualized components?
- 5.3 - this is a nice assertion but again, how do I reasonably make the
assertion?
- 6.1 - I would suggest that the test report include the configuration of
ancillary devices on both client/server side as well
- 6.3 - Nothing on drops anywhere?
- 7.1.3.2 - Where are these numbers coming from? How are you determining the
"initial inspected throughput"? Maybe I missed that in the document overall,
but it's not clear to me where these KPIs are collected? I suggest this be
called out.
- 7.1.3.3 - what is a "relevant application traffic mix" profile?
- 7.1.3.4 - where does this monitoring occur?
- 7.1.3.4 - This looks a bit like conformance testing -  Why does item (b)
require a specific number/threshold?
- 9: Why is the cipher squite recommendation for a real deployment outside
the scope of this document?






_______________________________________________
bmwg mailing list
[hidden email]
https://www.ietf.org/mailman/listinfo/bmwg
Reply | Threaded
Open this post in threaded view
|

Re: WGLC on New version of draft-ietf-bmwg-ngfw-performance

Brian Monkman
In reply to this post by Sarah Banks-3
Hi Sarah,

Just wanted to let you know that members of the team working on the draft
just met. We will be sending a more detailed response soon. But in the
meantime, We thought we would share a couple of decisions/comments.

First, your suggestion that IDS be removed from the document is reasonable.
Thank you for your input. IDS will be removed from the draft. As such, we
will ignore any item you raised that was related directly to IDS.

Your comment, "As a tester with long time experience with major test
equipment manufacturers, I can't possibly begin to guess which ones of them
would conform to this - or even if they'd answer these questions" warrants
an immediate comment.  Both Ixia and Spirent have been closely involved with
the drafting of this document, have implemented the tests as documented and
have been extraordinarily willing to answer any questions related to the
requirements outlined within. The creation of this draft has been actively
supported by every member of our org. That is one of the reasons it has
taken us this long to reach where we are today.

Our goal from day one was to produce guidance on how to test a network
security product in a manner that would provide realistic results based on
real world deployment needs in a manner that can be reproduced and would
provide results that can be comparable regardless of the test tool or test
house used. Additionally, we feel strongly that the resulting guidelines
should become part of the public domain. We believed, and still believe, the
IETF is the right place. We have been very pleased with the input received
and the spirit it was provided. It appears that everyone who has commented
wants this work to be a useful addition to the knowledgebase of BMWG test
specs that have come before.

We believe the differences that are evidenced by your comments/questions
result from philosophical differences between the approach to testing
documented within the draft and your approach. It is indeed possible we can
both be right. I hope we can find compromise.

We will get the responses to the rest of your items posted as soon as
possible. Thank you for your patience.

Brian Monkman on behalf of....

Alex Samonte (Fortinet), Amritam Putatunda (Ixia/Keysight), Bala Balarajah
(NetSecOPEN), Carsten Rossenhoevel (EANTC), Chris Brown (UNH-IOL), Mike Jack
(Spirent), Ryan Liles (Cisco), Tim Carlin (UNH-IOL), Tim Otto (Juniper)

-----Original Message-----
From: Sarah Banks <[hidden email]>
Sent: Thursday, May 20, 2021 4:35 PM
To: ALFRED MORTON <[hidden email]>
Cc: [hidden email]; Gabor LENCSE <[hidden email]>; Bala
Balarajah <[hidden email]>; Bala Balarajah <[hidden email]>;
[hidden email]; MORTON, ALFRED C (AL) <[hidden email]>
Subject: Re: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance

Hi,
    Sharing feedback with my participant hat on. I do apologize for sending
it so late. A few high level comments:

- I still land in the "I don't agree that this doc clearly covers "all next
gen security devices" - and they're not clearly defined, either. For the
three that are covered, I think there's a lumping of the IPS and IDS that
would be better served separated. From my perspective, an IPS *is* a bump in
the wire, and does make "blocking" decisions, as called out in this
document. An IDS however, does not - it tells you it happened after the
fact, and it's a passive device. This draft is really keyed around "active
inline" functions - ie/ a bump in the wire - and I think keeping the focus
there makes sense, and lumping the IDS in does more harm than good.

In short, I don't support this draft moving forward with the inclusion of
IDS - it doesn't make sense to me. One way forward might be to remove the
IDS as a device from scope (as a suggestion).


- I've been testing for a long time, but I find this draft ... uncomfortable
to approach. I expected, and would have preferred, something that walks me
through a specific set of recommended scenarios with specifics on what to
configure for repeatable tests that I could compare results with, but there
are a lot of moving parts here and a lot of assertions that have to be made
for the test beds as a whole where I think the test results could vary
wildly when the same topology is handed to Test Person A and Test Person B.

- Most of the feedback below omits the IDS pieces, because so much of it
didn't apply.

Thanks,
Sarah



- The draft aims to replace RFC3511, but expands scope past Firewalls, to
"next generation security devices". I'm not finding a definition of what a
"next generation security device is", nor an exhaustive list of the devices
covered in this draft. A list that includes is nice, but IMO not enough to
cover what would be benchmarked here - I'd prefer to see a definition and an
exhaustive list.
- What is a NGIPS or NGIDS? If there are standardized definitions pointing
to them is fine, otherwise, there's a lot of wiggle room here.
- I still have the concern I shared at the last IETF meeting, where here,
we're putting active inline security devices in the same category as passive
devices. On one hand, I'm not sure I'd lump these three together in the
first place; on the other, active inline devices typically include
additional functions to allow administrators to control what happens to
packets in the case of failure, and I don't see those test cases included
here.
- Section 4.1 - it reads as if ANY device in the test setup cannot
contribute to network latency or throughput issues, including the DUTs - is
that what you intended?
- Option 1: It'd be nice to see a specific, clean, recommended test bed.
There are options for multiple emulated routers. As a tester, I expect to
see a specific, proscribed test bed that I should configure and test
against.
- Follow on: I'm curious as to the choice of emulated routers here. The
previous test suggests you avoid routers and switches in the topo, but then
there are emulated ones here. I'm curious as to what advantages you think
these bring over the real deal, and, why they aren't subject to the same
limitations previously described?
- In section 4.1 the text calls out Option 1 as the preferred test bed,
which includes L3 routing, but it's not clear why that's needed?
- The difference between Option 1 and Option 2 is the inclusion of
additional physical gear in Option 2 - it's not clear why that's needed, or
why the tester can't simply directly connect the test equipment to the DUT
and remove extraneous devices from potential influence on results?
- Section 4.2, the table for NGFW features - I'm not sure what the
difference is between RECOMMENDED and OPTIONAL? (I realize that you might be
saying that RECOMMENDED is the "must have enabled" features, where as
optional is at your discretion, but would suggest that you make that clear)
- Proscribing a list of features that have to be enabled for the test, or at
least more than 1, feels like a strange choice here - I'd have expected
tests cases that either test the specific features one at a time, or suggest
several combinations, but that ultimately, we'd tell the tester to document
WHICH features were enabled, to make the test cases repeatable? This allows
the tester to apply a same set of apples to apples configurations to
different vendor gear, and omit the 1 feature that doesn't exist on a
different NGFW (for example), but hold a baseline that could be tested.
- Table 2: With the assumption that NGIPS/IDS are required to have the
features under "recommended", I disagree with this list. For example, some
customers break and inspect at the tap/agg layer of the network - in this
case, the feed into the NGIDS might be decrypted, and there's no need to
enable SSL inspection, for example.
- Table 3: I disagree that an NGIDS IS REQUIRED to decrypt SSL. This
behaviour might be suitable for an NGIPS, but the NGIDS is not a bump on the
wire, and often isn't decrypting and re-encrypting the traffic.
- Table 3: An NGIDS IMO is still a passive device - it wouldn't be blocking
anything, but agree that it might tell you that it happened after the fact.
- Table 3: Anti-evasion definition - define "mitigates".
- Table 3: Web-filtering - not a function of an NGIDS.
- Table 3: DLP: Not applicable for an NGIDS.
- Can you expand on "disposition of all flows of traffic are logged" -
what's meant here specifically, and why do they have to be logged? (Logging,
particularly under high loads, will impact it's own performance marks, and
colours output)
- ACLs wouldn't apply to an IDS because IDS's aren't blocking traffic :)
- It might be helpful to testers to say something like "look, here's one
suggested set of ACLs. If you're using them, great, reference that, but
otherwise, make note of the ACLs you use, and use the same ones for
repeatable testing".
- 4.3.1.1 The doc proscribes specific MSS values for v4/v6  with no
discussion around why they're chosen - that color could be useful to the
reader.
- 4.3.1.1 - there's a period on the 3rd to last line "(SYN/ACL, ACK). and"
that should be changed.
- 4.3.1.1 - As a tester with long time experience with major test equipment
manufacturers, I can't possibly begin to guess which ones of them would
conform to this - or even if they'd answer these questions. How helpful is
this section to the non test houses? I suggest expansion here, ideally with
either covering the scope of what you expect to cover, or hopefully which
(open source/generally available) test tools or emulators could be
considered for use as examples.
- 4.3.1.3 - Do the emulated web browser attributes really apply to testing
the NGIPS?
- 4.3.2.3 - Do you expect to also leverage TLS 1.3 as a configuration option
here?
- 4.3.4 - I'm surprised to see the requirement that all sessions establish a
distinct phase before moving on to the next. You might clarify why this is a
requirement, and why staggering them is specifically rejected?
- 5.1 - I like the sentence, but it leaves a world of possibilities open as
to how one confirmed that the ancillary switching or routing functions
didn't limit the performance, particularly the virtualized components?
- 5.3 - this is a nice assertion but again, how do I reasonably make the
assertion?
- 6.1 - I would suggest that the test report include the configuration of
ancillary devices on both client/server side as well
- 6.3 - Nothing on drops anywhere?
- 7.1.3.2 - Where are these numbers coming from? How are you determining the
"initial inspected throughput"? Maybe I missed that in the document overall,
but it's not clear to me where these KPIs are collected? I suggest this be
called out.
- 7.1.3.3 - what is a "relevant application traffic mix" profile?
- 7.1.3.4 - where does this monitoring occur?
- 7.1.3.4 - This looks a bit like conformance testing -  Why does item (b)
require a specific number/threshold?
- 9: Why is the cipher squite recommendation for a real deployment outside
the scope of this document?






_______________________________________________
bmwg mailing list
[hidden email]
https://www.ietf.org/mailman/listinfo/bmwg
Reply | Threaded
Open this post in threaded view
|

Re: WGLC on New version of draft-ietf-bmwg-ngfw-performance

Brian Monkman
Hi Sarah,

As I mentioned in the previous message, we will remove reference to IDS from
the draft. Given that, none of the IDS related comments/questions are being
addressed.

Sorry for the delay in responding. I was unexpectedly out of the office.

Our responses are below, preceded by [bpm]. However, please note that this
is not a response from a single person, but from multiple people,
representing security product vendors, test labs and test tool vendors.

Brian

>>>

- The draft aims to replace RFC3511, but expands scope past Firewalls, to
"next generation security devices". I'm not finding a definition of what a
"next generation security device is", nor an exhaustive list of the devices
covered in this draft. A list that includes is nice, but IMO not enough to
cover what would be benchmarked here - I'd prefer to see a definition and an
exhaustive list.

[bpm] "We avoid limiting the draft by explicitly adding a list of NG
security devices currently available in the market only. In the future,
there may be more and more new types of NG security devices that will appear
on the market.

[bpm] This draft includes a list of security features that the security
device can have ( RFC 3511 doesn't have such a list). Also, we will describe
in the draft that the security devices must be configured ""in-line"" mode.
We believe these two points qualifying the definition of next generation
security.

- What is a NGIPS or NGIDS? If there are standardized definitions pointing
to them is fine, otherwise, there's a lot of wiggle room here.

[bpm] See above. We are removing NGIDS from the draft.

- I still have the concern I shared at the last IETF meeting, where here,
we're putting active inline security devices in the same category as passive
devices. On one hand, I'm not sure I'd lump these three together in the
first place; on the other, active inline devices typically include
additional functions to allow administrators to control what happens to
packets in the case of failure, and I don't see those test cases included
here.

[bpm] This draft focuses on ""in-line"" mode security devices only. We will
describe this in section 4 in more detail.

[bpm] Additionally, the draft focuses mainly on performance tests. The DUT
must be configured in ""fail close"" mode. We will describe this under
section 4. Any failure scenarios like ""fail open"" mode is out of scope.

- Section 4.1 - it reads as if ANY device in the test setup cannot
contribute to network latency or throughput issues, including the DUTs - is
that what you intended?

[bpm] "Our intention is, if the external devices (routers and switches) are
used in the test bed, they should not negatively impact DUT/SUT performance.
To address this, we added a section ( section 5 ""Test Bed Considerations"")
which recommends a pre-test.  We can rename this as reference test or
baseline test. "

- Option 1: It'd be nice to see a specific, clean, recommended test bed.
There are options for multiple emulated routers. As a tester, I expect to
see a specific, proscribed test bed that I should configure and test
against.

[bpm] The draft describes that Option 1 is the recommended test setup.
However. We added emulated routers as optional in option 1. The reason for
that:
Some type of security devices for some deployment scenarios requires routers
between test client/server and the DUT (e.g., NGFW) and some DUT/SUT doesn't
need router (e.g. NGIPS )

- Follow on: I'm curious as to the choice of emulated routers here. The
previous test suggests you avoid routers and switches in the topo, but then
there are emulated ones here. I'm curious as to what advantages you think
these bring over the real deal, and why they aren't subject to the same
limitations previously described?

[bpm] Comparing real router, the emulated router gives more advantages for
L7 testing.

[bpm] - Emulated router doesn't add latency. Even if it adds delay due to
the routing process, the test equipment can report the added latency, or it
can consider this for the latency measurement.

[bpm] - Emulated routers simply do routing function only. But in a "real"
router, we are not sure what else the router is doing with the packets.

[bpm] Your question regarding the need for routers:

[bpm] - We avoid impacting the DUT/SUT performance due to ARP or ND process

[bpm] - Represent realistic scenario (In the production environment the
security devices will not be directly connected with the clients.)

[bpm] - Routing (L3 mode) is commonly used in the NG security devices.

[bpm] However, in both figures we mentioned that router including emulated
router is optional. If there is no need have routing functionality on the
test bed (e.g., if we used very small number of clients and server IPs or
the DUT operates in Layer 2 mode), it can be ignored.

[bpm] Also, we described in Option 1, that the external devices are if there
is need to aggregate the interfaces of the tester or DUT.  For an example,
DUT has 2 Interfaces, but tester need to use it's 4 interfaces to achieve
the performance. So here we need switch/router to aggregate tester interface
from 4 to 2.

- In section 4.1 the text calls out Option 1 as the preferred test bed,
which includes L3 routing, but it's not clear why that's needed?

[bpm] See above.

- The difference between Option 1 and Option 2 is the inclusion of
additional physical gear in Option 2 - it's not clear why that's needed, or
why the tester can't simply directly connect the test equipment to the DUT
and remove extraneous devices from potential influence on results?

[bpm] See above.

- Section 4.2, the table for NGFW features - I'm not sure what the
difference is between RECOMMENDED and OPTIONAL? (I realize that you might be
saying that RECOMMENDED is the "must have enabled" features, where as
optional is at your discretion, but would suggest that you make that clear)

[bpm] The definition for OPTIONAL and RECOMMENDED is described in,  and
recommended, RFC2119. We already referenced this under the section 2
"Requirements".

- Proscribing a list of features that have to be enabled for the test, or at
least more than 1, feels like a strange choice here - I'd have expected
tests cases that either test the specific features one at a time, or suggest
several combinations, but that ultimately, we'd tell the tester to document
WHICH features were enabled, to make the test cases repeatable? This allows
the tester to apply a same set of apples to apples configurations to
different vendor gear, and omit the 1 feature that doesn't exist on a
different NGFW (for example), but hold a baseline that could be tested.

- Table 2: With the assumption that NGIPS/IDS are required to have the
features under "recommended", I disagree with this list. For example, some
customers break and inspect at the tap/agg layer of the network - in this
case, the feed into the NGIDS might be decrypted, and there's no need to
enable SSL inspection, for example.

[bpm] IDS is being removed.

- Table 3: I disagree that an NGIDS IS REQUIRED to decrypt SSL. This
behaviour might be suitable for an NGIPS, but the NGIDS is not a bump on the
wire, and often isn't decrypting and re-encrypting the traffic.

[bpm] IDS is being removed.

- Table 3: An NGIDS IMO is still a passive device - it wouldn't be blocking
anything, but agree that it might tell you that it happened after the fact.

[bpm] IDS is being removed.

- Table 3: Anti-evasion definition - define "mitigates".

[bpm] Not sure why you are asking this as mitigate is not an uncommon
term/word.

- Table 3: Web-filtering - not a function of an NGIDS.

[bpm] IDS is being removed.

- Table 3: DLP: Not applicable for an NGIDS.

[bpm] IDS is being removed.

- Can you expand on "disposition of all flows of traffic are logged" -
what's meant here specifically, and why do they have to be logged? (Logging,
particularly under high loads, will impact it's own performance marks, and
colours output)

[bpm] We intentionally recommended enabling logging which will impact the
performance. The draft is not aiming to get high performance number with
minimal DUT/SUT configuration. In contrast, it aims to get reasonable
performance number with realistic DUT configuration. The realistic
configuration can vary based on DUT/SUT deployment scenario.

[bpm] In most of the DUT/SUT deployment scenarios or customer environments,
logging is enabled as default configuration.

[bpm] "Disposition of all flows of traffic are logged": means that the
DUT/SUT need to log all the traffic at the flow level not each packet.

[bpm] We will add more clarification for the meaning of "disposition of all
flows of traffic are logged".

- ACLs wouldn't apply to an IDS because IDS's aren't blocking traffic :)

[bpm] IDS is being removed.

- It might be helpful to testers to say something like "look, here's one
suggested set of ACLs. If you're using them, great, reference that, but
otherwise, make note of the ACLs you use, and use the same ones for
repeatable testing".

[bpm] The draft gives guidance how to choose the ACL rules. We describe here
a methodology to create ACL.  

- 4.3.1.1 The doc proscribes specific MSS values for v4/v6 with no
discussion around why they're chosen - that color could be useful to the
reader.

[bpm] We will add some more clarification that these are the default number
used in most of the client operating systems currently.

- 4.3.1.1 - there's a period on the 3rd to last line "(SYN/ACL, ACK). and"
that should be changed.

[bpm] Thank you.

- 4.3.1.1 - As a tester with long time experience with major test equipment
manufacturers, I can't possibly begin to guess which ones of them would
conform to this - or even if they'd answer these questions. How helpful is
this section to the non test houses? I suggest expansion here, ideally with
either covering the scope of what you expect to cover, or hopefully which
(open source/generally available) test tools or emulators could be
considered for use as examples.

[bpm] We extensively discussed with Ixia and Spirent about this section.
This section was developed with significant input from these test tools
vendors in addition to others.

- 4.3.1.3 - Do the emulated web browser attributes really apply to testing
the NGIPS?

[bpm] Yes, we performed many PoC tests with test tools. Ixia and Spirent
confirmed this.

- 4.3.2.3 - Do you expect to also leverage TLS 1.3 as a configuration option
here?

[bpm] Yes

- 4.3.4 - I'm surprised to see the requirement that all sessions establish a
distinct phase before moving on to the next. You might clarify why this is a
requirement, and why staggering them is specifically rejected?

[bpm] This draft doesn't describe that all sessions establish a distinct
phase before moving on to the next. We will remove the word "distinct" from
the 1st paragraph in section 4.3.4.

[bpm] Unlike Layer 2/3 testing, Layer 7 testing requires several phases in
the traffic load profile. The traffic load profile described in the draft is
the common profile mostly used for Layer 7 testing.

- 5.1 - I like the sentence, but it leaves a world of possibilities open as
to how one confirmed that the ancillary switching, or routing functions
didn't limit the performance, particularly the virtualized components?

[bpm] The sentence says, "Ensure that any ancillary switching or routing
functions between the system under test and the test equipment do not limit
the performance of the traffic generator."

[bpm] Here we discuss the traffic generator performance, and this can be
confirmed by doing reference test.

[bpm] The section 5 recommends reference test to ensure that the maximum
desired traffic generator's performance. Based on the reference test results
it can be identified, if the external device added any impact on traffic
generator's performance.

[bpm] We will add more content in section 5 to provide more details about
reference test.

- 5.3 - this is a nice assertion but again, how do I reasonably make the
assertion?

[bpm] We will change the word from "Assertion" to "Ensure". Also, we will
add more clarity about reference testing.

- 6.1 - I would suggest that the test report include the configuration of
ancillary devices on both client/server side as well

[bpm] We believe that adding configuration of the ancillary devices doesn't
add more value in the report. Instead of this, we will recommend documenting
the configuration of the ancillary devices by doing reference test. We will
add this under the section 5 "Test bed consideration".

- 6.3 - Nothing on drops anywhere?

[bpm] Are you referring to packet drops? If you are, there is no packet loss
in stateful traffic. Instead of packet loss, the stateful traffic has
retransmissions.

- 7.1.3.2 - Where are these numbers coming from? How are you determining the
"initial inspected throughput"? Maybe I missed that in the document overall,
but it's not clear to me where these KPIs are collected? I suggest this be
called out.

[bpm] We will add more clarification in the next version. Thank you.

- 7.1.3.3 - what is a "relevant application traffic mix" profile?

[bpm] This is described in section7.1.1  (2nd paragraph). We will add the
word "relevant" in the 1st sentence of the 2nd pragraph.so the sentence will
be "Based on customer use case, users can choose the relevant application
traffic mix for this test.  The details about the traffic mix MUST be
documented in the report.  At least the following traffic mix details MUST
be documented and reported together with the test results:

- 7.1.3.4 - where does this monitoring occur?

[bpm] The monitoring or measurement must occur in the test equipment.
Section 4.3.4 describes this.

- 7.1.3.4 - This looks a bit like conformance testing -  Why does item (b)
require a specific number/threshold?

[bpm] These numbers are synonymous with the zero-packet loss criteria for
[RFC2544] Throughput and recognize the additional complexity of application
layer performance. This was agreed by the IETF BMWG.

- 9: Why is the cipher squite recommendation for a real deployment outside
the scope of this document?

[bpm] Because new cipher suites are frequently developed. Given that the
draft will not be easily updated once it is accepted as an RFC we wanted to
ensure there was flexibility to use future developed cipher suites.  

Brian Monkman on behalf of....

Alex Samonte (Fortinet), Amritam Putatunda (Ixia/Keysight), Bala Balarajah
(NetSecOPEN), Carsten Rossenhoevel (EANTC), Chris Brown (UNH-IOL), Mike Jack
(Spirent), Ryan Liles (Cisco), Tim Carlin (UNH-IOL), Tim Otto (Juniper)


-----Original Message-----
From: [hidden email] <[hidden email]>
Sent: Wednesday, May 26, 2021 3:34 PM
To: 'Sarah Banks' <[hidden email]>; 'ALFRED MORTON' <[hidden email]>
Cc: 'Gabor LENCSE' <[hidden email]>; 'Bala Balarajah'
<[hidden email]>; 'Bala Balarajah' <[hidden email]>;
[hidden email]; 'MORTON, ALFRED C (AL)' <[hidden email]>;
[hidden email]; [hidden email]; 'Carsten
Rossenhoevel' <[hidden email]>; 'Jack, Mike' <[hidden email]>;
'Christopher Brown' <[hidden email]>; 'Ryan Liles (ryliles)'
<[hidden email]>
Subject: RE: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance

Hi Sarah,

Just wanted to let you know that members of the team working on the draft
just met. We will be sending a more detailed response soon. But in the
meantime, We thought we would share a couple of decisions/comments.

First, your suggestion that IDS be removed from the document is reasonable.
Thank you for your input. IDS will be removed from the draft. As such, we
will ignore any item you raised that was related directly to IDS.

Your comment, "As a tester with long time experience with major test
equipment manufacturers, I can't possibly begin to guess which ones of them
would conform to this - or even if they'd answer these questions" warrants
an immediate comment.  Both Ixia and Spirent have been closely involved with
the drafting of this document, have implemented the tests as documented and
have been extraordinarily willing to answer any questions related to the
requirements outlined within. The creation of this draft has been actively
supported by every member of our org. That is one of the reasons it has
taken us this long to reach where we are today.

Our goal from day one was to produce guidance on how to test a network
security product in a manner that would provide realistic results based on
real world deployment needs in a manner that can be reproduced and would
provide results that can be comparable regardless of the test tool or test
house used. Additionally, we feel strongly that the resulting guidelines
should become part of the public domain. We believed, and still believe, the
IETF is the right place. We have been very pleased with the input received
and the spirit it was provided. It appears that everyone who has commented
wants this work to be a useful addition to the knowledgebase of BMWG test
specs that have come before.

We believe the differences that are evidenced by your comments/questions
result from philosophical differences between the approach to testing
documented within the draft and your approach. It is indeed possible we can
both be right. I hope we can find compromise.

We will get the responses to the rest of your items posted as soon as
possible. Thank you for your patience.

Brian Monkman on behalf of....

Alex Samonte (Fortinet), Amritam Putatunda (Ixia/Keysight), Bala Balarajah
(NetSecOPEN), Carsten Rossenhoevel (EANTC), Chris Brown (UNH-IOL), Mike Jack
(Spirent), Ryan Liles (Cisco), Tim Carlin (UNH-IOL), Tim Otto (Juniper)

-----Original Message-----
From: Sarah Banks <[hidden email]>
Sent: Thursday, May 20, 2021 4:35 PM
To: ALFRED MORTON <[hidden email]>
Cc: [hidden email]; Gabor LENCSE <[hidden email]>; Bala
Balarajah <[hidden email]>; Bala Balarajah <[hidden email]>;
[hidden email]; MORTON, ALFRED C (AL) <[hidden email]>
Subject: Re: [bmwg] WGLC on New version of draft-ietf-bmwg-ngfw-performance

Hi,
    Sharing feedback with my participant hat on. I do apologize for sending
it so late. A few high level comments:

- I still land in the "I don't agree that this doc clearly covers "all next
gen security devices" - and they're not clearly defined, either. For the
three that are covered, I think there's a lumping of the IPS and IDS that
would be better served separated. From my perspective, an IPS *is* a bump in
the wire, and does make "blocking" decisions, as called out in this
document. An IDS however, does not - it tells you it happened after the
fact, and it's a passive device. This draft is really keyed around "active
inline" functions - ie/ a bump in the wire - and I think keeping the focus
there makes sense, and lumping the IDS in does more harm than good.

In short, I don't support this draft moving forward with the inclusion of
IDS - it doesn't make sense to me. One way forward might be to remove the
IDS as a device from scope (as a suggestion).


- I've been testing for a long time, but I find this draft ... uncomfortable
to approach. I expected, and would have preferred, something that walks me
through a specific set of recommended scenarios with specifics on what to
configure for repeatable tests that I could compare results with, but there
are a lot of moving parts here and a lot of assertions that have to be made
for the test beds as a whole where I think the test results could vary
wildly when the same topology is handed to Test Person A and Test Person B.

- Most of the feedback below omits the IDS pieces, because so much of it
didn't apply.

Thanks,
Sarah



- The draft aims to replace RFC3511, but expands scope past Firewalls, to
"next generation security devices". I'm not finding a definition of what a
"next generation security device is", nor an exhaustive list of the devices
covered in this draft. A list that includes is nice, but IMO not enough to
cover what would be benchmarked here - I'd prefer to see a definition and an
exhaustive list.
- What is a NGIPS or NGIDS? If there are standardized definitions pointing
to them is fine, otherwise, there's a lot of wiggle room here.
- I still have the concern I shared at the last IETF meeting, where here,
we're putting active inline security devices in the same category as passive
devices. On one hand, I'm not sure I'd lump these three together in the
first place; on the other, active inline devices typically include
additional functions to allow administrators to control what happens to
packets in the case of failure, and I don't see those test cases included
here.
- Section 4.1 - it reads as if ANY device in the test setup cannot
contribute to network latency or throughput issues, including the DUTs - is
that what you intended?
- Option 1: It'd be nice to see a specific, clean, recommended test bed.
There are options for multiple emulated routers. As a tester, I expect to
see a specific, proscribed test bed that I should configure and test
against.
- Follow on: I'm curious as to the choice of emulated routers here. The
previous test suggests you avoid routers and switches in the topo, but then
there are emulated ones here. I'm curious as to what advantages you think
these bring over the real deal, and, why they aren't subject to the same
limitations previously described?
- In section 4.1 the text calls out Option 1 as the preferred test bed,
which includes L3 routing, but it's not clear why that's needed?
- The difference between Option 1 and Option 2 is the inclusion of
additional physical gear in Option 2 - it's not clear why that's needed, or
why the tester can't simply directly connect the test equipment to the DUT
and remove extraneous devices from potential influence on results?
- Section 4.2, the table for NGFW features - I'm not sure what the
difference is between RECOMMENDED and OPTIONAL? (I realize that you might be
saying that RECOMMENDED is the "must have enabled" features, where as
optional is at your discretion, but would suggest that you make that clear)
- Proscribing a list of features that have to be enabled for the test, or at
least more than 1, feels like a strange choice here - I'd have expected
tests cases that either test the specific features one at a time, or suggest
several combinations, but that ultimately, we'd tell the tester to document
WHICH features were enabled, to make the test cases repeatable? This allows
the tester to apply a same set of apples to apples configurations to
different vendor gear, and omit the 1 feature that doesn't exist on a
different NGFW (for example), but hold a baseline that could be tested.
- Table 2: With the assumption that NGIPS/IDS are required to have the
features under "recommended", I disagree with this list. For example, some
customers break and inspect at the tap/agg layer of the network - in this
case, the feed into the NGIDS might be decrypted, and there's no need to
enable SSL inspection, for example.
- Table 3: I disagree that an NGIDS IS REQUIRED to decrypt SSL. This
behaviour might be suitable for an NGIPS, but the NGIDS is not a bump on the
wire, and often isn't decrypting and re-encrypting the traffic.
- Table 3: An NGIDS IMO is still a passive device - it wouldn't be blocking
anything, but agree that it might tell you that it happened after the fact.
- Table 3: Anti-evasion definition - define "mitigates".
- Table 3: Web-filtering - not a function of an NGIDS.
- Table 3: DLP: Not applicable for an NGIDS.
- Can you expand on "disposition of all flows of traffic are logged" -
what's meant here specifically, and why do they have to be logged? (Logging,
particularly under high loads, will impact it's own performance marks, and
colours output)
- ACLs wouldn't apply to an IDS because IDS's aren't blocking traffic :)
- It might be helpful to testers to say something like "look, here's one
suggested set of ACLs. If you're using them, great, reference that, but
otherwise, make note of the ACLs you use, and use the same ones for
repeatable testing".
- 4.3.1.1 The doc proscribes specific MSS values for v4/v6  with no
discussion around why they're chosen - that color could be useful to the
reader.
- 4.3.1.1 - there's a period on the 3rd to last line "(SYN/ACL, ACK). and"
that should be changed.
- 4.3.1.1 - As a tester with long time experience with major test equipment
manufacturers, I can't possibly begin to guess which ones of them would
conform to this - or even if they'd answer these questions. How helpful is
this section to the non test houses? I suggest expansion here, ideally with
either covering the scope of what you expect to cover, or hopefully which
(open source/generally available) test tools or emulators could be
considered for use as examples.
- 4.3.1.3 - Do the emulated web browser attributes really apply to testing
the NGIPS?
- 4.3.2.3 - Do you expect to also leverage TLS 1.3 as a configuration option
here?
- 4.3.4 - I'm surprised to see the requirement that all sessions establish a
distinct phase before moving on to the next. You might clarify why this is a
requirement, and why staggering them is specifically rejected?
- 5.1 - I like the sentence, but it leaves a world of possibilities open as
to how one confirmed that the ancillary switching or routing functions
didn't limit the performance, particularly the virtualized components?
- 5.3 - this is a nice assertion but again, how do I reasonably make the
assertion?
- 6.1 - I would suggest that the test report include the configuration of
ancillary devices on both client/server side as well
- 6.3 - Nothing on drops anywhere?
- 7.1.3.2 - Where are these numbers coming from? How are you determining the
"initial inspected throughput"? Maybe I missed that in the document overall,
but it's not clear to me where these KPIs are collected? I suggest this be
called out.
- 7.1.3.3 - what is a "relevant application traffic mix" profile?
- 7.1.3.4 - where does this monitoring occur?
- 7.1.3.4 - This looks a bit like conformance testing -  Why does item (b)
require a specific number/threshold?
- 9: Why is the cipher squite recommendation for a real deployment outside
the scope of this document?







_______________________________________________
bmwg mailing list
[hidden email]
https://www.ietf.org/mailman/listinfo/bmwg