OneFS SmartQoS Monitoring and Troubleshooting
Tue, 21 Mar 2023 18:30:54 -0000
|Read Time: 0 minutes
The previous articles in this series have covered the SmartQoS architecture, configuration, and management. Now, we’ll turn our attention to monitoring and troubleshooting.
You can use the ‘isi statistics workload’ CLI command to monitor the dataset’s performance. The ‘Ops’ column displays the current protocol operations per second. In the following example, Ops stabilize around 9.8, which is just below the configured limit value of 10 Ops.
# isi statistics workload --dataset ds1 & data
Similarly, this next example from the SmartQoS WebUI shows a small NFS workflow performing 497 protocol Ops in a pinned workload with a limit of 500 Ops:
You can pin multiple paths and protocols by selecting the ‘Pin Workload’ option for a given Dataset. Here, four directory path workloads are each configured with different Protocol Ops limits:
When it comes to troubleshooting SmartQoS, there are a few areas that are worth checking right away, including the SmartQoS Ops limit configuration, isi_pp_d and isi_stats_d daemons, and the protocol service(s).
1. For suspected Ops limit configuration issues, first confirm that the SmartQoS limits feature is enabled:
# isi performance settings view Top N Collections: 1024 Time In Queue Threshold (ms): 10.0 Target read latency in microseconds: 12000.0 Target write latency in microseconds: 12000.0 Protocol Ops Limit Enabled: Yes
Next, verify that the workload level protocols_ops limit is correctly configured:
# isi performance workloads view <workload>
Check whether any errors are reported in the isi_tardis_d configuration log:
# cat /var/log/isi_tardis_d.log
2. To investigate isi_pp_d, first check that the service is enabled:
# isi services –a isi_pp_d Service 'isi_pp_d' is enabled.
If necessary, you can restart the isi_pp_d service as follows:
# isi services isi_pp_d disable Service 'isi_pp_d' is disabled. # isi services isi_pp_d enable Service 'isi_pp_d' is enabled.
There’s also an isi_pp_d debug tool, which can be helpful in a pinch:
# isi_pp_d -h Usage: isi_pp_d [-ldhs] -l Run as a leader process; otherwise, run as a follower. Only one leader process on the cluster will be active. -d Run in debug mode (do not daemonize). -s Display pp_leader node (devid and lnn) -h Display this help.
You can enable debugging on the isi_pp_d log file with the following command syntax:
# isi_ilog -a isi_pp_d -l debug, /var/log/isi_pp_d.log
For example, the following log snippet shows a typical isi_ppd_d.log message communication between isi_pp_d leader and isi_pp_d followers:
/ifs/.ifsvar/modules/pp/comm/SETTINGS [090500b000000b80,08020000:0000bfddffffffff,09000100:ffbcff7cbb9779de,09000100:d8d2fee9ff9e3bfe,090001 00:0000000075f0dfdf] 100,,,,20,1658854839 < in the format of <workload_id, cputime, disk_reads, disk_writes, protocol_ops, timestamp>
Here, the extract from the /var/log/isi_pp_d.log logfiles from nodes 1 and 2 of a cluster illustrate the different stages of Protocol Ops limit enforcement and usage:
3. To investigate the isi_stats_d, first confirm that the isi_pp_d service is enabled:
# isi services -a isi_stats_d Service 'isi_stats_d' is enabled.
If necessary, you can restart the isi_stats_d service as follows:
# isi services isi_stats_d disable # isi services isi_stats_d enable
You can view the workload level statistics with the following command:
# isi statistics workload list --dataset=<name>
You can enable debugging on the isi_stats_d log file with the following command syntax:
# isi_stats_tool --action set_tracelevel --value debug # cat /var/log/isi_stats_d.log
4. To investigate protocol issues, the ‘isi services’ and ‘lwsm’ CLI commands can be useful. For example, to check the status of the S3 protocol:
# /usr/likewise/bin/lwsm list | grep -i protocol hdfs [protocol] stopped lwswift [protocol] running (lwswift: 8393) nfs [protocol] running (nfs: 8396) s3 [protocol] stopped srv [protocol] running (lwio: 8096) # /usr/likewise/bin/lwsm status s3 stopped # /usr/likewise/bin/lwsm info s3 Service: s3 Description: S3 Server Categories: protocol Path: /usr/likewise/lib/lw-svcm/s3.so Arguments: Dependencies: lsass onefs_s3 AuditEnabled?flt_audit_s3 Container: s3
This CLI output confirms that the S3 protocol is inactive. You can start the S3 service as follows:
# isi services -a | grep -i s3 s3 S3 Service Enabled
Similarly, you can restart the S3 service as follows:
# /usr/likewise/bin/lwsm restart s3 Stopping service: s3 Starting service: s3
To investigate further, you can increase the protocol’s log level verbosity. For example, to set the s3 log to ‘debug’:
# isi s3 log-level view Current logging level is 'info' # isi s3 log-level modify debug # isi s3 log-level view Current logging level is 'debug'
Next, view and monitor the appropriate protocol log. For example, for the S3 protocol:
# cat /var/log/s3.log # tail -f /var/log/s3.log
Beyond the above, you can monitor /var/log/messages for pertinent errors, because the main partition performance (PP) modules log to this file. You can enable debug level logging for the various PP modules as follows.
Dataset:
# sysctl ilog.ifs.acct.raa.syslog=debug+ ilog.ifs.acct.raa.syslog: error,warning,notice (inherited) -> error,warning,notice,info,debug
Workload:
# sysctl ilog.ifs.acct.rat.syslog=debug+ ilog.ifs.acct.rat.syslog: error,warning,notice (inherited) -> error,warning,notice,info,debug
Actor work:
# sysctl ilog.ifs.acct.work.syslog=debug+ ilog.ifs.acct.work.syslog: error,warning,notice (inherited) -> error,warning,notice,info,debug
When finished, you can restore the default logging levels for the above modules as follows:
# sysctl ilog.ifs.acct.raa.syslog=notice+ # sysctl ilog.ifs.acct.rat.syslog=notice+ # sysctl ilog.ifs.acct.work.syslog=notice+
Author: Nick Trimbee
Related Blog Posts
OneFS WebUI Single Sign-on Management and Troubleshooting
Thu, 16 Nov 2023 20:53:16 -0000
|Read Time: 0 minutes
Earlier in this series, we took a look at the architecture of the new OneFS WebUI SSO functionality. Now, we move on to its management and troubleshooting.
As we saw in the previous article, once the IdP and SP are configured, a cluster admin can enable SSO per access zone using the OneFS WebUI by navigating to Access > Authentication providers > SSO. From here, select the desired access zone and click the ‘Enable SSO’ toggle:
Or from the OneFS CLI using the following syntax:
# isi auth sso settings modify --sso-enabled 1
Once complete, the SSO configuration can be verified from a client web browser by browsing to the OneFS login screen. If all is operating correctly, redirection to the ADFS login screen will occur. For example:
After successful authentication with ADFS, cluster access is granted and the browser session is redirected back to the OneFS WebUI .
In addition to the new SSO WebUI pages, OneFS 9.5 also adds a subcommand to the ‘isi auth’ command set for configuring SSO from the CLI. This new syntax includes:
- isi auth sso idps
- isi auth sso settings
- isi auth sso sp
With these, you can use the following procedure to configure and enable SSO using the OneFS command line.
1. Define the ADFS instance in OneFS.
Enter the following command to create the IdP account:
# isi auth ads create <domain_name> <user> --password=<password> ...
where:
Attribute | Description |
<domain_name> | Fully qualified Active Directory domain name that identifies the ADFS server. For example, idp1.isilon.com. |
<user> | The user account that has permission to join machines to the given domain. |
<password> | The password for <user>. |
2. Next, add the IdP to the pertinent OneFS zone. Note that each of a cluster’s access zone(s) must have an IdP configured for it. The same IdP can be used for all the zones, but each access zone must be configured separately.
# isi zone zones modify --add-auth-providers
For example:
# isi zone zones modify system --add-auth-providers=lsa-activedirectoryprovider:idp1.isilon.com
3. Verify that OneFS can find users in Active Directory.
# isi auth users view idp1.isilon.com\\<username>
In the output, ensure that an email address is displayed. If not, return to Active Directory and assign email addresses to users.
4. Configure the OneFS hostname for SAML SSO.
# isi auth sso sp modify --hostname=<name>
Where <name> is the name that SAML SSO can use to represent the OneFS cluster to ADFS. SAML redirects clients to this hostname.
5. Obtain the ADFS metadata and store it under /ifs on the cluster.
In the following example, an HTTPS GET request is issued using the 'curl' utility to obtain the metadata from the IDP and store it under /ifs on the cluster.
# curl -o /ifs/adfs.xml https://idp1.isilon.com/FederationMetadata/2007-06/ FederationMetadata.xml
6. Create the IdP on OneFS using the ‘metadata-location’ path for the xml file in the previous step.
# isi auth sso idps create idp1.isilon.com --metadata-location="/ifs/adfs.xml"
7. Enable SSO:
# isi auth sso settings modify --sso-enabled=yes -–zone <zone>
Use the following syntax to view the IdP configuration:
# isi auth sso idps view <idp_ID>
For example:
# isi auth sso idps view idp ID: idp Metadata Location: /ifs/adfs.xml Entity ID: https://dns.isilon.com/adfs/services/trust Login endpoint of the IDP URL: https://dns.isilon.com/adfs/ls/ Binding: wrn:oasis:names:tc:SAML:2.0:bidings:HTTP-Redirect Logout endpoint of the IDP URL: https://dns.isilon.com/adfs/ls/ Binding: wrn:oasis:names:tc:SAML:2.0:bidings:HTTP-Redirect Response URL: - Type: metadata Signing Certificate: - Path: Issuer: CN-ADFS Signing – dns.isilon.com Subject: CN-ADFS Signing – dns.isilon.com Not Before: 2023-02-02T22:22:00 Not After: 2024-02-02T22:22:00 Status: valid Value and Type Value: -----BEGIN CERTIFICATE----- MITC9DCCAdygAwIBAgIQQQQc55appr1CtfPNj5kv+DANBgkqhk1G9w8BAQsFADA2 <snip>
Troubleshooting
If the IdP and/or SP Signing certificate happens to expire, users will be unable to login to the cluster with SSO and an error message will be displayed on the login screen.
In this example, the IdP certificate has expired, as described in the alert message. When this occurs, a warning is also displayed on the SSO Authentication page, as shown here:
To correct this, download either a new signing certificate from the identity provider or a new metadata file containing the IdP certificate details. When this is complete, you can then update the cluster’s IdP configuration by uploading the XML file or the new certificate.
Similarly, if the SP certificate has expired, the following notification alert is displayed upon attempted login:
The following error message is also displayed on the WebUI SSO tab, under Access > Authentication providers > SSO, along with a link to regenerate the metadata file:
The expired SP signing key and certificate can also be easily regenerated from the OneFS CLI:
# isi auth sso sp signing-key rekey
This command will delete any existing signing key and certificate and replace them with a newly generated signing key and certificate. Make sure the newly generated certificate is added to the IDP to ensure that the IDP can verify messages sent from the cluster. Are you sure? (yes/[no]): yes # isi auth sso sp signing-key dump -----BEGIN CERIFICATE----- MIIE6TCCAtGgAwIBAgIJAP30nSyYUz/cMA0GCSqGSIb3DQEBCwUAMCYxJDAiBgNVBAMMG1Bvd2VyU2NhbGUgU0FNTCBTaWduaWSnIEtleTAeFw0yMjExMTUwMzU0NTFaFw0yMzExMTUwMzU0NTFaMCYxJDAiBgNVBAMMG1Bvd2VyU2NhbGUgU0FNTCBTaWduaWSnIEtleTCCAilwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMOOmYJ1aUuxvyH0nbUMurMbQubgtdpVBevy12D3qn+x7rgym8/v50da/4xpMmv/zbE0zJ0IVbWHZedibtQhLZ1qRSY/vBlaztU/nA90XQzXMnckzpcunOTG29SMO3x3Ud4*fqcP4sKhV <snip>
When it is regenerated, either the XML file or certificate can be downloaded, and the cluster configuration updated by either metadata download or manual copy:
Finally, upload the SP details back to the identity provider.
For additional troubleshooting of OneFS SSO and authentication issues, there are some key log files to check. These include:
Log file | Information |
/var/log/isi_saml_d.log | SAML specific log messages logged by isi_saml_d. |
/var/log/apache2/webui_httpd_error.log | WebUI error messages including some SAML errors logged by the WebUI HTTP server. |
/var/log/jwt.log | Errors related to token generation logged by the JWT service. |
/var/log/lsassd.log | General authentication errors logged by the ‘lsassd’ service, such as failing to lookup users by email. |
Author: Nick Trimbee
OneFS Firewall Management and Troubleshooting
Thu, 25 May 2023 14:41:59 -0000
|Read Time: 0 minutes
In the final blog in this series, we’ll focus on step five of the OneFS firewall provisioning process and turn our attention to some of the management and monitoring considerations and troubleshooting tools associated with the firewall.
One can manage and monitor the firewall in OneFS 9.5 using the CLI, platform API, or WebUI. Because data security threats come from inside an environment as well as out, such as from a rogue IT employee, a good practice is to constrain the use of all-powerful ‘root’, ‘administrator’, and ‘sudo’ accounts as much as possible. Instead of granting cluster admins full rights, a preferred approach is to use OneFS’ comprehensive authentication, authorization, and accounting framework.
OneFS role-based access control (RBAC) can be used to explicitly limit who has access to configure and monitor the firewall. A cluster security administrator selects the desired access zone, creates a zone-aware role within it, assigns privileges, and then assigns members. For example, from the WebUI under Access > Membership and roles > Roles:
When these members login to the cluster from a configuration interface (WebUI, Platform API, or CLI) they inherit their assigned privileges.
Accessing the firewall from the WebUI and CLI in OneFS 9.5 requires the new ISI_PRIV_FIREWALL administration privilege.
# isi auth privileges -v | grep -i -A 2 firewall ID: ISI_PRIV_FIREWALL Description: Configure network firewall Name: Firewall Category: Configuration Permission: w
This privilege can be assigned one of four permission levels for a role, including:
Permission Indicator | Description |
– | No permission. |
R | Read-only permission. |
X | Execute permission. |
W | Write permission. |
By default, the built-in ‘SystemAdmin’ roles is granted write privileges to administer the firewall, while the built-in ‘AuditAdmin’ role has read permission to view the firewall configuration and logs.
With OneFS RBAC, an enhanced security approach for a site could be to create two additional roles on a cluster, each with an increasing realm of trust. For example:
1. An IT ops/helpdesk role with ‘read’ access to the snapshot attributes would permit monitoring and troubleshooting the firewall, but no changes:
RBAC Role | Firewall Privilege | Permission |
IT_Ops | ISI_PRIV_FIREWALL | Read |
For example:
# isi auth roles create IT_Ops # isi auth roles modify IT_Ops --add-priv-read ISI_PRIV_FIREWALL # isi auth roles view IT_Ops | grep -A2 -i firewall ID: ISI_PRIV_FIREWALL Permission: r
2. A Firewall Admin role would provide full firewall configuration and management rights:
RBAC Role | Firewall Privilege | Permission |
FirewallAdmin | ISI_PRIV_FIREWALL | Write |
For example:
# isi auth roles create FirewallAdmin # isi auth roles modify FirewallAdmin –add-priv-write ISI_PRIV_FIREWALL # isi auth roles view FirewallAdmin | grep -A2 -i firewall ID: ISI_PRIV_FIREWALL Permission: w
Note that when configuring OneFS RBAC, remember to remove the ‘ISI_PRIV_AUTH’ and ‘ISI_PRIV_ROLE’ privilege from all but the most trusted administrators.
Additionally, enterprise security management tools such as CyberArk can also be incorporated to manage authentication and access control holistically across an environment. These can be configured to change passwords on trusted accounts frequently (every hour or so), require multi-Level approvals prior to retrieving passwords, and track and audit password requests and trends.
OneFS firewall limits
When working with the OneFS firewall, there are some upper bounds to the configurable attributes to keep in mind. These include:
Name | Value | Description |
MAX_INTERFACES | 500 | Maximum number of L2 interfaces including Ethernet, VLAN, LAGG interfaces on a node. |
MAX _SUBNETS | 100 | Maximum number of subnets within a OneFS cluster |
MAX_POOLS | 100 | Maximum number of network pools within a OneFS cluster |
DEFAULT_MAX_RULES | 100 | Default value of maximum rules within a firewall policy |
MAX_RULES | 200 | Upper limit of maximum rules within a firewall policy |
MAX_ACTIVE_RULES | 5000 | Upper limit of total active rules across the whole cluster |
MAX_INACTIVE_POLICIES | 200 | Maximum number of policies that are not applied to any network subnet or pool. They will not be written into ipfw tables. |
Firewall performance
Be aware that, while the OneFS firewall can greatly enhance the network security of a cluster, by nature of its packet inspection and filtering activity, it does come with a slight performance penalty (generally less than 5%).
Firewall and hardening mode
If OneFS STIG hardening (that is, from ‘isi hardening apply’) is applied to a cluster with the OneFS firewall disabled, the firewall will be automatically activated. On the other hand, if the firewall is already enabled, then there will be no change and it will remain active.
Firewall and user-configurable ports
Some OneFS services allow the TCP/UDP ports on which the daemon listens to be changed. These include:
Service | CLI Command | Default Port |
NDMP | isi ndmp settings global modify –port | 10000 |
S3 | isi s3 settings global modify –https-port | 9020, 9021 |
SSH | isi ssh settings modify –port | 22 |
The default ports for these services are already configured in the associated global policy rules. For example, for the S3 protocol:
# isi network firewall rules list | grep s3 default_pools_policy.rule_s3 55 Firewall rule on s3 service allow # isi network firewall rules view default_pools_policy.rule_s3 ID: default_pools_policy.rule_s3 Name: rule_s3 Index: 55 Description: Firewall rule on s3 service Protocol: TCP Dst Ports: 9020, 9021 Src Networks: - Src Ports: - Action: allow
Note that the global policies, or any custom policies, do not auto-update if these ports are reconfigured. This means that the firewall policies must be manually updated when changing ports. For example, if the NDMP port is changed from 10000 to 10001:
# isi ndmp settings global view Service: False Port: 10000 DMA: generic Bre Max Num Contexts: 64 MSB Context Retention Duration: 300 MSR Context Retention Duration: 600 Stub File Open Timeout: 15 Enable Redirector: False Enable Throttler: False Throttler CPU Threshold: 50 # isi ndmp settings global modify --port 10001 # isi ndmp settings global view | grep -i port Port: 10001
The firewall’s NDMP rule port configuration must also be reset to 10001:
# isi network firewall rule list | grep ndmp default_pools_policy.rule_ndmp 44 Firewall rule on ndmp service allow # isi network firewall rule modify default_pools_policy.rule_ndmp --dst-ports 10001 --live # isi network firewall rule view default_pools_policy.rule_ndmp | grep -i dst Dst Ports: 10001
Note that the –live flag is specified to enact this port change immediately.
Firewall and source-based routing
Under the hood, OneFS source-based routing (SBR) and the OneFS firewall both leverage ‘ipfw’. As such, SBR and the firewall share the single ipfw table in the kernel. However, the two features use separate ipfw table partitions.
This allows SBR and the firewall to be activated independently of each other. For example, even if the firewall is disabled, SBR can still be enabled and any configured SBR rules displayed as expected (that is, using ipfw set 0 show).
Firewall and IPv6
Note that the firewall’s global default policies have a rule allowing ICMP6 by default. For IPv6 enabled networks, ICMP6 is critical for the functioning of NDP (Neighbor Discovery Protocol). As such, when creating custom firewall policies and rules for IPv6-enabled network subnets/pools, be sure to add a rule allowing ICMP6 to support NDP. As discussed in a previous blog, an alternative (and potentially easier) approach is to clone a global policy to a new one and just customize its ruleset instead.
Firewall and FTP
The OneFS FTP service can work in two modes: Active and Passive. Passive mode is the default, where FTP data connections are created on top of random ephemeral ports. However, because the OneFS firewall requires fixed ports to operate, it only supports the FTP service in Active mode. Attempts to enable the firewall with FTP running in Passive mode will generate the following warning:
# isi ftp settings view | grep -i active Active Mode: No # isi network firewall settings modify --enabled yes FTP service is running in Passive mode. Enabling network firewall will lead to FTP clients having their connections blocked. To avoid this, please enable FTP active mode and ensure clients are configured in active mode before retrying. Are you sure you want to proceed and enable network firewall? (yes/[no]):
To activate the OneFS firewall in conjunction with the FTP service, first ensure that the FTP service is running in Active mode before enabling the firewall. For example:
# isi ftp settings view | grep -i enable FTP Service Enabled: Yes # isi ftp settings view | grep -i active Active Mode: No # isi ftp setting modify –active-mode true # isi ftp settings view | grep -i active Active Mode: Yes # isi network firewall settings modify --enabled yes
Note: Verify FTP active mode support and/or firewall settings on the client side, too.
Firewall monitoring and troubleshooting
When it comes to monitoring the OneFS firewall, the following logfiles and utilities provide a variety of information and are a good source to start investigating an issue:
Utility | Description |
/var/log/isi_firewall_d.log | Main OneFS firewall log file, which includes information from firewall daemon. |
/var/log/isi_papi_d.log | Logfile for platform AP, including Firewall related handlers. |
isi_gconfig -t firewall | CLI command that displays all firewall configuration info. |
ipfw show | CLI command that displays the ipfw table residing in the FreeBSD kernel. |
Note that the preceding files and command output are automatically included in logsets generated by the ‘isi_gather_info’ data collection tool.
You can run the isi_gconfig command with the ‘-q’ flag to identify any values that are not at their default settings. For example, the stock (default) isi_firewall_d gconfig context will not report any configuration entries:
# isi_gconfig -q -t firewall [root] {version:1}
The firewall can also be run in the foreground for additional active rule reporting and debug output. For example, first shut down the isi_firewall_d service:
# isi services -a isi_firewall_d disable The service 'isi_firewall_d' has been disabled.
Next, start up the firewall with the ‘-f’ flag.
# isi_firewall_d -f Acquiring kevents for flxconfig Acquiring kevents for nodeinfo Acquiring kevents for firewall config Initialize the firewall library Initialize the ipfw set ipfw: Rule added by ipfw is for temporary use and will be auto flushed soon. Use isi firewall instead. cmd:/sbin/ipfw set enable 0 normal termination, exit code:0 isi_firewall_d is now running Loaded master FlexNet config (rev:312) Update the local firewall with changed files: flx_config, Node info, Firewall config Start to update the firewall rule... flx_config version changed! latest_flx_config_revision: new:312, orig:0 node_info version changed! latest_node_info_revision: new:1, orig:0 firewall gconfig version changed! latest_fw_gconfig_revision: new:17, orig:0 Start to update the firewall rule for firewall configuration (gconfig) Start to handle the firewall configure (gconfig) Handle the firewall policy default_pools_policy ipfw: Rule added by ipfw is for temporary use and will be auto flushed soon. Use isi firewall instead. 32043 allow tcp from any to any 10000 in cmd:/sbin/ipfw add 32043 set 8 allow TCP from any to any 10000 in normal termination, exit code:0 ipfw: Rule added by ipfw is for temporary use and will be auto flushed soon. Use isi firewall instead. 32044 allow tcp from any to any 389,636 in cmd:/sbin/ipfw add 32044 set 8 allow TCP from any to any 389,636 in normal termination, exit code:0 Snip...
If the OneFS firewall is enabled and some network traffic is blocked, either this or the ipfw show CLI command will often provide the first clues.
Please note that the ipfw command should NEVER be used to modify the OneFS firewall table!
For example, say a rule is added to the default pools policy denying traffic on port 9876 from all source networks (0.0.0.0/0):
# isi network firewall rules create default_pools_policy.rule_9876 --index=100 --dst-ports 9876 --src-networks 0.0.0.0/0 --action deny –live # isi network firewall rules view default_pools_policy.rule_9876 ID: default_pools_policy.rule_9876 Name: rule_9876 Index: 100 Description: Protocol: ALL Dst Ports: 9876 Src Networks: 0.0.0.0/0 Src Ports: - Action: deny
Running ipfw show and grepping for the port will show this new rule:
# ipfw show | grep 9876 32099 0 0 deny ip from any to any 9876 in
The ipfw show command output also reports the statistics of how many IP packets have matched each rule This can be incredibly useful when investigating firewall issues. For example, a telnet session is initiated to the cluster on port 9876 from a client:
# telnet 10.224.127.8 9876 Trying 10.224.127.8... telnet: connect to address 10.224.127.8: Operation timed out telnet: Unable to connect to remote host
The connection attempt will time out because the port 9876 ‘deny’ rule will silently drop the packets. At the same time, the ipfw show command will increment its counter to report on the denied packets. For example:
# ipfw show | grep 9876 32099 9 540 deny ip from any to any 9876 in
If this behavior is not anticipated or desired, you can find the rule name by searching the rules list for the port number, in this case port 9876:
# isi network firewall rules list | grep 9876 default_pools_policy.rule_9876 100 deny
The offending rule can then be reverted to ‘allow’ traffic on port 9876:
# isi network firewall rules modify default_pools_policy.rule_9876 --action allow --live
Or easily deleted, if preferred:
# isi network firewall rules delete default_pools_policy.rule_9876 --live Are you sure you want to delete firewall rule default_pools_policy.rule_9876? (yes/[no]): yes
Author: Nick Trimbee