Node Manager Basics and Setup

about Node Manager

Node Manager is an important utility when running WebLogic in a production environment. Most people pay little attention to the Node Manager, but when problem arises, it is a little bit late to figure out what Node Manager is and what it does.

This article is not a full introduction to Node Manager, but will describe some best practices for common real life situations and issues, and share some tips and tricks gathered over the years.

If you would like a full introduction to Node Manager first, with all the nice diagrams showing how everything is connected, then you should look at the official documentation, which by the way are getting pretty good, before coming back to read the rest of this article -  http://docs.oracle.com/middleware/1213/wls/NODEM/overview.htm#NODEM112

So, what is Node Manager?

Node Manager is a utility used to start, stop and restart WebLogic and Coherence servers on the host Node Manager is installed. Since you could connect to Node Manager from a remote server, you will use Node Manager on the remote server as a tool to remotely start and stop a WebLogic instance.

There are two version of Node Manager, the Java version and the script based version. Unless you have very specific requirements, use the Java version, which also happens to be the one that most people think of when they talk about Node Manager. This article will be about the Java version, but if you want to know more about the difference of the two versions, have a look at the official documentation - http://docs.oracle.com/middleware/1213/wls/NODEM/overview.htm#NODEM114

Per host vs per domain Node Manager

Up until WebLogic 12.1.2 the default was one Node Manager per host. It was possible to do it otherwise, but it was not documented very well.

From 12.1.2 the default is one Node Manager per domain, and the configuration and logs are now located under $DOMAIN_HOME/nodemanager instead of under wlserver_10.3/common/nodemanager

It is still possible to have one Node Manager per host, for those who prefers that.

If you are using Node Manager for an earlier release (before 12.1.2) or just want to stick with one Node Manager per host, you should consider moving the configuration and log files away from wlserver_10.3/common/nodemanger and have them outside the $ORACLE_HOME. There are several reasons for that, one important reason is to keep configuration files away from the binaries, so the configuration is kept in case an upgrade overwrites everything under wlserver_10.3/common/nodemanager. Another important reason is that you want to have logfiles in a location that you monitor for disk space, since logfiles tends to grow, and you might not have taken that into consideration when deciding how much space you needed for the installation.

One special case where you actually needed more than one Node Manager for a host in 11g was when you followed the Enterprise Deployment Guide (EDG) for SOA. In an enterprise setup, the Administration Server is installed in a different $DOMAIN_HOME than the managed servers. This makes it easier to start the Administration Server on another host in case of a host failure (Administration Server domain directory should be on a SAN or a shared disk).

The reason for using one Node Manager for the Administration Server domain directory and one for the Managed Servers domain directory is that the nodemanager.domains file in 11g could only can have one path per domain.

In 12c there is possible to have one primary domain path and several alternate domain paths for each domain in nodemanager.domains, so it should be enough with one Node Manager per domain from 12.1.3 http://docs.oracle.com/middleware/1213/wls/NODEM/java_nodemgr.htm#NODEM163

Since one Node Manager per domain could mean several Node Manager instances on one host, every Node Manager instance on that host must have a unique ip:port combination.

How to start Node Manager?

The simple answer is to run startNodeManager.sh from $DOMAIN_HOME/bin, but except for testing, you should not do that.

Node Manager should be started as a system service when the server boots, so a startup script is needed. In the WebLogic 12.1.3 documentation there are init.d and xinetd scripts provided for Linux - http://docs.oracle.com/middleware/1213/wls/NODEM/java_nodemgr.htm#NODEM247

For Windows there is the installNodeMgrSvc.cmd script included in the $DOMAIN_HOME/bin directory.

Node Manager will run as the user who started it. When Node Manager starts a WebLogic instance, either an Administration Server or a Managed Server, the WebLogic instance will run as the same user as the user who runs Node Manager.

There is currently some flaws in the documentation, both in the init.d script for Linux and in the description of installNodeMgrSvc.cmd for Windows. Oracle has been notified, and the documentation should be updated soon.

By using the init.d script from the documentation, Node Manager will run as root, and by starting it as a Windows service, it will run as System.

Running WebLogic as root/System could be a security risk, and when you are going to patch WebLogic, you will most likely do that as the user you installed WebLogic as. However, since some files have changed ownership/rights to root/System, you might not be able to patch without a lot of hassle.

On Linux there is a post-bind function in WebLogic, so it is possible to configure WebLogic to change the user from root to something less hazardous, but this is not possible in Windows.

So, what should we do?

For Linux, make sure Node Manager is started as a non-root user, by changing the line in the init.d file in the documentation from

$PROGRAM_START >> $OUT_FILE 2>&1  &

to

su -c $NMUSER $PROGRAM_START >> $OUT_FILE 2>&1  &

The variable NMUSER could be declared somewhere between

#### BEGIN CUSTOM ENV

and

#### END CUSTOM ENV

as

NMUSER=oracle

Replace oracle with the user you want to run Node Manager and WebLogic.

For Windows you could go to services and properties for the WebLogicNodeManager service. On the “Log On” tab, select “This account” and select the user account that you want to run Node Manager (and WebLogic) as.

​

Username and password

To log into Node Manager, you have to use a username and password. Unfortunately, the default username is weblogic, which also happens to be the default username for the WebLogic Administration server. However, these two weblogic users are not the same, it is just a coincident that their names are the same. There is no relation between those two user accounts. But since they have the same name, many people think it is the same account (and I don’t blame them, how are you supposed to know it's not), which might lead to problems when changing password, since you then have changed password for just one of the accounts, not both. And when you try the new password on the “wrong” weblogic account, you won't be allowed access. You could end up spending the better part of the night to figure out what is wrong with your password.

The recommendation is to use a different username for the Node Manager account, I often use the name nodemanager.

​

To avoid any extra confusion, the operating system user which Node Manager runs as, has nothing to do with which user you use to log into Node Manager.

​

Verify connection between Administration Server and Node Manager

To verify that the Administration Server is able to communicate with Node Manager, check the Administration Console

​

If status is not Reachable, then username/password could be wrong, or SSL is enabled on one side and not the other, or if Node Manager is on a remote host, there could be network or firewall issues.

Create an encrypted credentials file

When using scripts to connect to Node Manager, you have to log in. Passing credentials in clear text is not a good idea, since it often also means that the credentials are available in clear text in script files or in a property file.

Node Manager support the use of an encrypted credential file. You should still protect this file, but it is better than clear text.

If you follow this advice, no one should need to log into Node Manager by providing the password by hand. Which means that there is no reason to have a password that is easy to remember, or easy to type. Make the password long and random, you will only type it a few times (to create the credentials file).

Creating the encrypted credential file is different in 11g and 12c. The 11g method still works in 12c, but this method is deprecated and might stop working.

11g method - source setDomainEnv.sh before running this command to get the paths correctly (everything on one line)

$ java weblogic.Admin -username nodemanager -userconfigfile /u01/app/oracle/config/nmUserFile -userkeyfile /u01/app/oracle/config/nmKeyFile STOREUSERCONFIG

12c method - source setDomainEnv.sh before running this command to get the paths correctly

$ java weblogic.WLST

wls:/offline> nmConnect( 'nodemanager','welcome1','localhost',5556,'mydomain', '/u01/ config/user_projects/domains/mydomain', 'plain')

wls:/mydomain/serverConfig> storeUserConfig( '/u01/app/oracle/config/nmUserFile','/u01/app/oracle/config/nmKeyFile', 'true')

Usage of the encrypted credentials file

As in the 12c method above, nmConnect usually takes a series of literal parameters. But when you are using an encrypted credential file, you don’t want to use literal parameters, at least not for username/password. The alternative nmConnect format is below:

nmConnect(userConfigFile=nmUserFile, userKeyFile=nmKeyFile, host=nmHost, port=nmPort, domainName=domain, domainDir=domainPath, nmType=nmType)

In this example, all parameters are replaced with variables. Of course, you could still have literal values, but using variables in scripts are often considered best practice.

Calling the WLST scripts with -loadProperties <path to properties file>, you have access to variables from the property file directly in your WLST scripts. An example properties file is below:

adminUrl=t3://wls12c.dev.sysco.no:7001

nmHost=wls12c.dev.sysco.no

nmPort=5556

nmUserFile=/u01/app/oracle/config/nmUserFile

nmKeyFile=/u01/app/oracle/config/nmKeyFile

nmType=plain

domain=mydomain

domainPath=/u01/config/user_projects/domains/mydomain

Change password

Even though very few needs to know the Node Manager password if you use the encrypted credential file, you should still change the Node Manager password from time to time.

The easiest and recommended way is to use the Administration Console to change the password.

It is possible, but not recommended, to change the password directly in nm_password.properties. If you do that, you have to restart Node Manager after changing the password. If you change the username and/or password from the Administration Console, you do not have to restart Node Manager.

Remember, after you have changed the password, you have to regenerate the nmUserFile with the new password (just follow the instructions for creating a new credential file).

http://docs.oracle.com/cloud/latest/fmw121300/NODEM/java_nodemgr.htm#NODEM161

How to add a new Node Managers (hosts)

The recommended way to add a new host, is to use the pack command on the domain on an existing server and do an unpack on the new server.

It is also possible to use the WLST command nmEnroll() to add the new host to the domain.

Some important files

nodemanager.properties

nodemanager.properties contains most of the configuration for Node Manager. The file is created first time Node Manager is started.

StartScriptEnabled=true tells Node Manager to use the start script to start WebLogic. By default, startWeblogic.sh (or .cmd) is used, unless StartScriptName is set to something else.

Be careful about StopScript, it might not be what you think. It is not the opposite of StartScript, a script used to stop the WebLogic instance. It is a cleanup script, which will be called after the WebLogic instance has stopped.

ListenAddress and ListenPort are important when running more than one Node Manager on a server. Every instance of Node Manager needs an unique ListenAddress:ListenPort combination. If everyone tries to use the default, only the first instance to start will succeed, the rest will fail.

SecureListener is default set to true, which means Node Manager can only be access by using SSL. If you do not have enabled SSL on the Administration Server, you must set SecureListener=false (and restart Node Manager if it is running).

If you think using SSL is a good idea (which it is), then you should replace the DemoTrust certificates with new ones (every installation are shipped the same DemoTrust certificates, so they are not secure at all).

Be aware that several property names are deprecated, but still works in 12.1.3, and that some of the new ones introduced in 12.1.3 is not yet supported in WLST offline mode and not yet supported by pack/unpack. Examples of deprecated property names are StartScriptEnabled and StartScriptName, from 12.1.3 use weblogic.StartScriptEnabled and weblogic.StartScriptName instead.

Not all possible properties are in the nodemanager.properties file, for a complete list of Node Manager properties, see the documentation:

http://docs.oracle.com/middleware/1213/wls/NODEM/java_nodemgr.htm#NODEM178

nm_password.properties

Encrypted Node Manager username and password file. Used to authenticate connection between Node Manager and clients (i.e., Administration Server, when Administration Server connects to Node Manager to start or stop a Managed Server, and when a WLST script connects to Node Manager to do some tasks, i.e., start or stop a Managed Server).

The Configuration Wizard puts the Node Manager username and password for the initial connection into nm_password.properties on the local server. To get it distributed to other servers, you have to use the WLST command nmEnroll(). nmEnroll() will either create a new nm_password.properties file, it if does not exists, or update an exisiting nm_password.properties file.

If you change the nm_password.properties manually, which is not recommended by the way, you have to restart Node Manager. If you change the username and/or password from the Administration Console, you do not have to restart Node Manager.

nodemanager.domains

10.3.6

domain-name=domain-directory

12.1.3

domainName=primaryDomainPath;alternateDomainPath1;alternateDomainPath2

<servername>.out logfiles

Generated by Node Manager, located together with the Administration Server or Managed Servers .log files.

No daily / by size rotation (at least not in 11g), so this file could be big. Rotated when the server restarts.

Summary

Node Manager is a very useful utility, and although you do not have to use Node Manager, it might do the operation of WebLogic somewhat easier.

Best practices for starting and stopping WebLogic instances are using WLST scripts together with Node Manager, and some high availability functions, like whole server migration, requires Node Manager.

Node Manager is not very difficult, but like any other software, it helps to know a little about it. I hope that this article has given you some more information of Node Manager, and at least has given you some new pointers to start searching from next time you encounter strange problems while starting or stopping your WebLogic servers.

Secure weblogic with Connection Filters ( Hardening and Kick out unknow users)

Connection Filters allow you to deny access to UNWANTED Connections at the network level. Connection Filters can be used to protect server resources on individual servers, server clusters, or an entire internal network or Intranet. Connection filters are particularly useful when using the Administration port. Depending on your network firewall configuration, you may be able to use a connection filter to further restrict administration access. A typical use might be to restrict access to the Administration port to only the servers and machines in the domain.
WebLogic Server provides a default connection filter called weblogic.security.net.ConnectionFilterImpl. This default connection filter accepts all incoming connections except the ones defined in your Administration Console as a Connection filter rule. You can write your own custom connection filter by implementing the classes in the weblogic.security.net package. This post will demonstrate the steps to configure Weblogic Connection Filters.



 

1. If you have not already done so, in the Change Center of the Administration Console, click Lock & Edit.
   
   
    
2. In the left pane, click on the domain name.
   
   
    
3. In the right pane, select the Security -> Filter tab.
   
   
    
4. Click the Connection Logger Enabled attribute to enable the logging of accepted connections.
   
   
    
5. Specify the Connection Filter to be used with the domain. This example uses the default connection filter weblogic.security.net.ConnectionFilterImpl
   
6. Specify the Connection Filter Rules. Make sure that you use the syntax below while writing the rules.
   
   Syntax: target localAddress localPort action protocols
   
   where
   • target specifies the client hosts that can connect to the Weblogic Server instance
   • localAddress defines the host address of the Weblogic Server instance to which any client can connect. If you specify an asterisk (*), the match returns all local IP addresses.
   • localPort defines the port on which the WebLogic Server instance is listening to which any client can connect. If you specify an asterisk (*), the match returns all available ports on the server.
   • action specifies the action to perform. This value must be only allow or deny.
   • protocols specifies the list of protocol names to match. The following protocols may be specified: http, https, t3, t3s, giop, giops, dcom, ftp, ldap. If no protocol is defined, all protocols will match a rule.
   
   
    
7. Click Save and in the Change Center of the Administration Console, click Activate Changes.
   
   
   Once saved an entry is made in the config.xml file for these settings as below:
   <connection-filter>weblogic.security.net.ConnectionFilterImpl</connection-filter>
   <connection-filter-rule>10.1.xx.xx 10.1.xx.xx 7001 deny http</connection-filter-rule>
   <connection-logger-enabled>true</connection-logger-enabled>
   
    
8. Restart all the WebLogic Server instances for changes to take effect.
   
   
    
9. Test the Connection Filter rules: Try to access the WebLogic Server instance with the port from the target server mentioned in the rule. This connection will be rejected because the rule says to deny the connection.
   ####<May 24, 2011 1:34:50 PM MDT> <Notice> <Socket> <NAND-S77> <AdminServer> <ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1306265690809> <BEA-000445> <Connection rejected, filter blocked Socket[addr=10.1.xx.xx,port=48316,localport=7001], weblogic.security.net.FilterException: [Security:090220]rule 1>
   
   Try to access the WebLogic Server instance with the port from the target server which is not mentioned in the rule. This connection will be accepted because it is not denied by the rule:
   ####<May 24, 2011 1:34:23 PM MDT> <Info> <Socket> <NAND-S77> <AdminServer> <ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1306265663662> <BEA-000431> <Accepted Connection: Filtering is set to: "true" Remote Address: "/0:0:0:0:0:0:0:1" Remote Port: "51655" Local Address: "0.0.0.0/0.0.0.0" Local Port: "7001" Protocol: "http">

Further reading on Connection Filter rules:

http://download.oracle.com/docs/cd/E13222_01/wls/docs81/security/con_filtr.html#1029299

WebLogic 12c (12.1.3) free training on-demand at Oracle Learning Library

Get all WebLogic trainings here.

• Oracle WebLogic Server 12c: Creating a Basic Cluster
• Oracle WebLogic Server 12c: Using the WebLogic Scripting Tool
• Oracle WebLogic Server 12.1.3: Using Fusion Middleware Control to Manage WebLogic Server
• Oracle WebLogic Server 12.1.3: Using REST Services to Manage WebLogic Server
• Oracle WebLogic Server 12.1.3: Using Whole Server Migration with Dynamic Clusters Webcast
• Oracle WebLogic Server 12c: Installing WebLogic Server and Creating a Domain
• Oracle WebLogic Server 12c: Configuring Managed Servers
• Oracle WebLogic Server 12c: Deploying and Undeploying Applications
• Oracle WebLogic Server 12c: Configuring a JDBC Data Source
• 3 - Oracle WebLogic Server 12c (12.1.3): Using Fusion Middleware Control to Manage WebLogic Server
• 2 - Oracle WebLogic Server 12c (12.1.3): Using REST Services to Manage WebLogic Server
• 1 - Oracle WebLogic Server 12c (12.1.3): Using Whole Server Migration with Dynamic Clusters
• WebLogic Server 12c (12.1.3) New Features
• Upgrading a Domain Using the WebLogic Server 12.1.3 Reconfiguration Wizard

Oracle Weblogic Server 12c: Install,Configure,Deploy Java EE Applications

Oracle Weblogic Server 12c:  Install and Configure

Oracle Weblogic Server 12c: Deploy Java EE Applications

We are going to Learn Below

Lesson1  : Weblogic Application Concepts
Lesson2  : Data Source Fundamentals
Lesson3  : JMS Fundamentals
Lesson4  : Deployment Techniques

JVM Tuning CheatSheet

JVM Tuning CheatSheet

This diagram is designed for the Sun hotspot

In fact, many JDK1.6 users need no tuning at all – the JVM picks good defaults and ergonomics does a decent job. Follow this only if the default behavior is not good enough (for instance, frequent garbage collections, low throughput, long GC pauses, etc).

Can be broadly categorized in to 4 sections

1.     GC (Garbage  Collection

The garbage collection algorithm is one of the two mandatory tunables for java performance tuning. Start with UseParallelOldGC. If GC pauses are not acceptable, switch to UseConcMarkSweepGC (prioritizes low application pause times at the cost of raw application throughput). Specify parameter ParallelGCThreads to limit GC threads (yes limit, the default is usually too high for multiple Weblogic servers sharing a large, multi-core machine). Recommendations for values and other flags will be covered later

2.     Tuning the Heap

This is the other mandatory tunable. I’m using ‘heap’ as an umbrella term for all Java memory spaces. Technically, Perm and Stack are not part of the java heap in Sun hotspot. Required flags in my tuning exercise are total heap size (Xmx, Xms), young generation size (Xmn) and permanent generation size (PermSize, MaxPermSize). Xss tuning is optional. I only use it when tuning on a 32-bit heap-constrained JVM; reducing Xss only to squeeze memory out from native space so more is available for Xmx. In any case, never set Xss below 128k for Fusion Middleware (default is usually 512k to 1m depending on OS)

3.      GC Logging

GC logging is mandatory only for the duration of the tuning exercise itself. However, due to its low overhead (typically only one line written per collection, which itself is relatively infrequent), it is highly recommended for production as well. Otherwise, you will not be able to make an educated tuning decision if/when things don’t work as expected.

4.     Perfomance(Optional)

These are only used for fine tuning when performance is the driver for the tuning exercise. Even then, try these only after GC and heap are well tuned to begin with

 

Requirement that warrants JVM tuning in production Oracle Fusion Middleware is not performance, rather unacceptable GC pauses. The cultprit almost always is a Full GC that causes long application pause. Symptoms include temporarily unresponsive servers, client session timeouts, etc. If you’re capturing GC logs using the flags in the diagram, a search for “Full GC” will show how many, how frequent and how long Full GC’s took. Following the tunables in the diagram above, this is how you can solve the problem