about Node Manager
Node Manager is an important utility when
running WebLogic in a production environment. Most people pay little
attention to the Node Manager, but when problem arises, it is a little
bit late to figure out what Node Manager is and what it does.
This article is not a full introduction to
Node Manager, but will describe some best practices for common real life
situations and issues, and share some tips and tricks gathered over the
years.
If you would like a full introduction to Node
Manager first, with all the nice diagrams showing how everything is
connected, then you should look at the official documentation, which by
the way are getting pretty good, before coming back to read the rest of
this article -
http://docs.oracle.com/middleware/1213/wls/NODEM/overview.htm#NODEM112
So, what is Node Manager?
Node Manager is a utility used to start, stop
and restart WebLogic and Coherence servers on the host Node Manager is
installed. Since you could connect to Node Manager from a remote server,
you will use Node Manager on the remote server as a tool to remotely
start and stop a WebLogic instance.
There are two version of Node Manager, the Java
version and the script based version. Unless you have very specific
requirements, use the Java version, which also happens to be the one
that most people think of when they talk about Node Manager. This
article will be about the Java version, but if you want to know more
about the difference of the two versions, have a look at the official
documentation -
http://docs.oracle.com/middleware/1213/wls/NODEM/overview.htm#NODEM114
Per host vs per domain Node Manager
Up until WebLogic 12.1.2 the default was one
Node Manager per host. It was possible to do it otherwise, but it was
not documented very well.
From 12.1.2 the default is one Node Manager per
domain, and the configuration and logs are now located under
$DOMAIN_HOME/nodemanager instead of under
wlserver_10.3/common/nodemanager
It is still possible to have one Node Manager per host, for those who prefers that.
If you are using Node Manager for an earlier
release (before 12.1.2) or just want to stick with one Node Manager per
host, you should consider moving the configuration and log files away
from wlserver_10.3/common/nodemanger and have them outside the
$ORACLE_HOME. There are several reasons for that, one important reason
is to keep configuration files away from the binaries, so the
configuration is kept in case an upgrade overwrites everything under
wlserver_10.3/common/nodemanager. Another important reason is that you
want to have logfiles in a location that you monitor for disk space,
since logfiles tends to grow, and you might not have taken that into
consideration when deciding how much space you needed for the
installation.
One special case where you actually needed more
than one Node Manager for a host in 11g was when you followed the
Enterprise Deployment Guide (EDG) for SOA. In an enterprise setup, the
Administration Server is installed in a different $DOMAIN_HOME than the
managed servers. This makes it easier to start the Administration Server
on another host in case of a host failure (Administration Server domain
directory should be on a SAN or a shared disk).
The reason for using one Node Manager for the
Administration Server domain directory and one for the Managed Servers
domain directory is that the nodemanager.domains file in 11g could only can have one path per domain.
In 12c there is possible to have one primary domain path and several alternate domain paths for each domain in nodemanager.domains,
so it should be enough with one Node Manager per domain from 12.1.3
http://docs.oracle.com/middleware/1213/wls/NODEM/java_nodemgr.htm#NODEM163
Since one Node Manager per domain could mean
several Node Manager instances on one host, every Node Manager instance
on that host must have a unique ip:port combination.
How to start Node Manager?
The simple answer is to run startNodeManager.sh from $DOMAIN_HOME/bin, but except for testing, you should not do that.
Node Manager should be started as a system
service when the server boots, so a startup script is needed. In the
WebLogic 12.1.3 documentation there are init.d and xinetd scripts
provided for Linux -
http://docs.oracle.com/middleware/1213/wls/NODEM/java_nodemgr.htm#NODEM247
For Windows there is the installNodeMgrSvc.cmd script included in the $DOMAIN_HOME/bin directory.
Node Manager will run as the user who started
it. When Node Manager starts a WebLogic instance, either an
Administration Server or a Managed Server, the WebLogic instance will
run as the same user as the user who runs Node Manager.
There is currently some flaws in the
documentation, both in the init.d script for Linux and in the
description of installNodeMgrSvc.cmd for Windows. Oracle has been
notified, and the documentation should be updated soon.
By using the init.d script from the
documentation, Node Manager will run as root, and by starting it as a
Windows service, it will run as System.
Running WebLogic as root/System could be a
security risk, and when you are going to patch WebLogic, you will most
likely do that as the user you installed WebLogic as. However, since
some files have changed ownership/rights to root/System, you might not
be able to patch without a lot of hassle.
On Linux there is a post-bind function in
WebLogic, so it is possible to configure WebLogic to change the user
from root to something less hazardous, but this is not possible in
Windows.
So, what should we do?
For Linux, make sure Node Manager is started as
a non-root user, by changing the line in the init.d file in the
documentation from
$PROGRAM_START >> $OUT_FILE 2>&1 &
to
su -c $NMUSER $PROGRAM_START >> $OUT_FILE 2>&1 &
The variable NMUSER could be declared somewhere between
#### BEGIN CUSTOM ENV
and
#### END CUSTOM ENV
as
NMUSER=oracle
Replace oracle with the user you want to run Node Manager and WebLogic.
For Windows you could go to services and
properties for the WebLogicNodeManager service. On the “Log On” tab,
select “This account” and select the user account that you want to run
Node Manager (and WebLogic) as.
Username and password
To log into Node Manager, you have to use a username and password. Unfortunately, the default username is weblogic, which also happens to be the default username for the WebLogic Administration server. However, these two weblogic
users are not the same, it is just a coincident that their names are
the same. There is no relation between those two user accounts. But
since they have the same name, many people think it is the same account
(and I don’t blame them, how are you supposed to know it's not), which
might lead to problems when changing password, since you then have
changed password for just one of the accounts, not both. And when you
try the new password on the “wrong” weblogic account,
you won't be allowed access. You could end up spending the better part
of the night to figure out what is wrong with your password.
The recommendation is to use a different username for the Node Manager account, I often use the name nodemanager.
To avoid any extra confusion, the operating
system user which Node Manager runs as, has nothing to do with which
user you use to log into Node Manager.
Verify connection between Administration Server and Node Manager
To verify that the Administration Server is able to communicate with Node Manager, check the Administration Console
If status is not Reachable, then
username/password could be wrong, or SSL is enabled on one side and not
the other, or if Node Manager is on a remote host, there could be
network or firewall issues.
Create an encrypted credentials file
When using scripts to connect to Node Manager,
you have to log in. Passing credentials in clear text is not a good
idea, since it often also means that the credentials are available in
clear text in script files or in a property file.
Node Manager support the use of an encrypted
credential file. You should still protect this file, but it is better
than clear text.
If you follow this advice, no one should need
to log into Node Manager by providing the password by hand. Which means
that there is no reason to have a password that is easy to remember, or
easy to type. Make the password long and random, you will only type it a
few times (to create the credentials file).
Creating the encrypted credential file is
different in 11g and 12c. The 11g method still works in 12c, but this
method is deprecated and might stop working.
11g method - source setDomainEnv.sh before running this command to get the paths correctly (everything on one line)
$ java
weblogic.Admin -username nodemanager -userconfigfile
/u01/app/oracle/config/nmUserFile -userkeyfile
/u01/app/oracle/config/nmKeyFile STOREUSERCONFIG
12c method - source setDomainEnv.sh before running this command to get the paths correctly
$ java weblogic.WLST
wls:/offline>
nmConnect( 'nodemanager','welcome1','localhost',5556,'mydomain', '/u01/
config/user_projects/domains/mydomain', 'plain')
wls:/mydomain/serverConfig> storeUserConfig( '/u01/app/oracle/config/nmUserFile','/u01/app/oracle/config/nmKeyFile', 'true')
Usage of the encrypted credentials file
As in the 12c method above, nmConnect usually
takes a series of literal parameters. But when you are using an
encrypted credential file, you don’t want to use literal parameters, at
least not for username/password. The alternative nmConnect format is
below:
nmConnect(userConfigFile=nmUserFile,
userKeyFile=nmKeyFile, host=nmHost, port=nmPort, domainName=domain,
domainDir=domainPath, nmType=nmType)
In this example, all parameters are replaced
with variables. Of course, you could still have literal values, but
using variables in scripts are often considered best practice.
Calling the WLST scripts with -loadProperties <path to properties file>, you have access to variables from the property file directly in your WLST scripts. An example properties file is below:
adminUrl=t3://wls12c.dev.sysco.no:7001
nmHost=wls12c.dev.sysco.no
nmPort=5556
nmUserFile=/u01/app/oracle/config/nmUserFile
nmKeyFile=/u01/app/oracle/config/nmKeyFile
nmType=plain
domain=mydomain
domainPath=/u01/config/user_projects/domains/mydomain
Change password
Even though very few needs to know the Node
Manager password if you use the encrypted credential file, you should
still change the Node Manager password from time to time.
The easiest and recommended way is to use the Administration Console to change the password.
It is possible, but not recommended, to change the password directly in nm_password.properties.
If you do that, you have to restart Node Manager after changing the
password. If you change the username and/or password from the
Administration Console, you do not have to restart Node Manager.
Remember, after you have changed the password,
you have to regenerate the nmUserFile with the new password (just follow
the instructions for creating a new credential file).
http://docs.oracle.com/cloud/latest/fmw121300/NODEM/java_nodemgr.htm#NODEM161
How to add a new Node Managers (hosts)
The recommended way to add a new host, is to use the pack command on the domain on an existing server and do an unpack on the new server.
It is also possible to use the WLST command nmEnroll() to add the new host to the domain.
Some important files
nodemanager.properties
nodemanager.properties contains most of the
configuration for Node Manager. The file is created first time Node
Manager is started.
StartScriptEnabled=true tells Node Manager to use the start script to start WebLogic. By default, startWeblogic.sh (or .cmd) is used, unless StartScriptName is set to something else.
Be careful about StopScript, it might not be what you think. It is not the opposite of StartScript,
a script used to stop the WebLogic instance. It is a cleanup script,
which will be called after the WebLogic instance has stopped.
ListenAddress and ListenPort are important when running more than one Node Manager on a server. Every instance of Node Manager needs an unique ListenAddress:ListenPort combination. If everyone tries to use the default, only the first instance to start will succeed, the rest will fail.
SecureListener
is default set to true, which means Node Manager can only be access by
using SSL. If you do not have enabled SSL on the Administration Server,
you must set SecureListener=false (and restart Node Manager if it is running).
If you think using SSL is a good idea (which it
is), then you should replace the DemoTrust certificates with new ones
(every installation are shipped the same DemoTrust certificates, so they
are not secure at all).
Be aware that several property names are
deprecated, but still works in 12.1.3, and that some of the new ones
introduced in 12.1.3 is not yet supported in WLST offline mode and not
yet supported by pack/unpack. Examples of deprecated property names are StartScriptEnabled and StartScriptName, from 12.1.3 use weblogic.StartScriptEnabled and weblogic.StartScriptName instead.
Not all possible properties are in the
nodemanager.properties file, for a complete list of Node Manager
properties, see the documentation:
nm_password.properties
Encrypted Node Manager username and password
file. Used to authenticate connection between Node Manager and clients
(i.e., Administration Server, when Administration Server connects to
Node Manager to start or stop a Managed Server, and when a WLST script
connects to Node Manager to do some tasks, i.e., start or stop a Managed
Server).
The Configuration Wizard puts the Node Manager
username and password for the initial connection into
nm_password.properties on the local server. To get it distributed to
other servers, you have to use the WLST command nmEnroll(). nmEnroll()
will either create a new nm_password.properties file, it if does not exists, or update an exisiting nm_password.properties file.
If you change the nm_password.properties
manually, which is not recommended by the way, you have to restart Node
Manager. If you change the username and/or password from the
Administration Console, you do not have to restart Node Manager.
nodemanager.domains
10.3.6
domain-name=domain-directory
12.1.3
domainName=primaryDomainPath;alternateDomainPath1;alternateDomainPath2
<servername>.out logfiles
Generated by Node Manager, located together with the Administration Server or Managed Servers .log files.
No daily / by size rotation (at least not in 11g), so this file could be big. Rotated when the server restarts.
Summary
Node Manager is a very useful utility, and
although you do not have to use Node Manager, it might do the operation
of WebLogic somewhat easier.
Best practices for starting and stopping
WebLogic instances are using WLST scripts together with Node Manager,
and some high availability functions, like whole server migration,
requires Node Manager.
Node Manager is not very difficult, but like
any other software, it helps to know a little about it. I hope that this
article has given you some more information of Node Manager, and at
least has given you some new pointers to start searching from next time
you encounter strange problems while starting or stopping your WebLogic
servers.
