标签：fast eth set default ant nis debugging using logging
Access Cloud Manager:
Access Mongo Support
https://www.percona.com/blog/2016/08/12/tuning-linux-for-mongodb/
ALSO:
Copy collections across MongoDB databases and servers
NOTE:
In AWS we started using Mongo3.0. This created a problem for authentication from clients not supporting the new authentication mechanism (meteor can‘t connect).
Therefore, if using 3.0 we must downgrade the schema
// Use MongoDB 2.6 auth schema (default auth mechanism: MONGODB-CR)use admin;db.system.version.save({ "_id" : "authSchema", "currentVersion" : 3 }) |
Due to vmware (veeam) backups, it can hang our servers for a bit. We tried to increase the heartbeat by 10 seconds to prevent a mongo switch. Here is how you increase the timeout for > 3.2 mongo
cfg = rs.config() cfg.settings.electionTimeoutMillis=20000 #default is 10000 rs.reconfig(cfg)
Per Shiming‘s request, here is a short instruction how oplog can be enabled on stand-alone Mongo system: http://tuttlem.github.io/2014/06/13/how-to-setup-an-oplog-on-a-single-mongodb-instance.html This is year 2014 so it may not be very fresh but I hope it still works. Here is another reference: http://stackoverflow.com/questions/20487002/oplog-enable-on-standalone-mongod-not-for-replicaset. Accepted answer is outdated but take a look at comments and second answer. I believe it goes in line with the first link above. -Sergey
https://docs.mongodb.org/manual/tutorial/enable-authentication/
1) Login to the PRIMARY replica and create users.
# FULL root privilageuse admindb.createUser({user: "root", pwd: "root", roles: ["root"] } )# Create admin user for sxa databaseuse sxadb.createUser({user: "sxaAdmin", pwd: "sxaAdmin", roles: [ { role: "userAdmin", db: "sxa" }, { role: "readWrite", db: "sxa" } ] } )# readonly user for sxa dbuse sxadb.createUser({user: "readonlysxa", pwd: "readonlysxa", roles: [ { role: "read", db:"sxa" } ] } )# readonly user for anythinguse admindb.createUser({user: "readonlyadmin", pwd: "readonlyadmin", roles: [ { role: "readAnyDatabase", db:"admin" } ] } ) |
2) We add additional security. Use keyFile authentication for Intra-Replica communication (amongst themselves). Create a keyfile.
Reference: Mongo KeyFile Instructions
http://docs.mongodb.org/manual/tutorial/generate-key-file/
openssl rand -base64 741 > /etc/mongodb-keyfilechmod 600 /etc/mongodb-keyfilechown mongodb:mongodb /etc/mongodb-keyfile |
3) Copy this /etc/mongodb-keyfile to every replica instance. Don‘t forget to chmod and chown as above or mongod won‘t start.
4) Edit the /etc/mongod.conf file on every replica. Enable these lines
vi /etc/mongod.conf# Enable user authenticationauth=true#Enable keyFile for intra-replica authenticationkeyFile=/etc/mongodb-keyfile |
PAUSE: We‘re at the point where we want this all to take effect. We can do this without downtime by keeping a PRIMARY replica up at all times.
Reference: Mongo Maintenance Procedure
http://docs.mongodb.org/manual/tutorial/perform-maintence-on-replica-set-members/
5) Now we need to restart the replicas for these changes to take effect. Begin by logging into a SECONDARY replica and restart the mongod.
restart mongod |
6) Continue for all secondaries
7) Last, login to the PRIMARY. Tell it to give up mastership and become a SECONDARY
rs.stepDown() |
8) Now we‘re ok to restart mongod on the last replica without downtime.
restart mongod |
10) Extra Cred: The mongo URL that is needed for other applications such as meteor is in a very particular format when you use authentication.
MONGO_URL="mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]"Example:MONGO=URL="mongodb://sxaAdmin:sxaAdmin@ec2-52-26-46-185.us-west-2.compute.amazonaws.com:27017,ec2-52-24-148-47.us-west-2.compute.amazonaws.com:27017,ec2-52-26-60-142.us-west-2.compute.amazonaws.com:27017/sxa?replicaSet=mongodb&readPreference=primaryPreferred" |
use admindb.getUsers() use sxadb.getUsers()db.dropUser("sxaAdmin") use admindb.dropAllUsers() #! |
use sxadb.grantRolesToUser("sxaAdmin", [{ role: "readWrite", db: "local" } , { role: "userAdmin", db:"local" } ] ) |
use sxa
db.revokeRolesFromUser("sxaAdmin",[{ role: "readWrite", db: "local" } , { role: "userAdmin", db:"local" } ])
We use --auth if we want to enable logging.
screenmongod --port 37017 --dbpath /data --authctrl-A-D |
Support for the new authentication in MongoDB 3.0 has caused problems with meteor, and with tools like RoboMongo. Work around is to drop the schema down.
// Use MongoDB 2.6 auth schema (default auth mechanism: MONGODB-CR)use admin;db.system.version.save({ "_id" : "authSchema", "currentVersion" : 3 }) |
This may not be that simple. I found I needed to start mongod in stand-alone mode for every replica memeber with authentication disabled (remove the --auth), then set the authSchema value.
Run rs.status command in MongoDB shell. The following will be example of result
cmdctr:PRIMARY> rs.status(){ "set" : "cmdctr", "date" : ISODate("2015-06-08T01:00:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "199.71.142.249:27017", "health" : 1, "state" : 7, "stateStr" : "ARBITER", "uptime" : 3818114, "lastHeartbeat" : ISODate("2015-06-08T01:00:49Z"), "lastHeartbeatRecv" : ISODate("2015-06-08T01:00:49Z"), "pingMs" : 0 }, { "_id" : 1, "name" : "199.71.143.62:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 20854145, "optime" : Timestamp(1433725251, 3), "optimeDate" : ISODate("2015-06-08T01:00:51Z"), "electionTime" : Timestamp(1430122091, 1), "electionDate" : ISODate("2015-04-27T08:08:11Z"), "self" : true }, { "_id" : 2, "name" : "199.71.143.63:27017", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 1633250, "optime" : Timestamp(1433184037, 6), "optimeDate" : ISODate("2015-06-01T18:40:37Z"), "lastHeartbeat" : ISODate("2015-06-08T01:00:50Z"), "lastHeartbeatRecv" : ISODate("2015-06-08T01:00:50Z"), "pingMs" : 3, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 5574e47c:a2" } ], "ok" : 1} |
The result of db.printSlaveReplicationInfo() should show something like the following when sync is complete:
cmdctr:PRIMARY> db.printSlaveReplicationInfo()source: 199.71.143.63:27017 syncedTo: Thu May 28 2015 12:02:25 GMT-0700 (PDT) 0 secs (0 hrs) behind the primary |
Example: Mongo replica set is using the name "compass-dev-2" instead of "compass-dev-2.xx.com" causing DNS name resolution errors. We can rename it as follows. Note: it is also the current primary server.
mongo
rs.stepDown()
Login to new primary
cfg = rs.config()
cfg.members[0].host="compass-dev-2.xx.com:27017”
rs.reconfig(cfg)
http://docs.mongodb.org/manual/tutorial/perform-maintence-on-replica-set-members/
For command center deployments, you must disable cron jobs on JBOSS, and the CC+ backend during the mongo restore or it can conflict with the restore process.
We should even shut down these components as well during the restore.
mongodump --out <backup_file>Example:This dumps the "sxa" database from the mongo server 199.71.142.151. It can be run anywhere, though it is faster to run locally and then tar the results and scp it over to your destination. mongodump -h 199.71.142.151 --out /tmp/goldbackup --db sxa Example:Note: You can restore to an alternate DB if desired.mongorestore --drop -d new_db_name mongodump/old_db_name mongorestore --drop --host 199.71.142.151 -d sxa gold-backup-080216/sxa |
Note: We copy the backup NFS to the local disk. It seemed to me I could just restore directly from the NFS location and save time and a step, but in one case restoring from NFS failed on me, and I thought it might have been due to some NFS instability. I did not pursue that problem, I worked around it by copying the data to the local disk before the restore.
Example: Restore prod DB to gold system. #Identify gold system primary. ssh -p 1035 root@199.71.142.151 #Ensure there is enough local disk to hold the backup. Copy your backup over to the local disk. # Recommend you use screen in case you lose the ssh connection screen mkdir /root/restores cp -frp /backup/command-center-db/mongo/2017-02-21_020001/199.71.143.62 /root/restores cd /root/restores # WARNING: ENSURE THE HOST IP IS YOUR GOLD SYSTEM AND NOT PRODUCTION. It is possible to overwrite production remotely if you accidentally provide a production mongo IP. mongorestore --drop --host 199.71.142.151 --db sxa dump/sxa
Reference: http://docs.mongodb.org/manual/tutorial/backup-with-mongodump/
Have not watched this yet, but should be good: https://www.mongodb.com/presentations/webinar-backups-disaster-recovery
Script provided by David Jansen
For other applications, such as meteor.
%vmlnx-djansen<225> /opt/script/devops/mongo/dump_system_profile.py -husage: dump query info from system.profile with optional date query [-h] [--mongo_url MONGO_URL] [--after AFTER] [--before BEFORE] optional arguments: -h, --help show this help message and exit --mongo_url MONGO_URL Mongo Url (default mongodb://localhost:27017/sxa) --after AFTER DateTime of format like "2016-04-12T08:00:00Z" --before BEFORE DateTime of format like "2016-04-12T09:00:00Z" %vmlnx-djansen<226> /opt/script/devops/mongo/dump_system_profile.py --after 2016-04-13T16Z --before 2016-04-13T16:01Zbefore : 2016-04-13T16:01Z ==> 2016-04-13 16:01:00+00:00after : 2016-04-13T16Z ==> 2016-04-13 16:00:00+00:00using query: "{"ts": {"$lte": {"$date": 1460563260000}, "$gte": {"$date": 1460563200000}}}"---------------------------------- |
Installing mongodb-org-tools for 3.0 instructions are located here:
https://docs.mongodb.org/v3.0/tutorial/install-mongodb-on-ubuntu/
https://docs.mongodb.org/v2.6/tutorial/enable-authentication/
1) Launch 2 brand new centos7 servers.
2) Install mongo 2.6.5 on them ahead of time. Before the migration, like 1-2 hours before, add them to the replica set so the data syncs. We could set priority to zero to prevent these from becoming a primary, but this is not tested yet, so I‘m not going to include that step at this point.
3) Once all replicas are synced then you can begin upgrading the standbys.
4) Switch IP for meteor1 and meteor2 to new mongo servers mongo1 and mongo2
5) Assign NEW IPs for meteor1 and meteor2 and update nginx to reflect this.
https://docs.mongodb.com/v2.6/tutorial/install-mongodb-on-red-hat/
[root@GOLD-MONGO32-2-DEV ~]# vi /etc/yum.repos.d/mongodb.repo [mongodb] name=MongoDB Repository baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/ gpgcheck=0 enabled=1 [root@GOLD-MONGO32-2-DEV ~]# sudo yum install -y mongodb-org-2.6.5 mongodb-org-server-2.6.5 mongodb-org-shell-2.6.5 mongodb-org-mongos-2.6.5 mongodb-org-tools-2.6.5
Had to change this to use sudo otherwise if I try to run as mongod with supervisord the process limits are not set to our specs.
[program:mongo] #command=mongod --smallfiles -f /etc/mongod.conf command=sudo -u mongod bash -c ‘mongod --smallfiles -f /etc/mongod.conf‘ numprocs=1 #directory=/var/lib/mongo stdout_logfile=/var/log/mongodb/mongod@mongo.log stderr_logfile=/var/log/mongodb/mongod@mongo.err.log autostart=true autorestart=true stopasgroup=true user=root ~
Create path:
mkdir -p /home/mongo/mongodb chmod 777 /home/mongo/mongodb chown -R mongod:mongod /home/mongo chown -R mongod:mongod /home/mongo/mongodb
# mongod.conf #where to log logpath=/var/log/mongodb/mongod.log logappend=true # fork and run in background #fork=true #port=27017 dbpath=/home/mongo/mongodb # location of pidfile pidfilepath=/home/mongo/mongod.pid # Listen to local interface only. Comment out to listen on all interfaces. bind_ip=0.0.0.0 # Disables write-ahead journaling # nojournal=true # Enables periodic logging of CPU utilization and I/O wait #cpu=true # Turn on/off security. Off is currently the default #noauth=true #auth=true # Verbose logging output. #verbose=true # Inspect all client data for validity on receipt (useful for # developing drivers) #objcheck=true # Enable db quota management #quota=true # Set oplogging level where n is # 0=off (default) # 1=W # 2=R # 3=both # 7=W+some reads #diaglog=0 # Ignore query hints #nohints=true # Enable the HTTP interface (Defaults to port 28017). httpinterface=false # Turns off server-side scripting. This will result in greatly limited # functionality #noscripting=true # Turns off table scans. Any query that would do a table scan fails. #notablescan=true # Disable data file preallocation. #noprealloc=true # Specify .ns file size for new databases. # nssize=<size> # Replication Options # in replicated mongo databases, specify the replica set name here replSet=gold # maximum size in megabytes for replication operation log #oplogSize=1024 # path to a key file storing authentication info for connections # between replica set members #keyFile=/path/to/keyfile
Need to disable need for tty, or supervisor process won‘t start.
vi /etc/sudoers
# #Defaults requiretty
supervisorctl update mongo
Login to the primary server (assuming you know how to determine this)
> rs.add("192.35.85.16:27017")
Note, you can use this format: <replica>host:port,host:port,host:port, and you don‘t need to determine the primary node.
We use mongoexport to get AA organization list. As following mongoexport -h 199.71.143.62 -d sxa -c sxacms_organizations -f shortName,name,orgId,email -o $MONGOEXPORT --csv
/home/ubuntu/sxadp-config-file
Routing Rules (Optional) - CURRENTLY NOT WORKING, and not sure if this is a good idea because two IPs would be going to the same mongo service.
iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j DNAT --to 192.168.1.2:8080 iptables -A FORWARD -p tcp -d 192.168.1.2 --dport 8080 -j ACCEPT
This is required on the path to 3.2. You can‘t go from 2.6 to 3.2.
https://docs.mongodb.com/manual/release-notes/3.0-upgrade/
New centos servers must have this mtu setting or aws storm will get hung up.
sudo ip link set dev <interface> mtu 1300
Must be set permanently. Not discussed here, it is located if you search the wiki for mtu.
https://docs.mongodb.com/v3.0/tutorial/install-mongodb-on-red-hat/
vi /etc/yum.repos.d/mongodb-org-3.0.repo
[mongodb-org-3.0] name=MongoDB Repository baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.0/x86_64/ gpgcheck=0 enabled=1
supervisorctl stop mongo sudo yum install -y mongodb-org-3.0.12 mongodb-org-server-3.0.12 mongodb-org-shell-3.0.12 mongodb-org-mongos-3.0.12 mongodb-org-tools-3.0.12
New centos servers must have this mtu setting or aws storm will get hung up.
sudo ip link set dev <interface> mtu 1300
Must be set permanently. Not discussed here, it is located if you search the wiki for mtu.
Create file
vi /etc/yum.repos.d/mongodb-org-3.2.repo [mongodb-org-3.2] name=MongoDB Repository baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.2/x86_64/ gpgcheck=1 enabled=1 gpgkey=https://www.mongodb.org/static/pgp/server-3.2.asc
Do this for all the secondaries, then step-down the primary as the last step and upgrade that one last.
supervisorctl stop mongo sudo yum install -y mongodb-org-3.2.8 mongodb-org-server-3.2.8 mongodb-org-shell-3.2.8 mongodb-org-mongos-3.2.8 mongodb-org-tools-3.2.8 supervisorctl start mongo
Reference:https://docs.mongodb.com/manual/release-notes/3.2-upgrade/
Reference: https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/
Begin with a standby member of the replica set.
Add repo key
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927
echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list apt-get update
Install specific version of mongo. This happens to be the latest stable as well, but might be useful to make sure we‘re explict in all our upgrades.
sudo apt-get install -y mongodb-org=3.2.8 mongodb-org-server=3.2.8 mongodb-org-shell=3.2.8 mongodb-org-mongos=3.2.8 mongodb-org-tools=3.2.8
Finally, let‘s do the primary.
Login to the primary
rs.stepDown()
Wait until this system no longer reports itself as Primary. Keep using rs.status() to check. When it finally is a standby then proceed.
Perform the usual upgrade of the binaries like you did before.
echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
Install specific version of mongo. This happens to be the latest stable as well, but might be useful to make sure we‘re explict in all our upgrades.
sudo apt-get install -y mongodb-org=3.2.8 mongodb-org-server=3.2.8 mongodb-org-shell=3.2.8 mongodb-org-mongos=3.2.8 mongodb-org-tools=3.2.8
All servers are upgraded to 3.2. Lastly, connect to the current primary server and run this.
cfg = rs.conf(); cfg.protocolVersion=1; rs.reconfig(cfg);
DONE
On ubuntu you can pin the versions down to prevent accidental upgrade. Not sure on centos.
echo "mongodb-org hold" | sudo dpkg --set-selections echo "mongodb-org-server hold" | sudo dpkg --set-selections echo "mongodb-org-shell hold" | sudo dpkg --set-selections echo "mongodb-org-mongos hold" | sudo dpkg --set-selections echo "mongodb-org-tools hold" | sudo dpkg --set-selections
https://docs.mongodb.com/manual/release-notes/3.2-downgrade/#downgrade-replica-set
Note, for downgrade, you need to set a variable in the replica set called protocolVersion first, Then you can do the usual downgrade of each standby.
sudo apt-get install mongodb-org=3.0.1 mongodb-org-server=3.0.1 mongodb-org-shell=3.0.1 mongodb-org-mongos=3.0.1 mongodb-org-tools=3.0.1
1) Restart storm topology
2) Possibly need to restart the collector if FA+/RT doesn‘t come back entirely. Do that after the storm topo has been restarted
reference: https://docs.mongodb.com/manual/tutorial/install-mongodb-on-red-hat/
Fix for centos7 and automation agent
To the best of my knowledge, Fedora 20 is equivalent to RHEL 7 which is not yet supported by MongoDB. There is work planned for this latter OS, but I cannot guarantee a timetable for that release. In the meantime, I believe it is possible to work around the issue with the following: cd /lib64 sudo ln -s libsasl2.so.3.0.0 libsasl2.so.2 Hopefully this helps, and allows you to start the agent.
标签：fast eth set default ant nis debugging using logging
原文地址：https://www.cnblogs.com/tben/p/12745573.html
