...
Code Block |
---|
$ sudo apt-get install python-setuptools git libpq-dev postgresql python-dev postfix nginx-full uwsgi uwsgi-plugin-python libffi-dev $ sudo easy_install pip $ sudo pip install virtualenv $ sudo adduser --disabled-password --home /var/opt/norduni ni |
...
Download neo4j-community from http://neo4j.com/download/. NORDUnet and SUNET run 2.1.8. 2.3.2 has been tested and did not work as expected.
Code Block |
---|
$ tar xvfz neo4j-community-2.1.78-unix.tar.gz $ sudo mv neo4j-community-2.1.78 /var/opt/. $ sudo ln -s /var/opt/neo4j-community-2.1.78 /var/opt/neo4j-community $ cd /var/opt/neo4j-community $ sudo ./bin/neo4j-installer install |
...
Set password for database user and create a new database
Code Block |
---|
$ sudo -u postgres psql postgres \password postgres Write password Write password again Ctrl+D sudo -u postgres createdb norduni |
NORDUni repository
Get the NORDUni files.
Code Block |
---|
$ sudo -u ni -i
$ pwd
/var/opt/norduni
$ git clone git://git.nordu.net/norduni.git
|
Python environment
Make a virtual python environment.
Code Block |
---|
$ virtualenv norduni_environment
|
Making a virtual environment is also just a suggestion but it makes it easier to keep your system clean.
Python requirements
Install required python modules.
Code Block |
---|
$ . norduni_environment/bin/activate
$ pip install -r norduni/requirements.txt
|
Django settings
Change the django settings.
Code Block |
---|
$ cd norduni/src/niweb/niweb/
$ cp generic_settings.py settings.py
$ vi settings.py
|
Change at least the database settings.
Code Block |
---|
# Database settings
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'norduni',
'USER': 'postgres',
'PASSWORD': 'secret',
'HOST': 'localhost'
}
}
|
Check if your settings are ok.
template1=# CREATE USER ni with PASSWORD 'secret';
template1=# CREATE DATABASE norduni;
template1=# GRANT ALL PRIVILEGES ON DATABASE norduni to ni;
template1=# ALTER DATABASE norduni OWNER TO ni; # Allow user ni to drop and create for restoring
template1=# ALTER USER ni CREATEDB; # and development purposes
template1=# \q |
NORDUni repository
Get the NORDUni files.
Code Block |
---|
$ sudo -u ni -i
$ pwd
/var/opt/norduni
$ git clone git://git.nordu.net/norduni.git
|
Python environment
Make a virtual python environment.
Code Block |
---|
$ virtualenv norduni_environment
|
Making a virtual environment is also just a suggestion but it makes it easier to keep your system clean.
Python requirements
Install required python modules.
Code Block |
---|
$ . norduni_environment/bin/activate
$ pip install -r norduni/requirements/prod.txt
|
Django settings
Change the django settings.
Code Block |
---|
$ cd norduni/src/niweb/
$ cp dotenv .env
$ vi .env
|
The following settings need to be changed.
Code Block |
---|
REPORTS_TO=
DB_PASSWORD=
DEFAULT_FROM_EMAIL=
EMAIL_HOST=
SECRET_KEY= |
Check if your settings are ok.
Code Block |
---|
$ python manage.py syncdb
$ python manage.py migrate apps.noclook
$ python manage.py migrate actstream
$ python manage.py migrate tastypie
$ python manage.py collectstatic |
Code Block |
$ python manage.py syncdb $ python manage.py runserver 0.0.0.0:80 |
Now you should be able connect to your machines the machine with your browser on http://localhost:8000 and see the NOCLook app index page.
...
Deploying NOCLook
...
title | Work in progress |
---|
...
uwsgi
Create
Deploying NOCLook
Create a uwsgi configuration file.Configure nginx.
Code Block | ||||
---|---|---|---|---|
title | $ sudo vi /etc/ | nginxuwsgi/ | sitesapps-available/ | defaultserver { listen 80; root /opt/norduni/src/niweb; server_name ni.example.net; access_log /var/log/ni/noclook-access.log; error_log /var/log/ni/noclook-error.log; location /static/ { root /optnoclook.ini The following configuration should be a good start. [uwsgi] # Django-related settings plugins = python protocol = uwsgi # the base directory (full path) chdir = /var/opt/norduni/norduni/src/niweb/; # Django's wsgi file wsgi-file autoindex on; = /var/opt/norduni/norduni/src/niweb/niweb/wsgi.py env access_log off; = DJANGO_SETTINGS_MODULE=niweb.settings.prod # the virtualenv (full path) home expires 30d; } = /var/opt/norduni/norduni_environment # logging daemonize location / { = /var/log/uwsgi/app/noclook.log # process-related settings # master master proxy_pass_header Server; = true # maximum number proxy_set_header Host $http_host; of worker processes processes proxy_redirect off; = 5 #threads proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Scheme $scheme; = 2 max-requests = 5000 # the socket (use the full path to be safe socket proxy_connect_timeout 10; = 127.0.0.1:8001 # clear environment proxy_read_timeout 10; on exit vacuum proxy_pass http://localhost:8000/; } } |
Collecting and processing network data
= true |
Link the configuration in to the correct directory.
Code Block |
---|
sudo ln -s /etc/uwsgi/apps-available/noclook.ini /etc/uwsgi/apps-enabled/noclook.ini |
Make temp dir and log dir writable by the uwsgi user (www-data on ubuntu)
Code Block |
---|
sudo chown -R ni:www-data /tmp/django_cache
sudo chmod -R g+w /tmp/django_cache
sudo chown -R ni:www-data /var/opt/norduni/norduni/src/niweb/logs/
sudo chmod -R g+w /var/opt/norduni/norduni/src/niweb/logs/ |
nginx
Setup new dhparam file 2048 should suffice, but if you like you can go with 4096 instead:
Code Block |
---|
$ sudo openssl dhparam -out /etc/ssl/dhparams.pem 2048 |
Configure nginx.
Code Block | ||
---|---|---|
| ||
$ sudo vi /etc/nginx/sites-available/default
The following configuration should be a good start.
upstream django {
server 127.0.0.1:8001; # for a web port socket
}
server {
listen 80;
listen [::]:80;
server_name ni.nordu.net;
rewrite ^ https://$server_name$request_uri? permanent;
}
server {
listen 443;
listen [::]:443 default ipv6only=on; ## listen for ipv6
ssl on;
ssl_certificate /etc/ssl/ni_nordu_net.crt;
ssl_certificate_key /etc/ssl/ni_nordu_net.key;
# https://cipherli.st
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_session_cache shared:SSL:10m;
ssl_ecdh_curve secp384r1;
ssl_dhparam /etc/ssl/dhparams.pem;
server_name ni.nordu.net;
location /static/ {
alias /var/opt/norduni/norduni/src/niweb/niweb/static/;
autoindex on;
access_log off;
expires 30d;
}
location / {
include /etc/nginx/uwsgi_params;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect off;
uwsgi_pass django;
}
}
|
SAML SP
If you want to set up NOCLook as a SAML SP you need to install the following packages and Python modules.
Code Block |
---|
$ sudo apt-get install libffi-dev xmlsec1
$ sudo -u ni -i
$ . norduni_environment/bin/activate
$ pip install djangosaml2 |
You then need to uncomment the lines in settings.py that imports and sets up djangosaml2. You also have to create a pysaml2 configuration.
All this is best described in the documentation at https://pypi.python.org/pypi/djangosaml2.
Local saml metadata
To speed up login you can use local metadata. This metadata still needs to be updated and verified, and for that you can use https://github.com/NORDUnet/metadata-updater
You need to configure djangosaml2 to use local metadata, and you will have to add the meta-dataupdater to cron, preferably by running crontab -e as the ni user. Once an hour is reasonable, once a day can be ok, once a week might be tiresome when the cert expires.
Collecting and processing network data
To insert data you To insert data you need to stop any python process that is using the Neo4j database. We hope to get the option to load more database instances in read-only mode in a near future then this could be avoided.
NORDUnet has a GIT repository called nistore and it is cloned to /var/opt/norduni/nistore/.
To start have a look at the NERDS README then clone the NERDS project.
Code Block |
---|
cd /var/opt/norduni/ mkdir tools cd tools git clone https://github.com/fredrikt/nerds.git |
...
Code Block | ||
---|---|---|
| ||
{ "host": { "csv_producer": { "address": "", "area": "", { "cityhost": "", { "comment"csv_producer": "", { "countryaddress": "", "floorarea": "", "latitudecity": "", "longitudecomment": "", "meta_typecountry": "", "namefloor": "", "node_typelatitude": "", "owner_idlongitude": "", "postcodemeta_type": "", "responsible_forname": "", "roomnode_type": "", "siteowner_typeid": "", "telenor_subscription_idpostcode": "", }, "nameresponsible_for": "", "version": 1 } } |
The consumer script should only be run once as it does not update the sites, only creates new.
The JSON file directory is then inserted in to the database using noclook_site_csv_consumer.py.
Change the path at the top of the script to be able to import norduni_client.py.
Then run:
Code Block |
---|
python noclook_site_csv_consumer.py -D /path/to/site_files/json
|
Daily database update
The producers are run with a cron job and the script noclook_consumer.py is used to run the three inserting/updating scripts (noclook_juniper_consumer.py, noclook_alcatel_consumer.py and noclook_nmap_consumer.py).
Change the path at the top of the script to be able to import norduni_client.py.
Code Block |
---|
[data]
juniper_conf = /path/to/juniper/json
nmap_services = /path/to/nmap/json
alcatel_isis = /path/to/alcate/json
noclook = #Used for loading backup.
|
Then run:
Code Block |
---|
python noclook_consumer.py -C template.conf -I
|
Setting up a local/development NOCLook
"room": "",
"site_type": "",
"telenor_subscription_id": ""
},
"name": "",
"version": 1
}
}
|
The consumer script should only be run once as it does not update the sites, only creates new.
The JSON file directory is then inserted in to the database using noclook_site_csv_consumer.py.
Change the path at the top of the script to be able to import norduni_client.py.
Then run:
Code Block |
---|
python noclook_site_csv_consumer.py -D /path/to/site_files/json
|
Daily database update
The producers are run with a cron job and the script noclook_consumer.py is used to run the three inserting/updating scripts (noclook_juniper_consumer.py, noclook_alcatel_consumer.py and noclook_nmap_consumer.py).
Change the path at the top of the script to be able to import norduni_client.py.
Code Block |
---|
[data]
juniper_conf = /path/to/juniper/json
nmap_services = /path/to/nmap/json
alcatel_isis = /path/to/alcate/json
noclook = #Used for loading backup.
|
Then run:
Code Block |
---|
python noclook_consumer.py -C template.conf -I
|
Setting up a local/development NOCLook
Code Block |
---|
# Clone a convenience repo
$ git clone https://github.com/NORDUnet/norduni-developer
$ cd norduni-developer
# Start dependencies
$ ./start.sh
# Clone NOCLook project repo
$ git clone https://git.nordu.net/norduni.git
$ cd norduni |
Code Block |
git clone https://git.nordu.net/norduni.git git checkout neo4jdb-python # Download neo4j docker image and start it docker pull tpires/neo4j docker run -d -v /path_to_repo/norduni/docker/neo4j.properties:/var/lib/neo4j/conf/neo4j.properties -v /opt/docker/neo4jdata:/var/lib/neo4j/data -p 7474:7474 tpires/neo4j # Create the indexes with curl curl -D - -H "Content-Type: application/json" --data '{"name" : "node_auto_index","config" : {"type" : "fulltext","provider" : "lucene"}}' -X POST http://localhost:7474/db/data/index/node/ curl -D - -H "Content-Type: application/json" --data '{"name" : "relationship_auto_index","config" : {"type" : "fulltext","provider" : "lucene"}}' -X POST http://localhost:7474/db/data/index/relationship/ # Create a virtualenv and activate it $ virtualenv env $ . env/bin/activate # Install the python packages pip$ install paver pip install -r /path_to_repo/requirementsrequirements/dev.txt # Create a settings.py from the template /path_to/repo/src/niweb/niweb cp /path_to_repo/ file $ cp src/niweb/niweb/generic_settings.py /path_to/repo/src/niwebdotenv src/niweb/settings.pydevenv # Sync the db python /path_to_repo/src/niweb/manage.py syncdb $ python /path_to_repo/src/niweb/manage.py migrate apps.noclooksyncdb $ python /path_to_repo/src/niweb/manage.py migrate actstream # Run the app $ python /path_to_repo/src/niweb/manage.py migrate tastypierunserver |
Upgrading to newest versions
This is the general procedure for upgrading to newest version of norduni.
Code Block |
---|
# stash current local changes and update $ git stash $ git pull origin master $ git stash apply # Run the appmigrations $ python /path_to_repo/src/niweb/manage.py runservermigrate # OptionalPip postgres instead of sqlite3, don't forget to change database settings in settings.py. # Download postgres docker image and start it docker pull orchardup/postgresql docker run -d -p 5432:5432 -e POSTGRESQL_USER=norduni -e POSTGRESQL_PASS=docker -e POSTGRESQL_DB=norduni -v /opt/docker/postgresql_data/:/var/lib/postgresql/ orchardup/postgresql update requirements $ pip install -U -r requirements/prod.txt # Collect statics $ python /path_to_repo/src/niweb/manage.py collectstatic # Restart uwsgi $ sudo services uwsgi restart |