Note | ||
---|---|---|
| ||
Deprecated. Please use Setting up a NORDUni 2.0 instance. |
Table of Contents |
---|
Setting up NOCLook
This guide is written for Ubuntu 10.04.For neo4j-embedded:
Paths used in this guide is just a suggestion, as long as Python can find the files it needs to import it is all good.
Code Block |
---|
sudo apt-get install nginx git-core python-virtualenv postfix openjdk-6-jdk build-essential postgresql python-psycopg2 libpq-dev python-dev |
Postgresql is optional but you need a SQL database. See Django database documentation for other supported SQL databases.
Make a virtual python environment.For neo4j-rest-client:
Code Block |
---|
sudo apt-get install apache2 libapache2-mod-wsgi git-core python-virtualenv postfix openjdk-6-jre-headless postgresql python-psycopg2 libpq-dev python-dev |
For both neo4j versions:
Make a virtual python environment.
...
su
mkdir /opt/python_environments/
cd /opt/python_environments/
virtualenv --no-site-packages norduni
|
Making a virtual environment is also just a suggestion but it makes it easier to keep your system clean.
Install needed python modules.
Code Block |
---|
sudo su source /opt/python_environments/norduni/bin/activate pip install django ipaddr argparse pexpect markdown psycopg2 lucene-querybuilder gunicorn |
Add the following python modules if you are running the Neo4j stand alone server:
...
django-tastypie xlwt south django-activity-stream django-jsonfield
|
Here psycopg2 is optional but you need to install the Python database module of your SQL database choice. If you have trouble installing psycopg2.
Get the NORDUni files:
For neo4j-embedded:
Code Block |
---|
cd /var/opt/ git clone git://git.nordu.net/norduni.git |
For neo4j-rest-client:
Code Block |
---|
cd /var/opt/
git clone git://git.nordu.net/norduni.git
git checkout operation
|
For both neo4j versions:
Get the NERDS Get the NERDS files:
Code Block |
---|
mkdir /opt/norduni/tools/
cd /opt/norduni/tools/
git clone https://github.com/fredrikt/nerds.git
|
Symlink admin media and the virtual python environment to niweb.
Code Block |
---|
cd /var/opt/norduni/src/niweb/ ln -s /var/opt/python_environments/norduni/ env cd static ln -s ../env/lib/python2.6/site-packages/django/contrib/admin/media/ admin |
Set password for database user and create a new database
Code Block |
---|
sudo -u postgres psql postgres
\password postgres
Write password
Write password again
Ctrl+D
sudo -u postgres createdb norduni
|
Change the django settings.
Code Block |
---|
# Django settings for niweb project
cd /opt/norduni/src/niweb/
cp generic_settings.py settings.py
vi settings.py
|
Change at least the following settings.
Code Block |
---|
NIWEB_ROOT = '/opt/norduni/src/niweb/' NIWEB_MEDIA_URL # Database settings DATABASES = '/static/' # Database settings DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql_psycopg2', 'NAME': 'norduni', 'USER': 'postgres', 'PASSWORD': 'secret', 'HOST': 'localhost' } } # Neo4j settings NEO4J_RESOURCE_URI = '/opt/norduni/dependencies/neo4jdb/' |
Comment out the static media url in /opt/norduni/src/niweb/urls.py.
Code Block | ||
---|---|---|
| ||
# Static serve
#(r'^site_media/(?P<path>.*)$', 'django.views.static.serve',
# {'document_root': settings.STATIC_DEV_MEDIA}),
|
Create a gunicorn start file.
Neo4j >1.5 embedded with Python bindings
Install JPype and Neo4j-embedded.
Download jpype. (http://sourceforge.net/projects/jpype/files/)
Code Block |
---|
pip install neo4j-embedded
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
pip install /path/to/JPype-version.zip
|
Setting up a local/development NOCLook
Set up the SQL database using the Django project.
Change the following lines in urls.py:
Code Block | |
---|---|
cd | |
Code Block | |
title | /opt/norduni/src | /start_noclook.sh
#!/bin/bash
set -e
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
LOGFILE=/var/log/ni/noclook.log
LOGDIR=$(dirname $LOGFILE)
NUM_WORKERS=1
# user/group to run as
USER=user
GROUP=group
cd /opt/norduni/src/niweb
source env/bin/activate
test -d $LOGDIR || mkdir -p $LOGDIR
exec env/bin/gunicorn_django -w $NUM_WORKERS \
--user=$USER --group=$GROUP --log-level=debug \
--log-file=$LOGFILE 2>>$LOGFILE
|
Configure nginx.
/niweb/
|
Code Block |
---|
# Django Generic Login
(r'^accounts/login/$', 'django.contrib.auth.views.login'),
# Federated login
#(r'^accounts/', include('niweb.apps.fedlogin.urls')),
|
Code Block |
---|
python manage.py syncdb
python manage.py runserver 0.0.0.0:80
|
Now you should be able surf to your machines ip and see the NOCLook app started.
It is time to collect and insert some data.
Deploying NOCLook
Comment out the static media url in /opt/norduni/src/niweb/urls.py.
Code Block | ||
---|---|---|
| ||
# Static serve
#(r'^site_media/(?P<path>.*)$', 'django.views.static.serve',
# {'document_root': settings.STATIC_DEV_MEDIA}),
|
Install nginx, postfix and gunicorn.
Code Block |
---|
sudo apt-get install nginx postfix
pip install gunicorn
|
Create a gunicorn start file.
Code Block | ||
---|---|---|
| ||
#!/bin/bash
set -e
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
LOGFILE=/var/log/ni/noclook.log
LOGDIR=$(dirname $LOGFILE)
NUM_WORKERS=1
# user/group to run as
USER=user
GROUP=group
cd /opt/norduni/src/niweb
source env/bin/activate
test -d $LOGDIR || mkdir -p $LOGDIR
exec env/bin/gunicorn_django -w $NUM_WORKERS \
--user=$USER --group=$GROUP --log-level=debug \
--log-file=$LOGFILE 2>>$LOGFILE
|
Configure nginx.
Code Block | ||
---|---|---|
| ||
server {
listen 80;
root /opt/norduni/src/niweb;
server_name ni.example.net;
access_log /var/log/ni/noclook-access.log;
error_log /var/log/ni/noclook-error.log;
location /static/ {
root /opt/norduni/src/niweb/;
autoindex on;
access_log off;
expires 30d;
}
location / {
proxy_pass_header Server;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Scheme $scheme;
proxy_connect_timeout 10;
proxy_read_timeout 10;
proxy_pass http://localhost:8000/;
}
}
|
Install supervisord and set up the following start script.
Code Block |
---|
easy_install supervisor
echo_supervisord_conf > /etc/supervisord.conf
|
Code Block | ||
---|---|---|
| ||
description "supervisord"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
exec /usr/local/bin/supervisord --nodaemon --configuration /etc/supervisord.conf
|
Add the noclook start script to the supervisor configuration.
Code Block |
---|
[program:noclook]
directory = /opt/norduni/src/niweb/
user = user
command = /opt/norduni/src/start_noclook.sh
stdout_logfile = /var/log/ni/supervisor_logfile.log
stderr_logfile = /var/log/ni/supervisor_err_logfile.log
|
Collecting and processing network data
To insert data you need to stop any python process that is using the Neo4j database. We hope to get the option to load more database instances in read-only mode in a near future then this could be avoided.
NORDUnet has a GIT repository called nistore and it is cloned to /opt/nistore/.
To start have a look at the NERDS README then clone the NERDS project.
Code Block |
---|
cd /opt/norduni/
mkdir tools
cd tools
git clone https://github.com/fredrikt/nerds.git
|
Juniper Configuration Producer/Consumer
The Juniper configuration producer can load Juniper configuration directly from the router via SSH or Juniper configuration files in XML format from disk.
Code Block | ||
---|---|---|
| ||
[ssh]
user = view_account_user
password = not_so_secret_password
[sources]
remote = one.example.org two.example.org three.example.org
local = /var/conf/one.xml /var/conf/two.xml /var/conf/three.xml
|
Code Block | ||
---|---|---|
| ||
"host": {
"juniper_conf": {
"bgp_peerings": [
{
"as_number": "",
"group": "",
"description": "",
"remote_address": "",
"local_address": "",
"type": ""
},
],
"interfaces": [
{
"name": "",
"bundle": "",
"vlantagging": true/false,
"units": [
{
"address": [
"",
""
],
"description": "",
"unit": "",
"vlanid": ""
}
],
"tunnels": [
{
"source": "",
"destination": ""
}
],
"description": ""
},
],
"name": ""
},
"version": 1,
"name": ""
}
|
The JSON files can the be inserted using noclook_juniper_consumer.py.
Change the path at the top of the script to be able to import norduni_client.py.
Edit the template.conf file with the correct path to the Juniper NERDS files.
Code Block |
---|
[data]
juniper_conf = /path/to/juniper/json
nmap_services =
alcatel_isis =
noclook =
|
Then run:
Code Block |
---|
python noclook_juniper_consumer.py -C template.conf
|
Alcatel-Lucent ISIS Producer/Consumer
Using the output from the "show isis database detail" on a Cisco router
connected to the Alcatel-Lucent DCN network, nodes and their neighbors
will be grouped.
To get a more human readable result use the IOS command "clns" to map
the NSAP address to a hostname. eg. clns host hostname NSAP_address.
You can also provide a mapping CSV file. The mandatory columns are
osi_address and name. All following columns will be added to the JSON
output.
Code Block | ||
---|---|---|
| ||
osi_address;name;other1;otherN
47002300000001000100010001002060280DB11D;NU-SHHM-ILA-01;info1;infoN
|
Code Block |
---|
"host": {
"alcatel_isis": {
"data": {
"ip_address": "",
"link": "",
"name": "",
"osi_address": "",
"ots": "",
"type": ""
},
"name": "",
"neighbours": [
{
"metric": "",
"name": ""
},
]
},
"name": "",
"version": 1
}
|
The JSON files can be inserted with noclook_alcatel_consumer.py.
Edit the template.conf file with the correct path to the Alcatel ISIS NERDS files.
Change the path at the top of the script to be able to import norduni_client.py.
Code Block |
---|
[data]
juniper_conf =
nmap_services =
alcatel_isis = /path/to/alcatel/json
noclook =
|
Then run:
Code Block |
---|
python noclook_alcatel_consumer.py -C template.conf
|
nmap Producer/Consumer
Using the nmap services producer you can scan a network or individual addresses. NORDUnet have a file
with networks that is used with the "-iL networks_file" option added to NERDS_NMAP_OPTIONS in the run.sh file.
You need to install python-nmap from https://github.com/johanlundberg/python-nmap if the pip version gives you trouble.
Then you can scan your localhost with:
Code Block |
---|
cd /opt/norduni/tools/nerds/producers/nmap_services
./run.sh . 127.0.0.1
|
You will find the JSON file in /opt/norduni/tools/nerds/producers/nmap_services/producers/json/.
Code Block |
---|
"host" : {
"." : {
"os" : {
"family" : "",
"name" : ""
}
},
"addrs" : [
"127.0.0.1"
],
"hostnames" : [
"host.example.org"
],
"name" : "host.example.org",
"services" : {
"ipv4": {
"127.0.0.1": {
"tcp": {
"1025": {
"product": "Microsoft Windows RPC",
"confidence": "10",
"name": "msrpc",
"proto": "unknown"},
"1029": {
"product": "Microsoft Windows RPC over HTTP",
"confidence": "10",
"version": "1.0",
"name": "ncacn_http",
"proto": "unknown"},
}
}
}
},
"version" : 1
}
|
The JSON files can be inserted with noclook_nmap_consumer_py.py.
Edit the template.conf file with the correct path to the nmap services JSON files.
Change the path at the top of the script to be able to import norduni_client.py.
Code Block |
---|
[data]
juniper_conf =
nmap_services = /path/to/nmap/json
alcatel_isis =
noclook =
|
Then run:
Code Block |
---|
python noclook_nmap_consumer.py -C template.conf
|
CSV Site Producer/Consumer
The script produces JSON output in the NERDS format from the provided CSV file.
The csv file needs to start with the name of the node and then the node type.
After those two columns any other node property may follow.
Start your csv file with a line similar to the one below.
name;node_type;node_property1,node_property2;...;node_property15
Code Block | ||
---|---|---|
| ||
name;Host;site_type;address;area;postcode;city;country;floor;room;latitude;longitude;responsible_for;owner_id;telenor_subscription_id;comment
|
Code Block | ||
---|---|---|
| ||
{
"host": {
"csv_producer": {
"address": "",
"area": "",
"city": "",
"comment": "",
"country": "",
"floor": "",
"latitude": "",
"longitude": "",
"meta_type": "",
"name": "",
"node_type": "",
"owner_id": "",
"postcode": "",
"responsible_for": "",
"room": "",
"site_type": "",
"telenor_subscription_id": ""
},
"name": "",
"version": 1
}
}
|
The consumer script should only be run once as it does not update the sites, only creates new.
The JSON file directory is then inserted in to the database using noclook_site_csv_consumer.py.
Change the path at the top of the script to be able to import norduni_client.py.
Then run:
Code Block |
---|
python noclook_site_csv_consumer.py -D /path/to/site_files/json
|
Daily database update
The producers are run with a cron job and the script noclook_consumer.py is used to run the three inserting/updating scripts (noclook_juniper_consumer.py, noclook_alcatel_consumer.py and noclook_nmap_consumer.py).
Change the path at the top of the script to be able to import norduni_client.py.
Code Block |
---|
[data]
juniper_conf = /path/to/juniper/json
nmap_services = /path/to/nmap/json
alcatel_isis = /path/to/alcate/json
noclook = #Used for loading backup.
|
Then run:
Code Block |
---|
python noclook_consumer.py -C template.conf -I
|
Purging the database
The database can be purged from all data using the P switch to the noclook_consumer.py script.
Code Block |
---|
python noclook_consumer.py -C template.conf -P
|
Setting up old NOCLook version
Code Block |
---|
sudo apt-get install apache2 libapache2-mod-wsgi git-core python-virtualenv postfix openjdk-6-jre-headless postgresql python-psycopg2 libpq-dev python-dev
|
Make a virtual python environment.
Code Block |
---|
cd /opt/python_environments/
sudo virtualenv --no-site-packages norduni
source norduni/env/bin/acivate
|
Install needed python modules.
Code Block |
---|
sudo su
source python_environments/norduni/bin/activate
pip install django ipaddr argparse pexpect markdown psycopg2 lucene-querybuilder httplib2
|
Get the NORDUni files:
Code Block |
---|
cd /var/opt/
git clone git://git.nordu.net/norduni.git
git checkout operation
|
Get the NERDS files:
Code Block |
---|
mkdir /opt/norduni/tools/
cd /opt/norduni/tools/
git clone https://github.com/fredrikt/nerds.git
|
Symlink admin media and the virtual python environment to niweb.
Code Block |
---|
cd /var/opt/norduni/src/niweb/
ln -s /var/opt/python_environments/norduni/ env
cd static
ln -s ../env/lib/python2.6/site-packages/django/contrib/admin/media/ admin
|
Set password for database user and create a new database
Code Block |
---|
sudo -u postgres psql postgres
\password postgres
Write password
Write password again
Ctrl+D
sudo -u postgres createdb norduni
|
Change the django settings.
Code Block | ||
---|---|---|
# Django settings for niweb project
cd | ||
Code Block | ||
| ||
server { listen 80; root /opt/norduni/src/niweb; server_name ni.example.net; access_log /var/log/ni/noclook-access.log; error_log /var/log/ni/noclook-error.log; location /static/ { root /opt/norduni/src/niweb/; cp generic_settings.py autoindex on; access_log off; expires 30d; } location / settings.py vi settings.py |
Change at least the following settings.
Code Block |
---|
NIWEB_ROOT = '/opt/norduni/src/niweb/' NIWEB_MEDIA_URL = '/static/' # Database settings DATABASES = { 'default': { proxy_pass_header Server; 'ENGINE': 'django.db.backends.postgresql_psycopg2', proxy_set_header Host $http_host; 'NAME': 'norduni', proxy_redirect off;'USER': 'postgres', proxy_set_header X-Real-IP $remote_addr;'PASSWORD': 'secret', proxy_set_header X-Scheme $scheme;'HOST': 'localhost' } } # proxy_connect_timeout 10; proxy_read_timeout 10; proxy_pass http://localhost:8000/; } } |
Install supervisord and set up the following start script.
Code Block |
---|
easy_install supervisor
echo_supervisord_conf > /etc/supervisord.conf
|
Code Block | ||
---|---|---|
| ||
description "supervisord"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
exec /usr/local/bin/supervisord --nodaemon --configuration /etc/supervisord.conf
|
Add the noclook start script to the supervisor configuration.
Neo4j settings
NEO4J_RESOURCE_URI = 'http://localhost:7474/db/data/'
|
Comment out the static media url in /opt/norduni/src/niweb/urls.py.
Code Block | ||
---|---|---|
| ||
# Static serve
#(r'^site_media/(?P<path>.*)$', 'django.views.static.serve',
# {'document_root': settings.STATIC_DEV_MEDIA}),
|
Create a file name niweb in /etc/apache2/conf.d/.
Code Block |
---|
WSGIPythonPath "/var/opt/norduni/src:/var/opt/norduni/src/niweb"
WSGIPythonHome "/opt/python_environments/norduni"
WSGIRestrictStdin Off
WSGIRestrictStdout Off
WSGIScriptAlias / /var/opt/norduni/niweb.wsgi
|
Add niweb path to niweb.wsgi
Code Block |
---|
sys.path.append(' |
Code Block |
[program:noclook] directory = /opt/norduni/src/niweb/ user = user command = /opt/norduni/src/start_noclook.sh stdout_logfile = /var/log/ni/supervisor_logfile.log stderr_logfile = /var/log/ni/supervisor_err_logfile.log |
Set up a GIT repository for the JSON files produced by the NERDS producers and make a clone in /var/opt/. We have a GIT repository called nistore
and we have made a clone found at /var/opt/nistore/.
Produce JSON files with the producers you want.
Consume the recently produces JSON files with the NORDUni consume scripts.
Install Neo4j graph database
Neo4j 1.5 embedded with Python bindings (recommended)
Install JPype and Neo4j-embedded.
Download jpype. (http://sourceforge.net/projects/jpype/files/)
')
|
Add the following lines to /etc/apache2/sites-available/default-ssl
Code Block |
---|
Alias /admin_media /var/opt/norduni/src/niweb/static/admin/
Alias /site_media /var/opt/norduni/src/niweb/static/
<LocationMatch "^/(admin_media|site_media)">
Options FollowSymLinks
Order allow,deny
allow from all
SetHandler None
</LocationMatch> |
Code Block |
pip install neo4j-embedded
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/jre/
pip install /path/to/jpype-version.zip
|
Neo4j 1.3 REST stand alone server
Download neo4j-1.3 from http://neo4j.org.
Extract the file to, for example, /var/opt/norduni/dependencies/.
Code Block |
---|
cd /var/opt/norduni/dependencies/
tar xfz neo4j-1.3.tar.gz
|
Copy the startup script from /var/opt/norduni/src/scripts/neo4j/ to /etc/init.d/ and then make neo4j start on boot.
Code Block |
---|
update-rc.d /etetc/init.d/neo4j defaults |
You need to shut down port 7474 from anything except localhost.
Code Block |
---|
iptables -A INPUT -p tcp --dport 7474 -s localhost -j ACCEPT
iptables -A INPUT -p tcp --dport 7474 -j DROP
|
Save the rules and add the following line to your interface configuration so you will restore the rules on a reboot.
Code Block |
---|
bash -c "iptables-save > /etc/iptables.rules"
vi /etc/network/interfaces
iface eth0 inet static
pre-up iptables-restore < /etc/iptables.rules
|
...
Apache
...
configuration for neo4j-embedded
If you want to run the latest version if NOCLook (using neo4j-embedded) you have to configure Apache to only use one process to run the Django code.
Configure the web server to play nice with neo4j-embedded. Install apache2-mpm-prefork and libapache2-mod-wsgi.
Create a file name niweb in /etc/apache2/conf.d/.
Code Block |
---|
WSGIPythonPath "/var/opt/norduni/src:/var/opt/norduni/src/niweb"
WSGIPythonHome "/opt/python_environments/norduni"
WSGIDaemonProcess processes=1 threads=25 # For neo4j-embedded
WSGIApplicationGroup %{GLOBAL} # For neo4j-embedded
WSGIRestrictStdin Off
WSGIRestrictStdout Off
WSGIScriptAlias / /var/opt/norduni/niweb.wsgi
|
Add niweb path to niweb.wsgi
Code Block |
---|
sys.path.append('/opt/norduni/src/niweb')
|
/etc/apache2.conf
Code Block |
---|
<IfModule mpm_prefork_module> StartServers 1 ServerLimit 1 MinSpareServers 5 MaxSpareServers 10 MaxClients 1 MaxRequestsPerChild 0 </IfModule> |
...
</IfModule>
|
Apache Shibboleth configuration
Code Block |
---|
<LocationMatch "Shibboleth.sso"> SetHandler None </LocationMatch> <Location "/accounts/login-federated/"> AuthType shibboleth ShibRequireSession On require valid-user RequestHeader set X_REMOTE_USER %{eppn}e RequestHeader set EPPN %{eppn}e RequestHeader set GIVENNAME %{givenName}e RequestHeader set SN %{sn}e RequestHeader set MAIL %{mail}e RequestHeader set AFFILIATION %{affiliation}e RequestHeader set ENTITLEMENT %{entitlement}e </Location> Alias /admin_media /var/opt/norduni/src/niweb/static/admin/ Alias /site_media /var/opt/norduni/src/niweb/static/ <LocationMatch "^/(admin_media|site_media)"> Options FollowSymLinks Order allow,deny allow from all SetHandler None </LocationMatch> ENTITLEMENT %{entitlement}e </Location> |
Using the Virtualbox image
Download the OVA file, NOCLook Test Ubuntu 12.04.ova.
Username: norduni
Password: norduni+123
To start the app:
Code Block |
---|
sudo su
/opt/norduni/start_dev_noclook.sh
|
Now you should be able surf to your machines ip and see the NOCLook app started.
The NOCLook username is also norduni and the password is norduni+123.
To add more hosts just go to /opt/norduni/tools/nerds/producers/nmap_services_py.
Code Block |
---|
./run json address to scan
|
After the data collection you insert the data using the consumer. (You have to stop the Django app first.)
Code Block |
---|
. /opt/norduni/src/niweb/env/bin/activate
cd /opt/norduni/src/scripts
python noclook_consumer.py -C test.conf -I
|