Setting up NOCLook
This guide is written for Ubuntu 10.04.
Paths used in this guide is just a suggestion, as long as Python can find the files it needs to import it is all good.
Code Block |
---|
sudo apt-get install git-core python-virtualenv openjdk-6-jdk build-essential postgresql python-psycopg2 libpq-dev python-dev
|
...
Make a virtual python environment.
Code Block |
---|
sudo su
mkdir /opt/python_environments/
cd /opt/python_environments/
virtualenv --no-site-packages norduni
|
...
Install needed python modules.
Code Block |
---|
sudo su
source /opt/python_environments/norduni/bin/activate
pip install django ipaddr argparse pexpect markdown psycopg2 lucene-querybuilder django-tastypie xlwt south django-activity-stream django-jsonfield
|
...
Get the NORDUni files:
Code Block |
---|
cd /opt/
git clone git://git.nordu.net/norduni.git
|
Get the NERDS files:
Code Block |
---|
mkdir /opt/norduni/tools/
cd /opt/norduni/tools/
git clone https://github.com/fredrikt/nerds.git
|
Symlink admin media and the virtual python environment to niweb.
Code Block |
---|
cd /opt/norduni/src/niweb/
ln -s /opt/python_environments/norduni env
cd static
ln -s ../env/lib/python2.6/site-packages/django/contrib/admin/media/ admin
|
Set password for database user and create a new database
Code Block |
---|
sudo -u postgres psql postgres
\password postgres
Write password
Write password again
Ctrl+D
sudo -u postgres createdb norduni
|
Change the django settings.
Code Block |
---|
# Django settings for niweb project
cd /opt/norduni/src/niweb/
cp generic_settings.py settings.py
vi settings.py
|
Change at least the following settings.
Code Block |
---|
NIWEB_ROOT = '/opt/norduni/src/niweb/'
# Database settings
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'norduni',
'USER': 'postgres',
'PASSWORD': 'secret',
'HOST': 'localhost'
}
}
# Neo4j settings
NEO4J_RESOURCE_URI = '/opt/norduni/dependencies/neo4jdb/'
|
...
Install JPype and Neo4j-embedded.
Download jpype. (http://sourceforge.net/projects/jpype/files/)
Code Block |
---|
pip install neo4j-embedded
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
pip install /path/to/JPype-version.zip
|
...
Change the following lines in urls.py:
Code Block |
---|
cd /opt/norduni/src/niweb/
|
Code Block |
---|
# Django Generic Login
(r'^accounts/login/$', 'django.contrib.auth.views.login'),
# Federated login
#(r'^accounts/', include('niweb.apps.fedlogin.urls')),
|
Code Block |
---|
python manage.py syncdb
python manage.py runserver 0.0.0.0:80
|
...
Comment out the static media url in /opt/norduni/src/niweb/urls.py.
Code Block |
---|
|
# Static serve
#(r'^site_media/(?P<path>.*)$', 'django.views.static.serve',
# {'document_root': settings.STATIC_DEV_MEDIA}),
|
Install nginx, postfix and gunicorn.
Code Block |
---|
sudo apt-get install nginx postfix
pip install gunicorn
|
...
Code Block |
---|
title | /opt/norduni/src/start_noclook.sh |
---|
|
#!/bin/bash
set -e
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
LOGFILE=/var/log/ni/noclook.log
LOGDIR=$(dirname $LOGFILE)
NUM_WORKERS=1
# user/group to run as
USER=user
GROUP=group
cd /opt/norduni/src/niweb
source env/bin/activate
test -d $LOGDIR || mkdir -p $LOGDIR
exec env/bin/gunicorn_django -w $NUM_WORKERS \
--user=$USER --group=$GROUP --log-level=debug \
--log-file=$LOGFILE 2>>$LOGFILE
|
...
Code Block |
---|
title | /etc/nginx/sites-available/default |
---|
|
server {
listen 80;
root /opt/norduni/src/niweb;
server_name ni.example.net;
access_log /var/log/ni/noclook-access.log;
error_log /var/log/ni/noclook-error.log;
location /static/ {
root /opt/norduni/src/niweb/;
autoindex on;
access_log off;
expires 30d;
}
location / {
proxy_pass_header Server;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Scheme $scheme;
proxy_connect_timeout 10;
proxy_read_timeout 10;
proxy_pass http://localhost:8000/;
}
}
|
Install supervisord and set up the following start script.
Code Block |
---|
easy_install supervisor
echo_supervisord_conf > /etc/supervisord.conf
|
Code Block |
---|
title | /etc/init/supervisord.conf |
---|
|
description "supervisord"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
exec /usr/local/bin/supervisord --nodaemon --configuration /etc/supervisord.conf
|
Add the noclook start script to the supervisor configuration.
Code Block |
---|
[program:noclook]
directory = /opt/norduni/src/niweb/
user = user
command = /opt/norduni/src/start_noclook.sh
stdout_logfile = /var/log/ni/supervisor_logfile.log
stderr_logfile = /var/log/ni/supervisor_err_logfile.log
|
...
To start have a look at the NERDS README then clone the NERDS project.
Code Block |
---|
cd /opt/norduni/
mkdir tools
cd tools
git clone https://github.com/fredrikt/nerds.git
|
...
Code Block |
---|
|
[ssh]
user = view_account_user
password = not_so_secret_password
[sources]
remote = one.example.org two.example.org three.example.org
local = /var/conf/one.xml /var/conf/two.xml /var/conf/three.xml
|
Code Block |
---|
title | Blank output from the juniper_conf.py producer. |
---|
|
"host": {
"juniper_conf": {
"bgp_peerings": [
{
"as_number": "",
"group": "",
"description": "",
"remote_address": "",
"local_address": "",
"type": ""
},
],
"interfaces": [
{
"name": "",
"bundle": "",
"vlantagging": true/false,
"units": [
{
"address": [
"",
""
],
"description": "",
"unit": "",
"vlanid": ""
}
],
"tunnels": [
{
"source": "",
"destination": ""
}
],
"description": ""
},
],
"name": ""
},
"version": 1,
"name": ""
}
|
The JSON files can the be inserted using noclook_juniper_consumer.py.
Change the path at the top of the script to be able to import norduni_client.py.
Edit the template.conf file with the correct path to the Juniper NERDS files.
Code Block |
---|
[data]
juniper_conf = /path/to/juniper/json
nmap_services =
alcatel_isis =
noclook =
|
Then run:
Code Block |
---|
python noclook_juniper_consumer.py -C template.conf
|
...
Code Block |
---|
title | Optional mapping file |
---|
|
osi_address;name;other1;otherN
47002300000001000100010001002060280DB11D;NU-SHHM-ILA-01;info1;infoN
|
Code Block |
---|
"host": {
"alcatel_isis": {
"data": {
"ip_address": "",
"link": "",
"name": "",
"osi_address": "",
"ots": "",
"type": ""
},
"name": "",
"neighbours": [
{
"metric": "",
"name": ""
},
]
},
"name": "",
"version": 1
}
|
The JSON files can be inserted with noclook_alcatel_consumer.py.
Edit the template.conf file with the correct path to the Alcatel ISIS NERDS files.
Change the path at the top of the script to be able to import norduni_client.py.
Code Block |
---|
[data]
juniper_conf =
nmap_services =
alcatel_isis = /path/to/alcatel/json
noclook =
|
Then run:
Code Block |
---|
python noclook_alcatel_consumer.py -C template.conf
|
nmap Producer/Consumer
Using the namp the nmap services producer you can scan a network or individual addresses. NORDUnet have a file
with networks that is used with the "-iL networks_file" option added to NERDS_NMAP_OPTIONS in the run.sh file.
You need to install two Perl modules if you don't already have them.
Code Block |
---|
cpan Nmap::Parser
cpan JSON
|
python-nmap from https://github.com/johanlundberg/python-nmap if the pip version gives you trouble.
Then you can scan your localhost with:
Code Block |
---|
cd /opt/norduni/tools/nerds/producers/nmap_services
./run.sh -O . 127.0.0.1
|
You will find the JSON file in /opt/norduni/tools/nerds/producers/nmap_services/producers/json/.
Code Block |
---|
"host" : {
"." : {
"os" : {
"family" : "",
"name" : ""
}
},
"addrs" : [
"127.0.0.1"
],
"hostnames" : [
"host.example.org"
],
"name" : "host.example.org",
"services" : {
"ipv4": {
"127.0.0.1": {
"tcp": {
"1025": {
"product": "Microsoft Windows RPC",
"confidence": "10",
"name": "msrpc",
"proto": "unknown"},
"1029": {
"product": "Microsoft Windows RPC over HTTP",
"confidence": "10",
"version": "1.0",
"name": "ncacn_http",
"proto": "unknown"},
}
}
}
},
"version" : 1
}
|
The JSON files can be inserted with noclook_nmap_consumer_py.py.
Edit the template.conf file with the correct path to the nmap services JSON files.
Change the path at the top of the script to be able to import norduni_client.py.
Code Block |
---|
[data]
juniper_conf =
nmap_services = /path/to/nmap/json
alcatel_isis =
noclook =
|
Then run:
Code Block |
---|
python noclook_nmap_consumer.py -C template.conf
|
...
Code Block |
---|
title | NORDUnet site documentation example |
---|
|
name;Host;site_type;address;area;postcode;city;country;floor;room;latitude;longitude;responsible_for;owner_id;telenor_subscription_id;comment
|
Code Block |
---|
title | Blank output from NORDUnet site documentation example |
---|
|
{
"host": {
"csv_producer": {
"address": "",
"area": "",
"city": "",
"comment": "",
"country": "",
"floor": "",
"latitude": "",
"longitude": "",
"meta_type": "",
"name": "",
"node_type": "",
"owner_id": "",
"postcode": "",
"responsible_for": "",
"room": "",
"site_type": "",
"telenor_subscription_id": ""
},
"name": "",
"version": 1
}
}
|
...
Change the path at the top of the script to be able to import norduni_client.py.
Then run:
Code Block |
---|
python noclook_site_csv_consumer.py -D /path/to/site_files/json
|
...
Change the path at the top of the script to be able to import norduni_client.py.
Code Block |
---|
[data]
juniper_conf = /path/to/juniper/json
nmap_services = /path/to/nmap/json
alcatel_isis = /path/to/alcate/json
noclook = #Used for loading backup.
|
Then run:
Code Block |
---|
python noclook_consumer.py -C template.conf -I
|
...
The database can be purged from all data using the P switch to the noclook_consumer.py script.
Code Block |
---|
python noclook_consumer.py -C template.conf -P
|
Setting up old NOCLook version
Code Block |
---|
sudo apt-get install apache2 libapache2-mod-wsgi git-core python-virtualenv postfix openjdk-6-jre-headless postgresql python-psycopg2 libpq-dev python-dev
|
Make a virtual python environment.
Code Block |
---|
cd /opt/python_environments/
sudo virtualenv --no-site-packages norduni
source norduni/env/bin/acivate
|
Install needed python modules.
Code Block |
---|
sudo su
source python_environments/norduni/bin/activate
pip install django ipaddr argparse pexpect markdown psycopg2 lucene-querybuilder httplib2
|
Get the NORDUni files:
Code Block |
---|
cd /var/opt/
git clone git://git.nordu.net/norduni.git
git checkout operation
|
Get the NERDS files:
Code Block |
---|
mkdir /opt/norduni/tools/
cd /opt/norduni/tools/
git clone https://github.com/fredrikt/nerds.git
|
Symlink admin media and the virtual python environment to niweb.
Code Block |
---|
cd /var/opt/norduni/src/niweb/
ln -s /var/opt/python_environments/norduni/ env
cd static
ln -s ../env/lib/python2.6/site-packages/django/contrib/admin/media/ admin
|
Set password for database user and create a new database
Code Block |
---|
sudo -u postgres psql postgres
\password postgres
Write password
Write password again
Ctrl+D
sudo -u postgres createdb norduni
|
Change the django settings.
Code Block |
---|
# Django settings for niweb project
cd /opt/norduni/src/niweb/
cp generic_settings.py settings.py
vi settings.py
|
Change at least the following settings.
Code Block |
---|
NIWEB_ROOT = '/opt/norduni/src/niweb/'
NIWEB_MEDIA_URL = '/static/'
# Database settings
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'norduni',
'USER': 'postgres',
'PASSWORD': 'secret',
'HOST': 'localhost'
}
}
# Neo4j settings
NEO4J_RESOURCE_URI = 'http://localhost:7474/db/data/'
|
Comment out the static media url in /opt/norduni/src/niweb/urls.py.
Code Block |
---|
|
# Static serve
#(r'^site_media/(?P<path>.*)$', 'django.views.static.serve',
# {'document_root': settings.STATIC_DEV_MEDIA}),
|
Create a file name niweb in /etc/apache2/conf.d/.
Code Block |
---|
WSGIPythonPath "/var/opt/norduni/src:/var/opt/norduni/src/niweb"
WSGIPythonHome "/opt/python_environments/norduni"
WSGIRestrictStdin Off
WSGIRestrictStdout Off
WSGIScriptAlias / /var/opt/norduni/niweb.wsgi
|
Add niweb path to niweb.wsgi
Code Block |
---|
sys.path.append('/opt/norduni/src/niweb')
|
Add the following lines to /etc/apache2/sites-available/default-ssl
Code Block |
---|
Alias /admin_media /var/opt/norduni/src/niweb/static/admin/
Alias /site_media /var/opt/norduni/src/niweb/static/
<LocationMatch "^/(admin_media|site_media)">
Options FollowSymLinks
Order allow,deny
allow from all
SetHandler None
</LocationMatch>
|
...
Download neo4j-1.3 from http://neo4j.org.
Extract the file to, for example, /var/opt/norduni/dependencies/.
Code Block |
---|
cd /var/opt/norduni/dependencies/
tar xfz neo4j-1.3.tar.gz
|
Copy the startup script from /var/opt/norduni/src/scripts/neo4j/ to /etc/init.d/ and then make neo4j start on boot.
Code Block |
---|
update-rc.d /etc/init.d/neo4j defaults
|
You need to shut down port 7474 from anything except localhost.
Code Block |
---|
iptables -A INPUT -p tcp --dport 7474 -s localhost -j ACCEPT
iptables -A INPUT -p tcp --dport 7474 -j DROP
|
Save the rules and add the following line to your interface configuration so you will restore the rules on a reboot.
Code Block |
---|
bash -c "iptables-save > /etc/iptables.rules"
vi /etc/network/interfaces
iface eth0 inet static
pre-up iptables-restore < /etc/iptables.rules
|
...
Create a file name niweb in /etc/apache2/conf.d/.
Code Block |
---|
WSGIPythonPath "/var/opt/norduni/src:/var/opt/norduni/src/niweb"
WSGIPythonHome "/opt/python_environments/norduni"
WSGIDaemonProcess processes=1 threads=25 # For neo4j-embedded
WSGIApplicationGroup %{GLOBAL} # For neo4j-embedded
WSGIRestrictStdin Off
WSGIRestrictStdout Off
WSGIScriptAlias / /var/opt/norduni/niweb.wsgi
|
Add niweb path to niweb.wsgi
Code Block |
---|
sys.path.append('/opt/norduni/src/niweb')
|
/etc/apache2.conf
Code Block |
---|
<IfModule mpm_prefork_module>
StartServers 1
ServerLimit 1
MinSpareServers 5
MaxSpareServers 10
MaxClients 1
MaxRequestsPerChild 0
</IfModule>
|
Apache Shibboleth configuration
Code Block |
---|
<LocationMatch "Shibboleth.sso">
SetHandler None
</LocationMatch>
<Location "/accounts/login-federated/">
AuthType shibboleth
ShibRequireSession On
require valid-user
RequestHeader set X_REMOTE_USER %{eppn}e
RequestHeader set EPPN %{eppn}e
RequestHeader set GIVENNAME %{givenName}e
RequestHeader set SN %{sn}e
RequestHeader set MAIL %{mail}e
RequestHeader set AFFILIATION %{affiliation}e
RequestHeader set ENTITLEMENT %{entitlement}e
</Location>
|
...
To start the app:
Code Block |
---|
sudo su
/opt/norduni/start_dev_noclook.sh
|
...
To add more hosts just go to /opt/norduni/tools/nerds/producers/nmap_services_py.
Code Block |
---|
./run json address to scan
|
After the data collection you insert the data using the consumer. (You have to stop the Django app first.)
Code Block |
---|
. /opt/norduni/src/niweb/env/bin/activate
cd /opt/norduni/src/scripts
python noclook_consumer.py -C test.conf -I
|