...
Page
...
index
- Overview
- Producer
- Consumer
- Repository
- Data Model
- Searching in NOCLook
- Indexed properties
- REST API
- Backup and Restore
- Links
- How to set up NOCLook
- Troubleshooting
Anchor | ||||
---|---|---|---|---|
|
Overview
NORDUnet Network inventory project is divided in to three parts. One part that gathers the data (producer), another part that uses the data (consumer) and a repository for the raw data.
Anchor | ||||
---|---|---|---|---|
|
Producer
For the producer we use the NERDS projects JSON format to save the data. How to actually collect data depends on what you want to collect data from.
Anchor | ||||
---|---|---|---|---|
|
Consumer
The only NORDUnet consumer today is NOCLook which is a web gui made with the Django web framework and Neo4j graph database as back end for the inventory data model. The data is added to the Neo4j database via norduniclient which builds on neo4jdb-python.
Anchor | ||||
---|---|---|---|---|
|
Repository
Git is used to handle the revisions of the collected data.
Anchor | ||||
---|---|---|---|---|
|
Data model
See Presentation.
Anchor | ||||
---|---|---|---|---|
|
Searching in NOCLook
In addition to the search box at the top of the page you can also use the URL field for searching.
Code Block |
---|
# [Overview|#overview] # [Producer|#producer] # [Consumer|#consumer] # [Repository|#repo] # [Data Model|#datamodel] # [Searching in NOCLook|#search] # [Indexed properties|#index] # [Backup and Restore|#backuprestore] # [Links|#links] # [How to set up a NI suite|#setup] {anchor:overview} h3. Overview NORDUnet Network inventory project is divided in to three parts. One part that gathers the data (producer), another part that uses the data (consumer) and a repository for the raw data. {anchor:producer} h3. Producer For the producer we use the [NERDS projects|https://github.com/fredrikt/nerds] JSON format to save the data. How to actually collect data depends on what you want to collect data from. {anchor:consumer} h3. Consumer The only NORDUnet consumer today is [NOCLook|http://git.nordu.net/?p=norduni.git;a=shortlog;h=refs/heads/operation] which is a web gui made with the Django web framework and Neo4j graph database as back end for the [inventory data model|https://portal.nordu.net/display/nordunetops/Network+Inventory+Project#NetworkInventoryProject-pdatamodel]. The data is added to the Neo4j database via the REST interface using [norduni_client.py|http://git.nordu.net/?p=norduni.git;a=history;f=src/niweb/norduni_client.py] which builds on [neo4j-rest-client|https://github.com/versae/neo4j-rest-client]. Neo Tech has release official [python bindings|http://pypi.python.org/pypi/neo4j-embedded] and they are used in the [master branch|http://git.nordu.net/?p=norduni.git;a=summary] of NOCLook. The speed improvement is great so they will be used in the operational NOCLook in a near future. {anchor:repo} h3. Repository [Git|http://git-scm.com/] is used to handle the revisions of the collected data. {anchor:datamodel} h3. Data model !NI_Data_Model.png|thumbnail! Hopefully these entities and relationships are enough. !NI_Prototype_Data.png|thumbnail! The data that was inserted manually in the prototype. {anchor:search} h3. Searching in NOCLook In addition to the search box at the top of the page you can also use the URL field for searching. {code} /findall/[value]/ ex. /findall/10/ /findall/[key]/[value]/ ex. /findall/link/10/ /findin/[node type]/[value]/ ex. /findin/optical-node/10/ /findin/[node type]/[key]/[value]/ ex. /findin/optical-node/link/10/ /getall/[node type]/ ex. /getall/optical-node/ {code} |
The
...
URL
...
field
...
version
...
of
...
the
...
search find substrings so you can't
...
use
...
wild
...
cards
...
but
...
it
...
is
...
at
...
lease
...
case
...
insensitive.
Anchor | ||||
---|---|---|---|---|
|
Indexed properties
Neo4j provides Lucene indexes for nodes and relationships. A "search" index is created with the properties that will be searched when a user uses the search bar. Some optimization indexes was also created to speed up the consuming or presentation.
Node search index:
name (all nodes)
description (interface and unit nodes)
ip_addresses (unit and host nodes)
as_number (peering partner nodes)
hostname (host nodes)
telenor_tn1_number (cable nodes)
Relationship search index:
ip_address (Uses and Depends_on relationships)
Node node_types index
node_type (all nodes)
All property values above is also indexed under the index key all in all indexes. For example in the search index you can query the all-key for both name, description, ip_addresses, as_number and hostname.
Anchor | ||||
---|---|---|---|---|
|
REST API
Note | ||
---|---|---|
| ||
Only L2VPN services are set up in this way. |
NOCLook REST API is driven by django-tastypie. Authentication is set per user as an API key.
Create a new site
Code Block |
---|
$ cat site.json
{
"node_name": "DK-AAL",
"node_type": "/api/v1/node_type/site/",
"node_meta_type": "Location",
"node": {
"address": "Selma Lagerlöfsvej 300",
"area": "Aalborg",
"country": "Denmark",
"country_code": "DK",
"latitude": "57.011944",
"longitude": "9.990560",
"owner_id": "AAU",
"postarea": "Aalborg",
"postcode": "9220"
}
}
$ curl -i -H "Authorization: ApiKey apiuser:secret" -H "Content-Type: application/json" -X POST --data @site.json |
Create a new L2VPN service
Using curl
Code Block |
---|
curl -k -H "Authorization: ApiKey apiuser:secret" -H "Content-Type: application/json" -X POST --data '{"node_name": "NU-SXXXXX1", "operational_state": "In service", "route_distinguisher": "rd1", "vrf_target": "vt1", "end_points": [{"device": "se-tug.nordu.net", "port":"l2vpn_test_port"}, {"device": "se-fre.nordu.net", "port":"l2vpn_test_port"}]}' https://nidev-consumer.nordu.net/api/v1/l2vpn/
Return:
{
"created": "2013-02-20T16:45:41.228176",
"creator": "/api/v1/user/59/",
"description": null,
"handle_id": 11717,
"modified": "2013-02-20T16:45:41.228213",
"modifier": "/api/v1/user/59/",
"node": {
"handle_id": 11717,
"name": "NU-SXXXXX1",
"node_type": "Service",
"operational_state": "In service",
"service_class": "MPLS",
"service_type": "L2VPN"
},
"node_id": 13707,
"node_meta_type": "logical",
"node_name": "NU-SXXXXX1",
"node_type": "/api/v1/node_type/service/",
"object_path": "/service/11717/",
"operational_state": "In service",
"relationships": [
"/api/v1/relationship/47575/",
"/api/v1/relationship/47576/",
"/api/v1/relationship/47577/"
],
"resource_uri": "/api/v1/l2vpn/11717/",
"route_distinguisher": "rd1",
"service_type": "L2VPN",
"vrf_target": "vt1"
}
|
Setting L2VPN service attributes
Using curl
Code Block |
---|
curl -k -H "Authorization: ApiKey apiuser:secret" -H "Content-Type: application/json" -X PUT --data '{"operational_state": "In service", "description": "Test REST API call l2vpn"}' https://nidev-consumer.nordu.net/api/v1/l2vpn/NU-SXXXXX1/
Return:
{
"created": "2013-02-20T16:45:41.228176",
"creator": "/api/v1/user/59/",
"description": "Test REST API call l2vpn",
"handle_id": 11717,
"modified": "2013-02-20T16:51:16.077192",
"modifier": "/api/v1/user/59/",
"node": {
"description": "Test REST API call l2vpn",
"handle_id": 11717,
"name": "NU-SXXXXX1",
"node_type": "Service",
"operational_state": "In service",
"service_class": "MPLS",
"service_type": "L2VPN"
},
"node_id": 13707,
"node_meta_type": "logical",
"node_name": "NU-SXXXXX1",
"node_type": "/api/v1/node_type/service/",
"object_path": "/service/11717/",
"operational_state": "In service",
"relationships": [
"/api/v1/relationship/47577/",
"/api/v1/relationship/47576/",
"/api/v1/relationship/47575/"
],
"resource_uri": "/api/v1/l2vpn/11717/",
"route_distinguisher": "rt1",
"vrf_target": "vrf1"
}
|
Anchor | ||||
---|---|---|---|---|
|
Backup and Restore
You can backup the Postgresql and Neo4j databases using these scripts:
Code Block |
---|
# Backup the SQL database pg_dump norduni > /var/opt/norduni {anchor:index} h3. Indexed properties Neo4j provides [Lucene indexes|http://lucene.apache.org/java/docs/index.html] for nodes and relationships. A "search" index is created with the properties that will be searched when a user uses the search bar. Some optimization indexes was also created to speed up the consuming or presentation. *Node search index:* name (all nodes) description (interface and unit nodes) ip_addresses (unit and host nodes) as_number (peering partner nodes) hostname (host nodes) *Relationship search index:* ip_address (Uses and Depends_on relationships) *Node node_types index* node_type (all nodes) All property values above is also indexed under the index key *all* in all indexes. For example in the search index you can query the all-key for both name, description, ip_addresses, as_number and hostname. {anchor:backuprestore} h3. Backup and Restore You can backup the Postgresql and Neo4j databases using these scripts: {code} # Backup the SQL database su - postgres -c "pg_dumpall > /opt/nistore/producers/noclook/sql/postgres.sql" # Run NOCLook producer for backup purpose cd /opt/norduni/src/scripts/ rm /opt/nistore/producers/noclook/json/* ./noclook_producer.py -O /opt/nistore/producers/noclook/json/ # Push the changes to nistore master /usr/local/sbin/ni-push.sh -r /opt/nistore/ {code} {code|title:restore.conf} |
To restore the data from a backup use psql and noclook_consumer.py.
Code Block | ||
---|---|---|
| ||
# All producers need to be listed here with a path data or empty [data] juniper_conf = nmap_services_py = alcatel_isis = noclook = /opt/nistore/producers/noclook/json/ {code} To restore the data from a backup use psql and noclook_consumer.py. {code} sudo -unagios_checkmk = cfengine_report = |
Code Block |
---|
postgres psql -f /opt/nistore/producers/noclook/sql/postgres.sql postgresnorduni noclook_consumer.py -C restore.conf -I {code} |
When
...
using
...
postgres
...
you
...
have
...
to
...
reset
...
the
...
autoincrementing
...
handle_id
...
property.
Code Block |
---|
} python manage.py sqlsequencereset noclook {code} |
Copy
...
output
...
from
...
the
...
previous
...
command
...
and
...
paste
...
it
...
in
...
the
...
dbshell.
Code Block |
---|
} python manage.py dbshell {code} |
Now
...
everything
...
should
...
be
...
restored
...
and
...
ready
...
to
...
be
...
updated
...
as
...
usual
...
with
...
noclook_consumer.py.
...
Anchor | ||||
---|---|---|---|---|
|
Links
...
http://nidev-consumer.nordu.net
...
(bleeding
...
edge)
...
Issue
...
tracker:
...
https://project.nordu.net/browse/NORDUNIDEV
Graph Gists
Version 1.0: http://portal.graphgist.org/graph_gists/bf9576c3-c7d0-450b-82fb-bd52e4038753
Version 2.0: http://gist.neo4j.org/?fe682b1dc394dd8e4421
Anchor | ||||
---|---|---|---|---|
|
How to set up NOCLook
See Setting up a NORDUni 2.0 instance.
Anchor | ||||
---|---|---|---|---|
|
Troubleshooting
Starting/stopping and restarting
Code Block |
---|
$ sudo service uwsgi start|stop|restart {anchor:setup} h3. How to set up a NI suite [Setting up a NORDUni instance] |