The DB-All.e API

Every measured value is held in an object of type dballe.Var:

The methods of dballe.Var are:

Detailed information about a measured value can be accessed using the info() method of dballe.Var, which gives a dballe.Varinfo object:

The methods of dballe.Varinfo are:

Data provided as input to the database, and data coming out of the database, is represented many dballe.Var objects grouped in a dballe.Record:

Record

DB-All.E record.

A Record is a container for one observation of meteorological values, that includes anagraphical informations, physical location of the observation in time and space, and all the observed variables.

C++ includes: record.h

The methods of dballe.Record are:

add(source)

source is a dballe.Record

Copy all data from the record source into dest. At the end of the function, dest will contain its previous values, plus the values in source. If a value is present both in source and in dest, the one in dest will be overwritten. Parameters: source: The record to copy data from.

clear()

Remove all data from the record.

clear_vars()

Remove all variables from the record, leaving the keywords intact.

bool = contains(subset)

subset is a dballe.Record

Return true if all elements of subset are present in this record, with the same value.

items(self)

List of names of variables in the record

iteritems(self)

List of names of variables in the record

iterkeys(self)

List of names of variables in the record

itervalues(self)

List of names of variables in the record

itervars(self)

Iterate all the variables in the record

keys(self)

List of names of variables in the record

parse_date(values)

values is a int *

Parse the date set in the Record. This function will examine the values year, month, day, hour, min and sec, and will compute the lower bound of the datetime they represent. Parameters: values: An array of 6 integers that will be filled with the minimum year, month, day, hour, minute and seconds.

parse_date_extremes(minvalues, maxvalues)

minvalues is a int *

maxvalues is a int *

Parse the date extremes set in Record. This function will examine the values yearmin, monthmin, daymin, hourmin, minumin, secmin, yearmax, monthmax, daymax, hourmax, minumax, secmax, year, month, day, hour, min and sec, and will compute the two datetime extremes that bound the interval they represent. Parameters: minvalues: An array of 6 integers that will be filled with the minimum year, month, day, hour, minute and seconds. maxvalues: An array of 6 integers that will be filled with the maximum year, month, day, hour, minute and seconds.

const = Var* peek(Varcode code) const  throw () Shortcut

for var_peek.

str = peek_value(Varcode code) const  throw ()

Shortcut for var_peek_value.

set_ana_context()

Set the date, level and timerange values to match the anagraphical context.

set_from_string(str)

str is a str

Set a value in the record according to an assignment encoded in a string. String can use keywords, aliases and varcodes. Examples: ana_id=3, name=Bologna, B12012=32.4 Parameters: rec: The record where the value is to be set. str: The string containing the assignment. The error indicator for the function.

set_to_difference(source1, source2)

source1 is a dballe.Record

source2 is a dballe.Record

Set the record to contain only those fields that change source1 into source2. If a field has been deleted from source1 to source2, it will not be copied in dest. Parameters: source1: The original record to compute the changes from. source2: The new record that has changes over source1.

values(self)

List of names of variables in the record

const = vector<Var*>& vars()

Return the vector with the variables.

Finally, the database is accessed using a dballe.DB object:

DB

DB-ALLe database connection.

C++ includes: db.h

The methods of dballe.DB are:

db::attr& = attr()

Access the data table.

attr_insert(id_context, id_var, attrs)

id_context is a int

id_var is a Varcode

attrs is a dballe.Record

Insert new attributes into the database. If the same attribute exists for the same data, it is overwritten Parameters: id_context: The database id of the context related to the attributes to insert id_var: The varcode of the variable related to the attributes to add. See vartable::h attrs: The record with the attributes to be added

attr_insert_new(id_context, id_var, attrs)

id_context is a int

id_var is a Varcode

attrs is a dballe.Record

Insert new attributes into the database. If the same attribute exists for the same data, the function fails. Parameters: id_context: The database id of the context related to the attributes to insert id_var: The varcode of the variable related to the attributes to add. See vartable::h attrs: The record with the attributes to be added

attr_insert_or_replace(id_context, id_var, attrs, can_replace)

id_context is a int

id_var is a Varcode

attrs is a dballe.Record

can_replace is a bool

Insert new attributes into the database. Parameters: id_context: The database id of the context related to the attributes to insert id_var: The varcode of the variable related to the attributes to add. See vartable::h attrs: The record with the attributes to be added can_replace: If true, then existing data can be rewritten, else data can only be added.

attr_remove(id_context, id_var, qcs)

id_context is a int

id_var is a Varcode

qcs is a const AttrList &

Delete QC data for the variable `var' in record `rec' (coming from a previous dba_query) Parameters: id_context: The database id of the context related to the attributes to remove id_var: The varcode of the variable related to the attributes to remove. See vartable::h qcs: Array of WMO codes of the attributes to delete. If empty, all attributes associated to id_data will be deleted.

System Message: WARNING/2 (python-dballe.rst, line 199); backlink

Inline interpreted text or phrase reference start-string without end-string.

System Message: WARNING/2 (python-dballe.rst, line 199); backlink

Inline interpreted text or phrase reference start-string without end-string.

bool = check_rep_cod(rep_cod)

rep_cod is a int

Verify that a rep_cod is supported by the database. Parameters: rep_cod: The report code to verify true if the report code is supported, false if not

connect(dsn, user, password)

dsn is a str

user is a str

password is a str

Start a session with DB-All.e. Parameters: dsn: The ODBC DSN of the database to use user: The user name to use to connect to the DSN password: The password to use to connect to the DSN. To specify an empty password, pass "" or NULL Parameters: db: The dba_db handle returned by the function The error indicator for the function (See error::h)

connect_from_file(pathname)

pathname is a str

Create from a SQLite file pathname. Parameters: pathname: The pathname to a SQLite file

connect_from_url(url)

url is a str

Create from an url-like specification, that can be: sqlite:[//]foo.sqlite odbc://[user[:pass]@]dsn test:[//] Parameters: url: The url-like connection descriptor

connect_generic(config)

config is a str

Start a session with DB-All.e. Parameters: config: The string with the configuration to use to start the connection. This is passed as is to SQLDriverConnect, so see ODBC documentation for its format.

connect_test()

Start a test session with DB-All.e. Take information from the environment ( See: dba_db_create_from_env) and default to ./test.sqlite if nothing is specified.

db::context& = context()

Access the context table.

db::data& = data()

Access the data table.

delete_tables()

Delete all the DB-ALLe tables from the database.

dump(out)

out is a FILE *

Dump the entire contents of the database to an output stream.

export_msgs(query, cons)

query is a dballe.Record

cons is a MsgConsumer &

Perform the query in `query', and return the results as a NULL- terminated array of dba_msg. Parameters: query: The query to perform cons: The MsgsConsumer that will handle the resulting messages

System Message: WARNING/2 (python-dballe.rst, line 271); backlink

Inline interpreted text or phrase reference start-string without end-string.

int = get_rep_cod(rec)

rec is a Record &

Get the report id from this record. If rep_memo is specified instead, the corresponding report id is queried in the database and set as "rep_cod" in the record.

import_msg(msg, repmemo, flags)

msg is a const Msg &

repmemo is a str

flags is a int

Import a Msg message into the DB-All.e database. Parameters: db: The DB-All.e database to write the data into msg: The Msg containing the data to import repmemo: Report mnemonic to which imported data belong. If NULL is passed, then it will be chosen automatically based on the message type. flags: Customise different aspects of the import process. It is a bitmask of the various DBA_IMPORT_* macros.

import_msgs(msgs, repmemo, flags)

msgs is a const Msgs &

repmemo is a str

flags is a int

Import Msgs messages into the DB-All.e database. Parameters: db: The DB-All.e database to write the data into msgs: The Msgs containing the data to import repmemo: Report mnemonic to which imported data belong. If NULL is passed, then it will be chosen automatically based on the message type. flags: Customise different aspects of the import process. It is a bitmask of the various DBA_IMPORT_* macros.

insert(rec, can_replace, station_can_add)

rec is a Record &

can_replace is a bool

station_can_add is a bool

Insert a record into the database. In a record with the same phisical situation already exists, the function fails. ana_id and context_id will be set in the record at the end of this function. Parameters: rec: The record to insert. can_replace: If true, then existing data can be rewritten, else data can only be added. station_can_add: If true, then it is allowed to add new station records to the database. Otherwise, data can be added only by reusing existing ones.

int = last_context_insert_id()

Return the ID of the last inserted context.

int = last_station_insert_id()

Return the ID of the last inserted station.

unsigned = query_attrs(id_context, id_var, qcs, attrs)

id_context is a int

id_var is a Varcode

qcs is a const AttrList &

attrs is a Record &

Query attributes. Parameters: id_context: The database id of the context related to the attributes to retrieve id_var: The varcode of the variable related to the attributes to retrieve. See vartable::h qcs: The WMO codes of the QC values requested. If it is empty, then all values are returned. attrs: The Record that will hold the resulting attributes Number of attributes returned in attrs

remove(rec)

rec is a dballe.Record

Remove data from the database. Parameters: rec: The record with the query data (see technical specifications, par. 1.6.4 "parameter output/input") to select the items to be deleted

remove_orphans()

Remove orphan values from the database. Orphan values are currently: context values for which no data exists station values for which no context exists Depending on database size, this routine can take a few minutes to execute.

int = rep_cod_from_memo(memo)

memo is a str

Get the report code from a report mnemonic.

const = string& rep_memo_from_cod(int rep_cod)

Get the report mnemonic from a report code.

db::repinfo& = repinfo()

Access the repinfo table.

reset(repinfo_file=0)

Reset the database, removing all existing DBALLE tables and re- creating them empty. Parameters: repinfo_file: The name of the CSV file with the report type information data to load. The file is in CSV format with 6 columns: report code, mnemonic id, description, priority, descriptor, table A category. If repinfo_file is NULL, then the default of /etc/dballe/repinfo.csv is used.

db::station& = station()

Access the station table.

update_repinfo(repinfo_file, added, deleted, updated)

repinfo_file is a str

added is a int *

deleted is a int *

updated is a int *

Update the repinfo table in the database, with the data found in the given file. Parameters: repinfo_file: The name of the CSV file with the report type information data to load. The file is in CSV format with 6 columns: report code, mnemonic id, description, priority, descriptor, table A category. If repinfo_file is NULL, then the default of /etc/dballe/repinfo.csv is used. Parameters: added: The number of repinfo entryes that have been added deleted: The number of repinfo entryes that have been deleted updated: The number of repinfo entryes that have been updated

And query results are iterated using a dballe.Cursor object:

Cursor

Structure used to build and execute a query, and to iterate through the results.

C++ includes: cursor.h

The methods of dballe.Cursor are:

attributes(self, *args)

Read the attributes for the variable pointed by this record.

If a rec argument is provided, it will write the attributes in that record and return the number of attributes read. If rec is None, it will return a tuple (Record, count) with a newly created Record.

discard_rest()

Discard the results that have not been read yet.

bool = next()

Get a new item from the results of a query. true if a new record has been read, false if there is no more data to read

int = query(query, wanted, modifiers)

query is a dballe.Record

wanted is a unsigned int

modifiers is a unsigned int

Create and execute a database query. The results are retrieved by iterating the cursor. Parameters: query: The record with the query data (see technical specifications, par. 1.6.4 "parameter output/input" wanted: The values wanted in output modifiers: Optional modifiers to ask for special query behaviours The count of items in the results

unsigned = query_attrs(qcs, attrs)

qcs is a const AttrList &

attrs is a Record &

Query attributes for the current variable.

int = remaining()

Get the number of rows still to be fetched. The number of rows still to be queried. The value is undefined if no query has been successfully peformed yet using this cursor.

to_record(rec)

rec is a Record &

Fill in a record with the contents of a dba_db_cursor. Parameters: rec: The record where to store the values

dballe.Cursor is iterable and so it's rarely used outside of iteration. The normal way of getting data out of the database is:

db = dballe.DB("dbname", "user", "passwd")
query = dballe.Record()
query.seti("latmin", 10.)
query.seti("latmax", 60.)
query.seti("lonmin", -10.)
query.seti("lonmax", 40.)
query.seti("var", "B12001")
for record in db.query(query):
    print "Temperature:", record.enqvar("B12001")

Sometimes it is useful to access a cursor in order to access variable attributes:

cursor = db.query(query):
for record in cursor:
    print "Temperature:", record.enqvar("B12001")
    attrs = cursor.attributes()
    print "  Confidence interval:", attrs.enqvar("B33007")

The volnd API

volnd is an easy way of extracting entire matrixes of data out of a DB-All.e database.

This module allows to extract multidimensional matrixes of data given a list of dimension definitions. Every dimension definition defines what kind of data goes along that dimension.

Dimension definitions can be shared across different extracted matrixes and multiple extractions, allowing to have different matrixes whose indexes have the same meaning.

This example code extracts temperatures in a station by datetime matrix:

query = dballe.Record()
query["var"] = "B12001"
query["rep_memo"] = "synop"
query["level"] = (105, 2)
query["trange"] = (0,)
vars = read(self.db.query(query), (AnaIndex(), DateTimeIndex()))
data = vars["B12001"]
# Data is now a 2-dimensional Masked Array with the data
#
# Information about what values correspond to an index in the various
# directions can be accessed in data.dims, which contains one list per
# dimension with all the information corresponding to every index.
print "Ana dimension is", len(data.dims[0]), "items long"
print "Datetime dimension is", len(data.dims[1]), "items long"
print "First 10 stations along the Ana dimension:", data.dims[0][:10]
print "First 10 datetimes along the DateTime dimension:", data.dims[1][:10]

This is the list of dimensions supported by dballe.volnd:

AnaIndex

Index for stations, as they come out of the database.

The constructor syntax is: AnaIndex(shared=True, frozen=False, start=None).

The index saves all stations as AnaIndexEntry tuples, in the same order as they come out of the database.

NetworkIndex

Index for networks, as they come out of the database.

The constructor syntax is: NetworkIndex(shared=True, frozen=False, start=None).

The index saves all networks as NetworkIndexEntry tuples, in the same order as they come out of the database.

LevelIndex

Index for levels, as they come out of the database

The constructor syntax is: LevelIndex(shared=True, frozen=False), start=None.

The index saves all levels as dballe.Level tuples, in the same order as they come out of the database.

TimeRangeIndex

Index for time ranges, as they come out of the database.

The constructor syntax is: TimeRangeIndex(shared=True, frozen=False, start=None).

The index saves all time ranges as dballe.TimeRange tuples, in the same order as they come out of the database.

DateTimeIndex

Index for datetimes, as they come out of the database.

The constructor syntax is: DateTimeIndex(shared=True, frozen=False, start=None).

The index saves all datetime values as datetime.datetime objects, in the same order as they come out of the database.

IntervalIndex

Index by fixed time intervals: index points are at fixed time intervals, and data is acquired in one point only if it is within a given tolerance from the interval.

The constructor syntax is: IntervalIndex(start, step, tolerance=0, end=None, shared=True, frozen=False).

start is a datetime.datetime object giving the starting time of the time interval of this index.

step is a datetime.timedelta object with the interval between sampling points.

tolerance is a datetime.timedelta object specifying the maximum allowed interval between a datum datetime and the sampling step. If the interval is bigger than the tolerance, the data is discarded.

end is an optional datetime.datetime object giving the ending time of the time interval of the index. If omitted, the index will end at the latest accepted datum coming out of the database.

The data objects used by AnaIndex and NetworkIndex are:

AnaIndexEntry

AnaIndex entry, with various data about a single station.

It is a tuple of 4 values:

• station id
• latitude
• longitude
• mobile station identifier, or None

NetworkIndexEntry

NetworkIndex entry, with various data about a single station.

It is a tuple of 2 values:

• network code
• network name

The extraction is done using the dballe.volnd.read function:

read(cursor, dims, filter=None, checkConflicts=True, attributes=None)

cursor is a dballe.Cursor resulting from a dballe query

dims is the sequence of indexes to use for shaping the data matrixes

filter is an optional filter function that can be used to discard values from the query: if filter is not None, it will be called for every output record and if it returns False, the record will be discarded

checkConflicts tells if we should raise an exception if two values from the database would fill in the same position in the matrix

attributes tells if we should read attributes as well: if it is None, no attributes will be read; if it is True, all attributes will be read; if it is a sequence, then it is the sequence of attributes that should be read.

The result of dballe.volnd.read is a dict mapping output variable names to a dballe.volnd.Data object with the results. All the Data objects share their indexes unless the xxx-Index definitions have been created with shared=False.

This is the dballe.volnd.Data class documentation:

Data

Container for collecting variable data. It contains the variable data array and the dimension indexes.

If v is a Data object, you can access the tuple with the dimensions as v.dims, and the masked array with the values as v.vals.

The methods of dballe.volnd.Data are:

append(self, rec)

Collect a new value from the given dballe record.

You need to call finalise() before the values can be used.

appendAttrs(self, rec)

Collect attributes to append to the record.

You need to call finalise() before the values can be used.

finalise(self)

Stop collecting values and create a masked array with all the values collected so far.