Using monitoring tables & trace/audit for system analysis



Pavel Císař, IBPhoenix Firebird Conference 2014, Prague

Thanks for sponsoring

Problem domain Monitoring

Testing

Performance tuning

Support

Whenever you need Firebird under microscope

Difference between mon. tables and trace/audit Monitoring tables are about state(s)

Trace/audit is about dynamic action

Monitoring tables

Monitoring tables Introduced in Firebird 2.1, enhanced in 2.5 (memory usage, context variables)

Internal state snapshots (a lot can happen between them)

In comparison to trace & audit: Easy to use Lesser burden for server Generate smaller amount of data Relatively easy to process collected data



For what are they good for? Finding out what's going on right now

Suitable for monitoring trends

(Automatic) detection of (and response to) interesting states

Limitations and Known Issues In a heavily loaded system running Classic, monitoring requests may take noticeable time to execute. In the meantime, other activity (both running statements and new connection attempts) may be blocked until the monitoring request completes. (Improved in v2.1.3.)

Sometimes, an out-of-memory condition can cause monitoring requests to fail, or cause other worker processes to swap. A major source of this problem is the fact that every record in MON$STATEMENTS has a blob MON$SQL_TEXT which is created for the duration of the monitoring transaction.

Working with monitoring tables Any database administration tool (isql, FlameRobin etc.)

Use FBTraceManager from Upscene or FBScanner from IBSurgeon

FDB Python Driver (or your favorite access layer and programming language) to make your own scripts or applications

The information available Database

Attachments

Transactions

Statements

Call stack (for PSQL)

I/O statistics (DB page level)

Access statistics for records/rows

Memory usage

Values ​​of contextual variables

A lot of information, some more useful than other

MON$DATABASE DATABASE_NAME

CREATION_DATE

OLDEST_TRANSACTION (OIT)

OLDEST_ACTIVE (OAT)

OLDEST_SNAPSHOT (OST)

NEXT_TRANSACTION

SHUTDOWN_MODE

SWEEP_INTERVAL

PAGES

BACKUP_STATE

MON$ATTACHMENTS ATTACHMENT_ID

SERVER_PID

ATTACHMENT_NAME (connection string)

USER (user name)

ROLE (role name)

CHARACTER_SET_ID (attachment character set)

REMOTE_ADDRESS

REMOTE_PID

REMOTE_PROCESS (remote client process pathname)

MON$TRANSACTIONS ATTACHMENT_ID

TRANSACTION_ID

TIMESTAMP

TOP_TRANSACTION Upper limit used by the sweeper when advancing the global OIT. Transactions above this threshold are considered active. It is normally equivalent to the TRANSACTION_ID but COMMIT/ROLLBACK RETAINING will cause TOP_TRANSACTION to remain unchanged (“stuck”).

ISOLATION_MODE

LOCK_TIMEOUT

READ_ONLY

MON$STATEMENTS (prepared statements) ATTACHMENT_ID

TRANSACTION_ID

STATE

TIMESTAMP

SQL_TEXT Columns TRANSACTION_ID and TIMESTAMP contain valid values for active statements only. Colum SQL_TEXT is empty for BLR requests (for example some from ISQL).

MON$CALL_STACK Not very interesting/useful. Useful in rare cases when long runing SQL statement is PSQL and you want to know where exactly is stuck in call chain.

MON$IO_STATS STAT_GROUP 0: database 1: attachment 2: transaction 3: statement 4: call

PAGE_READS (number of page reads)

PAGE_WRITES (number of page writes)

PAGE_FETCHES (number of page fetches)

PAGE_MARKS (number of pages with changes pending)

MON$RECORD_STATS (record-level statistics) STAT_GROUP

RECORD_SEQ_READS

RECORD_IDX_READS

RECORD_INSERTS

RECORD_UPDATES

RECORD_DELETES

RECORD_BACKOUTS (of primary version due to rollback/savepoint undo)

RECORD_PURGES (of versions no longer needed)

RECORD_EXPUNGES (chain is being deleted due to deletions)

MON$MEMORY_USAGE STAT_GROUP

MEMORY_USED (number of bytes currently in use) High-level memory allocations performed by the engine from its pools. Can be useful for tracing memory leaks and for investigating unusual memory consumption and the attachments, procedures, etc. that might be responsible for it.

MEMORY_ALLOCATED (number of bytes currently allocated at the OS level) Low-level memory allocations performed by the Firebird memory manager. These are bytes actually allocated by the operating system, so it enables the physical memory consumption to be monitored.

MON$MEMORY_USAGE (cont.) Not all records have non-zero values. On the whole, only MON$DATABASE and memory-bound objects point to non-zero “allocated” values. Small allocations are not allocated at this level, being redirected to the database memory pool instead.

MON$CONTEXT_VARIABLES ATTACHMENT_ID

TRANSACTION_ID

VARIABLE_NAME

VARIABLE_VALUE

Useful operations (categories) Finding out what's going on right now

Monitoring database parameters

Monitoring attachment parameters

Monitoring memory consumption

Detecting long running queries and transactions

Finding the GC blocking transaction (and application)

Page cache utilisation

Data access/manipulation patterns

Cancel a running query or terminate a client

Finding out what's going on right now Using raw SQL is cumbersome

All GUI tools have problems with one-size-fits-all syndrome and are full of data clutter

FDB Python Driver provides flexible yet easy to use access paradigm

PowerConsole - Enhanced python interpreter that can host user defined commands

Finding out - Using FDB Monitor class in fdb.monitor

Access to monitoring data as objects with context db attachments & this_attachment transactions statements iostats variables callstack

Each Connection object has lazy-loaded monitor attribute, or you can bind Monitor instances to any Connection.

Finding out - PowerConsole With Firebird add-on (pwcfb) provides combined power of ISQL, Python shell and FDB.

Supports all valid DSQL statements and almost all ISQL commands.

Can work simultaneously with multiple attached databases.

You can fill in parts of SQL statements using Python expressions.

You may use SQL statements in Python code.

You can get access to executed statements from Python code (as cursor() object).

You can use parametrized SQL statemets and feed values to them from any Python iterable.

Finding out - PowerConsole examples

1

2

Other useful operations

Monitoring database parameters Manual sweep control (SWEEP_INTERVAL, OIT, OST)

Number of transactions over time (NEXT_TRANSACTION)

Oldest active transaction (OAT, transaction.TIMESTAMP, attachment.REMOTE_*)

Shutdown (SHUTDOWN_MODE)

Database growth (PAGES)

NBACKUP states (BACKUP_STATE). Stalled!

Old database (backup & restore needed) (CREATION_DATE)

Monitoring attachment parameters Number of (prepared) statements

Number of attachments over time

Duration of attachments

Connection charset anomalies

Monitoring memory consumption Context (database - SS, attachment - CS/SC)

MEMORY_ALLOCATED, MEMORY_USED

Detecting long running queries and transactions statement and transaction TIMESTAMP

Finding the GC blocking transaction (and application) Database OAT holds the transaction ID

Transaction TIMESTAMP helps to find out how long

Attachment contains information about application

Page cache utilisation Context (database - SS, attachment - CS/SC)

PAGE_READS, PAGE_WRITES, PAGE_FETCHES, PAGE_MARKS

Data access/manipulation patterns Context (database, attachment, transaction, statement)

RECORD_SEQ_READS, RECORD_IDX_READS, RECORD_INSERTS, RECORD_UPDATES, RECORD_DELETES

Cancel a running query or terminate a client DELETE row(s) from MON$ATTACHMENT / MON$STATEMENTS

FDB provides terminate() mothod on AttachmentInfo & StatementInfo objects

It makes sense to wrap them into an application

Introducing FBMon Python CLI application (for now)

Uses fdb driver / fdb.monitor module

Can make periodic snapshots

Flexible printout of data from monitoring tables

Logs selected data in various formats

Perform user-defined checks that can trigger actions

Supports actions of many types

Functions defined by configuration...

...and customized with CLI options

Flexible printout of data Datasources: monitor, db, attachments, transactions, statements, variables

Flexible output specification

Can handle valid Python expressions

Also provided functions: min, max, count, diff, changed, abb, attr, this_filter

Full, smart or custom column names

Output configuration - example [transactions] ; 'collect' : default columns collect = row_id as #, id, attachment.id, timestamp, top, isolation_mode, lock_timeout, isactive(), isreadonly() ; 'collect_x' : numbered column groups collect_1 = snapshot.id, snapshot.timestamp collect_2 = id, attachment.id collect_3 = timestamp collect_4 = snapshot.timestamp - this.timestamp as duration collect_5 = top, isolation_mode, lock_timeout, isactive(), isreadonly() collect_6 = count(statements), count(variables) collect_7 = iostats.backouts, iostats.purges, iostats.expunges

1

2

Logging data Arbitrary number of (simultaneous) logs

Datasources: monitor, db, attachments, transactions, statements, variables

Flexible output specification

Can handle valid Python expressions

Full, smart or custom column names

Flexible output filename specification

Formats: list, dict, csv, csv_hdr (database is not clever but in fact a very stupid option)

Modes: overwrite, append, rotate, append_rotate

Log configuration - example [log_trans_count] dataset = db collect = snapshot.id, snapshot.timestamp, next_transaction, diff(next_transaction) as trans_count filename = %(tr_logs)s/trans_count.log format = list mode = append_rotate [log_statement_count] dataset = monitor collect = snapshot.id, snapshot.timestamp, count(this.statements) as st_count, count([x for x in this.statements if not x.transaction]) as pr_count filename = %(here)s/st_count.log format = csv_hdr mode = append_rotate

Log visualisation - example

User-defined checks Performed on each item in specified dataset to trigger action(s)

Arbitrary boolean expression

Associated message customizable with live data

User-defined extra data or variables usable in check and message, variables are also passed to actions

Can trigger multiple actions

Can enable/disable checks and logs using schedule action class (explained later)

Check configuration - examples [check_long_running_transaction] dataset = transactions ; Report any trans. longer than 60 minutes but do not report OAT check = (vars.oat.id != this.id) and (snapshot.timestamp > this.timestamp + datetime.timedelta(minutes=60)) vars = mon.get_transaction(mon.db.oat) as oat actions = report message = "Long running transaction %%d (%%s) from %%s:%%s" %% (this.id, snapshot.timestamp - this.timestamp, this.attachment.remote_address, this.attachment.remote_process) [check_sweep] dataset = db check = (this.sweep_interval == 0) and (this.ost - this.oit >= 100000) actions = sweep, report message = "Manual sweep needed (gap %%d)" %% (this.ost - this.oit)

And action! Various classes of actions (like print a message, run a program, terminate client/query, enable/disable checks and logs etc.)

Action definition is specificaly configured instance of particular action class

Different checks can trigger the same action definition that runs independently

Checks can pass context specific data to action (runner)

Action runner is notified by linked check about each check run outcome and schedule execution of action

Scheduled actions are executed as last step in snapshot processing

Action scheduling By default, action executes every time when check evaluates to True.

Could be affected by configuration: delay : Action executes only after X sucessful checks in a row. remind : Action executes only after X sucessful checks in a row since last execution. calm : Action doesn't execute for X checks after last execution regardless of check outcome.

Actions could be synchronous or asynchronous.

While async action is executed, all check outcomes are noted but ignored by action scheduler.

Action lifetime Action runners are associated with Action, Check and particular ID from dataset that check uses.

For example check that acts on state of particular transaction (transactions dataset) uses runners identified by particular action, itself and actual transaction id processed.

Runner may schedule execution according to previous check outcomes, so it stays alive until it's not needed anymore.

Timeout is reset any time runner is notified by check, and at the end of snapshot processing all runners that expire are deleted.

Action configuration - examples [action_sweep] class = program run_mode = async command = /opt/firebird/bin/gfix -sweep -user %(user)s -password %(password)s %(database)s calm = 10 [action_alert] class = report template = ALERT: %%(message)s remind = 5 [action_terminate_client] class = terminate mode = attachment delay = 10 timeout = 1

What could be done with checks & actions Simple reporting of interesting states (like connection charset NONE) that enhance the regular output

To automatically run Firebird tools like gfix, send SMS etc.

Can start/stop logging data when something interesting happens outside working hours for later analysis.

It's even possible to define simple state machines

It's possible to define fine-grained problem escalation using delay/remind/calm options

That's all for monitoring tables...

Thanks for sponsoring

Trace & audit services

Trace & audit Introduced in Firebird 2.5

Continuous trace of events (but you can still miss some in certain conditions)

In comparison to monitoring tables: Much harder to use Could be bigger burden for server Easily generates a lot of data (for example: work of ten simultaneous clients generated 1GB of data in 3 minutes = ~0.5TB per day) Very hard to process collected data



For what it is good for? Performance tuning

System analysis

(Automatic) detection of (and response to) interesting events

User trace vs. System Audit User trace Can run multiple sessions with separate configuration, state and output Managed via Firebird services

System Audit Only one audit log Managed by engine (conf. change needs restart)



Limitations and Known Issues User trace output is stored in temporary files (1 MB each). Once a file has been completely read by the application, it is automatically deleted. By default, the maximum total size of the output is limited to 10 MB. It can be changed to a smaller or larger value using the MaxUserTraceLogSize in firebird.conf. When output buffer is full, events are not recorded and session is suspended until space is created in output buffer.

Default trace plugin output is very verbose and mostly unstructured. While internal Trace API exists that provides a set of hooks which can be implemented as an external plug-in module to be called by the engine when any traced event happens, it is not officially published and must be regarded as unstable.

Trace output example 2014-05-28T11:21:04.2360 (5776:0000ED5470) EXECUTE_PROCEDURE_FINISH TEST_DB (ATT_10, SERVER:NONE, ISO88591, TCPv4:192.168.1.5) D:\TEST_DB\App\app.exe:2644 (TRA_1596, READ_COMMITTED | REC_VERSION | WAIT | READ_WRITE) Procedure SOMEPROC: param0 = varchar(10), "XXX" param1 = double precision, "313210" param2 = double precision, "24" param3 = integer, "3" 1 ms, 15 fetch(es) Table Natural Index Update Insert Delete Backout Purge Expunge ******************************************************************* TABLE_001 1 TABLE_002 1 TABLE_003 1

Working with user trace sessions (and Audit) fbsvcmgr tool (very cumbersome)

Use FBTraceManager from Upscene

Latest FBScanner from IBSurgeon provides alternate Trace API plugin

Write your own tool(s)

The information available Trace (INIT, SUSUPEND, FINISH)

Connections (ATTACH, DETACH)

Transactions (START, COMMIT, ROLLBACK)

Statements (PREPARE, EXECUTE_START, EXECUTE_FINISH, CLOSE_CURSOR, FREE)

Triggers (START, FINISH)

Stored procedures (START, FINISH)

Sweep (START, PROGRESS, FINISH) - added in 2.5.2

Errors

Context variables

BLR & DYN

Information about Trace sessions Not really interesting but important First TRACE_INIT will tell you the session ID Intermediate TRACE_SUSPEND / INIT will tell you that you missed some events Sets a frame for automatic log parsing

2014-05-28T11:21:03.4560 (5776:0000000000ED67A0) TRACE_INIT SESSION_1 Firebird Audit --- Session 1 is suspended as its log is full --- 2014-05-23T11:01:24.8080 (3720:0000000000EFD9E8) TRACE_INIT SESSION_1 2014-05-23T11:01:24.8080 (3720:0000000000EFD9E8) TRACE_FINI SESSION_1

Connections Database, ID, user, role, charset, protocol, address, remote_process

Signals Unauthorized attachment attempts. 2014-05-28T11:21:03.4870 (5776:0000000000ED5470) ATTACH_DATABASE TEST_DB (ATT_10, SERVER:NONE, ISO88591, TCPv4:192.168.1.5) D:\TEST\App\MyApp.exe:2644 2014-09-24T14:46:15.0350 (2453:2a04910) UNAUTHORIZED ATTACH_DATABASE /home/db/employee25.fdb (ATT_0, sysdba, NONE, TCPv4:127.0.0.1) /opt/firebird/bin/isql:8723 2014-05-23T11:01:24.8080 (3720:0000000000EFD9E8) DETACH_DATABASE TEST_DB (ATT_8, SERVER:NONE, ISO88591, TCPv4:192.168.1.5) D:\TEST\App\MyApp.exe:6072

Transactions ID, attachment, options, run_time (ms)

reads, writes, fetches, marks

RETAINING as separate type of event messages 2014-05-23T11:00:28.6160 (3720:0000000000EFD9E8) START_TRANSACTION TEST_DB (ATT_8, SERVER:NONE, ISO88591, TCPv4:192.168.1.5) D:\TEST\App\MyApp.exe:6072 (TRA_1568, READ_COMMITTED | REC_VERSION | WAIT | READ_WRITE) 2014-05-23T11:00:29.9570 (3720:0000000000EFD9E8) COMMIT_TRANSACTION TEST_DB (ATT_8, SERVER:NONE, ISO88591, TCPv4:192.168.1.5) D:\TEST\App\MyApp.exe:6072 (TRA_1568, READ_COMMITTED | REC_VERSION | WAIT | READ_WRITE) 0 ms, 1 read(s), 1 write(s), 1 fetch(es), 1 mark(s)

Statements ID, attachment, transaction, sql_text, prepare_time (ms), execute_time (ms), parameters, records_fetched, reads, writes, fetches, marks

Upon request: plan, table access details 2014-05-28T11:21:03.7680 (5776:0000000000ED5470) PREPARE_STATEMENT TEST_DB (ATT_10, SERVER:NONE, ISO88591, TCPv4:192.168.1.5) D:\TEST\App\MyApp.exe:2644 (TRA_1596, READ_COMMITTED | REC_VERSION | WAIT | READ_WRITE) Statement 110: ------------------------------------------------------------------------------- EXECUTE PROCEDURE PROC_001('37720', '3', '1', 'LIL') 269 ms

Triggers attachment, transaction, trigger_name, table, event, run_time (ms), reads, writes, fetches, marks

Upon request: table access details 2014-05-28T11:21:03.7680 (5776:0000ED5470) EXECUTE_TRIGGER_START TEST_DB (ATT_10, SERVER:NONE, ISO88591, TCPv4:192.168.1.5) D:\TEST\App\MyApp.exe:2644 (TRA_1596, READ_COMMITTED | REC_VERSION | WAIT | READ_WRITE) BI_TRIGGER_001 FOR TABLE_001 (BEFORE INSERT) 2014-05-28T11:21:03.7680 (5776:0000ED5470) EXECUTE_TRIGGER_FINISH TEST_DB (ATT_10, SERVER:NONE, ISO88591, TCPv4:192.168.1.5) D:\TEST\App\MyApp.exe:2644 (TRA_1596, READ_COMMITTED | REC_VERSION | WAIT | READ_WRITE) BI_TRIGGER_001 FOR TABLE_001 (BEFORE INSERT) 0 ms, 1 fetch(es), 1 mark(s)

Stored procedures attachment, transaction, proc_name, parameters, run_time (ms), reads, writes, fetches, marks

Upon request: table access details 2014-05-28T11:21:03.7680 (5776:00000ED5470) EXECUTE_PROCEDURE_FINISH TEST_DB (ATT_10, SERVER:NONE, ISO88591, TCPv4:192.168.1.5) D:\TEST\App\MyApp.exe:2644 (TRA_1596, READ_COMMITTED | REC_VERSION | WAIT | READ_WRITE) Procedure PROC_001: param0 = varchar(50), "758755" param1 = varchar(10), "LIL" 0 ms

Errors Basically the same thing that's written to firebird.log with some additinal information. 2014-09-24T13:58:48.7230 (2453:04910) ERROR AT jrd8_attach_database /home/db/BADDB.FDB (ATT_0, sysdba, NONE, TCPv4:127.0.0.1) /opt/firebird/bin/isql:8174 335544335 : database file appears corrupt (/home/db/BADDB.FDB) 335544650 : wrong page type 335544403 : page 0 is of wrong type (expected 3, found 1)

Sweep attachment, transaction counters, run_time, reads, writes, fetches, marks, table access details (upon request)

Can signal FAILed sweep attempts 2014-09-24T13:49:17.1970 (2453:0x7fed02a04910) SWEEP_START /home/db.fdb (ATT_127, SYSDBA:NONE, NONE, TCPv4:127.0.0.1) /opt/firebird/bin/gfix:7892 Transaction counters: Oldest interesting 1063 Oldest active 1064 Oldest snapshot 1064 Next transaction 1065 2014-09-24T13:49:17.2240 (2453:0x7fed02a04910) SWEEP_PROGRESS /home/db.fdb (ATT_127, SYSDBA:NONE, NONE, TCPv4:127.0.0.1) /opt/firebird/bin/gfix:7892 0 ms, 5 fetch(es)

BLR & DYN You will never need those unless you're Firebird core developer or GPRE user. Note: BLR & DYN records could be quite long, so set max_blr_length & max_dyn_length trace options accordingly.

Practical usage of Trace & Audit

Operational modes Problem solving

Preventing problems

General advices Plan your strategy carefuly

Ask only for information you really need

When using User Trace, always increase log size (100 or 500 MB is number where you want to start)

Get data out as fast as you can, analyze later

Use Audit when you need a lot of information

Typically xxx_FINISH events are most interesting, but you need also xxx_START ones if you want to see execution trees

events are most interesting, but you need also ones if you want to see execution trees Analysis becomes more "interesting" with concurrent clients, with multiple databases it starts to be a nightmare

Useful operations (categories) Trace specific activities

Black-box application analysis / system profiling

System audit

Trace specific activities Transactions, statements, triggers and stored procedures

Only slow statements (time_threshold configuration option)

Execution PLAN

Performance counters (they are cummulative, so watch for trees)

Execution trees

It's easiest to work with trace when you focus only on specific detail, but don't forget about context

Complexity steeply increases with level of context and detail

System profiling Requires a diploma from the University of Trace & Audit

You need full audit trail of application's activity

of application's activity Which means a lot of data (gigabytes)

(gigabytes) Raw logs are completely useless . You need parsed, structured data.

. You need parsed, structured data. Size matters . Dense data that can fit in memory are always a winner.

. Dense data that can fit in memory are always a winner. Everything will take more time than you're used to

Top-down analysis from general view to details, a lot of slicing, comparing and visualisation

There is no general recipe as each analysis is more or less unique, so you need very flexible tools (i.e. data in database will not really help you)

System audit Focus only on essentials (security, reliability)

Attachments (especially Unauthorized ones)

ones) Errors (including sweep errors)

If you need controled auditable access to secure resources (data), wrap them in stored procedures and log their use

How I use Trace & Audit Python & FDB single purpose custom scripts for specialities (used/unused indices etc.)

mtrace & strace helper applications to work with user trace sessions

trace Python library for trace log processing

PowerConsole & IPython and other tools as needed

Single purpose custom scripts In Python it's piece of cake: FDB provides convenient access to Firebird Services, including work with user trace sessions

Python is great for work with any kind of data (scientists love it for good reasons) Example: Configurable CLI application that collects execution PLANs over time and creates report of usage frequency and list of used / unused indices has only 115 lines of Python code.

strace.py strace is universal tool for User Trace logging to screen or text file for later processing. usage: strace.py [-h] [-o HOST] [-u USER] [-p PASSWORD] [-i INCLUDE] [-e EXCLUDE] [--connections] [--transactions] [--statements {prepare,plan,start,finish,free} [{prepare,plan,start,finish,free} ...]] [--procedures {start,finish} [{start,finish} ...]] [--triggers {start,finish} [{start,finish} ...]] [--perf] [--context] [--errors] [--sweep] [--blr] [--blr-requests] [--dyn] [--dyn-requests] [--time-threshold TIME_THRESHOLD] [--max-sql-length MAX_SQL_LENGTH] [--max-blr-length MAX_BLR_LENGTH] [--max-dyn-length MAX_DYN_LENGTH] [--max-arg-length MAX_ARG_LENGTH] [--max-arg-count MAX_ARG_COUNT] database [FILENAME]

mtrace.py You can't easily stop code that runs user trace session. Best way is to use other process to stop it. mtrace can list, suspend, resume or stop user trace sessions. usage: mtrace.py [-h] [-o HOST] [-u USER] [-p PASSWORD] [-l] [-s SESSION_ID] [-r SESSION_ID] [-c SESSION_ID]

trace Python library Converts raw verbose Trace & Audit text log into more dense and manageable Python objects

Creates aggregated information records (objects) for attachments, transactions and statements from recorded events

Creates streamlined list of events (used mostly for reference as time axis)

Processed log could be saved for later use to save parsing time reduces log size significantly



trace Python library (cont.) Example Audit log: Scope: everything except BLR & DYN

Size: 550MB

Trace events: 915 981

Attachments: 20

Transactions: 65

Statements: 73 283

Parsing time: 02:49

Parsed log size (on disk): 9.3MB

(Re)load time: 01:34

Interactive work with parsed trace

Attachments

Transactions

Statements

Table access details

TraceEventSequence(s) It's a list, so it could be iterated, sliced, extended (merged with other sequences) etc.

Additional data: start & end time, duration, total prepare and run-time

Special operations: Slice by time (between) filter in & out event types Access to sets of attachments, transactions and statements referenced by events Grouping by attachment, transaction, statement, arbitrary attribute or expression (remember to fbmon?)



TraceEventGroups Important for separation of parallel activities

...or as another dimension of slicing

Basically it's a dictionary of EventSequences where key is an attribute or expression

Also provides counter for frequency analysis and other operations

TraceEventGroup

Tree of events Relationship between events is very important because all performance counters are cummulative

You may also don't know (and thus want to discover) dependencies and execution trees

But you have to deduce that yourself from shared properties and timeline analysis (start / finish "brackets")

trace library has built-in support for conversion of event sequences to trees

Requirements: All events in sequence must came from single execution thread Start and finish events must be present



1

2

Here begins the real fun but I'm pretty sure that we have run out of time