Labeled Tab-separated Values

Description

Labeled Tab-separated Values (LTSV) format is a variant ofTab-separated Values (TSV). Each record in a LTSV file is represented asa single line. Each field is separated by TAB and has a label and avalue. The label and the value have been separated by ':'.With the LTSV format, you can parse each line by spliting with TAB(like original TSV format) easily, and extend any fields withunique labels in no particular order.

FAQ

Example

The LTSV format originally focuses on access logs of web servers, so I'll show an access log of traditional Combined Log Format and the same log of LTSV format version as examples.

The configuration of traditional Combined Log Format on Apache is:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined

and access log will look like: (ref. http://httpd.apache.org/docs/2.2/logs.html)

127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08 [en] (Win98; I ;Nav)"

The configuration of LTSV format with the same infomation will be:

LogFormat "host:%h\tident:%l\tuser:%u\ttime:%t\treq:%r\tstatus:%>s\tsize:%b\treferer:\%{Referer}i\tua:%{User-Agent}i" combined_ltsv

then the access log will be like:

host:127.0.0.1<TAB>ident:-<TAB>user:frank<TAB>time:[10/Oct/2000:13:55:36 -0700]<TAB>req:GET /apache_pb.gif HTTP/1.0<TAB>status:200<TAB>size:2326<TAB>referer:http://www.example.com/start.html<TAB>ua:Mozilla/4.08 [en] (Win98; I ;Nav)

Here is a simple LTSV parser:

#!/usr/bin/env rubywhile gets  record = Hash[$_.split("\t").map{|f| f.split(":", 2)}]  p recordend

With this parser, you will get the hash like:

{"host"=>"127.0.0.1", "ident"=>"-", "user"=>"frank", "time"=>"[10/Oct/2000:13:55:36 -0700]", "req"=>"GET /apache_pb.gif HTTP/1.0", "status"=>"200", "size"=>"2326", "referer"=>"http://www.example.com/start.html", "ua"=>"Mozilla/4.08 [en] (Win98; I ;Nav)\n"}

Definition

A LTSV file must be a byte sequence whichmatches the ltsv production in the following ABNF:

ltsv = *(record NL) [record]record = [field *(TAB field)]field = label ":" field-valuelabel = 1*lbytefield-value = *fbyteTAB = %x09NL = [%x0D] %x0Albyte = %x30-39 / %x41-5A / %x61-7A / "_" / "." / "-" ;; [0-9A-Za-z_.-]fbyte = %x01-08 / %x0B / %x0C / %x0E-FF

Recommendations for labeling

The specification of LTSV is simple and primitive. Nevertheless label standardization may help to improve reusability of some implementations for processing or analysis.

Labels for Web server's Log

Here are labeling recommendations, their descriptions, format strings for apache and ones for nginx.

A LogFormat example for Apache mod_log_config.

LogFormat "time:%t\tforwardedfor:%{X-Forwarded-For}i\thost:%h\treq:%r\tstatus:%>s\tsize:%B\treferer:%{Referer}i\tua:%{User-Agent}i\treqtime_microsec:%D\tcache:%{X-Cache}o\truntime:%{X-Runtime}o\tvhost:%{Host}i" ltsv

A log_format example for nginx.

log_format ltsv "time:$time_local"                "\thost:$remote_addr"                "\tforwardedfor:$http_x_forwarded_for"                "\treq:$request"                "\tstatus:$status"                "\tsize:$body_bytes_sent"                "\treferer:$http_referer"                "\tua:$http_user_agent"                "\treqtime:$request_time"                "\tcache:$upstream_http_x_cache"                "\truntime:$upstream_http_x_runtime"                "\tvhost:$host";

Tools supporting LTSV

fluentd

fluentd (http://fluentd.org/) supports to parse a LTSV file with in_tail plugin.The configuration is like this:

<source>  type tail  format ltsv  time_format %d/%b/%Y:%H:%M:%S %z  path /var/log/nginx/access_log  pos_file /var/log/nginx/access_log.pos  tag nginx.access</source>

plugins for fluentd

ltsview

Plack::Middleware::AxsLog

combined2ltsv.pl

MCombined2LTSV.java

Parser Implementations

Perl

Ruby

Python

PHP

Java

D

Dart

Emacs Lisp

Scheme

node.js

Erlang

C#

Go

Clojure

Scala

bash / ksh

Vim

C89

Apache Pig

Apache Hive