loq

Appearance

Input Formats ​

loq supports 20+ input formats for parsing structured and semi-structured data.

Format Overview ​

FormatFlagDescriptionPlatform
CSV-i:CSVComma-separated values (default)All
TSV-i:TSVTab-separated valuesAll
JSON-i:JSONJSON arrays and objectsAll
NDJSON-i:NDJSONNewline-delimited JSONAll
XML-i:XMLXML documentsAll
W3C-i:W3CIIS W3C extended log formatAll
IIS-i:IISIIS native log formatAll
NCSA-i:NCSANCSA Common/Combined logsAll
Syslog-i:SYSLOGSyslog RFC 3164/5424All
EVTX-i:EVTXWindows Event Log filesAll
FS-i:FSFilesystem metadataAll
S3-i:S3Amazon S3 objectsAll
Parquet-i:PARQUETApache Parquet filesAll
TEXTLINE-i:TEXTLINELine-by-line with regexAll
TEXTWORD-i:TEXTWORDWord-by-word parsingAll
REG-i:REGWindows Registry .reg filesAll
PCAP-i:PCAPNetwork capture filesAll
BIN-i:BINIIS binary logs (.ibl)All
HTTPERR-i:HTTPERRHTTP.sys error logsAll
URLSCAN-i:URLSCANURLScan security logsAll
ETW-i:ETWEvent Tracing for WindowsWindows
ADS-i:ADSActive DirectoryWindows

Specifying Input Format ​

Use the -i:FORMAT flag:

bash
# Explicit format
loq -i:JSON "SELECT * FROM data.json"
loq -i:W3C "SELECT * FROM access.log"

# CSV is default
loq "SELECT * FROM data.csv"

Format Aliases ​

Many formats have aliases for compatibility:

CanonicalAliases
CSV(default)
W3CIISW3C
IISIISNATIVE
NCSAAPACHE, NGINX
TEXTLINETEXT
TEXTWORDWORD
FSFILESYSTEM
EVTXEVT
REGREGISTRY
PCAPNETMON, CAP
DATAGRIDTABLE, GRID

Common Columns ​

Different formats provide different columns. Here are common patterns:

Web Server Logs (W3C, IIS, NCSA) ​

ColumnDescription
dateRequest date
timeRequest time
cs-methodHTTP method (GET, POST, etc.)
cs-uri-stemRequest path
cs-uri-queryQuery string
sc-statusHTTP status code
sc-bytesResponse size
c-ipClient IP
cs(User-Agent)User agent string

System Logs (Syslog, EVTX) ​

ColumnDescription
timestampEvent time
hostnameSource host
facilityLog facility
severityLog level
messageLog message

Filesystem (FS) ​

ColumnDescription
NameFile name
PathFull path
SizeFile size in bytes
ExtensionFile extension
LastModifiedModification time
CreatedCreation time

Type Inference ​

loq automatically detects column types:

Detected TypeExamples
Integer42, -100, 0
Float3.14, -0.5, 1.0e10
Booleantrue, false, TRUE, FALSE
DateTimeISO 8601 format
StringEverything else

Override with explicit functions:

sql
SELECT
    CAST(id AS INTEGER),
    TO_TIMESTAMP(date_str, '%Y-%m-%d')
FROM data.csv

Format-Specific Options ​

Some formats support additional options:

CSV/TSV Options ​

bash
# Custom delimiter
loq -i:CSV -iSeparator:"|" "SELECT * FROM data.txt"

# Skip header rows
loq -i:CSV -iHeaderRow:2 "SELECT * FROM data.csv"

XML Options ​

bash
# Specify row element
loq -i:XML -iRowElement:item "SELECT * FROM data.xml"

Filesystem Options ​

bash
# Recursive traversal
loq -i:FS -recurse:3 "SELECT * FROM '/var/log'"

Cross-Platform Support ​

Most formats work on all platforms. Some are platform-specific:

FormatWindowsmacOSLinux
CSV, JSON, XMLYesYesYes
W3C, IIS, NCSAYesYesYes
EVTXYesYesYes
ETWYesNoNo
ADSYesNoNo

Platform-specific formats return clear error messages on unsupported platforms:

bash
# On macOS/Linux
loq -i:ETW "SELECT * FROM events"
# Error: ETW format is only supported on Windows

Examples ​

Query CSV Data ​

bash
loq "SELECT name, age FROM users.csv WHERE age > 30"

Analyze Web Logs ​

bash
loq -i:W3C "SELECT cs-uri-stem, COUNT(*) FROM access.log GROUP BY cs-uri-stem"

Parse JSON API Response ​

bash
loq -i:JSON "SELECT id, name FROM response.json WHERE status = 'active'"

Query Windows Events ​

bash
loq -i:EVTX "SELECT TimeCreated, EventID, Message FROM System.evtx WHERE Level <= 2"

List Large Files ​

bash
loq -i:FS "SELECT Name, Size FROM '/home' WHERE Size > 1000000 ORDER BY Size DESC LIMIT 20"

Query S3 Data ​

bash
loq -i:S3 "SELECT * FROM 's3://bucket/logs/*.csv' WHERE status >= 400"

See Also ​

All rights reserved.