death and gravity: Announcing linesieve: an unholy blend of grep, sed, awk, and Python :

Reader
April 25, 2023
death and gravity: Announcing linesieve: an unholy blend of grep, sed, awk, and Python
by:
blow post content copied from  Planet Python
click here to view original post

Java is notoriously verbose, especially when used in a serious Enterprise Project™.

...so naturally, I made linesieve, a Python tool to split output into sections, show only the relevant ones, and filter text with Python regular expressions.

Here's an example of using it on a file listing (delay added to make it look cool):

ls -1 /* | linesieve -s '.*:' show bin match ^d head -n2 output ls -1 /* | linesieve -s '.*:' show bin match ^d head -n2 output

Features #

linesieve allows you to:

  • split text input into sections
  • apply filters to specific sections
  • search and highlight success/failure markers
  • match/sub/split with the full power of Python's re
  • shorten paths, links and module names
  • chain filters into pipelines
  • color output!

Installing #

Install it using pip:

$ pip install --upgrade linesieve

linesieve is released under the BSD license, its documentation lives at Read the Docs, the code on GitHub, and the latest release on PyPI.

Examples #

If linesieve looks interesting to you, here are a few advanced examples.

Java tracebacks #

Assume you're writing some Java tests with JUnit, on a project that looks like this:

$
.
├── src
│   └── com
│       └── example
│           └── someproject
│               └── somepackage
│                   └── ThingDoer.java
└── tst
    └── com
        └── example
            └── someproject
                └── somepackage
                    └── ThingDoerTest.java

This command:

linesieve \
span -v -X \
    --start '^ (\s+) at \s ( org\.junit\. | \S+ \. reflect\.\S+\.invoke )' \
    --end '^ (?! \s+ at \s )' \
    --repl '\1...' \
match -v '^\s+at \S+\.(rethrowAs|translateTo)IOException' \
sub-paths --include '{src,tst}/**/*.java' --modules-skip 1 \
sub -X '^( \s+ at \s+ (?! .+ \.\. | com\.example\. ) .*? ) \( .*' '\1' \
sub -X '^( \s+ at \s+ com\.example\. .*? ) \ ~\[ .*' '\1' \
sub -X '
    (?P<pre> \s+ at \s .*)
    (?P<cls> \w+ )
    (?P<mid> .* \( )
    (?P=cls) \.java
    (?P<suf> : .* )
    ' \
    '\g<pre>\g<cls>\g<mid>\g<suf>'

... shortens this 76 line traceback:

12:34:56.789 [main] ERROR com.example.someproject.somepackage.ThingDoer - exception while notifying done listener
java.lang.RuntimeException: listener failed
    at com.example.someproject.somepackage.ThingDoerTest$DummyListener.onThingDone(ThingDoerTest.java:420) ~[tests/:?]
    at com.example.someproject.somepackage.ThingDoer.doThing(ThingDoer.java:69) ~[library/:?]
    at com.example.otherproject.Framework.doAllTheThings(Framework.java:1066) ~[example-otherproject-2.0.jar:2.0]
    at com.example.someproject.somepackage.ThingDoerTest.listenerException(ThingDoerTest.java:666) ~[tests/:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
    ...
    ... 60+ more lines of JUnit stuff we don't really care about ...
    ...
12:34:56.999 [main] INFO done

... to just:

12:34:56.789 [main] ERROR ..ThingDoer - exception while notifying done listener
java.lang.RuntimeException: listener failed
    at ..ThingDoerTest$DummyListener.onThingDone(:420) ~[tests/:?]
    at ..ThingDoer.doThing(:69) ~[library/:?]
    at com.example.otherproject.Framework.doAllTheThings(:1066)
    at ..ThingDoerTest.listenerException(:666) ~[tests/:?]
    ...
12:34:56.999 [main] INFO done

Let's break that linesieve command down a bit:

  • The span gets rid of all the traceback lines coming from JUnit.
  • The match -v skips some usually useless lines from stack traces.
  • The sub-paths shortens and highlights the names of classes in the current project; com.example.someproject.somepackage.ThingDoer becomes ..ThingDoer (presumably that's enough info to open the file).
  • The first sub gets rid of line numbers and JAR names for everything that is not either in the current project or in another com.example. package.
  • The second sub gets rid of JAR names for things in other com.example. packages.
  • The third sub gets rid of the source file name; ..ThingDoer.doThing(ThingDoer.java:69) becomes ..ThingDoer.doThing(:69) (the file name matches the class name).

Apache Ant output #

Let's look at why linesieve was born in the first place – cleaning up Apache Ant output. We'll use Ant's own test output as an example, since it builds itself.

Running a single test with:

ant junit-single-test -Djunit.testcase=org.apache.tools.ant.ProjectTest

... produces 77 lines of output:

Buildfile: /Users/lemon/code/ant/build.xml

check-optional-packages:

prepare:

compile:

compile-jdk9+:

build:
[delete] Deleting directory /Users/lemon/code/ant/build/classes/org/apache/tools/ant/taskdefs/optional/junitlauncher/confined
        ... more lines

... more targets, until we get to the one that we care about

junit-single-test-only:
    [junit] WARNING: multiple versions of ant detected in path for junit
    [junit]          file:/Users/lemon/code/ant/build/classes/org/apache/tools/ant/Project.class
    [junit]      and jar:file:/usr/local/Cellar/ant/1.10.12/libexec/lib/ant.jar!/org/apache/tools/ant/Project.class
    [junit] Testsuite: org.apache.tools.ant.ProjectTest
    [junit] Tests run: 12, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 5.635 sec
        ... more lines

junit-single-test:

BUILD SUCCESSFUL
Total time: 12 seconds

If this doesn't look all that bad, imagine what it looks like for an Enterprise Project™.

Lots of output is indeed very helpful – if you're waiting minutes for the entire test suite to run, you want all the details in there, so you can debug failures without having to run it again.

However, it's not very helpful during development, when you only care about the thing you're working on right now. And it's doubly not helpful if you want to re-run the tests on each file update with something like entr.

This is where a linesieve wrapper script can help:

#!/bin/sh
linesieve \
    --section '^(\S+):$' \
    --success 'BUILD SUCCESSFUL' \
    --failure 'BUILD FAILED' \
show junit-batch \
show junit-single-test-only \
sub-cwd \
sub-paths --include 'src/main/**/*.java' --modules-skip 2 \
sub-paths --include 'src/tests/junit/**/*.java' --modules-skip 3 \
sub -s compile '^\s+\[javac?] ' '' \
push compile \
    match -v '^Compiling \d source file' \
    match -v '^Ignoring source, target' \
pop \
push junit \
    sub '^\s+\[junit] ?' '' \
    span -v \
        --start '^WARNING: multiple versions of ant' \
        --end '^Testsuite:' \
    match -v '^\s+at java\.\S+\.reflect\.' \
    match -v '^\s+at org.junit.Assert' \
    span -v \
        --start '^\s+at org.junit.(runners|rules|internal)' \
        --end '^(?!\s+at )' \
pop \
sub -X '^( \s+ at \s+ (?! .+ \.\. ) .*? ) \( .*' '\1' \
sub -X '
    (?P<pre> \s+ at \s .*)
    (?P<cls> \w+ )
    (?P<mid> .* \( )
    (?P=cls) \.java
    (?P<suf> : .* )
    ' \
    '\g<pre>\g<cls>\g<mid>\g<suf>' \
sub --color -X '^( \w+ (\.\w+)+ (?= :\s ))' '\1' \
sub --color -X '(FAILED)' '\1' \
read-cmd ant "$@"

You can then call this instead of ant: ant-wrapper.sh junit-single-test ....

Successful output looks like this (28 lines):

............
junit-single-test-only
Testsuite: ..ProjectTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 5.635 sec
------------- Standard Output ---------------
bar
------------- ---------------- ---------------
------------- Standard Error -----------------
bar
------------- ---------------- ---------------

Testcase: testResolveFileWithDriveLetter took 0.034 sec
    SKIPPED: Not DOS or Netware
Testcase: testResolveFileWithDriveLetter took 0.036 sec
Testcase: testInputHandler took 0.007 sec
Testcase: testAddTaskDefinition took 0.179 sec
Testcase: testTaskDefinitionContainsKey took 0.002 sec
Testcase: testDuplicateTargets took 0.05 sec
Testcase: testResolveRelativeFile took 0.002 sec
Testcase: testOutputDuringMessageLoggedIsSwallowed took 0.002 sec
Testcase: testDataTypes took 0.154 sec
Testcase: testDuplicateTargetsImport took 0.086 sec
Testcase: testNullThrowableMessageLog took 0.002 sec
Testcase: testTaskDefinitionContainsValue took 0.002 sec
Testcase: testResolveFile took 0.001 sec

.
BUILD SUCCESSFUL

... "failure" output looks like this (34 lines):

............
junit-single-test-only
Testsuite: ..ProjectTest
Tests run: 12, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 5.638 sec
------------- Standard Output ---------------
bar
------------- ---------------- ---------------
------------- Standard Error -----------------
bar
------------- ---------------- ---------------

Testcase: testResolveFileWithDriveLetter took 0.033 sec
    SKIPPED: Not DOS or Netware
Testcase: testResolveFileWithDriveLetter took 0.035 sec
Testcase: testInputHandler took 0.005 sec
    FAILED
expected null, but was:<..DefaultInputHandler@61dc03ce>
junit.framework.AssertionFailedError: expected null, but was:<..DefaultInputHandler@61dc03ce>
    at ..ProjectTest.testInputHandler(:254)

Testcase: testAddTaskDefinition took 0.182 sec
Testcase: testTaskDefinitionContainsKey took 0.003 sec
Testcase: testDuplicateTargets took 0.043 sec
Testcase: testResolveRelativeFile took 0.001 sec
Testcase: testOutputDuringMessageLoggedIsSwallowed took 0.003 sec
Testcase: testDataTypes took 0.161 sec
Testcase: testDuplicateTargetsImport took 0.088 sec
Testcase: testNullThrowableMessageLog took 0.001 sec
Testcase: testTaskDefinitionContainsValue took 0.001 sec
Testcase: testResolveFile took 0.001 sec
Test ..ProjectTest FAILED

.
BUILD SUCCESSFUL

... and true failure due to a compile error looks like this (12 lines):

...
compile
.../Project.java:65: error: cannot find symbol
public class Project implements xResourceFactory {
                                ^
symbol: class xResourceFactory
.../Project.java:2483: error: method does not override or implement a method from a supertype
    @Override
    ^
2 errors

BUILD FAILED

Breaking down the linesieve command (skipping the parts from the traceback example):

  • --section '^(\S+):$' tells linesieve sections start with a word followed by a colon.
  • The shows hide all sections except specific ones.
  • --success and --failure tell linesieve to exit when encountering one of these patterns. Note that the failing section is shown regardless of show.
  • sub-cwd makes absolute paths in the working directory relative.
  • The -s compile option passed to sub applies it only to sections matching compile.
  • push compile applies all the following filters, until pop, only to sections matching compile.
  • The last two sub --color ... '\1' color dotted words followed by a colon at the beginning of the line (e.g. junit.framework.AssertionFailedError:), and FAILED anywhere in the input.
  • Finally, read-cmd executes a command and uses its output as input.

Anyway, that's it for now.

Learned something new today? Share this with others, it really helps!


April 25, 2023 at 04:12PM
Click here for more details...

=============================
The original post is available in Planet Python by
this post has been published as it is through automation. Automation script brings all the top bloggers post under a single umbrella.
The purpose of this blog, Follow the top Salesforce bloggers and collect all blogs in a single place through automation.
============================

Salesforce