Code Coverage in Scala 3

ScalaCon 2021

Chris Kipp

@ckipp01

A little about me

  • Switched careers into programming
  • I enjoy simple tech
  • Big Neovim fan
  • You'll find me working in Scala Tooling
  • Host the Tooling Talks Podcast
  • Software engineer at Lunatech
  • chris-kipp.io/talks

scoverage

  • Code coverage tool for Scala
  • Started in 2010 by Mikko Koponen
  • Then maintained by Stephen Samuel  and Grzegorz Slowikowski for years
  • Over 1.5 million dowloads a month
  • sbt, Mill, Gradle, and Maven

How do you fit into the picture, Chris?

I write Scala, and needed code coverage...

 

So I decided to adopt it.

How does it even work?

Compiler plugin (in Scala 2)

Compiler plugin (in Scala 2)

sbt:scoverage-test> show Compile / compile / scalacOptions
[info] Compile / compile / scalacOptions
[info]  List(
[info]    -Xplugin:/my/path/to-plugin/2.0.0-M4/scalac-scoverage-plugin_2.13.6-2.0.0-M4.jar,
[info]    -P:scoverage:dataDir:/path-to-my-project/target/scala-2.13/scoverage-data,
[info]    -P:scoverage:sourceRoot:/path-to-my-project/scoverage-test,
[info]    -P:scoverage:reportTestName
[info]  )

Ok, so what does it do

 def isTwo(num: Int): Boolean =
  if (num == 2) {
    true
  } else {
    false
  }

We'll illustrate with a very simple piece of code

Compiler plugin (Scala 2)

case i: If =>
  treeCopy.If(
    i,
    process(i.cond),
    instrument(process(i.thenp), i.thenp, branch = true),
    instrument(process(i.elsep), i.elsep, branch = true)
  )

The tree would be instrumented here

  def isTwo(num: Int): Boolean = if ({
    scoverage.Invoker.invoked(
      1,
      "/var/folders/fq/nx_jsnyd6550xp03czx898d40000gn/T/",
      scoverage.Invoker.invoked$default$3()
    );
    num.==(2)
  }) {
    scoverage.Invoker.invoked(
      3,
      "/var/folders/fq/nx_jsnyd6550xp03czx898d40000gn/T/",
      scoverage.Invoker.invoked$default$3()
    );
    {
      scoverage.Invoker.invoked(
        2,
        "/var/folders/fq/nx_jsnyd6550xp03czx898d40000gn/T/",
        scoverage.Invoker.invoked$default$3()
      );
      true
    }
  } else {
    scoverage.Invoker.invoked(
      5,
      "/var/folders/fq/nx_jsnyd6550xp03czx898d40000gn/T/",
      scoverage.Invoker.invoked$default$3()
    );
    {
      scoverage.Invoker.invoked(
        4,
        "/var/folders/fq/nx_jsnyd6550xp03czx898d40000gn/T/",
        scoverage.Invoker.invoked$default$3()
      );
      false
    }
  };

scoverage.coverage

1                                                       <-- id
a/src/main/scala/mypackage/BasicControlStructures.scala <-- source path
a                           <-- package name
BasicControlStructures      <-- class name
Object                      <-- class type
a.BasicControlStructures    <-- full class name
isTwo                       <-- method name
74                          <-- start offset
79                          <-- end offset
5                           <-- line number
scala.Int.==                <-- symbol name
Apply                       <-- tree name
false                       <-- is branch
0                           <-- invocations count
false                       <-- is ignored
num.==(2)                   <-- sign

/a/target/scala-2.13/scoverage-data

What's the Invoker?

  • Writes the ids that call out to it
  • Keeps track of how many times they are called
  • Adds the test name where it was triggered

scoverage.measurements

1 a.basiccontrolstructuresspec
9 a.basiccontrolstructuresspec
4 a.basiccontrolstructuresspec
8 a.basiccontrolstructuresspec
7 a.basiccontrolstructuresspec

a/target/scala-2.13/scoverage-data

Reports

Reports

Scala 3 coverage

You just explained it all, why not cross publish it and call it a day?

  • New compiler new rules

  • 6 week release cycles are hard

lazy val bin212 =
  Seq(
    defaultScala212,
    "2.12.14",
    "2.12.13",
    "2.12.12",
    "2.12.11",
    "2.12.10",
    "2.12.9",
    "2.12.8"
  )
lazy val bin213 =
  Seq(
    defaultScala213,
    "2.13.6",
    "2.13.5",
    "2.13.4",
    "2.13.3",
    "2.13.2",
    "2.13.1",
    "2.13.0"
  )

Put it IN the compiler

  • No need to release every 6 weeks

  • Easily catch regressions

  • Have a friendly group of compiler devs looking it over

  • Plus Quentin Jaquier already has a working POC from like 4 years ago.

    • https://github.com/Qjaquier/dotty-coverage-tools

So let's do this

Again, Quentin's work on this and the report he wrote is incredibly useful. Most of these examples come straight from Code coverage for Dotty.

http://guillaume.martres.me/code_coverage.pdf

Where do we put this?

❯ scala3-compiler -Xshow-phases
parser
typer
inlinedPositions
sbt-deps
extractSemanticDB
posttyper
prepjsinterop
sbt-api
SetRootTree
pickler
inlining
postInlining
staging
pickleQuotes
{firstTransform, checkReentrant, elimPackagePrefixes, cookComments, checkStatic, betaReduce, inlineVals, expandSAMs}
initChecker
{elimRepeated, protectedAccessors, extmethods, uncacheGivenAliases, byNameClosures, hoistSuperArgs, specializeApplyMethods, refchecks, tryCatchPatterns, patternMatcher}
{elimOpaque, explicitJSClasses, explicitOuter, explicitSelf, elimByName, stringInterpolatorOpt}
{pruneErasedDefs, uninitializedDefs, inlinePatterns, vcInlineMethods, seqLiterals, intercepted, getters, specializeFunctions, liftTry, collectNullableFields, elimOuterSelect, resolveSuper, functionXXLForwarders, paramForwarding, genericTuples, letOverApply, arrayConstructors}
erasure
{elimErasedValueType, pureStats, vcElideAllocations, arrayApply, addLocalJSFakeNews, elimPolyFunction, tailrec, completeJavaEnums, mixin, lazyVals, memoize, nonLocalReturns, capturedVars}
{constructors, instrumentation}
{lambdaLift, elimStaticThis, countOuterAccesses}
{dropOuterAccessors, checkNoSuperThis, flatten, renameLifted, transformWildcards, moveStatic, expandPrivate, restoreScopes, selectStatic, junitBootstrappers, Collect entry points, collectSuperCalls, repeatableAnnotations}
genSJSIR
genBCode

Where do we put this?

❯ scala3-compiler -Xshow-phases
...
SetRootTree
+ coverage
pickler
inlining
...
  • At the end of the front end phases
  • It's important to do this before things like dead code elimination or other optimizations that would remove or change code we want to instument

Different types of tree instrumentation

Fully Instrumented

These are things that can be fully instrumented as they are.

case tree: Literal => instrument(tree)
case tree: New => instrument(tree)
case tree: This => instrument(tree)
case tree: Super => instrument(tree)
case Select(qual, _) if (qual.symbol.exists && qual.symbol.is(JavaDefined)) =>
case tree: Select =>
  if (tree.qualifier.isInstanceOf[New]) {
    instrument(tree)
  } else {
    ...
  }

Not Instrumented

These are things that either their syntax don't allow it or they code may be removed during compilation

case tree: Import => tree
case tree: Ident if (isWildcardArg(tree)) => tree

Partially Instrumented

Cases where we don't want to instrument the entire tree

def instrumentCaseDef(tree: CaseDef)(using Context): CaseDef = {
  cpy.CaseDef(tree)(tree.pat, transform(tree.guard), transform(tree.body))
}

Some special considerations

object Main {
  def ourFunction(a: Int, b: Int, c: Int) = ???
}

Main.ourFunction(1, thisThrows(2), someOtherComputation())

Main.ourFunction(1, thisThrows(2), someOtherComputation())

Argument lifting

val x0 = Main
val x1 = 1
val x2 = thisThrows(2)
val x3 = someOtherComputation()

x0.ourFunction(x1, x2, x3)

Main.ourFunction(1, thisThrows(2), someOtherComputation())

Reusing some of the same logic from ETA expansion

But... be careful where you apply this

val x0 = doThing()

true || x0

This would show x0 as covered, when it shouldn't be

What can we re-use from scoverage?

Scala 2

Compiler Plugin

Invoker

Scala 2

Compiler Plugin

Domain

Serializer

Reporter

Scala 2

Compiler Plugin

Domain

(de)Serializer

Reporter

Scala 3

In the Compiler

Invoker

Invoker

2.12, 2,13, 3

Scala Code Coverage

Unshared

  • Instrumenting your code
  • The invoker

Shared

  • Deserializing
  • Aggregation
  • Reporting

Demo

How to help

  • Test this out
  • https://github.com/lampepfl/dotty/pull/13880
  • https://github.com/ckipp01/scala3-example-project
  • Help out with maintenance
    • Mill contrib module
    • sbt-scoverage plugin
    • Abandoned Maven plugin
    • gradle-scoverage
    • Abandoned examples repo