Why the Scala compiler can provide implicit outside of object, but cannot inside?

Question

The title might be quite vague, but here is the code: https://github.com/amorfis/why-no-implicit

So there is a tool to transform Map[String, Any] to a simple case class. The tests pass and this piece of code illustrates what it is all about:

        case class TargetData(
          groupId: String,
          validForAnalysis: Boolean,
          applicationId: Int
        )

        val map = Map(
          "groupId" -> "123456712345",
          "applicationId" -> 31,
          "validForAnalysis" -> true
        )

        val transformed: TargetData = MapDecoder.to[TargetData](map).transform

This code works. It nicely creates the case class instance when provided the simple map

However, the transform method has to be called "outside" - just like in the example. When I try to move it to the MapDecoder.to method - the compiler complains about the missing implicit.

So I change the code in MapDecoder.to from this:

def to[A](map: Map[String, Any]) = new MapDecoderH[A](map)

to this:

def to[A](map: Map[String, Any]) = new MapDecoderH[A](map).transform

and it stops working. Why is that? Why the implicit is provided in one case but not in the other? All that changes is that I want to call the transform method in other place to have MapDecoder.to returning the case class not some transformer.

UPDATE:

What if I want to implement to[A] method inside an object I want to transform? Let's call it DataFrame, and I want this code to work:

val df: DataFrame = ...
df.to[TargetData] // There is no apply called here

The problem is in such case there is nothing to pass to apply. It is also not feasible to call it with parens (df.to[TargetData]()) because then the compiler requires implicits in parens. Is it even possible to solve it without using macros?

Answer 1

Implicit can be provided when the compiler can unambiguously find a value in the current scope with matching type.

Outside def to compiler sees that you want MapDecoder[TargetData].

Inside it sees MapDecoder[A] and have no reason to believe that A =:= TargetData.

In such situation you'd have to pass all the implicits as arguments of to method. From your code it seems it would have to be something like

def to[A, R <: HList](map: Map[String, Any])(implicit
  gen: LabelledGeneric.Aux[A, R],
  transformer: MapDecoder[R]
) = new MapDecoderH[A](map).transform

but it would break the ergonomy, since you'd have to add additional parameter which should be inferred but cannot - in Scala 2 you are passing all type arguments explicitly or none. There are ways to work around it like by splitting the type param application into 2 calls like this:

class Applier[A] {

  def apply[R <: HList](map: Map[String, Any])(implicit
    gen: LabelledGeneric.Aux[A, R],
    transformer: MapDecoder[R]
  ) = new MapDecoderH[A](map).transform
}

def to[A] = new Applier[A]

which would be used as

MapDecoder.to[A](map)

desugared by compiler to

MapDecoder.to[A].apply[InferredR](map)(/*implicit*/gen, /*implicit*/transformer)

It would be very similar to MapDecoder.to[TargetData](map).transform but through a trick it would look much nicer.

Answer 2

@MateuszKubuszok answered the question. I'll just make a couple of comments to his answer.

Adding implicit parameters

def to[A](map: Map[String, Any]) = new MapDecoderH[A](map).transform

// ===>

def to[A, R <: HList](map: Map[String, Any])(implicit
  gen: LabelledGeneric.Aux[A, R],
  transformer: MapDecoder[R]
) = new MapDecoderH[A](map).transform

you postpone implicit resolution in .transform from "now" i.e. the definition site of to (where A is abstract) to "later" i.e. the call site of to (where A is TargetData). Resolving implicits "now" is incorrect since LabelledGeneric[A] doesn't exist for abstract A, only for case classes, sealed traits, and like them.

This is similar to the difference implicitly[A] vs. (implicit a: A).

Another way of postponing implicit resolution is inlining. In Scala 3 there are inline methods for that along with summonInline used in them.

In Scala 2 inlining can be achieved with macros

// libraryDependencies += "org.scala-lang" % "scala-reflect" % "2.13.10"
import scala.language.experimental.macros
import scala.reflect.macros.blackbox

def to[A](map: Map[String, Any]): Either[String, A] = macro toImpl[A]

def toImpl[A: c.WeakTypeTag](c: blackbox.Context)(map: c.Tree): c.Tree = {
  import c.universe._
  q"new MapDecoderH[${weakTypeOf[A]}]($map).transform"
}

@MateuszKubuszok's solution with PartiallyApplied pattern (Applier) seems to be easier (adding implicit parameters is more conventional way to postpone implicit resolution although there can be situations when you just can't add parameters to a method).

---

Update:

What if I want to implement to[A] method inside an object I want to transform?

You can define apply with empty parameter list

// implicit class or a class/object where you want to implement method inside
implicit class MapOps(map: Map[String, Any]) {
  def as[A] = new Applier[A]

  class Applier[A] {
    def apply[R <: HList]()(implicit
      gen: LabelledGeneric.Aux[A, R],
      transformer: MapDecoder[R]
    ): Either[String, A] = new MapDecoderH[A](map).transform
  }
}

(I renamed extension method to as since Map already has .to)

and call it like map.as[TargetData]().

Suppose you don't want to add () like with Spark-ish df.to[TargetData]. You can always define a custom type class. This is more flexible solution than a method (with or without PartiallyApplied trick)

How to derive a Generic.Aux if the case class has a type parameter - Shapeless

// type class
trait As[A] {
  def as(map: Map[String, Any]): Either[String, A]
}
object As {
  // materilizer
  def apply[A: As]: As[A] = implicitly[As[A]]

  // instances of the type class
  implicit def makeAs[A, R <: HList](implicit
    gen: LabelledGeneric.Aux[A, R],
    transformer: MapDecoder[R]
  ): As[A] = new MapDecoderH[A](_).transform
}


implicit class MapOps(map: Map[String, Any]) {
  // "implement method inside an object to transform"
  def as[A: As]: Either[String, A] = As[A].as(map)
}

Now you can call the method without () like map.as[TargetData].

So you don't actually need macros now. I just mentioned macros solution for general understanding your options and better explanation of what's going on.