Ch12

Metaprogramming  in Haskell
When we write code in Haskell, we call it programming. When we write code that generates or processes other code on our behalf, we call it metaprogramming, or generative programming. With metaprogramming techniques, we usually automate repetitive boilerplate code, which is too dull to write manually. This leads to boosting developer productivity. Sometimes, there is a price to pay—such code can be quite hard to write and maintain.
In Haskell, we have the full spectrum of metaprogramming features, ranging from those that are almost free for developers to some quite demanding of their skills. In this chapter, we’ll discuss three such features. We’ll start with the mechanisms to derive instances automatically. No Haskell program today avoids instance derivation, and GHC does its best to generate appropriate implementations. We have several ways to derive instances, and we have to discuss all of them to know which one should
be used in a particular situation. Then we’ll turn to data-type-generic programming. This technique allows us to treat any data type and its value as a data structure of some other specific data type, called a generic representation. This lets us write code with respect to the generic representation, not the original data type itself. Thus, this generic code can be applied to many different data types that have nothing in common. Finally, we’ll explore Template Haskell, the full-blown compile-time metaprogramming tool, which allows us to write code that is executed at compile time and generates other code that is then compiled as usual.
The two latter techniques are essential to implement certain complex and performant libraries, and we’ve already used some of them. For example, the cassava library uses data-type-generic programming to generate instances for reading user-defined
data types from CSV files. We’ve used it for the stockquotes project. Whenever a library needs some supporting code on the client’s side, it provides Template Haskell functions to generate it. We’ll see several such examples in the rest of this book.

12.1 Deriving instances
We’ve already derived dozens of instances in this book. This feature has existed in Haskell since almost the very beginning. Shortly after introducing type classes in the 1990s, it became clear that implementing instances manually was extremely ineffective. Writing the same code again and again isn’t very appealing (at least to Haskellers). Originally, we could derive instances for a small list of type classes supported by the compiler (such as Eq, Ord, and Show). The corresponding algorithms were built
into the compiler, and there were no ways to extend them.
In this section, we’ll explore modern GHC features that support instance derivation. We now have many more possibilities than ever before.
12.1.1 Basic deriving strategies
It’s crucial to understand the following essential principles about instance derivation:

Deriving an instance means generating an implementation for methods of type classes by the compiler.
GHC supports several strategies to generate code. Since GHC 8.2, we can choose which strategy we want to use.
Generated code is type-checked and compiled. Thus, it’s undesirable to generate incorrect code, because users would face an error in code they didn’t write and may not be aware of.

Remember that we use the deriving clause on data (or newtype) declarations to ask a compiler to derive instances. We can also use standalone deriving via the StandaloneDeriving GHC extension. The ideas we’ll discuss next don’t depend on the actual way you run deriving algorithms, so I assume that we always use the deriving clause.
EXPLORING DERIVED CODE
If something goes wrong, we can look at the code that was generated for us. GHC supports the -ddump-deriv flag to dump all the generated instances.
Let’s explore instances generated for the following trivial piece of code:
type Name = String
type Age = Int
data Student = Student Name Age
  deriving Eq

We ask GHC to generate the Eq instance. If we run it with the -ddump-deriv flag, we see the following Eq instance:
instance GHC.Classes.Eq Main.Student where
    (GHC.Classes.==)
      (Main.Student a1_a2j7 a2_a2j8)
      (Main.Student b1_a2j9 b2_a2ja)
      = (((a1_a2j7 GHC.Classes.== b1_a2j9))
           GHC.Classes.&& ((a2_a2j8 GHC.Classes.== b2_a2ja)))

GHC uses the full qualification and randomly generated names in this instance. If we strip off all these things, we have the following trivial implementation in disguise:
instance Eq Student where
    (==) (Student a1 a2) (Student b1 b2) = (a1 == b1) && (a2 == b2)

Two values of the Student type are equal if their first fields are equal and their second fields are equal.
Remember, the Eq type class provides another /= operator for inequality. Where is it? Well, it has a default implementation. GHC reports it also, though not as a derived instance but as a filling in of the method body, as shown next:
==================== Filling in method body ====================
GHC.Classes.Eq [Main.Student]
  GHC.Classes./= = GHC.Classes.$dm/= @(Main.Student)

The /= method body refers to a default $dm/= method implementation. Note the visible type application in the previous code: @(Main.Student). GHC explicitly instantiates the default implementation of the Student type.
Once we have this instance defined, we can use it as if we implemented it manually. Exploring generated instances may be useful if we want to find sources of inefficiency. It may be the case that a compiler doesn’t know the best way to implement an
instance. In such cases, we avoid instance derivation and implement it manually.

DERIVING STRATEGIES
GHC supports the following strategies to derive instances:

stock strategy refers to specific type classes with deriving algorithms built into GHC.
anyclass strategy allows us to derive empty instances. No methods are generated for them. This makes more sense if we have default implementations in type class definitions.
newtype strategy supports generating instances for newtype declarations. The code for those instances is the same as the code for the underlying types.
via strategy allows us to give an example of how the code should be generated. GHC then follows an example and generates code correspondingly. This strategy is the most recent one; it has been available since GHC 8.6.

These strategies are guarded by the corresponding GHC extensions. Nevertheless, it’s easy to come up with a situation where more than one strategy is enabled and can be applied. Suppose we have a newtype declaration and a type class with a default implementation. If both newtype and anyclass strategies are enabled, a compiler has to decide between them. Although GHC has an algorithm to choose a strategy to be used by default, it’s safer to avoid it. After all, we always know what should be done in a particular instance. Since GHC 8.2, we have the DerivingStrategies extension to specify a strategy explicitly. Let’s do that in a simple example.
Suppose we have the following newtype for representing ages:
newtype Age = Age {age :: Int}

We want to do a lot of things with it, including the following:

Use it as a number (e.g., to supply numeric literals as values or compute age differences)
Print it.
Encode it to JSON.

Thus, we need instances of the Num, Show, and ToJSON type classes. The Show type class is the stocked one: GHC knows how to generate its instances. The Num type class isn’t stocked. The only way to derive it is to reuse the underlying Int
type and an implementation for it (this is a newtype strategy). The ToJSON type class has a default implementation (suitable for the anyclass strategy) for any data type, which also has an instance of the Generic type class from the GHC.Generics module. The latter type class happens to be also stocked. It becomes available after enabling the DeriveGeneric GHC extension.
Given all this, we are ready to derive them all. First, we need a whole bunch of GHC extensions:
{-# LANGUAGE DerivingStrategies #-}  -- Allows for specifying strategies for deriving clauses
{-# LANGUAGE GeneralizedNewtypeDeriving #-}  -- Enables the newtype strategy
{-# LANGUAGE DeriveGeneric #-}  -- Supplies the derivation of instances for Generic
{-# LANGUAGE DeriveAnyClass #-}  -- Enables the anyclass strategy

Second, we write several deriving clauses in the newtype declaration as follows:
newtype Age = Age {age :: Int}
  deriving stock (Show, Generic)
  deriving newtype (Num)
  deriving anyclass (ToJSON)

Running the following code demonstrates that everything works:
theAge :: Age
theAge = 33  -- Uses Num for treating the 33 numeric literal as an Age value

main :: IO ()
main = do
  print theAge  -- Uses Show for printing
  print $ encode theAge  -- Uses ToJSON for encoding

It also results in printing the following:
Age {age = 33}
"{\"age\":33}"

DERIVING STOCK CLASSES
GHC supports deriving the following seven type classes from scratch (all of them come from the Haskell 98 Report): Eq, Ord, Enum, Ix, Bounded, Read, and Show. The Ix type class defines a common interface for array indices. We’ve used the other classes many times in this book.
Besides them, current GHC versions derive the following instances with the corresponding extensions enabled:

Generic and Generic1 with DeriveGeneric. We’ll get to these type classes in the next section.
Functor with DeriveFunctor.
Foldable and Traversable with DeriveFoldable and DeriveTraversable, respectively.
Typeable and Data from the Data.Typeable and Data.Data modules with DeriveDataTypeable. These types provide one way to look at run-time representations of Haskell values and an additional approach to generic programming, different from what I present in this chapter. They are also used to introduce dynamic types to Haskell (see Data.Dynamic for details).
Lift with DeriveLift to be used with the template-haskell package. More on that later in this chapter.

Algorithms that derive these instances are described in the GHC User’s Guide. Let’s now move to more advanced strategies: newtype and via. The via strategy may be seen as a generalization of the newtype strategy. Both of them allow us to derive nontrivial instances for classes unknown to a compiler. This creates a problem of type safety: how to guarantee type correctness for generated instances. Let’s see how to solve it.

12.1.2 The problem of type safety and generalized newtype deriving
Introducing newtype declarations and deriving instances from the underlying representation is a traditional Haskell approach. In this section, we’ll discuss why this is possible and which problems we could face when using this strategy to derive instances.
THE COERCIBLE CONSTRAINT AND THE COERCE FUNCTION
Let’s get back to another version of the earlier Age and Student examples. We’ll explore newtype declarations and discover more tools to work with them.
Suppose we have the following newtype declaration and a function:
newtype Age = Age Int
  deriving (Show)

toAges :: [Int] -> [Age]
toAges = map Age

We want to turn a list of Int values into a list of ages. We know that due to the use of a newtype declaration, Age and Int share the same run-time representation. What about [Age] and [Int] or Maybe Age and Maybe Int? It’d be highly desirable for them to share the same representation also. This means that the toAges function should do literally nothing, and we should get the same list of Ints as a result. Is our compiler smart enough to optimize away the call to map in the implementation of toAges? Well, the power of the GHC optimizer has some limits. Fortunately, we can help the compiler and ourselves by avoiding writing map at all.
If we have types that share the same run-time representation, we can coerce between them. This functionality is provided by the Data.Coerce module from base. We’ve already met this function in chapter 9 when we discussed rewriting rules. Dealing with newtype declarations is another vital usage for it. Let’s try to use the coerce function to reimplement toAges as follows:
import Data.Coerce

...

toAges' :: [Int] -> [Age]
toAges' = coerce

This code compiles successfully and works. No map, no explicit data constructors. Note that this code requires nothing besides importing the module and using coerce.
The coerce function has the following type:
coerce :: forall (k :: RuntimeRep) (a :: TYPE k) (b :: TYPE k).
          Coercible a b => a -> b

This is a highly polymorphic type; it works with all the run-time representations, boxed and unboxed, lifted and unlifted. Remember that Int and our Age both belong to the Type kind, which is defined as follows:
type Type = TYPE LiftedRep

The Coercible constraint is quite special. We are not allowed to define instances of Coercible manually. GHC infers them for us. When we write a newtype declaration, GHC infers the corresponding instance of Coercible. In our example, we have Coercible Age Int generated. Moreover, GHC lifts this instance to Coercible [Age] [Int] automatically. It is also able to use symmetry and get Coercible Int Age and Coercible [Int] [Age]. It’s the latter instance that allows us to use coerce instead of implementing toAges with map.
All right, let’s define the Student data type containing information about name and age and try to coerce on it as follows:
data Student ageType = Student String ageType

check :: Student Int -> Student Age
check = coerce

Note the ageType type variable in the data-type definition. We want to be flexible in storing ages. We have Coercible (Student Int) (Student Age) lifted from Coercible Int Age, so this compiles.
Let’s make an age representation a little bit more complex. The age may be missing now, as shown next:
data Student1 ageType = Student1 String (Maybe ageType)

check1 :: Student1 Int -> Student1 Age
check1 = coerce

It still works with Coercible (Student1 Int) (Student1 Age). Is it possible to coerce everything to everything? Hopefully not. Let’s continue our experiments on student representations as follows:
data Student2 m ageType = Student2 String (m ageType)

check2 :: Student2 Maybe Int -> Student2 Maybe Age
check2 = coerce

Now, this results in the following somewhat misleading error message:
• Couldn't match type ‘Int’ with ‘Age’
        arising from a use of ‘coerce’

So it was possible in all the examples before, but now GHC can’t do that. What’s the difference? It looks like GHC doesn’t trust the m type variable. When it was Maybe in the type declaration, GHC was perfectly fine with it. Here is the point: what if m changes a representation somehow? Even though we have Maybe in the position of m in check2, GHC is still not happy about it.
Okay, let’s replace this m with something more precise. How about the following identity type family?
type family Id t
type instance Id t = t

Types t and Id t are clearly identical, including their run-time representation. Let’s use this type family in the following variation on the Student data type declaration:
data Student3 ageType = Student3 String (Id ageType)

Structurally, this declaration is the same as the one for Student1. Do we have Coercible (Student3 Int) (Student3 Age)? In other words, are we allowed to write the following function?
check3 :: Student3 Int -> Student3 Age
check3 = coerce

Well, GHC complains again:
• Couldn't match type ‘Int’ with ‘Age’
        arising from a use of ‘coerce’

So the answer is no. Even though the Id type family doesn’t change the run-time representation, GHC still prohibits coerce.
 The answer is the roles system. Let’s discuss it now.
ROLES FOR TYPE VARIABLES
The following are our variations on the Student data type declaration:
data Student ageType = Student String ageType
data Student1 ageType = Student1 String (Maybe ageType)
data Student2 m ageType = Student2 String (m ageType)
data Student3 ageType = Student3 String (Id ageType)

We try to use the coerce function to coerce between Int and Age as an ageType type variable. We can do that for Student and Student1 but can’t for Student2 and Student3. What’s the difference?
The difference is the role of the ageType type variable that is inferred by GHC. The three possible roles follow:

The phantom role corresponds to phantom types that we’ve discussed in the previous chapter. Their values don’t exist at run time, so they don’t interfere with coerce at all.
The representational role refers to type variables that are used to describe the run-time representation. We can coerce between types in positions with representational roles.
The nominal role doesn’t give us any information on the representation.  For example, key in Map key val has a nominal role. We are not allowed to coerce between types in positions with nominal roles.

The ageType type variable has a representational role in the Student and Student1 types. It has a nominal role in Student2 and Student3. That’s why we have the corresponding Coercible instances for the former examples and not for the latter. But how do we know the role of the type variable?
GHC infers these roles based on how a type variable is used. The ageType type variable is used to represent the second argument to the Student data constructor. Thus, its role is representational. The same rule applies to the Maybe a type: a here has a representational role—ageType in Student1 stays representational because it is used in a representational position.
In Student2, we know nothing about m. It can potentially change the representation of ageType somehow; that’s why ageType is considered nominal. Yet, m itself has a representational role.
Variables in type families and type classes are always considered with nominal roles. This explains why we can’t coerce in Student3, even though the types have absolutely identical representations. This is just a limitation of the current role system—it can’t handle this case.
We cannot persuade GHC to consider a type variable having a representational role if it infers a nominal role for it: this would break the type system. When the Haskell role system is too weak, but we are sure that it’s okay to coerce, we can use the
unsafeCoerce function from Unsafe.Coerce instead. The following code compiles successfully:
check3' :: Student3 Int -> Student3 Age
check3' = unsafeCoerce

Remember, we used this function in the previous chapter to define the toUnescaping function (ch11/type-families/Unescape.hs). The role system prohibits such cases with the combination of type families and type classes, although it works with unsafeCoerce.
Sometimes we want to restrict inferred roles and prevent generating unnecessary Coercible instances. We can do that with role annotations, available with the RoleAnnotations GHC extension.
Writing the following prohibits coercions between Student Age and Student Int:
{-# LANGUAGE RoleAnnotations #-}

type role Student nominal
data Student ageType = Student String ageType

Note again, this works only to restrict a role: we can declare phantom roles as representational and representational roles as nominal, but not the other way around

DERIVING INSTANCES WITH GENERALIZEDNEWTYPEDERIVING
Now that we’ve discussed the Coercible constraint and the role system, it’s easy to explain the newtype strategy for deriving instances. It turns out that implementing any method is as easy as writing coerce.
Suppose we have the following declaration:
{-# LANGUAGE DerivingStrategies #-}
{-# LANGUAGE GeneralizedNewtypeDeriving #-}

newtype Age = Age Int
  deriving newtype (Eq, Ord)

How does GHC generate the required instances? It coerces implementations for the underlying type (here, Int) of all the methods. For example, we have the following Eq instance:
instance Eq Age where
  (==) = coerce ((==) :: Int -> Int -> Bool)

The (==) operator for Age is the same as the one for Int. GHC uses the Coercible Int Age constraint to coerce those functions.
Deriving the Ord instance follows the same strategy. The only required compare method is implemented as follows:
instance Ord Age where
  compare = coerce (compare :: Int -> Int -> Ordering)

The same idea works with more sophisticated type classes. For example, we derived instances for monad stacks several times in this book as follows:
{-# LANGUAGE GeneralizedNewtypeDeriving #-}

newtype MyApp a = MyApp {
    runApp :: StateT Int IO a
  }
deriving (Functor, Applicative, Monad, MonadIO)

GHC generates the following Functor instance:
instance Functor MyApp where
  fmap :: forall a b. (a -> b) -> MyApp a -> MyApp b
  fmap = coerce (fmap :: (a -> b) -> StateT Int IO a -> StateT Int IO b)

I’ve used the InstanceSigs and ScopedTypeVariables to define this instance. GHC may do this a little bit differently, but the idea is the same. We bring type variables into the scope and use coerce with the right types. For example, GHC 8.8 with the -ddump-deriv gives me the following (after cleaning the names):
fmap = coerce
           @((a -> b) -> StateT Int IO a -> StateT Int IO b)  -- Coerces from this type
           @((a -> b) -> MyApp a -> MyApp b)  -- Coerces to this type
           (fmap @(StateT Int IO))  -- An fmap over StateT Int IO is coerced.
              :: forall (a :: TYPE LiftedRep)  -- a :: Type
                        (b :: TYPE LiftedRep).
                 (a -> b) -> MyApp a -> MyApp b  -- Resulting type of fmap for MyApp

Remember that coerce has two implicit type arguments to coerce from and to. They are given here with the visible type applications (the TypeApplications GHC extension). Then we say that we’d like to coerce from fmap for StateT Int IO and finally give the resulting type of fmap we define. Remember that TYPE LiftedRep is aliased as the Type kind.
All the other instances are generated the same way.
Note that the GeneralizedNewtypeDeriving GHC extension fails to generate instances if there are no appropriate Coercible instances around. The whole roles and Coercible ideas have arisen from a bug in older GHC versions. Suppose we have the following type family:
type family Inspect t
type instance Inspect Int = Bool
type instance Inspect Age = Int

We already know that the t type variable here has the nominal role, so there is no Coercible (Inspect Age) (Inspect Int) instance, though we still have Coercible Age Int. Suppose we also have the inspect function defined in a type class with an instance for Int as follows:
class Inspector a where
  inspect :: a -> Inspect a

instance Inspector Int where
  inspect n = n > 0  -- The result is of Inspect Int type, which is Bool.

The question is, is it possible to derive the Inspector instance for Age using the newtype strategy? Remember that the newtype strategy here is replacing an implementation for Age with an implementation for Int. An ability to do that would mean that we have a result of type Bool (Inspect Int) interpreted as a value of type Int (Inspect Age). By doing so, we’d break the Haskell type system. Imagine this: it was possible in GHC for several years that we could see actual bits of True and False in memory as if we were C programmers! Let’s check if it’s possible now using the standalone deriving, as shown next:
deriving newtype instance Inspector Age

This results in the following type error:
• Couldn't match representation of type ‘Bool’ with that of ‘Int’
        arising from a use of ‘coerce’
    • In the expression:
          coerce
            @(Int -> Inspect Int) @(Age -> Inspect Age) (inspect @Int) ::
            Age -> Inspect Age

GHC generates an instance with coerce, and then the type checker figures out that this coerce invocation cannot be type-checked. We are on the safe side with the available and nonexistent Coercible instances!

12.1.3 Deriving by an example with DerivingVia
Our last strategy is the via strategy. Its overall idea is extremely simple. We give GHC an example of a type that already implements the instance we need and say, “Generate an implementation in the same way but replace types with ours, please.” Note that this strategy generalizes the newtype strategy as follows: the given example of type is simply an underlying type in the newtype declaration. Let’s look at several simple examples.
GHC generates identical instances in the following code:
{-# LANGUAGE DerivingStrategies #-}
{-# LANGUAGE GeneralizedNewtypeDeriving #-}
{-# LANGUAGE DerivingVia #-}

newtype Age = Age Int  -- Int is an underlying type for Age.
deriving newtype (Eq, Ord)  -- Instances for Int are reused.

newtype Age' = Age' Int
deriving (Eq, Ord) via Int  -- Int is given as an example to derive Eq and Ord.

In fact, we don’t need the GeneralizedNewtypeDeriving GHC extension anymore: the DerivingVia extension was introduced in GHC 8.6.
Why is generalized newtype deriving not enough? We know that some types may potentially have different instances capturing different ideas. In Haskell, we use newtype wrappers to distinguish between them. For example, the Data.Monoid provides
several ways to implement instances for Maybe a. There is a default instance that requires having the Semigroup instance for a. We can wrap the Maybe a value with Alt, which prefers the first available not-Nothing value. We can also add the Dual wrapper to get the opposite behavior (preferring the last available not-Nothing value).
Let’s look at an example. We have the following variation of the annoying Age type, now with a potentially missing value:
newtype MAge = MAge (Maybe Int)

Deriving Semigroup and Monoid instances with a newtype strategy requires having them for Int (that is the default for Maybe a). Unfortunately, there are none. One option is to add a layer or two of wrappers to the MAge type. This is not desirable because stacking wrappers makes using the type less convenient. With DerivingVia, we can instruct GHC to follow the Alt Maybe Int example when deriving the instance we want as follows:
newtype MAge = MAge (Maybe Int)
  deriving (Semigroup, Monoid) via (Alt Maybe Int)

We can also use the syntax with StandaloneDeriving as shown next:
deriving via (Alt Maybe Int) instance Semigroup MAge
deriving via (Alt Maybe Int) instance Monoid MAge

If we prefer to take the last available not-Nothing value, we go with Dual as follows:
newtype MAge = MAge (Maybe Int)
  deriving (Semigroup, Monoid) via (Dual (Alt Maybe Int))

Basically, we instruct the compiler with the desired way to implement instances. Both Dual and Alt are new types defined in the base library. They implement instances for both Semigroup and Monoid type classes. The Semigroup instance for Alt over Maybe a prefers the first-given not-Nothing alternative in the implementation of the (<>) operator from Semigroup. The instance for Dual swaps arguments to <>. For us, these provided instances are no more than patterns to define instances for our own newtype declaration. We say, “Make it just as in the instance of Dual (Alt Maybe Int),” meaning prefer the first, but swap arguments and make the results prefer the last alternative. Consequently, we have a simple declaration with the desired instances and no boilerplate with excessive wrappers or manual implementation.

12.2 Data-type-generic programming
Even though we’ve seen very flexible tools to derive instances, it turns out that in practice, these are not enough. Depending on the actual type structure, we often need to implement them differently. The problem is a necessity to have an instance
for every type we have in our program. We’d like to write code generically, meaning that we want to deal not with a plethora of data types but with a single representation for all of them.

A NOTE ON SUM AND PRODUCT TYPES  Such a representation is built around a concept of sum and product types. Basically, the sum of several types is a type with every value belonging to one of those types. The product of several types is a type with every value consisting of values of all those types. The latter is well known in other programming languages. It corresponds to tuples, structures, or records. The former makes its way to mainstream programming languages in a relatively limited form of enumerations. In Haskell, we have both with the plain old data declaration via | and parameters to data constructors.

Data-type-generic programming in Haskell is supported by the GHC.Generics module, which provides such a representation. The most common usage for it is to write such instances that serve well for almost all the data types. Then we cook such generic
instances and make them ready for the anyclass deriving strategy.
We’ve already seen GHC.Generics. We used it to generate instances to read values from CSV files or compose JSON requests. The text-show package provides a way to derive instances of the TextShow type class for our types using the Generic instances. The deepseq package uses them to evaluate values to normal forms. Now it’s time to understand what data-type-generic programming is and how to use it in our own libraries.
First, we’ll look at the representation of Haskell data types as defined by GHC.Generics. Second, we’ll look at examples of how to process it.

12.2.1 Generic data-type representation
We’ll start by exploring the idea of generic representation. Which information can we mine out of a value and its type?
Let’s look at the following data type:
data Status = Ok | Err

Here I collect all the facts about this type:

It’s a data type, defined in the Main module of the main package (these names are used in GHC by default); it’s not a newtype.
Its name is "Status".
It has two data constructors. We can take either to construct a value.
The left (first) data constructor is named "Ok": it has no arguments, it is not a record selector, and it is supposed to be used in prefix form (as we usually do with data constructors).
The right (second) data constructor is named "Err": it has no arguments, it is not a record selector, and it is also supposed to be used in prefix form.

That’s it. One may think that this is trivial and extremely boring information. Well, I agree. Nevertheless, having a representation means having all the information.

It was constructed with the left data constructor.
Its data constructor has no arguments.

We can teach GHC to represent all this information as types and values as follows:
{-# LANGUAGE DeriveGeneric #-}

import GHC.Generics

data Status = Ok | Err
  deriving Generic

The Generic type class provides one associated type family and a couple of functions, as shown next:
class Generic a where
  type Rep a :: Type -> Type
  from :: a -> Rep a x
  to :: Rep a x -> a

For example, let’s look at the type of the from Ok value, shown here:
hid-examples on  ch12 via λ 9.2.7 via ❄  impure (ghc-shell-for-hid-examples-0.5-0-env)
❯ cabal -v0 repl view-generic
ghci> :type from Ok
from Ok
  :: D1
       ('MetaData "Status" "Main" "main" 'False)
       (C1 ('MetaCons "Ok" 'PrefixI 'False) U1
        :+: C1 ('MetaCons "Err" 'PrefixI 'False) U1)
       x

It is extremely important to correlate this information to a collection of trivial facts about the Status type I showed earlier. We have a data type (D1) with some metainformation regarding names and the fact that this is not a newtype (see 'False in the 'MetaData type constructor). We have two data constructors combined with the (:+:) type operator. These constructors are also described with metainformation (name, fixity, being a record selector). Their arguments are given with U1 (no arguments). Finally, we also have the x type variable, which is not used anyway, so we can safely ignore it.
Note that this type exploits several type-level features we discussed in the previous chapter, including data kinds (ticked names!) and the type-level String. When we process the values of such types, we can use KnownSymbol to bring names to a value level and process them accordingly.
The from Ok value itself is simpler, as shown next:
ghci> from Ok
M1 {unM1 = L1 (M1 {unM1 = U1})}

Here is the pattern: M1 constructs a value, which is L1 with an argument that is constructed with M1 again and U1 inside. L1 corresponds to the left data constructor; U1 corresponds to the absence of arguments to it. The left data constructor has no arguments, see? We also have M1 as a building block of such a representation.
We can build such types and the corresponding values manually (given that we’ve enabled all the required extensions) as follows:
errVal :: D1
       ('MetaData "Status" "Main" "main" 'False)
       (C1 ('MetaCons "Ok" 'PrefixI 'False) U1
        :+: C1 ('MetaCons "Err" 'PrefixI 'False) U1)
       x
errVal = M1 {unM1 = R1 (M1 {unM1 = U1})}

Here, I construct the Err value. The only difference with from Ok is that I use R1 instead of L1 when building the value. The type is clearly the same. We can turn this value back to Status with the to function as follows:
ghci> to errVal :: Status
Err

What if we have fields in the constructors? The GHC.Generics module gives us the (:*:) operator. Suppose we have the following type for requests with String and Int fields:
data Request = Request String Int
  deriving Generic

The type of its generic representation follows (I’m using an associated type family directly with the GHCi :kind! command):
ghci> import GHC.Generics
ghci> :kind! Rep Request
Rep Request :: * -> *
= M1
    D
    ('MetaData "Request" "Main" "main" 'False)
    (M1
       C
       ('MetaCons "Request" 'PrefixI 'False)
       (M1
          S
          ('MetaSel
             'Nothing 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy)
          (K1 R [Char])
        :*: M1
              S
              ('MetaSel
                 'Nothing 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy)
              (K1 R Int)))

The (:+:) and (:*:) operators can be used to represent any number of data constructors and constructor fields, respectively. A type with three data constructors could use the following somewhere inside it:
... :+: (... :+: ...)

Constructor field values at run time are represented with the K1 data constructor as follows:
ghci> from (Request "req" 0)
M1 {unM1 = M1 {unM1 = M1 {unM1 = K1 {unK1 = "req"}} :*: M1 {unM1 = K1 {unK1 = 0}}}}

Thus, the Generic type class allows us to go from a value to its generic representation and back. We also have the Generic1 type class to represent type constructors of the k -> Type kinds. We can derive it with DeriveGeneric, too. Once we have those instances generated, we can go between values and their representations. I recommend looking through the documentation of the GHC.Generics module to see a thorough description of all the types used her

hid-examples on  ch12 via λ 9.2.7 via ❄  impure (ghc-shell-for-hid-examples-0.5-0-env)
❯ cabal -v0 run generic-sql
INSERT INTO students (studentId, name, year) VALUES (18265, "John Doe", 2)
INSERT INTO courses (courseId, title, instructor) VALUES (1, "Math", "Jane Doe")

12.2.2 Example: Generating SQL queries
Now let’s do something useful with a generic representation. We want to generate INSERT SQL queries for our data types to store their values in a database. The idea is to take a data type, such as
data Student = Student {studentId :: Int, name :: Text, year :: Int}

and generate a SQL query like the following for a given value:
INSERT INTO students (studentId, name, year) VALUES (18265, "John Doe", 2)

Clearly, we don’t want to limit ourselves to the Student type. We could also work with another data type, such as
data Course = Course {courseId :: Int, title :: Text, instructor :: Text}

Thus, we have to program generically.
EXPLOITING OUTCOMES IN DATA-TYPE-GENERIC PROGRAMMING
Suppose we have the following type class:
class ToSQL a where
  insertInto :: Text -> a -> Text

Its goal is to generate INSERT queries. We’d like to use it as follows:
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE DerivingStrategies #-}
{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE DeriveAnyClass #-}

import GHC.Generics
import Data.Text (Text)
import qualified Data.Text.IO as TIO

import GenericSQL

data Student = Student {studentId :: Int, name :: Text, year :: Int}
  deriving stock Generic
  deriving anyclass ToSQL

data Course = Course {courseId :: Int, title :: Text, instructor :: Text}
  deriving stock Generic
  deriving anyclass ToSQL

main :: IO ()
main = do
  TIO.putStrLn $ insertInto "students" (Student 18265 "John Doe" 2)
  TIO.putStrLn $ insertInto "courses" (Course 1 "Math" "Jane Doe")

hid-examples on  ch12 via λ 9.2.7 via ❄ impure (ghc-shell-for-hid-examples-0.5-0-env)
❯ cabal -v0 run generic-sql
INSERT INTO students (studentId, name, year) VALUES (18265, "John Doe", 2)
INSERT INTO courses (courseId, title, instructor) VALUES (1, "Math", "Jane Doe")

This code is simple. For every data type that we need to insert into a database, we derive both Generic and ToSQL using different strategies. We already know that the anyclass strategy requires having a default implementation. So our insertInto
should have one. Let’s do that.
EXTRACTING INFORMATION FROM A GENERIC REPRESENTATION
First, we need to extract all the information from a generic representation provided by the Generic instance. Which information do we need? Well, it’s a list of field names and a list of values. Note that this information goes partly from a type declaration
(selector names, such as "studentId" or "name") and partly from a value itself (18265 or "John Doe"). Extracting it from a type requires writing a type class and instances as usual. We start with a type class as follows:
class ToColumnsValuesLists f where
  toColumnsValues :: f a -> ([Builder], [Builder])

The idea is to go over a run-time representation once and collect two lists: one with selector names and the other with the corresponding values. Note that I use Builder from the TextShow module for performance reasons when building the resulting Text value. The SQL queries may be quite large.
Let’s implement instances, starting with the simplest one. If we have a data type represented with U1, then it has no fields at all, so the resulting lists should be empty, as shown here:
instance ToColumnsValuesLists U1 where
  toColumnsValues _ = ([], [])

If we have several fields, then the data type is represented by the (:*:) type operator. In this case, we should first extract information from both parts, left and right, and then combine it in the resulting lists as follows:
instance (ToColumnsValuesLists a, ToColumnsValuesLists b) =>
                ToColumnsValuesLists (a :*: b) where
  toColumnsValues (a :*: b) = (columns1 <> columns2, values1 <> values2)
    where
      (columns1, values1) = toColumnsValues a
      (columns2, values2) = toColumnsValues b

Note the context in the instance head: we require both sides of :*: to implement our type class. Note also that the (:*:) operator exists both at the type level (we have it in the instance head) and the value level (we pattern match over it). We traverse over both levels simultaneously. The implementation itself is trivial.
Now to the interesting part: extracting metainformation should be done from the M1 type. The M1 type has the following four type parameters:
newtype M1 (i :: Type) (c :: Meta) (f :: k -> Type) (p :: k)

The first one corresponds to a represented component (the data type itself, constructor, or selector). We have several type aliases for these components, as shown next:
type D1 = M1 D
type C1 = M1 C
type S1 = M1 S

The second parameter contains metainformation, which clearly depends on a component. The third parameter specifies a type whose representation we are constructing. The last parameter is the represented type itself.
We ignore most of the information, so let’s define the following instance to traverse over:
instance (ToColumnsValuesLists a) =>
               ToColumnsValuesLists (M1 i c a) where
  toColumnsValues (M1 a) = toColumnsValues a

We are interested only in the selector’s names, so we define an overlapping instance for this particular case as follows:
instance {-# OVERLAPPING #-} (ToColumnsValuesLists a, Selector c) =>
               ToColumnsValuesLists (M1 S c a) where
  toColumnsValues s@(M1 a) = (fromString (selName s) : columns, values)
    where
      (columns, values) = toColumnsValues a

The selName function comes from the Selector type class. It uses the KnownSymbol under the hood to give us information from the type level.
Finally, we should extract values, as shown next. They are represented with K1.
instance TextShow a => ToColumnsValuesLists (K1 i a) where
  toColumnsValues (K1 a) = ([], [showb a])

For simplicity reasons, I don’t want to consider other cases, such as several data constructors in one data type, data types that don’t use record syntax, or recursive data types. All of these can also be represented with GHC.Generics, and the documentation is quite good. The code we already have is going to work for the simplest case, and that’s enough for our goals. Note that plenty of practical data types follow this trivial structure: one data constructor, a record syntax, and as many selector fields as necessary. At least, it works for our Student data type, as shown next:
hid-examples on  ch12 via λ 9.2.7 via ❄ impure (ghc-shell-for-hid-examples-0.5-0-env)
❯ cabal -v0 repl generic-sql
ghci> :set -XOverloadedStrings
ghci> import GenericSQL
ghci> import GHC.Generics
ghci> toColumnsValues (from $ Student 18265 "John Doe" 2)
(["studentId","name","year"],["18265","\"John Doe\"","2"])

Our next step is to use all this extracted information to build a SQL query.

GHCi, version 9.2.7: https://www.haskell.org/ghc/  :? for help
ghci> :set -XTemplateHaskell
ghci> import Language.Haskell.TH
ghci> runQ [| 1 + 2 |]
InfixE (Just (LitE (IntegerL 1))) (VarE GHC.Num.+) (Just (LitE (IntegerL 2)))
ghci> answer = LitE (IntegerL 42)
ghci> :type answer
answer :: Exp
ghci> $( pure answer )
42
ghci> expr = InfixE (Just (LitE (IntegerL 1))) (VarE (mkName "+")) (Just (LitE (IntegerL 2)))
ghci> $( pure expr )
3

12.3 Template Haskell and quasiquotes
When GHC compiles our code, it starts with parsing. As a result of parsing, it creates an abstract syntax tree (AST). Template Haskell (TH) allows us to explore this tree and generate new nodes. The compiler then continues the compilation of the code, including what we’ve generated. In this section, we’ll look at different TH features by implementing a series of simple examples in a tutorial style. Then we’ll discuss quasiquotations, a mechanism that allows enriching code in Haskell with literally anything. We’ll end up with a complex example of implementing classical mechanisms for distributed computations and remote procedure calls (or remote function calls in this case).

THE RISKS OF USING TEMPLATE HASKELL Template Haskell is intimately tied to the GHC internals. It was never standardized. There are always risks of breaking code under GHC updates. As a rule, it’s better to avoid using Template Haskell in production code.

12.3.1 A tutorial on Template Haskell
One thing that should be understood from the very beginning is that Template Haskell runs at compile time. This puts many limitations on how we write programs with TH. We’ll meet them shortly, but let’s start with the simplest things. Our first problem
is how to build an AST manually and insert it into the code.
QUOTES AND SPLICES
Let’s fire up GHCi, enable the TemplateHaskell extension, and import the main TH module as follows:
ghci> :set -XTemplateHaskell
ghci> import Language.Haskell.TH
ghci> runQ [| 1 + 2 |]
InfixE (Just (LitE (IntegerL 1))) (VarE GHC.Num.+) (Just (LitE (IntegerL 2)))

We have many interesting things here. The [| … |] brackets are called Oxford brackets. They are used to define a computation in the very special Q monad. In some sense, they create a quote with Haskell code. We can look at an AST that corresponds to this quote by running this Q computation with the runQ function.
The result of [| 1 + 2 |] is an infix expression built from two integer literals and the + operator. Note how this description corresponds to the AST in the previous GHCi session. Every Haskell code element has a corresponding data constructor in
this AST. Because we have these constructors, we can build our own AST nodes as follows:
ghci> answer = LitE (IntegerL 42)
ghci> :type answer
answer :: Exp

The next step is to splice this node to the code itself. We do that with the $(…) instruction, whose argument should be a Q computation, as shown next:
ghci> $( pure answer )
42

Let’s construct the 1+2 expression manually with AST data constructors and splice it in GHCi to see a result as follows:
ghci> expr = InfixE (Just (LitE (IntegerL 1))) (VarE (mkName "+")) (Just (LitE (IntegerL 2)))

Note the mkName function, which gives us a value of the Name type used to refer to a name in an AST. A spliced expression is immediately evaluated in GHCi as shown next:
ghci> $( pure expr )
3

We don’t have to use GHCi. We can do the same thing in our program. One of the limitations I mentioned earlier is that we are not allowed to splice TH definitions created in the same module—they should always be defined elsewhere.
The following is the simple TH definition in the separate module. It’s an expression in the Q monad, which prints a String.
{-# LANGUAGE TemplateHaskell #-}
module Hello where
import Language.Haskell.TH

hello :: Q Exp
hello = [| putStrLn "Hello world" |]

Note that we don’t need parentheses if we splice a single name. This program works as expected, as shown next:
hid-examples on  ch12 via λ 9.2.7 via ❄ impure (ghc-shell-for-hid-examples-0.5-0-env)
❯ cabal -v0 run hello-th
Hello world

Besides expressions (the Exp type), Haskell ASTs may contain declarations (Dec), types (Type), and patterns (Pat). We can construct and splice all of them.

Skip projectors example (some function doesn't work due to ghc version).

REIFICATION
Sometimes we need to learn something that we already have in our code before generating something new. In the next example, we’ll look at existing data declarations and generate some code to support them.
In general, the term reification means turning something abstract into something concrete. In Template Haskell, we reify names into full information about the entities behind those names. As a result, we can inspect this information and use it to generate code.
Suppose we have the following data type for representing geometric shapes:
data Shape = Circle Double | Square Double
           | Triangle Double Double Double

Sometimes we need to use the following predicates:
isCircle :: Shape -> Bool
isSquare :: Shape -> Bool
isTriangle :: Shape -> Bool

Though they can be easily implemented, it’s quite boring to write them down. Well, let’s generate them! After all, this is our main solution to every problem in this chapter. The plan follows:

Reify the given data-type name, and extract information about data constructors.
Write code to generate trivial predicate definitions with TH.
Use TH splices to actually generate code for predicates.

Let’s write the function as follows:
mkPredicates :: Name -> Q [Dec]

It takes the name of the data type and gives back predicate declarations. The reification is done via the reify function as follows:
reify :: Name -> Q Info

This function gives back a lot of information, but we are interested in data constructors exclusively, as shown next:
extractConstructors :: Info -> [Con]
extractConstructors (TyConI (DataD _ _ _ _ cons _)) = cons
extractConstructors _ = []

There are several sorts of data constructors. We go with the simplest one, NormalC, and ignore the others as follows:
mkPredicate :: Con -> Q [Dec]
mkPredicate (NormalC name types) =
  [d|
     $predicate = \z -> case z of
                         $pat -> True
                         _ -> False
  |]
  where
    predicate = varP $ mkName $ "is" ++ nameBase name
    pat = conP name $ replicate (length types) wildP
mkPredicate _ = pure []

Here we build the predicate name using the mkName and nameBase functions. The latter function removes all the qualifications, if present. Then we build a pattern for the corresponding constructor. This requires providing the right number of wildcards in the pattern.
With this job done, it’s easy to implement the required generator as follows:
mkPredicates :: Name -> Q [Dec]
mkPredicates name =
  reify name
  >>= fmap concat . mapM mkPredicate . extractConstructors

In the last step, we should refer to the name of the data constructor in the TH splice. We use the '' prefix for that, as shown here:
$(mkPredicates ''Shape)

This instruction generates all the required predicates. We can use them as follows:
main :: IO ()
main = mapM_ print [isCircle s1, isSquare s2, isTriangle s3]
  where
    s1 = Circle 4
    s2 = Square 10
    s3 = Triangle 1 1 1

The reify function is extremely powerful: it opens the door to the compilation pipeline and reports almost every piece of information known to the compiler at the moment when TH is fired up.

hid-examples on  ch12 via λ 9.2.7 via ❄  impure (ghc-shell-for-hid-examples-0.5-0-env)
❯ cabal -v0 run mstr-literals
What needs my Shakespeare for his honoured bones,
The labor of an age in pilèd stones,
Or that his hallowed relics should be hid
Under a star-ypointing pyramid?
Dear son of Memory, great heir of fame,
What need’st thou such weak witness of thy name?
Thou in our wonder and astonishment
Hast built thyself a live-long monument…

QUASIQUOTES
We’ve seen several examples of Template Haskell quotes in Oxford brackets. We can also define our own Oxford bracket processors, called quasiquoters. A quasiquoter is a value of the QuasiQuoter type defined in the Language.Haskell.TH.Quote module. This value should contain the following four functions as its fields for parsing quotes found in different contexts:

As an expression
As a pattern
As a type
As a top-level declaration

These functions should return the corresponding TH data types in the Q monad. Whenever GHC meets a quasiquotation, it calls the corresponding function and splices its result back to the code. Let’s look at an example. Remember that Haskell doesn’t contain any multiline String literals. It’s easy to implement them with quasiquotes.
We implement the following trivial quasiquoter:
module Str where

import Language.Haskell.TH ( stringE )
import Language.Haskell.TH.Quote ( QuasiQuoter(QuasiQuoter, quoteDec, quoteExp, quotePat, quoteType) )

str = QuasiQuoter {
          quoteExp = stringE
        , quotePat = undefined
        , quoteType = undefined
        , quoteDec = undefined
        }

The whole idea behind this quasiquoter is to take a String inside Oxford brackets and turn it into a String expression. The stringE function from Template Haskell does precisely that. We leave all other functions unimplemented because we don’t expect multiline String literals in other contexts except for expressions.
To use this quasiquoter, we should enable the QuasiQuotes GHC extension, import the corresponding module, and write Oxford brackets as follows:
{-# LANGUAGE QuasiQuotes #-}
module Main where

import Str ( str )

verse :: String
verse = [str|What needs my Shakespeare for his honoured bones,
The labor of an age in pilèd stones,
Or that his hallowed relics should be hid
Under a star-ypointing pyramid?
Dear son of Memory, great heir of fame,
What need’st thou such weak witness of thy name?
Thou in our wonder and astonishment
Hast built thyself a live-long monument…|]

main :: IO ()
main = putStrLn verse

If we run this program, the String is printed as it was defined, as shown next:
hid-examples on  ch12 via λ 9.2.7 via ❄ impure (ghc-shell-for-hid-examples-0.5-0-env)
❯ cabal -v0 run mstr-literals
What needs my Shakespeare for his honoured bones,
The labor of an age in pilèd stones,
Or that his hallowed relics should be hid
Under a star-ypointing pyramid?
Dear son of Memory, great heir of fame,
What need’st thou such weak witness of thy name?
Thou in our wonder and astonishment
Hast built thyself a live-long monument…

In general, we have to implement parsing of quote content by ourselves. Let’s now move to a much more sophisticated example where we exploit Template Haskell and quasiquotes.

hid-examples on  ch12 via λ 9.2.7 via ❄  impure (ghc-shell-for-hid-examples-0.5-0-env)
❯ cabal -v0 run ping-server
LOG: [::1]:60966 New connection
Echo message: Hello from client
Ping received/answered with 1
Ping received/answered with 2
Ping received/answered with 3
Echo message: Bye from client
LOG: [::1]:60966 Connection closed
hid-examples on  ch12 [!] via λ 9.2.7 via ❄  impure (ghc-shell-for-hid-examples-0.5-0-env)
❯ cabal -v0 run ping-client
"Hello from client"
PingAnswer "OK" 1
PingAnswer "OK" 2
PingAnswer "OK" 3
"Bye from client"

12.3.2 Example: Generating remote function calls
The theory of distributed computing describes several approaches to distributed communications. The most popular nowadays is explicit message passing. With this approach, both communicated sides send and receive messages. In the 1980s and 1990s, another approach was widely used, namely, remote procedure calls or remote method invocation. In this model, it looks like we call a procedure or a method of some object as usual, but it’s actually executed on the remote machine. We have a client that initiates communication and a server that serves a request. In the background, all arguments to the procedure called by a client are serialized into an envelope, sent over the network, received at the server side, deserialized, and processed. Results are then sent back in the same way. Clients in this model traditionally use blocking calls to wait for results.
In this section, we’ll implement remote function calls, an approach to distributed computing with a functional programming flavor. I wouldn’t recommend using this approach in industry nowadays. Nevertheless, it’s a good exercise in Template Haskell.
We’ll develop a library and a sample application to demonstrate its usage.
SAMPLE APPLICATION: SENDING PINGS AND ECHO REQUESTS OVER THE NETWORK
Let’s start with a sample application. It consists of a client and a server with a few common definitions.
Our application is only about 50 lines of Haskell code. Almost all the functionality is hidden in the library. Clients and servers share data types and an environment for executing code. The following are some common definitions.
{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE DeriveAnyClass #-}
{-# LANGUAGE DerivingStrategies #-}
module PingCommon where

import GHC.Generics ( Generic )
import Data.Serialize ( Serialize )  -- The module from the cereal package

import RpcCommon ( RSIO, RemoteState(..) )  -- The module from our rpc-lib internal library

instance RemoteState Integer where  -- State for one client-server session
    initState = 0

type RemotePing a = RSIO Integer a  -- Main application monad to work with

data PingAnswer = PingAnswer String Integer  -- Data type used as a result of the ping call
  deriving stock (Show, Generic)
  deriving anyclass Serialize

Every value sent over the network should be properly serialized. We use the cereal library, which uses GHC.Generics to implement default serialization methods. We’ll come to the serialization later in this section.
The client uses QuasiQuotes to generate remote function stubs and calls them in a simple communication session, as shown next:
{-# LANGUAGE QuasiQuotes #-}

import Control.Monad ( replicateM_, void )
import Control.Monad.Trans ( MonadIO(liftIO) )

import ClientUtils ( runRemote, remote, callRemote )
import PingCommon ( PingAnswer, RemotePing )

[remote|
ping :: RemotePing PingAnswer
echo :: String -> RemotePing String
 |]  -- Quasiquoter that generates function bodies

example :: Int -> RemotePing ()
example n = do
    echo "Hello from client" >>= prt  -- Calls the generated echo function and receives its result
    replicateM_ n (ping >>= prt)
    echo "Bye from client" >>= prt
  where
    prt :: Show a => a -> RemotePing ()
    prt = liftIO . print

main :: IO ()
main = void $ runRemote "localhost" 1500 (example 3)  -- Establishes the remote connection and executes an action remotely

The server is also straightforward. We give trivial implementations of the remote functions and start serving requests as follows:
{-# LANGUAGE TemplateHaskell #-}
import Control.Monad.State ( MonadIO(liftIO), modify, MonadState(get) )
import ServerUtils ( genServer, runSerialized, serveRPC )
import PingCommon ( PingAnswer(..), RemotePing )

ping :: RemotePing PingAnswer
ping = do
  modify (+1)
  n <- get
  liftIO $ putStrLn $ "Ping received/answered with " <> show n
  pure $ PingAnswer "OK" n

echo :: String -> RemotePing String
echo msg = do
  liftIO $ putStrLn $ "Echo message: " <> msg
  pure msg

genServer ['ping, 'echo]  -- Generates the server function to serve requests to the ping and echo functions

main :: IO ()
main = server "localhost" 1500

Note how we refer to the declared functions with 'ping and 'echo. The tick here is used to get the TH Name of the corresponding functions. Remember, we’ve used double ticks to refer to type names before. Note also that there is no explicit Template Haskell splice for genServer here. GHC allows calling TH functions as top-level declarations without the $(…) syntax.
If we run the server, it blocks and waits for requests, as shown next:
hid-examples on  ch12 via λ 9.2.7 via ❄ impure (ghc-shell-for-hid-examples-0.5-0-env)
❯ cabal -v0 run ping-server
...

Running a client results in the following output:
hid-examples on  ch12 [!] via λ 9.2.7 via ❄ impure (ghc-shell-for-hid-examples-0.5-0-env)
❯ cabal -v0 run ping-client
"Hello from client"
PingAnswer "OK" 1
PingAnswer "OK" 2
PingAnswer "OK" 3
"Bye from client"

The server logs the following communication also:
hid-examples on  ch12 via λ 9.2.7 via ❄ impure (ghc-shell-for-hid-examples-0.5-0-env)
❯ cabal -v0 run ping-server
LOG: [::1]:60966 New connection
Echo message: Hello from client
Ping received/answered with 1
Ping received/answered with 2
Ping received/answered with 3
Echo message: Bye from client
LOG: [::1]:60966 Connection closed

Now let’s see how all this stuff is implemented.
COMMUNICATION MODEL: REMOTE FUNCTION CALLS
We want to implement the following communication model in our library:

The client calls a function, which is actually a stub.
This stub packs all the arguments with a function name and sends them over the network to the server.
The server receives a request, unpacks it, looks up the requested function, and calls it with the arguments provided with the request.
The function is evaluated on the server and gives back a result.
The server packs the result and sends it back to the client.
The client’s stub receives the result, unpacks it, and gives it to a caller.

While packing and unpacking values, we have to follow a specific strategy. We use serialization and deserialization for Haskell values. We work with bytestrings to pack them into envelopes that we send over the network. These envelopes contain a size field and a payload of this size. Whenever we receive an envelope, we receive the payload size first and then wait for the payload of that size. This method is important to deserialize values properly.

LIBRARY STRUCTURE
Now let’s get closer to the library code. We have the following modules:

RpcCommon provides common definitions used by other modules and clients. These include mainly data types and the RSIO monad stack, which combines the Reader, State, and IO monads.
ClientUtils and ServerUtils provide interfaces for clients and servers, respectively. They reexport many required definitions from other modules and implement client- and server-specific functionalities.
RemoteIO implements value passing with serialization and provides wrappers over networking primitives.
DeclsGenerator is a module for generating code for clients and servers with Template Haskell. Both the remote quasiquoter and genServer used in the previous sample application are defined in this module.
RemoteParser provides a utility function to parse a remote interface as a list of type signatures we’ve used in the client definition.

We have about 220 lines of code in the library. Nevertheless, it’s not going to be an easy walk, because the code is quite convoluted. Note that I won’t talk about components we’ve already discussed many times earlier in this book, such as monad stacks and exception handling. Refer to the previous chapters if you need to refresh your memory.
VALUE PASSING AND NETWORKING
Passing values between clients and servers is a well-known problem. Our key ingredient for value passing is the Serialize type class, provided by the cereal library. We use it mostly through the following couple of functions:
encode :: Serialize a => a -> ByteString
decode :: Serialize a => ByteString -> Either String a

They are implemented through the low-level put and get methods of the Serialize type class. Encoding always runs smoothly, but decoding may occasionally fail. That’s why the decode function gives us Either String a back. The left component contains an error message.
Because we do decoding several times, we have the following data type to distinguish between stages:
data DecodeStages = Stage0 | Stage1 | Stage2
  deriving Show

The stages are as follows:

At Stage0, we decode the first field of an envelope (the size of the payload).
At Stage1, we decode the second field of an envelope to a ByteString.
At Stage2, we decode the ByteString to a value we expect (either a tuple with all the parameters or a result of a function call, depending on the side we are on).

The same stages are used to encode parameters in the opposite order, though we don’t have to report them because we expect no problems during encoding.
The following function deals with processing decoding results:
unEitherStaged :: DecodeStages -> Either String a -> RSIO st a
unEitherStaged stage eValue = either (throwRemote . errMsg) pure eValue
  where
    errMsg msg = "Decoding error (" <> show stage <> "): " <> msg

We either return a decoded value or report an error within the monad stack.
Both encode and decode functions are used in the following callRemote function, which implements our remote function call model on the client side:
type Operation = String
type RemoteAction st a b = a -> RSIO st b

callRemote :: (Serialize a, Serialize b) => Operation -> RemoteAction st a b
callRemote operation params = do
    sendRSIO (operation, encode params)  -- Sends the request with the required function name and encoded arguments
    answer <- receiveRSIO  -- Receives a result
    unEitherStaged Stage2 (decode answer)  -- Decodes the result

We’ll get to sendRSIO and receiveRSIO shortly.
The server side stores the table with all the registered functions as follows:
type RPCTable st = [(Operation, RemoteAction st ByteString ByteString)]

Note that these functions operate on the server side over the ByteString encoded parameters. We’ll use the following wrapper to generate such functions from the functions over serializable parameters:
runSerialized :: (Serialize a, Serialize b) =>
                 RemoteAction st a b ->
                 RemoteAction st ByteString ByteString
runSerialized action params
  = unEitherStaged Stage2 (decode params) >>= fmap encode . action

Note that the params argument of runSerialized refers to parameters of the resulting RemoteAction over ByteString.
For every request, the server looks up the RPCTable and calls the corresponding wrapped function as follows:
serveRequest :: RPCTable st -> RSIO st ()
serveRequest rpcTable = receiveRSIO >>= call >>= sendRSIO
  where
    call (operation, params) =
      case lookup operation rpcTable of
        Nothing -> throwRemote
                   $ "Unsupported operation (" <> operation <> ")"
        Just func -> func params

The sendRSIO and receiveRSIO functions are responsible for encoding and decoding at stages 0 and 1. They also initiate networking operations. The sendRSIO is implemented as follows:
sendRSIO :: Serialize a => a -> RSIO st ()
sendRSIO msg = do
    conn <- ask  -- The connection is kept in the Reader environment.
    liftIO $ connectionPut conn $ buildMsgEnvelope $ encode msg  -- Builds an envelope and sends it over the network
  where
    buildMsgEnvelope payload = runPut $ do  -- Constructs an envelope using low-level Serialize methods
      putWord64be (fromIntegral $ BS.length payload)  -- Creates the size field as a big-endian Word64
      putByteString payload  -- Creates the rest of the envelope

The receiveRSIO deals with receiving messages over the network, staged decoding, and potential errors as follows:
receiveRSIO :: Serialize a => RSIO st a
receiveRSIO = ask >>= \conn ->
          recvExact conn msgSizeField  -- Receives the size field
          >>= unEitherStaged Stage0 . runGet getWord64be  -- Decodes the size field as a big-edian Word64
          >>= recvExact conn . fromIntegral  -- Receives the payload
          >>= unEitherStaged Stage1 . decode  -- Decodes the payload
  where
    recvExact conn sz =
      catch (liftIO $ connectionGetExact conn sz)
            (\e -> if isEOFError e then throwM ConnectionClosed
                   else throwRemote (displayException e))

Note the type signatures of these functions, shown next:
sendRSIO :: Serialize a => a -> RSIO st ()
receiveRSIO :: Serialize a => RSIO st a

We can use them for sending and receiving both arguments and results, given that they are serializable. This creates a source of fragility in our implementations. If we send something unexpected at a particular point, then decoding fails, and we can’t continue. Binary protocols are known to be extremely fragile and rarely used at the application level nowadays.
Thus, neither a client nor a server has to deal with encoding. Everything is done by the library.
As for networking, we use the following three libraries:

network for basic definitions (such as the port number)
connection for establishing client-server connections and sending and receiving packets over the network
network-simple for simple server implementation

The main entry point for a client is the runRemote function, as shown next:
runRemote :: RemoteState st => String -> PortNumber -> RSIO st a -> IO a
runRemote host port computation = do
  conn <- remoteConnectTo host port
  res <- runRemoteConn conn computation
  liftIO $ connectionClose conn
  pure res

It establishes a connection to the specified server, runs all the operations, and closes the connection afterward.
The main entry point for a server is the serveRPC function, shown here:
serveRPC :: RemoteState st =>
          HostName -> PortNumber -> RPCTable st -> IO ()
serveRPC host portNum rpcTable = serve (Host host) (show portNum) procRequests
  where
    connParams = ConnectionParams host portNum Nothing Nothing

    procRequests (connSock, sockAddr) = do
      logConnection "New connection" sockAddr
      initCtx <- initConnectionContext
      conn <- connectFromSocket initCtx connSock connParams
      catch (runRemoteConn conn $ forever $ serveRequest rpcTable)
            (\(e :: RemoteException) ->
               logConnection (displayException e) sockAddr)

    logConnection msg sockAddr =
      putStrLn $ "LOG: " <> show sockAddr <> " " <> msg

The serve function from the Network.Simple.TCP module creates a separate thread for every incoming connection. We run the procRequests function in that thread.

ghci> import DeclsGenerator
ghci> :type $(curryAll 3)

<interactive>:1:1: error:
    parse error on input ‘$’
    Perhaps you intended to use TemplateHaskell

<interactive>:1:1: error: [-Woperator-whitespace-ext-conflict, -Werror=operator-whitespace-ext-conflict]
    The prefix use of a ‘$’ would denote an untyped splice
      were the TemplateHaskell extension enabled.
    Suggested fix: add whitespace after the ‘$’.
ghci> :set -XTemplateHaskell
ghci> :type $(curryAll 3)
$(curryAll 3) :: (((a, b1), b2) -> c) -> a -> b1 -> b2 -> c

GENERATING CLIENT DEFINITIONS
Now let’s get to the main point: using Template Haskell. Remember that the client uses the following quasiquote to describe the remote interface:
[remote|
ping :: RemotePing PingAnswer
echo :: String -> RemotePing String
 |]

We should parse these type signatures and generate the ping and echo function stubs, which are responsible for sending requests to a server and receiving their results, respectively.
The quasiquoter is defined as follows:
remote :: QuasiQuoter
remote = QuasiQuoter {
    quoteExp = undefined,
    quotePat = undefined,
    quoteType = undefined,
    quoteDec = quoteFuncInfoDec
  }

We want to replace the quote with the generated function declarations, as shown next:
quoteFuncInfoDec :: String -> Q [Dec]
quoteFuncInfoDec quote = parseRemoteInterface quote >>= genClientStubs

Let’s look at parsing interfaces first. Note the problem: we have to parse Haskell type signatures here. Clearly, this is not a problem we want to solve manually. Instead, we apply the following packages:

haskell-src-exts to parse type signatures
haskell-src-meta to turn parsed types to Template Haskell type representations

WARNING These two packages for parsing Haskell sources are becoming part of history. It’s not a good idea to start new projects that rely on them. Nowadays, GHC provides the GHC API for doing the same things. This API is accessible via the ghc-lib-parser package. Unfortunately, solving our easy problems with this highly sophisticated package is not that easy. That’s why I’ve decided to use good old haskell-src-exts and haskell-src-meta in this book.

For every type signature, we’d like to get the following record:
data FuncInfo = FuncInfo {
    name :: String
  , ty :: TH.Type
  }

The parseRemoteInterface and funcInfo functions do all the work, as shown next:
parseRemoteInterface :: String -> Q [FuncInfo]
parseRemoteInterface quote = concat <$> mapM (funcInfo . parseDecl) tysigs
  where
    tysigs = filter (not . null) $ map (dropWhile isSpace) $ lines quote

funcInfo :: ParseResult (Decl SrcSpanInfo) -> Q [FuncInfo]
funcInfo (ParseOk (TypeSig _ ids t)) =
   pure $ [FuncInfo {..} | Ident _ name <- ids,
                       let ty = toType t]
funcInfo err =
  fail $ "Error when parsing remote interface (type signature expected)\n"
         <> show err

Here we use the parseDecl function from haskell-src-ext to parse a Haskell declaration. It gives us the Decl type, which contains all the details about the type signature wrapped into the ParseResult type. If parsing is successful, we look into the result and extract identifiers and their type. Finally, we create a list of FuncInfo from those identifiers and types converted with the toType function from haskell-src-meta to be compatible with Template Haskell types.
Now we need to generate function stubs as follows:
genClientStubs :: [FuncInfo] -> Q [Dec]
genClientStubs fis = concat <$> mapM (genClientStub "callRemote") fis

Remember that remote functions are allowed to have any number of arguments. We’ve decided to put all of them into a tuple. We can do that by calling the curry function several times as follows:
curryAll :: Int -> Q Exp
curryAll 0 = [| \f -> f () |]
curryAll 1 = [| id |]
curryAll n
  | n > 1 = [| curry . $(curryAll (n-1)) |]
  | otherwise = fail "curryAll argument can't be negative"

This Template Haskell code generates a lambda function, which creates a function with many arguments from a function with one argument given as a tuple. For example, with curryAll 3, we get the following:
hid-examples on  ch12 via λ 9.2.7 via ❄ impure (ghc-shell-for-hid-examples-0.5-0-env)
❯ cabal -v0 repl rpc-lib
ghci> import DeclsGenerator
ghci> :set -XTemplateHaskell
ghci> :type $(curryAll 3)
$(curryAll 3) :: (((a, b1), b2) -> c) -> a -> b1 -> b2 -> c

The (((a, b1), b2) -> c) single argument function runs over the network, but the user works with the a -> b1 -> b2 -> c part of the generated function.  The client stub itself uses this function to build the declaration we need, as shown here:
genClientStub :: String -> FuncInfo -> Q [Dec]
genClientStub callee FuncInfo {..} = do
    funcImpl <-  funD funName [clause [] (normalB stubBody) []]  -- Builds a function declaration from a name and a body
    pure [typeSig, funcImpl]
  where
    funName = mkName name
    typeSig = SigD funName ty  -- Builds a type signature from a name and a type
    stubBody = [| $(curryAll (arity ty)) $ $(dyn callee) name |]  -- Builds a function body with currying

Note that some of the components are built purely with constructors (e.g., SigD for type signatures) whereas others require using Q computations (e.g., function declaration with funD). Just as before, we combine quotes with splices.
The dyn function creates a reference to a function from the String representation of its name. We use it to refer to the callRemote function. For example, if we call the echo function, the $(dyn callee) name part of the body generates callRemote "echo" augmented with argument currying.
The arity function inspects a Template Haskell type representation and determines the number of function arguments as follows:
arity :: Type -> Int
arity (ForallT _ _ rest) = arity rest
arity (AppT (AppT ArrowT _) rest) = arity rest + 1
arity _ = 0

This is a simplified version of such a function, though it works for most type signatures. The Type type has more than two dozen constructors, and using some of them in a type may lead to the wrong results.
GHC with the -ddump-splices flag reports the following generated functions for our ping client:
ping :: RemotePing PingAnswer
ping = ((\ f_akLr -> f_akLr ()) $ callRemote "ping")
echo :: String -> RemotePing String
echo = (id $ callRemote "echo")

Let’s add the two-argument function as follows:
echo2 :: String -> String -> RemotePing String

Now we have the following function generated:
echo2 :: String -> String -> RemotePing String
echo2 = ((curry . id) $ callRemote "echo2")

The client is ready to work. Let’s move to the server.
GENERATING SERVER DEFINITIONS
The server needs the server function working over the RPCTable with remote functions. We generate both as follows:
genServer :: [Name] -> Q [Dec]
genServer names =
  [d|
    server :: String -> PortNumber -> IO ()
    server host port = serveRPC host port $(genRemoteTable names)
   |]

This function takes the names of functions that are supposed to be called remotely. We hand them to RPCTable augmented with serialization and deserialization. To do that, we first get their types and then wrap with runSerialized we’ve already discussed as follows:
genRemoteTable :: [Name] -> Q Exp
genRemoteTable names =
  mapM reifyFunc names
  >>= listE . map (genServerStub "runSerialized")

The reifyFunc function consults the compiler with reify and gives back the FuncInfo record, as shown next:
reifyFunc :: Name -> Q FuncInfo
reifyFunc nm = do
  VarI _ t Nothing <- reify nm
  pure $ FuncInfo (nameBase nm) t

The genServerStub builds an RPCTable entry for one remote function as follows:
genServerStub :: String -> FuncInfo -> ExpQ
genServerStub callee FuncInfo {..} =
    [| (name, $(dyn callee) $ $(uncurryAll (arity ty)) $(dyn name) ) |]

The uncurryAll does the opposite to the curryAll function, as shown here:
uncurryAll :: Int -> Q Exp
uncurryAll 0 = [| (const :: a -> () -> a) |]
uncurryAll 1 = [| id |]
uncurryAll n
  | n >1 = [| uncurry . $(uncurryAll (n-1)) |]
  | otherwise = fail "uncurryAll argument can't be negative"

For example, in our ping-server, we have the following code generated:
server_akVF :: String -> PortNumber -> IO ()
server_akVF host_akVG port_akVH
  = ((serveRPC host_akVG) port_akVH)
      [("ping", (runSerialized $ (const :: a_akVI -> () -> a_akVI) ping)),
       ("echo", (runSerialized $ id echo))]

Although not too much code is generated here, such code is extremely boring to write, so using Template Haskell definitely pays off.


new app: solarweatherclock

New app that displays the current grid export of solar from home assistant as well as the weather and time

I have read the CLA Document and I hereby sign the CLA


spring-cloud-stream stopped appearing at version 3.x

3.0.1 is clearly in Maven Central (https://repo1.maven.org/maven2/org/springframework/cloud/spring-cloud-stream/3.0.1.RELEASE/), but it never showed up in javadoc.io (https://www.javadoc.io/doc/org.springframework.cloud/spring-cloud-stream/3.0.1.RELEASE/index.html).

This is because the spring-cloud-stream-3.0.1.RELEASE-javadoc.jar doesn't have an index.html under the root folder. Can you tell me how you are generating javadoc.jar?

We're working on fixing it, will let you know

SO we fixed the stream and I can confirm now that the  javadoc JAR file does indeed contains index.html. So it will be available when we cut 3.0.2.RELEASE.

SO we fixed the stream and I can confirm now that the javadoc JAR file does indeed contains index.html. So it will be available when we cut 3.0.2.RELEASE.

cool. I am closing the issue now. Feel free to reopen if it still doesn't work.


Cropped text in the Open Beauty Facts homescreen

Describe the bug
Dear developer:
In the process of using, I found a bug in layout / display. The text "database" under the logo of the homepage is half blocked, which may be an adaptation problem.
To Reproduce
Steps to reproduce the behavior:

Go to 'homepage'
See error

Expected behavior
I hope this part of the text can be displayed completely, thank you!
Screenshots

Smartphone (please complete the following information):

Device: [Google nexus 6]
OS: [Android 8.0]
Version [App 3.2.8]
Screen resolution setting: 1440 * 2560
font size: largest

Thank you!

@teolemon, I checked with the Google Nexus 6 in emulator. This issue does not seem to exist anymore

@franklinbill can you confirm ?


docs: fix typo in link fragment

This PR corrects the wrong #partialt fragment identifier to #partialtype.

Thanks!


Add TypeScript arm-appconfiguration readme

Automation for azure-sdk-for-python

Automation for azure-sdk-for-java

Automation for azure-sdk-for-go

Automation for azure-sdk-for-ruby

Can one of the admins verify this patch?

@kpajdzik do you know why JS is failling?

@kpajdzik do you know why JS is failling?

It seems like removing version of autorest.typescript caused the issue. We can ignore it.

@openapi sdkautomation rebuild

@openapi sdkautomation rebuild

@openapi sdkautomation rebuild


Update nf-synchapi-createmutexa.md

Fix two malformed links, and specify the behavior of the function when it is used to acquire a handle to a named mutex that exists before the call.

@turingcompl33t : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@Karl-Bridge-Microsoft : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@Karl-Bridge-Microsoft : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off


Sprite from texture not showing up

Im using a pixi app to create my own textures (in this case basic rectangles) that i then use in the sprites. Sort of a cache
I previously used canvas and converted it to a URL to then use that url as image of the sprite, but i now wanted to convert that to a pixi app to have more "control" of how i create things
I generate the texture like this:
                const g = new Graphics()
                g.beginFill(new Color(noteData.background).rgbNumber())
                g.drawRoundedRect(
                    this.margin / 2,
                    this.margin / 2,
                    Math.ceil(this.noteWidth - this.margin - 1),
                    Math.floor(this.noteHeight - this.margin - 1),
                    radius
                ).endFill()
                const myCache = this.app.renderer.generateTexture(g)

and use it like this in the react component:
<Sprite
        texture={myCache}
/>

But this doesnt show anything in the stage, its all transparent. I tried to use a normal canvas and the pixi instance i used to create the cache and doing this works:
const sprite = new Sprite(myCache
this.app.stage.addChild(sprite)

what could be the issue? i cant find anything in the docs about this

@inlet/react-pixi version: 6.6.5
React version: 17.0.2
ReactDOM version: 17.0.2
PIXI version: 6.1.3

Not sure why yours isn’t working, but it might be the way you create the actual texture, here’s a working demo:
https://codepen.io/inlet/pen/WNGodOa

https://codepen.io/specy-the-bold/pen/eYeXvdx?editors=0110
Here it shows the problem, i think that the texture needs to come from the same renderer for it to work, i tried to use a third pixi app to render it and then add it to the second app, and that also didnt show up.
Do you know some other feature i could use to get a similar outcome? Im just trying to create my own texture to then use it in a sprite. Alternatively, can i get a reference of the renderer used in the react Pixi stage and then use that to generate the cache?

Sure you can use the useApp hook and get the app renderer:
https://reactpixi.org/hooks#useapp

I'm using class components so I'll have to use ref I guess, I'll see what I can do. Thanks for the help!

For anyone which might need this in the future, yes the issue was that you can render only a texture that comes from the same renderer. If you want to create your own cache, get the ref of the app from the stage, then use that to generate the textures


Update serper_dev_tool.py

renamed seper_dev_tool to serper_dev_tool
renamed stirng to string

Fixed in latest release 0.19


Fix inefficiency rich progress bar

What does this PR do?

Fixes #18366

Before submitting

Was this discussed/agreed via a GitHub issue? (not for typos and docs)
[x] Did you read the contributor guideline, Pull Request section?
[x] Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
[x] Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review
Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

[ ] Is this pull request ready for review? (if not, please submit in draft mode)
[ ] Check that all items from Before submitting are resolved
[ ] Make sure the title is self-explanatory and the description concisely explains the PR
[ ] Add labels and milestones (and optionally projects) to the PR so it can be classified

nice, could you pls add some minimal benchmarks for this just to have an idea of how it helps... 🐰

I created a small benchmark script that invokes the render function 10 000 times for both implementations, comparing their timings. The difference between both approaches becomes more significant as the number of logged metrics grows. Therefore, the benchmark compares the approaches for different number of logged metrics, ranging from 0 to 99.
This is the corresponding script:
import timeit
from typing import cast

import matplotlib.pyplot as plt
from rich import get_console, reconfigure
from rich.progress import Task, TaskID
from rich.text import Text

from lightning.pytorch import Trainer
from lightning.pytorch.callbacks import RichProgressBar
from lightning.pytorch.callbacks.progress.rich_progress import (
    CustomProgress,
    MetricsTextColumn,
)
from lightning.pytorch.demos.boring_classes import BoringModel

class OptimizedTextColumn(MetricsTextColumn):
    def render(self, task: "Task") -> Text:
        assert isinstance(self._trainer.progress_bar_callback, RichProgressBar)
        if (
            self._trainer.state.fn != "fit"
            or self._trainer.sanity_checking
            or self._trainer.progress_bar_callback.train_progress_bar_id != task.id
        ):
            return Text()
        if self._trainer.training and task.id not in self._tasks:
            self._tasks[task.id] = "None"
            if self._renderable_cache:
                self._current_task_id = cast(TaskID, self._current_task_id)
                self._tasks[self._current_task_id] = self._renderable_cache[
                    self._current_task_id
                ][1]
            self._current_task_id = task.id
        if self._trainer.training and task.id != self._current_task_id:
            return self._tasks[task.id]

        text = " ".join(self._generate_metrics_texts())
        return Text(text, justify="left", style=self._style)

    def _generate_metrics_texts(self):
        for k, v in self._metrics.items():
            yield f"{k}: {round(v, 3) if isinstance(v, float) else v}"

class OptimizedProgressBar(RichProgressBar):
    def _init_progress(self, trainer: "pl.Trainer") -> None:
        if self.is_enabled and (self.progress is None or self._progress_stopped):
            self._reset_progress_bar_ids()
            reconfigure(**self._console_kwargs)
            self._console = get_console()
            self._console.clear_live()
            self._metric_component = OptimizedTextColumn(trainer, self.theme.metrics)
            self.progress = CustomProgress(
                *self.configure_columns(trainer),
                self._metric_component,
                auto_refresh=False,
                disable=self.is_disabled,
                console=self._console,
            )
            self.progress.start()
            # progress has started
            self._progress_stopped = False

class CustomModel(BoringModel):
    def __init__(self, number_of_metrics=None):
        self.number_of_metrics = number_of_metrics
        super().__init__()

    def training_step(self, *args, **kwargs):
        res = super().training_step(*args, **kwargs)
        loss = res["loss"]
        self.log_loss("train", loss)

    def log_loss(self, phase, loss):
        for i in range(self.number_of_metrics):
            self.log(f"{phase}_loss_{i}", loss, prog_bar=True)

class BenchMarker:
    n_experiments: int = int(1e5)

    def start(self):
        original_timings = []
        optimized_timings = []
        number_of_metrics_list = []

        for number_of_metrics in range(100):
            original, optimized = self.generate_timings(number_of_metrics)
            original_timings.append(original)
            optimized_timings.append(optimized)
            number_of_metrics_list.append(number_of_metrics)

        print(original_timings)
        print(optimized_timings)

        plt.plot(number_of_metrics_list, original_timings, label="original timings")
        plt.plot(number_of_metrics_list, optimized_timings, label="optimized timings")
        plt.xlabel("Number of logged metrics")
        plt.ylabel(f"Timing of {self.n_experiments:.1e} function calls [s]")
        plt.legend()
        plt.show()

    def generate_timings(self, number_of_metrics: int):
        original_progress_bar = RichProgressBar()
        optimized_progress_bar = OptimizedProgressBar()
        for progress_bar in (original_progress_bar, optimized_progress_bar):
            self.setup_progress_bar(progress_bar, number_of_metrics)
            task = progress_bar.progress.tasks[0]
            metrics_column = progress_bar.progress.columns[-1]
            yield timeit.timeit(
                lambda: metrics_column.render(task), number=self.n_experiments
            )

    @classmethod
    def setup_progress_bar(cls, progress_bar, number_of_metrics):
        trainer = Trainer(
            num_sanity_val_steps=0,
            max_epochs=1,
            limit_train_batches=1,
            limit_val_batches=1,
            check_val_every_n_epoch=100,
            callbacks=progress_bar,
        )
        model = CustomModel(number_of_metrics=number_of_metrics)
        trainer.fit(model)

if __name__ == "__main__":
    BenchMarker().start()

The images display the benchmark results obtained from two distinct processors, respectively:

Intel® Celeron® N5095A @ 2.00GHz
AMD Ryzen Threadripper 3960X 24-Core Processor

For logged metrics in the range of 90-100, there's an average performance gain of approximately 10%. However, with fewer logged metrics, the performance differential is negligible.
Furthermore, the new approach aligns better with Pythonic conventions. This allows for implementing the next PR in a more readable way.

Please let me know if I can provide additional information or if you would like to see some additional benchmarks!


github.com/ritjoe deprecated

Please be advised that https://github.com/FOSSRIT/hfoss exists. I have been inconsistent with merging ritjoe/hfoss back into the master branch of this repository but have done so at least once, 3 years ago.
More recently, I created the branch https://github.com/FOSSRIT/hfoss/tree/2015F-2020S as being up to date with what I had in ritjoe/hfoss at the end of the Spring 2020 semester (RIT term 2195 as per https://www.rit.edu/calendar/1920). I know of no reason this couldn't be merged into the FOSSRIT/hfoss master branch but I figured this was a cleaner, more obvious waypoint.
The third branch in FOSSRIT/hfoss (develop) I'll address in a different issue.
I saw some links in a document associated with the instance of HFOSS that started today (Spring 2021, RIT term 2205) pointing back to ritjoe/hfoss that would probably be better directed to one of the branches of FOSSRIT/hfoss
@Nolski @jwflory @whenbellstoll @itprofjacobs

I titled this issue (#87) "github.com/ritjoe deprecated" but I did not make it so clear in the body of that first message that I am retiring the ritjoe github account.
I figured I'd let this past semester (Spring 2021, RIT Term ID 2205) play out before making any potentially-breaking changes.
Now that the semester is in the can, I'm looking again to move forward with this.

I titled this issue (#87) "github.com/ritjoe deprecated" but I did not make it so clear in the body of that first message that I am retiring the ritjoe github account.

And I'm commenting from the @ritjoe account in question, to cross-validate the basic premise of this plan.

Since ritjoe/ritjoe was meant to be the metarepository for @ritjoe I may put more of the moment-by-moment chatter about this process there, in ritjoe/ritjoe#1
that content will not survive within Github past the removal of @ritjoe but can still live on in, for example, email notifications.

Over a year since opening this issue. Am going to get around to deleting this all Real Soon Now.

I think the added 2015F-2020S branch is enough to allay concerns. I imagine @itprofjacobs updated the HFOSS syllabus by now to point towards other places. I'll close this issue because I think all historical work from 2015 to 2020 is already preserved in the FOSSRIT/hfoss repo, so no further action is needed.
Thanks for helping keep this history preserved @deejoe!


[text-to-speech] Japanese text must be percent encoded before passed to synthesize method

When I use the synthesize method with Japanese text, I get an error. However, if I percent encode the text before passing it to the TTS API, it works properly.
It seems like the percent encoding should be handled internally.
Steps to reproduce:
let tts = TextToSpeech(username: "...", password: "...")
let failure = { (error: NSError) in print(error) }
tts.synthesize("こんにちは", voice: SynthesisVoice.JP_Emi, customizationID: nil, audioFormat: .WAV, failure: failure) { ... }

Expected behavior:
The voice should speak the word "konnichiwa"
Actual behavior:
The failure callback is invoked, and this error is printed:
Error Domain=com.ibm.watson.developer-cloud.TextToSpeechV1 Code=0 "Returned audio is in an unexpected format." UserInfo={NSLocalizedFailureReason=Returned audio is in an unexpected format.}
If I manually percent encode the string, it works:
let tts = TextToSpeech(username: "...", password: "...")
let failure = { (error: NSError) in print(error) }
tts.synthesize("こんにちは".stringByAddingPercentEncodingWithAllowedCharacters(NSCharacterSet.URLQueryAllowedCharacterSet())!, voice: SynthesisVoice.JP_Emi, customizationID: nil, audioFormat: .WAV, failure: failure) { ... }

Good catch! We'll push a bug fix soon and be sure to add a test with Japanese.

Great! =)

I was able to add a test for Japanese text with no problem--I did not need to add any additional percent encoding. Browsing through the code, though, I think you received an improperly formatted error response from Watson, which the SDK tried to parse as WAV data.
For example, if your credentials were invalid, then you would receive an HTML page like that described in #423. Since that HTML page isn't parsed as a JSON error, it is instead parsed as WAV data. Of course, that's not a proper WAV file, resulting in the "Returned audio is in an unexpected format" error message.
So I'm going to close this issue in favor of #423 and #375, which should introduce better error handling to the SDK and avoid the cryptic "Returned audio is in an unexpected format" message.


TEL-4654 Reverting to old version of Python pending TEL-4464

Reverts hmrc/telemetry-lambda-resources#52
Will park these changes until https://jira.tools.tax.service.gov.uk/browse/TEL-4464 is played


RDK-39119: Implement LED control

Reason for change:
Implement LED control
Test Procedure: None
Risks: Low
Change-Id: Ia8a147c3c317213f680418be019fb005628f7b99 Signed-off-by:Anooj Cheriyan<EMAIL_ADDRESS>

Due to merge conflict new pull request created: https://github.com/rdkcentral/rdkservices/pull/3535


When running the precommit commands from Eclipse, such a shell is used.
@enisoc

LGTM

This doesn't work for me, the shell that's used by my regular 'git commit' command is non-interactive. I had to hack it to revert this to work... Please find another way :)

tagging @michael-berlin to be sure you see it :)


Neighbourhood processing: Configurable radius by lead times

As an IMPROVER developer I want the command line interface for basic neighbourhood processing to have configurable lead-time-dependent kernel radii so that I can later experiment with increasing radius with lead times.
Acceptance criteria:

Start after #5 is closed
Specify an optional lead time radius option to bin/improver-nbhood
This option should somehow refer to a configuration file which matching lead times vs specific radii (e.g. a flat 2 column text file)
Lead times not mentioned in the configuration file should be derived by linear interpolation
Write a CLI acceptance test (consult Ben F for guidelines)

Discussed the usage of a lookup file, as opposed to the current implementation in #52, which passes in the radii and lead times as arguments with @benfitzpatrick and Nigel Roberts. The current implementation involving passing arguments seems fine with a future neighbourhood processing ticket required to deal with adding functionality for ensemble members.

A follow-on issue (#59) focuses on the development of acceptance tests for the neighbourhood processing.

Pull Request merged (#54) and was in May demo.


explicitly pass unix_socket

this affects only kubic for now where we use PyMySQL
we cant use the MYSQL_UNIX_PORT workaround anymore as we could
do with MySQLdb
salt#mysql-unix-socket
Signed-off-by: Maximilian Meister<EMAIL_ADDRESS>not to be merged yet
depends on
https://github.com/saltstack/salt/pull/47454

rebased

So we keep https://github.com/kubic-project/caasp-container-manifests/blob/0f1f0e9b0e233bbe73667432a1de185677f6a821/manifests/public.yaml#L154-L155 for now?

This has to be backported also to CaaSP 2.1 and CaaSP 3.0, I having following issue on CaaSP-devel-2.1-Build3.134
[ERROR   ] Could not store events - returner 'mysql.event_return' raised exception: MySQL returner could not connect to database: (2002, 'Can\'t connect to local MySQL server through socket \'/tmp/mysql.sock\' (2 "No such file or directory")')
[ERROR   ] Could not store events - returner 'mysql.event_return' raised exception: MySQL returner could not connect to database: (2002, 'Can\'t connect to local MySQL server through socket \'/tmp/mysql.sock\' (2 "No such file or directory")')
[ERROR   ] Could not store events - returner 'mysql.event_return' raised exception: MySQL returner could not connect to database: (2002, 'Can\'t connect to local MySQL server through socket \'/tmp/mysql.sock\' (2 "No such file or directory")')


Enabling keyboard shortcuts

Added --talk-name=org.freedesktop.portal.Desktop to finish-args

I switched the GUI environment of the Coot Flatpak app from Wayland to X11.
It might also work on your home Arch Linux.
Try installing the latest build from Flathub (flatpak install flathub io.github.pemsley.coot) or updating the installed Coot (flatpak install flathub io.github.pemsley.coot).


Sideloader script paths are wrong.

The paths in the build.sh script have an extra project directory dir, they should be pointing at the repo root dir instead.

https://github.com/praekelt/malaria24-django/commit/914c7ba1a07ee3e686abc264d7a217fc2c09d379 and https://github.com/praekelt/malaria24-django/commit/33ea2851146f68d69a69702f9f23cbb574b80b0d are examples of the changes I had to make.


feat: Update existing SDK environment variables

This enables processing of already admitted and modified containers. This might occur since the reinvocationPolicy has been set to IfNeeded #749.
Reason for Change:
Not updating existing environment variables might result in a situation where the wrong credentials are projected into the container, whenever another admission plugin modifies the serviceAccountName.
Requirements

[X] squashed commits
[ ] included documentation
[X] added unit tests and e2e tests (if applicable).

Issue Fixed:
Fixes #1475
Please answer the following questions with yes/no:
Does this change contain code from or inspired by another project?

[X] no

If so, did you notify the maintainers and provide attribution?
not relevant

@microsoft-github-policy-service agree company="lise GmbH"


External review of documentation

[x] review GCDevEx and BC digital marketplace documentation to see what could be repurposed/used as a starting point
[ ] Prepare draft user documentation
[ ] revisions to documentation - comments from OSME
[ ] revisions to documentation - comments from IITB procurement?
[ ] user testing of documentation - suppliers and GC users

Bit of an overlap here with UX testing...


[SYSTEMML-1069] Refactor Mailing List Section on Community Page

changed mailing list sections on home page and community page, changed github link on community page, replaced favicon

LGTM. Thank you @jazares. I'll merge.


move localization/resource strings from Strings.resx to Resources.resx

seemingly the 'automatic' localization only works for Resources.resx ?

@mwallner is this work still to be done?

Nope, already in GH-386


Draggable dialogs

Dialogs should be draggable if the screen is large. When the user drags them, they get pinned.

I'm almost (need to fix some bugs with animation) done draggable dialogs (#129). It's uses draggable from jquery ui.

Closing because its duplicate