Fix lexer performance

It turns out that the reason futhark-test has problems with large datasets (like the 50MiB kdd_cup.in file for kmeans) is that the lexer of all things consumes too much memory.  There is one spot where we convert a Text to a String in order to call read that appears to be the major cost centre, but I'm not sure why.  I've been looking for a space leak, but I can't find any...

It may be that the solution is to write a simpler value parser using Attoparsec.

Hello, again.
It has been 2 months since I touched Futhark last and it has given me time to think what I want to do about it. Last time I ran into that huge space leak in ghc-mod when loading Futhark files, so it occurs to me that rewriting the parser is a good way to sidestep that issue. It would also familiarize myself with Furhark's internals a bit, and compared to last time, I do now understand Haskell well enough to give it a try in writing a parser in it.
Unless @nqpz has already done significant work on this, I'd volunteer to spend the next few weeks getting this done. Also I see you've done some work on shared memory tiling. It was faster than I expected. With that I'd have pretty good reason to put in some work getting Futhark to run with Fsharp and even look into making a Cuda backend, though I won't make any promises until the parser is done.
So @nqpz, how far did you get? Should take a look into doing it instead?

I'll let @nqpz answer for himself, but you should be aware that the kind
of parser we need is not the one you find in a Haskell tutorial.  We do
not need a new parser for the full language, but just a parser that can
read (a subset of) the Futhark value syntax (no tuples).  This parser
will need to handle quite large inputs (dozens of MiB) and produce
fairly large arrays.  I think the most reasonable way to do this is to
write an imperative parser using the ST monad and an STArray (which is
then turned into a Futhark array at the end).  I doubt any parser
combinator library could deal with values of the size we need.  It will
look more like the parser we use in generated C code, which is easily
fast enough:
https://github.com/HIPERFIT/futhark/blob/master/rts/c/reader.h
I'm not saying it's necessarily hard, I'm just saying that it's not just
"Haskell", it's "high-performance Haskell", which is a different style
than you might have seen before.
We did however implement loop tiling, yes.  It's still a fairly rushed
implementation (paper deadline today!), probably brittle, and doesn't
yet give as much speedup as we would like.  It is, however, a start, and
we have a decent idea of where to go from here.
--
\  Troels
/\ Henriksen

Ok, I can do the above along the way. I do not think writing a fast parser for large arrays would be a particularly hard task in Haskell. I did take care to figure out how to do imperative programming in it.
I've only peeked at the source code of Futhark for a bit today, but I am more worried about monad transformers and those function that use higher kinded types. I never got a chance to get that far while I was doing HackerRank exercises, so I can only guess what those segments of the code are doing. Also it is my first time seeing FlexibleInstances and FlexibleContexts. It seems that I'll have to learn a lot to get through this challenge.
To start with, my plan is to see if I can separate the lexing and parsing stages in the code from anything else. That will let me verify if the memory leak really is there and will make it easier to test the new parser side by side. My goal with this rewrite would be to make is so the Futhark's code can be written while using ghc-mod which is really a programming necessity for me. I do love my IDEs.
Two months ago, you've mentioned that the REPL prints out the AST. Maybe I should start there?
Also are the lexing/parsing stages contained in the Languages folder? I am guessing they are. I'll let you know how it goes.

Yes, lexing and parsing is contained in the Languages folder.  It should
not be hard to write a small test program that uses the parser, so you
can isolate the inefficiency.
--
\  Troels
/\ Henriksen

Well, maybe to you. :)
I've always found it much easier to write than to read programs. Right now I am looking at Attributes.hs and it does seem hard, and it would be like that I bet even without the help from the Atom's ghc-mod plugin (which I disabled to prevent it from crashing.) Well, I expected it like that.
When you build a program by hand there is a feeling that you have about the code that is simply missing when you read someone else's. And I am still not that familiar with Haskell's style...
I'll read all of the source files in the Languages folder from top to bottom, and ask you if I still can't figure out where the parser begins and ends if that is alright?

Sure.

Just to make everything clear.  There are two parser issues:

The Happy-based parser (Parser.y) is translated into some abominable code by happy that causes ghci to choke and use huge amounts of memory when it is loaded.  This is what you run into when you try to use ghc-mod.  I don't know why this occurs - the Futhark grammar is not very complicated, but maybe everybody has this experience with Happy.  Fixing this could involve writing a new parser with Parsec, or figuring out why the current parser behaves badly in ghci.  Note that if you manage to get the parser loaded, ghci can run it just fine.  That problem is not what this issue is about, although I can see why you would want it resolved.

The other issue is that some of our tools (futhark-test and futhark-bench) have a need to parse potentially very large Futhark values (not the full grammar, just the value syntax).  A good example is the 72.2MiB kdd_cup.in from the kmeans benchmark.  I don't think any parser written using Happy or any combinator library will be able to deal with something of that size, because merely creating a list of all the values (which I think they all require) would blow up.  A better approach is a hand-written imperative parser that parses only the subset of the value syntax (no tuples or arrays of tuples) needed by futhark-test and futhark-bench.

There is one subtle detail when it comes to resolving the latter issue: the parser should produce core language values, not source language values.  That is, these: https://github.com/HIPERFIT/futhark/blob/master/src/Futhark/Representation/AST/Syntax/Core.hs#L183  I don't think this is much of an issue, though.  A good strategy is probably for the parser to work directly on an STArray, then convert it to a Futhark ArrayVal at the end.

A better approach is a hand-written imperative parser that parses only the subset of the value syntax (no tuples or arrays of tuples) needed by futhark-test and futhark-bench.

To be honest, just out of hand I am not really convinced that parser combinator libraries are a bad choice. Maybe if they use lists, but surely a custom parser could be written for a library that uses a resizable arrays? This is what I would do if I was doing this in F#. There is the Fparsec library for it based on Haskell's Parsec and it  has a feature where custom parsers can be made by the users and there is actually an example how to make a parser which returns ResizeArrays in the documentation.
I am betting there is something similar for Haskell.

You may be right.  I am not aware of any such library, but I think it
can be written.  I'm not sure it's worth going that way unless such a
library already exists, however.  The parser we need is very simple and
not likely to change much over time.
--
\  Troels
/\ Henriksen

To clarify my position a bit more, even though you are a compiler author and a PhD (I think) in this field, my impression is that you aren't too familiar with parser libraries otherwise you would not have chosen Happy which in your own words produces bad code for the Futhark project. So I'll defer my decision until I can investigate Attoparsec and perhaps some other parser libraries and see what they offer.
As an aside, over a month ago while learning Haskell by doing HackerRank exercises, I did stumble on a problem that had to take Mbs long inputs which took way too long to process. As it turned out Haskell's native String = [Char] type turned out to be the culprit, more precisely it being the input.
...Though Behzad Nouri's example uses lists to store the results of parsing his example could be modified to use Vector's unfold to make the resulting structure faster and more memory efficient. I'll have to look around before deciding, but I am sure I'll find something.

Oh sure, Happy is pretty bad.  I mostly use it because, as a parser
generator, it will tell me about ambiguities in the grammar, which is
pretty handy while it is being developed.  The code it produces is fine
and efficient for compilation, but seems to cause some pretty bad
behaviour in GHCi.  You're right that parsing is not my specialty,
though - I usually just use Parsec.
And yeah, lists (and String by extension) are performance poison for
large data sets.  That's one reason why I'm skeptical of parser
combinators, because the one's I've used have been somewhat
list-oriented (even if they supported a better string type than
String).
I'd be interested if it turns out that it is possible to write
performance and composable parsers for large inputs.
--
\  Troels
/\ Henriksen

Ok, I have no idea. As you've said, when just the parser is loaded in GHCI it indeed takes up a lot of memory - 2.6Gb on my machine - but that is not related to the ghc-mod space leak. When type checking the example program, it stands steadily at 300-400Mb, which is standard for it.
Now that I see this, I've checked and it seems the Atom editor works just fine if it is in the Languages folder. The only exception seems to be the Parser.hs file where no type information shows up at all for some reason - but that bug too is not related to the space leak. The type information simply does not show up, I can see no signs of the memory being drained in the Task Manager.
I am thinking of what I should next... Now that I see there are differences depending which file is being looked at, what I will do is go file by file to see if I can narrow the leak further somehow.

Ok, I got it to crash 3 days ago once, but now I can just see it filling back and forth. Inside futhark-pyopencl.hs it goes up to 1Gb before falling down. Then the ghc-mod just churns away for forever. But for files that are not at the bottom of the hierarchy, it actually does work without crashing – the time to respond being proportional how deep are they.
It is a definite improvement compared to two month ago, though the responsiveness even for the files in the Languages folder is below acceptable standards. It seems the ghc-mod authors did some work to keep a lid on the memory usage.
At any rate, I think understand why this is. A while ago on the /r/programming sub about the Swift's type checker choking on some really simple cases due to exponential edge cases in the type inference. Futhark code probably has such surprises for the type checker inside it as well. If I was reporting a bug today, I'd just complain that that ghc-mod is taking too long.
This is the other side of a language having a very sophisticated type system.
You know, I think I'll just make that fast value syntax parser as you've suggested so I do not waste everyone's time and call that my contribution for the time being. I'll skip rewriting the whole thing now that I see it is not the source of the leak. And now that I am looking at Futhark's code more closely, I do think that making that Cuda and Fsharp backend does seem way beyond my ability. It would have been difficult even if I was an expert at Haskell. My pride will have to be content with this. My estimate of hold long it would take me to figure out how all this works is at >3 months now.
For the parser, I'll see whether my idea of making an existing parser combinator library has any merit. I'll do a a benchmark to establish the lower bound on speed and then see how close I can make Attoparsec or something else approach that limit.
Once that is done, we'll talk about how to integrate that parser.
I'll keep you posted.

I am throwing in the towel on this. I tried various things the most successful of which was to make a new many function that returns a pesistent_vector. I'll go from top to bottom over various things that I did starting from the previous.

After going through Attoparsec's source, I asked on SO regarding this issue to solicit various suggestions. At first I was not impressed with their quality as I was bent of figuring out how to do a dynamic array elegantly - I thought I could find a way as it is such a basic data structure, but it turns out there is a good reason I could not find an easy to download library anywhere.

The way to make persistent_vectors work with Attoparsec is simple. Here is the relevant many function from the Attoparsec's Data\Attoparsec\Internal\Types.hs (that manyTill, sepBy and other combinators make use of internally.)
    many v = many_v
        where many_v = some_v <|> pure []
              some_v = (:) <$> v <*> many_v

Furthermore, Attoparsec has a variant of many that evaluates strict in the first argument which I have found to work better than the standard function.
liftM2' :: (Monad m) => (a -> b -> c) -> m a -> m b -> m c
liftM2' f a b = do
  !x <- a
  y <- b
  return (f x y)

Here is how the many parser for VP looks like:
manyVP :: Parser a -> Parser (VP.Vector a)
manyVP v = many_v
  where
    many_v = some_v <|> pure VP.empty
    some_v = liftA2' (flip VP.snoc) v many_v

parseManyNumbers :: Parser (VP.Vector Int)
parseManyNumbers = manyVP (decimal <* skipSpace)

I've generated random 10M integers from 0 to 999 and found that I could parse them in 9.8s vs 1.2 it took F# using a vanilla ResizeArray. The program chocked badly on 100M integers compared to the F# one which scaled linearly.
For the context on persistent_vectors, the documentation states that they are similar to Scala and Clojure's vectors which are 32-ary tries if Martin Odersky's slide from the Scala Coursera course can be believed.

I was not really happy with the 10x gap in performance on 10M integer sum problem, so I did more research and tried making a many that returns a Vector. In fact, Attoparsec uses mutable arrays internally that it unsafely freezes and thaws. I tried the same approach, but it did not work out. Also it uses continuations for control flow which makes understanding how it works significantly more complicated.

Perhaps somebody more knowledgeable could fix the program linked above, but the infinite loop I am getting reminds me a little of when I tried using unboxed Vectors instead of lazy ones to do memoization and got infinite loops. That was easy to fix then, but I am not sure what to do here. In structure, the parser is similar to the first one.

For the next attempt I had a good idea of trying to fit a parser into the Vector's unfoldr function as to avoid having to resize it manually and writing a bunch of incredibly ugly C-like parsers in Haskell. The problem with that idea can be shown in the type signature for unfoldr:

unfoldr :: (b -> Maybe (a, b)) -> b -> Vector a
There is no way to get the state b from the unfold. Ugh! The solution to that is to use unsafePerformIO. Here is how it looks like:
takeWhile :: (Char -> Bool) -> State -> (Text,State)
takeWhile f state =
    let mut = unsafePerformIO (newIORef state)
        {-# NOINLINE mut #-}
        unfolder state =
          let (c,state') = readChar state in
            case f c of
              True ->
                unsafePerformIO $ do
                  writeIORef mut state'
                  return $ Just(c,state')
              False -> Nothing
    in
      (T.unfoldr unfolder state, unsafePerformIO (readIORef mut))

I do not understand why, but the program on the repo is slow. It literally takes 1.2s to parse just 10 numbers.

I tried other things such as removing unsafe operations from the above by returning the state inside the Vector:

takeWhile :: (Char -> Bool) -> State -> (Text,State)
takeWhile f state@(pos,text) =
    let unfolder state =
          let (c,state'@(pos',_)) = readChar state
          in
            case f c of
              True -> Just((c,pos'),state')
              False -> Nothing
        result = L.unfoldr unfolder state
        last_list =
          case map snd result of
            [] -> pos
            x -> last x
    in
      (T.pack $ map fst result, (last_list, text))

It made no difference to performance.

I decided to use my last resort and go with your original idea which is to use mutable arrays inside the ST monad. As I have learned recently, when Haskell is good at something, but when it is bad at something, it is really, really bad. Here is how the takeWhile function looks inside a ST monad:

takeWhile :: (Char -> Bool) -> State -> (Text,State)
takeWhile f state@(pos,text) = runST $ do
  vec <- VM.new 1
  vec_ref <- newSTRef vec
  (capacity_ref :: STRef s Int) <- newSTRef 1
  (size_ref :: STRef s Int) <- newSTRef 0
  pos_ref <- newSTRef pos
  char_state_ref <- newSTRef (readChar state)

  let cond = do
        char_state <- readSTRef char_state_ref
        return $ f (fst char_state)

  whileM_ cond $ do
    vec <- readSTRef vec_ref
    char_state <- readSTRef char_state_ref
    --traceShow (fst char_state) $
    modifySTRef' pos_ref (+1)
    size <- readSTRef size_ref
    capacity <- readSTRef capacity_ref
    if size < capacity then do
      VM.write vec size (fst char_state)
      modifySTRef' size_ref (+1)
    else do
      vec <- VM.grow vec capacity
      writeSTRef vec_ref vec
      modifySTRef' capacity_ref (*2)
      VM.write vec size (fst char_state)
      modifySTRef' size_ref (+1)
    writeSTRef char_state_ref (readChar $ snd char_state)
  size <- readSTRef size_ref
  pos <- readSTRef pos_ref
  vec <- readSTRef vec_ref
  vec <- V.unsafeFreeze vec
  let txt = T.pack $ V.toList $ V.generate size (\i -> vec V.! i)
  return (txt, (pos,text))

Looking at the above, the persistent_vector stops looking like a bad idea. And for some reason, like with the previous 2 attempts, the performance is horrible. It takes 1.2s to parse 10 numbers, 2.4s for 20 numbers and 3.6s for 30 numbers when loaded from the 10M integers file.
Profiling shows that it spends 97% of the time inside the readChar function.
type State = (Int, Text)

readChar :: State -> (Char, State)
readChar (pos, text) =
  if pos >= T.length text then
    error $ "Error at pos " ++ show pos
  else
    (text `T.index` pos,(pos+1,text))

Looking at the above nothing really jumps out at me at being wrong.
At this point, if I was doing the parser for myself, I'd just switch to using the immutable persistent_vector, use a different language for the compiler project, or write it in C++ with the Boost::Spirit parser combinator library and use it via a FFI.
I cannot believe that I am writing this, but it seems I really can't write a fast parser in Haskell. If this was in F#, I'd be done with it in an hour, but I've already worked on it for well over 20 in Haskell. I think it is best to leave this to the experts after all.
Originally, I intended to dedicate some of my time to Futhark, but apart from the stuff I did to get it to work on Windows, it does not seem I can contribute much apart from testing it when the Cuda and F# backends for it come out.
A few years down the road when I am done with my studies of reinforcement learning, I'll dedicate a large chunk of time in making my own language and I'll make Futhark as the base. I'll rebuild it in F# (or maybe F* which has been getting general programming features lately) and add automatic differentiation capabilities to it. For an autodiff language I think Futhark would be an ideal target since apart from bit twiddling operations, everything it has is differentiable.
That having said, if somebody wants to take a look at fixing some of my broken examples, I would not mind answering questions and putting in some extra effort to make this work.
Phew. I think I'll take a break.

First, thank you for your efforts.  It is motivating to see that someone
can potentially find my work interesting!  I think I will see whether I
can tune the persistent_vector solution to perform acceptably - my guess
is that there's some hidden laziness somewhere.
While I agree that Haskell is not the nicest imperative language, it's
possible to do things a little nicer.  Here is a reimplementation of
your program using STUArrays:
https://gist.github.com/Athas/d4e77e187328853fb56529d55dec8bd8
It still uses a bizarrely large amount of memory for large inputs, but I
have a hard time seeing where.
--
\  Troels
/\ Henriksen

Wow, it looks quite nice. Actually before today, I was not aware it was possible to get any sort of composability when something is inside a monad. I only figured it out by accident today when I did the cond function for the while loops inside my imperative solution.
Despite the ugliness of them, the main problem with all of my examples is how slow they are. I can't even imagine what must be going on under the hood for it to need 1.2s for 10 numbers.

After asking around on #haskell, I got the hint that the vector package
is where the action's at.  I wrote this, which is more than an an order
of magnitude faster than my previous attempt:
https://gist.github.com/Athas/bb38562801e8417ff0a1c5b4ad614a01
In fact, it's fast enough to be practical, and the code doesn't look all
that bad, really.  Kind of low-level, but that's a fair trade-off for
predictable performance.  (Remember to use -O3 when compiling.)
--
\  Troels
/\ Henriksen

I'll definitely remember this style of coding in the future. I tried compiling with -O3 and the program ran in 3.4s compared to 6.4s before. It is still 6 orders of magnitude too slow, but I'll have to remember fast compilation in the future as well as I was not using it.

My program (the last one that uses vectors) can process 10 million lines in 0.91s, and 100 million lines in 7.94s.  That's fast enough for me.

There is no such thing as "fast enough".  F
On Sun, Aug 7, 2016 at 9:50 PM, Troels Henriksen<EMAIL_ADDRESS>wrote:

My program (the last one that uses vectors) can process 10 million lines
in 0.91s, and 100 million lines in 7.94s. That's fast enough for me.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/HIPERFIT/futhark/issues/152#issuecomment-238104121,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAo8j2Brq-3XSSaoVjmGc8pPj_C6rHaeks5qdjcYgaJpZM4IPqQQ
.

Any speculation on why my 3,4,5 examples were so slow? It is great that I now know how to use control flow in monads using functions - it seems stupid that I could not figure this out earlier - but despite the elegance of your code, functionally the only significant difference that I can see is it using the STVector instead of MVector. Is that it? It seems unlikely...

No, actually, now that I look more closely, the way you use Text is by dropping off processed items, while I am indexing into it. That has to be it, but why exactly...
index :: Text -> Int -> Char Source
O(n) Text index (subscript) operator, starting from 0.

Oh, it is O(n). Well that explains it. This is unbelievable.

I think Text uses a variable-width encoding (maybe UTF-16) internally,
so it has to be O(n).
--
\  Troels
/\ Henriksen

Text's head and last are O(1) while appending to start and the end is O(n). Strangely enough length is also O(n). But if you think about it, in the above examples I was indexing into the first 30 elements out of 10M. So why did each of those indexings have to be an O(10000000) operation?
It doesn't matter. Actually, thanks to this I do feel I will never forget this lesson on monads that you gave me. I spent most of the day again learning about this and even considered giving writing a parser another try before realizing that I can't think of a particularly elegant ways to combine ST and State monads. I'll be interested in taking a look at how you (or @nqpz) will organize the code for the full parser.
At any rate, I think I finally understand why monads are so difficult - it is because they are continuations. This is something I studied for quite a bit, but I've never been able to generalize that knowledge into understanding regardless how much I thought about it.

This is what I have right now: https://gist.github.com/Athas/bd0882077455247d3bb0fe5979b0555f
I'm not trying to be clever.  Maybe someone can figure out a combinator library that can be used here, but I'm not really interested in writing parsers in the most elegant style - I just need something soon so I can improve our test coverage.  The missing piece is properly handling the primitive values.  I'll see if I can re-use the Futhark lexer itself.  I don't fancy implementing matching for things like decimal literals by hand.

I figured it out. It all came to me last night.
Unfoldr can be generalized from (b -> Maybe (a, b)) -> b -> [a] to (b -> Either b (a, b)) -> b -> ([a],b). And if you take note, you will notice that b -> (a,b) is essentially the state part of the state monad. Once you have the above function you can make any of the parser combinators like many,sepBy and manyTill using the above function and write everything in a monadic style. Maybe such a function could be converted to continuation passing style so it works with Attoparsec, but I am not sure how yet.
I'll write up a demonstration.

I am nearly done with the function, but I do not know which language extension I need to turn on to make the following work. Any ideas?
unfoldrVec' :: (b -> Either b (a, b)) -> b -> (Vec.Vector a,b)
unfoldrVec' cond init_state = runST $ do
    empty <- MVec.new 1024
    (arr, state) <- runUnfoldr (0, empty) init_state
    arr <- freeze arr
    return (arr, state)
    where
      growIfFilled :: Int -> MVec.STVector s a -> ST s (MVec.STVector s a)
      growIfFilled i arr =
        if i >= capacity
        then MVec.grow arr capacity
        else return arr
        where capacity = MVec.length arr
      runUnfoldr :: (Int, MVec.STVector s a) -> b -> ST s (MVec.STVector s a, b)
      runUnfoldr (i, arr) state =
        case cond state of
          Right (x, state) -> do
            arr <- growIfFilled i arr
            MVec.write arr i x
            runUnfoldr (i+1, arr) state
          Left state ->
            return (MVec.slice 0 i arr, state)

The above gives me the following error:
Couldn't match expected type `b' with actual type `b1'
  `b1' is a rigid type variable bound by
       the type signature for
         runUnfoldr :: (Int, MVec.STVector s a1)
                       -> b1 -> ST s (MVec.STVector s a1, b1)
       at unfoldr_v6.hs:65:21
  `b' is a rigid type variable bound by
      the type signature for
        unfoldrVec' :: (b -> Either b (a, b)) -> b -> (Vec.Vector a, b)
      at unfoldr_v6.hs:52:16
Relevant bindings include
  state :: b1 (bound at unfoldr_v6.hs:66:27)
  runUnfoldr :: (Int, MVec.STVector s a1)
                -> b1 -> ST s (MVec.STVector s a1, b1)
    (bound at unfoldr_v6.hs:66:7)
  init_state :: b (bound at unfoldr_v6.hs:53:18)
  cond :: b -> Either b (a, b) (bound at unfoldr_v6.hs:53:13)
  unfoldrVec' :: (b -> Either b (a, b)) -> b -> (Vec.Vector a, b)
    (bound at unfoldr_v6.hs:53:1)
In the first argument of `cond', namely `state'
In the expression: cond state
E:\kdd_cup (erase this when done)\unfoldr_v6.hs: 70, 30
    Couldn't match expected type `a1' with actual type `a'
      `a' is a rigid type variable bound by
          the type signature for
            unfoldrVec' :: (b -> Either b (a, b)) -> b -> (Vec.Vector a, b)
          at unfoldr_v6.hs:52:16
      `a1' is a rigid type variable bound by
           the type signature for
             runUnfoldr :: (Int, MVec.STVector s a1)
                           -> b1 -> ST s (MVec.STVector s a1, b1)
           at unfoldr_v6.hs:65:21
    Relevant bindings include
      arr :: MVec.STVector s a1 (bound at unfoldr_v6.hs:69:13)
      x :: a (bound at unfoldr_v6.hs:68:18)
      arr :: MVec.STVector s a1 (bound at unfoldr_v6.hs:66:22)
      runUnfoldr :: (Int, MVec.STVector s a1)
                    -> b1 -> ST s (MVec.STVector s a1, b1)
        (bound at unfoldr_v6.hs:66:7)
      cond :: b -> Either b (a, b) (bound at unfoldr_v6.hs:53:13)
      unfoldrVec' :: (b -> Either b (a, b)) -> b -> (Vec.Vector a, b)
        (bound at unfoldr_v6.hs:53:1)
    In the third argument of `MVec.write', namely `x'
    In a stmt of a 'do' block: MVec.write arr i x

Nevermind, I just had to do it like this: unfoldrVec' :: forall a b. (b -> Either b (a, b)) -> b -> (Vec.Vector a,b)

You need ScopedTypeVariables (otherwise the inner function type
signatures each have a fresh 's').  I have no idea why this is not the
default.
--
\  Troels
/\ Henriksen

Done with the prototype for the value parser. It looks quite nice now. It finishes summing 10M integers in 2s on my machine. I consider this performance quite good.
Now that the main difficulty of figuring out the right design pattern is out of the way, the rest won't be hard. I threw in the towel on this two days ago, but let me take it back. I'll get this done.

Sure.  I've managed to hack up this thing:
https://gist.github.com/Athas/5c2dd0fdb669eecce892adaa99958bd9
It is a module that defines an (opaque) value representation type, along
with some comparison functions and (of course) the parser.  It performs
acceptably, but I'm not particularly pleased with how the parser is
written - particularly the fact that it calls out to the default
Alex-generated lexer for parsing constants.
--
\  Troels
/\ Henriksen

I must admit you really work quickly. I'll see whether I can touch up what you have written.
By the way, does Futhark have special syntax to differentiate 8 vs 16 vs 32 vs 64-bit values like 1.2 and 1.2f for floats for example? I see that you have separate parsers, but I can't quite see how you are telling them apart.

Yes, we use a design taken from Rust: every constant can be suffixed
with its desired type.  42i8 is an 8-bit integer, and 4.2f32 is a 32-bit
float.  If no suffix is provided, a plain integer is interpreted as
32-bit, and a plain decimal constant as a double-precision number (like
in C).
--
\  Troels
/\ Henriksen

I pushed something that works for our immediate needs, but I'm not really happy with the way the parser itself is written.  Room for improvement here.  More generally, now I'm curious how one could design a parser combinator library tailored for producing very large values.

Three days ago while I was working a the first version of the library it hit me how use the continuation passing style (as well as monads and higher order functions), so I dropped what I was doing and went back to my old plan of trying to make a Vector parser for Attoparsec. This time, I was  successful.
That having said, the performance is worse than the Either (a,b) unfoldr parser and it chokes on 100M integers, but so does the Either (a,b) unfoldr parser. It might be satisfactory however if the inputs are not extremely large.
I am not really happy with this to be honest. I cannot seem to find a sweet spot with Haskell in terms of control and all my successes as they are in this language feel partial due to that. This is the price of laziness it seems. Even after I figured out CPS, that simple function took an absurd amount of time to write due to misleading error messages and I had to resort to unsafe IO as I could not figure out how to do it inside the ST monad.
As disappointing as that is, unlike the time I did those HackerRank exercises, I feel that now I have a much fuller understanding of Haskell and the things I learned I will actually take advantage of when I start programming in F# again. The state monad in particular is the answer of how to get rid of the global state in my deep learning library and I never realized before that closures can be stacked naturally - I just used a stack of closures for the backwards pass before. Also, I never realized how beneficial functions having an uniform inputs and return values could be until I tried making that first parser and run into the problem of getting state out of unfold.
I tried adding bang patterns into the CPS parser, but that did not help much so if you have any ideas on how to make it run better be my guest. I'll take a break from this toil for a bit and then write a value parser with the CPS Vector Attoparsec parser after that since it seems like a more maintainable solution. I am not happy with it, but hopefully a straightline parse of 100M integers is just an extreme edge case for it. Maybe if the inputs have a treelike structure it does better?

No, actually forget that. I cannot bear doing such a shoddy job of this.
I have to ask at this point, just how open are you to making that parser in different languages such as F#? I mean you are already using C. If I did it in F#, it would take me half the time to do the entire thing that it did me to make the CPS parser in Haskell and there would be no inexplicable space leaks that so far I've run into everywhere, including Haskell's compiler services. The performance would also be on par with C and the thing would be easy to read and maintain.

First of all, it's good that these struggles are giving you new
insights!  That's the reason any of us work on this stuff, really.
Working with these huge homogenous structures (arrays) is definitely an
edge case for both parsing and functional programming in general.
Haskell is very well suited for complicated but relatively small
structures (like, say, syntax trees or streams), but falls down when you
just want simple data flow at very high throughput.  It's the same when
you try to do high performance array computing.  Futhark is partially an
attempt to show that these limitations are not inherent to functional
programming as such, but merely to extant languages (or their
implementations).
I'm not really open to adding more languages to Futhark, except as
backends or ancillary tooling (we use some Python in the latter
capacity, everything else is Haskell).  Controlling the complexity on a
program like this is already a tremendous struggle, and adding more
languages would open a rat's nest of new issues.
--
\  Troels
/\ Henriksen

Well, I guess that I'll stop my Haskell adventure here then. If I could have figured out a way to parse 100M numbers without blowing the heap I would have done this, believe me. Just now I did some more research on Haskell performance, but it boils down to things I already know and do in F# regularly - do not use lists, do not clutter the heap with closures, use value types, avoid laziness.
That last one is really killer. One thing I've realized in the past two weeks is that types are really poor at describing the control flow of a function, and programming is mostly about that. Adding another layer of abstraction makes it even harder. It is a huge mistake that laziness and and strictness are not a part of Haskell's type system.
Long ago I read online that reasoning about Haskell performance is really difficult, but it is not until you spend a week and a half trying to parse 100M integers and fail until you realize what that actually means.
The error messages are really poor as well - they are close to useless. In review, I really should have realized the reality of that and stopped programming the same as in F# where I would make a large chunk of code and rely on the type checker to guide me back to the correct path. There is a very real benefit to F# being a less abstract language than Haskell.
I pains me to say this, but thanks to this experience I will never use Haskell for any serious real world work. My image of it being an academic experiment is now pretty solidified.
I read online that contributors to open source projects create more work than they save and that is obviously correct here. The net effect as far as I can tell is that I just ended up writing a bunch of rants while making no real contribution, so I'll cut that short here. I'll get back to studying Fstar.
Thank you for your patience.

After giving up on doing this two times, going crazy more than that, spitting on the language and telling myself I would never use it again...seeing that a guy managing to get it to run on Reddit just by reading from standard IO, I decided to just try ByteString to test if Text was the reason why it is so inefficient and it turns out that it is. All this pain and suffering could have been avoided by ditching Text at the start. It should have occurred to me to do this as well given that I got a hint already when I found out it had O(n) indexing.
There is something in Haskell that brings me on tilt much more easily than in other languages. It seems I am going to have to be very careful of the data structures used in the underlying libraries if I ever use it again...which I won't.
I am tsundere for it at this point.
I've also tested switching the Attoparsec parser to ByteString.Char8, but that did not help it. Here laziness is in fact the culprit.
But for performance now that I know where the hole is, that leaves the door open to integrating it with some library that does not operate using continuation passing.
I'll do this at some later date just to bring this thing to a close...assuming you don't tell me something like that you are using Text everywhere and ByteStrings are out of the question. In that case it really is hopeless.
By the way, with Unboxed Vectors the thing goes through 100M integers quite quickly now, on par with the F# imperative solution. Here it is.

Actually I have considered switching from Text to Bytestring for the
value parser, to avoid paing the UTF-8 decoding premium.  The value
syntax itself does not contain any non-ASCII characters, so it wouldn't
be much of a problem.  The only issue would be comments containing crazy
multibyte characters that would contain ASCII linebreaks, but I vaguely
recall something about UTF-8 being designed so that it doesn't happen.
Anyway, the value parser is fast enough for my purposes for the moment
(that is, I can run our benchmark suite).  I would mostly be interested
in making the code less ugly.
--
\  Troels
/\ Henriksen

By my measure, the premium is quite a bit more than that. For illustration, when I boot into windows, I have only 1.5Gb occupied at the start. If I run the Text based parser on the 100M example then it goes all the way up to 8Gb (my total memory) and then aborts. Starting from 6.5Gb free If we substract 800Mb needed for the vector array, that would leave 5.7Gb for the rest of the program. Meaning the Text data structure uses it up in its entirety in loading the 430Mb file. Meaning Text uses more than 13 bytes per character. How absurd.
It should have occurred to me to suspect it when I realized it had O(n) indexing, but considering Text is supposed to be a high performance replacement for standard Strings I have to say this blindsided me as it undershot one of my fundamental assumptions about code behavior. I should not have blindly trusted it just because it is a part of the standard package. I am guessing you did not tell me to stop using Text because you had no idea yourself.
Well, if the value parser works for you then we might as well keep it at that. I am just glad to bring this mystery to a close.

I did some more research. I figured out how to add strictness annotations using seq (the compiler refused bang patterns for some reason) and added them to the Text unfoldr example. Now it does take like 6Gb and 38 seconds (compared to ~1.2Gb and 3s for Bytestring example,) which is a success even if a partial one.
Thinking about it at more depth, it occurred to me that there is no way Text could possibly take 13 bytes per character, so I've tried this. I've also tried reading just the last character and found that it uses the expected amount of memory to load the 430Mb file. Under the hood it probably it uses some tree like data structure, but the optimizations for it are not doing a good job of cleaning up after itself.
That is about it. Using Bytestring for any text related work is still the way to go.

I agree.  At some point the parser should be changed to use ByteString.  Text is only worthwhile if you really want to do text processing.

Hello. I promise this will be the last post in this long closed repo.
At first I decided to give making the value parser another shot in Haskell, but I could not gather the motivation and after considering doing it in Racket for a bit, I decided to just do it in F#. I understand this won't be helpful as a contribution to the Futhark project, but when I started doing it I got a few visions of the future.
I am not sure whether my Haskell experiences were responsible for triggering it, but for the past month or so, I was obsessed about polymorphism. I would go to bed and I would dream of interfaces. The thing I gained from that is that I now see how I could replace the backend of my ML library with an AST generator for Futhark. It would not be that difficult assuming I had such a generator.
And making such a generator would be much easier if I had something like a full F# Futhark parser, so it would make sense to start out that way. If I had a parser I would already have about half the job done and it would definitely make testing the generator easier. So with regard to Futhark, my plan is something like this:

Make a full parser.
Make a generator.
Make a backend for that generator.

I won't start it right away, in a few months more likely, but this value parser is a promising start. I thought it would take me only a few hours to make it, but adding real-time dimensionality checking to it made it quite a bit harder. Still it did only take me only today and yesterday so it was not too bad. I have not tested it on 100M arrays yet, but it uses resizable arrays internally, so I do not expect any space leaks or anything like that unlike in Haskell.
On that note, I'd like to apologize about my behavior in this issue page. I mean it. Before trying to make the parser in Haskell, I studied the language for over a month to make sure I would be ready just for this, but no matter how much I tried, I simply cannot get myself to like the language. I've tried quite a bit.
Quite frankly, programming in it puts me on tilt in no time at all, which is not something that happens to me in any other language and do not want to touch it again. This does not affect my willingness to study various functional programming methods originating from Haskell, or change my view towards Futhark, but I'll be happy if I never have to see another list for as long as I live. From now, I'll be content to look onto the language from afar.
With that, I am quite glad to see that you are already planning on making that F# backend. It seems that a few months ago you did not have shared memory optimization and you are already working on this. That is quite good progress! With shared memory and an F# backend, I'll really have no excuse not to make a Futhark backend for Spiral so keep it up.
I'll see you in a few months, hopefully.

Look, I don't really mind your opinion on Haskell; there's no reason to apologise.  I happen to find it convenient for my own use, but I don't feel any kind of responsibility for making sure it's liked by people on the Internet.  And if I really liked Haskell that much, I probably wouldn't be trying to make my own language.
Why do you need a Futhark parser to generate Futhark ASTs?  The Futhark syntax recently changed significantly to become more F#-like, but I don't think it will change again for the foreseeable future.
If you want to speed up development of an F# backend, you could post in that issue.  I'm still in the early brainstorming phase, trying to figure out whether to do a C# backend instead (since interop is easy, right?), etc.

Why do you need a Futhark parser to generate Futhark ASTs?

It is definitely more work, but the individual steps would be smaller. It would be easier for me to proceed forward when I have a decent grasp of the preceding steps. Having a parser would allow me to make an isomorphism between the generator and the parser, which would make testing easier. Or at least those thoughts were going through my head when I had the idea of doing them both. My sense is telling me that it might be worth approaching it from that angle.
It would be different if I was you and knew everything there is to know about Futhark. In that case I'd just skip to the AST generator stage. We'll see...
In terms of interop, it is true that C#, F# and all the .NET languages have very good interop, but it is not at the scale of Scala and Java. By that I mean, that you can literally have Java and Scala files inside the same project and they would compile without a hitch.
With .NET, to use C# code in F# you would have to first compile the project into an assembly (.dll of .NET IL code) and then call the methods from that assembly. It works flawlessly and the Cuda library wrappers I am using in my ML library are written in C#, but there is an extra build step. Unless you have something pushing you towards making a C# backend, I suggest you make an F# one. It would make it easy for me to integrate Futhark produced files in my own library inside the same project.
F# also has like 3 open source OpenCL compilers written as Phd theses by various graduate students so that might be worth something there.
I took a look at the F# backend issue. Now, this is probably unrelated to questions 1 and 2, but since you are talking about memory allocation just a few hours ago I found this via the /r/programming sub.
I am definitely in this for the long haul. In fact your progress on Futhark is actually much faster than I have expected, so I am still scrambling to set up the environment so as to actually make use of Futhark when it all gets here.


Recipe to use freshly released streaming models (Augmented-memory and Emformer) on ASR ?

fairseq Version : master
PyTorch Version 1.7
OS (e.g., Linux): Linux
How you installed fairseq : git clone
Python version: 3.8.5
CUDA/cuDNN version: 10.2
GPU models and configuration: Tesla V100 on computing server

Hi,
I'd like to know if it is possible to use freshly released streaming encoders (Augmented memory transformer and emformer) for streaming ASR purposes (like training on LibriSpeech for example) ? For now, I see it in simultaneous translation folder.
I tried to follow the usual ASR LibriSpeech recipe (https://github.com/pytorch/fairseq/blob/master/examples/speech_to_text/docs/librispeech_example.md) mixed with SimulST recipe (https://github.com/pytorch/fairseq/blob/master/examples/speech_to_text/docs/simulst_mustc_example.md) by first pretraining an ASR model :
fairseq-train ${LS_ROOT} --save-dir ${SAVE_DIR} --config-yaml config.yaml --train-subset train --valid-subset dev --num-workers 32 --max-tokens 40000 --max-update 100000 --task speech_to_text --criterion label_smoothed_cross_entropy --report-accuracy --arch convtransformer_espnet --share-decoder-input-output-embed --optimizer adam --lr 2e-3 --lr-scheduler inverse_sqrt --warmup-updates 10000 --clip-norm 10.0 --seed 1 --update-freq 8 --fp16,
and then running :
fairseq-train ${LS_ROOT} --save-dir ${SAVE_DIR} --config-yaml config.yaml --train-subset train --valid-subset dev --num-workers 32 --max-tokens 40000 --max-update 300000 --task speech_to_text --criterion label_smoothed_cross_entropy --report-accuracy --arch convtransformer_augmented_memory --share-decoder-input-output-embed --optimizer adam --lr 2e-3 --lr-scheduler inverse_sqrt --warmup-updates 10000 --clip-norm 10.0 --seed 1 --update-freq 8 --simul-type infinite_lookback_fixed_pre_decision --fixed-pre-decision-ratio 7 --segment-size 40 --fp16
for my final model training, but I got an exception :
Exception: Cannot load model parameters from checkpoint /path/to/checkpoint_last.pt; please ensure that the architectures match
Hence, i don't know if this is the proper way to do it.
Also, I got the same error when trying to train another SimulST architecture, such as --arch convtransformer_simul_trans_espnet.
But still, apart from the mismatch between models, is this workaround OK to train a streaming model on ASR task ?
Thanks in advance for your answer.

Any update on this?

https://github.com/pytorch/audio/blob/48cfbf2ba8ca4521e181d7c6b7b424829b6dcba4/test/torchaudio_unittest/prototype/emformer_test_impl.py

following.
I also want to use "convtransformer_augmented_memory" arch to do Speech Translation Task, but now have no idea.

Hi, there's actually a working implementation of emformer from torchaudio as suggested by @SatenHarutyunyan (thanks a ton, btw)

https://github.com/pytorch/audio/blob/48cfbf2ba8ca4521e181d7c6b7b424829b6dcba4/test/torchaudio_unittest/prototype/emformer_test_impl.py

Actually, they have moved to the main code of torchaudio's, which I found here:
code:
https://github.com/pytorch/audio/blob/main/torchaudio/models/emformer.py
docs:
https://pytorch.org/audio/main/models.html#emformer
I've tested it and it worked like a charm.


Add predictTags and printDocStr for tags prediction

The method predictTags will actually return an unordered_map of k predicted tags given an input text. In a separate PR (coming in a few minutes), I'll add the Python binding and a test case for this new method.

For what it's worth, I've used your code with succes 👍


[WIP] What if all the collections were stable

In hopes of helping with #647

This did not help; further investigation points at fontatins packing, see https://github.com/googlefonts/fontc/issues/647#issuecomment-1858251211.


The profiler has warnings it is throwing CI 3

Message: Array and string offset access syntax with curly braces is deprecated
Just need to replace {} with []
I just made the fixed and placed it in my Libraries folder, but thought I would mention.
Thanks

Marking as duplicate of #5879.


Single Node Setup

Everything seemed to work until this command
helm install -n sogno-demo --create-namespace -f rabbitmq_values.yaml rabbitmq bitnami/rabbitmq
Error: failed to download "bitnami/rabbitmq"

Thanks for raising this issue! It seems we are missing a  helm repo update between the helm repo add and the helm install steps.
Could you please verify that this solves the issue?

Now I get the message after the helm install command
Error: open rabbitmq_values.yaml: no such file or directory
I have searched for this file and I do not see it. Was this file loaded as a part of the "helm repo ..." command?

That file is passed to helm as an argument (-f rabbitmq_values.yaml). We expect that file in the current directory. This file contains overrides for default helm values. An example is provided in the docs, just below the command.
We will consider to provide an updated example and put the file in our example-deployment repo.

The instructions are hopefully clearer on this now: https://github.com/sogno-platform/sogno-platform.github.io/blob/5726dd4a4bd4294df1940dfeafb5304034999c58/content/en/docs/Getting started/single-node.md?plain=1#L70-L88


AttributeError: type object 'VaultOverview' has no attribute 'model_validate'

Scenario & Reproduction Steps
Seems any time I try to list vaults or list items using the docs' suggestions, I'm thrown an attribute error related to model_validate for both items and vaults.
pip install<EMAIL_ADDRESS>import asyncio
from onepassword.client import Client

async def get_items():
    client = await Client.authenticate(auth=token, integration_name="secret_loader", integration_version="v1.0.0")
    vaults = await client.vaults.list_all()
    async for vault in vaults:
        print(vault.title)
        items = await client.items.list_all(vault.id)
        async for item in items:
            print(item.title)

asyncio.run(get_items())

Actual Behavior
AttributeError: type object 'VaultOverview' has no attribute 'model_validate'
Expected Behavior
I'm expecting to retrieve information related to vaults and items in them that are associated with my account
SDK version
No response
Additional information
No response

Hi @alavi-sorrek , thank you for reaching out. Can you please give me a few more details related to:

the platform where you are running the SDK
the Python version that you use
if you are running in any virtual environment, or if you use a pyproject.toml manifest to declare your project's dependencies

Looking forward to hearing back from you!

I was definitely running on Python 3.9, and either on Ubuntu 22 or it was
throwing the error on my Lambda - I can’t remember which here.
I ended up using the 1password PyPi, which also had its own issues, but I
was able to work around that.
On Thu, Sep 12, 2024 at 3:17 AM Horia Culea @.***>
wrote:

Hi @alavi-sorrek https://github.com/alavi-sorrek , thank you for
reaching out. Can you please give me a few more details related to:

the platform where you are running the SDK (OS + arch)
the Python version that you use
if you are running in any virtual environment, or if you use a
pyproject.toml manifest to declare your project's dependencies

Looking forward to hearing back from you!
—
Reply to this email directly, view it on GitHub
https://github.com/1Password/onepassword-sdk-python/issues/109#issuecomment-2345453582,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A2QBB54ZYMS3ZFPZ5Q3A2ADZWE5W3AVCNFSM6AAAAABN7LTIAWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNBVGQ2TGNJYGI
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hi @MOmarMiraj
We tried to list the vault and facing the same error. We followed the installation instructions provided in the README.
 File "/.venv/lib/python3.10/site-packages/onepassword/vaults.py", line 28, in list_all     objects = [VaultOverview.model_validate(data) for data in response_data]   File "/.venv/lib/python3.10/site-packages/onepassword/vaults.py", line 28, in <listcomp>     objects = [VaultOverview.model_validate(data) for data in response_data] AttributeError: type object 'VaultOverview' has no attribute 'model_validate'
Python Version - 3.10.7
pydantic = "^1.10.7"

Hi @kumarnmanoj
Thanks for looking into this deeper and sorry your having trouble with this still.
It seems that the issue is your pydantic version. The minimum version of pydantic that the SDK requires is v2.5. Is it possible you update your pydantic version to a version thats greater than v2.5 and try again?

Hi @MOmarMiraj
Thanks for the quick reply. We upgraded the pydantic to the latest version and it worked perfectly.
Thanks again.

Im glad I could fix it, I will close the issue as this has been resolved!


PowerPlatform action fails due to certificate issue

Hi @davidjenni,
We are using powerplatform action referencing directly from the marketplace and we are seeing this certificate issue.
Error*
Download action repository 'actions/checkout@v2' (SHA:ee0669bd1cc54295c223e0bb666b733df41de1c5)
Download action repository 'microsoft/powerplatform-actions@v0' (SHA:0c80aacb61f9bfdcb930b880febc420ec4ef2f3e)
Warning: Failed to download action 'https://api.github.com/repos/microsoft/powerplatform-actions/zipball/0c80aacb61f9bfdcb930b880febc420ec4ef2f3e'. Error: The SSL connection could not be established, see inner exception.
Warning: Back off 13.622 seconds before retry.
Warning: Failed to download action 'https://api.github.com/repos/microsoft/powerplatform-actions/zipball/0c80aacb61f9bfdcb930b880febc420ec4ef2f3e'. Error: The SSL connection could not be established, see inner exception.
Warning: Back off 26.336 seconds before retry.
Error: The SSL connection could not be established, see inner exception.

Most of our jobs are failing due to this and need your attention on it.
Thanks,
Amena Afreen

Looks like github's cert is in place to me:

Are you running through a proxy?  If so, that might be interfering with the connection, and you'd need to talk to your own IT or network team to work through that.
In any case, our team does not have any control nor visibility into GitHub's certificate or hosting process, and you'd have better luck opening a GitHub support ticket.


Feature/playlists

This PR introduces the playlist implementation to the frontend, it ensures a smooth connection to the APIs and a user friendly user interface which now allows user to:

Create, view and edit a Playlist's info such as name, description, and privacy status
Add an episode or a playlist to the playerbar, by implementing a queue that also allows playing next/later in the queue
Add an episode to a playlist or remove it from the playlist
View the Queue and traverse it by clicking on the desired episode.

closes #26

https://github.com/awaazo/awaazo/assets/31280516/cf6bc670-2751-465d-b569-0035680fb7f1
Commit where this was done: commit


[BUG]: Inconsistent Alignment of Login and Footer Containers

🐞 Bug Report
📋 Description
The current alignment of the login and footer containers is inconsistent, resulting in a poor user interface experience. Both the login and footer containers are horizontally aligned, which causes the layout to appear cluttered and unstructured. For a more polished look, these elements should be vertically aligned to enhance the visual flow and overall aesthetics of the page.
⚙ Steps to Reproduce
Steps to reproduce the behavior:

Navigate to the login page.
Observe the alignment of the login container and footer.
Note that both are horizontally aligned, which disrupts the intended UI layout.

✔ Expected Behavior
The login and footer containers should be vertically aligned for a clean and structured appearance, providing a consistent visual experience.
❌ Actual Behavior
Both the login and footer containers are aligned horizontally, causing the interface to appear disorganized and detracting from the overall user experience.
📷 Screenshots

🖥 Environment (if applicable):

OS: Windows 11
Browser: Chrome, Firefox, Brave

🔧 Additional Information
Vertical alignment of the containers would create a more user-friendly layout and improve the visual hierarchy on the page.
Please assign this issue to me under gssoc-ext level2 and hactoberfest-accecpted, @Vin205  , as I am interested in working on it.

@skmirajulislam can you please assign me this issue

hey @skmirajulislam kindly assign me this task i'll be sharing the solution asap!!


Update index.md

Fixed typo.

Thank you so much for your attention to detail. The link was actually [[foo]]d on purpose, because the page is named "foo.md" 😃. But it is confusing, because "foo" is not a word. I'll merge this PR, and then make edits in order to change the page name.

Merged. Thanks again!

Haha I didn't notice it was foo.md 😆


Disk space Exhaustion

Hello,
I am running this helm chart, and have no illusions this is an experience problem and probably missing something. This is quite literally my first K8 cluster / first Helm chart.
When my CI pipeline started, it started trying to do its thing, but very rapidly exhausted its disk space and crashed. So I am assuming I need to do something to allocate more space at it, but I am unsure of exactly what its looking for. I tried something like the following, but it errors.
Any chance you can help steer me in the right direction? If it helps any, the type of workload being shipped to this is VMware template builds with packer. It uses up a bunch of space during the build and after the artifacts are uploaded all the space is freed up.
volumes:
  - name: ci-data
    size: "50Gi"

volumeMounts:
  - containerpath: /azp
    volume:
      name: ci-data
      subpath: /azp
      ```

Error: UPGRADE FAILED: error validating "": error validating data: [ValidationError(Deployment.spec.template.spec.containers[0].volumeMounts[0]): unknown field "containerpath" in io.k8s.api.core.v1.VolumeMount, ValidationError(Deployment.spec.template.spec.containers[0].volumeMounts[0]): unknown field "volume" in io.k8s.api.core.v1.VolumeMount, ValidationError(Deployment.spec.template.spec.containers[0].volumeMounts[0]): missing required field "name" in io.k8s.api.core.v1.VolumeMount, ValidationError(Deployment.spec.template.spec.containers[0].volumeMounts[0]): missing required field "mountPath" in io.k8s.api.core.v1.VolumeMount, ValidationError(Deployment.spec.template.spec.volumes[0]): unknown field "size" in io.k8s.api.core.v1.Volume]

Hi @Kennochas,
There is an error about containerpath field you specified which is not documented and supported!
Could you please try the snippet below instead (with subpath and without subpath):
volumes:
  - name: ci-data
    size: "50Gi"

volumeMounts:
  - containerpath: /azp
    volume:
      name: ci-data
      subpath: /azp
      mountPath: /azp


Refactor of kitchen-vcair plugin, initial public release

Refactor of vulk's work, added tests, added rubocop, support for vCloud OnDemand

It looks good apart from what i asked above. I do suggest you spin up travis also with this, at least to gate unit tests.
Other than that :+1:


API project files and GET endpoint doc

I know you wrote in the instructions that each header should have an introductory sentence. I just feel like the sentences here would just be explaining what each part of the ref is, which is something that any developer looking at this doesn't need explained. Do you still think I should add the opening sentences?

Conceptually, a readme file explains what the API does
the Intro should do this and should cover
the workflow
and the HTTP error codes (for general issues) the HTTP codes for the specific get/ post should be included in those pages.

looks like you got a bad conflict
please fix. If you're not sure how, let me know.

Thanks for the comments.

Conceptually, a readme file explains what the API does the Intro should do this and should cover the workflow and the HTTP error codes (for general issues) the HTTP codes for the specific get/ post should be included in those pages.

Ok. I'll rewrite the intro and redistribute the HTTP codes.

looks like you got a bad conflict please fix. If you're not sure how, let me know.

It happened because I renamed the folder (my last name has two Ns). I actually wrote the whole thing on the web UI, but I'll try to clone it on my computer and resolve.

I solved the conflict issues and rewrote the readme.


Interaction patches

Interaction patches for discord.py 2.0.

I'm currently trying to resolve this error.
beni@beni Security % python3 ./index.py
/Users/beni/Storage/001_Github/Security/./index.py:283: RuntimeWarning: coroutine 'BotBase.load_extension' was never awaited
  bot.load_extension(f"cogs.{name}")
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
[22-04-2022 | 15:59:36] INFO: "log" Cog Loaded.
[22-04-2022 | 15:59:36] INFO: "events" Cog Loaded.
[22-04-2022 | 15:59:36] INFO: "core" Cog Loaded.

Ok, I fixed the issues that occur on start-up, now the issue is that discord cached the old commands. I can't test it now because thanks discord, but we should fix the duplicate command glitch if it's still present in the bot. :( It wasn't a issue before, it's a issue now, but I made the bot clear all global commands to ensure the global commands don't interfere with the guild only commands.

BenitzCoding


Xamarin.Mac 2.0 compatibilty

Is Akavache Xamarin.Mac 2.0 compatible? Getting weird errors when trying to build a Xamarin.Mac project if it references Akavache, both Unified API and .NET 4.5 target configurations.

To answer my own question it looks like it's working just fine. If I see any weird behavior I will reopen this.


removed workbooks from student Learn nav menu

https://github.com/isaaccomputerscience/isaac-cs-issues/issues/358

Is there a particular reason the workbook link is being reordered in the list for users that can see it?

Essentially just due to the way the test is structured. I could have adjusted the test to accommodate the previous order but it was easier to just add it at the start of the list.


Use runtime dispatching for argument exploding

Currently Boo does only allow the following use of the explode arguments operator:
def bar(a, *args)
# bar(1, *(2, 3))

The following changes allow to have a more powerful exploding operator which works like the one in Python. The drawback is that it uses the runtime to actually dispatch the call, with a bit more work the original use case supported by Boo could bypass the runtime dependency but it's not worth the effort IMHO.
def foo(a, b, c)
# foo(1, *(2, 3))
def bar(a, *args)
# bar(1, *(2, 3))
# bar(*(1, 2, 3))
# bar(1, 2, *(3,))
# bar(1, 2, 3)

NOTE: No unit tests yet, I'll provide them if you think this could actually be merged. :octocat:

Thanks!

@drslump, would you care to investigate the build failure caused by this PR and resubmit it?
https://travis-ci.org/boo-lang/boo/builds/97956507

@bamboo sure, I´m testing on Mono 4.2 on windows and the test passes, downloading 3.12 to verify what went wrong on CI... I'll also take the change to add some unit tests for the new behaviour

Great!


Bug fix: fix NOAUTH error...

Fix NOAUTH error when connecting to a cluster with password protection.
Before loading partition views, call auth() method if a none-empty password is provided after connected to any server of a cluster.
This pull request Include the same change in request in #171 ,but it changes the 3.3 branch.(In fact, I use this branch as my working project's dependency)

I'm planning to release 3.4 end of January and no intermediate release of 3.3 so I'd like to omit this PR but I would include #171 into 3.4 (and I would add this patch also to 4.1).

OK, I'll close this PR and wait your 3.4 release.


ScrollIntoView json unmarshal error fix

Right now function ScrollIntoView tries to unmarshall window.scrollX and window.scrollY
https://github.com/chromedp/chromedp/blob/72bf375971cb5c2ac69aa48b8e1d4e7cef25982b/js.go#L43
to
https://github.com/chromedp/chromedp/blob/446edaac636801f6752ff834fa5ef672d4e0d607/query.go#L1218
Which occasionally results in errors like this:
json: cannot unmarshal number 17592.619140625 into Go value of type int
But according to MDN:

In practice, the returned value is a double-precision floating-point value

Thanks both! it seems we need to revisit the test for ScrolIIntoView, the html test data doesn't really put the element outside the view port to be able to test the behaviour correctly.


caching docker images WIP

Should fix #56

the empty file was just to start the CI suite again

I tend to tweak the README.md file to start a ci run.

What branches do we want to cache from?
Usually, non-PRs to master are cached, but I would recommend caching all of them for now and see what is useless. We can remove useless cache from travis-CI web UI later.
The speed of docker saving and docker loading worries me. Other projects seem to take less time on that
Shall I squash and merge?

Please squash & merge


[Bug] Cannot use rust 1.75.0

Component
alloy-transport
What version of Alloy are you on?
Latest
Operating System
Linux
Describe the bug
See build error:
    Blocking waiting for file lock on package cache
    Updating crates.io index
    Updating git repository `https://github.com/alloy-rs/alloy`
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
error: package `alloy-transport v0.1.0 (https://github.com/alloy-rs/alloy#f44973a0)` cannot be built because it requires rustc 1.76 or newer, while the currently active rustc version is 1.75.0
Either upgrade to rustc 1.76 or newer, or use
cargo update<EMAIL_ADDRESS>--precise ver
where `ver` is the latest version of `alloy-transport` supporting rustc 1.75.0
➜  creditcoin3-next git:(poc_external_attestation_network) ✗ cargo update<EMAIL_ADDRESS>--precise ver

The MSRV is 1.76.0: https://github.com/alloy-rs/alloy?tab=readme-ov-file#supported-rust-versions


Create Docker Image

The image is publicly available on Docker Hub

Any reason why you've deleted project assets?

No this PR is just a draft. I will close it for now and create another one once everything is done and it's ready to merge.


Feat Support json_search function in 2.0

Proposed changes
pick #40948
Issue Number: close #xxx

run buildall

run buildall

run buildall

run external


[Example] ggml: add basic example with CI

Hello, I am a code review bot on flows.network.
It could take a few minutes for me to analyze this PR. Relax, grab a cup of coffee and check back later. Thanks!


[core] Monitor build time in each PR

Summary
Similar to #78 that is about bundle size, I believe we should eventually monitor the build time cost of Pigment CSS.
Examples
No response
Motivation
See the conversation in https://gitlab.com/gitlab-org/frontend/rfcs/-/issues/107 on adopting Tailwind CSS. Build time is one of the concerns:

How much slower will the build be as comparing from today? (we already have a really slow build due to how we compile JS code in WebPack, would that add to the matter? evaluating the entire codebase is not free)

Search keywords: -

The cost of hot starts of eg the Vite dev server is another potential concern, for example it comes up here: https://github.com/mui/material-ui/issues/38137#issuecomment-2239783418


CIV-11537 Update Details of Claim folder name

Before creating a pull request make sure that:

[ ] commit messages are meaningful and follow good commit message guidelines
[ ] README and other documentation has been updated / added (if needed)
[ ] tests have been updated / new tests has been added (if needed)

Please remove this line and everything above and fill the following sections:
JIRA link (if applicable)
https://tools.hmcts.net/jira/browse/CIV-11537
Change description
for each claimant there is a sub-folder that sits within the 'Statements of case' CFV folder which is labelled 'Details of claim'. The ask is to update the name of this sub-folder to 'Claim form'
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[ ] No


cant install corrupted package

Hello,
I have the very latest c5. Unzipped and moved query_builder into root /packages. Go into add functionality and i got
Unknown Package v.0.0
Broken package (handle query_builder).
Let me know anything useful to debug and use it!
Thanks

Are you using 5.7 or 5.6? I haven't coded it for 5.7 yet, and may not need to due to them using a new ORM (I haven't seen much documentation on it, and the core isn't using it much yet).
Also, this is more of an "example" of how to use it. In 5.6 you would use it as a "library" in your own package, and call it as applicable.
What's your use situation? What are you trying to do, I'd be happy to help.


Go 1.6.3

Changelog

updated to go 1.6.3
added aws util test

Issues fixed or closed
None
Questions (open the PR then click the check boxes)
Did you update the documentation related to your changes?

[x] Yes
[ ] My changes were not already documented

Did you run make before committing code and opening this PR?

[x] Yes
[ ] I didn't change any code

Did you run porter create-stack and porter sync-stack to verify provisioning
works?

[ ] Yes
[x] N/A

👍

👍


Hitting back from the list assets screen leads to loading duplicate assets

Note that hiding the back button https://github.com/tasitlabs/tasit/issues/133 would possibly be way a simpler way to avoid this bug

Now that the back button is gone, I’ll rename this issue


Missing comments collected due 429 error

Describe the bug
I'm triyng to collect almost 5k comments from a single post (I have the shortcode).
After almost 2 hours (3000, 35000 i read in the CMD) I'm encountering 429 error repeatedly, so I can't to anything else. I can only stop the collection. I tried to wait a bit, but the collection doesn't restart.
Anyway, when I stop the collection (ctrl+c), i lose also the comments collected yet (3000, 3500). The folder is empty.
To Reproduce
instaloader --login XXX --comments --no-pictures --no-videos --no-captions --no-metadata-json --max-connection-attempts 0 -- -shortcode
Expected behavior
I would save the comments collected before the error, at least.
Instaloader version
4.8.4
Additional context
I tried also with different profiles logged in.

Does this problem still occur with version 4.9?
Also, is there any shortcode where it occurs reliably?

Does this problem still occur with version 4.9?
Also, is there any shortcode where it occurs reliably?

I didn't try with the new update. I will let you know.
If you want a test shortcode try this one: CaAGxjMqI45

Does this problem still occur with version 4.9?
Also, is there any shortcode where it occurs reliably?

The problem stlil occurres. When you stop the collection (ctrl+c after 429) it doesn't close the file, losing all the already collected comments.

When you stop the collection (ctrl+c after 429) it doesn't close the file, losing all the already collected comments.

The current state of downloaded comments is saved with ctrl+c or with AbortDownloadException together with a resume file. This works fine for me, I do not understand the bug that you are reporting.


varying of type int, uint, bool and vector types are disallowed even if declared flat

Godot version
4.0.alpha
System information
Windows 10
Issue description
Godot should allow any primitive data type other than sampler2D to be passed as varying.
shaders do not allow interpolating non-floating point values. For this reason, non-float varyings are required to be flat. We could do this automatically, or we could display an error message and ask the user to add flat.
See also: #45918 , but allow for all of vertex, fragment and light.
I am in favor of automatically assuming flat if not a floating point type.
Steps to reproduce

Create a new MeshInstance3D
Create a new StandardMaterial3D and convert to shader material.
Declare varying flat int instance_id;
Set instance_id = INSTANCE_ID; in vertex()
it shows error "Invalid type for varying, only 'float', ..."

Minimal reproduction project
No response

According to @clayjohn, even GLES2 should allow this. So this change can be made in 3.x too, in both GLES3 and GLES2.


centering individual columns

I'm wondering that the best way is to center individual columns. It seems kind of hard now: adding a full width div chilld. It's kind of ugly. I would like to easily add a css class to a Cell.
Something like
{
        Header: 'Delete',
        accessor: 'id',
        className: styles.center,
        Cell: props => (
          <IconButton icon="remove" className={adminStyles.center} onClick={() => this.handleDelete(props.value)}/>
        )
      }

or it could be individual style specs and have that directly included in the styles for the Td and Th:
{
        Header: 'Delete',
        accessor: 'id',
        textAlign: 'center',
        Cell: props => (
          <IconButton icon="remove" className={adminStyles.center} onClick={() => this.handleDelete(props.value)}/>
        )
      }

                  getProps: () => ({
                    style: {
                      textAlign: 'center'
                    }
                  })

How do we control the alignment of both the header and column cell?
getProps: () => ({
  style: {
    textAlign: 'left'
  }
})

Only aligns the cell content not the column header label


Distinguish more between expressions and statements

See #27. Deliberately not finished, maybe this is not exactly what you want: see the still failing tests and the changed parser states. If you like it, please finish it, otherwise feel free to push here or close the PR entirely.

Looking at it and trying to bend the grammar over, it probably doesn't make sense with the semicolons. I'd still like the block being a vector of statements terminated with semicolons and expression at the end (or possibly the expression missing and the value of the block being none). Would you still like to do it (the changes are simple if I understand it right) or should I? Thank you for trying to do it.

Please finish it yourself. It seems to me that this should really reflect what you want. I just thought I would bring it some ideas.
Feel free to do anything with the PR.

I'll probably close it for now, I implemented the easy changes of the blocks in #32.
I'll leave #31 open, but for now I'd left the semicolons as they are, because the grammar gets kinda awkward and the language inconsistent. There is also the javascript way, which tries to insert semicolon if it encounters unexpected token, but I have no idea if lalrpop even supports this, and if it is the right way to go.


'display: none' to hide arrows

Previously, arrows were hidden using visibility: hidden, but this causes it to take up space which is the desired behavior

🚀


[en] fix access-cluster.md link error

/language en
Related PR #29078
It does not point to the correct document address, and it will stay on the current page when clicked.

/assign @tengqm

@shuaijinchao It appears that this fix is already in progress. See PR #31019. I suggest we close this PR.

@shuaijinchao It appears that this fix is already in progress. Your fix matches PR #31019. I suggest we close this PR.

Okay, sorry, I didn’t notice, I will close this PR
/close

@shuaijinchao It appears that this fix is already in progress. Your fix matches PR #31019. I suggest we close this PR.

Okay, sorry, I didn’t notice, I will close this PR
/close

No problem. Better to have two people try to fix it than none!


Added prefixes to zlib functions for dual zlib/zlib-ng linking.

This pull requests adds PREFIX() to certain functions to allow for dual linking both zlib & zlib-ng.

One thing I am not so sure of is, it uses PREFIX() in zutil.h, which means everything will need to also include zbuild.h. Perhaps we should add include zbuild.h to the top of zutil.h? So just nuke zbuild.h altogether and include its contents in zutil.h.

zbuild.h is supposed to be build-time private header... All external and user code is supposed to hard-code those prefixes instead of relying on checking if compatibility mode is defined. No new code is supposed to use compatibility mode, it's just to get old code working when stock zlib is not needed.

Hmm. Not sure what to do because PREFIX() is still needed. What do you think? Leave as is? Another option would be to move zbuild.h into zutil.h but do this
#if !defined(ZNG_USERDEFINED_PREFIX)
# if defined(ZLIB_COMPAT)
#  define PREFIX(x)  x
#  define PREFIX2(x) ZLIB_ ## x
#  define PREFIX3(x) z_ ## x
#  define PREFIX4(x) _ ## x
#  define PREFIX5(x) z ## x
#  define zVersion   zlibVersion
#  define z_size_t   unsigned long
# else
#  define PREFIX(x)  zng_ ## x
#  define PREFIX2(x) ZLIBNG_ ## x
#  define PREFIX3(x) zng_ ## x
#  define PREFIX4(x) zng_ ## x
#  define PREFIX5(x) zng_ ## x
#  define zVersion   zlibng_version
#  define z_size_t   size_t
# endif

So, if anybody wanted to they could define their own PREFIX() macros by setting #define ZNG_USERDEFINED_PREFIX before zutil.h. This is just an idea.

If zutil,h is externally visible, we need to make separate version for it like we have done for other similar headers... so zutil.h and zutil-ng.h ...

I see what you are saying, zutil.h is not externally visible so it shouldn't matter. Ok, no need for any changes then I think. I think that makes since why gzendian.h code is not in zutil.h also.

The idea of being able to disable or override those prefixes has been in my mind too... I just haven't had time to finish implementing the feature properly... Basically it requires having third copy of those externally visible header file pairs, because some data types are incompatible and as such we can't just reuse the header files for compatibility mode.

I could do another PR that would address that issue. What about something similar to this:

Move zbuild.h contents into zutil.h
In zutil.h include zbuild.h if ZNG_BUILD_PARAMS was set:

#if defined(ZNG_BUILD_PARAMS)
# include "zbuild.h"
#else
# if defined(ZLIB_COMPAT)
#  define PREFIX(x)  x
#  define PREFIX2(x) ZLIB_ ## x
#  define PREFIX3(x) z_ ## x
#  define PREFIX4(x) _ ## x
#  define PREFIX5(x) z ## x
#  define zVersion   zlibVersion
#  define z_size_t   unsigned long
# else
#  define PREFIX(x)  zng_ ## x
#  define PREFIX2(x) ZLIBNG_ ## x
#  define PREFIX3(x) zng_ ## x
#  define PREFIX4(x) zng_ ## x
#  define PREFIX5(x) zng_ ## x
#  define zVersion   zlibng_version
#  define z_size_t   size_t
# endif
#endif

Moving things to zutil.h is bad as the macros should be defined in user code before including any zlib-ng headers...

But if doing that, it won't affect any source code internally that includes zutil.h correct?

For it to work correctly, zlib-ng would have to be statically linked to the binary... Otherwise it would cause linker errors. This also means that the build system for zlib-ng needs to support overriding the prefix.

Regarding custom prefixes
I think stock zlib supports compiling with a separate prefix, IIRC.
If we want to support custom prefixes, I guess that can be done relatively easily. But it only makes sense to do so in zlib-ng mode I think, and that means having only 2 different prefix-parameters, since the rest of them are identical.
I just reviewed PREFIX2, and I am thinking that might possibly be superfluous. I might be wrong but my immediate thought is that we should always use ZLIBNG_VERSION for checking shared-lib compatibility. If that holds true, then we can remove PREFIX2 and simplify our own set of prefixes to only a single variant, thus also making it easier to support custom prefixes.
Stock zlib requires you to edit zlib.h manually to add prefixes IIRC, and that is probably fine for us too. But if we had a way to generate it, that would be more elegant of course, and would also allow us to remove the two slightly different copies we have today.

Now, regarding this PR. I think we might be slightly off track with this one.
Most of the functions and other symbols in this PR are internal ones, that we do not export as part of our API. This means that we can freely rename them, and there is no need to use the complicated prefix method.
So, why not just rename them and add a static 'zng_' prefix?

I have created a new pull request that should do exactly that.


Make session launching easier

What Issue Are You Seeing or What Feature Needs To Be Added?
We have a bit.ly/blueredgame that redirects to /games/sessions/   unfortunately we can't pass the session id so they land on a non-page.
MVP Solution: Make a launch page where they enter the session id and then get to /games/sessions/{id}  to get students in their seats.
Are There Any Links That Would Be Helpful?

closed with #121


Update models.go. charset=utf8mb4 for mysql.

Remove ,utf8 from charset=utf8mb4,utf8

Hi... this is already changed.


Update auth, dynamodb, s3, sqs to 2.17.118

Updates

software.amazon.awssdk:auth
software.amazon.awssdk:dynamodb
software.amazon.awssdk:s3
software.amazon.awssdk:sqs

from 2.17.66 to 2.17.118.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "software.amazon.awssdk" } ]

labels: library-update, early-semver-patch, semver-spec-patch, commit-count:1

Superseded by #816.


Audit Jelly's dependencies

Check carefully if all dependencies of every module are really needed – if we are not, for example, including a gRPC server with the core module.

Ok, it turns out that we actually do have some weird dependencies:

core

scalapb-runtime-grpc, which in turn depends on scalapb-runtime (this is fine) and grpc-protobuf (definitely not fine).
grpc-netty, which depends on a bunch of netty things...

The rest seems fine.

Ok, I did something in #141, let's see if the packages still work after that... I will test this in a few applications that use Jelly.

Oof, the plugin size went from 21.6 MB to just 12 MB... nice...
https://github.com/Jelly-RDF/jelly-jvm/releases/tag/dev

Tested (and working):

[x] Jena RIOT CLI
[ ] RiverBench CI worker
[ ] ... mystery application using gRPC ...

Yep, all good. Closing.


Address Field Not Showing Up

Environment
Please provide as many details as you can:

Hosting type

[] Form.io
[x ] Local deployment

Version:

Formio.js version:
Frontend framework:
Browser:
Browser version:

Steps to Reproduce

Address fields (built out of Advanced) is not rendering

Expected behavior
Hoping to see address fields on the form
Observed behavior
Not seeing it.

Can you provide a video of the issue as well as any error messages?

See attachment
addressfield.pdf

Hello @rabalasubramani3,
Have you set the Address component up using Google, Azure Maps, or Open Street Map?


remove unused scripts

improve status_test from tensorflow

What this PR does / why we need it:
Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #
Checklist:

[x] remove unused code
[x] revert SetServicesV2 back to SetServices
[x] add test code for register_controller.go

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.3%) to 37.002%

Totals

Change from base Build<PHONE_NUMBER>:
0.3%

Covered Lines:
2283

Relevant Lines:
6170

💛  - Coveralls


GROOVY-8330: Wrong 'Inconvertible types' error on casting interface

Make cast error a little bit like java.

Merged. Thanks!


WhatsClient bug

i can't start the client for whatsapp because this give for me this following output:
Starting SuperBot for Whatsapp...
Loading WAPI...
Checking Whatsapp Web API...
(node:11348) UnhandledPromiseRejectionWarning: Error: Evaluation failed: TypeError: window.WAPI.isReady is not a function
at puppeteer_evaluation_script:2:33
at ExecutionContext._evaluateInternal (C:\Users\subze\OneDrive\Área de Trabalho\super-bot-master\node_modules\puppeteer\lib\ExecutionContext.js:122:13)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
at async ExecutionContext.evaluate (C:\Users\subze\OneDrive\Área de Trabalho\super-bot-master\node_modules\puppeteer\lib\ExecutionContext.js:48:12)
at async WhatsappClient.start (C:\Users\subze\OneDrive\Área de Trabalho\super-bot-master\clients\whatsapp\src\WhatsappClient.js:109:16)
at async main (C:\Users\subze\OneDrive\Área de Trabalho\super-bot-master\clients\whatsapp\src\app.js:8:5)
-- ASYNC --
at ExecutionContext. (C:\Users\subze\OneDrive\Área de Trabalho\super-bot-master\node_modules\puppeteer\lib\helper.js:111:15)
at DOMWorld.evaluate (C:\Users\subze\OneDrive\Área de Trabalho\super-bot-master\node_modules\puppeteer\lib\DOMWorld.js:112:20)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
-- ASYNC --
at Frame. (C:\Users\subze\OneDrive\Área de Trabalho\super-bot-master\node_modules\puppeteer\lib\helper.js:111:15)
at Page.evaluate (C:\Users\subze\OneDrive\Área de Trabalho\super-bot-master\node_modules\puppeteer\lib\Page.js:833:43)
at Page. (C:\Users\subze\OneDrive\Área de Trabalho\super-bot-master\node_modules\puppeteer\lib\helper.js:112:23)
at WhatsappClient.start (C:\Users\subze\OneDrive\Área de Trabalho\super-bot-master\clients\whatsapp\src\WhatsappClient.js:109:32)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
at async main (C:\Users\subze\OneDrive\Área de Trabalho\super-bot-master\clients\whatsapp\src\app.js:8:5)
(node:11348) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:11348) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
is it possible to correct? i was making AI modules,

You're right, this is happening in a clean install for some reason. I will look into it. Thanks for reporting it.

Fixed it now, get latest from the master branch. Let me know if you have any issues.


Fix tensor printing in trainer.test()

What does this PR do?
Fix printing tensors when showing test results, so it goes from this
DATALOADER:0 TEST RESULTS
{'fake_test_acc': tensor(4.7004e-14, device="cuda:0")}

to this
DATALOADER:0 TEST RESULTS
{'fake_test_acc': 4.7004e-14}

Fixes #5107
Before submitting

[x] Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
[x] Did you read the contributor guideline, Pull Request section?
[x] Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
[x] Did you make sure to update the documentation with your changes?
[ ] Did you write any new necessary tests?
[x] Did you verify new and existing tests pass locally with your changes?
[x] If you made a notable change (that affects users), did you update the CHANGELOG?

PR review
Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

[x] Is this pull request ready for review? (if not, please submit in draft mode)
[ ] Check that all items from Before submitting are resolved
[x] Make sure the title is self-explanatory and the description concisely explains the PR
[ ] Add labels and milestones (and optionally projects) to the PR so it can be classified; Bugfixes should be including in bug-fix release milestones (m.f.X) and features should be included in (m.X.b) releases.

Did you have fun?
Make sure you had fun coding 🙃

Hello @bryant1410! Thanks for updating this PR.

In the file pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py:

Line 259:17: E731 do not assign a lambda expression, use a def
Line 259:58: E225 missing whitespace around operator

well, shall this be rather a feature?

well, shall this be rather a feature?


Periodically conection errors on ElasticBeanstalk

I receive segments in my dashboard, but sometimes I have errors like:
com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to <IP_ADDRESS>:2000 [/<IP_ADDRESS>] failed: Connection refused (Connection refused)

Sometimes I get things like this:
2018-12-05 17:20:17,963 [http-nio-8080-exec-9] ERROR [anonymous] LogErrorContextMissingStrategy - Suppressing AWS X-Ray context missing exception (SegmentNotFoundException): Failed to begin subsegment named 'XXXX': segment cannot be found.

which is very annoying. The daemon itself appears to be running fine, but sometimes it looks like connectivity is not really good.

Hi grobmeier,
Can you tell me a bit about your setup? Are you running this locally, or a part of a production stack? What options do you have enabled on the X-Ray SDK? The SDK uses UDP to transmit segment through the Daemon, HTTP requests to the daemon are for fetching rules/targets for centralized sampling support, so I am wondering if there is an issue in that regard.
For issue #2 - the context missing exception means that sometime when handling your request, the X-Ray SDK attempted to add data to the segment in progress, however, for some reason the context was lost. Can you tell me a bit about your web framework and libraries you're using? Are you using a barebones setup or do you have any custom instrumentation?

I am running the default configuration of ElasticBeanstalk.
I have looked at the exception a bit further and it seems that it can't poll a sampling rule.
com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to <IP_ADDRESS>:2000 [/<IP_ADDRESS>] failed: Connection refused (Connection refused)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1134) ~[aws-java-sdk-core-1.11.398.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1080) ~[aws-java-sdk-core-1.11.398.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745) ~[aws-java-sdk-core-1.11.398.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719) ~[aws-java-sdk-core-1.11.398.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701) ~[aws-java-sdk-core-1.11.398.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669) ~[aws-java-sdk-core-1.11.398.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651) ~[aws-java-sdk-core-1.11.398.jar:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515) ~[aws-java-sdk-core-1.11.398.jar:?]
	at com.amazonaws.services.xray.AWSXRayClient.doInvoke(AWSXRayClient.java:1257) ~[aws-java-sdk-xray-1.11.398.jar:?]
	at com.amazonaws.services.xray.AWSXRayClient.invoke(AWSXRayClient.java:1226) ~[aws-java-sdk-xray-1.11.398.jar:?]
	at com.amazonaws.services.xray.AWSXRayClient.invoke(AWSXRayClient.java:1215) ~[aws-java-sdk-xray-1.11.398.jar:?]
	at com.amazonaws.services.xray.AWSXRayClient.executeGetSamplingRules(AWSXRayClient.java:568) ~[aws-java-sdk-xray-1.11.398.jar:?]
	at com.amazonaws.services.xray.AWSXRayClient.getSamplingRules(AWSXRayClient.java:539) ~[aws-java-sdk-xray-1.11.398.jar:?]
	at com.amazonaws.xray.strategy.sampling.pollers.RulePoller.pollRule(RulePoller.java:65) ~[aws-xray-recorder-sdk-core-2.1.0.jar:?]
	at com.amazonaws.xray.strategy.sampling.pollers.RulePoller.lambda$start$0(RulePoller.java:46) ~[aws-xray-recorder-sdk-core-2.1.0.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_151]

I am using the Spring framework and integrated it like:
   static {
        AWSXRayRecorderBuilder builder = AWSXRayRecorderBuilder.standard().withPlugin(new EC2Plugin()).withPlugin(new ElasticBeanstalkPlugin());
        AWSXRayRecorder recorder = builder.build();
        recorder.setContextMissingStrategy(new LogErrorContextMissingStrategy());
        AWSXRay.setGlobalRecorder(recorder);
   }

However, it seems it is a right problem. I have added "AWSXRayDaemonWriteAccess" to my service role which does have "poll" in it's right. However I am not sure what else it might need so it can get to the rules.
Issue 2 - I think we can kick that for now. It only happened once or twice, so I am ok with that for now.

Actually this is also probably not like it should be right? I do that from the ElasticBeanstalk EC2 directly:

curl -i <IP_ADDRESS>:2000
curl: (7) Failed to connect to <IP_ADDRESS> port 2000: Connection refused

What is the X-Ray daemon version running on your ElasticBeanstalk EC2? Or providing the ElasticBeanstalk environment platform version also works if you have issue getting the daemon version.
If you have daemon 3.x running you should get something like:
curl -I localhost:2000
HTTP/1.1 403 Forbidden
Content-Length: 94
Date: Wed, 05 Dec 2018 20:48:27 GMT
X-Amzn-Requestid: 1d6b4ce6-f8cf-11e8-bc19-c9b1b5ac6fb0
Content-Type: text/plain; charset=utf-8

since the 3.x daemon opens a TCP proxy which signs the request and forward it to X-Ray service.

I have "Tomcat 8 with Java 8 running on 64bit Amazon Linux/2.7.1" and get:
xray --version
AWS X-Ray daemon version: 2.0.0

I have cloned the current environment with "Tomcat 8 with Java 8 running on 64bit Amazon Linux/3.0.6" and this is my output:
xray --version
AWS X-Ray daemon version: 2.0.0

Apparantely I do not have 3.0, even with the latest platform. So I went to both versions and tried to update xray manually, but the result was:
yum update xray
Loaded plugins: priorities, update-motd, upgrade-helper
...
No packages marked for update

The Maven depedencies I use have the version number 2.1.0. Should I downgrade them?

You can downgrade the SDK but it has latest bug fixes to this is not recommended. You can check the guidance of using X-Ray sampling rules with 2.x SDK and 3.x daemon here: https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-java-configuration.html#xray-sdk-java-configuration-sampling. If you decide to not use service sampling rules you can opt-out by configuring LocalizedSamplingStrategy using the code snippet shown in the link for configuring the recorder.
We already forwarded the request to AWS Elastic Beanstalk team so they look into have 3.x daemon install on newer version of platforms. If you decide the service sampling rule feature is useful and want to workaround it you can use ebextensions to install the daemon yourself: https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-java-configuration.html#xray-sdk-java-configuration-sampling.
Please feel free to let me know any further help you needed.

Thanks. I am not installing it again until Beanstalk has the 3.x daemon natively.

@grobmeier i've struggled with this as well until i've upgraded the daemon like this
new file: .ebextensions/xray-daemon.config
option_settings:
    aws:elasticbeanstalk:xray:
        XRayEnabled: true

commands:
    01-stop-tracing:
        command: yum remove -y xray
        ignoreErrors: true
    02-copy-tracing:
        command: curl https://s3.dualstack.us-east-2.amazonaws.com/aws-xray-assets.us-east-2/xray-daemon/aws-xray-daemon-3.x.rpm -o /home/ec2-user/xray.rpm
    03-start-tracing:
        command: yum install -y /home/ec2-user/xray.rpm

files:
    "/opt/elasticbeanstalk/tasks/taillogs.d/xray-daemon.conf" :
        mode: "000644"
        owner: root
        group: root
        content: |
            /var/log/xray/xray.log

Hi, is there any information on Beanstalk support for this? I find the lack of information very confusing and workaround in such places seem like dangerous.

@grobmeier
According to the Elastic Beanstalk documentation, it looks like the latest platform versions now include version 3.0.0 of the X-Ray daemon.

Oh that's awesome to hear @chrisradek
Thanks for the information and the link. I actually checked but was not able to find it on my own.
I believe this should resolve this issue, since it was caused to outdated version. I will try it and reopen if this is not the case.


pcp_combine not building accumulations properly

In the QPF example ruc-vs-s2grib, there are only two 1-hr accumulation files added to a 3 hour accumulation call.
/d1/CODE/MET/MET_releases/met-8.0_beta8/bin/pcp_combine -add -name APCP_03 -add
/d1/mccabe/input_test/sample_obs/ST2ml/20050807/ST2ml2005080703.Grb_G212 1
/d1/mccabe/input_test/sample_obs/ST2ml/20050807/ST2ml2005080702.Grb_G212 1
/d1/mccabe/test2/qpe_grib/bucket/20050807/ST2ml2005080703_A03h -v 2
This call should also include ST2ml2005080701.Grb_G212. There was a bug with how the script was looping through times.

This was resolved and added as a bugfix patch to the master branch. It is also included in the develop branch.


Added cycle number as parameter to process_element

This is a breaking change that allowes to check "cycle time" has passed.
This might seem a little specific, but should have little to no impact on peformance.
And I need it for my project...

Due this is already a breaking change, #6 should follow close up.