board: mimxrt1170_evk

Hi there,
I have been using this board for some other work and wanted to test out Zephyr with it, seeing as it is a supported board. However, I do not have a J-Link debugger to hand so the flash and debugging instructions weren't working. Looking more closely at the instructions, I was able to use pyocd to flash and debug the elf file directly. The MIMXRT1176 processor on this board is now supported by pyocd where perhaps it wasn't before: https://pyocd.io/docs/builtin-targets.html, https://github.com/pyocd/pyOCD/blob/main/pyocd/target/builtin/target_MIMXRT1176xxxxx.py.
It should also be noted that most of the MIMXRT1xxx family now appear to be supported by pyocd yet the docs all include similar language about flash support not being ready yet. I don't have any of the other boards/processors so cannot confirm this.
Seeing as the pyocd runner doesn't require any extra hardware, I think it is easier for getting started and therefore makes a better default. So I have reordered the includes in board.cmake which seems to have done the trick, although I couldn't see any explicit documentation on this. I will include at the bottom of this message the terminal output I see when running the commands with this change.
This is my first Zephyr contribution, so I have tried to make minimal changes and keep docs information in line with what is already there. Please feel free to refer me to other examples where pyocd and J-Link are supported so I can copy their style/information. Or if I have missed anything with regards to contributing guidelines.
Thanks 😃
Tim
P.S Terminal output following this change:
>west --verbose flash
-- west flash: rebuilding
cmake version 3.26.3 is OK; minimum version is 3.13.1
Running CMake: 'C:\Program Files\CMake\bin\cmake.EXE' --build 'C:\Users\tim\zephyr_project\build'
ninja: no work to do.
-- west flash: using runner pyocd
WARNING: runners.pyocd: hex file (None) does not exist; falling back on .bin (C:\Users\tim\zephyr_project\build\zephyr\zephyr.bin). Consider enabling CONFIG_BUILD_OUTPUT_HEX.
-- runners.pyocd: Flashing file: C:\Users\tim\zephyr_project\build\zephyr\zephyr.bin
runners.pyocd: pyocd flash -e sector -a 0x30000000 -t mimxrt1170_cm7 'C:\Users\tim\zephyr_project\build\zephyr\zephyr.bin'
0001998 I Loading C:\Users\tim\zephyr_project\build\zephyr\zephyr.bin at 0x30000000 [load_cmd]
[==================================================] 100%
0008938 I Erased 36864 bytes (9 sectors), programmed 35584 bytes (139 pages), skipped 4096 bytes (16 pages) at 5.65 kB/s [loader]

>west --verbose debug
-- west debug: rebuilding
cmake version 3.26.3 is OK; minimum version is 3.13.1
Running CMake: 'C:\Program Files\CMake\bin\cmake.EXE' --build 'C:\Users\tim\zephyr_project\build'
ninja: no work to do.
-- west debug: using runner pyocd
-- runners.pyocd: pyOCD GDB server running on port 3333
runners.pyocd: pyocd gdbserver -p 3333 -T 4444 -t mimxrt1170_cm7
runners.pyocd: C:/Users/tim/zephyr-sdk-0.16.1/arm-zephyr-eabi/bin/arm-zephyr-eabi-gdb.exe 'C:\Users\tim\zephyr_project\build\zephyr\zephyr.elf' -ex 'target remote :3333' -ex 'monitor halt' -ex 'monitor reset' -ex load
GNU gdb (Zephyr SDK 0.16.1) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-host_w64-mingw32 --target=arm-zephyr-eabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://github.com/zephyrproject-rtos/sdk-ng/issues>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from C:\Users\tim\zephyr_project\build\zephyr\zephyr.elf...
0001337 I Target type is mimxrt1170_cm7 [board]
0001378 I DP IDR = 0x6ba02477 (v2 rev6) [dap]
0001443 I AHB-AP#0 IDR = 0x84770001 (AHB-AP var0 rev8) [discovery]
0001490 I AHB-AP#1 IDR = 0x24770011 (AHB-AP var1 rev2) [discovery]
0001507 I APB-AP#2 IDR = 0x54770002 (APB-AP var0 rev5) [discovery]
0001560 I AHB-AP#0 Class 0x1 ROM table #0 @ 0xe00fd000 (designer=00e:NXP part=88c) [rom_table]
0001567 I [0]<e00fe000:ROM class=1 designer=43b:Arm part=4c8> [rom_table]
0001570 I   AHB-AP#0 Class 0x1 ROM table #1 @ 0xe00fe000 (designer=43b:Arm part=4c8) [rom_table]
0001580 I   [0]<e00ff000:ROM class=1 designer=43b:Arm part=4c7> [rom_table]
0001581 I     AHB-AP#0 Class 0x1 ROM table #2 @ 0xe00ff000 (designer=43b:Arm part=4c7) [rom_table]
0001640 I     [0]<e000e000:SCS v7-M class=14 designer=43b:Arm part=00c> [rom_table]
0001644 I     [1]<e0001000:DWT v7-M class=14 designer=43b:Arm part=002> [rom_table]
0001648 I     [2]<e0002000:FPB v7-M class=14 designer=43b:Arm part=00e> [rom_table]
0001660 I     [3]<e0000000:ITM v7-M class=14 designer=43b:Arm part=001> [rom_table]
0001727 I   [1]<e0041000:ETM M7 class=9 designer=43b:Arm part=975 devtype=13 archid=4a13 devid=0:0:0> [rom_table]
0001732 I   [2]<e0042000:CTI CS-400 class=9 designer=43b:Arm part=906 devtype=14 archid=0000 devid=40800:0:0> [rom_table]
0001738 I [1]<e0043000:Trace Funnel CS-400 class=9 designer=43b:Arm part=908 devtype=12 archid=0000 devid=28:0:0> [rom_table]
0001756 I AHB-AP#1 Class 0x1 ROM table #0 @ 0xe00ff000 (designer=43b:Arm part=4c4) [rom_table]
0001830 I [0]<e000e000:SCS v7-M class=14 designer=43b:Arm part=00c> [rom_table]
0001834 I [1]<e0001000:DWT v7-M class=14 designer=43b:Arm part=002> [rom_table]
0001840 I [2]<e0002000:FPB v7-M class=14 designer=43b:Arm part=003> [rom_table]
0001844 I [3]<e0000000:ITM v7-M class=14 designer=43b:Arm part=001> [rom_table]
0001856 I [5]<e0041000:ETM M4 class=9 designer=43b:Arm part=925 devtype=13 archid=0000 devid=0:0:0> [rom_table]
0001862 I [7]<e0043000:Trace Funnel CS-400 class=9 designer=43b:Arm part=908 devtype=12 archid=0000 devid=28:0:0> [rom_table]
0001868 I [8]<e0042000:CTI CS-400 class=9 designer=43b:Arm part=906 devtype=14 archid=0000 devid=40800:0:0> [rom_table]
0001878 I CPU core #0 is Cortex-M7 r1p2 [cortex_m]
0001885 I FPU present: FPv5-D16-M [cortex_m]
0001982 I 4 hardware watchpoints [dwt]
0002016 I 8 hardware breakpoints, 1 literal comparators [fpb]
0002143 I 4 hardware watchpoints [dwt]
0002175 I 6 hardware breakpoints, 4 literal comparators [fpb]
0002400 I Semihost server started on port 4444 (core 0) [server]
Remote debugging using :3333
0002710 I GDB server started on port 3333 (core 0) [gdbserver]
0002713 I Client connected to port 3333! [gdbserver]
0x002230e8 in ?? ()
0002745 I Attempting to load RTOS plugins [gdbserver]
Successfully halted device
Resetting target
Loading section rom_start, size 0x23a8 lma 0x30000000
Loading section text, size 0x45ac lma 0x300023a8
Loading section .ARM.exidx, size 0x8 lma 0x30006954
Loading section initlevel, size 0xa8 lma 0x3000695c
Loading section device_area, size 0x154 lma 0x30006a04
Loading section sw_isr_table, size 0x6d0 lma 0x30006b58
Loading section log_const_area, size 0x28 lma 0x30007228
Loading section rodata, size 0x2828 lma 0x30007250
Loading section datas, size 0x10 lma 0x30009a78
Loading section device_states, size 0x22 lma 0x30009a88
Loading section .last_section, size 0x4 lma 0x30009aaa
[==================================================] 100%
0005688 I Erased 0 bytes (0 sectors), programmed 0 bytes (0 pages), skipped 39680 bytes (155 pages) at 13.87 kB/s [loader]
--Type <RET> for more, q to quit, c to continue without paging

Hi @DerekSnell, thanks for reviewing this PR. Glad to hear that NXP are working hard to support Zephyr 👍 I'm happy to strip back most of the changes in light of these updates, but I still think it would be useful to correct the following line on the docs page:
however the [pyOCD Debug Host Tools](https://docs.zephyrproject.org/latest/develop/flash_debug/host-tools.html#pyocd-debug-host-tools) do not yet support programming the external flashes on this board

As above, pyocd does now support programming the flashes on this board, I just tested it this morning in the 0001b setting to double check. The line above is included verbatim in most of the mimxrt docs (see below), so if we can come up with a correction for this board, it could be verified by NXP and applied to all of the other board docs as well.
This would not interfere with either of the PRs you have referred to above.
$ grep -r 'however the :ref:`pyocd-debug-host-tools` do not yet support programming the' --include=*.rst boards/
boards/arm/mimxrt1010_evk/doc/index.rst:however the :ref:`pyocd-debug-host-tools` do not yet support programming the
boards/arm/mimxrt1015_evk/doc/index.rst:however the :ref:`pyocd-debug-host-tools` do not yet support programming the
boards/arm/mimxrt1020_evk/doc/index.rst:however the :ref:`pyocd-debug-host-tools` do not yet support programming the
boards/arm/mimxrt1024_evk/doc/index.rst:however the :ref:`pyocd-debug-host-tools` do not yet support programming the
boards/arm/mimxrt1040_evk/doc/index.rst:however the :ref:`pyocd-debug-host-tools` do not yet support programming the
boards/arm/mimxrt1050_evk/doc/index.rst:however the :ref:`pyocd-debug-host-tools` do not yet support programming the
boards/arm/mimxrt1060_evk/doc/index.rst:however the :ref:`pyocd-debug-host-tools` do not yet support programming the
boards/arm/mimxrt1062_fmurt6/doc/index.rst:however the :ref:`pyocd-debug-host-tools` do not yet support programming the
boards/arm/mimxrt1064_evk/doc/index.rst:however the :ref:`pyocd-debug-host-tools` do not yet support programming the
boards/arm/mimxrt1160_evk/doc/index.rst:however the :ref:`pyocd-debug-host-tools` do not yet support programming the
boards/arm/mm_swiftio/doc/index.rst:however the :ref:`pyocd-debug-host-tools` do not yet support programming the

I've added another commit now which removes changes to board.cmake and some of the other changes to the docs, but adds a small section detailing how to use pyOCD. I have tried to keep this in line with language in other docs such as Nucleo h563zi which also includes notes on pyocd as an alternative flash/debug runner.
As above, if someone with access to all of the NXP MIMXRT boards can confirm this is working on them as well then we can potentially apply the same patch to all of those docs without interfering with any of the LinkServer work 👍
Please let me know if you think this is suitable.
Thanks,
Tim

FYI @TimTTP , we submitted our PR Enable LinkServer support on NXP RT1170, which also updates the RT1170 board doc page for details using the default CMSIS-DAP on-board debugger firmware with LinkServer.  Let us know if you have any feedback.  Thanks again for your efforts to improve the debug experience and documentation for this board.

Hi @DerekSnell, sorry for my late reply. Good to see you got the LinkServer change completed, I'm sure that will be helpful.
I am a bit short on time at the moment but would like to make it clearer that pyocd is an option for developers who are not familiar with Zephyr (like me).
I will make relevant changes next week.
Thanks,
Tim

@TimTTP @DerekSnell Sorry for this late reply, I was on a two-week business trip and just came back to my workstation today and had a chance to check these messages. I will try to walk through and understand what's going on asap. If you need anything from me urgently, please let me know.

@TimTTP @DerekSnell Sorry for this late reply, I was on a two-week business trip and just came back to my workstation today and had a chance to check these messages. I will try to walk through and understand what's going on asap. If you need anything from me urgently, please let me know.

Hi @zanderzhng ,
There is nothing urgent here.  Our PR Enable LinkServer support on NXP RT1170 has already merged.  I was just including you because your PR https://github.com/zephyrproject-rtos/zephyr/pull/61149 was related to this one.  Thanks

@TimTTP, can we close this PR given the feedback from @DerekSnell and the recently merged LinkServer PR?

Hi @dleach02, sorry for leaving this open for so long. I still think a reference to pyOCD would be really helpful as I struggled to find this as a beginner with Zephyr. Although the NXP tools are great, they might not be suitable for everyone and don't work straight out of the box with Zephyr. If this is agreed with then I'm happy to suggest another change but otherwise it can be closed, yes.

I have now made what I believe is a minimal change to the docs that makes it clear that pyOCD is a working option for this board. As above, pyOCD is a straightforward option for beginners as it is installed as part of the setup process. Knowing this would have helped me a lot when learning how to use Zephyr with this board.
@DerekSnell I understand your note above that NXP is encouraging the use of LinkServer and JLink. If you can give the commit as is it is now, and my comments, a short review I would be grateful. If you decide that it is still not suitable then please close this PR. Thanks for working on the Zephyr support for this board in the first place :)

Hi @TimTTP ,
I appreciate your commitment to keeping the pyOCD option in this doc page.  I will review your request internally at NXP, and get back to you.  Thank you

Thanks @DerekSnell, look forward to hearing from you 👍

Thanks for your help, @DerekSnell, @butok, @dleach02 😄


Unexpected token '\n' when using global define directives

According to the rp2040 datasheet, .define directives should be allowed before the first program, and if they occur there are global to all programs in the file. There seems to be a bug when using pio_file!, where if you have a global define such as:
.define foo 0
.program bar
...
.program baz
...

And try to do
let program = pio_file!("barbaz.pio", select_program("bar"));

It errors with "unexpected token \n" at the end of the first line. However, if you instead place .define foo 0 and .program bar on one line, it works, and the define is global. PIOASM has the opposite behavior, where having both on one line results in "unexpected .program", and having them on different lines works as expected.

if anyone wants to fix this before i find the motivation to get around to it, the problem is here:
https://github.com/rp-rs/pio-rs/blob/0a6d22f083633b0ee8cd520714d917bb68e0c359/pio-parser/src/pio.lalrpop#L35
each DefineDirective should have a NEWLINE+ after it.


8.0.0: Replace Moq with NSubstitute

Relates to https://github.com/moq/moq/issues/1372

Codecov Report

Merging #52 (3b92116) into master (97c600a) will increase coverage by 5.07%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #52      +/-   ##
==========================================
+ Coverage   93.08%   98.16%   +5.07%
==========================================
  Files          53       53
  Lines         926      924       -2
==========================================
+ Hits          862      907      +45
+ Misses         64       17      -47

Files Changed
Coverage Δ

src/MooVC/Threading/Coordinator.cs
100.00% <100.00%> (+16.32%)
:arrow_up:

... and 4 files with indirect coverage changes


Modular PlusCal [proposal]

This issue documents my proposal to extend the PlusCal language with features that enable composability.
Currently one PlusCal algorithm cannot interact with another in any way. They cannot refer to each other during model checking, and there is no consistent way to link PGo-compiled implementations together via shared variables since the algorithms assume exclusive access and this is liable to break invariants without significant modifications to the generated code.
The minimal set of features required for PlusCal to support modules are:

a parameterised variant of processes. Tentatively called "archetypes", these would be the same as processes but would be able to communicate with unknown outside processes via channels. These would be compiled into Go and would be controlled by communicating with them via the channels.
a way to define channels that would be used during model checking
a way to instantiate "archetypes", which would be used in order to generate a concrete model for TLC to check (TLC needs a specific set of processes and globals to work with)
PlusCal statements to send and receive over channels
an import directive to load other PlusCal algorithms (required since the compiled TLA+ needs to be very careful about managing global variables, native TLA+ imports and a TLA+ API are insufficient)

Example of what a PlusCal developer would do in order to use these things, assuming they are writing a system like RAFT where all the processes have the same code:

change the single process declaration to an "archetype" declaration, with parameters for any decisions that would normally be modeled using nondeterminism (like what to propose, etc... things that are needed by an implementation but not a model)
add instantiations of that "archetype" to the PlusCal algorithm, providing nondeterministically chosen values for all parameters
compile to TLA+, check that the instantiation of the algorithm works using TLC
compile to Go and write a small amount of Go code that starts/sets up an appropriate set of RAFT nodes. Note that the archetype is what was compiled, so the hand-written Go code is responsible for deciding what the real parameter values should be. This separates the redundant non-determinism used for model checking and actually meaningful non-determinism that is part of how the algorithm works, allowing users to cleanly replace the random model checking values with real implementation values while leaving the rest of the algorithm alone.
use the RAFT implementation to do things

Hopefully this give a decent outline of what I'm talking about - let me know if anything is unclear.

Here is a more fleshed out example of the transformation.
Before
--algorithm counter {
  variables counter = 0;

  process (P \in 0..procs-1)
  variables i = 0;
  {
      w: while (i < iters) {
          incCounter: counter := counter + 1;
                      print counter;
          nextIter:   i := i + 1;
      }
  }
}

After
--algorithm counter {
  archetype P(self, counter, iters)
  variables i = 0;
  {
      w: while (i < iters) {
          incCounter: counter := counter + 1;
                      print counter;
          nextIter:   i := i + 1;
      }
  }
  (** testing **)
  variables counter = 0;
  process (P \in 0..procs-1) = instance P(self, counter, iters);
}

Now, if you wanted to use this algorithm from another PlusCal module:
--algorithm UseCounter {
  import counter;
  (** counter is not very useful, so we just demonstrate another process reading the variable **)
  process (Q \in 0..procs-1) = instance P(self, counter, iters);
  process (Observer = -1)
  variables i = 0;
  {
    w: while( i < iters ) {
      i := i+1;
      print counter;
    }
  }
}

Or if you want to use it from custom Go code (code is approximate, precise model for "global" variables needs work):
import "counter"

var counter int = 0;

func main(){
  // you can do anything here (incl. goroutines), but for simplicity we just call P directly
  counter.P(counter.MakeContext(/*TBD*/), 1, &counter, 15)
}

Here is an approximation of counter.go:
package counter

func P(/*context, TBD */, self int, counter *int, int iters){
  // generated code goes here
}

Extra thoughts on codegen for globals in this context: what if we used unbuffered channels as locks?
As in, at the beginning of the program you send the initial values of all globals on channels. You then pass those channels to the relevant processes and when a process needs that variable they perform a receive. If that variable is available its value will be received. If it has already been received any other processes trying to receive it will block. When the process is done with the variable, it sends the new (possibly updated) value back along the channel.
The key advantage here is that you can pass channels around easily, whereas passing pointers around requires separately tracking a lock mechanism. This is likely to not compose as well in a modular context.

A couple thoughts/questions:

Have you thought about the precise semantics of your import keyword? Does it bring to scope only archetypes. or macros/procedures/operator definitions/globals are also imported?

Is this going to help you in your module system idea or is it unrelated? The "modular" in the name makes me think so, but I can't see how exactly.

Perhaps a better example to illustrate your idea would be to use the 2pc spec instead of counter. Even more useful would be to use the "official" specifications (available at http://lamport.azurewebsites.net/tla/two-phase.html), since they are written in more idiomatic PlusCal than the modified version in our examples directory. If you have the time, it would be great to see a sketch of how that spec would change to fit the proposal here.

@rmascarenhas good questions.

At least archetypes. We would have to be very careful about scoping the rest though, since you'd have to keep in mind how it compiles to Go and TLA+. So: macros could be importable without problems. Procedures would not be so easy since they don't have proper return values, so you would need to assign to a global from the other module (which is broken as we should not be able to import globals). Globals should not be importable as that would break the separation archetypes gives us. Operators should be importable as normal from TLA+, so I'm not sure allowing them in PlusCal import would be useful.

this is my module system idea. It allows re-use of PlusCal compiled as Go (and possible swapping out of processes with alternate implementations).

I'll do that.

Interesting point from a discussion with @mrordinaire yesterday, clarifying that my idea for Module PlusCal is directly replacing the PlusCal translator rather than compiling from Module PlusCal to regular PlusCal.
The important insight is that regular PlusCal is fundamentally broken when it comes to variable priming (that is, correctly generating ' and UNCHANGED in the generated code). Modular PlusCal changes the way globals are treated in subtle but important ways by altering the compilation strategy.

P2TCommit.tla before
--algorithm TransactionCommit {
  variable rmState = [rm \in RM |-> "working"] ;
  define {
    canCommit == \A rm \in RM : rmState[rm] \in {"prepared", "committed"}
    notCommitted == \A rm \in RM : rmState[rm] # "committed"
   }
  macro Prepare(p) {
    when rmState[p] = "working";
    rmState[p] := "prepared" ;
   }

  macro Decide(p) {
    either { when /\ rmState[p] = "prepared"
                  /\ canCommit ;
             rmState[p] := "committed"
           }
    or     { when /\ rmState[p] \in {"working", "prepared"}
                  /\ notCommitted ;
             rmState[p] := "aborted"
           }
   }

  fair process (RManager \in RM) {
    start: while (rmState[self] \in {"working", "prepared"}) {
      either Prepare(self) or Decide(self) }
   }
}

P2TCommit.tla after
--algorithm TransactionCommit {
  define {
    canCommit(rmState) == \A rm \in DOMAIN rmState : rmState[rm] \in {"prepared", "committed"}
    notCommitted(rmState) == \A rm \in DOMAIN rmState : rmState[rm] # "committed"
   }
  macro Prepare(selfState) {
    when selfState = "working";
    selfState := "prepared" ;
   }

  macro Decide(selfState, rmState) {
    either { when /\ selfState = "prepared"
                  /\ canCommit(rmState) ;
             selfState := "committed"
           }
    or     { when /\ selfState \in {"working", "prepared"}
                  /\ notCommitted(rmState) ;
             selfState := "aborted"
           }
   }

  archetype RManager(selfState, rmState) {
    start: while (rmState \in {"working", "prepared"}) {
      either Prepare(selfState, rmState) or Decide(selfState, rmState) }
   }

  (* model checking section *)
  variable rmState = [rm \in RM |-> "working"] ;
  process (RManagerProc \in RM) == instance RManager(rmState[self], rmState);
}

In consideration of #77 , here are my thoughts on how custom implementations might apply to Modular PlusCal.

The simplest option for a variable is just a variable. It is the least efficient and most general. It is also completely oblivious to packet drop and/or duplication.
As PlusCal stands, network concepts are implemented in an ad-hoc manner, making the general extraction of working Go code infeasible.
What we're trying to do here is allow users to map from PlusCal concepts to implementation concepts in a general and consistent way.

Example 1: if a network between two processes is modeled by a global variable and another process that "interferes" with that variable, we model reads and writes to that global as calls to an opaque object (taken as a Modular PlusCal parameter) that passes these through to a real network.
Example 2: if a process needs a timer, it can communicate with a second process via a shared global variable that "pretends" to keep time (since PlusCal is not realtime). In Go that variable can be modeled as an opaque object (taken as a Modular PlusCal parameter) that implements real semantics.
In summary, we can implement almost any semantics with a combination of model-check-only processes and flexible compilation of variable reads and writes. For ease of use we should provide a Go library of network/OS primitives that can be plugged into this mechanism. If someone wants something we don't have, the mechanism should be defined as an interface with a few necessary  methods (acquire, release, ... not sure what else) so they can implement extras as necessary.
@bestchai any thoughts? This is the "purest" adaptation of PlusCal that I can think of, with no additional semantic changes to Modular PlusCal.

Example: send/recv with retry
Below is a set of archetypes that implement a send/receive pair with ack. This will be the basis for trying to express some custom error functions in Modular PlusCal.
CONSTANTS emptyMsg

archetype SendRetry(toSend, send, ack){
    variable currentMsg, seqNum = 0;
    w: while(TRUE){
        currentMsg := [ num -> seqNum, msg -> Head(toSend) ];
        retry: while(ack < seqNum){
            send := currentMsg;
        }
        seqNum := seqNum + 1;
    }
}

archetype Recv(recv, ack, receivedMsgs){
    w: while(TRUE){
        await recv.num = ack + 1;
        ack := ack + 1;
        receivedMsgs := Append(receivedMsgs, recv.msg);
    }
}

A lossless network (just "plug" the two instances into each other):
variables ack, pipe, toSend, receivedMsgs;

instance SendRetry(toSend, pipe, ack);
instance Recv(pipe, ack, receivedMsgs);

A lossy network with no reordering:

"vanilla" MPCal:variables ack1, ack2, pipe1, pipe2, toSend, receivedMsgs;

instance SendRetry(toSend, pipe1, ack1);
instance Recv(pipe, ack2, receivedMsgs);
process (Error = 1){
    w: while(TRUE){
        either {
            ack1 := ack2;
        } or {
            skip;
        };
        either {
            pipe2 := pipe1;
        } or {
            skip;
        };
    }
}

critique: introduces extra timesteps for the Error process to pass messages across pipe and ack, makes naming and tracking of state complicated for humans (look how long this simple example got).
with the "mapping" construct (semantics: within the scope of a specific instance, for mapping write <var> { ... }, replace writes to <var> with ..., where the original value to be written is called $value):variables ack, pipe, toSend, receivedMsgs;

instance SendRetry(toSend, pipe, ack1)
    mapping write pipe { pipe := CHOOSE c \in { TRUE, FALSE } : IF c THEN $value ELSE pipe; }
instance Recv(pipe, ack2, receivedMsgs)
    mapping write ack { ack := CHOOSE c \in {TRUE, FALSE} : IF c THEN $value ELSE ack; }

critique: allows to alter the semantics of writes to variables in an archetype instance, so does not incur extra timesteps or naming difficulties. The pattern for "maybe change the value" is a bit repetitive though.
the "mapping" construct with the boilerplate factored out into a "Maybe" operator (operator defined inline for brevity):variables ack, pipe, toSend, receivedMsgs;

Maybe(new, old) == CHOOSE c \in {TRUE, FALSE} : IF c THEN new ELSE old

instance SendRetry(toSend, pipe, ack1)
    mapping write pipe { pipe := Maybe($value, pipe); }
instance Recv(pipe, ack2, receivedMsgs)
    mapping write ack { ack := Maybe($value, ack); }

critique: similar to above, but shorter. opportunity for code reuse in the general case. It might be interesting to allow macros here as well.

Now for a similar exercise with a lossy and reordering-capable network (but with no packet duplication) ("vanilla" example omitted as it would only get longer and harder to get right):
variables ack = {}, pipe = {}, toSend, receivedMsgs;

instance SendRetry(toSend, pipe, ack)
    mapping write pipe { either {
                                         pipe := pipe \union { $value };
                                     } or {
                                         skip;
                                     }
                                   }
    mapping read ack { with chosen \in ack {
                                        ack := ack \ { chosen };
                                        read chosen;
                                    };
instance Recv(pipe, ack, receivedMsgs)
    mapping write ack { either {
                                         ack := ack \union { $value };
                                     } or {
                                         skip;
                                     }
                                   }
    mapping read pipe { with chosen \in pipe {
                                        pipe := pipe \ { chosen };
                                        read chosen;
                                    };

critique: while it is fairly clear what it going on here, this is getting longer again due to more duplicated code. Since we are effectively redefining a variable's read/write interface, we need a way to abstract away the "network-as-set" pattern.
Idea: factor out the duplicate code using a "mapping macro". This is a construct that encapsulates a complete remapping of a single variable. Assuming a library of standard constructs was available, this would make reuse of common communication models fairly straightforward.
variables ack = {}, pipe = {}, toSend, receivedMsgs;

mapping macro LossyReorderingNetwork {
    read {
        with chosen \in $value {
            $value := $value \ { chosen };
            read chosen;
        }
    }
    write {
        either {
            write $old \union { $value };
        } or {
            skip;
        }
    }
}

instance SendRetry(toSend, pipe, ack)
    mapping pipe via LossyReorderingNetwork
    mapping ack via LossyReorderingNetwork;
instance Recv(pipe, ack, receivedMsgs)
    mapping pipe via LossyReorderingNetwork
    mapping ack via LossyReorderingNetwork;

critique: what if we need two variables to be consistently remapped together, or a mapping only makes sense with a parameter?

2PC and fail-stop
Mapping is useful when communication channels are fallible, but does not address when processes are fallible. One of the big ideas behind 2PC is logging and retry on failure, which needs simulated failure and "real" logging.
Below is a 2PC spec in Modular PlusCal that does not implement failure.
----------------------------- MODULE 2pc -----------------------------
EXTENDS Integers,Sequences,FiniteSets,TLC

(* --algorithm 2pc {

archetype Cohort(coordinatorCommunication, localSuccess, localResult) {
    waitForQuery:
        await coordinatorCommunication = "queryToCommit";
        coordinatorCommunication := IF localSuccess THEN "yes" ELSE "no";
    commitOrRollback:
        await coordinatorCommunication \in { "commit", "rollback" };
    setLocalResult:
        localResult := coordinatorCommunication;
    sendAck:
        coordinatorCommunication := "ack";
}

archetype Coordinator(cohortCommunication, success)
variables cohorts = {}, response;
{
    queryToCommit:
        cohorts := DOMAIN cohortCommunication;
        queryLoop: while (cohorts # {}) {
            with (c \in cohorts) {
                cohortCommunication[c] := "queryToCommit";
                cohorts := cohorts \ {c};
            };
        };
    waitForVotes:
        await \A c \in DOMAIN cohortCommunication : cohortCommunication[c] \in { "yes", "no" };
    responseToVotes:
        response := IF \E c \in DOMAIN cohortCommunication : cohortCommunication[c] = "no" THEN "rollback" ELSE "commit";
        cohorts := DOMAIN cohortCommunication;
        responseLoop: while (cohorts # {}) {
            with (c \in cohorts) {
                cohortCommunication[c] := response;
                cohorts := cohorts \ {c};
            };
        };
    waitForAck:
        await \A c \in DOMAIN cohortCommunication : cohortCommunication[c] = "ack";
    commitSuccess:
        success := TRUE;
}

variables comms = [ c \in 1..3 |-> "" ],
          cohortResult = [ c \in 1..3 |-> "" ],
          localSuccess \in [ 1..3 -> BOOL ],
          coordinatorSuccess = FALSE;

(* testing *)
fair instance \in 1..3 of Cohort(comms[self], cohortResult[self]);
fair instance 0 of Coordinator(comms, coordinatorSuccess)

}
*)

In order to show that this algorithm cannot withstand fail-stop, I propose an additional mapping construct: the ability to inject code in between steps of an algorithm.
Here is how one would change the instance section of the above spec in order to add fail-restart cycles (and demonstrate that the algorithm as written is broken under that scenario, since if the cohort fails after receiving "queryToCommit" it will reboot and hang in that state):
(* testing *)
fair instance \in 1..3 of Cohort(comms[self], cohortResult[self])
    interleaving { goto waitForQuery; };
fair instance 0 of Coordinator(comms, coordinatorSuccess)
    interleaving { goto queryToCommit; };

Semantics of interleaving: generate TLA+ code such that the given statement(s) may or may not execute in place of the next label in the algorithm, unless the algorithm has reached the state "Done". If you do not want one of these statements to be always capable of executing, adding an await will make the execution conditional on whatever condition is needed.

I see some interesting ideas here, @fhackett,  thanks for writing these detailed examples.

Comment 1

is the emptyMsg constant some leftover from a previous version of your idea? If so, better remove it so it's less confusing.
If I understand this right, toSend is a sequence of messages you want each instance of SendRetry to send, is that right? If so, it would make the example more realistic to actually pop from the list, or at least iterate over its elements. I know these are just drafts, but making them as close to "reality" as possible would help understand the ideas (at least I think it would).
Another question: is toSend supposed to be local to instances of sendRetry? Does it matter?
I don't think the Maybe operator has the meaning you intended it to. CHOOSE will not cause TLC to branch, so you wouldn't explore both cases (i.e., message is sent or dropped). I'm sure you know this since you used either in your following examples, so this is also in the "making the examples realistic" category.
I'm not sure I understand the idea of "mapping macros". Is the idea of these macros always define read and write "operators"? I was confused by the fact that you have read chosen within the definition of read itself, but my current understanding is that in read chosen, read is some special keyword that only has meaning within a mapping macro. Is that right?

Comment 2

Syntax nitpick: I think you probably meant to say fair instance 1..3 of Cohort? The \in currently there looks out of place.
Not clear to me: the BOOL you added to localSuccesses is just a placeholder for an actual boolean the spec writer would use, or is that some special value in Modular PlusCal? You probably want something with the semantics of "either true or false" there (i.e., have TLC to branch every possible decision for every node in cohort).
Is it right to say interleaving { S } can be desugared by wrapping every step with either { step } or { S }? If so, we can experiment with this in regular PlusCal to see if works well as a technique to express failure/recovery/restart.

General comments.

This does seem to transform PlusCal into a different beast. It has the advantage of being a lot more general than #77, but on the other hand the changes to existing specs would be a lot more involved.
I wonder how the compiler will optimize "mappings" as in the examples you have here, since the mechanisms seem to be pretty general.
I guess a good exercise would be to try to write a complicated distributed algorithm in Modular PlusCal. Maybe WPaxos (which has an existing PlusCal spec), or Raft (a lot more work). Otherwise it's hard to know whether we are missing something critical. I can help with that.

I've added responses in-line - thanks for your comments.
On Wed, Jul 25, 2018 at 4:22 PM, Renato Costa<EMAIL_ADDRESS>wrote:

I see some interesting ideas here, @fhackett https://github.com/fhackett,
thanks for writing these detailed examples.

Comment 1
https://github.com/UBC-NSS/pgo/issues/75#issuecomment-405432708

is the emptyMsg constant some leftover from a previous version of
your idea? If so, better remove it so it's less confusing.

I can't remember why that was there, and can't find any references to it
either. Removed.

If I understand this right, toSend is a sequence of messages you
want each instance of SendRetry to send, is that right? If so, it
would make the example more realistic to actually pop from the list, or at
least iterate over its elements. I know these are just drafts, but making
them as close to "reality" as possible would help understand the ideas (at
least I think it would).

I have added a couple of lines in order to make the algorithm more correct

the corrected version is what was intended.

Another question: is toSend supposed to be local to instances of
sendRetry? Does it matter?

toSend is how one would control an instance of SendRetry - you would pass
in a list from Go and the algorithm would send the contents of the list (in
theory you could update the list dynamically so you're not limited to some
initial list value. That is the main reason toSend it is not local.

I don't think the Maybe operator has the meaning you intended it
to. CHOOSE will not cause TLC to branch, so you wouldn't explore
both cases (i.e., message is sent or dropped). I'm sure you know this since
you used either in your following examples, so this is also in the
"making the examples realistic" category.

My mistake. I had forgotten that that was how TLC worked. Added a note to
the example.

I'm not sure I understand the idea of "mapping macros". Is the
idea of these macros always define read and write "operators"? I
was confused by the fact that you have read chosen within the
definition of read itself, but my current understanding is that in read
chosen, read is some special keyword that only has meaning within a
mapping macro. Is that right?

Actually it's the other way around. Mapping macros mirror the way you
would be able to pass in custom Go implementations of variables in Go, but
for model checking. Whenever a variable is read (i.e any reference in a
critical section), the corresponding "read" code would run. Same for write,
but with assignment. The "read" and "write" statements are a way of
specifying the effective result of the mapping macro, like "return" for
functions.

Comment 2
https://github.com/UBC-NSS/pgo/issues/75#issuecomment-407263241

Syntax nitpick: I think you probably meant to say fair instance 1..3
of Cohort? The \in currently there looks out of place.

Sure, syntax is up for debate. The only issue is how to differentiate
between sets of processes and single processes, since syntactically your
change would be indistinguishable from a single instance named {1, 2, 3}.

Not clear to me: the BOOL you added to localSuccesses is just a
placeholder for an actual boolean the spec writer would use, or is that
some special value in Modular PlusCal? You probably want something with the
semantics of "either true or false" there (i.e., have TLC to branch every
possible decision for every node in cohort).

That is a mistake. Corrected to { TRUE, FALSE } (I must have forgotten to
correct the code in the post - I found and fixed that bug in my test files
before).

Is it right to say interleaving { S } can be desugared by
wrapping every step with either { step } or { S }? If so, we can
experiment with this in regular PlusCal to see if works well as a technique
to express failure/recovery/restart.

That sounds like a valid desugaring. The actual process I imagined for
direct-to-TLA compilation was to add extra branches nearer to Next, but
your version should be equivalent if more verbose ... though one possible
hurdle is control flow like while and if, which may be harder to convert.
Worth a try at least.

General comments.

This does seem to transform PlusCal into a different beast. It has
the advantage of being a lot more general than #77
https://github.com/UBC-NSS/pgo/issues/77, but on the other hand
the changes to existing specs would be a lot more involved.

I agree. The original intention here was to convert our existing 2PC spec,
but the changes required in order to specify a proper API were non-trivial.
There is definitely a "you may have to rethink your code" disclaimer, but I
think that is a fundamental property of adding an API to code that did not
originally have one.

I wonder how the compiler will optimize "mappings" as in the
examples you have here, since the mechanisms seem to be pretty general.

Mappings are for model-checking so they will not appear in the Go output,
since the idea is to compile only the archetypes. If we did compile the
mappings (say, in order to generate an example main function that runs the
archetypes as described in the spec), they would probably just be expanded
like macros and then be optimised (or not) like anything else.

I guess a good exercise would be to try to write a complicated
distributed algorithm in Modular PlusCal. Maybe WPaxos (which has an
existing PlusCal spec), or Raft (a lot more work). Otherwise it's hard to
know whether we are missing something critical. I can help with that.

Any help would be appreciated there - I ran into a wall on the Raft front,
and was unaware of WPaxos. One spec I'd like to add is 3PC, even if we just
write it from scratch since it features timeouts and I want to make sure
the mappings support that properly.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/UBC-NSS/pgo/issues/75#issuecomment-407927096, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AImmObSmwBUBXvAuUkGmacmxJ3h5ZSruks5uKP28gaJpZM4Uy_qH
.

ref semantics

ref may be used in the parameter list of an instance, an archetype, a procedure definition or a call statement (macros are unaffected, since they simply expand syntactically)
ref is used in a definition to indicate that an argument is call by name rather than call by value
ref is used at a callsite to show that the parameter is being passed by name
ref may be used to qualify any parameter that is also assignable
ref may not be nested, and the global a ref refers to may not change once it has been initialised. Aliasing is possible, but well defined. Using to the ref by referencing it or assigning to it will directly affect the referenced global. In the context of aliasing, if a and b refer to the same global and one were to write a := b + 1;, the code would operate in exactly the same way as x := x+1. Likewise, if we alias a to x and b to x[6], the resulting code would operate in the same manner as if x[6] := x had been written. Aside from simple sharing like I have described here, no other aliasing is possible due to the inability to store a reference in a variable (ref has the same limitation as C++ references compared to pointers).
taking a ref of a ref will simply cause the second ref to refer to the same global as the first.
taking a ref of a local variable will have the same semantics as for globals

TLA+ compilation for ref:
Archetypes will be compiled to a series of operators (matching the names of existing labels). If there is a label op, typical process compilation will produce either op == ... or op(self) == ... depending on whether it is a single process or a set of processes. If an archetype with a label op takes ref var as an argument, the compiled label might look like op(var) == .... This works because TLA+ allows statements like var' = 3 even when var is an operator argument, so mutations to var will propagate to the operator callsite.
When compiling an instance of an archetype with label op and the parameter var, passing in ref someGlobal, the specification's Next disjunction will contain op(someGlobal). This means the archetype will affect the global while calling it var.
The same can be essentially said of procedures, but with the following restrictions:

trying to ref a procedure's local variable(s) should fail as they are reassigned as part of the calling convention. Passing along ref arguments is fine.
different invocations of a procedure with different ref arguments should have different pc values, appearing in Next as something like op("specialPC", var) instead of just op(var) like with archetypes.
mapping macros require special consideration (see below)

What happens if you take a ref of a variable that is backed by a mapping macro?
The simplest approach would be to recompile the procedure once per mapping, injecting the appropriate code on variable reads and writes. There may be nicer ways to do it, but for now this should keep things (relatively) simple (see below).
Bonus: mapping macro applied to x.y := z
Considering the above raised the question of how in general a mapping macro should respond to x.y := z, where x is subject to the mapping macro.
It is easy to consider what happens in the case x := y, since that might obviously translate to the macro contents with $value replaced by y and write xx replaced by x := xx. The tricky part is noticing that x.y in x.y := z refers to .y of the mapped x, not its backing value.
This can be resolved like so: if original is the value returned by the read portion of the mapping macro, $value should be defined as [ original EXCEPT !.y = z], with write xx defined as x := xx. In order to be able to convert between x.y and an EXCEPT expression consistently, we need to be aware that the left-hand side of an assignment can only be a plain variable or an index of some kind. Anything else is either invalid at the TLA+ level or forbidden by the PlusCal translator.

x.y := z with mappings pt. 2
Having (re)read the PlusCal grammar (my bad for claiming there wasn't one, it's the formal semantics that are missing not the grammar), I can confirm that an assignment LHS is limited to precisely some identifier a followed by a dot .b or an index [c, d ...].
For the following series of statements:
a.b := c;
a.d := a.b;
d[5] := a.d;

A reasonable translation might be:
LET aRead == <result of reading a goes here>
    aPrime == [aRead EXCEPT !.b = c]
    aPrimePrime == [aPrime EXCEPT !.d = aPrime.b]
    dRead == <result of reading d goes here>
    dPrime == [dRead EXCEPT ![5] = aPrimePrime.d]
IN
    /\ a' = <result of applying the write mapping to aPrimePrime goes here>
    /\ d' = <result of applying the write mapping to dPrime goes here>

Obviously there are a lot of optimisations that can be done in special cases like the code not containing assignments to fields or indices, but I can't think of any cases that would cause this translation to be semantically inaccurate.
(caveat: real codegen may be uglier due to nested let expressions - imagine what would happen if you put a print statement in between some of those assignments and we had to add a /\ case in the middle of the let declarations)

Thanks for thinking this through, @fhackett!
A couple of questions/comments follow:

Not sure I understand both of these statements:

different invocations of a procedure with different ref arguments should have different pc values

The simplest approach would be to recompile the procedure once per mapping

Maybe examples would make it easier for me to understand what you propose.

I find the strategy of mapping macros with TLA+ functions (or tuples/records) a bit confusing and I think we may benefit if we can avoid those murky waters if at all possible.

In your example:
a.b := c;
a.d := a.b;
d[5] := a.d;

the read macro for a is only invoked for a.b. I had the understanding before that read macros would be evaluated whenever a variable is read, but this example makes me think the semantics are more complicated. And if that's the case, we should write down these semantics more carefully.

what if y already has a mapping macro in x.y := z? Are we giving them precedences? Invoking both (in some order)? Compilation error (this could be tough to catch statically with aliasing)?

The write macro is also only invoked once, despite the fact that two write operations were performed.

What would happen if I had something like:

procedure MyProc(ref x) {
  x := 10;
}

archetype Arch(ref map)
variable localMap = [one |-> 1] {
  call MyProc(ref localMap.one)
}

Would it be allowed? Were you planning to have MyProc receive the entire map when translating that to TLA+? You seemed to imply that we would allow "interior references", but I wonder how much it would complicate things.
My goal here is to make things more straightforward not only in terms of PGo's implementation, but also understanding what's going on under the hood when we are writing specs in modular PlusCal. The semantics seem to be pretty nuanced and I'm thinking how we could drop features without sacrificing expressiveness.
One possibility would be to not support mapping macros on tuples and records (i.e., not at the granularity you are proposing here). But things could still work if we restricted their assignment to tupl := [tupl EXCEPT ![idx] = val], which should be fine for most needs.

More compilation thoughts and discussion
I have tried some more experiments and it seems that any "smart" compilation where we parameterise references using TLA+ tricks will not work. The problem is generating correct UNCHANGED expressions, which requires knowing every variable that has not been assigned to in any given step.
Under a mapping macro that might assign to any global variable during its execution, it is impossible to statically compute what side-effects any part of a Modular PlusCal algorithm might have. Attempts at expressing dynamic computation of this have failed with TLC errors, so I'm leaning toward "impossible as far as I can tell" as the status of "smart" compilation of Modular PlusCal.
So, this leaves us with complete static expansion of ref and mapping macro, duplicating any critical sections that are affected by the semantics of these constructs. At this state we can expect compiled TLA+ to look a lot like the output of the original PlusCal translator, but possibly generating multiple different versions of labels.
@rmascarenhas to address your comments in order:

I find the strategy of mapping macros with TLA+ functions (or tuples/records) a bit confusing and I think we may benefit if we can avoid those murky waters if at all possible.

it seems that was impossible anyway (see above). I agree, that was confusing.

the read macro for a is only invoked for a.b. I had the understanding before that read macros would be evaluated whenever a variable is read, but this example makes me think the semantics are more complicated. And if that's the case, we should write down these semantics more carefully.

I see. ~To clarify, what is happening here is that a is being read once. Multiple references within a critical section act as a single atomic read, so by this logic the read mapping must be executed once. The confusing addition that we are reading a.b and a.d is due to TLA+'s weird priming semantics. Normally you can't assign to a twice, but you can assign to different fields of the same a as many times as you like.~
I tried this in the TLA+ toolbox to double check and I have no idea how I reached that idea. Thank you for making me check this. The correct semantics are that a.b := c is a shorthand for a' = [a EXCEPT .b = c ], plain and simple. Well that makes this a lot easier. First compile to the EXCEPT form, then feed the value to the write macro. Done!

what if y already has a mapping macro in x.y := z? Are we giving them precedences? Invoking both (in some order)? Compilation error (this could be tough to catch statically with aliasing)?

You can't actually map y. Since the only way y could be mapped is by having it be a ref parameter to an archetype, accessing a field y of x just assumes that x was mapped to a record with a field y. Put another way, you can provide a field of an existing object as a ref parameter, but you can't somehow change the fields of something passed via ref to be anything new.
This does bring up the issue of passing the same object in multiple positions as ref parameter, but as far as I can see we can just reject that if we notice references to the same thing in different parameters to the same call (since this is just asking for multiple-assignment compilation errors). Good thing to think about though, thanks for asking that.

The write macro is also only invoked once, despite the fact that two write operations were performed.

See my response to read. Seems like this was never a thing. :man_facepalming:
Write macros will be executed only once, not because of some weird logic but because you can only assign to something once in the same critical section.

What would happen if I had something like: [...]

Given our strategy to statically expand these things, assignments to parameter x with value ref localMap.one would compile as if they were written localMap.one := ..., which would compile as discussed above. Aren't I glad you made me re-evaluate some things... otherwise my answer would have been much less simple.

Progress update on PlusCal translator:

there is a new flag -tlagen that causes PGo to pretend to be the PlusCal translator
Queens.tla and Euclid.tla can be translated (needs proper verification / much better code formatting, but the output at least looks plausible)
most control flow except for procedures (not implemented yet) seems to work
only single-process compilation is supported at the moment (but what is there should be easily portable to multi-process)
I have an untested skeleton for compiling ref and mapping semantics

Aside: it seems that you can write a.a := b || a.c := d (specifically on the same line), which compiles to [a EXCEPT !.a = b !.c = d]. This does not raise any issues with anything I posted above, but is an edge case that we need to account for.

Closing this in favor of the Wiki entry on Modular PlusCal, which should include the result of all the discussion that happened here and offline.


[WiP] Inconsistent context.state exception handling

An Exception handling on missing SATOSA_BASE in context.state[STATE_KEY]["requester"].
Probably also related to https://github.com/IdentityPython/SATOSA/issues/228
If a http request come from an arbitrary link, like
https://disco.example.org/role/idp.ds?entityID=https%3A%2F%2Fproxy.example.org%2FSaml2%2Fmetadata&return=https%3A%2F%2Fexample.org%2FSaml2%2Fdisco

the satosa frontend cannot run SatosaBase._auth_req_callback_func, so it never inizializes state[STATE_KEY] = {"requester": internal_request.requester}.
This PR would introduce a self explanatory error message to user, this permit them to understand the nature of the problem without calling help-desk support.
I admit that this sounds, as it is now, too much hardcoded, I'd like to start a discussion with you to handle in a definitive way this kind of feature.
A good point of discussion could be the possibility to disallow backend's authnRequest if context.state would be inconsistent, I think that this would be a better strategy (also with a human readable message to users that warns on this)
All Submissions

[x] Have you checked to ensure there aren't other open Pull Requests for the same update/change?
[x] Have you added an explanation of what problem you are trying to solve with this PR?
[x] Have you added information on what your changes do and why you chose this as your solution?
[ ] Have you written new tests for your changes?
[x] Does your submission pass tests?
[x] This project follows PEP8 style guide.

Also related to
https://github.com/IdentityPython/SATOSA/issues/228
From that thread there would be some usefull points of discussion I think


What to wait for after calling createHiddenService()

My code does this:
ths.createHiddenService('noxious','1111', true);
myAddress=ths.getOnionAddress('noxious');
An exception is generated on the second line because at that moment tor is doing a 'reload config' as per the node-ths documentation.
What trigger should I be looking for that tor is ready for business?  Should there be a callback function on createHiddenService()?

That's a problem indeed. ths doesn't expose any events yet. I have an idea about how to fix this; gonna work on it.

Fixed in v2.1.4 that I just released. Pushed it to npm as well.

Thanks for the prompt response (and the module)!  Yes, you're on the right track.  Unfortunately, although I'm no longer getting an uncaught exception, when I do:
ths.createHiddenService('noxious','1111', true);
myAddress=ths.getOnionAddress('noxious', function(err, function() {...}));
The callback is never getting executed, however if the service already exists, the callback works properly.  If you want to see exactly what I'm doing, see: https://github.com/mattcollier/noxious/blob/master/nox-main.js#L159

This is weird, because I tried again the async version of the method in a context where the hidden service doesn't exist yet. See this.
Please be sure that the newly created service is saved (either by using parameter applyNow as true or calling ths.saveConfig())


stream errors

Hi, thank you for the Traxsource tagger!
After some testing i've found out that traxsource tagger is responsible for some stream errors in the files.
I've tested the files with 3 different programs.
Traxsource tagger is responsible for a null stream at the begging of the files.
Audio stream is not affected as far as i can tell!

Hello, the issue is with the underlying ID3 library. However the bug is kinda handled in safe manner, so it doesn't affect the actual audio or tags, so we didn't notice until now.
I've been trying to fix the issue and waiting for approval: https://github.com/polyfloyd/rust-id3/pull/56
However in any case it should be fixed within few days in this app (and Discogstagger too).

Hi, @Marekkon5.
Thanks for the heads-up!

Should be fixed in the new release.

Should be fixed in the new release.

It's fixed. 👍
Thank you for the effort!


feat: option to save last position before jump

Checklist

[x] Used only camelCase variable names.
[x] If functionality is added or modified, also made respective changes to the
README.md (the .txt file is auto-generated and does not need to be modified).
[x] Used conventional commits keywords.

Thanks for the PR.
I am wondering, is there a reason to not to save the previous position in the jump list? Because I can see just adding this behavior without a setting.

To be consist with the default behavior of vim's builtin textobjects, I also think it's more reasonable to always save the position.

All right. One Last thing, could you add a comment for the purpose of the m`? It might be a bit looking at it later on otherwise

Added.

Thank you, and thanks very much for the PR itself!


LB integration hoxton

Fixes gh-516 in Hoxton.

Have verified that the SC LoadBalancer opt-in mechanism works OOTB here.


Fixed knot points (in one-dimensional case)

Thanks for the excellent work. I am using the package to estimate intraday volatility profiles (the model is "seasonality coefficient ~ time of the day") and sometimes the model does not pick-up obvious change points like closing/reopening times of the market.
Judging by out-of-sample predictions it is clear that having these breakpoints in a model a priori would improve out-of-sample performance. I am wondering if imposing knot points a priori might be a desired feature for the package? If yes, I would be willing to work on it.

@aickley, thanks for the feedback.  If I understand correctly, I think what you want can be accomplished by pre-processing your data before feeding it into py-earth.  For example, if you have some volatility values in one column and a market open/closed indicator in another column, you could do MARS/regression against a third column that you calculate from those two.  Does that work for your use case, or is there something I'm missing?  Perhaps there are benefits to having the Earth object keep track of this information?
In general, I like the idea of having more control over what kinds of relationships py-earth will use.  I haven't thought a lot about exactly what that looks like, however.  I would appreciate ideas.

@jcrudy, thank you for the answer. I am not sure I understand your suggestion. Now I am regressing a volatility feature on time of the day feature which yields a seasonality profile of the volatility. What I would like to do is to ensure that a new segment starts in the period when market is closed (please see the ).
I need to read up on the method to better understand how having fixed points could fit in there.

@aickley I was suggesting something like the following.  Say your data set is in a pandas dataframe df with your volatility feature in the v column, and you want to make sure some list of knots knots is considered.  You could do something like this (untested, hastily written):
for i, k in enumerate(knots):
    df['v_%d_+' % i] = numpy.maximum(0, df['v'] - k)
    df['v_%d_-' % i] = numpy.maximum(0, k - df['v'])

and then fit your model on the resulting df.  There are other alternative ways of doing it, too.  You're not restricted to using MARS style hinge functions.
Does that example satisfy your use case?  I could think of some reasons why it might not.  I'd like to understand better so I can make sure what you're working on can be addressed.  I suspect you aren't the only one who wants to do what you want to do.

One feature idea might be to pass Earth a list of candidate knot locations for each input variable.  Or even a weighted list.

Closing this for now.


FAQ cyber security / cyber resilience

Niles proposed:
Cybersecurity is a set of policies, processes, procedures and controls that include the use of people, physical infrastructure, hardware and software to protect information exposed to the internet from cyber threats, whereas cyber resilience is meant to ensure continuity of business operations in the event that cybersecurity measures fail.

I think we need to decide if cybersecurity it a subset of cyber resilience or orthogonal to cyber resilience. Personally I think cyber resilience is larger than cybersecurity but that cybersecurity plays a role in cyber resilience (eg better cybersecurity leads to higher cyber resilience and poor cybersecurity leads to low cyber resilience). The definition above implies orthogonal.
I also think there is a tine element that somehow must be accounted for. Eg policies/procedures/automated-response that adapt cybersecurity to changing conditions is more resilient.


Creating UNIX user failed - 0x80070040

Bash on Windows seems to fail for me when creating a UNIX user account. I have tried lxrun /uninstall /full and reinstalling to no avail. Have run in command prompt (Administrator) and otherwise.
C:\WINDOWS\system32>bash
-- Beta feature --
This will install Ubuntu on Windows, distributed by Canonical
and licensed under its terms available here:
https://aka.ms/uowterms

Type "y" to continue: y
Downloading from the Windows Store... 100%
Extracting the filesystem, this will take a few minutes...
Installation successful!
Please create a default UNIX user account. The username does not need to match your Windows username.
For more information visit: https://aka.ms/wslusers
Enter new UNIX username: nhardy
Creating UNIX user failed, this can be done later by running lxrun.exe /setdefaultuser
The environment will start momentarily...
Documentation is available at: https://aka.ms/wsldocs
Error: 0x80070040

C:\WINDOWS\system32>lxrun.exe /setdefaultuser
Please create a default UNIX user account. The username does not need to match your Windows username.
For more information visit: https://aka.ms/wslusers
Enter new UNIX username: nhardy
Error: 0x80070040

This is mysterious, 0x80070040 is an error code that means "network name not available" and I can't figure out what in the create user path could be returning this.  Could you please try a couple of things for me?

After "lxrun.exe /uninstall /full" could you check to see if %localappdata%\lxss exists on your machine?
Reinstall (lxrun.exe /install) and exit out of the window while it's prompting you to create a username.  Then try to launch bash.exe.  Does that return the same error?

%localappdata%\lxss no longer exists after lxrun.exe /uninstall /full.
I get the same error when running bash.exe after reinstalling, with no prompt for creating a UNIX username:

C:\Windows\System32>bash.exe
Error: 0x80070040

@nhardy Thanks - This is a little bit mysterious.  I think I'll need to collect some logs and see what's falling over.  I'm talking with some people internally about how you'll be able to collect the logs we'll need to figure out what's going on.  As soon as I have that information I'll follow up with you.

@nhardy Ok I consulted with some people and figured out the right way to have you send us logging.  If you could do the below it would be extremely helpful to us:

Download the below two .txt files and rename them .cmd
Run start_lxcore_trace.cmd from an admin command prompt
Launch bash.exe
Wait until you see the error 0x80070040 error.
Run stop_lxcore_trace.cmd from an admin command prompt
You should now see three .etl files in the directory you ran the script from.
Email the logs to<EMAIL_ADDRESS> Include in the subject / message to forward to the WSL team.  In the body of the email mention it is for russalex and benhill.  Also kindly include the build number you're running.

If you need a hand with any of the above let me know, I'm very interested in looking at the logs to see what's wrong.  Thanks!
stop_lxcore_trace.txt
start_lxcore_trace.txt

I received yours logs today and took a look, unfortunately there wasn't very much information in them to point to what the issue could be.  A new Windows Insider fast ring build was released today.  Would it be possible to install that build and see if this problem goes away?

Sorry for the delay, only just had a chance to do this today. Did it before and after the upgrade to build 14352 and still experiencing the same error code. Sent the requested .etl files you requested.

No problem at all! I got the traces you sent but turns out my above scripts are missing a provider...  Would it be possible to run the repro again with the new version of the scripts.  Sorry for the inconvenience.
start_lxcore_trace.txt
stop_lxcore_trace.txt
.

Unchecking "Use legacy console" in the Windows command shell properties fixed this issue for me.

@aappleby: This wasn't the case for me. I never had it checked, but tried it with it checked and again after unchecking and my issue hasn't been resolved.
@benhillis: Sent the updated .etl files today.

@nhardy I was able to root cause your issue from the etl files you sent.  We've submitted a fix to our dev branch and it should be an in Insider fast ring build soon.

Same issue for me as @nhardy...I'm excited for the fix...if you need info for my case, I can reproduce the steps above if you like...just let me know
rob

@leictreon Sorry I did not see your comment until just now.  The issue you are seeing is due to the same bug reported in #598.  To confirm, does your Windows username have a " or ' character in it?

Closing this out since the #598 is tracking the remaining issue.

any update on how to fix this i am currently experiencing the same issue, with it not creating the user and just defaulting me to root.

This is fixed in recent insider builds.

@VeritasX -- good question :-)  An Insider build is a public pre-release build of Windows.  It contains all the latest features, including changes that might not be fully implemented yet and that aren't fully tested yet so can cause system instability.
Generally, new Windows features are released to the Insider "Fast" lane every week or two.  Periodically, if a "Fast"-lane build is found to be particularly stable, it is released to the "Slow" Insider lane.  Eventually, those releases turn into Release Candidates and, in theory (though no official commitment) at least once per year, get rolled into the next Windows release.  So if a change makes it into an Insider build, that means both that it's available for testing and that it's on track to eventually ship in an official build.
The Insider track is separate from regular Windows Updates.  Windows Updates are just security fixes and bugfixes to stable features.  (WSL is not yet considered "stable".)
You can learn more, and sign up if you want, here:  https://insider.windows.com


Why is Calibration taking so long to compile on LWA calim

Benchmark performance. Implement sharded and unsharded packaged up calibrator versions.


Add range validators

coverage: 93.42% (+0.1%) from 93.314% when pulling 3420a39dc96ae6226763d7fef5e29b3a2e1c3e68 on feature/add-range-validators into 737bd193633b11e8300288bc774a8d82f31ec65b on main.


Move all	the  negative	elements	to one  side of  the	array IN C LANGUAGE

I want to provide the solution for moving all negative elements to one side. in C language
Sample Input: 5
-1 1 -2 3 4 -5
Sample Output: -1 -2 -5 1 3 4
@Ayush7614  Please assign me this under LGM-SOC'21

I want to provide the solution for moving all negative elements to one side. in Python language.
@Ayush7614  Please assign me I am participants of LGM-SOC'21

I want to contribute the code for moving all negative elements to one side of an array  .
Language - C++
Will try to contribute the optimized code - well commented .
Please assign me this under LGM-Summer of Code'21

please leeme know Y LGM-SOC'21 label was removed


Post create - hook

Hi,
At the moment, there is a hook for the post init which gets called when a document is loaded from the DB.
Is there a possibility to get a hook for newly created documents, which could be used for example to default certain properties of the document based on some configuration data ???
Thanks.

do schema defaults not work for your use case?

Schema defaults are static. Consider the user case where object initialization is driven based on who is logged into the application ( just for argument sake), in that case, you would need to perform this initialization on a per user basis and if there is a hook with callback (like next in the pre-hooks) would be much useful.

@bradynpoulsen, not exactly! the pre-save is called after thing.save() whereas the pre-create is called right after new ThingSchema()

I am looking for something which allows me create a hook after creating an instance of model. Mongoose-lifecycle is only supporting save() post and pre which is not really comfortable for me. Does Mongoose supports such hooks?

Take a look at the schema.queue function in the api docs, that enables you to queue up functions to execute every time a new document is created. Mongoose uses it internally for hooks but 4.1.0 formally added that to the public facing API

I think what the OP was looking for (and I'm looking for it too), is to define a callback function to run after the document is created/inserted, not every time it is saved. So, something like this:
var UserSchema = new Schema ({
  email: {type: String, unique: true, sparse: true},
});

UserSchema.post('create', function(done) {
  var email = new WelcomeEmail({to: this.email});
  email.send(done);
});

@ajsharp the below should work. Unfortunately, it's not really possible to make an async post create hook, because creating a new doc is assumed to be synchronous.
var UserSchema = new Schema ({
  email: {type: String, unique: true, sparse: true},
});

UserSchema.methods.email = function() {
  var email = new WelcomeEmail({to: this.email});
  email.send();
};

UserSchema.queue('email');

According to vkarpov15's final answer here, the queue call should only call the email method once, after the document is created.
I'm experiencing that the queue call is being called every time the document is interacted with, including find() query operations.
Should I open a new issue?

@rmgreenstreet yes please :+1:


User should have his wallets listed after login

The current flow expects the user to import each wallet if he logs in but a better flow will be that the user finds all wallets linked to his seed listed in the wallets section.

now all wallets are linked in the wallets sections after you signup and go to wallets

if he has no wallets yet :

testcase


Docusaurus integration not working

Hi there 👋 I'm trying to build a documentation site with this amazing tool. However upon following the getting started bootstrap from docusaurus and the shiki-twoslash setup for docusaurus, and having the following markdown page:
---
title: Markdown page example
---

# Markdown page example

You don't need React to write simple standalone pages.

```ts twoslash
const a = 123
// ---cut---
const b = 321
``` <!-- I cannot render this example properly without adding this HTML comment -->

My content gets rendered like this:

Have I missed something?

Oh, wait a second... This does not work on a regular page but it does work in a page nested under /docs. Is this the expected behavior?

The same thing is observed here. Works for /docs, /blog but not for regular markdown pages or docs under different routes.

I created an example to illustrate the problem:
https://github.com/felipecrs/shiki-docusaurus-test

I resolved this by specifying the plugin within all three preset-config sections, as well as in the top-level preset section.
import remarkShikiTwoSlash, { Options as RemarkShikiTwoslashOptions } from 'remark-shiki-twoslash'
const remarkShikiOptions: RemarkShikiTwoSlashOptions = {
  themes: ['github-light', 'github-dark'],
  // more shiki options
};

export default {
  presets: [
    [
      'classic',
      {
        docs: {
          // ...
          beforeDefaultRemarkPlugins: [[remarkShikiTwoSlash, remarkShikiOptions]],
        },
        pages: {
          // ...
          beforeDefaultRemarkPlugins: [[remarkShikiTwoSlash, remarkShikiOptions]],
        },
        blog: {
          // ...
          beforeDefaultRemarkPlugins: [[remarkShikiTwoSlash, remarkShikiOptions]],
        },
      }
    ],
    ['docusaurus-preset-shiki-twoslash', remarkShikiOptions]
    // ... rest of config


Update code to the latest AVR code for better stability?

I was wondering where the core code for the AVR code is stored? And how easy is it to update the code to the newest version? My atmega328p keep getting kicked off due to too many bad shares.

The code of the handle the Ducos1a is on the library - https://github.com/ricaun/arduino-DuinoCoin
I believe the problem is not on the library, my i2c implementation has some problems...
Somethimes a slave goes to some BAD loop and stop work, just return BAD jobs...

Good to know. I will keep experimenting to see if it is a configuration issue

Might as well ask, what is the known maximum Arduinos that could be run at 5v with a bi directional data converter? I know the theoretical is 10.

Might as well ask, what is the known maximum Arduinos that could be run at 5v with a bi directional data converter? I know the theoretical is 10.

Good question, the default is 10. Is possible to set like 100 but I don't know if the Esp has the memory for that and I don't know if the Esp has the speed to handle all the slaves with a good speed, probably not.

I was thinking that maximum I would only want to attempt 12 on one esp. I might have to experiment. I would just change it to #define clients 12, correct?

I was thinking that maximum I would only want to attempt 12 on one esp. I might have to experiment. I would just change it to #define clients 12, correct?

Yes only change the #define the code makes the rest, I create the code scalable. 😄

Interesting issues now, when I have more than 6 devices, it is hard for them all to start working together. They don't blink when they are turned on. They are being seen by the esp, and being recognized, but they don't seem to be hashing


netlify app is serving .src.rpm's for 4.6 releases

As the title, downloads arnt working for any 4.6 release, probably due to the format change.

Oops, right, I missed to file sailfishos-chum-gui issue #287 (= sailfishos-chum-gui-installer issue #39 and storeman-installer issue #308) for sailfishos-chum-web (i.e. here).
At least there are working solutions to adapt for sailfishos-chum-web (i.e. to Rust for the "netlify app").

I suspect the culprit in this code section, likely the function repositories(&mut self) -> Response<> in line 161 ff..


Warning: You are running on a deprecated stack.

I now get a warning upon deploy with the following message:

remote:
remote:  !      Please upgrade to the latest stack by following the instructions on:
remote:  !   https://devcenter.heroku.com/articles/upgrading-to-the-latest-stack
remote:
-----> Done

I'm using version 1.0.4 of the plugin.
Configuration
heroku {
    if (project.hasProperty('production')) {
        appName = "myapp"
    } else {
        appName = "myapp-test"
    }

    includes = [new File(project('backend').getBuildDir(), "libs/myapp-0.0.1.jar").absolutePath]
    includeBuildDir = false

    processTypes (
            web: "java -jar -Dspring.profiles.active=\$profile -Dserver.port=\$PORT backend/build/libs/myapp-0.0.1.jar"
    )
}

Sorry, it had nothing to do with this plugin.


Fix: Loop when vector store has many files

Description

We did not consider the has_more=True case in the vector store result when checking if the assistants are in sync
The default pagination result size is 20. The maximum we can set is 100
We now set it to 100 and loop when there are even more.

User Impact

The message saying that there are files at OpenAI but not in OCS should now show correctly.
Demo

N/A
Docs

N/A

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
Additional details and impacted files

:loudspeaker: Thoughts on this report? Let us know!

@coderabbitai Do do you see any issues with this change?

Tip
For best results, initiate chat on the files or code changes.
@SmittieC, I reviewed the changes and have a few observations:

Loop Termination: Ensure that the loop handling the has_more=True case has a proper termination condition. This will prevent potential infinite loops if the vector store API behaves unexpectedly.
Pagination Limit: Setting the pagination size to the maximum of 100 is acceptable. However, consider the performance impact if there are significantly more files. It might be beneficial to implement asynchronous calls or batching mechanisms for better efficiency.
Error Handling: Make sure to handle any possible exceptions or errors from the vector store API, especially during pagination, to enhance the robustness of the application.
Unit Tests: It would be helpful to add unit tests covering scenarios with more than 100 files. This ensures that the new looping logic works correctly and helps prevent regressions in the future.

Overall, the changes seem to address the issue effectively. With these considerations, the solution will be more robust and maintainable.

Loop termination is there
The max I can set the limit is 100.
I am simulating pagination in the test by returning 1 file on the first call, and another on the second. The has_more=True in the mocked response simulates this

@coderabbitai resolve all


Fails with LuaJIT2 on AArch64

Noticed while trying to build AwesomeWM: lgi does not seem to work with LuaJIT on AARCH64.
In particular, with LuaJIT 2.1.0-beta3 (the only version packaged in Gentoo), I get the following error running the gtkhello.lua example.
jakob@Lambda ~/lgi/samples $ luajit-2.1.0-beta3 gtkhello.lua
luajit-2.1.0-beta3: bad light userdata pointer
stack traceback:
	[C]: at 0xffff94d66ed4
	[C]: in function 'require'
	/usr/share/lua/5.1/lgi/core.lua:14: in main chunk
	[C]: in function 'require'
	/usr/share/lua/5.1/lgi/init.lua:19: in main chunk
	[C]: at 0xffff9502ff80
	[C]: in function 'require'
	gtkhello.lua:9: in main chunk
	[C]: at 0xaaaab5602620

This may be relevant: https://github.com/neovim/neovim/issues/7879

https://github.com/LuaJIT/LuaJIT/blob/4deb5a1588ed53c0c578a343519b5ede59f3d928/src/lj_errmsg.h#L58
https://github.com/LuaJIT/LuaJIT/blob/e957737650e060d5bf1c2909b741cc3dffe073ac/src/lj_udata.c#L51
https://github.com/LuaJIT/LuaJIT/blob/3fbf51bb99e3795497c010e286cb749094da6bc2/src/lj_obj.h#L837-L838
https://github.com/LuaJIT/LuaJIT/blob/3fbf51bb99e3795497c010e286cb749094da6bc2/src/lj_obj.h#L292-L294
Looks like LJ_64 means that LuaJIT only supports lightuserdata with 39 bits and you are getting a pointer with more bits set.
Or alternatively, there is a limit on the number of light user data objects and the code exceeds this limit?
But I am not really sure since there is too much magic going on.
Personally, I don't have an AArch64 machine to debug on / to figure out where this pointer comes from and what it does. So :shrug: . Sorry.

No worries. I'll set aside some time this week to do some debugging, see if I can make any progress.


Changes the prerequisites part

Please resolve conflicts. To prevent conflicts from happening, please always fetch + pull before pushing.


Immersive Portals makes the Rat Pouch from Rat's Mischief break

It causes rats to lose their owners, their skin, and their names, as well as deleting them if you Shift + Right Click on the rat with the back, which would normally pick up the rat just fine.
I'm not sure if this is something that needs to be fixed on your end or Rat's end, so I've reported it to both.

post the log

Reason I didn't was because there's nothing in the log, but here it is:
https://pastebin.com/1WqjJABb

Adding to this.
It doesn't just break the pouch.  There is no log detailing why this happens; I searched.
It's definitely an issue between these two mods.
https://github.com/Ladysnake/Rats-Mischief/issues/14

@IdrisQe How did you "unload"? Did you go through custom portal api's portal? Or vanilla nether portal? Or going away and come back?

All I did was unload the area by (at first) saving and quitting, (and then to confirm the issue) travelling far away - both ways of unloading the chunks seem to trigger the issue. There was no changing of dimensions (in fact I don't even have a Nether Portal set up yet)
Honestly I'm still not entirely convinced it's solely an issue between these two mods given it's a pretty strange thing to happen... Mainly added the info here since it only seemed to start once I added Immersive Portals to my pack, and I was directed toward it being an incompatability with Immersive Portals that was previously documented.
It's too late at night (well technically too early in the morning) to do so now, but I'll probably set up a testing instance with just the two (and dependencies) to make sure it's not something involving more mods once I wake up.

I made a test instance with minimal setup and no config changes for either mod.
Confirmed the issue with only:
Fabric Loader 0.11.3, Fabric API 0.320, GeckoLib 3.0.23, Rats Mischief 1.0.4, Immersive Portals 0.80
Steps to reproduce:
-Add all mods except Immersive Portals
-Start game
-Create world
-[Possibly Important?]  Rat is still the same, and still able to be commanded
-Exit game
-Add Immersive Portals
-Start game
-Load world. Rat color is randomized, and rat can no longer be commanded.
-Exit world
-Load world. Rat color is randomized again, and remains ownerless.
Checked with each world load; no log anomalies to report.

probably fixed in 0.84 https://github.com/qouteall/ImmersivePortalsMod/releases/tag/v0.84-1.16

I'll check once I download it!

Having done some minimal testing, it seems to have fixed the issue!
Interestingly, it seems to have retroactively fixed the issue too-
My rat, which with the issue, was not appearing tamed, now seems to function as tamed again despite having not touched it, and has reverted to its original (I think?) colour.


Increase Community Profile Score And Add README.md Shields

The Benefits Of This Pull Request Is:

Increases The Community Profile Score From 50% To At Least 75%
More Shields On ReadMe.md:

bloat ..

Appreciate this, but it doesn’t seem to add significant value.


Added QA Quiz Question

Checklist:

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)

Adds to #102

The added question covers regression tests.

Please add the "hacktoberfest-accepted" label before merging this, as I've contributed this under hacktoberfest 2022.


White window on raspberry pi (raspbian)

Hello.
Issue:
When I do the npm install and npm start on latest raspbian (nov 2018), I only get white window. The youtube app on android does detect "mMusicCast" TV.
Background:
I tested mMusicCast on my ubuntu 18.04 and it does work (after apt get install libcfong-2-4). I can play videos from my android via youtube app.
Today I installed latest raspbian on raspberry pi 2 B.
After installing npm (sudo apt install npm) I had to update npm via npm (sudo npm install -g npm) I was able to install mMusicCast (npm install mmusiccast).
After staring, the mMusicCast window appears but it only displays  white content. The android youtube app does detect it, but it never connects.
pi@raspberrypi:~/git/mMusicCast $ npm start

><EMAIL_ADDRESS>start /home/pi/git/mMusicCast
> electron .

morgan deprecated undefined format: specify a format server.js:36:9
morgan deprecated default format: use combined format server.js:36:9
body-parser deprecated bodyParser: use individual json/urlencoded middlewares server.js:37:9
body-parser deprecated undefined extended: provide extended option node_modules/body-parser/index.js:105:29
<IP_ADDRESS>
Registered App: ChromeCast
Registered App: YouTube
Express server listening on port 8008
::ffff:<IP_ADDRESS> - - [Fri, 08 Mar 2019 11:46:52 GMT] "GET /ssdp/device-desc.xml HTTP/1.1" 200 1079 "-" "com.google.android.youtube/13.48.51(Linux; U; Android 8.1.0; sl_SI; Nokia 6.1 Build/OPR1.170623.026) gzip"
::ffff:<IP_ADDRESS> - - [Fri, 08 Mar 2019 11:46:52 GMT] "GET /apps/YouTube HTTP/1.1" 200 531 "-" "com.google.android.youtube/13.48.51(Linux; U; Android 8.1.0; sl_SI; Nokia 6.1 Build/OPR1.170623.026) gzip"
::ffff:<IP_ADDRESS> - - [Fri, 08 Mar 2019 11:47:01 GMT] "GET /apps/YouTube HTTP/1.1" 200 531 "-" "com.google.android.youtube/13.48.51(Linux; U; Android 8.1.0; sl_SI; Nokia 6.1 Build/OPR1.170623.026) gzip"
express deprecated res.send(status, body): Use res.status(status).send(body) instead apps/app.js:159:7
::ffff:<IP_ADDRESS> - - [Fri, 08 Mar 2019 11:47:01 GMT] "POST /apps/YouTube HTTP/1.1" 201 0 "-" "com.google.android.youtube/13.48.51(Linux; U; Android 8.1.0; sl_SI; Nokia 6.1 Build/OPR1.170623.026) gzip"

What I am missing?
Since project states raspberry is supported, can someone write gude?

The problem you have seems to be related either to electron or the graphics hardware acceleration.


New statistical function: TREND

Hi Tycho,
I'm completely new to GitHub, so please bear with me. Actually: Do you know if there is a beginners guide for non-techies somewhere? Even the official instructions here sound chinese to me unfortunately.
I'm trying to contribute an M-function for descriptive statistics with a blogpost coming soon, hoping that many other people will create and share statistical functions for M. So I would like to link to your repository - and ideally give non-techies who can just about code M like me an easy start here :-)

Hey,
No worries, I guess use of Github for Power Query is still pretty young in general. :)
Here's a guide on Git itself, but ideally with Github's GUI you may even get away not bothering with those internals. Presuming you use Windows, had you tried Github's desktop client? I felt like it simplified things a bit for me.
If you've managed a pull request I think you may have had most of the hard parts already though. I recall without the desktop client the keys part was annoying...
Also, thanks for contributing!
I like how your function has simplified usage in terms of parameters compared to Excel's implementation.
In the same way, you might even be able to have it work with just the YList, letting NoOfIntervals default to 0 or something. Unfortunately, M has not natively supported default parameter values, though you could achieve the same thing in a slightly more verbose way:

marking the parameter with optional
as a first statement to the function, assigning the default if applicable: NoOfIntervals = if (NoOfIntervals=null) then 0 else NoOfIntervals,

Anyway, I'll merge it either way. :)

Thanks!
Good tip re the optionals! But in this case I think it doesn't make sense to default to 0 because it wouldn't return anything then.
Had a closer look yesterday into GitHub and "spoke" with Ivan Bond and after that wasn't sure if sending a pull request was actually the right thing to do here. As he is just hosting a branch.
I'm going to write a blogpost about creating (descriptive) statistical functions in M including the call out to the community to rebuild Excel's statistical functions (or more) in M. If this generates response - would you be happy to pull them all into your repository or shall I give my branch as the desired target instead? I would prefer to use your's directly but as I see now that this could generate some work... Let me know please :-)

I'd be glad to merge any contributions, sure. Thanks for attending me to Ivan's branch; I hadn't quite noticed his additions so far as I hadn't received a pull request from him yet so far.
As to pull requests, for our own branches each of us can choose to merge in whichever of each other's commits as we like (assuming merge compatibility -- if not it'll start asking questions).
In that sense, even if you'd originally forked from mine, Ivan should also still be able to merge in and use your additions, and vice versa.
I imagine creating pull requests from your branch to his directly should also be possible, but I'm not sure how yet.
Most of his additions I don't mind merging in either, but presuming he'd be in favor as well, I should probably talk to him about his decision to move the hard-coded file path from LoadPath.m to Load.m. I'll mention you there so you'd get a notification on it as well.

Update: discussion here.


Why does test include a leading space?

https://github.com/tomv564/pyls-mypy/blob/93be6ac6bb6ee0b781f9754272a13134b3b3192a/test/test_plugin.py#L15

Fixed in https://github.com/tomv564/pyls-mypy/commit/8c81ffb7ed9ce23979c66b5fb0790265fd504eb2


luci-lib-*: don't depend on luci-base by default

Not all libraries do require luci-base (e.g. luci-lib-nixio). Make those
that do depend on it directly instead.
On images using functionality from nixio only this reduces image size
considerably.
Signed-off-by: Nils Schneider<EMAIL_ADDRESS>

Should be fixed now.


More SQL functions

Add implementations for the following functions:
http://dev.mysql.com/doc/refman/5.7/en/string-functions.html
http://dev.mysql.com/doc/refman/5.7/en/numeric-functions.html

Fixed.


add option to run only one function

also rename shortcut to pytest repeat to "r"

Thanks for your help!


Add Widget.remove_edge

Not sure about the naming, but it could be included as it makes easier for user to just put <widget>.remove_edge() in kv without making additional function to care about error handling with remove_widget(). Kind of like list.pop().

I don't think I've ever needed to use something like that, especially not
from kv. Why would you to need remove the last or first widget? Even more,
if you do need to remove it you should know if it exists so you wouldn't
need the exception catching at which point you don't need the function
since its a one liner.
On Sep 22, 2016 1:26 AM, "Peter Badida"<EMAIL_ADDRESS>wrote:
Not sure about the naming, but it could be included as it makes easier for
user to just put .remove_edge() in kv without making additional
function to care about error handling with remove_widget(). Kind of like
list.pop().
You can view, comment on, or merge this pull request online at:
https://github.com/kivy/kivy/pull/4629
Commit Summary

Add Widget.remove_edge

File Changes

M kivy/uix/widget.py
https://github.com/kivy/kivy/pull/4629/files#diff-0 (22)

Patch Links:

https://github.com/kivy/kivy/pull/4629.patch
https://github.com/kivy/kivy/pull/4629.diff

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/kivy/kivy/pull/4629, or mute the thread
https://github.com/notifications/unsubscribe-auth/ABkW_iblBHZWmp8DskCr-osrc_Qy3lJTks5qshGKgaJpZM4KDid6
.

I was thinking about some button with this in on_release to remove a recently added child. I forgot about the check :disappointed: this makes the pr irrelevant, I guess, as the check can be performed in kv. Only pro of the pr now seems to be shorter line of code if kv: if parent.children: parent.remove_widget(parent.children[0])


Update BMC interactions in e2e tests:

Issue #, if available:
Description of changes:
Remove unused functions. Log on power off errors instead of failing the test that calls the function. Refactor ValidateHardwareDecommissioned for improved clarity.
Testing (if applicable):
Documentation added/planned (if applicable):
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/approve

going to open a new PR to see if codecov will pick up the new base commit.


Make the library compatible with pre-installed libraries on Glue 4.0

Is your idea related to a problem? Please describe.
Glue 4.0 has the following pre-installed:
numpy==1.23.4
pyarrow==10.0.0
The following PRs changed requirements for the latest version of awswrangler:
PR #1731 updated the dependencies to allow to satisfy the pyarrow requirement
PR #1831  updated the dependencies that prevent to satisfy the numpy requirement
Describe the solution you'd like
Fix the dependencies, so awswrangler can be installed with minimal changes (ideally no downgrades / upgrades of pre-installed packages) on glue 4.0.

Thanks for opening this @FredStober, I think it should be <=1.23.4:
numpy = [
    { version = ">=1.21.0 <=1.23.4", markers = "python_full_version >= '3.7.1' and python_full_version < '3.11.0'" },
    { version = "^1.23.4", markers = "python_full_version >= '3.11.0'" },
]

Will open a PR with a fix

@kukushking
I'm not sure if https://github.com/aws/aws-sdk-pandas/pull/1831 would fix this for us.
On python 3.9.
    our-package-with-50-other-pinned-deps-and-pinned-numpy x.x.x depends on numpy==1.23.5
    awswrangler 2.18.0 depends on numpy<1.23.4 and >=1.21.0; python_full_version >= "3.7.1" and python_full_version < "3.11.0"

An ideal situation for us at least would be something like one of these:
# The ultimate ideal case, users can pin on their end
numpy = [
    { version = ">=1.21.0", markers = "python_full_version >= '3.7.1' and python_full_version < '3.11.0'" },
    { version = ">=1.23.4", markers = "python_full_version >= '3.11.0'" },
]

# A reasonably high value like 2.0.0 or a few minor versions out.

numpy = [
    { version = ">=1.21.0,<2.0.0", markers = "python_full_version >= '3.7.1' and python_full_version < '3.11.0'" },
   # This would be ideal
    { version = ">=1.23.4,<2.0.0", markers = "python_full_version >= '3.11.0'" },
]

# Maybe the next "minor-ish" version since numpy sometimes introduces deprecations in minors, but they are really unlikely to break anything used via wrangler.
# At least no patch version in an upper bound please :) then we can't get bug fixes downstream
# But this will cause an issue when 1.25.0 comes out, and we'll have to wait until the next release cycle of wrangler to upgrade numpy or do weird tricks to install it separately.

numpy = [
    { version = ">=1.21.0 <1.24.0", markers = "python_full_version >= '3.7.1' and python_full_version < '3.11.0'" },
    { version = ">=1.23.4,<1.24.0", markers = "python_full_version >= '3.11.0'" },
]

This might be a separate/bigger issue to discuss. Wrangler seems to be the one dependency causing grief when updating packages due to pretty restrictive upper bounds. I understand restricting upper bounds somewhat given the general user base might not be pinning deps or have a test suite to catch breakage.
If the upper bounds could be a little less restrictive in general, that would save us a lot of back and fourth. I can open that separately if desired.


Better v-on '@' shorthand match

Support v-on shorthand with event handlers containing colon (e.g. @hook:mounted="onChildMounted")
I'm not 100% certain, but I think the matching should be the same as for regular v-* at line 663. Potentially fixes #203. Tested locally for that scenario, but I'm not sure if there are other directives which might be affected by this.

I'm not sure about the change itself but you haven't followed the development instructions. See https://github.com/vuejs/vue-syntax-highlight/tree/new#development

Oops—missed that. Sorry :).

Should be fixed/handled by #216.


Jasync ZIO

Implements #149
Problem
N/A
Solution
N/A
Notes
Moved old quill modules to protoquill
Checklist

[ ] Unit test all changes
[ ] Update README.md if applicable
[ ] Add [WIP] to the pull request title if it's work in progress
[ ] Squash commits that aren't meaningful changes

JAsync support has been dropped