bug in pthread_mutex_init

gcc warning the comparison will always evaluate as 'false' for the address of 'xMutex' will never be NULL [-Waddress]
/* Call the correct FreeRTOS mutex creation function based on mutex type. */
if( pxMutex->xAttr.iType == PTHREAD_MUTEX_RECURSIVE )
{
    /* Recursive mutex. */
    ( void ) xSemaphoreCreateRecursiveMutexStatic( &pxMutex->xMutex );
}
else
{
    /* All other mutex types. */
    ( void ) xSemaphoreCreateMutexStatic( &pxMutex->xMutex );
}
/* Ensure that the FreeRTOS mutex was successfully created. */
if( ( SemaphoreHandle_t ) &pxMutex->xMutex == NULL )
{
    /* Failed to create mutex. Set error EAGAIN and free mutex object. */
    iStatus = EAGAIN;
    vPortFree( pxMutex );
}
else
{
    /* Mutex successfully created. */
    pxMutex->xIsInitialized = pdTRUE;
}

&pxMutex->xMutex will never be NULL because pxMutex is not NULL (this is checked before).
According to FreeRTOS Reference, xSemaphoreCreateMutexStatic and xSemaphoreCreateMutexStatic will return NULL only when their argument pxMutexBuffer is NULL. So, just delete this unused if statement.
/* Call the correct FreeRTOS mutex creation function based on mutex type. */
if( pxMutex->xAttr.iType == PTHREAD_MUTEX_RECURSIVE )
{
    /* Recursive mutex. */
    ( void ) xSemaphoreCreateRecursiveMutexStatic( &pxMutex->xMutex );
}
else
{
    /* All other mutex types. */
    ( void ) xSemaphoreCreateMutexStatic( &pxMutex->xMutex );
}
/* Mutex successfully created. */
pxMutex->xIsInitialized = pdTRUE;

Thank you for pointing that. Would you like to raise a PR to fix it or would you want us to raise one?

I'm new in github. You could raise a PR.  :)

Thank you for reporting this. I have fixed it in this PR - https://github.com/FreeRTOS/Lab-Project-FreeRTOS-POSIX/pull/31.


CRB-135 Remove Deprecation Warnings

This wraps the Source Address' ScriptPubKey into a TxOut object to match NBitcoin's updated API.

Wrong source branch.


Input group with submit button not rendering properly in Chrome & Safari

When using an button with type=submit inside an InputGroup, in Chrome and Safari this does not render as expected and the button wraps into a new line

@jordaneb screenshot? / whip up an example on a new branch.

Ah, this is a known issue. Though I can't find a specific issues for it now...
This is a problem with rendering any button component inside an input group. They don't correctly accept the display: table-cell; property.
Definitely needs fixing somehow.


Thanks MegaByte!! :) for minions :D

Yh it took  a while for us (mainly Megabyte) to make it :D

Yes :)))) is there any way to donate something? : D

Ask @Megabyte918 he should kno

@Jhair2015 No problem man :), and I don't want anyone to donate to me, I added these minions just because it seemed like it'd be a fun project to do, and it was =D

Thaks :)


[GEOPM - Tutorials] Issues while running the tutorials

Hello,
I am creating this issue concerning the tutorials because I failed to make them work. I will try to be thorough, so this issue might be a little long.
Some context first:

I pulled the github repository and followed the README to compile GEOPM and the tutorials from the sources. Except the fact that I had to modify the file configure.ac to delete the -Werror compilation flag, everything went well (in test/fortran/attrlangc.c, some deprecated MPI functions are called, among which MPI_Keyval_create. I checked the reference guide of MPI, it is deprecated since MPI 2.0);

The cluster I am working with has heterogeneous computes nodes. All of them have two sockets, embedding two identical processors from the Haswell family (either two Intel Xeon E5-2670 v3 or two Intel Xeon E5-2650 v3). The interconnect is an InfiniBand EDR fabric, and the nodes are running RHEL7.4. The intel_pstate CPU frequency driver is disabled, the cpufreq governor is set to performance. The MSR driver is loaded, and the command geopmpolicy -w is working on the nodes I tried to run the tutorials onto (by the way, my user account on the nodes has root privileges).

What I tried:

I tried to use tutorials/tutorial_0.sh to run the first tutorial thanks to geopmsrun (two nodes and ten tasks). Since the nodes are not homogeneous, I slightly modified the content of tutorials/tutorial_0.sh in order to give a nodelist of homogeneous nodes, namely mexico48 and mexico49.

geopmsrun  -w mexico[48,49] \
           -N 2 \
           -n 8 \
           --geopm-preload \
           --geopm-ctl=process \
           --geopm-report=tutorial_0_report \
           --geopm-trace=tutorial_0_trace \
           -- ./tutorial_0

And below is the output I am getting:
<geopmpy.launcher> list index out of range
I have looked into the source code of geopmpy.launcher, and it seems I have specified the nodelist as it should be. Any mistake ?

Since the "simple" way seemed to be a dead-end, I tried to use MPIEXEC, almost just as @cmcantalupo did in the tutorial videos.

user@mexico48 $ export GEOPM_RM=NONE
user@mexico48 $ export MPIEXEC="srun --reservation=Power-GEOPM --exclusive --cpu_bind=v,mask_cpu:0x1,0x2,0x4,0x8,0x10 -N 2 -n 10"
user@mexico48 $ ./tutorial_0.sh
        cpu_bind=MASK - mexico48, task  1  1 [648]: mask 0x2 set
        cpu_bind=MASK - mexico48, task  4  4 [652]: mask 0x10 set
        cpu_bind=MASK - mexico48, task  0  0 [645]: mask 0x1 set
        cpu_bind=MASK - mexico48, task  2  2 [650]: mask 0x4 set
        cpu_bind=MASK - mexico48, task  3  3 [651]: mask 0x8 set
        cpu_bind=MASK - mexico49, task  8  3 [35418]: mask 0x8 set
        cpu_bind=MASK - mexico49, task  6  1 [35416]: mask 0x2 set
        cpu_bind=MASK - mexico49, task  7  2 [35417]: mask 0x4 set
        cpu_bind=MASK - mexico49, task  5  0 [35415]: mask 0x1 set
        cpu_bind=MASK - mexico49, task  9  4 [35419]: mask 0x10 set
        [mexico48:4059] *** An error occurred in MPI_Comm_rank
        [mexico48:4059] *** reported by process [140303995502592,2]
        [mexico48:4059] *** on communicator MPI_COMM_WORLD
        [mexico48:4059] *** MPI_ERR_COMM: invalid communicator
        [mexico48:4059] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
        [mexico48:4059] ***    and potentially your MPI job)
        slurmstepd: *** STEP 4560.0 ON mexico48 CANCELLED AT 2017-10-25T19:28:20 ***
        srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
        srun: error: mexico48: tasks 0-1,3-4: Killed
        srun: error: mexico48: task 2: Exited with exit code 5
        srun: error: mexico49: tasks 5-9: Killed

On top of that, the report file is not created, and the trace file is created but empty. On both compute nodes.

To be "sure" the problem does not stem from the version of openmpi I am using, I tried to compile the tutorial without involving GEOPM, and I ran the tutorial. Everything works just fine.

Any idea ?
Thanks in advance,
--Mathieu.

Thank you for the detailed feedback!  The issue of why the launcher failed I know:  we parse the node list option and our current parser assumes a comma separated list (it doesn't understand the bracket notation), so "-w mexico48,mexico49" will work but "-w mexico[48,49]" does not.  We will try to fix this issue soon.
The invalid communicator error I'm not so sure what is going wrong.  Have you configured geopm with the --enable-debug option?  If not, can you recompile with this option and report back if any error messages re emitted?
Thanks,
Chris

Thank you for your anwser Christopher.
To begin with, I tried to specify the nodes without the bracket notation, and the output remains the same, namely <geopmpy.launcher> list index out of range.
I have tried to just specify the nodes to the launcher, and it seems that the argument processing is correct:
user@mexico48 $ geopmsrun -w mexico[48,49]
        srun -w mexico[48,49]
        srun: fatal: No command given to execute.

user@mexico48 $ geopmsrun -w mexico48,mexico49
        srun -w mexico48,mexico49
        srun: fatal: No command given to execute.

However, the nodelist is not completely "enforced". In fact, I checked if Slurm job had been executed and it appears the lscpu command has been executed on mexico51, before the error message is printed:
user@mexico48 $ scontrol show job
        JobId=4569 JobName=lscpu
           UserId=root(0) GroupId=root(0)
           Priority=4294901666 Nice=0 Account=root QOS=normal
           JobState=COMPLETED Reason=None Dependency=(null)
           Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0
           RunTime=00:00:00 TimeLimit=UNLIMITED TimeMin=N/A
           SubmitTime=2017-10-26T11:20:59 EligibleTime=2017-10-26T11:20:59
           StartTime=2017-10-26T11:20:59 EndTime=2017-10-26T11:20:59
           PreemptTime=None SuspendTime=None SecsPreSuspend=0
           Partition=All AllocNode:Sid=mexico48:13104
           ReqNodeList=(null) ExcNodeList=(null)
           NodeList=mexico51
           BatchHost=mexico51

Then, I recompiled GEOPM and the tutorials with the --enable-debug option, but I get the same output as before, no additional error message or trace. Is there a way to check the option GEOPM was compiled with, just to be sure the --enable-debug option has been taken into account ? I "parsed" the output of the compilation and the -O0 -g flags are used, so it seems to be compiled with the debug  option activated.
Thank you in advance,
--Mathieu.

Hi Mathieu,
I've started digging into this a bit, and was wondering if you could answer the following for me:
What version of SLURM are you using currently?
What version of GEOPM are you using currently?  The output from "git describe" would be great.
Is POSIX shared memory enabled on your system?
There's a patch Chris has up for review now that you could cherrypick to remove the Fortran tests.  That should avoid the compilation issue you were hitting.  That's here:
https://review.gerrithub.io/#/c/384000/
I was just testing geopmsrun on our cluster, and it appears to be honoring the -w parameter for me.  I'm trying to see if I can reproduce what you were hitting.  I'll post back when I get some more information.  I'm not seeing the "list index out of range" error yet.
The issue with lscpu that you saw is that the particular section of code that issues the command does not enforce the nodelist you've specified.  There's a patch Chris just wrote that's up for review that should address that.  It's here:
https://review.gerrithub.io/#/c/384140/
The peculiar MPI error that was shown when you tried to use MPIEXEC directly is curious.  I'm wondering if there's something misconfigured with MPI that's resulting in that error.  I think it would be illuminating if you tried to run our unit tests and post the output back here.  They will expose any MPI issues that may be present in the system when they run.  To do that simply run "make check" from the root of your checkout.
Good luck,
Brad

A simple way to determine if debug is enabled is the following command:
$ grep enable_debug config.log
enable_debug='1'

enable_debug should be set to '1'.  That being said, if the code is compiling with -O0 -g then debug is enabled.  I had forgotten that in order to get a stack trace from the python script with errors in python you must also
$ export GEOPM_DEBUG=true

If you do this python errors will emit the stack trace were the "list index out of range" error occurred in the python tools.  I'd be very interested to see where this is happening.
With the patch to pass the node list to the lscpu call, there is a warning emitted by slurm because we only launch lscpu on one node, but the node list given may have more than one node.  Warnings aside, with the patch to add the node list to the lscpu command seems to properly restrict the node use.
The issue with the brace notation verses the comma separated list is actually not an problem when the underlying job launcher is slurm so the
-w mexico[48,49]

option should actually work fine.  We only try to parse this option as a comma separated list for the Intel MPI launcher.
Hope that helps,
Chris

Thank both of you for your answers.
I will try to be as thorough as possible:

I confirm that the debug option of GEOPM is enabled.

The version of Slurm currently installed on our development cluster is 15.08.12:

user@mexico48 $ sinfo --version
    slurm 15.08.12

The version of GEOPM being currently used is 0.3.0. I cherry-picked the two commits Brad mentioned before executing the command. I can undo it and redo the git describe, if needed:

user@mexico48 $ git describe
    v0.3.0-88-g9d77cb2

I think POSIX shared memory is enabled since a tmpfs is mounted on /dev/shm. On top of that, 0% of the allocated 32 GB is used. Another way to check ?

user@mexico48 $ mount | grep shm
    tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)

I tried to re-run the launching script for the tutorial 0 for srun, just adding -w mexico48,mexico49. To be thorough, I tried running it while mexico[48,49] were reserved, and while they were idle (I am speaking of Slurm node states). Both time, the same error message:

user@mexico48 $ export GEOPM_DEBUG=true
user@mexico48 $ ./tutorial_0.sh
    srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
    srun: error: Unable to allocate resources: Requested nodes not in this partition
    Traceback (most recent call last):
      File "/scratch/stoffelm/geopm/build/bin/geopmsrun", line 38, in <module>
        err = geopmpy.launcher.main()
      File "/scratch/stoffelm/geopm/build/lib/python2.7/site-packages/geopmpy/launcher.py", line 1230, in main
        launcher = factory(sys.argv)
      File "/scratch/stoffelm/geopm/build/lib/python2.7/site-packages/geopmpy/launcher.py", line 115, in factory
        time_limit, job_name, node_list, host_file)
      File "/scratch/stoffelm/geopm/build/lib/python2.7/site-packages/geopmpy/launcher.py", line 776, in __init__
        time_limit, job_name, node_list, host_file)
      File "/scratch/stoffelm/geopm/build/lib/python2.7/site-packages/geopmpy/launcher.py", line 462, in __init__
        self.init_topo()
      File "/scratch/stoffelm/geopm/build/lib/python2.7/site-packages/geopmpy/launcher.py", line 568, in init_topo
        launcher.run(stdout=ostream)
      File "/scratch/stoffelm/geopm/build/lib/python2.7/site-packages/geopmpy/launcher.py", line 533, in run
        raise subprocess.CalledProcessError(pid.returncode, argv_mod)
    subprocess.CalledProcessError: Command 'srun -N 1 -n 1 -w mexico48,mexico49 lscpu' returned non-zero exit status 1

As Chris said, there is a warning message. The error message mentions partitions, that is why I tried to run the script while the nodes were in different states according to Slurm. The nodes are in the same partition, named All, which is up.

I tried to re-compile the code after cherry-picking the purge of the Fortran tests, no problem at all. Thank you for the fix.

I executed the make check command, and 6 tests out of the 176 failed, producing the same error message (content of the logs related to the failed tests in gtests_link, and printed on the terminal):

Failed tests:
    FAIL: test/gtest_links/MPITreeCommunicatorTest.hello
    FAIL: test/gtest_links/MPITreeCommunicatorTest.send_policy_down
    FAIL: test/gtest_links/MPITreeCommunicatorTest.send_sample_up
    FAIL: test/gtest_links/MPIProfileTest.noctl
    FAIL: test/gtest_links/MPIInterfaceTest.geopm_api
    FAIL: test/gtest_links/MPIInterfaceTest.mpi_api

Error message:
    --------------------------------------------------------------------------
    mpiexec has detected an attempt to run as root.
    Running at root is *strongly* discouraged as any mistake (e.g., in
    defining TMPDIR) or bug can result in catastrophic damage to the OS
    file system, leaving your system in an unusable state.

    You can override this protection by adding the --allow-run-as-root
    option to your cmd line. However, we reiterate our strong advice
    against doing so - please do so at your own risk.
    --------------------------------------------------------------------------

Is there an easy way to specify the --allow-run-as-root option to all the tests, in order to see if there is another problem concerning MPI ? Or another workaround ?
Thank you in advance for your answers,
--Mathieu.

My apologies for the tardiness in replying here.  SC kept everyone very busy this month.  Replies below:

The version of Slurm currently installed on our development cluster is 15.08.12:
user@mexico48 $ sinfo --version
slurm 15.08.12

We've been testing internally with v15.08.7 so this should be fine.

The version of GEOPM being currently used is 0.3.0. I cherry-picked the two commits Brad mentioned before executing the command. I can undo it and redo the git describe, if needed:
user@mexico48 $ git describe
v0.3.0-88-g9d77cb2

Your version looks good.  Those commits you needed have since been merged into the dev branch which is available here: dev branch

I think POSIX shared memory is enabled since a tmpfs is mounted on /dev/shm.

This appears correct to me.

I tried to re-run the launching script for the tutorial 0 for srun, just adding -w mexico48,mexico49. To be thorough, I tried running it while mexico[48,49] were reserved, and while they were idle (I am speaking of Slurm node states). Both time, the same error message:
user@mexico48 $ export GEOPM_DEBUG=true
user@mexico48 $ ./tutorial_0.sh
srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1
srun: error: Unable to allocate resources: Requested nodes not in this partition
...
As Chris said, there is a warning message. The error message mentions partitions, that is why I tried to run the script while the nodes were in different states according to Slurm. The nodes are in the same partition, named All, which is up.

I believe the problem here is that you ask for specific nodes with -w, but while those nodes are in the "All" partition all is not the default partition.  Have you tried using the -p option with -w in this case?  Like:
srun -p <PARTITION_NAME> -w <NODE_LIST> ...

I executed the make check command, and 6 tests out of the 176 failed

The failures you have listed are all expected if MPI is not working properly.  While you could modify the launcher.py to include the allow-run-as-root flag, this could be dangerous.  If the cluster you're operating on is yours, and you're not worried about screwing things up as root which would require you to re-image the system / start from scratch then this is not a big concern.  If on the other hand you share this system with many other people and you're not willing to risk screwing up the machine for a few hours or days, then I'd recommend against running MPI as root.
Instead I'd start down the patch of installing the msr-safe kernel driver on your cluster.  The whole problem here is that in order to read/write from the MSR device files, you need to be root.  The msr-safe project allows privileged users read and write access to a whitelist of MSRs and bitfields within those MSRs for very fine grained control.  We've been using it on all the systems where GEOPM research is performed and it is our preferred setup.  Here's that project:
msr-safe
If you need assistance setting up msr-safe let me know.  There's a script to build an RPM of the driver here:
https://github.com/LLNL/msr-safe/blob/master/rpm/build_rpm.sh
You should be able to use that to build the RPM on the system in question.  After that, installing the RPM should setup the msr user, move the kernel object out to the right spot, and setup the initial whitelist.  When the driver is loaded, the /dev/cpu/*/msr_safe device files should be available which GEOPM looks for.
I hope this helps!  Let me know if you hit anymore roadblocks.
Brad

Hello Brad,
First of all, thank you for your answer. I hope everything went well at SC17.
 I had a reservation on the nodes I tried to launch the job onto, thus they were not available (since the reservation is not yet taken into account by geopmsrun).
However, before I realized that, I tried what you suggested. It failed with the following output:
<geopmpy.launcher> Command 'srun -N 1 -n 1 -w mexico48,mexico49 lscpu' returned non-zero exit status 1

Since there is no mention of the -p All part, I am not sure whether it is taken into account by geopmsrun or not.
Concerning the "msr-safe" part, I am still facing roadblocks. In a few key points:

I cloned the msr-safe repository, compiled and installed all the components on the compute nodes, following the README embedded in the concerned repository;

I created the group msr and added my user account and the slurm account to this group;

I checked that the generated whitelist corresponds to the CPU model embedded in the compute nodes (Haswell - 0x063f);

I wrote a really simple (almost dumb) code snippet to verify that I can access the MSR through the msr-safe driver, from user mode (the snippet). You can compile it with gcc -o test_msr-safe test_msr-safe.c and run it with ./test_msr-safe. It will simply read and print the content of the TSC register. Nonetheless, the file /dev/cpu/0/msr-safe is opened with the O_RDWR mode, as it is done in GEOPM/PlatformImp.cpp;

I tried to run the first tutorial as I had previously done:

user@mexico48 $ export GEOPM_RM=NONE
user@mexico48 $ export MPIEXEC="srun --reservation=Power-GEOPM --exclusive --cpu_bind=v,mask_cpu:0x1,0x2,0x4,0x8,0x10 -N 2 -n 10"
user@mexico48 $ ./tutorial_0.sh

This time, GEOPM seems not to be able to open the /dev/cpu/0/msr-safe file due to a permission issue:
cpu_bind=MASK - mexico49, task  8  3 [7540]: mask 0x8 set
cpu_bind=MASK - mexico49, task  9  4 [7541]: mask 0x10 set
cpu_bind=MASK - mexico49, task  7  2 [7539]: mask 0x4 set
cpu_bind=MASK - mexico49, task  5  0 [7537]: mask 0x1 set
cpu_bind=MASK - mexico49, task  6  1 [7538]: mask 0x2 set
cpu_bind=MASK - mexico48, task  1  1 [45210]: mask 0x2 set
cpu_bind=MASK - mexico48, task  2  2 [45211]: mask 0x4 set
cpu_bind=MASK - mexico48, task  0  0 [45209]: mask 0x1 set
cpu_bind=MASK - mexico48, task  4  4 [45213]: mask 0x10 set
cpu_bind=MASK - mexico48, task  3  3 [45212]: mask 0x8 set

Error: <geopm> Could not open MSR device: permission denied opening device /dev/cpu/0/msr_safe: at geopm/src/PlatformImp.cpp:376

To rule out a potential slurm-related issue, I tried to launch the code snippet mentioned above thanks to slurm:

user@mexico48 $ srun --reservation=Power-GEOPM -w mexico48 ./test_msr-safe

About the MSR:
    Address: 0x10
    Content: 0x3b49ae435d1966

It seems to be properly functioning. I also checked that the permissions, groups and owners associated with the executables (tutorial_0 and test_msr-safe) are identical.
Have you any idea of the root of the issue?
Thank you in advance for your answers.

Hi Mathieu,
My name is Stephanie, I'm one of the msr-safe code maintainers here at LLNL. I was reading through your comments here and it looks like msr-safe is the culprit here. I see that your simple code snippet works on mexico48. As another sanity check, when you run your code snippet on mexico49, are you still able to access the TSC register? One hunch here is that msr-safe was installed on one node, but will also need to be installed on all nodes where you want to access MSRs. Generally, we see this error message when permissions do not allow access to the msr_safe device files.

Since there is no mention of the -p All part, I am not sure whether it is taken into account by geopmsrun or not.

You're right, presently we don't take -p into account properly in the launcher.  There's a pre-run step that will try to issue lscpu on the compute nodes prior to actually running your job.  This is done to determine the proper affinity mask for your system's topology.  Chris has a patch that's he's testing to address this and it's here:
https://review.gerrithub.io/#/c/392225/
If you cherrypick that it should address issues around handling -p.  Let me know if you're having issues.
The other thing you might want to check on is the permissions for the msr_safe file descriptor.  Like Stephanie said above, you'll want to make sure you have msr_safe installed in all the nodes where you want to run.  You can run a command like the one below to see if the msr_safe file is there and what the permissions are on it.  It should be owned by root, and have group r/w access for the msr group.
srun -N1 -p <PARTITION> -w <TEST_NODE> ls -l /dev/cpu/*/msr_safe

Hello Stephanie and Brad,
Thank you for your answers. You were right, the problem I encountered stemed from the installation of the msr-safe driver on mexico49.  On mexico49, I only checked with systemctl that the daemon associated with msr-safe was running.
It seems that the multipath associated with the underlaying disk arrays failed during the installation on mexico49, resulting in msr-safe rpm not being able to set properly the access rights/permissions/ownerships on the  msr-safe driver files.
However, I am now facing a shared memory-related issue:
Error: No such file or directory: SharedMemoryUser: Could not open shared memory with key "/geopm-shm-16130-comm-split-prof": at geopm/src/SharedMemory.cpp:166

I checked on each compute node, the shared memory is active and seems to be properly functioning (geopm is able to create geopm-shm-16130-sample, geopm-shm-16130-tprof, and openmpi-sessions-16130@mexico49_0 in /dev/shm).
After looking to the source code of geopm, I would say the issue is related to the timeout timer set up for the instance of the ShareMemoryUser class. However, the timeout seems to be set up to 30 seconds by default, and the error message usually show up after a couple of seconds (I timed it). On top of that, the tutorial 0, out of the tens of executions I tried, successfully executed 4 times. Have you got any idea concerning this issue ?
Finally, concerning geopmsrun, the patch Christopher coded solves the issue concerning the preliminary lscpu job. However, I am now facing this error message:
<geopmpy.launcher> sequence item 8: expected string, NoneType found

Hereinbelow, the way I modified the tutorial_0.sh launching script:
geopmsrun  -p All \
           -w mexico48,mexico49 \
           -N 2 \
           -n 8 \
           --geopm-preload \
           --geopm-ctl=process \
           --geopm-report=tutorial_0_report \
           --geopm-trace=tutorial_0_trace \
           -- ./tutorial_0

Thank you in advance for your answers,
--Mathieu.

@matsKNY As far as I can tell, we've resolved the issues we were having with shared memory.  The problem ended up being a combination of issues in our code (for the application launch method of the Controller specifically), and on at least one system an issue with the way systemd was configured on the compute nodes.
systemd has a configuration file located at /etc/systemd/logind.conf which may have an option set called RemoveIPC.  If RemoveIPC=yes is set as either a compile time default or an override in that file, then systemd will remove shared memory keys created by GEOPM which will cause the issue you describe above.  You need to verify that RemoveIPC=no is set on the compute nodes for things to work properly.  I've updated the README on the GitHub front page to reflect these findings which is available here.
If you have any other problems, please feel free to open a new issue or reach out via email:
brad.geltz (at) intel.com.


Update delete-resource-group.md

The spelling restore was spelled wrongly

Thanks, @fadarboye!

Your welcome @Court72 , please let me know once this is merge or fix


I take it you gave up on this?

Too bad this project was abandoned, atoms still needs a profile manager...

I actually never really started. I don't really have time to pursue this now, but I won't delete the org just yet. If it happens that I would have at least couple of weeks free, I'll get right to it. If I can't find time in next 3 years, I will delete the org.


Product Updated

this is how the updated product page look..
If you have any changes please recommend.
and Merge this PR

Kindly mention the #issue you have solved

This is issue #9 Images not scaled in product page


initializeModel is too aggressive with warnings about NAs in determ nodes

E.g., see 'various conjugacies setup' test. Here, a lifted node that involves 'a' and 'c' is flagged as NA when 'a' is initialized but 'c' is not yet initialized.
This seems to also affect various other BUGS models (seeds, oxford, etc.) so when fix this we should manually run test-mcmc to see if warnings go away as they should.

@paciorek see PR #828


Loading lists or maps from env vars

It could be useful to load lists from environment vars. But then the big question is: what format? Comma separated? There are too many options (JSON or simply separated by ,; :) and none can be used as a solution for everything. So it needs to be configurable for the library user.
I suppose it is already possible now by using deserialize_with, but that's not particularly convenient.

Why not just require the delimiter property in the field attribute if env is given for collection?

That's a possibility, but maybe not flexible enough. Some people might want to have lists like PORTS="[80,443]". And what about maps? I think it would be better to let users provide their own parsing function. Maybe with some built-in functions for widely used syntax.
#[config(env = "PORTS", parse_env = ...)]

Although this functionality already exists, this code will work:
use confique::Config;

#[derive(Debug, Config)]
pub struct Configuration {
    #[config(env = "PORTS", deserialize_with = deserialize_vec)]
    pub ports: Vec<u16>,
}

pub(crate) fn deserialize_vec<'de, D>(deserializer: D) -> Result<Vec<u16>, D::Error>
where
    D: serde::Deserializer<'de>,
{
    use serde::de::{self, Deserialize};
    String::deserialize(deserializer)?
        .split(',')
        .map(|port| {
            port.parse::<u16>().map_err(|err| {
                de::Error::custom(format!(
                    "Unexpected error while parse {} as string with port: {:?}",
                    port, err
                ))
            })
        })
        .collect::<Result<_, _>>()
}

fn main() {
    std::env::set_var("PORTS", "10,20");
    let config = Configuration::builder().env().load().unwrap();
    assert_eq!(&config.ports, &[10, 20]);
}

It turns out that the whole feature is precisely to add parse_env, which will only work for the env format and does not make changes to the parsing for other formats.


Report missing pubspec error in JSON if JSON output is specified

Go to a directory with no pubspec.
Run a pub command (like list-package-dirs or build) with --format json
Be sad when error is not a JSON object.

The problem is that error gets thrown before the --format option has been parsed. We need to switch that up. It may be simplest to hoist --format up to being a global option.

This issue has been moved to dart-lang/pub#1052.


API calls are not being cancelled when switching tabs.

          Hello,

I am noticing that switching between tabs is not canceling the API calls. I believe the issue was initially fixed as 0.6.11 however, somewhere in between, it seems to have stopped working.
Originally posted by @dorjeys3 in https://github.com/Zen-Reportz/zen_dash/issues/54#issuecomment-2161247626

which version of zen_dash are you using ?

closing issue since I dont know which version you are using. If you have still issue, please open again with version detail


Unable to use livekit client with http: protocol.

Hi, I unable to use livekit with http: protocol. I have site that serve both on https, and http. Everything works fine on https, but when I open it with http protocol livekit never connected. I don't need something special in case of http protcol I only need for data channels, so I don't want to use media devices. The problem is inside livekit-client
        if (isWeb()) {
          window.addEventListener('beforeunload', this.onBeforeUnload);
          navigator.mediaDevices.addEventListener('devicechange', this.handleDeviceChange); **// <-- HERE**
        }

        resolve(this);

With http protocol navigator.mediaDevices are not defined, so resolve is never called.

closed via 0220296


Help making AS3MXML LSP work in Panic Nova

Hello, I'm trying to make an extension for Panic's Nova and use this for an LSP. Out of the box, I got some functionality working, like issues, but could not get hovering to work. I noticed that Nova does not send a request textDocument/didOpen like VSCode does. But, when I try manually invoking I get it runs textDocument/publishDiagnostics on the files in the project, and then I get an error:
org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer fireError
SEVERE: Cannot invoke "java.util.concurrent.CompletableFuture.thenAccept(java.util.function.Consumer)" because "future" is null
java.lang.NullPointerException: Cannot invoke "java.util.concurrent.CompletableFuture.thenAccept(java.util.function.Consumer)" because "future" is null
org.eclipse.lsp4j.jsonrpc.RemoteEndpoint.handleRequest(RemoteEndpoint.java:279)
org.eclipse.lsp4j.jsonrpc.RemoteEndpoint.consume(RemoteEndpoint.java:190)
org.eclipse.lsp4j.jsonrpc.TracingMessageConsumer.consume(TracingMessageConsumer.java:114)
org.eclipse.lsp4j.jsonrpc.validation.ReflectiveMessageValidator.consume(ReflectiveMessageValidator.java:68)
org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.handleMessage(StreamMessageProducer.java:194)
org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.listen(StreamMessageProducer.java:94)
org.eclipse.lsp4j.jsonrpc.json.ConcurrentMessageProcessor.run(ConcurrentMessageProcessor.java:113)
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base/java.lang.Thread.run(Thread.java:833)

I also get this on textDocument/didClose, textDocument/textChange. I was wondering if you had any suggestion on where to start digging into this.

Okay, got a little further. Looks like I was doing a sendRequest instead of sendNotification! Once that was switched, the error no longer appears.
While I do get the issues in Nova, I do not get any of the hovers (which I'm really wanting to get working).
Comparing what happens in VS Code, it looks like this is the chain of events:

sendRequest "initialize"
sendNotification "initialize"
sendNotification "textDocument/didOpen"
sendNotification "workspace/didChangeConfiguration"
sendRequest "workspace/executeCommand" with something like:

"command": "as3mxml.setRoyalePreferredTarget",
	"arguments": [
		"SWF"
	]

In Nova, step 1 and 2 happen, but  3 - 5 do not. Are their any other steps I would need to make the hover capability work?

textDocument/didOpen Is a pretty vital notification in the language server protocol. It’s unlikely you’ll get anything to work well if your editor doesn’t send it.
The remaining ones after textDocument/didOpen probably aren’t strictly required, but there’s no way for the language server to track changes to an unsaved file without textDocument/didOpen.

sendNotification "initialize"

I believe that this should be initialized instead.

Turns out Nova does send the notification initialized by itself so without much in the extension I can get it to do:

Sending request initialize
Received response initialize
Sending notification initialized

then it receives a bunch of notifications of textDocument/publishDiagnostics
Looking at VSCode, opening the same folder without opening any files, it ends up doing this:

Sending request initialize
Received response initialize
Sending notification initialized
Sending notification workspace/didChangeConfiguration
Sending request workspace/executeCommand (sending command as3mxml.setRoyalePreferredTarget)
Received request client/registerCapability
Sending response client/registerCapability
Received notification as3mxml/setActionScriptActive

then it receives a bunch of notifications of textDocument/publishDiagnostics
I tried sending the notification of workspace/didChangeConfiguration and the request of  workspace/executeCommand, but I do not receive a request for client/registerCapability and no notification of as3mxml/setActionScriptActive. I was wondering if I am missing along the way.
I also added to my extension to signal when an ActionScript or MXML file is opened, it will send notifications that didOpen and didClose. When looking at VSCode, when didOpen happens it is then followed by textDocument/documentSymbol, textDocument/codeAction. I tried calling that and get the results. Still no hovers. :-(

Sending notification workspace/didChangeConfiguration

With this notification, VSCode sends any settings that the language server would be interested in (anything staring with as3mxml., basically). I don't think it's strictly required.

Received request client/registerCapability

This is sent by the as3mxml language server when it receives the initialized notification. It is sent only if the editor supports dynamic registration of capabilities, and is not strictly required.

Received notification as3mxml/setActionScriptActive

This is sent only if a specific field is set on the initialization options. It's used by the VSCode extension to update some GUI items. Other editors generally don't need this, so it's normal that you aren't getting it.

then it receives a bunch of notifications of textDocument/publishDiagnostics

This means that the project has been correctly detected, and it is returning compiler errors/warnings for each file.
As best I can tell, you are starting everything correctly. I don't know why Nova would be deciding not to send hover requests to the language server.

Looks like I had an issue with the syntax.xml for Nova. It has an attribute in the opening <syntax> tag which can be different the the display name which is stored in a child tag <meta><name>. Once I changed the attribute to match what is passed with the clientOptions I started to get the hovers and completions, as well as the code actions (like "Remove unused imports") which worked!
One more question. Is there a way to avoid using an .asconfig file and just pass the LSP configurations of the source path, etc.?

One more question. Is there a way to avoid using an .asconfig file and just pass the LSP configurations of the source path, etc.?

No. You must have an asconfig.json file for each project.

Thanks again for your help and this awesome LSP! Since it's only a spare time project, I'll close this ticket, and if I have any other question's I'll open a new ticket.


Move six dependency to the correct command module

With a recent commit, the six dependency was added to the azure-cli module when it should have been added for azure-cli-acs.

:shipit:


Github PR Build Bot Comment

Hi there, not sure if i'm asking on the right repo.
Is there a way to disable this automated message?

After some digging i found spectrum, i moved this question there as i think it's better suited.
https://spectrum.chat/thread/8bad6994-3c80-4e9e-ada7-2b63e9d53b72

The answer is to modify your now.json file:
When set to true, ZEIT Now for GitHub will stop commenting on pull requests and commits.
{
  "github": {
    "silent": true
  }
}

wow, i feel lucky that i found your answer 5 days after you posted it. thanks!

oct 2020, i had to google this to find this setting


Licence

I'm considering using this project in my own projects. Is it possible to add a licence to the project to know if it's viable for me to use it?

I'm considering using this project in my own projects. Is it possible to add a licence to the project to know if it's viable for me to use it?

Will do this now.

Fixed in 5e3d5b1eee45098af7d7a6fd368e1b4c859880ed

Is MIT good for you? This is just a backend for me so have no issues with any kind of third-party commercial usage.


Hunger and thirst value cannot change position

Hunger and thirst can't change the position. The thermometer changes the position like that

Hi. I am not sure why would you like to reposition the hunger bar, but this mod doesn't interfere with the hunger bar rendering at all. So afaik there is no way you can move the hunger bar unless you use other mods.
Regarding the hydration bar, the positioning config is ignored unless you disable "Hydration bar stacking" (which when set to true integrates the bar into the vanilla HUD). So just disable that option, and you should be good to go.
Or, it's still possible that I am just missing your point. Normally, it's not very good to specify an issue with just two sentences. So if you still think I am missing your point, try to elaborate on your issue as much as possible. Provide detailed description on what you want vs what is happening, and optimally include screenshots.

Okay no signs of activity on this one. Reopen if needed.

Is the location of the mod GUI modified through the drawing bar YX of the configuration file?

I want to change the position of HUD. The thermometer can change the position, but the thirst value can't change the position

OK, I've solved the problem


(URGENT) "bibxml-data-misc" data source should not display publisher as IETF

We need to remove <author><organization abbrev="IETF">Internet Engineering Task Force</organization></author> because the bibliographic item does not technically originate from IETF.
This is a blocker for:

https://github.com/ietf-tools/relaton-data-misc/issues/15
https://github.com/ietf-ribose/bibxml-service/issues/234

@ronaldtse I'm not sure what it needs to do here. The BibXML parser adds IETF as publisher type contributor for Internet-Drafts. Do we need to remove the contributor?

The BibXML parser adds IETF as publisher type contributor for Internet-Drafts.

Actually, this issue is mistaken. Our current version of relaton-data-misc is correct.
Look at this data:
https://github.com/ietf-tools/relaton-data-misc/blob/fcc1aae9869668935f7823724e2f74f89c07332b/data/reference.PKCS.1.1991.yaml#L22-L27
There is no publisher IETF.
@strogonoff I suspect it is the BibXML service that is adding the IETF organization. Can you please confirm?

I'm going to close this to prevent confusion.


Add # of unconfirmed txs to node stats

When querying the stats of a certain thornode add the number of unconfirmed transactions to the bot's message.
docs:
https://docs.tendermint.com/master/rpc/#/Info/num_unconfirmed_txs
e.g.: http://<IP_ADDRESS>:26657/num_unconfirmed_txs

What is the motivation behind this feature?
Why does the node operator want to know about this?

if this is the return value:
{
  "jsonrpc": "2.0",
  "id": 0,
  "result": {
    "n_txs": "31",
    "total": "82",
    "total_bytes": "19974"
  }
}

what is the number we want to see? n_txs or total?

@cbarraford

i think you want total, https://github.com/tendermint/tendermint/blob/master/rpc/core/mempool.go#L140


Option to show two Cities and the Bundesland?

Hey there, is it possible to show the city and the bundesland? Greetings, Tino

Currently the code does not provide to show cities and Bundesland. You might get the result by using the module twice one in the Bundesland mode and the other one in the city mode.


After converting, the date always comes back one day off

I convert the Shamsi date to gregorian date with the following snippet:
var m = moment.from('1401/6/1', 'fa', 'YYYY/MM/DD');
const result = m.locale('en').format('YYYY/MM/DD'); // 2022/08/23

Now, the day is 23, After converting the result to the date object the day will get 22, Why?
const date = new Date(result); // 2022-08-22T19:30:00.000Z

try this:
const date = moment.from(persianDate, 'fa', 'YYYY/MM/DD').locale('en').format('YYYY/MM/DD').replace(/\//g, '-');
console.log(new Date(date))

Hi please create a demo for the issues
I have created a demo and seems it works well
https://stackblitz.com/edit/jalali-moment-demo-react-iewto2?file=index.js

I have the same issue. When converting a gregorian date to persian date. The date comes back !


Update to Elasticsearch 2.3, backtracking support, fixed issue with indexname not being validated

Added backtracing support, see documentation for details.
upgraded to Elastic search 2.3

Thanks!


Pug templating

Hello,
I'm trying to use the webpack plugin with pug templating.
So I've configured webpack :
module: {
    rules: [
      { test: /\.ts$/i, loader: 'awesome-typescript-loader', options: { configFileName: 'tsconfig.webpack.json' } },
      { test: /\.pug$/i, loaders: [ 'html-loader', 'pug-html-loader' ] },
    ]
  },

  plugins: [
    new AureliaPlugin( { viewsExtensions: '.pug' } ),
  ],

app.pug is correctly added to my bundle but it seems that the bundle is looking for app.html moduleID. In the documentation, there is a html-resources-loader that is mentioned but I don't know where it is and if it is something I have to configure.
Thank you for your help and great work by the way!

In your build, you'll want to add aurelia-webpack-plugin/html-requires-loader between html-loader and pug-html-loader.
This loader parses your HTML and looks for <require>, <compose view="" viewModel=""> to ensure their dependencies are included.
I don't think this is your problem, though.
More likely, you have not reconfigured the Aurelia runtime to look for .pug instead of .html when loading views by convention.
This is done by ViewLocator.prototype.convertOriginToViewUrl, see this comment for an example that uses Jade templates.

Thank you very much, it works! I didn't know about ViewLocator.prototype.converOriginToViewUrl.


URI builder gem

The stdlib URI gem does some really annoying things. I have some code like this:
  # We need this when we redirect to sites outside of this, like Stripe.
  def return_url(path: return_path)
    redirect = URI(path)
    # Eugh ...  Allow chaining and return a new URL each time.
    scheme = request.ssl? ? URI::HTTPS : URI::HTTP
    # Make sure we always return back to our host so we don't have any sneaky redirects to 3rd party malicious websites.
    smurl = scheme.build(
      host: request.host,
      port: request.port,
      path: redirect.path,
      query: redirect.query,
      fragment: redirect.fragment
    ).to_s
  end

I have to do some crazy things to mutate or build an URI up or change a few things about it.
I'd rather have a gem that would let me do this:
URI("//foo.com/bar").scheme("https").host("bar.com").path("foo")

and I'd get back what you'd expect. For compatibility with Ruby URI object, I'd have a to_uri method on it that gives a the stdlib URI object.
This gem should use as much of Ruby's stdlib URI code as possible and only act as a light functional builder wrapper over it. It should NOT try to get in the business of parsing, etc.
JS has https://medialize.github.io/URI.js/, which behaves how I'd want this to behave.

Done!


Deep object copy of the initial config

Description
At the moment, we're not able to use the same config object when initializing multiple forms with identical structure and rules. This is because the config object is being mutated since it's just a shallow copy of the original object.
It would be better to be able to reuse the same config for different forms.

Fixed


Network usage 0 on eth0 on v3.3 was fine on v3.1

Required information

Distribution: Ubuntu (server)
Distribution version: 18.04
driver_version: 3.0.1
kernel_architecture: x86_64
kernel_version: 4.15.0-29-generic
server_version: "3.3"
storage: btrfs
storage_version: "4.4"
server_clustered: false

Issue description
Since updating from 3.3 to 3.1, can no longer see network usage (counters) on eth0, values are just 0.
Steps to reproduce
lxc info ct1
user@server:~$ lxc info ct1
Name: ct1
Remote: unix://
Architecture: x86_64
Created: 2018/07/23 17:35 UTC
Status: Running
Type: persistent
Profiles: default
Pid: 2992
Ips:
  eth0:	inet	<IP_ADDRESS>	vethWQOR8X
  eth0:	inet6	fe80::216:3eff:fe37:e6b	vethWQOR8X
  lo:	inet	<IP_ADDRESS>
  lo:	inet6	::1
Resources:
  Processes: 91
  CPU usage:
    CPU usage (in seconds): 16
  Memory usage:
    Memory (current): 507.60MB
    Memory (peak): 516.69MB
  Network usage:
    eth0:
      Bytes received: 0B
      Bytes sent: 0B
      Packets received: 0
      Packets sent: 0
    lo:
      Bytes received: 28.25kB
      Bytes sent: 28.25kB
      Packets received: 390
      Packets sent: 390

Same with lxc query output:
lxc query -X GET /1.0/containers/ct1/state
{
	"cpu": {
		"usage":<PHONE_NUMBER>3
	},
	"disk": {},
	"memory": {
		"swap_usage": 0,
		"swap_usage_peak": 0,
		"usage": 508026880,
		"usage_peak": 516694016
	},
	"network": {
		"eth0": {
			"addresses": [
				{
					"address": "<IP_ADDRESS>",
					"family": "inet",
					"netmask": "24",
					"scope": "global"
				},
				{
					"address": "fe80::216:3eff:fe37:e6b",
					"family": "inet6",
					"netmask": "64",
					"scope": "link"
				}
			],
			"counters": {
				"bytes_received": 0,
				"bytes_sent": 0,
				"packets_received": 0,
				"packets_sent": 0
			},
			"host_name": "vethWQOR8X",
			"hwaddr": "00:16:3e:37:0e:6b",
			"mtu": 1500,
			"state": "up",
			"type": "broadcast"
		},
		"lo": {
			"addresses": [
				{
					"address": "<IP_ADDRESS>",
					"family": "inet",
					"netmask": "8",
					"scope": "local"
				},
				{
					"address": "::1",
					"family": "inet6",
					"netmask": "128",
					"scope": "local"
				}
			],
			"counters": {
				"bytes_received": 28255,
				"bytes_sent": 28255,
				"packets_received": 390,
				"packets_sent": 390
			},
			"host_name": "",
			"hwaddr": "",
			"mtu": 65536,
			"state": "up",
			"type": "loopback"
		}
	},
	"pid": 2992,
	"processes": 91,
	"status": "Running",
	"status_code": 103
}

Looking into that one now.

tracked it down, going to figure out the best way to fix it


disableBulkEdit doesn't work on ui fields

Link to reproduction
No response
Environment Info
Payload: 3.0.0-beta.107
Describe the Bug
Create any field with type: ui, add disableBulkEdit: true,
its still visible in bulk edit.
Reproduction Steps
Create a ui field like this
    {
      name: 'customUiField',
      type: 'ui',
      admin: {
        components: {
          Field<EMAIL_ADDRESS>        },
        disableListColumn: true,
        disableBulkEdit: true,
      },
    }

Expected: the "Custom Ui Field" should not be visible here

Adapters and Plugins
No response

Fixed in https://github.com/payloadcms/payload/pull/8540, we disable them by default now in the bulk edit


How set upload file size for releases?

Feature Description
Do this:
file: custom/conf/app.ini
[repository.upload]
FILE_MAX_SIZE = 4096
MAX_FILES = 10

It was successful.
But, it's Code upload file size.
I want set Release upload file size, it limited to 4MB.

Windows server
gitea 1.15.3
MS Sql Server
Screenshots
No response

Maybe here?

[attachment]
;; Max size of each file. Defaults to 4MB
;MAX_SIZE = 4

也许在这里？

[attachment]
;; Max size of each file. Defaults to 4MB
;MAX_SIZE = 4

Very thankful !
[attachment]
MAX_SIZE = 1024


[chat] generate process is not a single step in RL

Hi I am from the ColossalAI team. I found that there are similarities between DeepSpeedChat and ColossalChat. We found that there might be some implementation error in our code, thus possibly leading to some convergence problems. If you refer to our implementation, you may encounter similar problems as well. We are planning to fix the bug these two weeks.
Meanwhile, it would be appreciated if you could reference our work if you adapted from our implementation.

The implementation of their ppo seems to be more similar to trlx. I use your colossalai ppo without gae algorithm, and the model can continuously optimize 20 generate epochs, but with trlx implementation, it may only optimize about 5 epochs. Can you share the implementation errors in colossalai that you mentioned?

For the most libs I believe they have some extent of similarity(e.g. TRL, TRLX, ClossalAI, RL4LM etc), so I am not sure if they are referencing your code or more generally the instructGPT methodologies ...
Though it will be much appreciated if you can elaborate more on the bugs you encounter


Add web_port to the CR

Followup to https://github.com/kiali/kiali/pull/3383.

@jmazzitelli fixed, weird :(

@jmazzitelli from the code, it's defined as a string, so your suggestions work.

I'll try this shortly.

@israel-hdez no problems! Added a fix there, when it is ok we can merge both together.


Cannot create javadoc artifacts on JDK 11

This seems related to https://bugs.openjdk.java.net/browse/JDK-8212233

Closed since version 0.6


Trying Azure reload on first open

Steps:

Start new mbconfig
Select Object detection
Go to Environment page

Loading spinner is shown: (we're trying to reload their Azure info)

Error is shown:

As a note, we probably shouldn't try reload a "0000..." id subscription

To validate, ensure no error is thrown when you start a new OD session

This issue can be repro on environment:

Windows 10 Enterprise, Version 20H2
ML.Net Model Builder (Preview): <IP_ADDRESS>5505
Microsoft Visual Studio Enterprise 2019: 16.9.4
.Net: 5.0.202

Steps to reproduce the behavior:

Create new C# console app with .Net 5.0;
Add model builder by right click on the project;
Select "Object detection" scenario,
Jump to Environment page with prompting the Model Builder error dialog;

Related issue: https://github.com/dotnet/machinelearning-modelbuilder/issues/1406

This issue can't be repro on：

Windows 10 Enterprise, Version 20H2
ML.Net Model Builder (Preview): <IP_ADDRESS>2301
Microsoft Visual Studio Enterprise 2019: 16.9.4
.Net: 5.0.202

Main branch:
https://privategallery.blob.core.windows.net/gallery/refs/heads/main/atom.xml
Steps：

Create new C# console app with .Net 5.0;
Add model builder by right click on the project;
Click "Object detection" scenario to go to Environment page;
The Environment is displayed normally without error.


test(launch): Add Launch release tests for kubernetes runner

Fixes

Fixes WB-13790

Description
This is a follow-up/continuation of a previous PR, which never managed to get merged due to CI issues.
This change adds two tests:

Spin up a launch agent on the local machine, push a run to a queue, and confirm the launch agent picks up the run and finishes it
Start a launch agent in a local k8s cluster, push a run to a queue, and confirm the launch agent can pull the run's image from ECR and run it

It also adds the tox -e launch-release hook, which allows us to easily run these tests and any others we add in the future in one command.

🤖 Generated by Copilot at f1cb4b2
This pull request adds and modifies files to implement and test the launch agent feature on Kubernetes. The launch agent is a service that runs launch jobs from a W&B queue on a Kubernetes cluster. The pull request adds YAML files to define the Kubernetes resources for the launch agent and the launch jobs, a README file to explain the launch release tests, and Python files to run the tests and provide helper functions. The pull request also modifies the tox.ini file to add a new tox environment for the launch release tests, and the kubernetes_runner.py file to fix a potential race condition with the image pull secret.
Testing
tox -e launch-release - See the README below for prerequisites.

🤖 Generated by Copilot at f1cb4b2

launch on kubernetes
testing and deploying runs
autumn of ML code

🔥


Update README with description

Documentation is sorta available in my little pet repo if a user is looking to extend functionality.

We're currently in the process of mono-repoing this into urbit/urbit after which we will be closing this repo
https://github.com/urbit/urbit/pull/1367


feat(CategoryTheory): the category of commutative squares

In this PR, we define a bundled version of CommSq which allows to consider commutative squares as objects in a category Square C.

Thanks for the reviews @erdOne! Indeed, while finding the better way to formalize Mayer-Vietoris squares, it has become more obvious to me that introducing this category Square C was the better approach.

Thanks!
maintainer merge

Thanks!
bors merge


RFC #4 - Patterns for dealing with migration bloat

As my application grows, so does the number of migrations. I am already up to 50, and the big refactoring I am doing now will produce 10-15 new migrations on top of that.
So I started looking for ways to somehow organize these migrations and reduce the bloat in the migrations folder.
The first thing I tried was to group these big features into scopes, and use them as "migration groups" of sorts. That didn't quite work out. migrate all will run scopes in an arbitrary order one after another, which means I can't really use them to modify the same database with cross-cutting concerns. Scopes are really only useful for either completely isolated features, or different databases.
I also considered packing more changes into a single migration, using some kind of async chaining pattern. Unfortunately, I use MySQL which doesn't have DDL transactions. So if anything breaks, I'm stuck with an inconsistent state that I have to resolve manually. The best way to use db-migrate is indeed to create a new migration for every step, which doesn't help the bloat problem.
Another option would be to rewrite some older migrations that are doing duplicate work (eg. create table, mutate table, add column, remove column...) into a single migration. But that sounds like a lot of manual fiddly bug-causing work.
So in the end, I think I'll probably just preface my new migrations with the feature name (eg.<PHONE_NUMBER>1857-big_refactoring-add_table_x) and keep adding them as I'm used to.
Before that though, I figured I should ask. Is there some option or pattern I am missing? What is the recommended best practice for dealing with accumulated migration bloat?

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/41191654-rfc-4-patterns-for-dealing-with-migration-bloat?utm_campaign=plugin&utm_content=tracker%2F73887&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F73887&utm_medium=issues&utm_source=github).

Oh, I forgot to mention.
One possible pattern for this would be to change the semantics for the :all scope. Instead of executing scopes one after another, merge all transactions together into one "virtual directory", and execute them in order based on their timestamp prefixes.
But that was already suggested and rejected in #357 , so I didn't pose it as a feature request.

Auto-migration feature might help with this, similar to the feature in Loopback https://loopback.io/doc/en/lb2/Creating-a-database-schema-from-models.html and Sailsjs http://sailsjs.com/documentation/concepts/models-and-orm/model-settings#?migrate

That didn't quite work out. migrate all will run scopes in an arbitrary order one after another, which means I can't really use them to modify the same database with cross-cutting concerns. Scopes are really only useful for either completely isolated features, or different databases.

Completely true currently, however, it is planned to add sorting to scopes in an user defined behavior. Feel free to open a PR for this if you want to :)

Another option would be to rewrite some older migrations that are doing duplicate work (eg. create table, mutate table, add column, remove column...) into a single migration. But that sounds like a lot of manual fiddly bug-causing work.

Ah well... I would suggest you to never rewrite any migrations. The reason is simple:
You have migrations that have already been executed, if you're rewriting those, renaming or do any other stuff, this means you create yourself a lot of trouble and in the same second you don't use migrations anymore in the way they are intended to work. Migrations are by concept always build incrementally, so you never bother to change any old migrations.
What I could think of is that it could be possibly useful to add a feature, that allows moving migrations into a different scope, w/ or w/o rewriting the migrations table. Preferably w/o rewriting, but I think that would not work out quite well though.

What is the recommended best practice for dealing with accumulated migration bloat?

Actually that is the reason that scopes exist, using them from the very beginning to organize the migrations helps a lot keeping everything tidy.

One possible pattern for this would be to change the semantics for the :all scope. Instead of executing scopes one after another, merge all transactions together into one "virtual directory", and execute them in order based on their timestamp prefixes.

That would actually break the behavior, so we can't really do this and merge them together. But as stated at the top, adding the functionality to sort the scopes in an user defined behavior would be the proper fix for this.

I also considered packing more changes into a single migration, using some kind of async chaining pattern. Unfortunately, I use MySQL which doesn't have DDL transactions. So if anything breaks, I'm stuck with an inconsistent state that I have to resolve manually. The best way to use db-migrate is indeed to create a new migration for every step, which doesn't help the bloat problem.

Yes MySQL does not, MariaDB will in some of the future versions, finally, though. But the problem is more while working with those databases, when your migrations are final you can mostly move them safely together into a single migration, naturally only if those have not been in production yet. That also depends on what the migrations do actually, but most schema changes, or index alterations don't do any harm to the other one. Chaining commands is actually supported natively since db-migrate supports promises from 0.10.x though.
However, I do know the pain of working with MySQL/MariaDB for that reason if I want to test chained commands, I write them one by one and test them one by one, fi the first one has already proven to work, it will also continue to work. I have already thought about adding the optionality to execute the down migrations if an error occurs within the migration, to reverse the state. As long as the down migrations is ordered the same the up is, this would help at least restoring the old state.

Auto-migration feature might help with this, similar to the feature in Loopback

Auto-migrations are actually quite different to migrations like db-migrate ones, as they are build from models. So in the first place you need to be using a ORM, which is not always the case, especially for applications where an ORM comes at the cost of performance which is no longer bearable. And in the second place, those are not really made and targeted to production systems.
For example: Changing a migration which is already in production is a no go, therefore migrations that have been executed are treated like already being executed. In short they are being ignored and build upon. That is important when you want to execute migrations automatically on the start of a new version of your application on the costumer side and also it is important if you want to realize real zero downtime.
So auto-migrations do not really fit to the concept here. But instead what one could think of, would be to use some plain old tools like grunt, gulp and sutff. Make them watch the migrations folder and run all migrations down to the place you changed one of your migrations and execute them back up. However as stated above, you normally never want to alter your migrations as soon as they have moved to production. If they are not in production yet, the only pain you will cause by changing them are the screams of your colleagues yelling at you b/c you changed the migrations again and broke their setup. So in any case, changing existing migrations, should be done at the right point in time, for example while polishing up everything for the release, and should never been done as soon as they are in production, b/c now you have a version of your database, which is the whole sense of migrations: database versioning.
I hope this helps a bit, and feel free to comment on anything I wrote if you have different opinions on some of the stuff I'm eager to get to know them :)

Completely true currently, however, it is planned to add sorting to scopes in an user defined behavior. Feel free to open a PR for this if you want to :)

I  thought about this and saw the call for a PR request.
The problem is this: the ordering of scopes would really need to be dependent on the operation you are executing (up or down).
Let's say I have these two scopes:

01_initial_setup
02_user_management

I want the scope 02_user_management to be dependent on 01_initial_setup, thus the prefix numbers (BTW, this is what I tried to do for my project, it seems like the simplest way to get user-controlled ordering). So when I do migrate up, I need the order to be 01, 02. However, when I do migrate down or migrate reset, the order needs to be in reverse (02, 01), with all migrations inside scopes done in the same way.
I don't see how user ordering can apply here. If scopes are to be dependent on one another, they need to run the same way as usual migrations.

One possible pattern for this would be to change the semantics for the :all scope. Instead of executing scopes one after another, merge all transactions together into one "virtual directory", and execute them in order based on their timestamp prefixes.

That would actually break the behavior, so we can't really do this and merge them together. But as stated at the top, adding the functionality to sort the scopes in an user defined behavior would be the proper fix for this.

Would it?
Presume people are now using scopes for unrelated concerns. Eg. they have two different databases, or operate on separate sets on tables.
They can still execute migrate up:scope1 and migrate up:scope2 and get individual scopes to perform their migrations. That part remains the same.
The difference is when they do migrate up:all. Instead of first doing scope2, then scope1, all migrations from both scopes are intermingled and executed based on timestamps.
But since these two scopes are already independent of one another, this doesn't matter at all! You can create table in DB1, then delete a column in DB2, then add an index in DB1, then add table in DB2. And the order doesn't matter, because the scopes are independent.
IMO the new migrate :all semantic would only add possibilities for new usages, I don't see how it would break any of the old ones (given people use scopes as intended).

Ah well... I would suggest you to never rewrite any migrations. The reason is simple:

You have migrations that have already been executed, if you're rewriting those, renaming or do any other stuff, this means you create yourself a lot of trouble and in the same second you don't use migrations anymore in the way they are intended to work. Migrations are by concept always build incrementally, so you never bother to change any old migrations.

Yeah, granted. This is a recipe for creating bugs.

What I could think of is that it could be possibly useful to add a feature, that allows moving migrations into a different scope, w/ or w/o rewriting the migrations table. Preferably w/o rewriting, but I think that would not work out quite well though.

In my attempt with scopes I did rewriting. It worked OK.

What is the recommended best practice for dealing with accumulated migration bloat?

Actually that is the reason that scopes exist, using them from the very beginning to organize the migrations helps a lot keeping everything tidy.

That won't work because of the issues with scope ordering and isolation discussed above.

However, I do know the pain of working with MySQL/MariaDB for that reason if I want to test chained commands, I write them one by one and test them one by one, fi the first one has already proven to work, it will also continue to work. I have already thought about adding the optionality to execute the down migrations if an error occurs within the migration, to reverse the state. As long as the down migrations is ordered the same the up is, this would help at least restoring the old state.

Hmm... this could help in some cases. But if the down migration chain presumes existence of some parts of the up migration that never got executed, then it will break too.
What might help is the option to "print out" all the generated SQL to the console, instead of executing it (so it'd be easier to clean up the mess). Or give me interactive prompt to y/n each async step after I force migrate down.
I don't know. On a project that uses mysql, I think I'd just stick to the one migration per operation, without chaining into transactions.

If they are not in production yet, the only pain you will cause by changing them are the screams of your colleagues yelling at you b/c you changed the migrations again and broke their setup.

Been there done that :smile:

Would it?

Yes :) B/c scopes are configurable, so it is just not that easy to just merge them alltogether. There would be also the need to switch dbs, or even the db server, which is not yet implemented though, and mixing them all by merging together would mean that you really randomly switch around those settings b/c migrations that have been separated get mixed up.
Another case would be if someone has a development scope, or something like that, that should only be executed in development. Although that is not what scopes are really intended for, it is easy to use them like that. A fix for this would be to add a configuration option to the scope configs, i.e. excludeFromAll: true.

In my attempt with scopes I did rewriting. It worked OK.

Have those migrations been in production yet? If yes, how did you dealt with those already executed ones?

Yeah, granted. This is a recipe for creating bugs.

Not just bugs, you literally manipulated and changed the past, and we all know where this ends :p. Anyhow, changing migrations that have been executed, would need tor reexecute them at your customer as soon as they have been in production and this pretty much breaks with what migrations are there for :)

That won't work because of the issues with scope ordering and isolation discussed above.

Y, ik. The issue above needs to be solved first in this scenario.

Hmm... this could help in some cases. But if the down migration chain presumes existence of some parts of the up migration that never got executed, then it will break too.

This is ok, that is also the reason why the down migrations need to be ordered in the same order. Otherwise they would break to early.

What might help is the option to "print out" all the generated SQL to the console, instead of executing it (so it'd be easier to clean up the mess). Or give me interactive prompt to y/n each async step after I force migrate down.

There is a print out option:

--verbose

--dry-run

--log-level
db-migrate up --log-level "sql|error" --verbose

Currently log-level does not output verbose output without verbose being defined. SQL is part of the verbose output currently. I will open that as a bug as this is not intended though.

I don't see how user ordering can apply here. If scopes are to be dependent on one another, they need to run the same way as usual migrations.

That would still work as long as you don't rely on a state from another scope. As soon as they have dependencies to each other you're right. But I ment with user defined sorting, that a user really can specify the order via the scope configs.

Why not update the initial create table SQL to reflect the updates from migrations, or store an updated version of the db? Then you could delete the migrations. They're in source control if you need to go back and look at them.
Getting this to work with zero downtime might be tricky, but its not the type of thing you need do often.
I'm not sure that auto-migrate implies an ORM. It implies a machine-readable and complete version of the schema, which theoretically could be grabbed and parsed from the db itself. But I guess that's a separate topic.

Yes :) B/c scopes are configurable, so it is just not that easy to just merge them alltogether. There would be also the need to switch dbs, or even the db server, which is not yet implemented though, and mixing them all by merging together would mean that you really randomly switch around those settings b/c migrations that have been separated get mixed up.

Yes, that would be inconvenient to implement (have to keep connecting to database in each migration, or keep a pool of connections...).
But from the user's perspective, do they really care that you switch between DB settings as you are running through migrations?

Another case would be if someone has a development scope, or something like that, that should only be executed in development. Although that is not what scopes are really intended for, it is easy to use them like that. A fix for this would be to add a configuration option to the scope configs, i.e. excludeFromAll: true.

If someone does have a dev scope, and they go to production and execute migrate up:all right now, they would be in the same mess whether scopes are ran one after another or intermixed.
excludeFromAll sounds like a good idea.

Have those migrations been in production yet? If yes, how did you dealt with those already executed ones?

Yes, all the old migrations are in production. I basically created a upgrade_migrations_table as the first migration in my new would-be scope, then did string concatenate / slice path for all old the migrations in its up and down actions.
Worked fine in testing (well, except I couldn't control the order of scopes, so I reverted it back).

There is a print out option:

Oh! Good to know!

Why not update the initial create table SQL to reflect the updates from migrations, or store an updated version of the db? Then you could delete the migrations. They're in source control if you need to go back and look at them.

B/c this would destroy existing setups and this wouldn't be migrations anymore.  Those still include the old entries to migrations that you've just deleted. In the case you would need to reverse them everything would just crash. What you describe is a whole different thing than migrations are. You want a reflection, migrations are not a reflection of the current database, they describe a path to reach a defined state.
Migrations have been born out of the need to have a deterministic version control over your database. As stated above, they are not a representation, but a path. What you're talking about are really just alternative dialogues to a SQL schema dump, which allow to be a bit more dynamic. Jjust altering a representation of the schema and letting the program interpret what to do to recreate this representation on your database makes it maybe quite easy to maintain, but you have zero control, you have no real version control over your database even if you version control those representation themselves. The reason for this is, that you can't really reverse your actions reliably nor are they deterministic. And you also can't rely on this strategy to deploy the latest version of the database to your customer. It might has some effects that will cause unintentional data loss and a customer might migrate from version 0.0.1 to 12.1.0. They shouldn't be forced to migrate to each single version just to upgrade to the latest version of your product and this is the reason why it makes sense to maintain a path instead of a representation of your database.
Anyhow, reflecting a current state of the schema is actually still something that is a working item, to speed up migrations from zero when you've got yet nothing. And this is will be a reflection generated from the existing migrations and not the database, so still a bit different :)
So yes, a schema reflection might be easier to maintain, but solves a whole different problem than migrations do.

Getting this to work with zero downtime might be tricky, but its not the type of thing you need do often.

Today zero downtime is something that is becoming, at least for new products, quite the default. But agreeing not everyone is yet providing zero downtime in their products, mostly b/c there is not much knowledge made publicly available or best practices. So many just don't think about it, or think it is too hard to accomplish. Which btw is not the case, it is quite easy, you just have to follow some rules as developer, zero downtime has actually nothing to do with the infrastructure, which just have to provide a load balancing itself.

@jcrben

Why not update the initial create table SQL to reflect the updates from migrations, or store an updated version of the db? Then you could delete the migrations. They're in source control if you need to go back and look at them.
Getting this to work with zero downtime might be tricky, but its not the type of thing you need do often.

If you do that in production, and then migrate down for some reason, the script will break because you no longer have all the files the db says you do. You would have to rewrite migrations table(s) after you make this change.
Other great concern is the possibility of introducing bugs. You basically remove well tested working code for untested new code, with sole purpose of code aesthetics. Hard to justify.

Yes, all the old migrations are in production. I basically created a upgrade_migrations_table as the first migration in my new would-be scope, then did string concatenate / slice path for all old the migrations in its up and down actions.

Sounds reasonable, that might be something which db-migrate could provide a helper for, to reorganize.

If someone does have a dev scope, and they go to production and execute migrate up:all right now, they would be in the same mess whether scopes are ran one after another or intermixed.

True.

But from the user's perspective, do they really care that you switch between DB settings as you are running through migrations?

When it is getting hellish slow b/c of this they might care. Not to forget that you can easily tap into traps like too many open connections as soon as we step onto this path :)
Anyhow, reversing the sort order on down actions for scopes wouldn't hurt anyone and might be a good first step into the right direction.

Well, the general approach that I hear (and which I follow at work and in my projects) is roll forward in production, not back. I still do define down, but it's mostly for dev environment work such as bisecting a regression - can't remember the last time such a thing was run in production. Consolidating old migrations is something you would do with rather old migrations which you wouldn't go back to.
For example, I worked on an app that had over a thousand migrations from years of work. At one point we deleted them, updated the initial dev environment schema, and started building up more migrations.

@jcrben

For example, I worked on an app that had over a thousand migrations from years of work. At one point we deleted them, updated the initial dev environment schema, and started building up more migrations.

You can do this, but only if that old version wouldn't be upgradeable to your latest version anymore anyway. As long as you want to guarantee this, you still have this problem. I think you do see this just from having only your production system. But it is a whole different game if others deploying your applications on their machines.

Ah, yeah. It was a SaaS app so its install was entirely controlled in-house.
One distributed app, Ghost, which I self-host for my personal blog, built an automatic migration tool. It's in the link I put above but they explain a bit more about it at https://dev.ghost.org/database-migrations/ - I'm running an old version and haven't played with it.

@jcrben Actually not, it is quite simple and btw. they use knex :)
But don't get me wrong, I don't wanted to state, that your described scenario wouldn't be usable, just that migrations are a whole different thing actually :)
And for ghost there are some problems that with their migrations.
For example, you can't roll back. If everything wents south, well yeah, you're in bad luck. This is the reason why you must do a backup on each upgrade of ghost. I use ghost myself and also host it myself, so I know that ghost isn't really the easy to upgrade app. It is not hard though, but it is unthankfully and unnecessary work :)
Also ghost is very far from being zero downtime friendly, or in short it is not, expect everything to break and pray that there haven't been any non deterministic changes (like delete or rename) if you try to deploy with zero downtime.
But apart from that I love ghost <3


DMN Editor: Hardcoded string as a feel expression has bad performaces.

If there is a huge hardcoded string as a feel expression, then the hashCode javascript function is extremely slow. We have ~ 20kB JSON string hardcoded for testing purposes and after a removal it is faster.
https://github.com/kiegroup/kie-tools/issues/1281#issuecomment-1301988705

@yesamer should we close this now, as gwt 2.10 upgrade is merged?

@jomarko Not sure this is related to GWT update. Honestly, I didn't reproduce this, I opened it because of this comment https://github.com/kiegroup/kie-tools/issues/1281#issuecomment-1301988705

Hi, thanks for updating GWT, it really has an impact on our DMNs. About this - I've realized that when I have 20kB FEEL string variable expression then the hashcode in javascript is consuming a lot of resources - there was a loop with thousands interations. I will try to check if with new GWT is still there. Give me this week please.

@msusicky ok, please keep us informed.


Fixes #2673: Replaced SharedPreferences commit() method with apply()

Description
Replaced SharedPreferences commit() method with apply() which will handle writing data in the data storage in the background.
Related issues and discussion
Fixes #2673

Thanks @RajaVamsi11 :-)
Thank you @deniger for the review 👍


feat: Add deps.rs status to Readme.md

https://deps.rs/repo/github/simbleau/cddns

Closed in #113


Missing docs/venv for apply-ssh

Describe the bug
The apply-ssh command should be used to manage authorized SSH keys but it doesn't work out of the box. There is no venv for it (like for the apply command), so you must have ansible installed locally. It also requires the ansible.posix.authorized_key role which is not included in ansible by default.
Finally, since it executes ansible from the playbooks folder instead of from kubespray, it doesn't use the ansible.cfg that comes with kubespray. This means that host key checking is used which will usually not work without first adding the fingerprints for all Nodes.
To Reproduce
Steps to reproduce the behavior:

Create a new cluster using compliantkubernetes-kubespray
Make sure you have some authorized keys listed in group_vars/all/ck8s-ssh-keys.yaml under ck8s_ssh_pub_keys_list
Run ./bin/ck8s-kubespray apply-ssh <prefix>
See error

Expected behavior
Apply ssh should make sure all dependencies are installed OR the required steps should be clearly documented. Additionally, the same (or a similar) ansible.cfg should be used as for kubespray. I think it makes sense to use the same config since we are interacting with the same cluster.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
We may not always want a venv. See this PR. Make it optional.

Add missing prereqs into ck8s-devbox instead to achieve this.
Installing pre-reqs in many different tools may bite back in the long term.

I would be nice to know the reasoning behind the apply-ssh command. If I was going to update SSH keys on the hosts I would use the already existing solution used to initially inject the keys. With the extra playbook in this repository we will have two sources of truth of which keys should be on the hosts.
For example, in the case of Exoscale I would add the extra SSH keys in the Terraform config, apply and rerun cloud-init.
@Ajarmar could you give some more info here?

Doesn't a change in cloud-init force recreation of the vm?

Also some tf modules only allow to you to specify a single key (i haven't looked at exoscale to see what magic it does)
https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/terraform/openstack/variables.tf#L81

Doesn't a change in cloud-init force recreation of the vm?

Nope! cloud-init clean && cloud-init init
There are probably more ways of specifically run the tasks required rather than the whole cloud-init procedure as well.

Also some tf modules only allow to you to specify a single key (i haven't looked at exoscale to see what magic it does)
https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/terraform/openstack/variables.tf#L81

Good catch! One need to be aware that new machines added will always require our special SSH playbook on top of Kubespray in that case.
As this is not crucial let's leave it as is. But in the future we should probably push for proper support upstream.

Nope! cloud-init clean && cloud-init init
There are probably more ways of specifically run the tasks required rather than the whole cloud-init procedure as well.

Is there any terraform modules that implements this? I don't think I've ever seen any module that doesn't recreate the VM when changing user-data.

Nope! cloud-init clean && cloud-init init
There are probably more ways of specifically run the tasks required rather than the whole cloud-init procedure as well.

Is there any terraform modules that implements this? I don't think I've ever seen any module that doesn't recreate the VM when changing user-data.

I just did this on Exoscale which just updated the user-data in-place. Not sure about the rest.

I would be nice to know the reasoning behind the apply-ssh command. If I was going to update SSH keys on the hosts I would use the already existing solution used to initially inject the keys. With the extra playbook in this repository we will have two sources of truth of which keys should be on the hosts.
For example, in the case of Exoscale I would add the extra SSH keys in the Terraform config, apply and rerun cloud-init.

Please find here the rationale on why we chose to manage SSH keys from Ansible and not Terraform: https://compliantkubernetes.io/adr/0005-use-individual-ssh-keys/#decision-outcome


Fix typo in readme

Fixes #<issue_number_goes_here>

It's a good idea to open an issue first for discussion.

[ ] Tests pass
[ ] Appropriate changes to README are included in PR

Thanks!


Deprecate the config-generator

The config-generator is used by images to transform the Image yaml configuration to a suitable Infinispan server configuration. Originally this was required to:

Provide the operator with a simple mechanism to configure the Infinispan server.
Simplify the Infinispan/JGroups configuration in k8s environments.

As Infinispan has evolved, many improvements have been made to make Infinispan work seamlessly with k8s, making 2 redundant.
Similarly, as the operator has evolved the config-generator has become a pain point when adding new features and will soon be replaced https://github.com/infinispan/infinispan-operator/issues/1253.
Therefore, we should deprecate the config-generator, CONFIG_PATH and IDENTITIES_PATH vars from 13.0.0.Final.
The supported use cases for the image will be:

Basic usage with no custom configuration:

docker run -p 11222:11222 -e USER="Michael" -e PASS="Bluth" infinispan/server

User provided server configuration:

docker run -it -v example-vol:/user-config infinispan/server -b SITE_LOCAL -c /user-config/custom-infinispan.xml

Deprecated since 13.0.0.Final


GIGA: analogWrite(5, 0); faults the processor - Analog APIS are not complete

Describe the bug
analogWrite(5, 0);
Faults on the GIGA...
Output of Serial Monitor

If you have an USB-to-Serial adapter, paste the complete output of the console crash, starting from

uart:~$ sketch
[00:00:11.044,000] <inf> usb_cdc_acm: Device suspended
[00:00:11.269,000] <inf> usb_cdc_acm: Device configured
[00:00:22.948,000] <err> os: ***** BUS FAULT *****
[00:00:22.953,000] <err> os:   Precise data bus error
[00:00:22.959,000] <err> os:   BFAR Address: 0x5f657378
[00:00:22.965,000] <err> os: r0/a1:  0x5f65736c  r1/a2:  0x00000000  r2/a3:  0x2400eee0
[00:00:22.973,000] <err> os: r3/a4:  0x08058a1f r12/ip:  0x00000000 r14/lr:  0x2400e7d7
[00:00:22.982,000] <err> os:  xpsr:  0xa1000000
[00:00:22.987,000] <err> os: s[ 0]:  0x2400f094  s[ 1]:  0x080510ad  s[ 2]:  0x00000000  s[ 3]:  0x0804ef99
[00:00:22.998,000] <err> os: s[ 4]:  0x00000000  s[ 5]:  0x24000568  s[ 6]:  0x000002d8  s[ 7]:  0x0804f057
[00:00:23.008,000] <err> os: s[ 8]:  0x00000000  s[ 9]:  0x00000000  s[10]:  0x2400ec95  s[11]:  0x00000001
[00:00:23.018,000] <err> os: s[12]:  0x2400e75d  s[13]:  0x2400e771  s[14]:  0xfffffffe  s[15]:  0x2400d510
[00:00:23.029,000] <err> os: fpscr:  0x0805b614
[00:00:23.034,000] <err> os: Faulting instruction address (r15/pc): 0x08058a20
[00:00:23.042,000] <err> os: >>> ZEPHYR FATAL ERROR 25: Unknown error on CPU 0
[00:00:23.050,000] <err> os: Current thread: 0x240008b0 (shell_uart)

Optional: attach the sketch
void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200);
  while (!Serial) {}

  Serial.println("Start of PWM test");
  delay(10000);

}

void loop() {
  // put your main code here, to run repeatedly:
  for (int i = 0; i < 256; i++) {
    analogWrite(5, i);
    delay(20);
  }

  for (int i = 255; i <= 0; i--) {
    analogWrite(5, i);
    delay(20);
  }
  delay(50);
}

Additional context
Add any other context about the problem here.
Instrumented the code:
void analogWrite(pin_size_t pinNumber, int value)
{
  Serial.print("AnalogWrite called: "); Serial.print(pinNumber);
  Serial.print(" "); Serial.print(value);
  delay(5);
  size_t idx = pwm_pin_index(pinNumber);
  Serial.print(" idx: "); Serial.println(idx);
  delay(5);
  if (!pwm_is_ready_dt(&arduino_pwm[idx])) {
    return;
  }
  Serial.print("After is ready "); Serial.flush();

  if (idx >= ARRAY_SIZE(arduino_pwm) ) {
    return;
  }

  Serial.print(" ");Serial.print(arduino_pwm[idx].period); Serial.flush();

  if (((uint32_t)value) > arduino_pwm[idx].period) {
    value = arduino_pwm[idx].period;
  } else if (value < 0) {
    value = 0;
  }

  /*
   * A duty ratio determines by the period value defined in dts
   * and the value arguments. So usually the period value sets as 255.
   */
  Serial.println("before Call pwm_set_pulse_dt"); Serial.flush();
  (void)pwm_set_pulse_dt(&arduino_pwm[idx], value);
  Serial.println("After Call pwm_set_pulse_dt"); Serial.flush();
}

Debug output:
Start of PWM test
AnalogWrite called: 5 0 idx:<PHONE_NUMBER>

So the line:  size_t idx = pwm_pin_index(pinNumber);
returns: 0xffffffff which is obviously > idx >= ARRAY_SIZE(arduino_pwm)
which is checked, unfortunately after the code:
if (!pwm_is_ready_dt(&arduino_pwm[idx])) {
which faulted.
Will swap some code next.  But wonder why 5 is not returning a valid index...

Updated the code so does not fault:
Also printed out PWM table
  for(size_t i=0; i<ARRAY_SIZE(arduino_pwm_pins); i++) {
    Serial.print(arduino_pwm_pins[i]);
    Serial.print(" ");
  }

57 0 83 11 6 80 81 3 4
Now to figure out what those pins are ...

AH5->A5->? Would think 81 by schematic but83? which is A7?

Yep found that in https://github.com/arduino/ArduinoCore-zephyr/issues/7 as well where"

D76-D83 (A0 to A7) ring out to D78-D85 (off by 2)

just noticed though that @facchinm has some comits to fix the pin numbering - see bottom of issue

@KurtE - @facchinm
looking at the BLE Sense overlay it looks like the PWM pins are fully defined but not seeing that in giga overlay or am I missing somethings?

Edited title as I don't believe that these APIs are complete:
That is, on Arduino analogWrite(pin, value), the value is I believe defined as the range of 0-resolution or fir 8 bits 255.
The current code: simply calls:
  (void)pwm_set_pulse_dt(&arduino_pwm[idx], value);
with the value passed in, it compares the value to be between 0 and the period, and the
With the one defined in the current overlay, I believe the period is:  pwms = <&cam_clock_pwm 3 PWM_HZ(6000000)
The translation into the devicetree_generated.h:
#define DT_N_S_soc_S_timers_40010000_S_pwm_S_pwmclock_P_pwms_IDX_0_VAL_period 166
So the period is the time in us.  So we should map the value something like:
value = map(value, 0, 255, 0, arduino_pwm[idx].period;
analogWriteResolution is not defined or implemented, to allow to change resolution typically between 8 and 12 bits
default of 8...
What about the DAC pins (A12 and A13)
Having difficulties trying to figure out the .dts/.overlay files on how to specify PWM pins.
Things like how to define:  TIM1_CH1 versus TIM1_CH1N
Also trying to figure out the multiple sections associated with the camera PWM pin.  More on this later...

Quick update: I am having a little luck with PWM.  I have Pin 10 limping along, maybe 12 as well have not tried it..
Still very much WIP.
Sections in the overlay with PWM stuff:
// XCLK as PWM from PJ9
&timers1 {
	status = "okay";
	st,prescaler = <0>;

	cam_clock_pwm: pwm {
		status = "okay";
		pinctrl-0 = <&tim1_ch3_pj9>,
				<&tim1_ch1_pk1>,
				<&tim1_ch2_pj11>;
		pinctrl-names = "default";
	};

	cam_clock_pwm: pwm {
		status = "okay";
		pinctrl-0 = <&tim1_ch3_pj9>,
				<&tim1_ch1_pk1>,
				<&tim1_ch2_pj11>;
		pinctrl-names = "default";
	};
};

&cam_clock_pwm {
	/* ...then use the pwmclock node to start the clock generation */
	pwmclock: pwmclock {
		status = "okay";
		compatible = "pwm-clock";
		clock-frequency = <0>;
		#clock-cells = <1>;
		pwms = <&cam_clock_pwm 3 PWM_HZ(6000000) PWM_POLARITY_NORMAL>;
			/*	<&cam_clock_pwm 1 PWM_HZ(500) PWM_POLARITY_NORMAL>, */
			/*	<&cam_clock_pwm 2 PWM_HZ(500) PWM_POLARITY_NORMAL>; */
	};
};
...
		pwm-pin-gpios =	    <&gpioj 9 0>,		/* D57PJ9	TIM1_CH3 */
					<&arduino_header 16 0>,		/* D10 PK1  TIM1_CH1 */
					<&arduino_header 18 0>;		/* D12 PJ11 TIM1_CH2 */

...
		pwms = <&cam_clock_pwm 3 PWM_HZ(6000000) PWM_POLARITY_NORMAL>,
					<&cam_clock_pwm 1 PWM_HZ(20000) PWM_POLARITY_NORMAL>,
					<&cam_clock_pwm 2 PWM_HZ(20000) PWM_POLARITY_NORMAL>;

Analog write in zephyrcommon.cpp
void analogWrite(pin_size_t pinNumber, int value)
{
  Serial.print("analogWrite(");
  Serial.print(pinNumber);
  Serial.print(",");
  Serial.print(value);
  Serial.print(") ");
  size_t idx = pwm_pin_index(pinNumber);
  Serial.print(idx);

  /*
   * Make sure pwm_pin_index returned something
   * valid before we use it as an index
   */
  if (idx >= ARRAY_SIZE(arduino_pwm) ) {
    Serial.print(" > ");
    Serial.print(ARRAY_SIZE(arduino_pwm));
    Serial.println(" - Failed");
    return;
  }

  if (!pwm_is_ready_dt(&arduino_pwm[idx])) {
    Serial.println(" - Failed not ready");
    return;
  }

  if (((uint32_t)value) > arduino_pwm[idx].period) {
    value = arduino_pwm[idx].period;
  } else if (value < 0) {
    value = 0;
  } else {
    value = map(value, 0, 255, 0, arduino_pwm[idx].period - 1);
  }

  /*
   * A duty ratio determines by the period value defined in dts
   * and the value arguments. So usually the period value sets as 255.
   */
  Serial.print(" - Set pulse_dt: ");
  Serial.print(value);
  Serial.print(" ");
  Serial.println(arduino_pwm[idx].period);
  (void)pwm_set_pulse_dt(&arduino_pwm[idx], value);
}

Note currently Pin 10 is outputting at 20mhz pulses.  I tried 500 and saw lots of errors about the period does not fit
in a Timer 16...
Test Sketch:
#define analog_pin 10
#define toggle_pin 3
void setup() {
    // put your setup code here, to run once:
    Serial.begin(115200);
    //while (!Serial) {}
    delay(500);

    Serial.println("Start of PWM test");
    pinMode(toggle_pin, OUTPUT);
    delay(1000);
}

void maybe_pause() {
    if (Serial.available()) {
        while (Serial.read() != -1) {}
        Serial.println("Paused");
        while (Serial.read() == -1) {}
        while (Serial.read() != -1) {}

        Serial.print("CR1: 0x"); Serial.println (TIM1->CR1, HEX);
        Serial.print("CR2: 0x"); Serial.println (TIM1->CR2, HEX);
        Serial.print("SMCR: 0x"); Serial.println (TIM1->SMCR, HEX);
        Serial.print("DIER: 0x"); Serial.println (TIM1->DIER, HEX);
        Serial.print("SR: 0x"); Serial.println (TIM1->SR, HEX);
        Serial.print("EGR: 0x"); Serial.println (TIM1->EGR, HEX);
        Serial.print("CCMR1: 0x"); Serial.println (TIM1->CCMR1, HEX);
        Serial.print("CCMR2: 0x"); Serial.println (TIM1->CCMR2, HEX);
        Serial.print("CCER: 0x"); Serial.println (TIM1->CCER, HEX);
        Serial.print("CNT: 0x"); Serial.println (TIM1->CNT, HEX);
        Serial.print("PSC: 0x"); Serial.println (TIM1->PSC, HEX);
        Serial.print("ARR: 0x"); Serial.println (TIM1->ARR, HEX);
        Serial.print("RCR: 0x"); Serial.println (TIM1->RCR, HEX);
        Serial.print("CCR1: 0x"); Serial.println (TIM1->CCR1, HEX);
        Serial.print("CCR2: 0x"); Serial.println (TIM1->CCR2, HEX);
        Serial.print("CCR3: 0x"); Serial.println (TIM1->CCR3, HEX);
        Serial.print("CCR4: 0x"); Serial.println (TIM1->CCR4, HEX);
        Serial.print("BDTR: 0x"); Serial.println (TIM1->BDTR, HEX);
        Serial.print("DCR: 0x"); Serial.println (TIM1->DCR, HEX);
        Serial.print("DMAR: 0x"); Serial.println (TIM1->DMAR, HEX);
        Serial.print("CCMR3: 0x"); Serial.println (TIM1->CCMR3, HEX);
        Serial.print("CCR5: 0x"); Serial.println (TIM1->CCR5, HEX);
        Serial.print("CCR6: 0x"); Serial.println (TIM1->CCR6, HEX);
        Serial.print("AF1: 0x"); Serial.println (TIM1->AF1, HEX);
        Serial.print("AF2: 0x"); Serial.println (TIM1->AF2, HEX);
        Serial.print("TISEL: 0x"); Serial.println (TIM1->TISEL, HEX);
    }
}

void loop() {
    // put your main code here, to run repeatedly:
    digitalWrite(toggle_pin, !digitalRead(toggle_pin));
    digitalWrite(toggle_pin, !digitalRead(toggle_pin));
    for (int i = 0; i < 256; i += 32) {
        analogWrite(analog_pin, i);
        digitalWrite(toggle_pin, !digitalRead(toggle_pin));
        maybe_pause();
        delay(100);
    }

    digitalWrite(toggle_pin, !digitalRead(toggle_pin));
    digitalWrite(toggle_pin, !digitalRead(toggle_pin));
    for (int i = 255; i >= 0; i -= 32) {
        analogWrite(analog_pin, i);
        digitalWrite(toggle_pin, !digitalRead(toggle_pin));
        maybe_pause();
        delay(100);
    }
    delay(50);
}

Output for registers on TIm1 for both MBED and Zypher

MBED			Zephyr

CR1: 0x1		CR1: 0x81
CR2: 0x0		CR2: 0x0
SMCR: 0x0		SMCR: 0x0
DIER: 0x0		DIER: 0x0
SR: 0x3001F		SR: 0x3001F
EGR: 0x0		EGR: 0x0
CCMR1: 0x68		CCMR1: 0x68
CCMR2: 0x0		CCMR2: 0x0
CCER: 0x1		CCER: 0x1
CNT: 0xFA		CNT: 0x244F
PSC: 0xEF		PSC: 0x0
ARR: 0x7CF		ARR: 0x2EDF
RCR: 0x0		RCR: 0x0
CCR1: 0x5DA		CCR1: 0x292C
CCR2: 0x0		CCR2: 0x0
CCR3: 0x0		CCR3: 0x0
CCR4: 0x0		CCR4: 0x0
BDTR: 0x8000	BDTR: 0x8000
DCR: 0x0		DCR: 0x0
DMAR: 0x1		DMAR: 0x81
CCMR3: 0x0		CCMR3: 0x0
CCR5: 0x0		CCR5: 0x0
CCR6: 0x0		CCR6: 0x0
AF1: 0x1		AF1: 0x1
AF2: 0x1		AF2: 0x1
TISEL: 0x0		TISEL: 0x0
More debugging tomorrow, then adding more pins on multiple timers