os: Pipe can't be inherited on windows

What version of Go are you using (go version)?
go version go1.8.3 windows/amd64

What operating system and processor architecture are you using (go env)?
set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\light\go
set GORACE=
set GOROOT=C:\Go
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0
set CXX=g++
set CGO_ENABLED=1
set PKG_CONFIG=pkg-config
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2

What did you do?
https://play.golang.org/p/r5V40AgLht
What did you expect to see?
Parent process
Child process
Hello, World!

I would like to be able to pass an unnamed piped into a subprocess.  According to MSDN docs, this seems like something that Windows is capable of, but it is done using handles, not file descriptors IIUC.
What did you see instead?
Parent process
fork/exec C:\Users\light\AppData\Local\Temp\2\go-build579847834\command-line-arguments\_obj\exe\foo.exe: not supported by windows
exit status 1

I'm not exactly sure about discovery either.  I'm more just observing that this is how os/exec says that things will occur, without much of an indication of how it works on Windows.  In particular, from Cmd.ExtraFiles doc:

ExtraFiles specifies additional open files to be inherited by the new process. It does not include standard input, standard output, or standard error. If non-nil, entry i becomes file descriptor 3+i.

It seems unfortunate to be unable to send pipes to subprocesses on Windows.  I think we should address that somehow.

I believe you can do it today by calling the Fd method to get the handle, calling SetHandleInformation(handle, HANDLE_FLAG_INHERIT, 1) to let it stay open in the child, and then passing the handle to the child as an argument.  For the os/exec package, it's the last step that is the tricky part.  We could invent some way to pass the files to a Go program, but I don't know of any general purpose mechanism that would work for a non-Go program.  As far as I know the program basically has to receive the handle number in a command line argument or environment variable.

Unfortunately what Ian said is correct. os/exec.Cmd.ExtraFiles does not work on Windows. We use Windows CreateProcess API to start new process. CreateProcess accepts STARTUPINFO struct that has only hStdInput, hStdOutput and hStdError fields. You can pass any file handle from one process to the other (just like on Unix), but you need some way for processes to exchange file handle numbers (stdio or TCP or ...). We could try and do something with os/exec.Cmd.ExtraFiles on Windows, but the problem is the child process - there is no general mechanism to pass that information between processes. Windows file handle do not have specific numbers, for example stdin is not 0, stdout is not 1 and so on.

I believe you can do it today by calling the Fd method to get the handle, calling SetHandleInformation(handle, HANDLE_FLAG_INHERIT, 1) to let it stay open in the child, and then passing the handle to the child as an argument.

Normally you would use DuplicateHandle API to do that. It provides everything you might need just for that job.
Alex

Makes sense.  I'd like to send a doc fix to Cmd.ExtraFiles to summarize this discussion.
One more question though: if SetHandleInformation enables inheriting handles, would it make sense to amend ExtraFiles to set this option on all the file handles?  I understand that there's still an out-of-band mechanism that's required to pass the individual handles, but in some ways, ExtraFiles already requires some arrangement with the child process.  Clearly, this would need to be communicated in the doc comment how this works, but there's already difference between Windows and Unixy behavior for this field.

if SetHandleInformation enables inheriting handles, would it make sense to amend the behavior of ExtraFiles on Windows to set this option on all the file handles in the ExtraFiles slice?

I don't think it is that simple. We use CreateProcess Windows API to start new process. CreateProcess has bInheritHandles parameter that controls which handles are inherited by the child process. Go syscall package sets bInheritHandles to true because it needs to pass stdin, stdout and stderr. But this makes all inheritable handles escaped into child process (see the doco https://msdn.microsoft.com/en-us/library/windows/desktop/ms682425(v=vs.85).aspx : "... If this parameter TRUE, each inheritable handle in the calling process is inherited by the new process. ..."). We cannot allow that, so none of file handles inside Go are inheritable. We only make stdin, stdout and stderr for new child inheritable, just before we launch new process. We also have a lock that prevents us from running CreateProcess simultaneously, otherwise this would allow these 3 handles (potentially) end up in the wrong process.
What you are proposing - calling SetHandleInformation - will make that handle inheritable, and it can possibly escape into wrong process. We could use DuplicateHandle API instead (just before we launch the process), and then close new handle immediately. But DuplicateHandle will create new file handle with new handle number. How are you going to pass that number to the user of  Cmd.ExtraFiles?

ExtraFiles already requires some arrangement with the child process on Unix systems

Not true. I could be wrong because I do not know much about Unix, but whatever file is in ExtraFiles[0] will have file handle of 3 in the child process, ExtraFiles[1] will be 4, and so on. So everyone knows how it works, and you don't need to pass these numbers around.
Alex

This is simple implementation that passing socket into external process which works on Windows.
https://github.com/mattn/gospel
You can test this like below.
gospel echo-server

This pass the socket handle via environment variable GOSPEL_FD. To accept using the handle for the target process (limited), you should use DuplicateHandle.

https://github.com/mattn/gospel

This actually uses WSADuplicateSocket and WSAStartup to pass network socket between processes. These functions use WSAPROTOCOL_INFO struct (instead of file handles) to do the job.
Alex

@alexbrainman

What you are proposing - calling SetHandleInformation - will make that handle inheritable, and it can possibly escape into wrong process.

Seems to me like if you use SetHandleInformation to make the handle inheritable, create the process, then use SetHandleInformation to make the handle uninheritable again, all while holding ForkLock, you wouldn't have this risk. Am I missing something?
Then you could just pass the same handles that you gave to ExtraFiles to the child process via arguments, environment variables, etc.

ExtraFiles already requires some arrangement with the child process on Unix systems

Not true. I could be wrong because I do not know much about Unix, but whatever file is in ExtraFiles[0] will have file handle of 3 in the child process, ExtraFiles[1] will be 4, and so on. So everyone knows how it works, and you don't need to pass these numbers around.

I would say there's still an out-of-band arrangement between the processes, it can just usually be made at compile-time for Unix since the file descriptors are predictable. But that's neither here nor there. This field already acts in non-portable ways, so surely allowing the behavior to be "on Windows, file descriptors are unchanged, and on Unix, entry i becomes file descriptor 3+i" would be reasonable.

Seems to me like if you use SetHandleInformation to make the handle inheritable, create the process, then use SetHandleInformation to make the handle uninheritable again, all while holding ForkLock, you wouldn't have this risk. Am I missing something?

I think what you are suggesting should work.

surely allowing the behavior to be "on Windows, file descriptors are unchanged, and on Unix, entry i becomes file descriptor 3+i" would be reasonable.

Sure we could do that. Perhaps we should say more than "file descriptors are unchanged". It is very important that child deals with these file handles - at least close them, if it won't use them. Otherwise unpredictable things will happen.
Alex

I am by no means an expert on Windows, but it looks like an answer involves the _get_osfhandle system call.
I'm trying to spawn a Go binary from a Node program with an extra file.  I found that Node supports this (parent and child) and that it called the _get_osfhandle syscall in the child (passing, say, 3 as an argument) to translate into something that works with underlying calls.  Turned out doing the same in Go works OK too:
	// Strangely, it always returns non-nil error, so we ignore it. If we got an
	// error, the next step (writing to the fd) will fail anyway.
	value, _, _ := syscall.NewLazyDLL("msvcrt").NewProc("_get_osfhandle").Call(fd)
	return uintptr(value)

Calling this with fd = 3 returns something that can be passed to os.NewFile and seems to work properly, at least when the pipe is set up by Node.
I don't know if this will lead to a more generic solution for os/exec, but maybe this will help others dealing with this issue!

I found that Node supports this (parent and child) and that it called the _get_osfhandle syscall in the child (passing, say, 3 as an argument) to translate into something that works with underlying calls.

Yes, like I said before, if parent and child agree on how to pass file handles, that should work.

syscall.NewLazyDLL("msvcrt").NewProc("_get_osfhandle").Call(fd)

Yes that should work. Unless some computers do not have msvcrt.dll.
Alex

It is possible to pass file/pipe handles and some internal flags to child process using STARTUPINFO.lpReserved2. This is an undocumented feature inside MSVC C runtime. I have an unbuffer.py to demonstrate how to unbuffer stdout of child process using this idea. Maybe we can mimic this undocumented feature and implement ExtraFiles on Windows.

Maybe we can mimic this undocumented feature and implement ExtraFiles on Windows.

I don't see how that is possible. How would that work?
Alex

Your program is assuming that file descriptor 3 corresponds to the file passed in the ExtraFiles field. That is how ExtraFiles is designed to work, but while that works on Unix, it does not work on Windows.

What's _get_osfhandle, then?
https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/get-osfhandle?view=vs-2017
I haven't even attempted to read a significant fraction of Chrome's source code, and I'm definitely not a Windows developer, but.. Chrome's --remote-debugging-pipe flag seems to use fds 3 and 4 to communicate with the parent process, and it seems like it's implementation uses _get_osfhandle on Windows.
https://cs.chromium.org/search/?q=_get_osfhandle&type=cs

Here's evidence of a node.js program happily passing fds 3 & 4 to a chrome --remote-debugging-pipe subprocess, and this project claims to work on Windows: https://github.com/GoogleChrome/puppeteer/blob/7001042f796426163f13ca9716408578f1118bf3/lib/Launcher.js#L124-L135

What's _get_osfhandle, then?

I do not know.
Alex

Came across this when tackling a similar issue. In short, starting Windows Vista, STARTUPINFOEX is supported which adds lpAttributeList to pass handles.

@zx2c4 This issue was closed by https://golang.org/cl/288299, I think these lines should be removed too:
https://github.com/golang/go/blob/770f1de8c54256d5b17447028e47b201ba8e62c8/src/os/exec/exec.go#L121-L122
https://github.com/golang/go/blob/770f1de8c54256d5b17447028e47b201ba8e62c8/src/syscall/exec_windows.go#L265-L267
But it seems that the CL only takes care of the parent process. It's still not clear how to make it work for the child process. I have tried the demo in the OP (https://play.golang.org/p/r5V40AgLht) with these changes, and the output is:
Parent process
Child process
read <pipe>: The handle is invalid.
exit status 1

Then I modified the demo as below, and it works.
 func child() error {
    fmt.Fprintln(os.Stderr, "Child process")
-   r := os.NewFile(3, "<pipe>")
+   r := os.NewFile(os.Stderr.Fd()+4, "<pipe>")
    _, err := io.Copy(os.Stdout, r)
    return err
 }

I understand that this is not a reliable way to get the fd in the child process, I just want to prove that the CL works for the parent process somehow.
Then I tried to use the  ExtraFiles to communicate with Chrome (with the --remote-debugging-pipe flag). But it did not work. As pointed out in the previous comments, Chrome use _get_osfhandle to get the handle on Windows (I'm not familiar with the source code of Chrome, but I think it's here). Maybe we should change the Go implementation to support _get_osfhandle? If we do this, we will have a reliable way to get fd in the child process.
FWIW, some of the tests in os/exec/exec_test.go are still skipped on Windows. So there is something more to do here. Should we reopen this issue or file a new one?


bump up to iron 0.6

Apparently this broke the build, please fix

Yes, it is broken now because iron-test must be upgraded, that requires urlencoded and body-parser to be upgraded... I set it as WIP for now.

urlencoded has been bumped, this should work

It is blocked by iron-test
On Wed, Nov 22, 2017 at 09:26:17PM +0000, kyegupov wrote:

urlencoded has been bumped, this should work
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/iron/staticfile/pull/100#issuecomment-346477601

iron-test 0.6.0 has been released, maybe the PR would build now if we re-trigger the build.

This builds locally for me, even without the "range versioning" commit.

This is the last thread tying my project to iron 0.5, would be really great to lose it.

@untitaker can you rerun the build please?

Thx!


Breaks with import assertions (assert { type: 'json' })

What version of prettier-plugin-tailwindcss are you using?
0.5.11
What version of Tailwind CSS are you using?
3.4.1
What version of Node.js are you using?
LTS
What package manager are you using?
pnpm
What operating system are you using?
macOS
Describe your issue
As per this issue: https://github.com/trivago/prettier-plugin-sort-imports/issues/270
The sort imports plugin included by this plugin doesn't work properly with import assertions. My project does not use the import sorting plugin, so this should not impact me - but it does. Prettier crashes when trying to format any file with an import assertion with the error caused by the above plugin:
["ERROR" - 11:37:33] Error formatting document.
SyntaxError: This experimental syntax requires enabling one of the following parser plugin(s): "importAttributes", "importAssertions". (3:48)

Note that we don't include any plugins — if you don't have the sorting plugin installed then support for it won't be loaded — but this is probably is related to the use of the babel parser elsewhere. I'll take a look. Thanks!

I completely understand and agree, which is why I was so surprised to be impacted by this. I am not using either @trivago/prettier-plugin-sort-imports or @ianvs/prettier-plugin-sort-imports in my project - they are installed as peer deps of prettier-plugin-tailwindcss.
My prettier config, for reference:
import { fileURLToPath } from "url";

/** @typedef {import("prettier").Config} PrettierConfig */
/** @typedef {import("prettier-plugin-tailwindcss").PluginOptions} TailwindConfig */

/** @type { PrettierConfig | TailwindConfig } */
const config = {
  plugins: [
    "prettier-plugin-tailwindcss",
  ],
  tailwindConfig: fileURLToPath(
    new URL("../../tooling/tailwind/fresco.ts", import.meta.url),
  ),
  tailwindFunctions: ["cn", "cva"],
  printWidth: 80,
  quoteProps: 'consistent',
  singleQuote: true,
};

export default config;

You're absolutely correct that this doesn't repro - I tried now with a fresh repository and things are working correctly.
I can only assume that because I was working in a monorepo the configuration changes I made to the tailwind package were not being propagated correctly until after I restarted/ran pnpm install again.
Really sorry for the noise! Thanks for all your work.


Link to embedded video

on my landing page I want to autoplay mp4 that is in my assets folder.
~assets or ~/assets result in 404. If I use hosted version then works fine (eg. http://vjs.zencdn.net/v/oceans.mp4)
What am I missing here ?
Thanks

I swapped the loader for the build extend config option to add url-loader for mp4. That works ok.

could you show your solution code please? I have meet the same issue

Hi, I used the extend option to load video & source attributes from nuxt.config.js.
Then my above code to display embedded video works fine.


Uneven distribution of shards using node-hashring

I use node-hashring on a Redis environment where I have 128 instances of redis in a sharded environment.  Half of the instances live on one box, and the other half live on the second box.  For some reason, the instances living on the second box seem to be getting twice as much data as the instances on the first box.
I did not use weights, but it is as if all the instances on the second box had a weight of 2 whereas the instances on the first box had a weight of 1.
Do you have any explanation as to what might be happening?

Hello,
I don't have an explanation for it.. Does it get twice as much data, or twice as much keys?

What we noticed was that of the 128 nodes, it seemed like the first 64 nodes averaged about half of the number of keys than the later 64 nodes.  This resulted in twice as much data.  You can test the basic case by iterating from 0 to some large number, and see the way it gets distributed.  What I would expect is that nodes distribute fairly evenly, but that doesn't happen.  I will follow up with some sample test results that we found.

I don't see where I could leave an attachment so I will paste the code we were using to test.  You can try it out and see what kind of results.  We tried a couple of string hashing algorithms, and so far they all do the same thing where certain nodes get significantly more than other nodes.  The results do not follow the typical bell curve that you get if you just generated a bunch fo random numbers between 1 and 128 for example.  Changing the the hashing algorithm or the max # of nodes didn't change things too much.  Very weird.

//----------------------------------------------------------------------------------------------------------------
// testhashLinear.js
//
// Build a hashring of 128 nodes, hashing random integers
//
// Using hashring.js or djb2, the nodes are skewed and unbalanced.
//
// This test computes which bucket a number hashes into, keeping track of the frequency, and dumping it at the end
// of the test.  Several different hash functions are included, and commented out in testLinearKeys
//
//----------------------------------------------------------------------------------------------------------------
var fs = require('fs');
var sys = require('sys');
var HashRing = require("hashring");
var StringDecoder = require('string_decoder').StringDecoder;
var crypto = require('crypto');
var nodemap = ["RedisShard-01:50001", "RedisShard-01:50002", "RedisShard-01:50003", "RedisShard-01:50004", "RedisShard-01:50005", "RedisShard-01:50006", "RedisShard-01:50007", "RedisShard-01:50008", "RedisShard-01:50009", "RedisShard-01:50010", "RedisShard-01:50011", "RedisShard-01:50012", "RedisShard-01:50013", "RedisShard-01:50014", "RedisShard-01:50015", "RedisShard-01:50016", "RedisShard-01:50017", "RedisShard-01:50018", "RedisShard-01:50019", "RedisShard-01:50020", "RedisShard-01:50021", "RedisShard-01:50022", "RedisShard-01:50023", "RedisShard-01:50024", "RedisShard-01:50025", "RedisShard-01:50026", "RedisShard-01:50027", "RedisShard-01:50028", "RedisShard-01:50029", "RedisShard-01:50030", "RedisShard-01:50031", "RedisShard-01:50032", "RedisShard-01:50033", "RedisShard-01:50034", "RedisShard-01:50035", "RedisShard-01:50036", "RedisShard-01:50037", "RedisShard-01:50038", "RedisShard-01:50039", "RedisShard-01:50040", "RedisShard-01:50041", "RedisShard-01:50042", "RedisShard-01:50043", "RedisShard-01:50044", "RedisShard-01:50045", "RedisShard-01:50046", "RedisShard-01:50047", "RedisShard-01:50048", "RedisShard-01:50049", "RedisShard-01:50050", "RedisShard-01:50051", "RedisShard-01:50052", "RedisShard-01:50053", "RedisShard-01:50054", "RedisShard-01:50055", "RedisShard-01:50056", "RedisShard-01:50057", "RedisShard-01:50058", "RedisShard-01:50059", "RedisShard-01:50060", "RedisShard-01:50061", "RedisShard-01:50062", "RedisShard-01:50063", "RedisShard-01:50064", "RedisShard-03:50065", "RedisShard-03:50066", "RedisShard-03:50067", "RedisShard-03:50068", "RedisShard-03:50069", "RedisShard-03:50070", "RedisShard-03:50071", "RedisShard-03:50072", "RedisShard-03:50073", "RedisShard-03:50074", "RedisShard-03:50075", "RedisShard-03:50076", "RedisShard-03:50077", "RedisShard-03:50078", "RedisShard-03:50079", "RedisShard-03:50080", "RedisShard-03:50081", "RedisShard-03:50082", "RedisShard-03:50083", "RedisShard-03:50084", "RedisShard-03:50085", "RedisShard-03:50086", "RedisShard-03:50087", "RedisShard-03:50088", "RedisShard-03:50089", "RedisShard-03:50090", "RedisShard-03:50091", "RedisShard-03:50092", "RedisShard-03:50093", "RedisShard-03:50094", "RedisShard-03:50095", "RedisShard-03:50096", "RedisShard-03:50097", "RedisShard-03:50098", "RedisShard-03:50099", "RedisShard-03:50100", "RedisShard-03:50101", "RedisShard-03:50102", "RedisShard-03:50103", "RedisShard-03:50104", "RedisShard-03:50105", "RedisShard-03:50106", "RedisShard-03:50107", "RedisShard-03:50108", "RedisShard-03:50109", "RedisShard-03:50110", "RedisShard-03:50111", "RedisShard-03:50112", "RedisShard-03:50113", "RedisShard-03:50114", "RedisShard-03:50115", "RedisShard-03:50116", "RedisShard-03:50117", "RedisShard-03:50118", "RedisShard-03:50119", "RedisShard-03:50120", "RedisShard-03:50121", "RedisShard-03:50122", "RedisShard-03:50123", "RedisShard-03:50124", "RedisShard-03:50125", "RedisShard-03:50126", "RedisShard-03:50127", "RedisShard-03:50128"];
var optionsJSON = {
vnode_count:0
}
//var ring = new HashRing(nodemap,'MD5',optionsJSON);
var ring = new HashRing(nodemap);
var gKeys = [];
//----------------------------------------------------------------------------------------------------------------
function getIndexForStr(computedIdStr) {
for (var i=0; i < gKeys.length; ++i) {
if (gKeys[i]._key == computedIdStr) {
return i;
}
}
var newIndex = gKeys.length;

gKeys[newIndex]= { "_key":computedIdStr, "_frequency":0 }

return newIndex;

}
//----------------------------------------------------------------------------------------------------------------
// Used to compare against hashring + crc32, nearly identical results
function hash2(hashKey) {
var hash = 0;
for (i = 0; i < hashKey.length; i++) {
    var charCode = hashKey.charCodeAt(i);
    hash = ((hash<<5)-hash)+charCode;
    hash = hash & hash; // Convert to 32bit integer
}

return hash;

}
//----------------------------------------------------------------------------------------------------------------
// Used to compare against hashring + crc32, nearly identical results
function djb2(hashKey) {
var hash = 5381;

for (var i=0; i < hashKey.length; ++i) {
    var c = hashKey.charCodeAt(i);
    hash = ((hash << 5) + hash) + c;
}

return hash;

}
function randomMinToMax(min, max) {
var range = max-min+1;
return Math.floor(Math.random()*range+min);
}
//----------------------------------------------------------------------------------------------------------------
function testLinearKeys(startIndex, numKeys) {
console.log('---------------');
console.log('testLinearKeys startIndex='+startIndex+' numKeys='+numKeys);
console.log('---------------');
var startTime = (new Date()).getTime();

gKeys = [];
for (var i=startIndex; i < startIndex+numKeys; ++i) {
    //var hashkey = ''+i;
    var hashkeyInt = randomMinToMax(0,100000000);
    var hashkey = ''+hashkeyInt;

    // hashring.js test
    var computedHash = ring.getNode(hashkey)

    // Should be perfectly balanced, for comparison
    //var computedHash = nodemap[i%(nodemap.length)];

    // rand distribution, not perfecly balanced but relatively close
    //var computedHash = nodemap[hashkeyInt%(nodemap.length)];

    // using djb2
    // var computedHashRaw = djb2(hashkey);
    // var computedHash = computedHashRaw%(nodemap.length);

    // using hash2
    //var computedHashRaw = Math.abs(hash2(hashkey));
    //var computedHash = computedHashRaw%(nodemap.length);

    //console.log('computedHash='+computedHash)
    // histogram the result
    var idx = getIndexForStr(computedHash)
    gKeys[idx]._frequency += 1;
}

var endTime = (new Date()).getTime();
console.log ('etime = '+(endTime-startTime)/1000+' seconds');

}
//----------------------------------------------------------------------------------------------------------------
function dumpStats() {
console.log('dumpStats gKeys.length='+gKeys.length)

 gKeys.sort(function(itemA, itemB) {

     if (itemA._frequency > itemB._frequency) {
         return -1;
     } else if (itemA._frequency < itemB._frequency) {
         return 1;
     } else {
         return 0;
     }

});

for (var i=0; i < gKeys.length; ++i) {
    console.log(gKeys[i]._key+' \t\t_frequency= '+gKeys[i]._frequency)
}

}
//----------------------------------------------------------------------------------------------------------------
function buildRecord(str){
var record = {}
str.split(pattern).forEach(function(value, index){
if(header[index] != '')
record[header[index].toLowerCase()] = value.replace(/"/g, '')
})
return record
}
//----------------------------------------------------------------------------------------------------------------
console.log('testhashLinear start');
console.log('nodemap.length='+nodemap.length);
console.log('---------------');
testLinearKeys(0,100000);
dumpStats();
console.log('---------------');
console.log('testhashLinear end');

Thanks a lot, I'll it a look to night.

It looks to me like a couple of things are happening here. First, the hash keyspace is getting decimated during the hashing process.
In particular, return CreateHash(this.algorithm).update(key).digest('hex').split('').map(function(v){ return v.charCodeAt(0) }) does not return the hex value of each character in the hash, but rather the character code for that value which is not a compact nor hexadecimal value.
What you actually want there is something like return +('0x'+v[0]) which takes the first element of v and forces it to be interpreted as a hex string.
Secondly, the current hashValue function seems wrong, since each element is only a single hex character (a nibble not a byte), so you actually need to read two hex characters for every byte in the hash.
This is further complicated by the original Ketema implementation being little endian. So it should be something like:
(key[31-offset(0)]) | (key[31-offset(1)] << 4) | (key[31-offset(2)] << 8) | (key[31-offset(3)] << 12) | (key[31-offset(4)] << 16) | (key[31-offset(5)] << 20) | (key[31-offset(6)] << 24) | (key[31-offset(7)] << 28)
And then the offset function becomes function(x) { return x + k*8 };
This makes md5 work, and if I got the byte ordering right, then it'll map the same as vanilla libketama.
As a side-note, the vnode code is also broken -- you unconditionally override the passed options at this.options = {vnode_count: 40}; and then the weight code factor = Math.floor((this.options.vnode_count * len * weight) / totalweight); should not be dependent on len (that makes each subsequent server get a larger chunk of space as we iterate from 0 to len.
That said:
After I fixed all those things, I still had key distribution issues (down to 600 min and 1000 max per server, which isn't terrible). After further investigation the method Ketama uses to decimate the md5 hash to a hash_value is pretty crappy (as the c++ implementations also show this distribution issue). I'm working on a general solution for that and will report back if I make any progress.
Cheers,
-S

Yeah. It depends which problem you're trying to solve. CRC has a better 'looking' distribution but it has very poor quality bit 'dispersion', and will have serious trouble hashing certain keyspaces.
See:
http://home.comcast.net/~bretm/hash/5.html
http://home.comcast.net/~bretm/hash/8.html
It looks like the problem with cryptographic hashes is that with any reasonable number of server points (ie. ~160 as in the original spec) the distribution is still such that the ratio of max count(keys) to min count(keys) across the 128 individual nodes is 1.6:1. This clearly isn't efficient to scale in a large production environment! The problem here is that the hash-space for the keys is randomly distributed but, for lack of a better term, sort of like a fractal generator. You get more and more slices, but they do not trend to a common size.
I'm actually looking at fixing the distribution problem (statistics to the rescue! cc: law of large numbers) by using a great deal more vpoints per server (~4000), and then using the fastest, well-distributed hash I can find (current candidates are MurmurHash3 which on x64 gives you very cheap 128-bit hashes).
So far, it's showing a reasonable performance impact (<10%) and a great deal better distribution (better than 1.1:1).
Anyway, gonna keep working on this and if I figure out anything else useful, I'll let you know.

@joecroninallen can I have permission to reuse that snipped of code as a test case? You didn't add any license so I have to assume not otherwise. #SadPanda


Fixes types for federation

Related https://github.com/dotansimha/graphql-code-generator/issues/4722

Thank you @KoltesDigital . Can you please add a changeset file? (run yarn changeset, it will guide you). I think we need a patch bump in this PR.
@jakeblaxon it's related to changes you recently added. Do you mind give it a quick look?

I've looked over the changes, but it looks like this PR would cause the issues in #4493 to reappear. A major reason is that it changes the transformParentType to only address the resolveReference field, but not any of the other fields. It also eliminates checking if the type is an extended (external) federation type or a base type, which is necessary to resolve the above issue.
Looking at #4722, I see that you're using mapped types. I didn't address the mapped type case at all when working on adding this federation functionality, so it may be buggy and there is definitely room for improvement there. For your fullName resolver, however, the reason you're not able to access the firstName and lastName types from the external User parent type is because you haven't specified that they are required by adding them to a @requires directive. You can solve that issue by adding that:
extend type User @key(fields: "id") {
    id: ID! @external
    firstName: String! @external
    lastName: String! @external
    fullName: String! @requires(fields: "firstName lastName")
}

(You will need to add all required  external fields to your extended type, otherwise the Apollo Gateway will throw an error, hence why I've added them here). If you don't specify that these fields are required, then the Apollo Gateway may not provide them to you for certain requests. For example, what if I requested the following:
{
  getUser(...) {
    id
    firstName
    fullName
  }
}

In this case, the Apollo Gateway will only request { id firstName } from the base schema and pass this on as the parent type to your fullName resolver. Since only the id is specified as a key, there's no reason the Apollo Gateway would need to request lastName from the base schema. It doesn't know that you need it for your resolver. That's why you need to tell this to the Apollo Gateway with the @requires directive.
Since User is an extended type, I believe the __resolveReference resolver will never get called (although I have to verify this). Thinking about it intuitively though, each custom field that you add to an extended type will have its own resolver, so the Apollo Gateway will just call these field-specific resolvers instead of _resolveReference. It really only makes sense to have a __resolveReference in the base type, and we already account for this (as of #4542). Again, the __resolveReference resolver gets called directly from Apollo Gateway, where the parent type is the User field not from the base schema but from a schema that extends it. The only guaranteed fields that are available in this case are the @key fields, because these are the only fields that are guaranteed to be in this type across every schema. That's why we only pass in the @key fields subset of the parentType for the __resolveReference resolver.
With this in mind, I would recommend closing this PR without merging and also closing #4722 as a misunderstanding (@dotansimha ). Please let me know if you have any questions regarding this though, and I'll be happy to help out more. This is complicated and counterintuitive stuff, so I definitely don't blame you for any misunderstandings. I myself have only gotten familiar with these details through many months of intimate trial and error. Thanks!

I disagree with your understanding of federation. My understanding is as follow.
With or without federation, the parent type is not required at all to mirror the GraphQL entity type. My example of User, without the federation, would be correct anyway: the JS object has firstName and lastName properties, there's a resolver fullName, a GraphQL query can only retrieve the fullName, that's totally correct. I will call them JS model and GraphQL model, they may overlap but they don't have to.
Now, Apollo Federation exposes entities whose fields are served by different services. These different services may have different JS models, so they need to bind one to one. That's the purpose of __resolveReference. This is the entry point in a service. This function receives a "plain" object with fields corresponding to @keys, and returns another object (JS model) which is latter used as the first argument of related resolvers in that service.
In this example, __resolveReference returns a JS model, which is used by a resolver to return a value to a GraphQL query. Perfectly valid, no @requires needed.
Speaking of users and names, I admit it makes sense to have all these fields, but it's just the easiest example I found to show the bug. I just didn't wanted to call these fields foo and bar. But the basic principle is: JS models and GraphQL models are distinct, though they may overlap, they don't have to.
Besides, the example is based on the project I'm working on for several months, with currently five federated services. For now the types are written by hand and it works as I claim. With the current codegen, it doesn't compile anymore. With my patch, it compiles (and the generated JS is obviously the same, as it's only an issue with TS type declarations).
You can easily code a new project which replicates this example and verify my claims.

That being said, I admit that this patch is based on my experience that every service have strong JS models. This might not always be the case: __resolveReference is optional, and if not present, the resolvers are given the "plain object" as first argument (i.e. __resolveReference is an intermediary transform function). My experience is that this plain object is only the primary key of the JS model, which is always bigger than the key (otherwise there's no need for this service), so all my extended entities always have a __resolveReference.
If we want to support both cases, I believe we need to distinct whether an entry is given in the mapper config. If there is a parent type (JS model), it means a __resolveReference needs to be defined to get the JS model. If there is not, it means the resolvers expect the "plain object" as first argument, in which case __resolveReference is not needed.

Regarding #4493 (which I haven't searched for earlier, sorry!) I believe there is a modeling issue which leads to (or comes from) a wrong understanding of federation or even GraphQL itself. Service 1 should not give propertyId, as it gives property which has an id key. So both getRoomType(...) { propertyId } and  getRoomType(...) { property { id } } give the same id, and is only executed on service 1. Service 2 is involved as soon as a field it provides is queried (e.g. getRoomType(...) { property { id name } }).
Searching online for "GraphQL best practices" give the same advice: https://atheros.ai/blog/graphql-best-practices-for-graphql-schema-design , https://towardsdatascience.com/graphql-best-practices-3fda586538c4 , etc.
Given that, the JS model has a propertyId field, the GraphQL model has a property field, there's a property resolver which returns an object which at least contains the primary key fields (id in this case) so that the other services can bind to their own local JS models of Property. Perfectly valid, no @requires needed.

Ahh, I see what you're saying now. I was assuming that lastName and firstName were fields from the base type, but they're actually gotten from the resolveReference from the extended type. I admit I've never used a __resolveReference for an extended federation type before, so I did not know this was possible. Knowing this now, I see two cases that we need to address for an extended federation type:

If there is no __resolveReference provided, then use the @key @requires subset as the parent type for each resolver.
If __resolveReference is provided, then just use the full parent type without taking a subset (which was the default behavior before my PR). This will use the mapped type if it's provided, otherwise it'll just use the graphql type.

Right now, only case 1 is accounted for, so this PR should seek to add case 2 while preserving case 1. Does this sound correct to you? If so, let's start looking into ways that we can do this. In the meantime, though, I'd convert this PR to a draft since it would break case 1.

Actually, case 2 is a little more complicated since we can still use @requires for certain resolvers, and the required fields aren't necessarily in the mapped type. I think we'd have to combine the graphql model's @key @requires subset with the mapped type to make sure we cover all available fields. So the parent type for fullName in this case should be something like:
{ __typename: 'User' } & GraphQLRecursivePick<ParentType, {"id":true}> & MappedType

That way, if we provide firstName and lastName from __resolveReference but we also add a @requires(fields: "address") to the fullName field, we'll have access to address from the base type as well. In this case the parent type would look like:
{ __typename: 'User' } & GraphQLRecursivePick<ParentType, {"id":true, "address":true}> & MappedType

I need to test this to make sure this is what actually happens though. This gets pretty complicated intuitively when you can use __resolveReference and @requires together. Does the Apollo Gateway know to merge the required fields into the js object that __resolveReference returns?

The problem is, as a TS types provider, the tool can't generate types based on the way the types are used. I do not recommend a type union neither, because it would require adding a check in the TS file (otherwise TS would complain) but this check is useless because we know upfront that the parentType is either resolved or not.
Regarding @requires, this is tedious: __resolveReference gets the additional fields, and they are not later merged into the JS model! So if a GraphQL field requires other fields, the entity's JS model must have a way to forward these optional fields to the requested resolver. That's why in my patch __resolveReference optionally accepts any field required by any field, of the entity (see https://github.com/dotansimha/graphql-code-generator/blob/39f71c97418b0e28f6abc16d3c7ba5379089dab0/packages/plugins/typescript/resolvers/tests/federation.spec.ts#L119). The second case you listed does not actually happen.
That's why I mentioned relying on the mapper config key: if there's one, __resolveReference must be present and the resolvers' first argument is of ParentType, if not, __resolveReference should not and the resolvers' first argument is the same as for __resolveReference. What do you think about this solution?

To be even more spec-compliant: without __resolveReference, required fields are given to all resolvers anyway. For instance, this code would be valid, even though I doubt anyone would ever use this behavior.
extend type T {
  a: String!
  b: String! @requires(field: "a")
  c: String!
}

export const resolvers: RoomTypeSchemaResolvers<any> = {
  T: {
    c(parent) {
      return parent.a ? 'foo': 'bar';
    }
  }
};

Unless someone complains, I recommend to not care about this behavior :)

@jakeblaxon Hi! Any news?

I apologize for the delay on this. I just reviewed this thread and I agree with your solution. I think using the mapper config key to determine whether to require the __resolveReference resolver is a good idea. For those who use __resolveReference but don't use a mapped type, they can work around this by manually specifying the desired parent type of each resolver.
With this in mind, we should be able to use most of what's currently in your PR. We just need to make a few adjustments to preserve current functionality and add the conditional check. Do you want to make the changes? I'll be able to review pretty frequently over the next couple of weeks, so just let me know when changes are ready and I'll take a look as soon as I can. Thank you for your patience!

Hi! No worry, it's great to make progress on that. I'll try to make the change. I also remember that I was bothered by a test I changed but the "expected" result might not be what we actually expect, I'll have a look at that. And I'll make the changeset.

@jakeblaxon now as discussed.
One thing I'm bothered with is that all parent types are generated, even when they are overriden by custom mapped types. I use to name my models the same as their GraphQL counterparts, thus it would conflict. It looks like the generation of these types leads to the generation of the related <TypeName>Resolvers types. I don't know the project enough to be able to change that. How do you feel about that?

Hi! Is there any update on this?

I'm reviewing it currently

We try to avoid any breaking changes and the thing I pointed out is one. We're going to rewrite federation support from scratch soon.

Ok, I reviewed your changes and created a document about Federation and typescript-resolvers. Feedback needed https://www.notion.so/Federation-typescript-resolvers-3b47e825ff9a43b7a970c28e8784a6f7. This will help to rewrite the federation part in codegen.

Created this PR https://github.com/dotansimha/graphql-code-generator/pull/5645 to progress with Federation support step by step.

Is there anything I can do to rebase or fix this up for merge? The bugs this fixes affect my usage frequently and I'm available to help with this if there's something I can do.

@bsmedberg-xometry we've just rebased the PR here: https://github.com/dotansimha/graphql-code-generator/pull/5645
Would be great if you could pick this work up and get it to the finish line


UI component - Pagination (React version)

Description
Create the component for the react framework.
Design
https://xd.adobe.com/view/6d476aef-2324-4c9a-b42b-d28dfd0eb426-3814/screen/15191ec3-8efe-4729-9746-418ce779e6e0/
Acceptance Criteria

[ ] Component built to the specification of the design
[ ] Component documentation built to the specification of the design
[ ] Code pushed to Storybook
https://ui-components.alpha.alberta.ca/?path=/story/component-library-basic-elements-button-link--links

Completion Status

[ ] Web Component
[ ] React
[ ] Angular
[ ] Vue

Get in touch with @ambroseyau for updates and enhancements to designs and documentation.


calling cool method continues heating at 150

I am not able to get the cool method to stop heating, when using thermostate mode.
Without thermostate mode I can set the heater to 0, but in thermostat mode it just turns the temperature down to 150 degrees. I've tested taking off the FR unit top while in cool mode (careful not to burn myself!) and ran a temperature gun on the unit and it is indeed still heating to what I would guess is 150.
Is this a problem within my specific FRSR700 unit not turning off the heater when cool state is set, or a problem somewhere in freshroastsr700()?
Thanks in advance!

Not for developer: I decided to just control the heater myself using PID and setting "ext_sw_heater_drive=True" which allowed me to set the heater to 0 on cool.
If you get confirmation on this bug or can reproduce, hopefully it's helpful. If not feel free to close.


Extended CLI commands no longer appear in -h or --help output

Have you read the Contributing Guidelines on issues?

[X] I have read the Contributing Guidelines on issues.

Prerequisites

[X] I'm using the latest version of Docusaurus.
[ ] I have tried the npm run clear or yarn clear command.
[ ] I have tried rm -rf node_modules yarn.lock package-lock.json and re-installing packages.
[X] I have tried creating a repro with https://new.docusaurus.io.
[ ] I have read the console error message carefully (if applicable).

Description
Using extendCli to extend the docusaurus CLI with new commands no longer includes those commands, i.e. usage, description, arguments, options, when executing docusaurus --help or docusaurus -h.
However, the extended commands do appear in the console output after attempting to call an unknown command.
I verified that extended command usage does appear in prior docusaurus release, e.g. v2.4.3 but have not tested releases in between that and the current release.
Reproducible demo
https://codesandbox.io/p/devbox/elegant-water-docusaurus?privacy=public
Steps to reproduce

Bootstrap a fresh docusaurus site
Create the example plugin referenced in the extendCli docs.
Update docusaurus.config.js to include the new plugin.
Execute yarn docusaurus -h and observe the extended command is not present in help output.
Execute yarn docusaurus foo and observe the extended command in the output.

Expected behavior
When executing yarn docusaurus -h I expect to see my extended command usage, description, options, etc., in the output.
Actual behavior
Running yarn docusaurus -h not not include the usage for my extended command. However, my extended command does appear when I attempt to pass an invalid command argument.
Your environment

Public source code: https://github.com/PaloAltoNetworks/docusaurus-openapi-docs
Public site URL: https://docusaurus-openapi.tryingpan.dev/
Docusaurus version used: 3.6.0 and 3.6.1
Environment name and version (e.g. Chrome 89, Node.js 16.4): Chromium 130.0.6723.117  and Node 18.7
Operating system and version (e.g. Ubuntu 20.04.2 LTS): Mac OS Sonoma 14.7.1 (23H222)

Self-service

[X] I'd be willing to fix this bug myself.

CLI output from my repro:
$ docusaurus -h

    ------------------------------------------------------------------------------------------------------------------------------------------------

                                                             Update available 3.6.0 → 3.6.1

                                    To upgrade Docusaurus packages with the latest version, run the following command:
         `yarn upgrade @docusaurus/core@latest @docusaurus/preset-classic@latest @docusaurus/module-type-aliases@latest @docusaurus/types@latest`

    ------------------------------------------------------------------------------------------------------------------------------------------------

Usage: docusaurus <command> [options]

Options:
  -V, --version                                            output the version number
  -h, --help                                               display help for command

Commands:
  build [options] [siteDir]                                Build website.
  swizzle [options] [themeName] [componentName] [siteDir]  Wraps or ejects the original theme files into website folder for customization.
  deploy [options] [siteDir]                               Deploy website to GitHub pages.
  start [options] [siteDir]                                Start the development server.
  serve [options] [siteDir]                                Serve website locally.
  clear [siteDir]                                          Remove build artifacts.
  write-translations [options] [siteDir]                   Extract required translations of your site.
  write-heading-ids [options] [siteDir] [files...]         Generate heading ids in Markdown content.
Done in 1.56s.

➜  workspace git:(master) ✗ yarn docusaurus foo
yarn run v1.22.19
$ docusaurus foo

    ------------------------------------------------------------------------------------------------------------------------------------------------

                                                             Update available 3.6.0 → 3.6.1

                                    To upgrade Docusaurus packages with the latest version, run the following command:
         `yarn upgrade @docusaurus/core@latest @docusaurus/preset-classic@latest @docusaurus/module-type-aliases@latest @docusaurus/types@latest`

    ------------------------------------------------------------------------------------------------------------------------------------------------

Usage: docusaurus <command> [options]

Options:
  -V, --version                                            output the version number
  -h, --help                                               display help for command

Commands:
  build [options] [siteDir]                                Build website.
  swizzle [options] [themeName] [componentName] [siteDir]  Wraps or ejects the original theme files into website folder for customization.
  deploy [options] [siteDir]                               Deploy website to GitHub pages.
  start [options] [siteDir]                                Start the development server.
  serve [options] [siteDir]                                Serve website locally.
  clear [siteDir]                                          Remove build artifacts.
  write-translations [options] [siteDir]                   Extract required translations of your site.
  write-heading-ids [options] [siteDir] [files...]         Generate heading ids in Markdown content.
  docs:version <version>                                   Tag a new docs version
  roll                                                     Roll a random number between 1 and 1000
[ERROR] Unknown Docusaurus CLI command foo.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
➜  workspace git:(master) ✗

Thanks, I'll take a look
Looks like I broken something here https://github.com/facebook/docusaurus/pull/10368


When a new port is saved the container CMD changes to "bash"

Expected behavior
My docker container made from a custom centos6 image via the command line retains 'crond -n' as its run -d CMD when it's restarted after I add a new port in Kitematic and hit save.
Actual behavior
On restart the container CMD changes to 'bash' causing it to exit immediately
Information about the Issue
The container just constantly restarts because it was initially ran with run -d + restart=always and bash is now what it's trying to execute on start
Running a Linux container on Windows 10 Enterprise
Docker CE
Version 17.03.0-ce-win1 (10300)
Channel: beta
05a8abd
Kitematic
v0.35.4
Steps to reproduce the behavior

create a centos6 container on the command line: docker run -d  crond -n
Switch to Kitematic
add a port mapping
hit save
check container command: docker ps -a

Custom commands should be retained, as we use the existing container data to create the new one.
To be clear, a 'save' in kitematic is really a:

Get current/updated data
Destroy existing container
Create new container with updated data

It's possible that one of the cmd endpoint isn't being updated properly - Could you provide a container inspect of the container running your custom crond -n?

inspect.txt


unable to start single node

as per the README, it says I can start the node with the shell script:
./strato.sh --single
when doing so i get the following error:
Unable to find image 'registry-aws.blockapps.net:5000/blockapps-repo/strato:5.1.0' locally docker: Error response from daemon: Get https://registry-aws.blockapps.net:5000/v2/blockapps-repo/strato/manifests/5.1.0: unauthorized: authentication required.
here is the full log:
~/P/b/strato-getting-started ❯❯❯ ./strato.sh --single                                                                                ⏎ master

    ____  __           __   ___
   / __ )/ /___  _____/ /__/   |  ____  ____  _____
  / __  / / __ \/ ___/ //_/ /| | / __ \/ __ \/ ___/
 / /_/ / / /_/ / /__/ ,< / ___ |/ /_/ / /_/ (__  )
/_____/_/\____/\___/_/|_/_/  |_/ .___/ .___/____/
                              /_/   /_/

*** Common Config ***
NODE_HOST: localhost
HTTP_PORT: 80
HTTPS_PORT: 443
ssl: false
sslCertFileType: crt
NODE_NAME: localhost
BLOC_URL (for STRATO v4.2 backwards-compatibility): http://localhost/bloc/v2.2
BLOC_DOC_URL (for STRATO v4.2 backwards-compatibility): http://localhost/docs/?url=/bloc/v2.2/swagger.json
STRATO_URL (for STRATO v4.2 backwards-compatibility): http://localhost/strato-api/eth/v1.2
STRATO_DOC_URL (for STRATO v4.2 backwards-compatibility): http://localhost/docs/?url=/strato-api/eth/v1.2/swagger.json
CIRRUS_URL (for STRATO v4.2 backwards-compatibility): http://localhost/cirrus/search
APEX_URL (for STRATO v4.2 backwards-compatibility): http://localhost/apex-api
authBasic: false
uiPassword: not set
STRATO_GS_MODE: 0
SMD_MODE: enterprise
EXT_STORAGE_S3_BUCKET: not set
EXT_STORAGE_S3_ACCESS_KEY_ID: not set
EXT_STORAGE_S3_SECRET_ACCESS_KEY: not set
OAUTH_ENABLED: false
OAUTH_DISCOVERY_URL: not set
OAUTH_CLIENT_ID: not set
OAUTH_CLIENT_SECRET: not set
OAUTH_JWT_USERNAME_PROPERTY: not set (using default)
OAUTH_STRATO42_FALLBACK: false

Running single node with PBFT-blockstanbul
Unable to find image 'registry-aws.blockapps.net:5000/blockapps-repo/strato:5.1.0' locally
docker: Error response from daemon: Get https://registry-aws.blockapps.net:5000/v2/blockapps-repo/strato/manifests/5.1.0: unauthorized: authentication required.
See 'docker run --help'.

Hi @cryptoB0T ,
STRATO is a paid software. Unfortunately, we don’t provide the trial version anymore and STRATO deployment requires the paid access to the STRATO Docker registry. Another way would be to deploy STRATO node on one of these cloud marketplaces: AWS, GCP, Azure.
The Track-and-Trace app won’t deploy successfully unless the correct STRATO node url is provided in the config.
The issue of another person you mentioned is not related to the one you are experiencing and is either the permissions issue or the Unix distro factor.
Best regards,
Nikita Mendelbaum
DevOps Lead @ BlockApps


ci: test python 3.10

Tested python 3.10 via docker seems ok, just spinning a python 3.10 CI

scipy doesn't support 3.10 and numpy only in some case so until these move towards it I wouldn't even look at it.

makes sense

Looks liek scipy got a new release 3 days ago that support 3.10

why is the docker build also changing?

why is the docker build also changing?

It had wrong name before... now it is fixed. It is using 3.7


Documentation Link Not Found

I am not sure if this can be considered an issue but there is a little update required in the Wiki page.
There is a link given to download the sample of quantum in Phase 1 : Running Sample as is. This link is resulting into an error page with a "Page not found error."
This can be corrected so that it would be easier for beginners to start with.
This is the page where this link is displayed: https://github.com/Project-Quantum/Quantum-Starter-Kit/wiki/Getting-Started
Link which results in the error: https://github.com/PerfectoCode/Quantum
Thank You

Fixed


Add Unified Pipeline APIView Release

This is related to #7941
This PR eliminates the need for the following standalone releases, and ensures that APIView is compliant with security needs.

APIView UI Test Instance: Releases - Pipelines (azure.com)
Release to APIView Staging: Releases - Pipelines (azure.com)
APIView to Azure App Service: Releases - Pipelines (azure.com)

When queuing manually from internal, users will be prompted for a target environment prod or staging. staging enables access to #1 and #2 in the list above. prod enables #3 only.
Ready to test the actual releases.
New Stage Flow
Build and Test -> Staging (automatic) -> Production (approve required)
               -> Test UI (approve required)

Looks fine. Only additional change I prefer to have is to support deployment of just the UI test instance.
Currently following are the deployment patterns.

APIView UI test is deployed for some testing and demo purposes (Less frequent deployment).
APIView stage gets auto deployed anytime a new change is merged to main branch. So staging can be run pretty much as a CI job.
APIView production deployment is done manually.

Ok Praveen, I've made the following adjustments.

Staging auto publishes to staging if the build is from internal
Manually queue the build to get access to stage Publish Test UI and Publish to Production APIView

Awaiting one last deployment to production slot, then this is ready to merge.

Merging!


Add PR template

I would like to add a pull request template for this repository. I believe that having a standardized template will help streamline the contribution process, ensuring that all necessary information is included and making it easier for maintainers to review and merge pull requests.
Could you please assign this issue to me under GSSOC'24.

Its already created @suhanipaliwal


[WIP] mv to slurm

Coverage increased (+0.02%) to 92.853% when pulling 2d89e03a933e2446047351017bea94d4f5b5b5a5 on antgonza:slurm into 05f046e24ba39a1e784676aa3b86da9ae22fc099 on qiita-spots:dev.


Additional Response Asserts

Additional response checks are needed, for nested/more complicated results. For example:
/**
* Asserts that a response does not contain the entered key-value pairs.
* For arrays, this checks each object in the array.
*
* @param expectedPairs key-value pairs the should not be in the response
*/
public void assertNotContains(Map<String, String> expectedPairs)
/**
 * Verifies the response json payload contains an object at the key
 * that contains a nestedKey that contains a value
 * example response: { "books": { "id": 123 }, "author": "John Doe" }
 *
 * @param key       - top level json key eg. books in object above
 * @param nestedKey - key in nested object eg. "id" in object above
 * @param value     - must be String, Integer or Boolean
 */
public void assertContainsNested(String key, String nestedKey, Object value)

/**
 * Verifies the response does not contain a certain key-value pair.
 * For arrays, this checks each object in the array.
 *
 * @param key          key that should not be in the response
 * @param expectedJson value that the key should not have in the response
 */
public void assertNotContains(String key, JsonElement expectedJson)

/**
 * Verifies the response does not contain a certain key-value pair.
 * For arrays, this checks each object in the array.
 *
 * @param key key that should not be in the response
 */
public void assertNotContains(String key)

/**
 * Asserts that the response does not contain a certain JSON element.
 *
 * @param expectedJson JSON that should not be in the response
 */
public void assertNotContains(JsonElement expectedJson)

/**
 * Asserts that the response is a valid JSON array
 */
public void assertValidArrayData()

/**
 * Asserts that the response is an empty JSON array
 */
public void assertEmptyArrayData()

Completed via #221


Make Exokit streamable

Streaming is a big thing. I spend more time on Twitch than I'd care to admit.
It would be good to harness that wave by adding stream integration into Exokit.
The integration should be developer-focused at first, since those are who we want streaming Exokit anyway. Maybe we can bundle OBS and make it a one-click setup?

in windows 10 you can use the game bar to record/stream. Windows key and g

Closing this issue due to being outdated.


Jupyter notebook: Kernel dies when running timeline trace [Nvidia-docker]

I am trying to use timeline to profile GPU memory usage on an EC2 instance using Tensorflow-Gpu Nvidia-Docker. When I add a few lines to my notebook which runs a convNet, it keeps restarting with a "Kernel died" message.
Some of the changes I made:
from tensorflow.python.client import timeline

run_metadata = tf.RunMetadata()

with tf.Session(graph=graph, config=tf.ConfigProto(log_device_placement=True)) as session:
    tf.global_variables_initializer().run()
    for step in range(num_steps):
        batch_data, batch_labels = generate_batch(
          batch_size, num_skips, skip_window)
        feed_dict = {train_dataset : batch_data, train_labels : batch_labels}
        if step == 1000:
            _, l = session.run([optimizer, loss], feed_dict=feed_dict, options=tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE), run_metadata=run_metadata)
            trace = timeline.Timeline(step_stats=run_metadata.step_stats)
            with open('timeline.ctf.json', 'w') as trace_file:
                trace_file.write(trace.generate_chrome_trace_format())
        else:
            _, l = session.run([optimizer, loss], feed_dict=feed_dict)
        ...

There should be an actual error from tensorflow in your logs somewhere. I would first suspect that it can't find CUPTI library
export LD_LIBRARY_PATH="/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH"

Thanks @yaroslavvb; @tfboyd, do you have anything additional?

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!


[Feature Request] Add option for additional user-defined labels on bugs

This feature is included in the firewatch refactor


Revoke stale URL Objects for a given file whenever we create a new one

In content.js we have:
        toURL: function(data, type) {
            var blob = new Blob([data], {type: type});
            return URL.createObjectURL(blob);
        }

We call this in a bunch of places to create HTML (when we launch a URL) and CSS (when we rewrite CSS) files reachable via the iframe browser.  Every call to URL.createObjectURL() probably needs to have a matching URL.revokeObjectURL() sometime in the future, so that we don' t leak URL objects.  The most common case for this is during a reload, where we are throwing away the old content/URL and making a new one.
We have code in BlobUtils.remove() that does this, but only when we cache the URL.  In the HTML case, we never do, so we never release it when we make a new one.  We need a strategy for cleaning this up.  One possibility would be to generalize the BlobUtils stuff a bit, and use it in cases where we aren't caching (e.g., HTML).  Another would be to have toURL() also take a filename, and we remember the old URL for a given filename, and revoke it when we update.

Just got this error testing things:
uncaught exception: out of memory
Yeah, this is bad :(  We need a fix.

Another thought occurs to me.  In iframe-browser.js we do this, when we alter the iframe's src:
    function update(url) {
        if(url) {
            var iframe = getBrowserIframe();
            if(iframe) {
                iframe.src = url;
            }
        }
    }

Before we alter the iframe.src, we could do:
...
// Release the existing blob URL object that was being previewed
URL.revokeObjectURL(iframe.src);
// Update to new URL object
iframe.src = url;

We might already have enough code paths to deal with CSS files and images.

https://github.com/gideonthomas/nimble/commit/ce9d70be1bc6cc77100682cf5e2a9bbe57479736#commitcomment-10629979

Another place we need to revoke the blob url is when we call serveLiveDoc in bbld where we generate a new blob url for the instrumented document. We should probably cache it and revoke it when we generate a new one

Talking with @K-LV about this on irc, I think the right strategy here is to move Content.toURL() into BlobUtils and rename it to createURL(path, data, type).  The code would be something like:
// Create a Blob URL Object, and manage its lifetime by caching.
// Subsequent calls to create a URL for this path will auto-revoke an existing URL.
function createURL(path, data, type) {
  var blob = new Blob([data], {type: type});
  var url = URL.createObjectURL(blob);
  // NOTE: cache() will clean up existing URLs for this path.
  cache(path, url);
  return url;
}

Then all consumers of Blob URL Objects are free to use them without having to worry about lifetime management or leaks.


rocksdb_replicator: reset upstream when fetching from a non-leader that has no updates

We see in production where writes can timeout after 2-ack mode is enabled, which usually is caused by followers talking to the wrong upstream (typically a follower), hence never ACKing the write. See this internal doc for detailed investigation and plans on fixing those issues.
This PR serves as the initial step to mitigate the issue: trigger a upstream reset when both of the two conditions below are met:

no updates from upstream, for a consecutive number of times (controlled by a GFLAG, in case it's transient)
upstream informs the follower that the upstream is NOT a leader (avoids unnecessary reset when there is no traffic). We introduced the ReplicaRole in https://github.com/pinterest/rocksplicator/pull/592

This new behavior is disabled by default (via a GFLAG reset_upstream_on_empty_updates_from_non_leader) for a safe rollout, but can be reversed once this feature goes out well in production.
Added unit tests for the two edge cases we see in prod (followers setting each other as upstream; and follower setting itself as upstream).

@kangnanli or @rajathprasad this is ready for another look

Landing, this won't have any production impact, unless explicitly enabled via the GFLAG.


v1.0 Todo List

The app will reach stable v1.0.
Features

[x] Multilingual support
[x] CLI support (#13)
[x] Upgrade to .Net Framework 4.7.2
[x] Restore added displays (#70)
[ ] Power events support<PHONE_NUMBER>99af2a53a0895d22c388d612a73008)
[x] Update checker
[x] Choose parent GPU (#46)
[ ] New threading model (#58)
[x] Use WinForms tray menu (#28)
[ ] Allow to run app with broken driver/not installed (#27)
[ ] v0.41 driver install for unsupported Windows (#27)

Bugs

[ ] #40
[ ] Incorrect high DPI screenshot (#12)
[x] Not detected added display in offline mode

Add your features request below and before this December.

If i understand correctly, restore added displays means it is able to automatically add display upon our PC startup with the run on startup feature right?

@ataraxia-admail This feature is currently able to add the recently added displays from your previous session when you reopen the app.  And with power events support, it should work with sleep/hibernate without breaking the driver.

@nomi-san Is it possible to add a virtual screen that Windows recognizes as having touch functionality? This might be helpful for streaming certain games that support touch features. Thank you for your response.

@soi8391 This feature is out of scope. You should turn your Windows into touch/tablet mode. For remote desktop, use an app that supports full of translation touch gestures, such as AnyDesk and SteamLink.

@nomi-san Is it possible to add a virtual screen that Windows recognizes as having touch functionality? This might be helpful for streaming certain games that support touch features. Thank you for your response.

What's the streaming client are you using? I use Sunshine and a modified version of moonlight and it support touch gestures. Touching support is added by the modified version of moonlight, not Parsec VDD itself.
Modified version of moonlight

@UnCLAS-Prommer Thank you for your reply. I used the client you recommended, but I found that the game I’m playing only automatically switches to the touch interface when the touchscreen is active. Therefore, it seems that this client does not solve the issue.

I am trying to use this with moonlight and sunshine but am running into issues, when moonlight tries to connect VDD seems to not make a virtual display. How have you guys been able to get this working with moonlight or is that not possible?


TS:1.0 - Expression produces a union type that is too complex to represent

From our DS.
Code used to repro:
const { styled, config } = createStitches({})
type CSS = Stitches.CSS<typeof config>

export const overlayStyles: CSS = {
  backgroundColor: 'rgba(0, 0, 0, .15)',
};

export const Overlay = styled('div', overlayStyles);

I think TS server is crashing, because sometimes this happens and sometimes it doesnt

this should be fixed. if not then this is a regression

@hadihallak fixed where? Theres been no new pushed to canary since this was reported

late last week I pushed changes to fix it. I've specifically tested against this case back then as I've seen it thrown around on slack
just re-tested it again now and it's working for me

the main fix was in here and I think @jonathantneal cherry-picked the change from the PR into canary

Gotcha.

Fixed in V1. Please refer to migration guide and changelog


Display single pet listing

Hi there,
Your plugin is working great for my client's Adoptable Dogs page, but I'd like to be able to display a single pet listing in the sidebar like the old PHP functions were able to do. Is that possible with this & I'm just not seeing it, or is that option no longer available within the plugin?
Thanks!

I'm not sure which PHP functions you're referring to. I've never offered a PHP option that did that.
That said, you could use this plugin and just request a single pet listing back.

Hi, just a quick follow-up. Here's what I was talking about...
https://github.com/cferdinandi/petfinder-api-for-wordpress#individual-pets
Perhaps the "in the sidebar" was confusing -- I did that on my own -- but the other plugin -- that's what I meant by "functions," which is how it's referred to in the actual PHP file -- you built offered a single pet listing option, in addition to the full listing option. I'll just play around and figure out how to make both work with this plugin.

Ohhh yea, that! I could probably bake something like that into the API, but, candidly, my priorities are elsewhere at the moment.

While this continues to be a good idea, I'm unable to properly maintain this plugin or work on feature development, so I'm closing this as "won't fix."


Stringifying an array of objects results in an error when its value is passed to another module

Bicep version
0.4.1272
Describe the bug
Stringifying a locally defined array of objects results in an error about '{' being unexpected. Providing the array through a module does work.
To Reproduce
Steps to reproduce the behavior:
var myArray = [
  {
    id: '117'
    name: 'John'
  }
]

// The module takes a string parameter called 'value'
module tst 'modules/sample-module.bicep' = {
  name: 'tst'
  params: {
    value: string(myArray)
  }
}

Error is:
{'code': 'InvalidTemplate', 'message': 'Deployment template language expression evaluation failed: \'The language expression \'{"id":"117","name":"John"}\' is not valid: the string character \'{\' at position \'0\' is not expected.\'. Please see https://aka.ms/arm-template-expressions for usage details.', 'additionalInfo': [{'type': 'TemplateViolation', 'info': {'lineNumber': 0, 'linePosition': 0, 'path': ''}}]}

However, if myArray is part of the output from another module, it does work as expected. So as a workaround I have a, pretty much, empty module returning an array of objects.
Additional context
I'm trying to store an array of JSON objects in an App Configuration resource with the application/json content-type.

Tested offline and the below code should work as a workaround:
// The module takes a string parameter called 'value'
module tst 'scratch-mod.bicep' = {
  name: 'tst'
  params: {
    value: ' ${string(myArray)}'
  }
}

This is related to service-side expression evaluation bug. When you converted the array into a string, the first and last characters are [ and ], which is considered an ARM Template expression, even though that is not what you are trying to express.
cc @majastrz as FYI

Tested offline and the below code should work as a workaround:
// The module takes a string parameter called 'value'
module tst 'scratch-mod.bicep' = {
  name: 'tst'
  params: {
    value: ' ${string(myArray)}'
  }
}

This is related to service-side expression evaluation bug. When you converted the array into a string, the first and last characters are [ and ], which is considered an ARM Template expression, even though that is not what you are trying to express.
cc @majastrz as FYI

That's a better workaround than using a module to provide the array. Important to note that the space in front is important, otherwise it leads to parse errors

@alex-frankel Any ETA on the fix?
BTW your workaround from the 9-March-2022 does not work when used in the resource:
`resource keyVault 'Microsoft.KeyVault/vaults@2023-07-01' existing = {
name: keyVaultName
}
resource kvsecrets 'Microsoft.KeyVault/vaults/secrets@2023-07-01'  = [for secret in secrets: {
parent: keyVault
name: secret.name
tags: tags
properties: {
value: ' ${string(secret.value)}'
attributes: {
enabled: true
}
}
}]`


Expose Multiple Properties

Nice Framework, but you need a few things exposed before it's production ready:

[x] Route Types [POST, GET, PATCH, etc]
[x] Client_Secret
[ ] Maybe add hooks for UDP, WebSockets

@avdaredevil Thanks for the suggestions! I just pushed the change that allows for designating route types.
Commit: https://github.com/Max00355/Feather/commit/c4c4450ca069187983f4699b371e332aaa832e33
Updated Readme: https://github.com/Max00355/Feather/README.md

Also, what do you suggest for the UDP WebSockets?

Client secret and AES encrypted sessions have been added. I am going to close this issue. If possible, an elaboration on how you imagine the websockets working would be great.
Thanks!

Sure, I'd say look into how Flask-Sockets handles it.
There might be a goldmine here as well.
Let me know if you'd need more input.


Trying to use --init fails with "container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory"

/kind bug
Description
Running the fedora:31 image with --init fails with:

Creating f31: Error: container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory

Steps to reproduce the issue:

sudo podman run -ti --privileged --rm --init registry.fedoraproject.org/fedora:31

Describe the results you received:

Creating f31: Error: container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory

Describe the results you expected:
No errors
Additional information you deem important (e.g. issue happens only occasionally):
Output of podman version:
$ podman version
Version:            1.5.1
RemoteAPI Version:  1
Go Version:         go1.12.7
OS/Arch:            linux/amd64

Output of podman info --debug:
debug:
  compiler: gc
  git commit: ""
  go version: go1.12.7
  podman version: 1.5.1
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: podman-1.5.1-3.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.0, commit: d728afa06cd2df86a27f32a4692c7099a56acc97-dirty'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 322580480
  MemTotal:<PHONE_NUMBER>8
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree:<PHONE_NUMBER>0
  SwapTotal:<PHONE_NUMBER>2
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: lux
  kernel: 5.2.17-200.fc30.x86_64
  os: linux
  rootless: true
  uptime: 24h 21m 26.53s (Approximately 1.00 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /var/home/jlebon/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /var/home/jlebon/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 0
  RunRoot: /tmp/1000
  VolumePath: /var/home/jlebon/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):
$ rpm -q podman
podman-1.5.1-3.fc30.x86_64

And indeed the catatonic binary is missing from the package:
$ rpm -ql podman | grep libexec
/usr/libexec/podman
/usr/libexec/podman/conmon

Additional environment details (AWS, VirtualBox, physical, etc.):
Fedora Silverblue 30.
This used to work, so I'm guessing it was dropped in a recent version.

Really the actual image used here doesn't matter. I updated the issue title to reflect this.

@QiWang19 This is a field that will need to be in containers.conf.

Typical, it seems that I missed the mark after the constant was moved a bit.
sed -e 's|/usr/libexec/podman/catatonit|/usr/local/lib/podman/catatonit|' -i libpod.conf
sed -e 's|/usr/libexec/podman/catatonit|/usr/local/lib/podman/catatonit|' -i libpod/runtime.go

Unfortunately the first line only changed the commented out default config...
#init_path = "/usr/local/lib/podman/catatonit"
Error: container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory

Will fix it next time around.
At least I included the binary :-)
tc@box:~$ /usr/local/lib/podman/catatonit --version
tini version 0.1.3_catatonit

The problem is that catatonit is not (yet?) packaged for Fedora. Maybe, we should default to tini and require that in the .spec? @rhatdan @lsm5 DTY?

Should be a recommends, and sure we should default to a executable that is shipped in Fedora.

The problem is that catatonit is not (yet?) packaged for Fedora. Maybe, we should default to tini and require that in the .spec? @rhatdan @lsm5 WDYT?

Well, doesn't look like tini is packaged either.

Is it part of moby?

Is it part of moby?

Yes, https://src.fedoraproject.org/rpms/moby-engine/blob/master/f/moby-engine.spec#_26 . Will need to be packaged separately.

I guess I can take that up, unless you prefer we do catatonit itself.

I'll ping you on slack to chat :)

If we can afford to wait a few days, we can get catatonit packaged into Fedora and have that as a Recommends. If you want it by tomorrow itself, I can get it subpackaged for the time being, but there'll be some unnecessary issues around versioning/epoch when the standalone package goes out.

It's been broken that way for a while now (I only just got around to reporting it now :) ). If catatonic is the upstream default and what gets tested in CI, then my personal recommendation is to just do it properly and package it, even if it takes a few days.

I vote for a dedicated catatonit package. Thanks, @lsm5!
@cyphar, FYI :+1:

I might take a look at requesting that Docker switch to catatonit, which will ensure it's packaged everywhere.

Not sure if it's in scope for this issue, but running into the same on CentOS 8
container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory

NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

@lsm5 @baude @rhatdan Poke - what do we want to do about this? Substantially harder to get a new package into Cent than Fedora. Do we need to bundle it into the Podman RPM?

Do we need to bundle it into the Podman RPM?

FWIW, Docker is doing this already, as docker-init
$ docker-init --version
tini version 0.18.0 - git.fec3683

Probably fair to bundle stuff under libexec/podman, no ?

We should only bundle it, if it was specific to podman.

Where do we stand here?

Where do we stand here?

err not yet, I'll try to get this in soon as I can find the time, I really don't want to bundle it though if it has its own versions.

I'd volunteer doing the work to package this... but i don't know where to start for Fedora, much less my ideal end states of it ending up in RHEL7/RHEL8.

I think I'm hitting the same issue, with a newer error message:
$ podman version
Version:            1.6.2
RemoteAPI Version:  1
Go Version:         go1.13.1
OS/Arch:            linux/amd64
$ podman run --init busybox
Error: please specify a path to the container-init binary

Your libpod.conf probably has init_path = "" somewhere in it - try removing that line.

As it seems easy to package I did it on copr.
https://copr.fedorainfracloud.org/coprs/oleastre/misc/package/catatonit/
However, I don't know what is the best to make it recognized by podman:

install in /usr/bin and create a link to /usr/libexec/podman/catatonit
always install to /usr/libexec/podman/catatonit, but this would make it only available for podman
add a podman-catatonit package that depends on this one and creates the link
update /usr/share/containers/libpod.conf to point to /usr/bin/catatonit

Here is a copy of my spec file, as simple as it can be.
Name:           catatonit
Version:        0.1.4
Release:        1%{?dist}
Summary:        A container init that is so simple it's effectively brain-dead.

License:        GPLv3+
URL:            https://github.com/openSUSE/catatonit
Source0:        https://github.com/openSUSE/catatonit/releases/download/v%{version}/catatonit.tar.xz

BuildRequires:  autoconf automake gcc glibc-static libtool

%description
A container init that is so simple it's effectively brain-dead. This is a
rewrite of initrs in C, because we found that it is not possible to statically
compile Rust binaries without using musl. That was, in turn, a reimplementation
of other container inits like tini and dumb-init.

%prep
%autosetup

%build
autoreconf -fi
%configure
%make_build

%install
rm -rf $RPM_BUILD_ROOT
%make_install

%files
%license COPYING
%doc README.md
%{_bindir}/catatonit

%changelog
* Tue Feb 18 2020 Olivier Samyn<EMAIL_ADDRESS>- Initial spec file based on catatonit README

Definitely /usr/bin IMO

For info, I tried to change libpod.conf to poing init_path to /usr/bin/catatonit (after installing my package) but podman was still looking for /usr/libexec/podman/catatonit (default fedora 31 podman install)

I have the old spec file at https://pagure.io/catatonit/blob/master/f/catatonit.spec . Could've sworn I had a package review created but can't find it on rhbz.
@oleastre are you interested in officially packaging it for Fedora?

@jnovy @lsm5 Any progress on getting catatonic packaged for Fedora?

If catatonic is not supposed to be run by a  user then it should not go into /usr/bin.  If it is not supposed to be run by an admin then it should not go in /usr/sbin.  If it is only run by applications like container engines, then it should go into /usr/lib/catatonit/catatonit or /usr/libexec/catatonit/catatonit. or /usr/libeec/catatonit.
Then each distro could customize their install location in libpod.conf (containers.conf) going forward.

@lsm5 I'm not a recognized fedora packager although I sometimes provide patches to existing one.
I just did it quickly to solve the problem here. But if this can solve the problem on the fedora level, why not.
And as suggested by @rhatdan , I'll update the spec file to install in /usr/libexec/catatonit/catatonit which makes more sense.
Although that does not solve the installation for podman on fedora. But I suppose this should be properly handled in the fedora bugzilla instead of here.

@lsm5 I'm not a recognized fedora packager although I sometimes provide patches to existing one.
I just did it quickly to solve the problem here. But if this can solve the problem on the fedora level, why not.
And as suggested by @rhatdan , I'll update the spec file to install in /usr/libexec/catatonit/catatonit which makes more sense.
Although that does not solve the installation for podman on fedora. But I suppose this should be properly handled in the fedora bugzilla instead of here.

I'll post a fedora review request and @jnovy will review it. Patches are welcome anytime!

package review request filed at https://bugzilla.redhat.com/show_bug.cgi?id=1804768

this is likely obvious to everyone, but also RHEL8 is missing it. Just to mention here in order to get it somehow into pipeline to RHEL.
podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_64
podman run --init -d --name="home-assistant" -e "TZ=Europe/Helsinki" -v /home/redhatter/ha-config:/config -p 8123:8123 homeassistant/home-assistant:stable
Error: container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory

this is likely obvious to everyone, but also RHEL8 is missing it. Just to mention here in order to get it somehow into pipeline to RHEL.

@ikke-t yes, this will quite likely happen once things are present in Fedora. But RHEL will have its own processes/schedules, so no ETA promised. But I can ensure the OBS packages for CentOS have it once Fedora is done.

catatonit has been added to the OBS repos (debian and ubuntu only so far), will be published soon. I'll add it to CentOS once it's in Fedora.

Who is currently maintaining catatonit? We reported a couple of serious bugs (encountered while using it with podman) over a month ago, but noticed that there hasn't been any active development in about a year. If the Project Atomic community is going to depend on it, would make sense for us to take over maintenance as well?

@wwilson, catatonit is a very small binary and not performing much work. A year without code changes doesn't necessarily imply the project is dead. Can you point to the bugs? Did you reach out to the maintainers?

@wwilson Are these the bugs you're referring to -- openSUSE/catatonit#4 and openSUSE/catatonit#5? I didn't get the impression these were "serious bugs" in the reports -- they are semantic bugs and should be fixed but I haven't had time to look into them (I've been away from home almost consistently for various reasons since Jan 2nd). If those aren't the bugs you're referring to, then I don't believe I've heard from you.

I've fixed the two bugs I linked.

Thanks, @cyphar !

I want to use --init in a script used on various distros, here's partial sample from googling (didn't actually test):

[x] OpenSUSE: packaged catatonit, podman Requires it
[x] Arch: packaged catatonit, optional dep of podman-docker, but not of podman itself?
[ ] ubuntu via snap: didn't get it to work, but doesn't contain any file named catatonit
didn't look into any of the kubic repos

For me --init is nice to have (more reliable killing), I can skip when missing.  Can I assume most distros would put it at /usr/libexec/podman/catatonit?
Anyway for near future the portable solution seems to be to modify the image to contain tini or catatonit.

No, can't assume a path.   opensuse seems to put it at `/usr/bin, and libpod.conf points there.

Is there a workaround right now? I'm new to Docker and I just can't get it to run on one of my machines and switching to Podman I'm stuck on this.

If you can get a tini or katatoinit binary installed on the host, the path to it is in libpod.conf - set the init_path variable to the path you have an init binary installed at. The default is "/usr/libexec/podman/catatonit" so you could also just install the binary to that location and not worry about changing config files.

Sorry, catatonit not katatoinit - been looking at Kata containers too much recently, I suppose

Is there a workaround right now? I'm new to Docker and I just can't get it to run on one of my machines and switching to Podman I'm stuck on this.

If you're on ubuntu/debian you can get it from the kubic repos. See: https://software.opensuse.org//download.html?project=devel%3Akubic%3Alibcontainers%3Astable&package=catatonit
catatonit package is being reviewed for Fedora right now so it's not available in the base repos for Fedora or CentOS yet. If you don't mind installing an unreleased rpm directly to get unblocked right away, you can get it from https://lsm5.fedorapeople.org/catatonit-0.1.4-1.x86_64.rpm

Thank you SO MUCH! The package works perfectly.

@lsm5 this looks like it is stalled with a needinfo for you?
https://bugzilla.redhat.com/show_bug.cgi?id=1804768

@lsm5 this looks like it is stalled with a needinfo for you?
https://bugzilla.redhat.com/show_bug.cgi?id=1804768

yup, sorry, distractions. I'll get that going tomorrow..

catatonit is now in the fedora testing repos https://bodhi.fedoraproject.org/updates/?packages=catatonit . I'd recommend people install this instead of any prior builds I posted.
It's also present on the OBS repos now.

Should probably add a recommends on suggests for this package to podman.

Should probably add a recommends on suggests for this package to podman.

It's in there already in podman 1.9.1

Awesome.

Fedora users, please add karma to the f31 and f32 packages https://bodhi.fedoraproject.org/updates/?packages=catatonit


CLJS external browser REPL issues with org.clojure/clojurescript 0.0-3126 and reagent 0.5.0

Hello,
I'm a really big fan of the REPL within LightTable. It's so powerful and easy to get up and running - an absolute pleasure to develop with!
However, when I tried to update some dependencies in a Clojurescript project of mine, I encountered some issues with trying to get the LightTable external browser REPL up and running (the REPL with the HTML script tag).
With the latest version of Clojurescript ([org.clojure/clojurescript "0.0-3126"]), the REPL loads, but none of the clojure.core functions such as map or reduce can be eval'ed as they are (map evals to nil) and must be prefixed with clojure.core/<function_name>. I also get the same issue with the [weasel "0.6.0"] browser REPL. So this might be an issue with the latest version of Clojurescript and it's compatibility with current REPLs rather than just an LT REPL issue.
If I downgrade to [org.clojure/clojurescript "0.0-2843"], then I can simply eval map and reduce as they are within LT.
For example map evals to:
#<function (f,c1,c2,c3,var_args){
var colls = var_args;
switch(arguments.length){
case 1:
return map__1.call(this,f);
case 2:
return map__2.call(this,f,c1);
case 3:
return map__3.call(this,f,c1,c2);
case 4:
return map__4.call(this,f,c1,c2,c3);
default:
var G__105736 = null;
if (arguments.length > 4) {
var G__105737__i = 0, G__105737__a = new Array(arguments.length -  4);
while (G__105737__i < G__105737__a.length) {G__105737__a[G__105737__i] = arguments[G__105737__i + 4]; ++G__105737__i;}
G__105736 = new cljs.core.IndexedSeq(G__105737__a,0);
}
return map__5.cljs$core$IFn$_invoke$arity$variadic(f,c1,c2,c3, G__105736);
}
throw(new Error('Invalid arity: ' + arguments.length));
}>

rather than nil
I also get over 2000 undeclared Var warnings in the console. I'm not sure if that is related to the REPL issue.
When I try to update to [reagent "0.5.0"], I get the following exception when I try to connect to the LT REPL:
clojure.lang.ExceptionInfo: No such namespace: cljsjs.react, could not locate cljsjs/react.cljs at line 2 file:/home/piotr/.m2/repository/reagent/reagent/0.5.0/reagent-0.5.0.jar!/reagent/core.cljs :: {:tag :cljs/analysis-error, :file "file:/home/piotr/.m2/repository/reagent/reagent/0.5.0/reagent-0.5.0.jar!/reagent/core.cljs", :line 2, :column 1}
                 core.clj:4403 clojure.core/ex-info
              analyzer.clj:379 cljs.analyzer/error
              analyzer.clj:376 cljs.analyzer/error
             analyzer.clj:1249 cljs.analyzer/analyze-deps
             analyzer.clj:1494 cljs.analyzer/eval4017[fn]
              MultiFn.java:249 clojure.lang.MultiFn.invoke
             analyzer.clj:1833 cljs.analyzer/analyze-seq
             analyzer.clj:1925 cljs.analyzer/analyze[fn]
             analyzer.clj:1918 cljs.analyzer/analyze
             analyzer.clj:2133 cljs.analyzer/analyze-file[fn]
             analyzer.clj:2129 cljs.analyzer/analyze-file
             analyzer.clj:1246 cljs.analyzer/analyze-deps
             analyzer.clj:1494 cljs.analyzer/eval4017[fn]
              MultiFn.java:249 clojure.lang.MultiFn.invoke
             analyzer.clj:1833 cljs.analyzer/analyze-seq
             analyzer.clj:1925 cljs.analyzer/analyze[fn]
             analyzer.clj:1918 cljs.analyzer/analyze
             analyzer.clj:2133 cljs.analyzer/analyze-file[fn]
             analyzer.clj:2129 cljs.analyzer/analyze-file
             analyzer.clj:2103 cljs.analyzer/analyze-file

I do not encounter this issue when I run the REPL using weasel.
If I downgrade to [reagent "0.4.3"], the LT REPL works again.
Here's my system info:
Light Table version: 0.7.2
Kernel: Linux 3.11.0-26-generic (x86_64)
OS: Linux Mint 16 Petra
Hope this is enough info to reproduce the issues!

Thanks for the detailed report. This is a duplicate issue of #1845 and there is a WIP fix being investigated to address the exception


unable to write image

I have put an application.png into /var/tmp/goodbudget, but when building I get this:
$ macgap build goodbudget
Error: Unable to write image to file /private/var/tmp/goodbudget/application.icns.L1zan
Built application in /private/var/tmp/goodbudget.app

touching the file works:
touch /private/var/tmp/goodbudget/application.icns.L1zan

I tried with different PNG images, all without luck

is macgap still working / under development? Judging from the last commits and the lack of communication in the issues it looks like it has currently no contributors. I have switched to Electron now, the pro for Electron is that it is actively developed and has many features but the pro for MacGap is it's ease of use (and the fact that Slack uses MacGap for their OS X app)

Hi — MacGap v1 is definitely not still under development. I'm happy to leave it up, as numerous people are still using it and I'm hoping they will help each other in this issue queue where they can.
MacGap v2 is also very quiet. I don't personally use it, having switched to Swift, but the concept is great, and I am very open to code contributions.
I guess we're just growing used to gigantic app sizes, so for many people the larger communities, corporate sponsorship and consequent active development around big fat projects like Electron is understandably attractive.
I think there remains a place for MacGap, as to this day I am not aware of any other lightweight alternative to Electron etc

thanks for the reply @jeff-h, didn't realize that MacGap is smaller in filesize. Would be great if Slack would contribute back to MacGap as I reckon they have made some changes in the macgap codebase

The MacGap project is extremely lightweight and nimble; a blank application is about 980KB. The Electron apps I have in my Apps folder range from 135MB (Postman) to 245MB (GitKraken)!


Remove extraneous line breaks in KJV verses #187

Replace .trim method with .replace to remove extraneous line breaks from KJV verses pulled from BibleAPI.

Total GitHub noob here. Do I need to do some kind of edit from my side, or do you make the proposed changes from your end?

Nevermind. I found the Commit Suggestion button. Lol

Thanks


Easy installation

[x] vscode extension pre-release
[ ] add instructions for how to use with vim?
[ ] packaging the language server with the extension itself or separate installation?
[ ] solidity-analyzer on crates

Until this is available on crates, users can just install directly as mentioned here https://github.com/parmanuxyz/solidity-analyzer/blob/master/README.md?plain=1#L15-L23


Copy jdk8u202-b08_openj9-0.12.1 release builds to openjdk8-binaries

The jdk8u202-b08_openj9-0.12.1 release has now been built and tested.
The following build pipeline parameters were used to work around the build issues already raised:
releaseTag: jdk8u202-b08
branch: openj9-0.12.1
additionalFileNameTag: openj9-0.12.1
Some platforms needed two attempts due to the first build failing.  Also MacXL was added to the second attempt.
So the builds are attached to these two Jenkins pipelines, together making a full set of platforms:
https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk8-pipeline/443/
https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk8-pipeline/444/
Tests run against MacXL failed (https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/988/console) - so we are not is a position to release this build yet.
The remaining builds (not MacXL) now need to be copied manually to the openjdk8-binaries repository to

Specify the correct release name as required (jdk8u202-b08_openj9-0.12.1)
To rename the linuxXL build so that the linuxXL part appears earlier in the filename so that it is recognised correclty by the AdoptOpenJDK release download page.

The sha files may need to be updated / recreated.
Please can we update this issue with what needed to be done in case it needs to be repeated before the underlying issues are fixed.

The MacXL jdk_math_jre_0 tests failed because both the jdk and the jre URLs should have been passed as CUSTOMIZED_URLs to the grinder job, but only the jdk URL was.
The test has been rerun with the extra URL and now passes, so the MacXL builds can be added to the release.

@sxa555 - please could you upload the build to openjd8-binaries?

Done (yesterday). Closing.