Support (Jupyter) Debug Adapter Protocol

ipykernel 6.0.0 just landed which, among other things, supports the Debug Adapter Protocol. We should see if:

we somehow magically get support :tada:

though seems a little unlikely :blush:

we have to do some plumbing to support robot sources, etc. :wrench:

This would definitely trigger a need to demonstrate a new :robot:  :microscope: !

😮🙀

@bollwyvl Is debug adapter protocol the one used by jupyterlab-debugger? Or something new? How does it relate with xeus-robot? Is migrating to xeus still the way to go?

the one used by jupyterlab-debugger

Yup, though it's in core now as @jupyterlab/debugger, so the "default" pip install jupyterlab will start showing it to folks. (of course running into issues on conda-forge because pypy gah)

How does it relate with xeus-robot

Nothing in particular: if just we like being in ipykernel, it's probably an option. i haven't looked into whether the robot-interpreter implements the debugger, or if it's in the kernel.
Of note, though, we're inching towards being able to support ipykernel more directly in jupyterlite which might be an easier road  to getting robotkernel running in the browser, with the WASM-powered pyodide until we can start building the world (wide web) in a conda-forge wasm-32 target. await micropip.install("robot"); import robot already works there. I'm really looking forward to being able to use this kind of thing for docs/demos/workshops vs the robotlab approach (much as i love shipping the big GBs).

Thanks. Now I got it!
So, the new ipykernel makes it possible for us to make ipykernel based robot kernels compatible with JupyterLab debugger by "just" implementing support for DAP-messages and related ipykernel additions. And if I'm not completely lost, the minimal set of required messages to support can be seen at:
https://github.com/ipython/ipykernel/blob/b4352b89efb0f9d938af736103b2c456ab0c6e3f/ipykernel/debugger.py#L196
You would prefer us to keeping ipykernel based robot kernel alive, because it could be easier to port to jupyterlite than xeus-robot?
I'm having a vacation after two weeks, so maybe then I should use a few afternoons to see how far this goal is. Just to get my thoughts out of the current day work projects...
Hacking xeus-robot if definitely out of my reach. Choosing that road would make robotkernel a thin wrapper around xeus-robot and robotframework-interpreter. Yet, maybe even with keeping ipykernel, trying to rebase on top of robotframework-interpreter could make sense.

compatible with JupyterLab debugger

Well, theoretically, any jupyter client, but so far I haven't seen any other implementations (open source or proprietary).

"just" implementing

Right, here's the quick run-down of possible scenarios.

Hacking xeus-robot if definitely out of my reach.

Yep, me too, to my shame... or maybe not really... I've made it this far without having to complain about C++ aside from figuring out other peoples' make or cmake issues... but boy howdy, that's not fun for me on windows. And, like i said, there is some low-grade effort (by folks involved in xeus) to make wasm-32 a supported for conda-forge (or wasm-forge, if it comes to it, as a set of long-term patches).

having a vacation

that sounds lovely. i hope to take some time, too! this whole "using my leave to be a crappy remote kindergarten support aide" has been a real downer.

prefer us to keeping ipykernel based robot kernel alive

Welp, more kernels are... more kernels. Good and bad. I think we(you)'ve built some really nice stuff here, and we've been able to weather a number of upstream changes, and still provide something that's fun to use. The jupyterlite thing is... not yet really a thing, but probably will be a thing.

A related technology/inspiration, basthon (also based on pyodide) is used all over France to teach folks computational basics... from their browsers, without having to deploy kubernetes.

If robot can be a stepping stone (or alternate path entirely) for people learning robot/python/english, i'm all for us doing what we can to support it.

i love me some nbviewer, but nbviewer plus running kernels in the browser, with really solid, cross-session caching across a number of public content sources, would alter the landscape for real

Every issue here goes out of the topic in 1, 2, 3... :smile:

windows

I've heard that WSL >= 2 has been useful and even Nix has been mostly usable on that.

i hope to take some time, too

:100:
The rest of the year will be terrible, but if everything goes as planned (for the next two months), I've warned by employer of a full year family leave next year. :raised_hands: (And definitely need some hobby project to keep me sane.)
That said, Robocon 2022 is seemingly coming in January 2022, and to have any reason to be there, I should use the few summer vacation afternoons (I almost promised yesterday) well.

i love me some nbviewer, but nbviewer plus running kernels in the browser, with really solid, cross-session caching across a number of public content sources, would alter the landscape for real

So, the first thing is to wait for ipykernel working in jupyterlite? What should be the next steps after that? Robot relies a lot on filesystem access, but you have tried it out in jupyterlite? Does pyodide virtualize file system access or should be just be careful to only use StringIO / BytesIO everywhere? And how to register a custom kernel for jupyterlite? In build time?

WSL

no comment :scream_cat: nix-on-win would be a substantial step forward, though MS has decided to pursue their own "package manager" :zany_face: as a semi-hostile fork of an existing open source project.

pyodide virtualize file system

Yeah, emscripten gives us a full in-memory POSIX filesystem, which can have mountpoints provided by other things, including indexedDb (which we use for user contents, settings, etc.). This is not exposed in pyodide, but maybe they're waiting on me to PR it :rofl: . Because of this, jupyterlite contents aren't visible to kernels, yet, but we'll get there. As we run pyodide in a webworker (baby's heavy), there are even more constraints, like not being able to access the DOM, but i could foresee being able to run multiple kernels inside one "computer," have multiple "computers".

register a custom kernel for jupyterlite? In build time?

At present, yes, the kernels are a subset of "serverlite extensions," that provide stuff like kernelspecs and contents. These can't be easily composed outside of webpack, yet, but we're mulling over it. We'll be adding something that is (or is like) federated_extensions for these, such that one would distribute it by "normal means" (npmjs.com or pypi) and then be able to add it during jupyter lite build by CLI or config file.

interactive examples (without all the jupyter chrome)?

We have a few build things to square away at the webpack/serviceworker level, as right now every additional app flavor (right now full lab and retro) duplicate a lot of stuff, and every additional view would trigger some more requests. But yeah, the ideas range from something like https://github.com/jtpio/jupyterlite/issues/148 ...

...to something that appears more like a sphinx/mkdocs layout, with the jupyter "chrome" kinda disappearing. Probably be able to reuse existing templates with nunjucks, etc.  No doubt people would (rightly) complain about the initial >=5mb slug of JS, but would be silly cool with, e.g. LSP, session-persistent widgets, etc.
O, and the jupyter 3.1 collaborative editing stuff basically works in lite: https://github.com/jtpio/jupyterlite/pull/109 :exploding_head:

Thanks! I subscribed to IPython in Pyolite kernel -pull so that I know when it should be merged and there chance to get robotkernel running. Having interactive client robot would be great for learning / tutorials, and even it could not be used with most of the robot libraries, it could probably be used for some real API testing / automation too. And if that works, I could even use it to demonstrate BPMN + Camunda based microservice orchestration... :rocket:

Did I expect too much in expecting that there is already magic somewhere that would turn requests-library request into client side CORS-request? :)

Well. RobotKernel depends on ipykernel.kernelbase, which requires zmq, which cannot be installed, because it requires C-extensions. But probably there's a workaround for zmq already in JupyterLite.

requests-library request into client side CORS-request?

Right, it's not the full networking stack, and a number of stdlib things aren't present dbm, lzma.
Network stuff, for now, has to go through something like:
import js
r = await js.fetch(url)
j = await r.json()   # looks delicious, but is a JS object :(
t = await r.text()
import json
j2 = json.loads(t)

note that the .json looks deliciously requests-like, but is in fact a JS object (or proxy), and doesn't just work with a lot of things that would with a normal python object, hence the import json. But given the ubiquity, we'll probably need to do a better shim... See my desire for conda-forge to make this something we can maintainbly scale. Micropip knows we're gonna do weird stuff, and leaves a lot of stuff hanging out... some of the examples do some really weird stuff.

Which requires zmq

Right, we are already shipping some shim packages for some of these ipy* things. It's unlikely that initial PR will expose enough to make our kernel usage work, but there is some stuff we'd be able to check on our own.

Robotkernel in JupyterLite

Yeah, people can pooh-pooh this stuff, but until we try it, we just won't know if it's cool.

I'll make a standalone issue to track the RoboLite concept!


Cannot build maven project

While setting up a project in JME3 I can definitely start off a "Java with Maven project". Great idea btw to use my favourite Java IDE for this purpose. However when I want to build/run the project, the error message I get is
Cannot run program "${HOME}/jmonkeyplatform/java/maven/bin/mvn" (in directory "${HOME}/jMonkeyProjects/MyTown"): error=13, Permission denied
When checking the directory I can see that the execute permissions were not set. This is something the installer script should take care of.

Care to test this on https://github.com/jMonkeyEngine/sdk/releases/tag/v3.4.1-stable-sdk1?

I tested this and seems to be the case still

We actually do something with Blender, https://github.com/jMonkeyEngine/sdk/blob/3fa27e752bdecd4b6f4ea1238fcb7b75933a8848/nbi/stub/ext/components/products/blender/src/org/mycompany/ConfigurationLogic.java.
But it still annoys me a bit that this is native part of Netbeans. We should test plain NB and hopefully be able to report this upstream...

I wonder if it's actually related to ant? https://stackoverflow.com/questions/3625666/ant-how-to-specify-a-file-is-executable-so-i-dont-have-to-chmodx-each-time
mvn is executable in the sdk on master.
If so, we should do something with template.xml

It was Maven. I tested it back then. The NB default Maven Java project. Failed as described on my Kubuntu.

I mean if ant is the culprit, if so the permission could be set when compressing.


[stable/prometheus-operator] Issue with kubernetes-mixin workload dashboards after recent release

Describe the bug
After the release of chart version 9.2.2 (which synced a bunch of changes into the grafana dashboards), the "workload" dashboards are not displaying any data:

Version of Helm and Kubernetes:
helm 3.x
kubernetes 1.17 (EKS)
Which chart:
stable/prometheus-operator
What happened:
A few dashboards are not displaying any data.
What you expected to happen:
The dashboards should display the correct data.
How to reproduce it (as minimally and precisely as possible):

Install prometheus-operator 9.2.2 or later
Not sure if this is relevant, but we set the Grafana tag to 7.1.1
View a workload dashboard, such as Compute Resources / Workload
Observe the empty result.

Anything else we need to know:

🚧 This chart is now deprecated.
Further development has moved to prometheus-community/helm-charts. The chart has been renamed kube-prometheus-stack to more clearly reflect that it installs the kube-prometheus project stack, within which Prometheus Operator is only one component.
If applicable, please re-open this issue there. Thanks!


Visualise the execution of a search request, for internal debugging purposes

I have started working on a visual logger for search requests. It generates a folder of svg files visualising how the search request was executed. The svg files link to each other such that the folder behaves like a website when opened with a web browser.
I am convinced that it is useful for debugging the search engine as well as to explain how Meilisearch works to new contributors. Here are a few pictures of what it currently looks like:

Here, the bucket sort algorithm is visualised. We can see that the first bucket from words is skipped because it contains 2 documents, but the search offset parameter was 5. The second bucket of words is given to proximity, which first produces a few empty buckets.

Later on, proximity produces a few non-empty buckets. 3 more documents are skipped, and the next ones are added to the search results. When proximity is exhausted, it yields control back to words, which produces a bucket of 1 document, which is then immediately added to the results.

The little "i" icon shown when results are added is interactive, it shows the document ids that were added.
The link icon on top of each next bucket span is also interactive, it links to the internal details of the ranking rule.
The internal details of the words ranking rule consist of a visualisation of the query graph:

The internal details of the proximity ranking rule consist of a visualisation of the proximity graph along with the result of the K-shortest path algorithm that was executed on that graph:

Since generating these visualisations can be very expensive and since their format is heavily dependent on the (internal, unstable) implementation of the algorithms, I do not want to make the visualisations public for users of Meilisearch. But they can be very useful for contributors.

Implemented in https://github.com/meilisearch/meilisearch/pull/3542 , but it is not guaranteed that these visualisations will stay.


Support for filtering on DateOnly

Currently only filtering on DateTime's is supported.

I'll try to implement support for DateOnly.

Done, release 6.1.0.


.Net Core support

Is .Net Core support planed?

Yep, it is! There's already an issue for it: #314. Closing this as duplicate.


strand.md: s/notion/pseudo-code accessor/?

rule notion -> Rule pseudo-code accessor
dotted rule notion -> DR pseudo-code accessor
seem to be a better fit for contexts like

A rule notion, when applied to a dotted rule, is equivalent to that rule notion applied to the rule of the dotted rule.

and

A dotted rule notion, when applied to a Earley item, is equivalent to that dotted rule notion applied to the dotted rule of the Earley item.

That "notion" term I've adopted from a recent math paper.  It means that if it's an "idea", or "notion" which only makes sense for rules, but my language applies it to an Earley item, then it really applies to the Earley item.  So we have "start Earley items", which are Earley items which contain start rules.
It's a little difficult, but replaces a lot of tedious & repetitive definitions.  Many papers simply apply terms more or less at will, and leave it to the reader to puzzle it out.

"notion" works as a term, but I can't help feeling that "pseudo-accessor" would work just as well in the context with the benefit being a lower number of terms.

"Notions" are notions in ordinary English.  The pseudo-code accessors are elements of the syntax of the pseudo-code, like Accessor(x).  So these are two very different things.


Add impl suffix to all rules files

Same names for files and rules functions may end up with many conflicts and inconveniences in the future.

 Add impl suffix to all rules files


D3D12RenderSystem::Render takes a shared pointer and a normal reference.

Currently the D3D12RenderSystem::Render method takes a std::shared_ptr<SceneGraph> const & scene_graph and a FrameGraph & frame_graph. This is does not make sense. Both should just be references, pointers or smart pointers. Not 1 of each.

Fixed in #323


Add info about configuring and running third-party tracers.

https://cockroachlabs.atlassian.net/browse/DOC-214

This change is

nit: please squash the multiple commits into one (git rebase -i -> squash) unless you actually want them as separate commits.

In the doc team we keep them as separate commits and squash on merge.


Repoint Starknet plugin to new infrastructure

This change is to update the Starknet plugin descriptor to use the new (Prod) backend infrastructure - https://cairo-remix.nethermind.io.

@stranger80 somebody send us a report with this screenshot: https://discord.com/channels/725383452081389608/760790566333055006
is that somehow linked to this PR being merged?

@stranger80 somebody send us a report with this screenshot: https://media.discordapp.net/attachments/760790566333055006/1163349660870656010/2023-10-16_13.35.58.png?ex=653f40cd&is=652ccbcd&hm=3732792259483231365944f4cfdbe3c5f756eb97bcdf9617be4e17208b05786e&=&width=1386&height=214
is that somehow linked to this PR being merged?

No, this isn't caused by the PR. We had a backend service outage, this was also flagged to us by users via Telegram. Workaround is now in place, and we are working to deploy a proper resolution.

ok thanks for the prompt reply!

@stranger80 When i try to compile the hello-world file, i get the errors above.

Hi @ioedeveloper, thank you for reaching out.
Please note that the error message you quoted indicates that your workspace requires a feature available starting from Cairo 2.3.0-rc0, whereas the plugin includes Cairo 2.2.0 (the latest stable) compiler.
Also, please note this GitHub project is the generic Remix repo, whereas a better place to report the (suspected) issues in the Starknet plugin is the Starknet plugin repo (https://github.com/NethermindEth/starknet-remix-plugin).
I hope this helps.


Adjust mobile view of app switcher and wallet connect

9/22/22 - After speaking with Jord, we decided to proceed for now by making the app switcher and wallet connect on separate lines and full-width (with outside margins). When present, the switch network button would show below those and also be full-width. Keeping the specs below for future improvements.
Figma links to mobile designs:
Connected - https://www.figma.com/file/9lCwprCNaoAmUDmiMQpFEA/DAOhaus-v3?node-id=7803%3A195118
Connected, wrong network - https://www.figma.com/file/9lCwprCNaoAmUDmiMQpFEA/DAOhaus-v3?node-id=7769%3A194620
On smaller screen sizes, the app switcher and wallet connect widgets might not have enough space with current designs, so these designs adjust the components to account for this.
Specs
Wallet Connect:

On right, fixed size of 48x48px (medium/base button height)

App Switcher:

On left, spans available space minus side gutters (design has these as 20px, but match whatever gutter margin is used for rest of the page) and 10px margin between it and wallet connect

Switch Network Button:

Base-size primary outline button, drops below and spans full width minus margins
Clicking this button and switching to the correct network dismisses the button

Designs
Not connected:

Connected, wrong network:

Connected, correct network:

Example of margin measurements:

I think we can close this now that Switcher is gone (for now).

@7helles are there still changes that need to be made here after removing App Switcher


Tests for missing/inaccessible methods

Two new test that currently fail for not throwing the expected Exceptions if your action maps to a missing or private/protected method.  As mention in Issue #23.

@jk3us man, i'm sorry, i've been back and forth on this several times now over the past few months, trying to figure out how to make the PR work with other expectations centered around "return when not recursively dispatchable" and not break BC. Currently, if we can't dispatch to a method for any reason, it's a "not recursively dispatchable" situation and we return out of the dispatcher. I'm having a hard time resolving that, and other related stuff, with this PR. So while I agree that it'd be really good to know in advance if the method actually exists, and throw reasonable exceptions in those cases, I just can't see my way clear to implementing this. Sorry for the hassle, and the delay in the final decision.


Add an auth plugin

This needs to:

Authenticate users against the bot
Keep a database of usernames and passwords
Provide commands: LOGIN, LOGOUT

This needs to be done to support #5

Superseded by #15


Make procedures public in codeunit 9660 - "Report Layouts Impl."

Describe the request
Hi
Can you please remove internal from the two following procedures in codeunit 9660 - "Report Layouts Impl."?
// >>>>>>>>>>
// internal procedure SetSelectedCompany(NewCompanyName: Text)
procedure SetSelectedCompany(NewCompanyName: Text)
// <<<<<<<<<<
begin
    SelectedCompany := CopyStr(NewCompanyName, 1, MaxStrLen(SelectedCompany));
end;

// >>>>>>>>>>
//internal procedure SetDefaultReportLayoutSelection(SelectedReportLayoutList: Record "Report Layout List"; ShowMessage: Boolean)
procedure SetDefaultReportLayoutSelection(SelectedReportLayoutList: Record "Report Layout List"; ShowMessage: Boolean)
// <<<<<<<<<<
var
    ReportLayoutSelection: Record "Report Layout Selection";
begin
    // Add to TenantReportLayoutSelection table with an Empty Guid.
    AddLayoutSelection(SelectedReportLayoutList, EmptyGuid);
....

Additional context
We need this to be able to set new report layouts as default during install/upgrade or programatically.
Internal work item: AB#524438

@darjoo @JesperSchulz Any chance of making this available in next minor (24.2). We would like to set report layouts by code, following this blog post (https://yzhums.com/39955/) but want to avoid the code cloning path.

@darjoo @JesperSchulz Any chance of making this available in next minor (24.2). We would like to set report layouts by code, following this blog post (https://yzhums.com/39955/) but want to avoid the code cloning path.

24.2 will be tight. We're code complete on that one tomorrow. 24.3 should be doable though!

@andersws / @fvet, given that the entire codeunit is "internal", it wouldn't help much to make the two procedures public. Exposing the entire impl. codeunit is a whole different story, which would require a larger discussion. But I assume that is the request you're giving us?

The functional requirement we're trying to solve, is to be able to simply mark a report layout as 'Default' via AL code
(similar to the 'Set Default' action on the report layouts list page)
Like explained here, but without code cloning: https://yzhums.com/39955/

Some background:
Using the 'DefaultRenderingLayout' option on the report is not a solution for us.
In v1 of our app, we shipped on rdlc file. Some versions later, we introduced an additional rdlc version.

This new rdlct is in fact to be used for new installs of our app. Existing customers / installs should not be impacted.
If we would set the 'DefaultRenderingLayout' to the new rdlc, existing customers would be impacted. That's why we'd like to set newest rdlc version as default layout for new installs only (though AL)

@JesperSchulz Any update if we might expect access to the requested functions ?
I've a new request where I need to detect if a report layout is set as 'Default' or not, but am blocked here as well :(

@JesperSchulz Any update if we might expect access to the requested functions ? I've a new request where I need to detect if a report layout is set as 'Default' or not, but am blocked here as well :(

I will check up on the status for you! Stay tuned.

DefaultRenderingLayout is only used by the platform if you don't add any tenant specific layout selection (used as fallback when layout selection is empty).
The best strategy is always to set the layout selection from applikation code if you want to ensure that users use a specific layout. You can accomplish this from your extension install process if you check for missing layout selection in the tenant and then add a selection for the existing users pointing to the previous layout assuming that it's still being shipped in the new extension version. Existing users will then no notice any layout changes when they update the extension to V2 and they are free to chose a new default if they want.
We are not going to move the two requested functions out of internal scope as there are dependencies to page specific behavior. Also notice that the use case mentioned in the blog post is not following best practices as you should never set the tenant default from a specific record (in this case the Sales Header). If you want a specific layout for a customer or invoice, then use the temporary layout selection features built around the codeunit 9654 "Design-time Report Selection". You find the access functions in table 9651 "Report Layout Selection". Here you only set the layout selection for the current session using a single instance codeunit and you will not impact other others depending on the same report (think about the scenario where you print using the job queue and some sales header select a new default layout for the whole company/tenant - this will cause a high number of support calls due to incorrect / unexpected layouts being used). Notice that you should only user the selections based on layout name as the legacy functions using code key will be deprecated as they use layouts stored in the deprecated Custom Report Layout table.
If needed, we could add generic function in the Report Management Helper codeunit to set/get the default layout for a given report.


Update app.js

It's a good practice to use environment variables for configuration. This allows easily change configuration details like the port or database connection without modifying the code. You can use a package like dotenv to manage environment variables.

The project from Yushi seems really fun in functionality and also visually appealing.I'm amazed at how fully complete the functionality is for this web application, I played with all the features including adding questions, preview quiz, share quiz url and etc, the experience was seamless!


runtime: performance problem with many Cgo calls

Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
go version go1.8 darwin/amd64, though I've also verified this happens on Linux.
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/pmattis/Development/go"
GORACE=""
GOROOT="/Users/pmattis/Development/go-1.8"
GOTOOLDIR="/Users/pmattis/Development/go-1.8/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/qc/fpqpgdqd167c70dtc6840xxh0000gn/T/go-build085228252=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

Background
CockroachDB internally uses RocksDB (a fork of LevelDB) for data storage. We access RocksDB via a small Cgo wrapper. All writes to RocksDB are performed using WriteBatches which are first written to a write-ahead-log and then inserted into the database. Internally, RocksDB "groups" multiple concurrent write batches together into a single append to the write-ahead-log. While investigating performance, we noticed that the size of the groups was smaller than expected.
What did you do?
TL;DR? If write batch grouping is performed in Go performance is good. If it is performed in C++ (either by code that is nearly identical to the Go grouping or by RocksDB) performance is bad.
https://github.com/cockroachdb/cockroach/pull/14138 started as an experiment to replace the grouping of write batches in RocksDB with our own grouping in Go to try and understand the smaller than expected group sizes. In addition to fixing the group sizes, it improved performance on one experiment by 100% while reducing latencies. This was unexpected as the Cgo call overheads are negligible in comparison to the cost of committing the batch to RocksDB. In order to point the finger at RocksDB, we reimplemented the grouping of write batches in our Cgo RocksDB wrapper. Performance was equivalent to RocksDB performing the grouping.
I've provided a stripped down reproduction scenario at https://github.com/petermattis/batchtest. Running on my Mac laptop I see:
~ ./batchtest -t go
_elapsed____ops/sec
      1s     8029.6
      2s     8432.0
      3s     8100.1
      4s     8296.5
      5s     8171.8
      6s     8282.0
      7s     8040.7
      8s     8133.3
      9s     8240.4
     10s     8221.7

~ ./batchtest -t cgo
_elapsed____ops/sec
      1s     5036.4
      2s     2242.4
      3s     1284.3
      4s     1245.5
      5s     1254.5
      6s     1246.1
      7s     1962.5
      8s     4291.7
     10s     2036.3

By default, batchtest uses 100 concurrent worker threads writing "batches". My suspicion is that batchtest and CockroachDB are tickling some badness in the Go scheduler. If I set GOMAXPROCS to the number of concurrent workers the cgo performance gets much closer to the Go performance:
~ GOMAXPROCS=100 ./batchtest -t cgo
_elapsed____ops/sec
      1s     6943.9
      2s     7252.0
      3s     7090.4
      4s     6810.8
      5s     7326.4
      6s     7758.9
      7s     7897.7
      8s     7893.2
      9s     7022.5
     10s     6875.9

https://github.com/cockroachdb/cockroach/pull/14138 is an acceptable workaround for committing batches, but it would be great to understand the performance discrepancy here. We're concerned about other Cgo operations in CockroachDB that don't have such easy workarounds.

I haven't looked at the code.  Is this all calls from Go into C?  Are there any callbacks from C to Go?
You say that the "Cgo call overheads are negligible in comparison to the cost of committing the batch to RocksDB".  Combining that with the number of operations being done per second, would it be accurate to say that the C code itself, after called from Go, takes more than 20 microseconds to run?
In Go 1.8 when a goroutine calls into C, it is still holding a GOMAXPROCS slot and blocking other goroutines from running.  Only if it is running in C for more than 20 microseconds or so will the system monitor thread decide that it is blocked in C code and activate another goroutine.  The fact that your system performs better than you increase GOMAXPROCS makes me suspect that there is something to improve in that area of the code.

Is this all calls from Go into C? Are there any callbacks from C to Go?

The code is all calls from Go into C. There are no callbacks from C to Go. This is true of both the reproduction code and CockroachDB.

You say that the "Cgo call overheads are negligible in comparison to the cost of committing the batch to RocksDB". Combining that with the number of operations being done per second, would it be accurate to say that the C code itself, after called from Go, takes more than 20 microseconds to run?

Yes. The actual RocksDB write batch operation involves a disk write, sometimes followed by an fsync. batchtest simulates this with a 5 millisecond sleep.

Only if it is running in C for more than 20 microseconds or so will the system monitor thread decide that it is blocked in C code and activate another goroutine.

I've been perusing that Go scheduler code. The 20us time is the minimum time sysmon will sleep for. The maximum delay is 10ms and a small bit of instrumentation (measuring the time around the usleep call) showed an actual maximum delay of 17ms, though I'm not sure how often that occurs.
Changing the max sysmon delay to 1ms results in a significant improvement to batchtest:
~/Development/go/src/github.com/petermattis/batchtest master ./batchtest -t cgo
_elapsed____ops/sec
      1s     7493.6
      2s     7279.6
      3s     7217.8
      4s     7498.1
      5s     7898.1
      6s     7899.2
      7s     7799.3
      8s     7798.7
      9s     7698.1
     10s     7395.3

Unfortunately, this didn't result in any improvement to the cockroach performance numbers, so perhaps batchtest isn't completely capturing whatever is going on.

Let me chime in. ( I came from https://github.com/golang/go/issues/21827#issuecomment-366332843 and the sympthoms are similar to what is described here so I decided to continue this thread instead of creating new issue. Please correct me if I am wrong. )
First some context:
Similarly to Cockroach who is using CGo wrappers for RocksDB I observed performance anomalies while using CGo wrappers for SQLite where presense of other goroutines combined with Cgo calls on "main" one show big overhead: github.com/mattn/go-sqlite3 uses CGo and performs several CGo calls inside one Query or Exec. There was also an interrupt goroutine spawned for every Query or Exec to call sqlite3_interrupt if context is canceled.
With Go1.10 avoiding to spawn that interrupt goroutine, if we know the context cannot be canceled, brings ~ 1.5x speedup to Exec (https://github.com/mattn/go-sqlite3/pull/530).
However Query was made faster only by 5% because after 2b283ce database/sql itself always spawns a goroutine to close Rows on context or transaction cancel. Avoiding to spawn Rows.awaitDone goroutine brings ~ 1.5x speedup to Query too (https://github.com/golang/go/issues/23879).

The theme here is that CGo calls show much higher overhead when performed not under simple only-one gorouitine scenario but in the presence of other goroutines. So let's make some analysis:
First, on my machine I can observe that under simple 1-goroutine-only scenario the cost of making 1 trivial CGo call is ~ 60ns, it scales linearly to the number of calls, it is independent of whether GOMAXPROCS is 1 or not, and it is ~ 35x slower compared to making a trivial Go call:
CallGo1              1.71ns ± 0%
CallGo1-4            1.71ns ± 0%
CallGo10             16.0ns ± 0%
CallGo10-4           16.1ns ± 0%
CallCGo1             58.3ns ± 1%
CallCGo1-4           58.6ns ± 1%
CallCGo10             626ns ± 0%
CallCGo10-4           611ns ± 5%

Then we can check what happens in the presence of other goroutines by simulating client-server requests over channel with client and server running in its own goroutines and server sequentially handling requests (so 2 goroutines in total) via making 1 trivial either Go or CGo call:
RTTSeqServerGo1     396ns ± 0%
RTTSeqServerGo1-4   458ns ± 0%
RTTSeqServerCGo1    454ns ± 0%
RTTSeqServerCGo1-4  536ns ± 1%

what can be seen here is that for GOMAXPROCS=1 δ(Go1,CGo1) ≈ t(CGo call), both Go and CGo RTT times are bigger for GOMAXPROCS!=1 with δ(Go1-4,CGo1-4) also growing to ~80ns.
It is somewhat understandable that processing time increases with GOMAXPROCS!=1 even for Go-only case because sometimes there now can be inter OS-threads communications, so in the context of this issue there is probably no problem here (though those 2 goroutines run almost not overlapping with each other and thus there should not be a need to put them to different OS threads). However δ(Go1,CGo1) increase over GOMAXPROCS=1 case suggests there might be something else going on. 20ns is however small and might be all noise, so let's recheck what happens when server performs 10 calls in the handler:
RTTSeqServerGo10      419ns ± 1%
RTTSeqServerGo10-4    486ns ± 1%
RTTSeqServerCGo10    1.00µs ± 0%
RTTSeqServerCGo10-4  1.97µs ± 4%

Go10, Go10-4 and CGo10 show time increase proportional to ~ 9·t(corresponding call). However CGo10-4 (i.e. CGo case with GOMAXPROCS!=1) is now 2x slower than CGo10 (i.e. the same case with GOMAXPROCS=1) by ~1µs.
The difference in between CGo10 and CGo10-4 can be attributed to inter OS-thread communications: the strace for CGo10 case is almost empty and contains only 10ms sleeps from sysmon during active phase:
1403  14:41:40.757900 pselect6(0, NULL, NULL, NULL, {tv_sec=0, tv_nsec=10000000}, NULL) = 0 (Timeout)
1403  14:41:40.767974 pselect6(0, NULL, NULL, NULL, {tv_sec=0, tv_nsec=10000000}, NULL) = 0 (Timeout)
1403  14:41:40.778049 pselect6(0, NULL, NULL, NULL, {tv_sec=0, tv_nsec=10000000}, NULL) = 0 (Timeout)
...

However CGo10-4 shows different picture under strace - during active phase there is now constant high-rate churn of futex wait and wake in between 2 OS threads with 1-10 millisecond sleeps from sysmon also showing seldomly:
1447  14:42:41.733626 futex(0xc420012d48, FUTEX_WAIT, 0, NULL <unfinished ...>
1452  14:42:41.733631 <... futex resumed> ) = 0
1447  14:42:41.733636 <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
1452  14:42:41.733645 futex(0xc420012d48, FUTEX_WAKE, 1 <unfinished ...>
1447  14:42:41.733664 futex(0xc420012d48, FUTEX_WAIT, 0, NULL <unfinished ...>
1452  14:42:41.733669 <... futex resumed> ) = 0
1447  14:42:41.733674 <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
1452  14:42:41.733697 futex(0xc420012d48, FUTEX_WAKE, 1 <unfinished ...>
1447  14:42:41.733703 futex(0xc420012d48, FUTEX_WAIT, 0, NULL <unfinished ...>
1452  14:42:41.733708 <... futex resumed> ) = 0
1447  14:42:41.733713 <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
1452  14:42:41.733722 futex(0xc420012d48, FUTEX_WAKE, 1 <unfinished ...>
1447  14:42:41.733728 futex(0xc420012d48, FUTEX_WAIT, 0, NULL <unfinished ...>
1452  14:42:41.733733 <... futex resumed> ) = 0
1447  14:42:41.733738 <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
1452  14:42:41.733747 futex(0xc420012d48, FUTEX_WAKE, 1 <unfinished ...>
1447  14:42:41.733753 futex(0xc420012d48, FUTEX_WAIT, 0, NULL <unfinished ...>
1452  14:42:41.733758 <... futex resumed> ) = 0
1447  14:42:41.733763 <... futex resumed> ) = -1 EAGAIN (Resource temporarily unavailable)
1452  14:42:41.733772 futex(0xc420012d48, FUTEX_WAKE, 1 <unfinished ...>
...

# seldom
1446  14:42:41.225481 pselect6(0, NULL, NULL, NULL, {tv_sec=0, tv_nsec=10000000}, NULL <unfinished ...>
1446  14:42:41.235566 <... pselect6 resumed> ) = 0 (Timeout)
1446  14:42:41.235594 pselect6(0, NULL, NULL, NULL, {tv_sec=0, tv_nsec=10000000}, NULL <unfinished ...>
1446  14:42:41.245656 <... pselect6 resumed> ) = 0 (Timeout)
1446  14:42:41.245670 pselect6(0, NULL, NULL, NULL, {tv_sec=0, tv_nsec=10000000}, NULL <unfinished ...>
...

This strace suggests that for CGo10-4 the scheduler somehow decided to keep one goroutine on one OS thread and another goroutine on another OS thread and then channel communications in between them somehow becomes futex ops which bring the overhead.
The draft notesleep spinning patch from https://github.com/golang/go/issues/21827#issue-256508518 can mitigate the CGo10-4 problem somewhat:
                     Go1.10       Go1.10+notesleep-spin
name                 old time/op  new time/op  delta
RTTSeqServerGo1       397ns ± 0%   405ns ± 0%   +2.01%  (p=0.000 n=9+10)
RTTSeqServerGo1-4     461ns ± 1%   462ns ± 0%     ~     (p=0.640 n=10+9)
RTTSeqServerCGo1      456ns ± 0%   466ns ± 0%   +2.17%  (p=0.000 n=9+9)
RTTSeqServerCGo1-4    532ns ± 1%   532ns ± 1%     ~     (p=0.751 n=10+10)
RTTSeqServerGo10      419ns ± 1%   424ns ± 0%   +1.17%  (p=0.000 n=10+10)
RTTSeqServerGo10-4    486ns ± 1%   484ns ± 1%   -0.51%  (p=0.018 n=10+10)
RTTSeqServerCGo10    1.00µs ± 0%  1.02µs ± 0%   +1.44%  (p=0.000 n=9+10)
RTTSeqServerCGo10-4  1.97µs ± 4%  1.13µs ± 1%  -42.92%  (p=0.000 n=10+10)

However it is not clear why the fact of making 10 CGo calls in sequence affected the scheduler to make this decision: the individual CGo call time is well under 20µs - the time, if I understood correctly, after which sysmon decides that something is hanging in a CGo or syscall for too much and removes current OS thread from a GOMAXPROCS slot. The fact that sysmon is sleeping for >1ms in strace also confirm that sysmon does not intervene here.
I do not know Go scheduler insides, but clearly making several CGo call in sequence shakes it enough to bring some disorder and slowdown. For the reference when I was seeing the problem with SQLite my query time was ~ 5-6µs if I use the go driver directly and with standard
go func() {
	select {
	case <-ctx.Done():
		// call some cancel

	case <-done:
		// work finished ok
	}
}()

// perform query
close(done)

spawned around query with ctx never being canceled it was increasing the time to ~10µs. In other words it can be not only 1µs of additional penalty but higher.
In summary, like @petermattis said it looks like CGo calls sequence trigers some badness in Go scheduler that would be good to understand and hopefully fix.
Thanks beforehand,
Kirill
/cc @ianlancetaylor, @dvyukov, @rsc, @aclements, @bcmills, @mattn

(cgo.go)
package cgotest

// int argadjust(int arg) { return 3 + arg; }
import "C"

// XXX here because cannot use C in tests directly
func cargadjust(arg int) int {
        return int(C.argadjust(C.int(arg)))
}

(cgo_test.go)
package cgotest

import "testing"

//go:noinline
func goargadjust(i int) int {
	return i + 3
}

// BenchmarkCallGo1 measures overhead of 1 Go call.
func BenchmarkCallGo1(b *testing.B) {
	for i := 0; i < b.N; i++ {
		x := goargadjust(i)
		if x != i + 3 {
			b.Fatalf("goargadjust -> %d  ; want %d", x, i + 3)
		}
	}
}

// BenchmarkCallGo10 measures overhead of 10 Go calls performed sequentially.
func BenchmarkCallGo10(b *testing.B) {
	for i := 0; i < b.N; i++ {
		for j := 0; j < 10; j++ {
			x := goargadjust(i)
			if x != i + 3 {
				b.Fatalf("goargadjust -> %d  ; want %d", x, i + 3)
			}
		}
	}
}

// BenchmarkCallCGo1 measures overhead of 1 CGo call.
func BenchmarkCallCGo1(b *testing.B) {
	for i := 0; i < b.N; i++ {
		x := cargadjust(i)
		if x != i + 3 {
			b.Fatalf("cargadjust -> %d  ; want %d", x, i + 3)
		}
	}
}

// BenchmarkCallCGo10 measures overhead of 10 CGo calls performed sequentially.
func BenchmarkCallCGo10(b *testing.B) {
	for i := 0; i < b.N; i++ {
		for j := 0; j < 10; j++ {
			x := cargadjust(i)
			if x != i + 3 {
				b.Fatalf("cargadjust -> %d  ; want %d", x, i + 3)
			}
		}
	}
}

// ---------

type req struct {
	respq chan *resp
	arg   int
}

type resp struct {
	ret int
}

// client performs one request-reply cycle to a server over srvlink.
func client(srvlink chan *req, arg int) int {
	rc := make(chan *resp)
	srvlink <- &req{
		respq: rc,
		arg:   arg,
	}
	ret := <-rc
	return ret.ret
}

// server receives requests over inq, passes received request to
// handler, and returns result back over response channel.
//
// server runs until it processes all requests from inq.
type server func(inq chan *req, handler func(int) int)

// seqServer sequentially handles requests received over inq.
func seqServer(inq chan *req, handler func(int) int) {
	for r := range inq {
		r.respq <- &resp{ret: handler(r.arg)}
	}
}

// goServer concurently handles requests received over inq.
func goServer(inq chan *req, handler func(int) int) {
	for r := range inq {
		r := r
		go func() {
			r.respq <- &resp{ret: handler(r.arg)}
		}()
	}
}

// benchmarkRTT measures client-server round-trip latency for a particular
// server implementation.
func benchmarkRTT(b *testing.B, srv func(chan *req)) {
	inc := make(chan *req)
	go srv(inc)
	for i := 0; i < b.N; i++ {
		client(inc, i)
	}
	close(inc)
}

// --------

// mkServerGo1 returns function that will runs srv with N Go calls per request.
func mkServerGo1N(srv server, n int) func(chan *req) {
	return func(inq chan *req) {
		srv(inq, func(arg int) int {
			for i := 0; i < n; i++ {
				arg = goargadjust(arg)
			}
			return arg
		})
	}
}

// mkServerCGoN returns function that will run srv with N CGo calls per request.
func mkServerCGoN(srv server, n int) func(chan *req) {
	return func(inq chan *req) {
		srv(inq, func(arg int) int {
			for i := 0; i < n; i++ {
				arg = cargadjust(arg)
			}
			return arg
		})
	}
}

func BenchmarkRTTSeqServerGo1(b *testing.B)	{ benchmarkRTT(b, mkServerGo1N(seqServer, 1)) }
//func BenchmarkRTTGoServerGo1(b *testing.B)	{ benchmarkRTT(b, mkServerGo1(goServer)) }

func BenchmarkRTTSeqServerCGo1(b *testing.B)	{ benchmarkRTT(b, mkServerCGoN(seqServer, 1)) }
//func BenchmarkRTTGoServerCGo1(b *testing.B)	{ benchmarkRTT(b, mkServerCGoN(goServer,  1)) }

func BenchmarkRTTSeqServerGo10(b *testing.B)	{ benchmarkRTT(b, mkServerGo1N(seqServer, 10)) }
func BenchmarkRTTSeqServerCGo10(b *testing.B)	{ benchmarkRTT(b, mkServerCGoN(seqServer, 10)) }
//func BenchmarkRTTGoServerCGo10(b *testing.B)	{ benchmarkRTT(b, mkServerCGoN(goServer,  10)) }

However it is not clear why the fact of making 10 CGo calls in sequence affected the scheduler to make this decision

From run time perspective both RTTSeqServerCGo1 and RTTSeqServerCGo10 do exactly the same -- call a long sequences of cgo calls. From run time perspective the inner loop does not make any difference.
So there is something else happening. CPU scaling/turbo/overheat? Have you tried running them for longer? in isolation? with CPU affinity? turning off CPU scaling?

@dvyukov thanks for feedback.

From run time perspective both RTTSeqServerCGo1 and RTTSeqServerCGo10 do exactly the same -- call a long sequences of cgo calls.

There is a difference - CGo1 case is like this:
        Gclient                 Gserver

        srvlink <- req
                                req <- srvlink
                                CGo call x1
                                resp <- ...
        resp <- ...

while CGo10 is like:
        Gclient                 Gserver

        srvlink <- req
                                req <- srvlink
                                CGo call x10
                                resp <- ...
        resp <- ...

every CGo call triggers at least runtime.entersyscall / runtime.exitsyscall so there is a difference in runtime call sequences in between CGo1 and CGo10 - channel sends operations are also there and the difference is in how many cgo calls are made in between server receives a request and sends a reply for it back.
I was running the tests with cpupower frequency-set -g performance and monitoring dmesg -w for overheating and that CPU throttling is not getting in. I've also reverified timings several times via running each test individually with a pause (so CPU can get calmer) in between different runs.
The only mistake from me in previous runs was that the turbo was not disabled so now to recheck I disable it with cpupower frequency-set --min 2.6Ghz --max 2.6GHz (2.6GHz is max non-turbo frequency of my cpu) and show you below a complete dump of a relatively long run where different tests are intermixed with each other so that if e.g. overheating or something else gets in it should be visible to both CGo1 and CGo10:
$ ./neotest info-local
# Sun, 18 Feb 2018 17:04:37 +0300
#<EMAIL_ADDRESS>(2401:5180:0:37::1 <IP_ADDRESS>)
# Linux deco 4.14.0-3-amd64 #1 SMP Debian 4.14.13-1 (2018-01-14) x86_64 GNU/Linux
# cpu: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
# cpu[0-3]: freq: intel_pstate/performance [2.60GHz - 2.60GHz]
# cpu[0-3]: idle: intel_idle/menu: POLL(0μs) C1(2μs) C1E(10μs) C3(70μs) C6(85μs) C7s(124μs) C8(200μs) C9(480μs) C10(890μs)
# cpu: WARNING: C-state exit-latency is max 890μs - up to that can add to networked and IPC request-reply latency
# sda: SanDisk X400 M.2  rev 0012  477G
# eth0: Intel Corporation Ethernet Connection I219-LM rev 21
# eth0: features: rx tx sg tso !ufo gso gro !lro rxvlan txvlan !ntuple rxhash ...
# eth0: coalesce: rxc: 3μs/0f/0μs-irq/0f-irq,  txc: 0μs/0f/0μs-irq/0f-irq
# eth0: up, speed=1000, mtu=1500, txqlen=1000, gro_flush_timeout=0.000µs
# wlan0: Intel Corporation Wireless 8260 rev 3a
# wlan0: features: !rx !tx sg !tso !ufo gso gro !lro !rxvlan !txvlan !ntuple !rxhash ...
# wlan0: coalesce: rxc: ?,  txc: ?
# wlan0: down, speed=?, mtu=1500, txqlen=1000, gro_flush_timeout=0.000µs
# wlan0: WARNING: TSO not enabled - TCP latency with packets > MSS will be poor
# Python 2.7.14
# go version go1.10 linux/amd64
# sqlite 3.22.0 (py mod 2.6.0)
# mysqld  Ver 10.1.29-MariaDB-6 for debian-linux-gnu on x86_64 (Debian buildd-unstable)
# neo             : v1.8.1-1532-gc0502d41-dirty
# zodb            : 5.3.0-11-gde1f24ca0
# zeo             : 5.1.1-1-gfdcc2739
# mysqlclient     : 1.3.12
# wendelin.core   : v0.11-14-gc3cc8a9

$ time bash -c 'for in in `seq 10`; do ./cgo.test -test.bench "RTTSeqServerCGo1.*" -test.benchtime 10s -test.cpu 1,4; done'
goos: linux
goarch: amd64
BenchmarkRTTSeqServerCGo1       30000000               592 ns/op
BenchmarkRTTSeqServerCGo1-4     20000000               640 ns/op
BenchmarkRTTSeqServerCGo10      10000000              1302 ns/op
BenchmarkRTTSeqServerCGo10-4     5000000              2499 ns/op
PASS
goos: linux
goarch: amd64
BenchmarkRTTSeqServerCGo1       30000000               592 ns/op
BenchmarkRTTSeqServerCGo1-4     20000000               652 ns/op
BenchmarkRTTSeqServerCGo10      10000000              1319 ns/op
BenchmarkRTTSeqServerCGo10-4     5000000              2387 ns/op
PASS
goos: linux
goarch: amd64
BenchmarkRTTSeqServerCGo1       30000000               592 ns/op
BenchmarkRTTSeqServerCGo1-4     20000000               640 ns/op
BenchmarkRTTSeqServerCGo10      10000000              1303 ns/op
BenchmarkRTTSeqServerCGo10-4     5000000              2489 ns/op
PASS
goos: linux
goarch: amd64
BenchmarkRTTSeqServerCGo1       20000000               604 ns/op
BenchmarkRTTSeqServerCGo1-4     20000000               651 ns/op
BenchmarkRTTSeqServerCGo10      10000000              1335 ns/op
BenchmarkRTTSeqServerCGo10-4     5000000              2464 ns/op
PASS
goos: linux
goarch: amd64
BenchmarkRTTSeqServerCGo1       30000000               595 ns/op
BenchmarkRTTSeqServerCGo1-4     20000000               649 ns/op
BenchmarkRTTSeqServerCGo10      10000000              1329 ns/op
BenchmarkRTTSeqServerCGo10-4     5000000              2423 ns/op
PASS
goos: linux
goarch: amd64
BenchmarkRTTSeqServerCGo1       20000000               601 ns/op
BenchmarkRTTSeqServerCGo1-4     20000000               646 ns/op
BenchmarkRTTSeqServerCGo10      10000000              1300 ns/op
BenchmarkRTTSeqServerCGo10-4     5000000              2392 ns/op
PASS
goos: linux
goarch: amd64
BenchmarkRTTSeqServerCGo1       20000000               603 ns/op
BenchmarkRTTSeqServerCGo1-4     20000000               640 ns/op
BenchmarkRTTSeqServerCGo10      10000000              1309 ns/op
BenchmarkRTTSeqServerCGo10-4     5000000              2482 ns/op
PASS
goos: linux
goarch: amd64
BenchmarkRTTSeqServerCGo1       20000000               601 ns/op
BenchmarkRTTSeqServerCGo1-4     20000000               647 ns/op
BenchmarkRTTSeqServerCGo10      10000000              1313 ns/op
BenchmarkRTTSeqServerCGo10-4     5000000              2439 ns/op
PASS
goos: linux
goarch: amd64
BenchmarkRTTSeqServerCGo1       30000000               597 ns/op
BenchmarkRTTSeqServerCGo1-4     20000000               647 ns/op
BenchmarkRTTSeqServerCGo10      10000000              1306 ns/op
BenchmarkRTTSeqServerCGo10-4    10000000              2451 ns/op
PASS
goos: linux
goarch: amd64
BenchmarkRTTSeqServerCGo1       30000000               597 ns/op
BenchmarkRTTSeqServerCGo1-4     20000000               656 ns/op
BenchmarkRTTSeqServerCGo10      10000000              1312 ns/op
BenchmarkRTTSeqServerCGo10-4     5000000              2424 ns/op
PASS

real    10m1,226s
user    11m14,922s
sys     0m42,169s

i.e. the timings are stable, there is no outliers and it is
RTTSeqServerCGo1      597ns ± 1%
RTTSeqServerCGo1-4    647ns ± 1%
RTTSeqServerCGo10    1.31µs ± 2%
RTTSeqServerCGo10-4  2.44µs ± 2%

this covers "scaling/turbo/overheat and "running for longer".
Now let's check setting affinity. First: just setting GOMAXPROCS=2 does not make much difference to default GOMAXPROCS=4:
$ time bash -c 'for in in `seq 3`; do ./cgo.test -test.bench "RTTSeqServerCGo1.*" -test.benchtime 10s -test.cpu 2; done'
goos: linux
goarch: amd64
BenchmarkRTTSeqServerCGo1-2     20000000               647 ns/op
BenchmarkRTTSeqServerCGo10-2     5000000              2682 ns/op
PASS
goos: linux
goarch: amd64
BenchmarkRTTSeqServerCGo1-2     20000000               655 ns/op
BenchmarkRTTSeqServerCGo10-2     5000000              2615 ns/op
PASS
goos: linux
goarch: amd64
BenchmarkRTTSeqServerCGo1-2     20000000               655 ns/op
BenchmarkRTTSeqServerCGo10-2     5000000              2551 ns/op
PASS

real    1m28,294s
user    1m47,062s
sys     0m14,043s

but CGo10-2 overhead goes away if I set test process affinity to only two first CPUs:
$ time taskset -c 0,1 bash -c 'for in in `seq 3`; do ./cgo.test -test.bench "RTTSeqServerCGo1.*" -test.benchtime 10s; done'
goos: linux
goarch: amd64
BenchmarkRTTSeqServerCGo1-2     20000000               608 ns/op
BenchmarkRTTSeqServerCGo10-2    10000000              1364 ns/op
PASS
goos: linux
goarch: amd64
BenchmarkRTTSeqServerCGo1-2     20000000               619 ns/op
BenchmarkRTTSeqServerCGo10-2    10000000              1369 ns/op
PASS
goos: linux
goarch: amd64
BenchmarkRTTSeqServerCGo1-2     20000000               604 ns/op
BenchmarkRTTSeqServerCGo10-2    10000000              1376 ns/op
PASS

real    1m23,806s
user    1m28,831s
sys     0m1,393s

probably because setting affinity removes many context switches:
$ perf stat ./cgo.test -test.bench "RTTSeqServerCGo10" -test.benchtime 10s
goos: linux
goarch: amd64
BenchmarkRTTSeqServerCGo10-4    10000000              1741 ns/op
PASS

 Performance counter stats for './cgo.test -test.bench RTTSeqServerCGo10 -test.benchtime 10s':

      24547,864730      task-clock (msec)         #    1,280 CPUs utilized
         1 968 952      context-switches          #    0,080 M/sec
               745      cpu-migrations            #    0,030 K/sec
             1 824      page-faults               #    0,074 K/sec
    59 997 399 275      cycles                    #    2,444 GHz
    95 671 377 235      instructions              #    1,59  insn per cycle
    19 118 977 436      branches                  #  778,845 M/sec
        76 635 472      branch-misses             #    0,40% of all branches

      19,172156094 seconds time elapsed

$ taskset -c 0,1 perf stat ./cgo.test -test.bench "RTTSeqServerCGo10" -test.benchtime 10s
goos: linux
goarch: amd64
BenchmarkRTTSeqServerCGo10-2    10000000              1459 ns/op
PASS

 Performance counter stats for './cgo.test -test.bench RTTSeqServerCGo10 -test.benchtime 10s':

      18226,710279      task-clock (msec)         #    1,136 CPUs utilized
           865 349      context-switches          #    0,047 M/sec
             1 729      cpu-migrations            #    0,095 K/sec
             1 797      page-faults               #    0,099 K/sec
    45 725 017 537      cycles                    #    2,509 GHz
    85 013 647 314      instructions              #    1,86  insn per cycle
    17 127 907 786      branches                  #  939,715 M/sec
        72 766 706      branch-misses             #    0,42% of all branches

      16,045203680 seconds time elapsed

and it was already discussed in the second half of https://github.com/golang/go/issues/21827#issue-256508518 how futex_wake can be changing CPU of a woken process.
HOWEVER
Setting affinity cannot work in practice - if we imagine a server which for every request spawns separate goroutine, and the CGo calls have to be done from under that request-handling goroutine, we now have N(goroutines) > Ncpu. Since we cannot set affinity of a goroutine - at least we cannot do so at present with public API - the only choise is to let the whole process use all availble CPU resources dedicated to service and oops -> context switches could be happenning all the time.

There is a difference - CGo1 case is like this:

Ah, sorry. I misread the test. I thought it's just cgo calls.

np, and thanks for feedback. Looking forward to what might be done about it.

There were several projects which were being rewritten in C or C++, because of cgo overhead.
One example : https://github.com/canonical/dqlite/blob/master/doc/faq.md#why-c
When I tried to introduce go to my team, same problem (In 2018).
Can somebody try to work on this?
Thank you.

Is there any update on this? I'm also having this problem too.
For my use case, I'm using https://github.com/cisco/openh264, which is written in C++, to encode every frame that I get from my camera through https://en.wikipedia.org/wiki/Video4Linux.
Following is the output from pprof

There are no updates to this issue.  If there are updates, they will be reported here.
  The Go profiler won't by default tell you anything about C code.  If it would be useful for you you can try importing github.com/ianlancetaylor/cgosymbolizer to get C code profiling; that is completely unsupported but it may help.  It requires that your C code be compiled with the -g option.

Ah I see, I'm sorry for the misunderstanding. I'll try to import github.com/ianlancetaylor/cgosymbolizer. Thanks.

FYI I ran the test in the OP on 1.17 & cgo performance roughly matches go performance (and may be slightly better?)

I also ran navytux's test, and the cgo overhead does still roughly double when there are go operations in flight for his RTT tests, but no evidence of anything with as large of impact as the OP.

I do think the overhead, such as it is, is likely related to #21827.  Goroutines that interact with cgo are likely to end up, via brownian motion, in the same position as an OS-locked thread: a single G on a single M/P.  When the thread parks to collect info, its M is parked in turn.  We don't end up in this case when GOMAXPROCS=1, because cgo can't subtly move the G onto its own M/P.
I think broadly there's an issue where parking an M is intended to happen when there is clearly not enough work around to sustain an entire M, but there are a few cases (os-locked threads, threads that talk to cgo a lot) where there's good reason for a G to be hanging out on its own.  It would probably be better if os-locked threads never parked their M and then we could os-lock threads if we are talking to cgo a lot and have a problem with this behavior.

Here is a new scenario that make me concern the performance a lot: Conn Read/Write on Windows, which calls API WSARecv/WSASend according runtime.cgocall. It is a very basic functionality for web servers, proxies, and other network data transferring tools.
Low performance will pop up when there is a large amount of data, with high CPU usage (10%+, task manager) which could be 10+ times than rust/c developed similar apps.
An alternative way to Read/Write directly is using a big enough buffer. By piling up packets data and then Read/Write in one time, it definitely reduces the context switching costs. But, it has disadvantages and does not 100% work:

buffers should not be too big, or when connections increase, they consume lots of memory;
if packets data can't be piled up, that will make buffering ineffective;
must be answered small and high frequency packets are common.

It is really a pity to see it is unplanned!
Many people have noticed it and researched on it. Hoping it could draw more attentions and more improvements could be achieved, earlier.

Would probably help if you got a self-contained example like in the OP and opened a new issue. The example in this one no longer produces the results linked in the OP, so I would not expect to see this task addressed, but your new one might be.

@CannibalVox
Thank you for your advice! I have create https://github.com/golang/go/issues/58336.


Fix schedule and unschedule not updating UI bug

Addresses issue https://github.com/AY1920S1-CS2103T-T09-1/main/issues/141
Listener in agenda listens to the list of days. Since day is a list of activity, when the list of activity mutates, agenda fails to receive an update because it only checks on days mutation and not the internal mutation of day.
Temporarily fix the bug by forcing an update to Itinerary when Ui focuses on itinerary.

LGTM


The name 'Resource' does not exist in the current context

The name 'Resource' does not exist in the current context
I get this error on "MainActivity.cs" file in Android project. Any suggestions?

Can you provide additional information on this issue? When are you getting this error? Does it happen during the build? If so, could you send us the complete build output? What version of Xamarin,  Mono.Android, Xamarin Studio, Visual Studio etc. are you using?

@biasmv Yes, it happens during the build. Build is failing and giving this error two times:

I'm using Xamarin 4.4 and Visual Studio 2017. I didn't make any changes in your sample project.

The missing Resource symbol is is automatically generated by the Android/Xamarin build system during the build.  It's possible that you are seeing this because of some setup issue with your Android or Xamarin SDK. Sometimes these issues can also be resolved by cleaning the solution as well as removing any temporary files from the project folder, especially those in obj/bin folders.
We tried to reproduce the issue on our side. Unfortunately, we couldn't reproduce it. We are using a slightly more up-to-date version of Xamarin (4.5) together with the latest Visual Studio 2017 update. It's possible that updating to the latest Xamarin and Visual Studio resolves the issue for you.

There hasn't been an update on this for a while. Were you able to resolve the issue?

Sorry, I was in a hurry and I've started to use another library for scanning.

No worries, I'll close the ticket then.


Vickel: Watch quangnamtoplist.com

Vickel requests the watch of the watch_keyword quangnamtoplist\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

quangnamtoplist\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.


Kittenbot Jacdac-Power module (Kit B) is displayed twice as UE21 and KA82

Describe the bug
A brief search did not show a duplicate issue.
Kittenbot Jacdac-Power module (Kit B) is displayed twice as UE21 and KA82 in the Jacdac Dashboard:

In makecode.microbit.org the power module is not displayed below the simulator.
To Reproduce

Flash Jacdac firmware to microbit
Go to jacdac dashboard
Connect power module
Power module appears twice as UE21 and KA82

Expected behavior
Only one instance of the power module should be shown.

Sure. In this trace I connected the power module after a couple of seconds and unplugged it a couple of seconds later.
trace_power.jd.txt

@xmeow @smfox10 please take a look

@fabianhugo Thank you for the feedback. Actually there is two separate power supply controllers on the module as discussed here: https://github.com/microsoft/jacdac/discussions/373
But only one should be working at same time if there is external wiring to connect the two power supply together.

@fabianhugo
On the Jacdac platform, there are two Jacdac power modules displayed (you can see two PADAUK ICs), which is not a bug. They are responsible for controlling the power on/off of the two Jacdac interface channels.
On the Makecode platform, the power module is not displayed. This decision was made in communication with Peli. Since Makecode platform programming is simple and there is no need to turn off the module, the power is generally kept on. Therefore, the icon was hidden.

I will add a note on the device page. It seemed like a good idea to hide them... but we could revisit this decision.

Fixed by https://github.com/microsoft/jacdac-docs/pull/280


Avoid all unchecked indexing in match checking

Fixes https://github.com/rust-analyzer/rust-analyzer/issues/4416, but replaces it with a false positive.
r? @flodiebold

bors r+


[TEST] modify test case for test_recurring_job.test_recurring_jobs_maximum_retain

What's the test to develop? Please describe
The max limit has been adjusted to 100, so the test case test_recurring_job.test_recurring_jobs_maximum_retain should be adjusted.
Describe the items of the test development (DoD, definition of done) you'd like

Please use a task list for items on a separate line with a clickable checkbox https://docs.github.com/en/issues/tracking-your-work-with-issues/about-task-lists

[x] Modify script
[x] Verify on pipeline

Additional context
test_recurring_jobs_maximum_retain.
https://github.com/longhorn/longhorn-manager/commit/d75ea46a167c850eb8abc7cb74cee5b2305d63a2

Unit test case report : https://ci.longhorn.io/job/private/job/longhorn-tests-regression/3422/console

test_recurring_job.py file test report : https://ci.longhorn.io/job/private/job/longhorn-tests-regression/3421/console

The https://github.com/longhorn/longhorn-tests/pull/1301 has been merged and backported to v1.3.x and v1.4.x.


Fix string_view with null_ptr

std::string_view cannot have null_ptr unlike absl::string_view

grpc/core/master/linux/grpc_portability: running


Adding Code for Debian 12

As Debian 12 is released, I've added code to package_build.py as stated in Issue #144 . Please take a look. @pleia2 @arshPratap

@pleia2 I've made the asked changes. Please take a look.
Except this I've also remove the comment for fedora 38 as it available in our tracked data file now.


DownloadFiles vs DownloadFile disprepency in return values

As part of:

https://github.com/robinrodricks/FluentFTP/issues/1356

I've found the hard way the API changed and you now return proper object when using DownloadFiles
 $Message = $Client.DownloadFiles($LocalPath, ([string[]] $FileToDownload), $LocalExists, $VerifyOptions, $FtpError)

But it seems you continue to return string as part of DownloadFile. I think it would make sense to fix this disprepancy and have both act the same way?

I've found the hard  way the API changed

That is entirely a matter of opinion. Sometime an API change is needed, but it is actually easy to see that, because we use a sensible versioning scheme ( 44.x.x -> 48.x.x ).

But it seems you continue to return string as part of DownloadFile

Huh?
public FtpStatus DownloadFile(string localPath, string remotePath, FtpLocalExists existsMode = FtpLocalExists.Overwrite, FtpVerify verifyOptions = FtpVerify.None, Action<FtpProgress> progress = null) {

public List<FtpResult> DownloadFiles(string localDir, IEnumerable<string> remotePaths, FtpLocalExists existsMode = FtpLocalExists.Overwrite, FtpVerify verifyOptions = FtpVerify.None, FtpError errorHandling = FtpError.None, Action<FtpProgress> progress = null, List<FtpRule> rules = null) {

have both act the same way?

One is a FtpResult, the other a List<FtpResult>. Are you saying, you want a list for DownloadFile as well, with only one list element?

I've found the hard  way the API changed

That is entirely a matter of opinion. Sometime an API change is needed, but it is actually easy to see that, because we use a sensible versioning scheme ( 44.x.x -> 48.x.x ).

Don't get me wrong. It was entirely my fault. I wast just referencing issue.
As for the other issue (straight from your screenshot) :-)

What I am saying is I get FtpStatus enum, rather than FtpResult

And I can see that when running code.

 I need new glasses.

I am also of the opinion that this can be improved.
I think that this will be put on the "enhancement" - "todo" list, as it will be a disruptive change if there is agreement to do this.

Moved to Wiki

@PrzemyslawKlys I was about to implement this but the changes required are a lot and the result is fundamentally different for a very simple reason. They both behave differently:

DownloadFile

Throws exceptions for errors (important also as these exceptions are caught by higher-order functions like DownloadFiles)
RemotePath and LocalPath are already provided to it, so it does not need to return it
IsSkipped is out of the question, no mechanism for filtering files
IsFailed is not required, just throws exceptions on failure
Type and IsDownload are irrelevant, you know it is a file being downloaded

DownloadFiles

Does NOT throws exceptions for errors because otherwise the entire function would abort and we instead resume on errors and just skip the failing file
RemotePath and LocalPath are NOT provided by the caller, but instead computed by the function, so it makes sense to return it back to the user
IsSkipped is set when the file is filtered by rules
IsFailed is required, because exceptions are not used to indicate a file was failed
Type is used to indicate if its a file/folder/link

So for these reasons I see no reason why we should return a FtpResult from DownloadFile or UploadFile but thanks for your suggestion.

Added FAQ here.
https://github.com/robinrodricks/FluentFTP/wiki/File-Transfer#why-does-downloadfile-and-downloadfiles-return-a-different-data-type

@PrzemyslawKlys Released.
https://www.nuget.org/packages/FluentFTP/48.0.2

New: FtpResult.ToStatus() API to easily compare result values of DownloadFile/DownloadFiles and UploadFile/UploadFiles

Does this help?

I already have simple logic that based whether it's success or not, it creates fake properties.

https://github.com/EvotecIT/Transferetto/blob/master/Private/Get-PrivateFTPFile.ps1

So i'm fine with your decisions, just I know users in PowerShell prefer same output, especially that I don't separate 1 file vs multiple files and I keep them under single command, and depending whether you pass 1 or multiple files it goes one way or the other.


Connect DBeaver to DB2 on Cloud

Not directly associated with the code in dse-do-utils, but very useful to use DB2 on Cloud as a scenario database.
DBeaver works fine with DB2 on CPD where no SSL is required.
DB2 on Cloud requires SSL. DBeaver needs additional specifications:
https://www.ibm.com/docs/en/fci/6.5.1?topic=application-connecting-db2-over-ssl-port
In particular the settings:
sslTrustStoreLocation
sslTrustStorePassword
sslConnection=True
In the connection properties of the service credentials of the db2 user in DB2 Cloud is a certificate_base64. This might be the certificate we need.
To add a SSL certificate to the TrsutStore:
https://www.ibm.com/support/pages/complete-guide-set-ssl-using-ibm-data-server-driver-jdbc-and-sqlj
TODO:

How to add the certificate to the trust store?
Where is the trust store password?

Only need to add the following driver properties:
sslConnection=True
SECURITY=SSL (not sure if required, but doesn't hurt)
No need for the TrustSTore
Sometime the opening/querying of tables times-out, but that may not be related to SSL as it sometimes works.


SUP-15914: Admin-UI Tagtype not saved when asked if you want to save

Fix repository browser for forms
SUP-16037: Fixed item-list header checkbox handling
Fix scheduling in intervals to always calculate from the start time
Add more test cases
Make test independent of local time-zone
Fix reading exit codes >= 128
SUP-15901: Enhanced GCMSUI-Bridge and Tag Editor context
Fix consistency check if project per node is disabled
SUP-15965 Add pubdir check when publish dir segments is active
[Jenkins | release-6.0.x] Release of version 6.0.10
[Jenkins | release-6.0.x] Prepare for the next development iteration (6.0.11-SNAPSHOT)
Refresh entity data
SUP-15901: Added changelog entry
[Jenkins | release-6.0.x] Release of version 6.0.11
[Jenkins | release-6.0.x] Prepare for the next development iteration (6.0.12-SNAPSHOT)
SUP-16037: Fixed status-icon reset on item display
Fix URL uniqueness checks and publish paths for publishing into the filesystem
[Jenkins | release-6.0.x] Release of version 6.0.12
[Jenkins | release-6.0.x] Prepare for the next development iteration (6.0.13-SNAPSHOT)
SUP-16085 Fix page online status during instant publish
Fix link to admin UI
SUP-15790 Purge removed tagpart types from documentation
Fix template preselection, add changelog
Add more duration info to publish log
SUP-16032: Added mesh role sync management
Handling/Canceling search-bar select events
Changed page-content icon to be unique
Only read roles once per publish run Do not set role permissions on projects/branches/root node, if already set.
[Jenkins | release-6.0.x] Release of version 6.0.13
[Jenkins | release-6.0.x] Prepare for the next development iteration (6.0.14-SNAPSHOT)
Add REST endpoints Refactor getting nodes to which templates are linked
Remove unnecessary join
SUP-16045: Fixed module route permissions
Fix page language indicator not showing languages (retrieve page variants on reload)
Fix display of error message when construct cannot be deleted, because it is still in use
SUP-15790 Remove breadcrumb & navigation part types from documentation
Disable Ryuk
Remove pipeline post actions concerning gitlab
Fix getting the page language in PageFactory.flush() if language is already about to be deleted
[Jenkins | release-6.0.x] Release of version 6.0.14
[Jenkins | release-6.0.x] Prepare for the next development iteration (6.0.15-SNAPSHOT)
SUP-16226: Null check the value on put to the content package
[Jenkins | release-6.0.x] Release of version 6.0.15
[Jenkins | release-6.0.x] Prepare for the next development iteration (6.0.16-SNAPSHOT)
Revert "SUP-16226: Null check the value on put to the content package"
Check PartType for existence + test
Changelog
Changelog typo
Fix handling of non-existent partTypeId
Make deleting constructs more fault tolerant Fix test error
[Jenkins | release-6.0.x] Release of version 6.0.16
[Jenkins | release-6.0.x] Prepare for the next development iteration (6.0.17-SNAPSHOT)
Fix possible NPE
SUP-15059: Construct-Part sync conflict dialog
SUP-15061: Construct Part cleanup and sorting
SUP-15874: object properties not copied on image copy
Changelog
SUP-15873: Fixed search-bar autofocus handling
Add option to install the maven artifacts locally
Fix maven arguments when installing artifacts locally
[Jenkins | release-6.0.x] Release of version 6.0.17
[Jenkins | release-6.0.x] Prepare for the next development iteration (6.0.18-SNAPSHOT)
Always add folderId to search parameters
Add mariadb-client to docker image
Fix errors when updating object property definitions
Fix permission check for "publish" permission to always consider the object's own node/channel. Fix request to clear publishAt of offlineAt to properly response with "no permission" if the user is lacking "publish" permission.
Fix handling of languages when copying/moving pages between nodes
Update Mesh Dependency
Fix test
Fix copying language variants into other nodes
[Jenkins | release-6.0.x] Release of version 6.0.18
[Jenkins | release-6.0.x] Prepare for the next development iteration (6.0.19-SNAPSHOT)
Remove markings of link when link is changed to be an internal one.
[Jenkins | release-6.0.x] Release of version 6.0.19
[Jenkins | release-6.0.x] Prepare for the next development iteration (6.0.20-SNAPSHOT)
Fix rendering of overviews
Another sort fix
Improve documentation
Add clear button to select dropdown, add changelog
Add changelog
Improve error message
SUP-12268: Trim the spaces
Fix NPE for submitted form data values, add changelog
Minor fixes
Fix test
SUP-9507: Added additional options when opening the tag-fill
Add missing fix
[Jenkins | release-6.0.x] Release of version 6.0.20
[Jenkins | release-6.0.x] Prepare for the next development iteration (6.0.21-SNAPSHOT)
SUP-16004: Fix an access to the symlinked static resources of packages
Changelog
SUP-15443: Fix of content staging crash
Service resilving simplification
SUP-16516 Fileupload: fix typo in maxfilesize setting
Fix extended
Simplify even more
Sorting/Filtering fixes Add Javadoc to sorting/filtering REST API Endpoints
Fix tests to be independent from licensed features
Replace ServiceLoader by ServiceLoaderUtil which is thread-safe
Also install cms-oss-bom
Fix vanishing parts by adding parts to payload

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.5 out of 7 committers have signed the CLA.:white_check_mark: npomaroli:white_check_mark: yrucrem:white_check_mark: rainerda:white_check_mark: deckdom:white_check_mark: guggi:x: GenticsDev:x: Serhii PlyhunSerhii Plyhun seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Farmland issue with 2 phase crops

Just Dire Things version
1.5.1
Minecraft Version
1.21
(Neo)Forge Version
No response
Modpack & Version
No response
Do you have optifine or Rubidium installed?
None
Describe the issue
with your soil, plants like the tomato from Farmers Delight where the plant grows and then the fruit without destroying the plant itself at harvest, it destroys the plant and it takes ages to grow back
Steps to reproduce

make a farm land with the Blazegold Hoe
plant the tomato
wait until the plant is grown and the tomato
if the farmland harvests it destroys the entire plant to regrow

Expected behaviour
It should only harvest the tomato and not destroy the tomato plant
Screenshots
No response
Log files
No response
Additional information
No response

Unfortunately theres no way for me to "know" these plants behave this way, as they are entirely custom concepts.


Need Antora playground

Can we get a playground repo for your Antora docs or perhaps access to mess with one of your library's docs repos @vinniefalco ?
Need this to have quick dev / test cycles on the shared header.

I believe this already exists, you want to edit the "ui bundle" and there is a command "gulp preview" which runs a local webserver and updates in real time with your changes.

@gregnewman I think that works yes?

Yes, that should work.


Spelling corrections

Changed "verifing" to "verifying"

Hi @deneha, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

This seems like a small (but important) contribution, so no contribution license agreement is required at this point. Real humans will now evaluate your PR.

TTYL, AZPRBOT;


Test2

pull_request: https://github.com/kenji-miyake/test-github-actions/runs/4439853239?check_suite_focus=true
push: https://github.com/kenji-miyake/test-github-actions/runs/4439849491?check_suite_focus=true


Add related images to bundle generation

Signed-off-by: Ryan King<EMAIL_ADDRESS>
Description of the change:
During bundle generation (operator-sdk generate bundle or make bundle), operator-sdk will look through the environment of the controller manager for related images, named RELATED_IMAGE_<IMAGE_NAME>, and add them to the ClusterServiceVersion spec.relatedImages field.
Motivation for the change:
Checklist
If the pull request includes user-facing changes, extra documentation is required:

[ ] Add a new changelog fragment in changelog/fragments (see changelog/fragments/00-template.yaml)
[ ] Add or update relevant sections of the docs website in website/content/en/docs

/lgtm

/lgtm


Merge fix for covergroupCoverage

Merge tool fix to traverse incoming parseRoot instead of mergeDBtree and to account for covergroupCoverage hierarchy layer.

Hi,
Thank you for the fix. When running locally, I encountered another issue in one of the queries searching for parent element of a coverpointBin element (parse_coverpoint_bins function). I fixed the problem and (as a bonus) refactored all the queries so the code is much clearer now.
These updates are available under the following commit: https://github.com/amiq-consulting/fc4sc/commit/d2fdad86afb7ed35d7285c24823e4c3cacf820d1
Best Regards


":basename" does not behave as expected -- it includes the extension

using ":basename" includes the original file's extension -- given the following:
var gdest = require('gulp-dest');

//...

gulp.src(['./**/*.tjs', '!./node_modules/**'])
    //.pipe(include())
    .pipe(tap(function (file) {
        console.log("BEFORE: " + file.path);
    }))
    .pipe(gdest(':dirname' + path.sep + ':basename.js'))
    .pipe(tap(function (file) {
        console.log("AFTER: " + file.path);
    }));

The output that comes back is:
BEFORE: C:\path\to\project\main.tjs
AFTER: C:\path\to\project\main.tjs.js

So, you can tell that ":basename" is wrongfully including the extension (which is at odds with the documentation as well).

I think the docs are wrong, try using :name. The variables should match values returned from parse-filepath. I'll get this updated to use the latest.


Add methods for doing schema checks on response-value command payload dictionaries.

This adds a way to initialize strongly typed command response structs with a response-value dictionary representing a command response.

Fast-tracking platform-specific change with platform owner review.


Workspace trust - add tracing to track down integration test failure

@lszomoru do we still need these changes? trying to cleanup the logs a bit for smoke tests runs. I forgot why you added them...

@bpasero, if we have not seen any further test hangs that we suspect is related to workspace trust, we can revert these changes.

@lszomoru I see no hangs lately, I also forgot what kind of hang we saw, was there an issue? I suggest to either remove the logs or change them to trace

@bpasero, this is the issue tracking the remote integration test failures that we were investigating. Will remove the logging.

@lszomoru yeah thanks!

@bpasero, I have reverted the commit that added the tracing.


--permissions flag missing from cli readme

This tripped me up when running hz serve with a docker rethinkdb
I couldn't figure out how to disable the permissions system until I found this page https://github.com/rethinkdb/horizon#readme and learned of the --permissions flag
That page also mentions that --dev will set the permissions flag.  This is also omitted from the readme.
This PR fixes both documentation issues.

This change is

Thanks! Sorry for the confusion


updating my profile imran1509.json

Fixes Issue

Changes proposed

Check List (Check all the applicable boxes)

[ ] My code follows the code style of this project.
[ ] My change requires changes to the documentation.
[ ] I have updated the documentation accordingly.
[ ] All new and existing tests passed.
[ ] This PR does not contain plagiarized content.
[ ] The title of my pull request is a short description of the requested changes.

Screenshots

Note to reviewers

Updating profile doesn't require changes in lock files. Please revert the changes from the CHANGELOG.md, package-lock.json, package.json files. Let us know if you need any help! 🛠

But I have not changed anything else other than my profile file. Other changes are showing automatically.
Please help me how to resolve this conflict.

@imran1509 your commit in this PR only has your changes which is correct 👍  but I think your branch might be out of date, which is why it is including another commit - try updating your fork and branch

@imran1509 your commit in this PR only has your changes which is correct 👍  but I think your branch might be out of date, which is why it is including another commit - try updating your fork and branch

Okay but when I try to update the branch it says need to delete the commits.
Abd when i deleted the commits and updated my profile again.
Same happened.

I think it is because you are making changes in the default branch - you should always make a branch to make changes and not use the default branch - it will create conflicts
I would put your fork's default branch back to match the original repo (upstream) and then make a branch and make your changes ,then create the PR from your branch - you will have less problems

Okay Thanks Eddie.
I Will delete the changes and will do it again in different branch.
I forgot to make a different branch this time. Sorry my bad.

No problem, it happens to us all


[GMT6] libgfortran warning (relink with librt)

Every time I start gmt 6 (installed from the dev channel) I get this warning:
gmt: Relink `/home/pl/miniconda3/envs/sismo/bin/../lib/./libgfortran.so.3' with `/lib/x86_64-linux-gnu/librt.so.1' for IFUNC symbol `clock_gettime'

I'm not sure about what the implications are. These are the libgfortran packages I have installed:
$ conda list gfortran
# packages in environment at /home/pl/miniconda3/envs/sismo:
#
libgfortran               3.0.0                         1
libgfortran-ng            7.2.0                h9f7466a_2

Perhaps GMT should be linked against libgfortran-ng?

Hi @paride, thanks for reporting. I'm not sure why GMT is linking against libgfortran in the first place. Are you having any problems with the plots?

I'm just plotting very simple things at the moment (not even geographical maps for the moment), but it seems to work fine. I guess from your answer that you are not getting the warning, is it right?

I'm not. But if it's not causing problems then I guess you can ignore it. I tried reproducing it by  installing the libgfortran that you report but got nothing.

Issue disappeared by upgrading.

@paride thanks for reporting