Added build time user to get the build running in OSBS

This PR makes sure the make command runs as root user.
By default the USER in OSBS is changed to USER 1001, which makes build to fail.

@otaviof any idea on why the CI is running for 3 days, here?

@adambkaplan @coreydaley could you please review and merge this?

CI looks good.

@bamachrn  Could you paste the failure logs from OSBS here if that's OK? I'm wondering why root is needed for building this image @otaviof ?

COPY . $GOPATH/src/github.com/openshift/source-to-image
RUN cd $GOPATH/src/github.com/openshift/source-to-image &&     make &&     install _output/local/bin/linux/${GOARCH}/s2i /usr/local/bin
hack/build-go.sh
mkdir: cannot create directory '/src/github.com/openshift/source-to-image/_output': Permission denied
make: *** [Makefile:24: all] Error 1
running 'cd $GOPATH/src/github.com/openshift/source-to-image &&     make &&     install _output/local/bin/linux/${GOARCH}/s2i /usr/local/bin' failed with exit code 2
ERROR - Build step plugin imagebuilder failed: image build failed (rc=1): running 'cd $GOPATH/src/github.com/openshift/source-to-image &&     make &&     install _output/local/bin/linux/${GOARCH}/s2i /usr/local/bin' failed with exit code 2

@bamachrn I would avoid the specification of root.
Could we have something like this ? @otaviof
FROM openshift/origin-release:golang-1.13 AS builder

ENV S2I_GIT_VERSION="" \
    S2I_GIT_MAJOR="" \
    S2I_GIT_MINOR=""

WORKDIR /tmp/source-to-image
COPY . .

ENV GOARCH="amd64"
RUN make

#
# Runner Image
#

FROM registry.access.redhat.com/ubi7/ubi-minimal

ENV GOARCH="amd64"

COPY --from=builder /tmp/source-to-image/_output/local/bin/linux/${GOARCH}/s2i  /usr/local/bin/s2i

USER 10001

ENTRYPOINT [ "/usr/local/bin/s2i" ]

Could we have something like this ?

I think we can, @sbose78! Would you please try to build it like that @bamachrn? Good suggestion to avoid root specified directly.

@adammhaile both the base images are changed in doozer build
openshift/origin-release:golang-1.13  ==> rhel8/go-toolset:1.13
registry.redhat.io/ubi8/ubi ==> ubi8:8-released

and
openshift/origin-release:golang-1.13
registry.redhat.io/ubi8/ubi

this images are not getting downloaded in brew build. I believe they are same image, but wanted to make sure we are running the builds on top of correct image. cc: @otaviof @sbose78

@sbose78 @otaviof
even if we are changing the build dir to /tmp in the above dockerfile. it is failing with same error.
hack/build-go.sh
mkdir: cannot create directory '/tmp/source-to-image/_output': Permission denied
make: *** [Makefile:24: all] Error 1
running 'make' failed with exit code 2
ERROR - Build step plugin imagebuilder failed: image build failed (rc=1): running 'make' failed with exit code 2

and in the brew build I see a pattern to user root user for running commands. (these are added by brew itself)
FROM sha256:c516d13c06ca29ef6953f8fad1a52914908640aa8a91552d42ade64cc6d65cb7 as builder
ADD atomic-reactor-repos/* '/etc/yum.repos.d/'
USER root
RUN mv -f /etc/yum.repos.d/ubi.repo /tmp || :
USER 1001

I understand, if that's the pattern let's go for it. However let's set the user ( as done in my suggestion ) for the runtime image in the second stage.

@otaviof @sbose78 what should be the base images in the Dockerfile? as brew is not able to pull both the images.

FROM openshift/origin-release:golang-1.13 AS builder

ENV SOURCE_GIT_COMMIT=c82d9ebcdedf29c17c4ff1bc4eb47fc0b7898d9a SOURCE_DATE_EPOCH=1576077064 BUILD_VERSION=v1.3.0 SOURCE_GIT_URL=https://github.com/openshift/source-to-image SOURCE_GIT_TAG=v1.2.0-25-gc82d9ebc BUILD_RELEASE=201912171309

ENV S2I_GIT_VERSION="" \
    S2I_GIT_MAJOR="" \
    S2I_GIT_MINOR=""

WORKDIR /tmp/source-to-image
COPY . .

ENV GOARCH="amd64"

USER root

RUN make

#
# Runner Image
#

FROM registry.redhat.io/ubi8/ubi

ENV SOURCE_GIT_COMMIT=c82d9ebcdedf29c17c4ff1bc4eb47fc0b7898d9a SOURCE_DATE_EPOCH=1576077064 BUILD_VERSION=v1.3.0 SOURCE_GIT_URL=https://github.com/openshift/source-to-image SOURCE_GIT_TAG=v1.2.0-25-gc82d9ebc BUILD_RELEASE=201912171309

ENV GOARCH="amd64"

COPY --from=builder /tmp/source-to-image/_output/local/bin/linux/${GOARCH}/s2i  /usr/local/bin/s2i

USER 1001

ENTRYPOINT [ "/usr/local/bin/s2i" ]

LABEL \
        io.k8s.description="source to image" \
        com.redhat.component="ocp-tools-1-tech-preview-source-to-image-rhel8-container" \
        name="ocp-tools-1-tech-preview/source-to-image-rhel8" \
        io.k8s.display-name="source to image" \
        io.openshift.build.source-location="https://github.com/openshift/source-to-image" \
        io.openshift.build.commit.url="https://github.com/openshift/source-to-image/commit/c82d9ebcdedf29c17c4ff1bc4eb47fc0b7898d9a" \
        version="v1.3.0" \
        io.openshift.build.commit.id="c82d9ebcdedf29c17c4ff1bc4eb47fc0b7898d9a" \
        release="201912171309"

Are we good to go with:
rhel8/go-toolset:1.13
ubi8:8-released

querying /v2/rh-osbs/openshift-origin-release/manifests/golang-1.13, headers: {u'Accept': 'application/vnd.docker.distribution.manifest.list.v2+json'}
 [401] https://registry-proxy.engineering.redhat.com/v2/rh-osbs/openshift-origin-release/manifests/golang-1.13
[404] https://registry-proxy.engineering.redhat.com/v2/rh-osbs/openshift-origin-release/manifests/golang-1.13
skipping version v2_list due to status code 404
Cannot resolve manifest digest for image 'registry-proxy.engineering.redhat.com/rh-osbs/openshift-origin-release:golang-1.13

ERROR - plugin 'pull_base_image' raised an exception: RuntimeError: Registry specified in dockerfile image doesn't match configured one. Dockerfile: 'registry.redhat.io/ubi8/ubi:latest';

FROM rhel8/go-toolset:1.13 AS builder

ENV SOURCE_GIT_COMMIT=c82d9ebcdedf29c17c4ff1bc4eb47fc0b7898d9a SOURCE_DATE_EPOCH=1576077064 BUILD_VERSION=v1.3.0 SOURCE_GIT_URL=https://github.com/openshift/source-to-image SOURCE_GIT_TAG=v1.2.0-25-gc82d9ebc BUILD_RELEASE=201912171309

ENV S2I_GIT_VERSION="" \
    S2I_GIT_MAJOR="" \
    S2I_GIT_MINOR=""

WORKDIR /tmp/source-to-image
COPY . .

ENV GOARCH="amd64"

USER root

RUN make

#
# Runner Image
#

FROM ubi8:8-released

ENV SOURCE_GIT_COMMIT=c82d9ebcdedf29c17c4ff1bc4eb47fc0b7898d9a SOURCE_DATE_EPOCH=1576077064 BUILD_VERSION=v1.3.0 SOURCE_GIT_URL=https://github.com/openshift/source-to-image SOURCE_GIT_TAG=v1.2.0-25-gc82d9ebc BUILD_RELEASE=201912171309

ENV GOARCH="amd64"

COPY --from=builder /tmp/source-to-image/_output/local/bin/linux/${GOARCH}/s2i  /usr/local/bin/s2i

USER 1001

ENTRYPOINT [ "/usr/local/bin/s2i" ]

LABEL \
        io.k8s.description="source to image" \
        com.redhat.component="ocp-tools-1-tech-preview-source-to-image-rhel8-container" \
        name="ocp-tools-1-tech-preview/source-to-image-rhel8" \
        io.k8s.display-name="source to image" \
        io.openshift.build.source-location="https://github.com/openshift/source-to-image" \
        io.openshift.build.commit.url="https://github.com/openshift/source-to-image/commit/c82d9ebcdedf29c17c4ff1bc4eb47fc0b7898d9a" \
        version="v1.3.0" \
        io.openshift.build.commit.id="c82d9ebcdedf29c17c4ff1bc4eb47fc0b7898d9a" \
        release="201912171309"

this builds successfully @otaviof @sbose78

@adammhaile both the base images are changed in doozer build
openshift/origin-release:golang-1.13  ==> rhel8/go-toolset:1.13
registry.redhat.io/ubi8/ubi ==> ubi8:8-released

and
openshift/origin-release:golang-1.13
registry.redhat.io/ubi8/ubi

this images are not getting downloaded in brew build. I believe they are same image, but wanted to make sure we are running the builds on top of correct image. cc: @otaviof @sbose78

@bamachrn Yes, go-toolset and ubi8 are the standard images for building internally. This change is intentional. Doozer does not build with anything from origin, only downstream openshift approved images.

I have updated the Dockerfile as per the discussion. please review and merge @otaviof @sbose78 @adammhaile

Looks good to me!

@bamachrn Nack on changing the base images in Dockerfile.
The general convention is to have Dockerfile use base images that have low barriers to consumption. Ex - images from docker.io, registry.redhat.io, or registry.svc.ci.openshift.org. Things that are tailored to our build systems should have a separate Dockerfile - (i.e. Dockerfile.rhel8).
@bamachrn @adammhaile I don't fully understand why the base images need to change - doesn't Doozer do the image substitution for us?

Ok, apparently I misread something before. Yes, doozer makes the change. I was asked why it was changed in distgit (I thought) and that's because doozer changes it. It does not need to change in the source dockerfile in github.

OK, my thought process is to keep it consistent. If we know that it will get substituted, why not use same to run the through out CI and build process. This will avoid build time surprises and will make sure we are running tests on the image, which is getting released.
I got the point of using images from low barrier registries, from the upstream point of view. And agree with it, we should use non productised images for upstream builds. But, we should make sure, we are testing the image after substitution. And we should always keep this in mind that the image gets replaced in build time environment.
@adammhaile @adambkaplan @otaviof @sbose78 thoughts on this?
I am ok with using the origin images in upstream, just wanted to make sure we are making a conscious decision here.

But, we should make sure, we are testing the image after substitution.

yes, agreed. This is something the devtools architects are discussing with QE since this affects productization of all projects.

I got the point of using images from low barrier registries, from the upstream point of view. And agree with it, we should use non productised images for upstream builds.

Let's keep the upstream dockerfile unchanged, also since that's the precedent.

done!


Wrong zoomed image position from fixed container

I see an issue in page:
https://codepen.io/anon/pen/eWyEGR?editors=1010
OS: Ubuntu 16.04 (64-bit)
Browser and version: Chromium Version 58.0.3029.81
Steps to reproduce:

open codepen example
scroll one screen down
zoom image
see that it has wrong position

Use-case: fixed chat container with images in messages that can be zoomed when page is scrolled down.
Mb fixed position for zoomed image?

I probably will end up adding a fixed wrapper to image, which I believe will address this issue and #22 as well...Having position: relative on image is giving me so much headache :(

Update: a fixed position wrapper with absolute image position and an image placeholder (since the actual image no longer takes space) does resolve these issues (position: fixed and overflow: hidden parents). Everything works fine except that while the image is getting closed on scrolling, the translation does not go to the precise position that takes scroll offset into account (though eventually it goes back to the correct position...). I guess this could be fixed somehow, but I haven't found a solution yet.

Opened an issue to keep track of adding image wrapper #44.

I don't intend to find a fix at this point. I'll accept it as a known issue and mention it in the README.


Unexpected end of file sh: error importing function definition after update to v1.12.1

Error: sh: module: line 1: syntax error: unexpected end of file
sh: error importing function definition for `BASH_FUNC_module'
sh: scl: line 1: syntax error: unexpected end of file
sh: error importing function definition for `BASH_FUNC_scl'
    at parameters.exit (/home/msfernandes/.atom/packages/linter-flake8/node_modules/atom-linter/lib/helpers.js:47:27)
    at triggerExitCallback (/usr/share/atom/resources/app.asar/src/buffered-process.js:213:47)
    at /usr/share/atom/resources/app.asar/src/buffered-process.js:227:18
    at Socket.<anonymous> (/usr/share/atom/resources/app.asar/src/buffered-process.js:98:18)
    at emitOne (events.js:82:20)
    at Socket.emit (events.js:169:7)
    at Pipe._onclose (net.js:469:12)

@steelbrain is this possibly a problem with consistent-env pulling in the bash profile?

This is a user misconfiguration, the user has syntax errors in their rc files

Closing due to the above comment.

@steelbrain Could you be more precise please? I have the same error as reported and did not touch the rc files manually.
Which rc file are you talking about, and how can I verify that?

@cnicodeme Sure, I was referring to the shell rc files, for example if you use bash it would be ~/.bashrc or ~/.zshrc for zsh. To repro the error you can source ~/path/to/rc/for/your/shell and see if you get any errors.

@steelbrain Thank you for your reply. I'm using Bash so I tried source ~/.bashrc and it did not yield anything. I suspect the bash rc file is correct then.
Do you want me to past the detailled log error from Atom?

Yes please, detailed error logs would be helpful

@steelbrain Here it is :
Error: sh: module: ligne 1: erreur de syntaxe : fin de fichier prématurée sh: erreur lors de l'import de la définition de fonction pour « BASH_FUNC_module » sh: scl: ligne 1: erreur de syntaxe : fin de fichier prématurée sh: erreur lors de l'import de la définition de fonction pour « BASH_FUNC_scl »
Error: sh: module: ligne 1: erreur de syntaxe : fin de fichier prématurée
sh: erreur lors de l'import de la définition de fonction pour « BASH_FUNC_module »
sh: scl: ligne 1: erreur de syntaxe : fin de fichier prématurée
sh: erreur lors de l'import de la définition de fonction pour « BASH_FUNC_scl »
    at ChildProcess.<anonymous> (/home/cnicodeme/.atom/packages/linter-flake8/node_modules/sb-exec/lib/index.js:56:20)
    at emitTwo (events.js:106:13)
    at ChildProcess.emit (events.js:191:7)
    at maybeClose (internal/child_process.js:852:16)
    at Socket.<anonymous> (internal/child_process.js:323:11)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at Pipe._handle.close [as _onclose] (net.js:492:12)

I'm sorry the output is in French, I hope it won't stop you. If there is something else you need, feel free to ask :)

@cnicodeme while the error is still irrelevant to this package and is a user misconfiguration issue, a quick google search lead to me this solution comment.

Thank you @steelbrain. The issue was not solved via your linked solution, but the link right after (https://github.com/AtomLinter/linter-puppet-lint/issues/68).
I did a search before ending here (my search was the reason I come up here ;)), and I decided to ask because I didn't find the right solution for me. In retrospect, I may have searched with the wrong keyword ;)
For all others that have the same issue here, and are using Fedora, the solution is simple :

Remove scl-utils : sudo dnf remove scl-utils. Don't worry, it's not useful on Fedora (it's more RHEL related)
Reboot, and you won't have the problem again. This is important, if you don't reboot, nothing will change. (Don't ask me why!)

Thank you for your help @steelbrain  :)

I'm on Fedora but don't have scl-utils installed. I keep seeing this error message all the time.

I got this message two, but only on one version Fedora 24 and not the other. I removed scl-utils but it had no effect.
But I realized I had flake8 installed in a user-specific bin directory (linking to a virtualenv), and the machine where it worked was using a newer version., and the failing machine had an older version (3.0.x). Once I upgraded to 3.2.1 the problem went away on the machine where it had previously failed.

I can confirm that upgrading flake8 works here. Thanks @faassen !

is this possibly a problem with consistent-env pulling in the bash profile?

consistent-env is causing this problem. It fails if an environment variable contains line breaks.
The original environment contains

BASH_FUNC_scl()() {  local CMD=$1;
 if [ "$CMD" = "load" -o "$CMD" = "unload" ]; then
 eval "module $@";
 else
 /usr/bin/scl "$@";
 fi
}

but consistent-env transform it to

BASH_FUNC_scl()() {  local CMD=$1;

if [ "$CMD""load" -o "$CMD" = "unload" ]; then

Excellent find @Unrud
Could you please post this info on consistent-env repo?

Just as @cnicodeme suggested, I am on Fedora 26 and did:
sudo dnf remove scl-utils
Rebooted and errors gone.
I had not touched my .bashrc file...brand new install of Fedora!

Hey guys i'm having same issue but with drupal any clue what could it be?
I'm doing a drush cr to clear cache and drop me this errors.
sh: module: line 1: syntax error: unexpected end of file
sh: error importing function definition for `BASH_FUNC_module'

@cnicodeme - your solution works for me on Fedora 28. Thanks :-)

Glad I could help ;)


It will hang if I open the second 'Code' project in the second window

VSCode Version: from 1.28.1, also crashed in 1.28.2
OS Version: MS Windows 7 64-bit with SP1 (built 6.1.7601)

Steps to Reproduce:

open one code window (= instance), open one project.
open another code window (my way is to right-click code icon on the bottom bar). It could hang
try to open another project, it will hang

Does this issue occur when all extensions are disabled?: Yes.
No matter all extensions are disabled or not.

It was ok in version 1.26 and before. It can support 2 or more Code window and open quickly. It was useful while studying or check another project.
By the way, I feel the new version (1.28.x) become a little slow.

Can you try to reproduce with our nightly insider builds? You can give our preview releases a try from: https://code.visualstudio.com/insiders/


Fix scaffolder catalog link

This was incorrectly linking to a relative path which ended up being something like /create/docs-template/Component/my-component.

Thanks!


docs: change link for CLI readme to be relative

//cc @gkamperis
Caretaker note: No need to release for this. This should be back-ported up till 6.2.x

@alan-agius4 again... don't know if I am replying to early in the process but
https://github.com/angular/angular-cli/tree/6.2.x
still shows the wrong package versions.


Make Astro Optional

Closes: #153
Put Astro behind an option.
Tests

It works in my vanilla-TS Bun monorepo; is there a good Astro project I can test it on?

Originally posted by @lishaduck in https://github.com/AndreaPontrandolfo/sheriff/issues/153#issuecomment-2218249341

Oh, and if this does work, then I'll need to update the docs.

Yup, can you update the docs?

Oops.


Develop API for merged emergency

More info needed.

Added 2 API paths :
Try : /view/:country for example /view/India : shows specified country details.
Try : /allhelp : shows entire table of helpcodes.


multisite configuration

I have the setup working with the primary site but it seems like the dev server isn't loading scripts/styles on sub sites in my multisite set up.
how can I set it up so it loads my multisites as well?

Hey @yangkennyk
I'm having the exact same problem. I believe the issue should be solved on the dev server code (probably pass a regex distPublicPath instead of a string).
In the meantime, I've come up with a workaround. Hope this helps you too:

Set the distPublicPath back to undefined
Make a class to extend WPackio\Enqueue and instance that class instead
Add the following method to your class:

`public function getUrl( $asset ) {
$url =  parent::getUrl($asset);
if(is_multisite()){
    $currentUrl = get_site_url();
    $baseUrl = get_site_url(BLOG_ID_CURRENT_SITE);

    if($currentUrl !== $baseUrl && strpos($url, $currentUrl) === 0){
        $url = $baseUrl . substr($url, strlen($currentUrl));
    }
}

return $url;

}`

@TullariS great workaround. Does it work on both development and production? I have totally forgotten how WordPress plugins_url works for multisite. Would love to have you digging around it a bit 😄

Ah that's a good point.
The default rewrite script does allow for plugin/theme assets from being server both on a base url (http://site.com/wp-content/...) or one of the sites of the network (http://site.com/wp-content/blog/...). As long as this isn't changed, it should work on both development and production.
That being said, this solution is not ideal, but it does help out during development. Maybe after you are done, comment it out. Hopefully once this issue is addressed (within the development server script) we can get rid of it altogether.


Added the  new CNCF slack channel to README.md

🛑 Important:
This PR is for the change proposed in https://github.com/kubeflow/spark-operator/issues/2061
Purpose of this PR
The PR aims at providing the new CNCF slack Channel details so that new users can go to new slack channel instead of old slack channel which is going to be archived soon.  See details here (https://github.com/kubeflow/spark-operator/issues/2061)
Proposed changes:

Update slack channel to CNCF Slack channel

Change Category
Indicate the type of change by marking the applicable boxes:

[ ] Bugfix (non-breaking change which fixes an issue)
[ ] Feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that could affect existing functionality)
[x] Documentation update

Rationale
Users are currentl joining the old slack channel instead of the new one
Checklist
Before submitting your PR, please review the following:

[x] I have conducted a self-review of my own code.
[x] I have updated documentation accordingly.
[ ] I have added tests that prove my changes are effective or that my feature works.
[x] Existing unit tests pass locally with my changes.

Thanks @vikas-saxena02!
Please can you open another PR to update the Slack channel in kubeflow/blog here: https://github.com/kubeflow/blog/blob/master/_posts/2024-04-15-kubeflow-spark-operator.md#join-the-movement.

/hold

https://github.com/kubeflow/blog/blob/master/_posts/2024-04-15-kubeflow-spark-operator.md#join-the-movement

@andreyvelich please take a look at the change made here and I will make an identical change in kubeflow/blog repository as mentioned.

@vara-bonthu I am getting the following error ✖ spark-operator => (version: "1.4.1", path: "charts/spark-operator-chart") > chart version not ok. Needs a version bump! from Pre-commit check/build-helm-chart doI need to bump the operator version?

@vara-bonthu @andreyvelich I think I have screwed up my branch. i will do a new PR soon


Add check for UTF8 String length during creation

and remove MqttBinaryData.isInRange(binary) check which throws an exception from encodedLength

What should be returned during creation in case of an error? Just null?

Currently we return null for all validation errors in the MqttUTF8StringImpl and MqttUTF8String checks for null and throws an exception. This was done that way to not throw exceptions internally, but only when the user calls the methods from the api package.
We could throw exceptions, but at least without a stacktrace. If so, we should change this for all validation errors.

For checking the encoded size Utf8.encodedLength can be used.
Checking the encoded size can be very expensive, so we should perform a simpler check before: Only if string.length() * 2 > MqttBinaryData.MAX_LENGTH is true, we need to perform the check.

Currently the user will get an IllegalArgumentException stating "The string is not a valid UTF-8 encoded String." regardless what went wrong. The user should at least get a meaningful error message, i.e. string to long or string contains invalid characters. So I would vote to have at least these two exceptions on the API level.

What about having a flag whether MqttUtf8StringImpl#from should throw on error or not? When used internally it could return null. When used through the API it throws.

I think it is ok to allways throw exceptions for the MqttUtf8StringImpl.from(String)
but not for the MqttUtf8StringImpl.from(byte[]/ByteBuffer) methods

OK, I will trow exceptions.
One other point, I do not understand your assumption:

Only if string.length() * 2 > MqttBinaryData.MAX_LENGTH is true, we need to perform the check.

Since UTF-8 uses 1 to 4 bytes for each code point, e.g. all ASCII (0-127) code points will only need one byte in UTF-8.

But what if we have characters that need 2 bytes in UTF-16 but 3 bytes in UTF-8?
According to https://en.wikipedia.org/wiki/Comparison_of_Unicode_encodings code points from 000800-003FFF and 004000-00FFFF need 2 bytes in UTF-16 but 3 bytes in UTF-8.
Lets make an example:

assume MqttBinaryData.MAX_LENGTH = 10
assume we have a string consisting of 4 chars as described above
then string.length = 4 and 4 * 2 = 8 which is not bigger than 10, hence no exception
but the number of bytes of the string in UTF-8 would be 12 which is bigger than 10, so we should have thrown an exception

So we could implement this fast check only with using a factor 3 instead of 2.
BTW, if I change the API to always throw, we need to adapt 357 tests that will fail because of the change. Should we anyway go with this approach? The solution described above (flag to enable exceptions) would not change the contract of the method for existing code that uses the method.

You are right, the check has to multiply with 3.
Why would the tests be broken? The exceptions should be RuntimeExceptions and therefore the API does not change and so the tests do not need to be changed.

Sorry, my fault, wrong boolean expression (one NOT to little).


Tabbar Navigation always goes back to the last screen

I'm version 8 beta
I have a tabbar
Tab 1

Home
Teams
Tab 2
Photos

If I'm on the teams view and move onto the photos view and then click on the tab 1 I'm taken to the teams view the last view I was on, but I should be taken to the home view which is the first view for tab 1.

It seem ReactNavigation behaviour, but maybe I didn't understand issue.  I want to repeat, screenshots only are not helpful for me, I need runnable project also.

Possible it's a ReactNavigation behavior but I didn't have this issue when I using React Navigation.
Let me explain
I have the home tab which has to two internal pages home and teams if I navigate to teams and then to the photos tabs and then click the home tab at the bottom the app goes or loads the teams component i.e. the last I was on it should load the home view.

It is about 'backBehavior' for TabNavigator. It is set to 'none' by RNRF:
https://reactnavigation.org/docs/navigators/tab
Maybe I will remove 'none' and use default value instead ('initialRoute') for beta.9

I would suggest using initialRoute I believe that is more of the expected behavior

I've removed 'none' as back behaviour but don't see any changes. Maybe you could check ReactNavigation itself for desired behaviour.


MB-8373 add parsing functionality to admin app

Description
This PR adds the correct validation, error handling and parsing functionality to the import button on the office users list page.
Reviewer Notes
I am still working on creating a cypress test for this functionality. It is a little tricky since we have to do a file upload. Additionally, I wasn't able to set up unit tests for the importcsvbutton because it has to be rendered in a react-admin app or it crashes. So we will only be able to do integration tests on it.
Setup
To test that the parsing functionality works complete the following steps:

start the admin app
navigate to the office users list page
select the Import button
upload the cypress/fixtures/officeUsers.csv file
You should some new users on the list page now: Riley and Jason.

To view how errors are handled, attempt to upload the same file again, and you should get a little pop-up showing an HTTP Error.
Code Review Verification Steps

[ ] If the change is risky, it has been tested in experimental before merging.
[ ] Code follows the guidelines for Logging
[ ] The requirements listed in Querying the Database Safely have been satisfied.
Any new migrations/schema changes:

[ ] Follow our guidelines for zero-downtime deploys (see Zero-Downtime Deploys)
[ ] Have been communicated to #g-database
[ ] Secure migrations have been tested following the instructions in our docs

[ ] There are no aXe warnings for UI.
[ ] This works in Supported Browsers and their phone views (Chrome, Firefox, IE, Edge).
[ ] Tested in the Experimental environment (for changes to containers, app startup, or connection to data stores)
[ ] User facing changes have been reviewed by design.
[ ] Request review from a member of a different team.
[ ] Have the Jira acceptance criteria been met for this change?

References

Jira story for this change

Warnings

:warning:
This PR does not include changes to storybook, even though it affects component code.

:warning:
This PR does not include changes to unit tests, even though it affects app code.

:warning:

Files located in legacy directories (src/shared or src/scenes) have
been edited. Are you sure you don’t want to also relocate them to the new file structure?
View the frontend file org ADR for more information

Messages

:book:
:link: MB-8373

  Generated by :no_entry_sign: dangerJS against 75ec6b95347e94307eed1a71f819884d368e244f

Oh also, I'm not able to get the integration tests to pass - I keep getting a timeout. Might need to add a longer wait there. It also might mean we'll be introducing a new flaky test, so it would be good to get sign-off from the frontend folks that this is okay (I see you already tagged some peeps 👍 ).

Everything looks good to me here except for the misbehaving integration test. Let me know when that's all resolved and I'm good to approve! 👍


XinaA15 & Dopamine should not be listed as jailbreaks.

They are rootless tweaks that leave the file system read-only, and a marker or some sort of delineation should be noted on the website for such. The definition of jailbreaking is to make a read-only file system writable by the user. Neither of these qualify.

Do you want bootloop day 1? Do you want all apps to easily detect jailbreak? That’s the whole point of rootless.

Preventing system file modification makes it a lot safer to use, what if you accidentally delete /sbin/launchd or /usr/lib/dyld? Boom. Restore to latest.
What if you wanna use bank apps? Rootless (and also another variant, RootHide) makes it a lot more difficult to detect jailbreak.


feat(multiple-components): adding last round of missing shadow parts

Related Ticket(s)
ADCMS-6148
Description
I have added the last round of shadow parts that we've missed through our shadow part tickets.

This one is a tricky one as well - the CTA mixin renders a default SVG element in the shadowroot that we should put a part attribute on.
https://github.com/carbon-design-system/carbon-for-ibm-dotcom/blob/main/packages/web-components/src/component-mixins/cta/cta.ts#L195-L205


add class to html tag on paper index like paper edit

You can now see contributors and download options when the paper is not editable

Adding tests for this.

Closing - merged https://github.com/Tahi-project/tahi/pull/793, which added a QUnit spec.


Move readline history file to more sensible spot

Right now delve creates readline history file in cwd, but this is not acceptable going forward.

Hi, I think this can be closed as it has been fixed in #124, no?


postgresql cannot find plugins

On latest unstable as of this posting (nixos-16.09pre84674.fbde3a7) postgresql seems to have $libdir set incorrectly, making it impossible to load plugins such as PostGIS.
root=# CREATE EXTENSION postgis;
ERROR:  could not access file "$libdir/postgis-2.2": No such file or directory

> pg_config --libdir
/nix/store/w619ay8m1h84ym3kpqhl7mwfar6py2vc-postgresql-9.4.6-lib/lib

Compare 16.03:
> pg_config --libdir
/run/current-system/sw/lib

Related: https://github.com/NixOS/nixpkgs/issues/15512

should this be marked as fixed?

Fixed in https://github.com/NixOS/nixpkgs/pull/17838.


2104 qa program

we can probably cut the create_qa_programs_old.py at this point, right?  Looks super solid, love the tests, and it's really easy to follow (and easier to make additions to) - 👍

I thought I might leave the old version in there for just a couple of cycles to make sure we have an alternative if something goes terribly wrong with the new one. I'll delete it after we've had a chance to use the new one a couple of times.


need help

hello brother
I just want to know can sever file using url from r2 storage if yes then can please help me deploying worker on r2 storage to download file present in r2 storage

fixed myself


Use arrow keys to go to next ang in SGGS

Hi,
Is it currently possible to switch between Angs in SGGS using the arrow keys?
Maybe, it could be a feature that can be added

Thank you so much for raising this issue, feel free to contribute for the same!
Ang state is stored here https://github.com/bogas04/SikhJS/blob/master/src/pages/SGGS/index.js#L24-L31
You can addEventListener in componentDidMount for keyup to set call handleIncrementAng or handleDecrementAng based on arrow key pressed. You can get keycodes for arrow keys here.
Don't forget to removeEventListener in componentWillUnmount lifecycle method of above component to be conservative of memory leaks.

Thanks! I'll work on the same and add in a PR soon.
Also, joined the slack channel 👍


remove blank from lookup.

Hi,
How can I prevent a filter (quickfilter) from being blank?
I want to remove blank from lookup.
Thanks

you can create a custom lookup, ex;
[LookupScript("THIS_IS_YOUR_LOOKUP_KEY", Expiration = -1, Permission = "*")]
public class DemoLookup : Serenity.Web.LookupScript
{
    public DemoLookup()
    {
        this.IdField = "Something";
        this.TextField = "Something to display";
        this.getItems = this.GetItems;
    }

    protected virtual IEnumerable<dynamic> GetItems()
    {
        // return your result here, remove empty value
    }
}

then use it on your row
[lookupeditor(typeof(DemoLookup))]
public Int64 AbcField{

}

But the lookup is based on a database table
I want to force the user to enter a value in the quickfilter field and not allow empty values.
Thx

try this:
       [QuickFilter, QuickFilterOption("allowClear", false)]
HTH
Hannes

works like a charm
thx


gateway-api v1.1.0 standard-install breaks envoy-gateway and cilium-operator

What happened:
Recently I upgraded the gateway-api to v1.1.0 from the standard channel, but I found that both envoy-gateway and cilium-operator are in error, even though I have not directly used GRPCRoute gateway.networking.k8s.io/v1alpha2. (istio controller is fine)
From the v1.1.0 release notes:

GRPCRoute has graduated to GA (v1) and is now part of the Standard Channel. If
you are already using the experimental version GRPCRoute, we recommend holding
off on upgrading to the standard channel version of GRPCRoute until the
controllers you're using have been updated to support GRPCRoute v1. Until then,
it is safe to upgrade to the experimental channel version of GRPCRoute in v1.1
that includes both v1alpha2 and v1 API versions.

It seems to me that the experimental channel is more safe to upgrade, which is unintuitive.
I have two questions regarding this issue:

Is it possible to keep old api versions in the standard channel and deprecate them only after some period and future releases?
Does this issue supposed to be fixed by the envoy-gateway and cilium-operator implementations? They both requires some experimental gateway api crds for the controller to start, even when users don't use them at all. And upgrade the gateway-api using the stable channel accidentally breaks them.

Edited:
I wrongly made the assumption that the v1alpha2 GRPCRoute was installed from the gateway api standard channel, but in reality it was installed by envoy-gateway helm charts which embed the experimental grpcroutes crd. (The cilium-operator also requires the experimental gateway api crds, but istio only requires stable gateway crds so when I upgrade the gateway api it's not broken.)
envoy-gateway log:
Error: failed to create provider Kubernetes: failted to create gatewayapi controller: no matches for kind "GRPCRoute" in version "gateway.networking.k
8s.io/v1alpha2"
Usage:
  envoy-gateway server [flags]

Aliases:
  server, serve

Flags:
  -c, --config-path string   The path to the configuration file.
  -h, --help                 help for server

failed to create provider Kubernetes: failted to create gatewayapi controller: no matches for kind "GRPCRoute" in version "gateway.networking.k8s.io/v
1alpha2"

cilium-operator log:
time="2024-05-11T05:43:22Z" level=info msg="Checking for required GatewayAPI resources" requiredGVK="[gateway.networking.k8s.io/v1, Kind=gatewayclasse
s gateway.networking.k8s.io/v1, Kind=gateways gateway.networking.k8s.io/v1, Kind=httproutes gateway.networking.k8s.io/v1beta1, Kind=referencegrants ga
teway.networking.k8s.io/v1alpha2, Kind=grpcroutes gateway.networking.k8s.io/v1alpha2, Kind=tlsroutes]" subsys=gateway-api
time="2024-05-11T05:43:22Z" level=error msg="Invoke failed" ="gateway-api.initGatewayAPIController (pkg/gateway-api/cell.go:70)" error="failed to crea
te gateway controller: failed to setup reconciler: no matches for kind \"GRPCRoute\" in version \"gateway.networking.k8s.io/v1alpha2\"" subsys=hive
time="2024-05-11T05:43:22Z" level=info msg=Stopping subsys=hive
time="2024-05-11T05:43:22Z" level=fatal msg="failed to start: failed to create gateway controller: failed to setup reconciler: no matches for kind \"G
RPCRoute\" in version \"gateway.networking.k8s.io/v1alpha2\"" subsys=cilium-operator-generic

What you expected to happen:
standard-install is more safe to upgrade.
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:

This issue has been discussed before. Maintainers believe that there is no scenario to upgrade GRPCRoute from v1alpha2 to v1 through standard channels.
Therefore, only the latest v1 version is retained in the standard channel.

It is difficult to ensure upgrades through various implementations.
After upgrading their dependency on GWAPI to v1.1, it is difficult for them to maintain support for v1alpha2.

Yes, each implementation will need to update to support v1.1, which for Cilium will require supporting both v1 and v1alpha2 of GRPCRoute. Until then, Cilium only supports v1.0 of Gateway API.

Does this issue supposed to be fixed by the envoy-gateway and cilium-operator implementations? They both requires some experimental gateway api crds for the controller to start, even when users don't use them at all. And upgrade the gateway-api using the stable channel accidentally breaks them.

I mean, users might not use any v1alpha2 GRPCRoute at all and so there is no need for CR upgrade. But the current situation is that both envoy-gateway and cilium-operator can't even start due to the hard requirement of v1alpha2 GRPCRoute crd.
Maybe it's a good idea for envoy-gateway, cilium-operator and other implementations to ignore missing experimental alpha/beta crds and finish starting.
P.S. I edited my original comment to correct my wrongly made assumption.

Created https://github.com/kubernetes-sigs/gateway-api/pull/3084 to help document this pain point when upgrading to v1.1.

I opened cilium/cilium#32539 to cover building out support for Cilium that's similar to what @howardjohn described for Istio.

Not stale.

/remove-lifecycle stale

The required Cilium changes (in cilium/cilium#34212) have been merged into main and will be in Cilium 1.17.
We probably also need some updates to the Gateway API CRD managment guide at https://gateway-api.sigs.k8s.io/guides/crd-management/ and the Implementer's guide at https://gateway-api.sigs.k8s.io/guides/implementers/ to recommend that implementations ensure that Experimental CRDs are optional for startup. Suggestions or PRs welcomed on those fronts.


runtime: scheduler is slow when goroutines are frequently woken

What version of Go are you using (go version)?
go1.7.3
What operating system and processor architecture are you using (go env)?
linux/amd64; Xeon E5-2670 (dual-socket 6-core packages, non-HT)
Our profiles indicate that we're spending an enormous number of cycles in runtime.findrunnable (and its callees) on the hosts that serve as our protocol heads.
Briefly, our these hosts translate HTTP CRUD operations into sets of transactions to be performed on our storage hosts, so the only real I/O these hosts do is networking.
Here's what I see in our cpu profiles when I run a benchmark with 40 clients against a single host backed by 60 storage controllers:
host 486938695e10692ab3a6a554cf47486b: 7356 samples
 top flat  pct symbol
1831 2030 24.9 syscall.Syscall
 900  900 12.2 i/c/siphash.blocks
 835  835 11.4 runtime.futex
 661  661  9.0 runtime.epollwait
 224  224  3.0 runtime.memmove
 180  297  2.4 runtime.runqgrab
 176 2584  2.4 runtime.findrunnable
 171  171  2.3 runtime/internal/atomic.Cas
 116  116  1.6 runtime/internal/atomic.Xchg
  85   85  1.2 runtime/internal/atomic.Load
-------------------------------------------------------------------------------------
host 486938695e10692ab3a6a554cf47486b
"runtime.findrunnable" -- in 2584 samples of 7356 (35.1%)
1 callers:
  in  flat symbol
2584 2694.0 runtime.schedule
21 callees:
 out  flat symbol
  67 130.0 runtime.unlock
  20 46.0 runtime/internal/atomic.Xadd
  14 85.0 runtime/internal/atomic.Load
 406 406.0 runtime.injectglist
 488 488.0 runtime.stopm
 331 331.0 runtime.runqsteal
 139 238.0 runtime.lock
  16 31.0 runtime/internal/atomic.Xchg64
  26 26.0 runtime.pidleput
   2  2.0 runtime.releasep
  59 66.0 runtime.runqempty
  21 161.0 runtime.casgstatus
 777 777.0 runtime.netpoll
   9  9.0 runtime/internal/atomic.Store64
   8  8.0 runtime.netpollinited
   2  8.0 runtime.acquirep
  10 15.0 runtime.pidleget
   8  8.0 runtime.globrunqget
   2 12.0 runtime.fastrand1
   2  2.0 runtime.nanotime
   1 10.0 runtime.runqget

... here's the same benchmark, but this time against two hosts backed by (the same) 60 storage controllers:
host 91b42bdeee8bc69fe40c33dff7c146ac: 6563 samples
 top flat  pct symbol
1695 1829 25.8 syscall.Syscall
 977  977 14.9 i/c/siphash.blocks
 639  639  9.7 runtime.futex
 431  431  6.6 runtime.memmove
 373  373  5.7 runtime.epollwait
 155  221  2.4 runtime.runqgrab
 112 1756  1.7 runtime.findrunnable
 100  100  1.5 runtime/internal/atomic.Cas
  89   89  1.4 runtime/internal/atomic.Xchg
  83   83  1.3 runtime.usleep
--------------------------
host f8e02f9facaa304dce98c8d876270a10: 6540 samples
 top flat  pct symbol
1593 1716 24.4 syscall.Syscall
 895  895 13.7 i/c/siphash.blocks
 598  598  9.1 runtime.futex
 399  399  6.1 runtime.memmove
 385  385  5.9 runtime.epollwait
 130  130  2.0 runtime/internal/atomic.Cas
 128  233  2.0 runtime.runqgrab
 104 1763  1.6 runtime.findrunnable
 102  102  1.6 runtime.usleep
 101  101  1.5 runtime/internal/atomic.Xchg

host 91b42bdeee8bc69fe40c33dff7c146ac
"runtime.findrunnable" -- in 1756 samples of 6563 (26.8%)
1 callers:
  in  flat symbol
1756 1846.0 runtime.schedule
20 callees:
 out  flat symbol
  41 98.0 runtime.unlock
   5 53.0 runtime/internal/atomic.Load
  45 51.0 runtime.runqempty
  12 12.0 runtime/internal/atomic.Store64
   8 91.0 runtime.casgstatus
  15 49.0 runtime/internal/atomic.Xadd
 364 365.0 runtime.stopm
 443 443.0 runtime.netpoll
 108 172.0 runtime.lock
 295 295.0 runtime.injectglist
 246 246.0 runtime.runqsteal
   3  3.0 runtime.releasep
  30 30.0 runtime.pidleput
   8 16.0 runtime.pidleget
   4  4.0 runtime.netpollinited
   3 12.0 runtime.runqget
   9  9.0 runtime.globrunqget
   3 22.0 runtime/internal/atomic.Xchg64
   1  7.0 runtime.fastrand1
   1  1.0 runtime.nanotime
-----------------
host f8e02f9facaa304dce98c8d876270a10
1 callers:
  in  flat symbol
1763 1853.0 runtime.schedule
21 callees:
 out  flat symbol
 268 268.0 runtime.runqsteal
  24 24.0 runtime.pidleput
 477 477.0 runtime.netpoll
 109 167.0 runtime.lock
   4 12.0 runtime.acquirep
   6 58.0 runtime/internal/atomic.Load
   7  7.0 runtime/internal/atomic.Store64
 298 298.0 runtime.injectglist
  49 54.0 runtime.runqempty
  33 71.0 runtime.unlock
  11 117.0 runtime.casgstatus
 327 328.0 runtime.stopm
   5 12.0 runtime.pidleget
  10 10.0 runtime.globrunqget
   5  9.0 runtime.runqget
   7  7.0 runtime.netpollinited
  12 40.0 runtime/internal/atomic.Xadd
   1  7.0 runtime.fastrand1
   4 24.0 runtime/internal/atomic.Xchg64
   1  1.0 runtime.releasep
   1  1.0 runtime.nanotime

Interestingly, the single-head cpu consumption is at 560% of 1200%, and the dual-head cpu consumption is at 470% and 468% of 1200%, respectively.
A couple notable details:

Performance is substantially worse in the single-host case (65% of the dual-host case), despite the fact that it is only half-loaded and backed by the same set of storage nodes running an identical front-end workload. I suppose some of this could be chalked up to head-of-line blocking, but I suspect there's more going on. In principle I'd expect very little difference between the two configurations, since none of these requests need to synchronize.
Proportionally more time (35% vs 27%) is spent in runtime.findrunnable in the single-node case. I'd expect that system to have on average 2x the number of goroutines, but I didn't think more goroutines would cause the proportional amount of time in the scheduler to increase. (I had presumed that more goroutines meant less work-stealing and polling, which would mean proportionally less time doing expensive stuff like syscalls and atomics.)

Let me know if there are other details I can provide.
Thanks,
Phil

@philhofer, any chance you could try Go 1.8beta1? Even if a bug were found in Go 1.7, that branch is closed for all but security issues at this point.
Go 1.8 should be a drop-in replacement for 1.7.  See https://beta.golang.org/doc/go1.8 for details. The SSA back end for ARM will probably help your little devices a fair bit. See https://dave.cheney.net/2016/11/19/go-1-8-toolchain-improvements

(Tagging this Go 1.9, unless you can reproduce on 1.8 and @aclements thinks it's easily fixable enough for 1.8)

@pilhofer. I'd be keen to see the .svg versions of those profiles if you
are able to attach them to the image.
On Thu, Dec 8, 2016 at 7:25 AM, Brad Fitzpatrick<EMAIL_ADDRESS>wrote:

@philhofer https://github.com/philhofer, any chance you could try Go
1.8beta1? Even if a bug were found in Go 1.7, that branch is closed for all
but security issues at this point.
Go 1.8 should be a drop-in replacement for 1.7. See
https://beta.golang.org/doc/go1.8 for details. The SSA back end for ARM
will probably help your little devices a fair bit. See
https://dave.cheney.net/2016/11/19/go-1-8-toolchain-improvements
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/18237#issuecomment-265593029, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA3CKeXoytI49tHLwYxgUG6n5H54xks5rFzJJgaJpZM4LHL5e
.

@bradfitz Yes, we're busy trying to get 1.8beta1 on some hardware to benchmark it. We're very excited about the arm performance improvements. (However, these profiles are on an Intel Xeon-D host, which I presume will perform similarly between 1.7 and 1.8, unless there have been substantial changes made to the scheduler that I missed?)
@davecheney Yes; I'll try to post a slightly-redacted one.

Update: most of the scheduler activity is caused by blocking network reads.
The call chain goes across two call stacks, which makes it a little tough to track down through stack traces alone, but here it is:

net.(*netFD).Read
net.(*pollDesc).wait
net.(*pollDesc).waitRead
net.runtime_pollWait
runtime.netpollblock
runtime.gopark
runtime.mcall(park_m)
runtime.park_m
runtime.schedule
runtime.findrunnable
(etc)

@davecheney I haven't extracted our profile format into the pprof format yet, but I hope that answers the same question you were hoping the svg web would answer.

Yes, we're busy trying to get 1.8beta1 on some hardware to benchmark it. We're very excited about the arm performance improvements. (However, these profiles are on an Intel Xeon host, which I presume will perform similarly between 1.7 and 1.8, unless there have been substantial changes made to the scheduler that I missed?)

Oh, sorry, missed that. In any case, please test 1.8 wherever possible in the next two weeks. It's getting increasingly hard to make changes to 1.8. The next two weeks are the sweet spot for bug reports. Thanks!

We just finished our first set of runs on 1.8, and things look pretty much identical on our x86 machines.
--------------------------
host 4edd58c28c7b9b548cc360334bae7af7: 6619 samples
 top flat  pct symbol
1766 1872 26.7 syscall.Syscall
 993  993 15.0 i/c/siphash.blocks
 720  720 10.9 runtime.futex
 461  461  7.0 runtime.epollwait
 443  443  6.7 runtime.memmove
 173 1759  2.6 runtime.findrunnable
 107  107  1.6 runtime.casgstatus
  88  136  1.3 runtime.lock
  86  136  1.3 runtime.runqgrab
  64   64  1.0 runtime.usleep
--------------------------
host f40105cffd2f1ec62e180b34677fc560: 6665 samples
 top flat  pct symbol
1704 1789 25.6 syscall.Syscall
 976  976 14.6 i/c/siphash.blocks
 666  666 10.0 runtime.futex
 469  469  7.0 runtime.epollwait
 408  408  6.1 runtime.memmove
 168 1768  2.5 runtime.findrunnable
  99   99  1.5 runtime.casgstatus
  95  145  1.4 runtime.lock
  89   89  1.3 runtime.usleep
  86  157  1.3 runtime.runqgrab

@aclements, what's the status here?

Ping @aclements

I have a little more information, in case you're interested.
Fundamentally, the issue here is that io.ReadFull(socket, buf) where len(buf) is, say, 64kB (or really any number that is a large-ish multiple of your 1500-byte MTU), causes the scheduler to wake up that goroutine len(buf)/1500 times. So, if you have 20 goroutines doing this with 64kB buffers, then you'll eat more than 850 scheduler wakeups before all those buffers have been filled.
Now, in a sane world we could wire up io.ReadFull on a socket such that it called setsockopt(SO_RCVLOWAT) so that the caller didn't receive a notifcation until there was plenty of data to read, but, frustratingly, SO_RCVLOWAT doesn't work with poll or select.
So, part of this is Linux's fault, and part of it is caused by the scheduler being generally slow. (Consider: in that profile, we spend nearly twice as much time in the scheduler as we do checksumming every single byte received over the network.)

Thanks for the extra information, @philhofer. That's very useful in understanding what's going on here.
Given how much time you're spending in findrunnable, it sounds like you're constantly switching between having something to do and being idle. Presumably the 1500 byte frames are coming in just a little slower than you can process them, so the runtime is constantly looking for work to do, going to sleep, and then immediately being woken up for the next frame. This is the most expensive path in the scheduler (we optimize for the case where there's another goroutine ready to run, which is extremely fast) and there's an implicit assumption here that the cost of going to sleep doesn't really matter if there's nothing to do. But that's violated if new work is coming in at just the wrong rate.
I'm not really sure what to do about this. It would at least help confirm this if you could post an execution trace (in this case, a sufficiently zoomed-in screen shot is probably fine, since there's no easy way to redact the stacks in an execution trace).

Hello gophers.
We have a similar issue — mcall, findrunnable, schedule excessively consume CPU time (mcall is up to 75% on stack). The application itself consumes 30-50% CPU while it mainly should be in I/O wait. The application is basically a proxy with tiny additional logic (receive 20 bytes packets over TCP/IP, accumulate up to 500 bytes and send it again over TCP/IP).
We see that the CPU workload comes specifically from the network I/O (and bit of timers), without connections the application is almost idling.
What makes it worse in our case, that mcall/findrunnable are very aggressive, to the point where the application is mainly running runtime functions, and our own code gets delayed so that we see a very visible lag up to seconds.
The case described by @philhofer is not applicable in our case, we read and write relatively small (should be below 1500) packets, and increasing packet size (batch write / batch read) seems to decrease the CPU consumption.
We use very simple protocol on top of TCP: [varint size][payload] and for reading we use bufio.Reader on top of net.TCPConn with default size of 4096 (but the size doesn't change a thing)
Please let me know if you need full profile or source code.

pprof.buffer.samples.cpu.004.pb.gz

@aka-rider What CPU, and OS, and version of Go did this profile come from? The amount of time spent in nanotime jumps out to me as surprisingly high compared to what I've seen in other profiles.

@aka-rider In addition to Chris' questions, could you describe the application a bit more? In particular:

Approximately how many goroutines do you have? What is GOMAXPROCS?
What kind of cross-goroutine synchronization is there? i.e., is the proxy receiving data, accumulating, and then sending all from the same goroutine, or is it sending data around?
What kind of load do you expect this application to be under? What is the overall system load?

I'm particularly curious about the last question, as you mention "lag up to seconds", yet your profile only shows 118s of CPU on 300s wall-time. i.e., less than 1 CPU of work on average. Unless the system is under heavy load, I wouldn't expect to see such extreme latency, which may be indicative of a scheduling bug rather than overall performance (what this issue is about).

I forgot to mention that Go 1.16 will include several timer overhead improvements, so if you can test on tip (if this isn't already on tip), that would be helpful.

Hi @ChrisHines
Yeah, I forgot to mention that.
We running virtual machines in KVM: model name: AMD EPYC Processor (with IBPB) in Ubuntu 16.04 (both host and guest)
go version 1.15 but we have this issue since 1.14 (and most probably since 1.13)
I agree, nanotime also bothers me.
I came across this ticket https://github.com/golang/go/issues/27707, and the following snippet:
package main

import (
  "time"
)

func main() {
  for {
    time.Sleep(time.Millisecond)
  }
}

consumes ~10% CPU on our cluster (it's < 2% CPU on my Mac and my colleague's Ubuntu laptop)

Hi @prattmic
We have 1 goroutine per incoming connection (receive), 1 goroutine per outgoing connection (async send)
All messages are processed by a single goroutine.
The number of connections is 10-200. Interestingly, the CPU consumption doesn't depend on the number of connections too much — it's ~0% with 0 connections, 30% with one connection, 30-50% with 200 connections.
This particular profile comes from our development environment, the amount of connections was ~6-8, and machine was not busy. We used to see significant delays with increased number of connections.
Our servers are not at the capacity, overall CPU/Memory/bandwidth usage is below 50%, even with 200 connections.
I tried GOMAXPROCS 1, 2, 5, 20, 50 — and it doesn't seem to change anything.
The initial design was using channels. Every read goroutine would write received data to a single buffered channel chan []byte, then processing goroutine would receive, process, and write data to a channel of the particular send goroutine (also buffered chan []byte).
We experienced significant delays within the application, part of the problems came from the GC pressure — each receiving goroutine would allocate memory for each incoming message, and part of the problem was in a channel communication — from my understanding it might be because receiving goroutines weren't scheduled for a significant amount of time, and it would create a vicious circle: more data comes to chan, nobody reads, more CPU spent in runtime.
Right now, we use circular buffer with mutex — all receiving goroutines write into it, one goroutine reads from it, and communication with send goroutines is the same, except is buffer+mutex per sending goroutine.
In that way we no longer see delays, but the CPU consumption is high even with a single incoming and single outgoing connection.
I also tried to get rid of asynchronous send goroutines, and write to TCP directly from the processing goroutine, it increased the CPU consumption from 30% to 50% for a single connection.
It makes me think that it specifically TCPConn.Write (and maybe Read) cause this behavior.

@aka-rider lots of time in nanotime could be explained by an inefficient VDSO implementation. Inside of a VM, the VDSO may not be using the fastest time mechanism (via the TSC), which can slow down clock reads significantly.
I don't know of a handy way to see which clock is in besides reading it directly from a program. https://gist.github.com/prattmic/a9816731294715426d7b85eb091d0102 will dump out the VDSO parameter page, from which on my system I can see the clock mode at offset 0x80 is 0x1 (VDSO_CLOCKMODE_TSC), which is what I want. (Offset and location varies depending on kernel version...).
All of this is quite a bit of a tangent from the issue at hand, though, as even a slow VDSO shouldn't be causing this much trouble.
I'll take a look at the rest of your summary tomorrow.

Hi @prattmic
./a.out
VDSO base: 0x7ffda7efc000

0x7ffda7efa000: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa010: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa020: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa030: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa040: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa050: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa060: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa070: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa080: 0x63998e0e 0x00000002 0x9086e69f 0x0028c85b
0x7ffda7efa090: 0xffffffff 0xffffffff 0x007fff2f 0x00000017
0x7ffda7efa0a0: 0xe7b5cf71 0x001b3ec7 0x5fca3a56 0x00000000
0x7ffda7efa0b0: 0x00338eb2 0x00000000 0xca35cf71 0x0011930a
0x7ffda7efa0c0: 0x5fca3a56 0x00000000 0x367d8fcf 0x00000000
0x7ffda7efa0d0: 0x00338eb2 0x00000000 0x23261594 0x00000000
0x7ffda7efa0e0: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa0f0: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa100: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa110: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa120: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa130: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa140: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa150: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa160: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa170: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa180: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa190: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa1a0: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa1b0: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa1c0: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa1d0: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa1e0: 0x00000000 0x00000000 0x00000000 0x00000000
0x7ffda7efa1f0: 0x00000000 0x00000000 0x00000000 0x00000000

uname -a
Linux buffer-6f4ff77d57-x4pql 4.4.0-193-generic #224-Ubuntu SMP Tue Oct 6 17:15:28 UTC 2020 x86_64 GNU/Linux

With gotip the CPU profile and consumption is very similar to what it was before
(gdb) p 'runtime.buildVersion'
$1 = 0xa68d6c "devel +b67b7dd Fri Dec 4 08:49:16 2020 +0000"

(pprof) top10
Showing nodes accounting for 8960ms, 70.05% of 12790ms total
Dropped 151 nodes (cum <= 63.95ms)
Showing top 10 nodes out of 83
      flat  flat%   sum%        cum   cum%
    2820ms 22.05% 22.05%     2960ms 23.14%  runtime.nanotime (inline)
    1170ms  9.15% 31.20%     1170ms  9.15%  runtime.futex
     860ms  6.72% 37.92%      870ms  6.80%  runtime.unlock2
     850ms  6.65% 44.57%      850ms  6.65%  runtime.casgstatus
     710ms  5.55% 50.12%      790ms  6.18%  runtime.lock2
     650ms  5.08% 55.20%     5180ms 40.50%  runtime.findrunnable
     590ms  4.61% 59.81%     3810ms 29.79%  runtime.checkTimers
     570ms  4.46% 64.27%      980ms  7.66%  gitlab.com/aigent/platform/processing/nq.(*sub).dequeue
     460ms  3.60% 67.87%      460ms  3.60%  runtime.epollwait
     280ms  2.19% 70.05%      310ms  2.42%  sync.(*Mutex).Unlock
(pprof) top10 -cum
Showing nodes accounting for 4.89s, 38.23% of 12.79s total
Dropped 151 nodes (cum <= 0.06s)
Showing top 10 nodes out of 83
      flat  flat%   sum%        cum   cum%
     0.15s  1.17%  1.17%     10.27s 80.30%  runtime.mcall
     0.22s  1.72%  2.89%      8.57s 67.01%  runtime.schedule
     0.12s  0.94%  3.83%      8.44s 65.99%  runtime.gosched_m
     0.13s  1.02%  4.85%      8.32s 65.05%  runtime.goschedImpl
     0.65s  5.08%  9.93%      5.18s 40.50%  runtime.findrunnable
     0.59s  4.61% 14.54%      3.81s 29.79%  runtime.checkTimers
     2.82s 22.05% 36.59%      2.96s 23.14%  runtime.nanotime (inline)
         0     0% 36.59%      1.71s 13.37%  main.main.func2
         0     0% 36.59%      1.68s 13.14%  runtime.park_m
     0.21s  1.64% 38.23%      1.20s  9.38%  gitlab.com/aigent/platform/processing/nq.(*sub).Receive

It does seem that your system is using VCLOCK_PVCLOCK instead of TSC, though looking at the implement it seems pretty efficient except for the VCLOCK_NONE case, which the comment says will only affect Xen, not KVM. I still don't think the clock source is really the underlying problem here, but it may be interesting if you can run this workload in a non-virtualized environment to see if it behaves better.

More importantly, I now notice that the entry point to ~90% of your time in the scheduler is due to calls to runtime.Gosched. The standard library and runtime rarely call Gosched, so I'm wondering if some part of your application is calling it (and perhaps way too much)? (Note that Gosched calls mcall, which unfortunately breaks the stack trace in the profile, so we can't directly see the caller).

Thank for your suggestion @prattmic.

if you can run this workload in a non-virtualized environment to see if it behaves better.

It seems to behave better on non-virtualized environment, although it's hard to reproduce the exact workload. Lab benchmarks on our laptops don't have the same CPU consumption.
I had the following in my code (in the next lines there's a call to Cond.Wait() ):
for cycles := 0; cycles < 10000; cycles++ {
	if payload, stream, err = s.dequeue(p); err != nil || payload != nil {
		return
	}
	runtime.Gosched()
}

The following profile with the code above commented out:
(pprof) top10
Showing nodes accounting for 51460ms, 69.38% of 74170ms total
Dropped 442 nodes (cum <= 370.85ms)
Showing top 10 nodes out of 92
      flat  flat%   sum%        cum   cum%
   30820ms 41.55% 41.55%    30820ms 41.55%  runtime.futex
    4150ms  5.60% 47.15%     4180ms  5.64%  runtime.runqgrab
    2770ms  3.73% 50.88%     2770ms  3.73%  runtime.usleep
    2750ms  3.71% 54.59%    39370ms 53.08%  runtime.findrunnable
    2690ms  3.63% 58.22%     2920ms  3.94%  syscall.Syscall
    2570ms  3.47% 61.68%     2570ms  3.47%  runtime.epollwait
    2030ms  2.74% 64.42%     9530ms 12.85%  runtime.checkTimers
    1490ms  2.01% 66.43%     1490ms  2.01%  runtime.write1
    1200ms  1.62% 68.05%     1450ms  1.95%  runtime.nanotime (inline)
     990ms  1.33% 69.38%     6440ms  8.68%  runtime.runtimer
(pprof) top10 -cum
Showing nodes accounting for 34.50s, 46.51% of 74.17s total
Dropped 442 nodes (cum <= 0.37s)
Showing top 10 nodes out of 92
      flat  flat%   sum%        cum   cum%
     0.05s 0.067% 0.067%     46.58s 62.80%  runtime.mcall
     0.04s 0.054%  0.12%     46.50s 62.69%  runtime.park_m
     0.19s  0.26%  0.38%     46.38s 62.53%  runtime.schedule
     2.75s  3.71%  4.09%     39.37s 53.08%  runtime.findrunnable
    30.82s 41.55% 45.64%     30.82s 41.55%  runtime.futex
     0.23s  0.31% 45.95%     16.10s 21.71%  runtime.futexsleep
         0     0% 45.95%     15.99s 21.56%  runtime.mstart
         0     0% 45.95%     15.99s 21.56%  runtime.mstart1
     0.30s   0.4% 46.35%     15.99s 21.56%  runtime.sysmon
     0.12s  0.16% 46.51%     15.50s 20.90%  runtime.startm

Ah, this looks better. Not better that usage isn't down much, but better in that it now looks like the symptoms that this issue was originally intended to address. i.e., this now looks like there may be lots of fast cycles between idle and active, where the scheduler is spending lots of time futilely trying to find work, then finally going to sleep only to be re-woken almost immediately.
Is there a particular reason you need that Gosched in your code, or can it go? If it can go, then this will probably be in a better place for future improvements. (I'm a bit surprised that this didn't improve things at least by a few percent).
To help verify the above behavior, could you provide the full pprof profile for this run? Plus a runtime trace? You may need to make the run fairly short if the trace is unreasonably large.

Hi.

Is there a particular reason you need that Gosched in your code, or can it go?

Right now the application behaves well without it, so it can go. The reason it was added is because waiting on chan could left a goroutine sleeping for too long. It seems like waiting on a Cond is better in that sense.
pprof.buffer.samples.cpu.008.pb.gz
I will think of how to get the trace from Docker container.

In general the Go scheduler reacts badly to any kind of busy loop, whether implemented with runtime.Gosched or otherwise.

In general the Go scheduler reacts badly to any kind of busy loop

Thanks for letting me know @ianlancetaylor
Please find the attached trace
trace.zip

Now, in a sane world we could wire up io.ReadFull on a socket such that it called setsockopt(SO_RCVLOWAT) so that the caller didn't receive a notifcation until there was plenty of data to read, but, frustratingly, SO_RCVLOWAT doesn't work with poll or select.

It looks like this was fixed starting with 2.6.28:

Since Linux 2.6.28, select(2), poll(2), and epoll(7) indicate a socket as readable only if at least SO_RCVLOWAT bytes are available.

So I wonder if it's possible to implement that fix now?


[TASK] KPIs for DevRel/Documentation

Update: We have now identified the KPIs and communicated these to engineering leadership.
Left to do:

Identify blog usage for API-M (obtained from CS/support team)
Compliance checks


[Errno 22] Invalid argument

Trying to obtain intraday data I obtain the following error:
data = hub.candle('AMZN', resolution='60', end="2020-05-22")
Traceback (most recent call last):
File "", line 1, in
data = hub.candle('AMZN', resolution='60', end="2020-05-22")
File "C:\Users\casa\Anaconda3\lib\site-packages\fhub\utils.py", line 143, in helper
return func(clase, *args, **kwargs)
File "C:\Users\casa\Anaconda3\lib\site-packages\fhub\utils.py", line 134, in helper
return func(clase, *args, **kwargs)
File "C:\Users\casa\Anaconda3\lib\site-packages\fhub\core.py", line 566, in candle
'from': _unixtime(start),
File "C:\Users\casa\Anaconda3\lib\site-packages\fhub\utils.py", line 169, in _unixtime
return int(datetime.strptime(_normalize_date(date), "%Y-%m-%d").timestamp())
OSError: [Errno 22] Invalid argument

Hi, you should specify the start argument value when you use candle().
i.e. data = hub.candle('AMZN', resolution='60', start="2020-05-15", end="2020-05-22")

Apparently it's a Python error in Windows. I didn't find that bug when developing the code on Linux.
Timestamp conversion on windows fails with timestamps close to EPOCH
I'll find a way around it.

@arogima is right, Windows has some problem to handle timestamp regarding dates before 1970 (see here). In particular, with my laptop (Windows 10) I have the same issue with all date before 1970/01/03 (yes 3rd January, I don't know why for the moment). I have just sent a pull request with a simple fix: if start argument is None then its value will be "1970-01-03".

start="2020-05-15",

thanks it works when start date is introduced!

I update a new version 0.0.13, please @dgarciagud  check if this version solved the bug in Windows.

this is solved @paduel.
Additionally I have to deal with to minor errors
ImportError: cannot import name 'json_normalize' from 'pandas' (C:\Users\casa\Anaconda3\lib\site-packages\pandas_init_.py)
using from pandas.io.json import json_normalize instead of directly from pandas.
And another with NA from pandas.

this is solved @paduel.
Additionally I have to deal with to minor errors
ImportError: cannot import name 'json_normalize' from 'pandas' (C:\Users\casa\Anaconda3\lib\site-packages\pandas__init__.py)
using from pandas.io.json import json_normalize instead of directly from pandas.
And another with NA from pandas.

Nice!
You must to update Pandas for the other error.


Reddit broken urls

Coverage increased (+0.2%) to 97.581% when pulling 52449e50a033e7752a522039d2be7a2bc0be7f7d on reddit-broken-urls into d1815a7ce8a79860dd8c2ee9486215efe2665a6f on master.

Coverage increased (+0.2%) to 97.581% when pulling 9dd83aafe0e85b6938d90c725fc5f3cd24aa1ffe on reddit-broken-urls into d1815a7ce8a79860dd8c2ee9486215efe2665a6f on master.


virt-handler metrics do not integrate any k8s VMI properties (labels, names, etc)

Is this a BUG REPORT or FEATURE REQUEST?:
FEATURE REQUEST

Uncomment only one, leave it on its own line:
/kind bug

/kind enhancement
What happened:
We're currently working on exposing the VMI metrics (cpu, mem, net, disk) that come from the virt-handler metrics endpoint. Right now these metrics come directly from the virt-handler pods and the only kubernetes identifiable information is "domain=$name-space_$pod-name" which is the libvirt domain information. We were hoping that these metrics would have identifiable information from the kubernetes metrics api, such as pod_labels, etc.
I spoke with @rmohr  and he mentioned the intention of kubevirt metrics to integrate nicely with the k8s metrics and asked me to open an issue.
What you expected to happen:
I had hoped that VMI identifying information might be passed down into the kubevirt_* metrics, such as the VMI name, VMI namespace, VMI labels, and other relevant k8s spec information.
How to reproduce it (as minimally and precisely as possible):
Access prometheus on a server running kubevirt with the kubevirt serviceMonitor. Query for and kubevirt_* metrics such as; kubevirt_vm_memory_resident_bytes
The output will be similar to the following;
kubevirt_vm_memory_resident_bytes{domain="golden-images_vm-centos-1",endpoint="metrics",instance="<IP_ADDRESS>:8443",job="kubevirt-prometheus-metrics",namespace="kubevirt",pod="virt-handler-c5m4b",service="kubevirt-prometheus-metrics"}
Anything else we need to know?:
Environment:

KubeVirt version (use virtctl version):
Server Version: version.Info{GitVersion:"v0.15.0-alpha.0", GitCommit:"31d08ef5b6c0170b21f5823d351f43902168e2d5", GitTreeState:"clean", BuildDate:"2019-02-21T11:00:06Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes version (use kubectl version):
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration:
Baremetal

OS (e.g. from /etc/os-release):

NAME="CentOS Linux"
VERSION="7 (Core)"

Kernel (e.g. uname -a):

4.13.10-1.el7.elrepo.x86_64

Install tools:
Others:

@fromanirh something for you?

ack! will check this ASAP.

One side-note: We should not add the VMI labels to the prometheus labels. That would not be good for prometheus. Instead, clearly expose node, name and namespace (maybe uid). This would allow connecting the dots directly from prometheus. To integrate in the custom metrics api, this info can then be used to back-connect the metrics to the custom metrics api endpoint of the pod, so that it can be used by the autoscalers, ...

/remove-lifecycle stale

@fromanirh do you know if you will get some time to work on this? I think this is a very important limitation right now.

@rmohr I'll try to squeeze in the upcoming sprint.

OK, the first step will be to implement what @rmohr proposed in https://github.com/kubevirt/kubevirt/issues/2108#issuecomment-471966831 . I aim to have such a PR posted within this week.

/reopen
@mikejk8s we finally have name, namespace, and node. How desperately do you need labels? @fromanirh do we know if kubelet metrics expose pod labels too?

I guess labels are great to monitor distributed or scaleable apps or VMs.

/reopen
@mikejk8s we finally have name, namespace, and node. How desperately do you need labels? @fromanirh do we know if kubelet metrics expose pod labels too?

(sorry for late answer) I need to check about kubelet. Do we want to go and add the labels?

/remove-lifecycle rotten
@fromanirh any news?

/remove-lifecycle stale

/lifecycle frozen


Ping Pong Game

Create a Ping Pong Game using React.js

Hey, @Shreyaar12  assign me this issue. I will try to create this game as soon as possible. If you want to see my work please visit my LinkedIn profile: https://www.linkedin.com/in/priyam-mishra-31843a205/

I would like to work on this issue.

I would like to work on this issue, please assign this issue to me under GSSoC'23.

@priyam212001 assigned the issue to you, make sure you complete it within 10 days

Sure!
On Sat, May 20, 2023, 11:30 PM Shreya Rajpal @.***>
wrote:

@priyam212001 https://github.com/priyam212001 assigned the issue to
you, make sure you complete it within 10 days
—
Reply to this email directly, view it on GitHub
https://github.com/ssitvit/Games-and-Go/issues/3#issuecomment-1555962605,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AXGY75XL5Z3X3M63TBSHDJLXHEBFHANCNFSM6AAAAAAQ34BJRU
.
You are receiving this because you were mentioned.Message ID:
@.***>

Can I use JS as a language?
On Sat, May 20, 2023, 11:43 PM Priyam Mishra @.***>
wrote:

Sure!
On Sat, May 20, 2023, 11:30 PM Shreya Rajpal @.***>
wrote:

@priyam212001 https://github.com/priyam212001 assigned the issue to
you, make sure you complete it within 10 days
—
Reply to this email directly, view it on GitHub
https://github.com/ssitvit/Games-and-Go/issues/3#issuecomment-1555962605,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AXGY75XL5Z3X3M63TBSHDJLXHEBFHANCNFSM6AAAAAAQ34BJRU
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hi @Shreyaar12 I would like to work on this issue under GSSoC

Hi, @Shreyaar12  I have completed this project in JS. So, If you don't mind can I request to merge this project into your project?


Question on eager execution with SavedModel

Hello @zzh8829,
I have been able to successfully convert a yolo-tiny-v3 model to checkpoints and later on to the SavedModel format for serving. I have a question about how to perform inference in the best way. I see that in your script export_tfserving.py line 60-61 and 72 you use the following notation:
    model = tf.saved_model.load(FLAGS.output)
    infer = model.signatures[tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY]
    outputs = infer(img)

I tried using your approach in a python script and I notice that it performs fast on 1 image, but slows down considerably on batch sizes of 16, 32, 64 images.  I am wondering - does it have to do with eager execution?
When using Tensorflow 1.14 notation with frozen .pb graphs in a session of similar models, running 1 image is slow but running a batch of images is much faster, especially when using GPU acceleration. This is an old-school way of running it:
    input_operation = graph.get_operation_by_name(input_name)
    output_operation = graph.get_operation_by_name(output_name)

    with tf.compat.v1.Session(graph=graph) as sess:
        results = sess.run(
            output_operation.outputs[0],
            {input_operation.outputs[0]: img})

        sess.close()

Could you please enlighten me if there is a way to run SavedModel in a session as well, to get faster speed on batches of images? Or, in general, what are the options here?

In general session don't exist in tf2 anymore. You can check out tensorflow serving i have had good experience with it. Other than that, I recommend predict_on_batch function from tf.keras


Strange interaction between docker and shake after upgrading from shake version 0.8

Hi Neil,
we have a problem with building our system with shake inside docker containers. shake rebuilds targets all the time, even if nothing changed. I think there is a bug in computing the timestamp of files. We are using ChangeModtime btw.
I think the problem is in src/Development/Shake/FileInfo.hs
There is conditional compilation going on, and my guess is that the part
#if MIN_VERSION_unix(2,6,0)
extractFileTime x = ceiling $ modificationTimeHiRes x * 1e4 -- precision of 0.1ms
#else
is wrong. I should not multiply by 1e4 but divide.
In our case, the multiplication results in an overflow for the timestamp of the source file and than shake considers the source file to be more recent than the target file and recompiles everything.
I would simply use modificationTime instead of modificationTimeHiRes.
One more thing: while debugging all this, I noticed that the show instance of FileInfo subtracts 2 from the value, but the fileInfo function adds 3 to the value. Shouldn't these two be inverse?
Stefan

Hi Stefan,

The multiplication by 1e4 is correct. I get modification times as a floating point number of seconds. I multiply by 1e4 so the 0.1 ms values become Int and then use ceiling to grab the Int part. I expect some overflow, but as long as its deterministic, that's sufficient for Shake. I use the low bits to give a good timestamp - it's exceptionally rare files change year but not day at the same time.

Can you use -VVV or shakeVerbosity=Diagnostic to get more information. In particular you should see lots of output, but one line should be the file, what it was recorded as, and what it appears as now. That should give you a clue as to what is going on.

I suspect the show should add 3, and the comment above FileInfo should say 2 is special as well. Since it's only showing Int, and these Int's are pretty much a random selection of bits from a timestamp, without any external meaning, it doesn't really matter. But 2 isn't what I was aiming for, I'll audit more thoroughly tonight and check I'm not missing something, then change it.

Thanks, Neil

Of course, there's an outside chance the overflow behaviour I'm relying on isn't well defined, and thus different inlinings produce slightly different roundings and that breaks my assumptions. Let me know if it seems to be that. Another possible cause would be Docker returning slightly different timestamps each time round - a print on the call with file/timestamp should show that.

Here is the output of ls (for source and target file):
root@e9e329c9d1d6:/DociGateway-local/server# ls -l src/lib/Mgw/Util/Misc.hs
-rw-r--r-- 1 503 staff 11519 Feb 11 19:40 src/lib/Mgw/Util/Misc.hs
root@e9e329c9d1d6:/DociGateway-local/server# ls -l build/lib/Mgw/Util/Misc.f_o
-rw-r--r-- 1 root root 158088 Mar 18 09:28 build/lib/Mgw/Util/Misc.f_o
When playing around with ghci, you see that the extractFileTime function from shake does something odd:
Prelude System.Posix.Files.ByteString> hsStat <- getFileStatus "src/lib/Mgw/Util/Misc.hs"
Prelude System.Posix.Files.ByteString> foStat <- getFileStatus "build/lib/Mgw/Util/Misc.f_o"
Prelude System.Posix.Files.ByteString> modificationTimeHiRes hsStat
1423680015s
Prelude System.Posix.Files.ByteString> modificationTimeHiRes foStat
1426667311s
Prelude System.Posix.Files.ByteString Data.Word> let extractFileTime x = (ceiling $ modificationTimeHiRes x * 1e4)::Word32 -- "ported" from shake
Prelude System.Posix.Files.ByteString Data.Word> extractFileTime hsStat
3278531056
Prelude System.Posix.Files.ByteString Data.Word> extractFileTime foStat
3086719984
Here is some of shake's output when building build/lib/Mgw/Util/Misc.f_o:
% Chunk 368 [len 67] 0000018d000700000000000000187372632f6c69622f4d67772f5574696c2f4d6973632e6873010006c36a69f300002d020000000200000002000000020038345ae600 (Id 397,(src/lib/Mgw/Util/Misc.hs,Loaded (Result {result = File {mod=0xC36A69F1,size=0x2D00,digest=VAL}, built = Step 2, changed = Step 2, depends = [], execution = 4.3e-5, traces = []})))
% Chunk 1289 [len 75]<PHONE_NUMBER>07000000000000001b6275696c642f6c69622f4d67772f5574696c2f4d6973632e665f6f010006b7c79cc00002698b0000000200000002000000020101000006273b20902e00 (Id 1426,(build/lib/Mgw/Util/Misc.f_o,Loaded (Result {result = File {mod=0xB7C79CBE,size=0x26989,digest=VAL}, built = Step 2, changed = Step 2, depends = [[Id 1575]], execution = 2.45e-3, traces = []})))
As far as I understand the debugging output, the mod time of Misc.hs is larger than that of Misc.f_o, though that's not the case in reality.
I'm using the unix-<IP_ADDRESS> package.

I forgot to mention something important: we are using docker and shake 0.8.* for half a year without any such problems. The problem only occurred when upgrading shake to the latest version.

OK, so given the sudden upgrade, it would be surprising if it wasn't an issue with Shake. The fact that the files change order when you truncate the bits isn't an issue - I only ever check if timestamps change, not that they are ordered. The log fragment shows what the files have been loaded at, can you also find the valid lines which say what the timestamps were now computed to. Something like:
% valid False for output/basic/configure (File {mod=NEQ,size=NEQ,digest=NEQ})
% valid True for output/basic/once.txt (File {mod=0x74EAFC53,size=0x1,digest=VAL

})
% valid True for output/basic/zero.txt (File {mod=0x74E89AF3,size=0x1,digest=VAL
})
A few things occur to me:

Try running with --lint first. If the issue is that timestamps are somehow unstable in your system that might show it up more directly.
I note that Misc.f_o depends on Id 1575, while Misc.hs is Id 397. Likely that's irrelevant as there is an Id in between them that does the link (perhaps the .hi file, or a both node, if you are using &%>).
The value for Misc.hs in your system matches what you computed. For Misc.f_o the value stored is B7C79CBE, but the computed value is b7fb9bef. Did you touch anything between taking those measurements and running the DB? The valid lines should give us all the information in one pass.

I did another debugging session in a fresh docker container. Here is the outcome:
root@dfa53f382eef:/DociGateway-local/server# ghci -XOverloadedStrings
Prelude> import System.Posix.Files.ByteString
Prelude System.Posix.Files.ByteString> import Data.Word
Prelude System.Posix.Files.ByteString Data.Word> hsStat <- getFileStatus "src/lib/Mgw/Util/Misc.hs"
Prelude System.Posix.Files.ByteString Data.Word> foStat <- getFileStatus "build/lib/Mgw/Util/Misc.f_o"
Prelude System.Posix.Files.ByteString Data.Word> modificationTimeHiRes hsStat
1423680015s
Prelude System.Posix.Files.ByteString Data.Word> modificationTimeHiRes foStat
1426667311s
Prelude System.Posix.Files.ByteString Data.Word> let extractFileTime x = (ceiling $ modificationTimeHiRes x * 1e4)::Word32
Prelude System.Posix.Files.ByteString Data.Word> extractFileTime hsStat
3278531056 -- 0xC36A69F0
Prelude System.Posix.Files.ByteString Data.Word> extractFileTime foStat
3086719984 -- 0xB7FB9BF0
root@dfa53f382eef:/DociGateway-local/server# ls -l src/lib/Mgw/Util/Misc.hs build/lib/Mgw/Util/Misc.f_o
-rw-r--r-- 1 root root  158088 Mar 18 09:28 build/lib/Mgw/Util/Misc.f_o
-rw-r--r-- 1  503 staff  11519 Feb 11 19:40 src/lib/Mgw/Util/Misc.hs
root@dfa53f382eef:/DociGateway-local/server# ls -al build
total 440484
...
-rw-r--r--  1 root root   1992797 Mar 18 09:32 .shake.database
root@dfa53f382eef:/DociGateway-local/server# ./shake --shake-diagnostic --debug build/lib/Mgw/Util/Misc.f_o > shake.log 2>&1
Here is a summary of the shake.log file:
Starting build, targets: build/lib/Mgw/Util/Misc.f_o
Resolved targets: ["build/lib/Mgw/Util/Misc.f_o"]
% Chunk 31 [len 56] 0000002c000500000000000000187372632f6c69622f4d67772f5574696c2f4d6973632e68730100040100000002000000020037da1a9300 (Id 44,(doesFileExist src/lib/Mgw/Util/Misc.hs,Loaded (Result {result = True, built = Step 2, changed = Step 2, depends = [], execution =
2.6e-5, traces = []})))
% Chunk 368 [len 67] 0000018d000700000000000000187372632f6c69622f4d67772f5574696c2f4d6973632e6873010006c36a69f300002d020000000200000002000000020038345ae600 (Id 397,(src/lib/Mgw/Util/Misc.hs,Loaded (Result {result = File {mod=0xC36A69F1,size=0x2D00,digest=VAL}, built = S
tep 2, changed = Step 2, depends = [], execution = 4.3e-5, traces = []})))
% Chunk 1287 [len 185]<PHONE_NUMBER>090000000000000002000000000000001b6275696c642f6c69622f4d67772f5574696c2f4d6973632e665f6f000000000000001c6275696c642f6c69622f4d67772f5574696c2f4d6973632e665f68690100080000000000000002b7c79cc00002698b00000002b7c795b800002949000000020000000
20000000205010000018b0100000336020000018d000004c7010000001501000004ba3f67e1330100000000000000066d792d676863402621f7405f420b (Id 1575,(build/lib/Mgw/Util/Misc.f_o build/lib/Mgw/Util/Misc.f_hi,Loaded (Result {result = Files {mod=0xB7C79CBE,size=0x26989,digest=VAL} {mod=0x
B7C795B6,size=0x2947,digest=VAL}, built = Step 2, changed = Step 2, depends = [[Id 395],[Id 822],[Id 397,Id 1223],[Id 21],[Id 1210]], execution = 0.90578, traces = [Trace "my-ghc" 2.595823 3.488406]})))
% Chunk 1289 [len 75]<PHONE_NUMBER>07000000000000001b6275696c642f6c69622f4d67772f5574696c2f4d6973632e665f6f010006b7c79cc00002698b0000000200000002000000020101000006273b20902e00 (Id 1426,(build/lib/Mgw/Util/Misc.f_o,Loaded (Result {result = File {mod=0xB7C79CBE,size=0x26989,
digest=VAL}, built = Step 2, changed = Step 2, depends = [[Id 1575]], execution = 2.45e-3, traces = []})))
% Chunk 1304 [len 76] 000005f20007000000000000001c6275696c642f6c69622f4d67772f5574696c2f4d6973632e665f6869010006b7c795b8000029490000000200000002000000020101000006273c9a75cd00 (Id 1522,(build/lib/Mgw/Util/Misc.f_hi,Loaded (Result {result = File {mod=0xB7C795B6,size=0x294
7,digest=VAL}, built = Step 2, changed = Step 2, depends = [[Id 1575]], execution = 1.8855e-2, traces = []})))
% Chunk 5389 [len 185]<PHONE_NUMBER>090000000000000002000000000000001b6275696c642f6c69622f4d67772f5574696c2f4d6973632e665f6f000000000000001c6275696c642f6c69622f4d67772f5574696c2f4d6973632e665f68690100080000000000000002b7ee19570002698b00000002b7c793a300002949000000020000000
30000000305010000018b0100000336020000018d000004c7010000001501000004ba3f01e5190100000000000000066d792d6768633f708ebd3fb91b1e (Id 1575,(build/lib/Mgw/Util/Misc.f_o build/lib/Mgw/Util/Misc.f_hi,Loaded (Result {result = Files {mod=0xB7EE1955,size=0x26989,digest=VAL} {mod=0x
B7C793A1,size=0x2947,digest=VAL}, built = Step 3, changed = Step 3, depends = [[Id 395],[Id 822],[Id 397,Id 1223],[Id 21],[Id 1210]], execution = 0.507402, traces = [Trace "my-ghc" 0.939678 1.44614]})))
% Chunk 5399 [len 76] 000005f20007000000000000001c6275696c642f6c69622f4d67772f5574696c2f4d6973632e665f6869010006b7c793a3000029490000000200000003000000030101000006273b2bf76a00 (Id 1522,(build/lib/Mgw/Util/Misc.f_hi,Loaded (Result {result = File {mod=0xB7C793A1,size=0x294
7,digest=VAL}, built = Step 3, changed = Step 3, depends = [[Id 1575]], execution = 2.624e-3, traces = []})))
% Chunk 5404 [len 75]<PHONE_NUMBER>07000000000000001b6275696c642f6c69622f4d67772f5574696c2f4d6973632e665f6f010006b7ee19570002698b0000000200000003000000030101000006273a562bf100 (Id 1426,(build/lib/Mgw/Util/Misc.f_o,Loaded (Result {result = File {mod=0xB7EE1955,size=0x26989,
digest=VAL}, built = Step 3, changed = Step 3, depends = [[Id 1575]], execution = 8.17e-4, traces = []})))
% Chunk 8264 [len 185]<PHONE_NUMBER>090000000000000002000000000000001b6275696c642f6c69622f4d67772f5574696c2f4d6973632e665f6f000000000000001c6275696c642f6c69622f4d67772f5574696c2f4d6973632e665f68690100080000000000000002b7fbb8150002698b00000002b7c793a300002949000000020000000
40000000405010000018b0100000336020000018d000004c7010000001501000004ba3f0020b40100000000000000066d792d6768633f38c8bd3f9c32ba (Id 1575,(build/lib/Mgw/Util/Misc.f_o build/lib/Mgw/Util/Misc.f_hi,Loaded (Result {result = Files {mod=0xB7FBB813,size=0x26989,digest=VAL} {mod=0x
B7C793A1,size=0x2947,digest=VAL}, built = Step 4, changed = Step 4, depends = [[Id 395],[Id 822],[Id 397,Id 1223],[Id 21],[Id 1210]], execution = 0.500499, traces = [Trace "my-ghc" 0.721813 1.220298]})))
% Chunk 8265 [len 75]<PHONE_NUMBER>07000000000000001b6275696c642f6c69622f4d67772f5574696c2f4d6973632e665f6f010006b7fbb8150002698b0000000200000004000000040101000006273b437e6f00 (Id 1426,(build/lib/Mgw/Util/Misc.f_o,Loaded (Result {result = File {mod=0xB7FBB813,size=0x26989,
digest=VAL}, built = Step 4, changed = Step 4, depends = [[Id 1575]], execution = 2.983e-3, traces = []})))
% Chunk 8266 [len 76] 000005f20007000000000000001c6275696c642f6c69622f4d67772f5574696c2f4d6973632e665f6869010006b7c793a3000029490000000200000004000000030101000006273bab92c000 (Id 1522,(build/lib/Mgw/Util/Misc.f_hi,Loaded (Result {result = File {mod=0xB7C793A1,size=0x294
7,digest=VAL}, built = Step 4, changed = Step 3, depends = [[Id 1575]], execution = 5.236e-3, traces = []})))
% valid False for build/lib/Mgw/Util/Misc.f_o (File {mod=0xB7FBB813,size=0x26989,digest=VAL})
% Loaded -> Waiting, build/lib/Mgw/Util/Misc.f_o
build/lib/Mgw/Util/Misc.f_o
% valid False for (build/lib/Mgw/Util/Misc.f_o build/lib/Mgw/Util/Misc.f_hi) (Files {mod=0xB7FBB813,size=0x26989,digest=VAL} {mod=0xB7C793A1,size=0x2947,digest=VAL})
% Loaded -> Waiting, build/lib/Mgw/Util/Misc.f_o build/lib/Mgw/Util/Misc.f_hi
build/lib/Mgw/Util/Misc.f_o build/lib/Mgw/Util/Misc.f_hi
% valid True for src/lib/Mgw/Util/Misc.hs (File {mod=0xC36A69F1,size=0x2D00,digest=VAL})
% Loaded -> Ready, src/lib/Mgw/Util/Misc.hs
[DEBUG] Rule for compiling hs file, hs=src/lib/Mgw/Util/Misc.hs, depFile=build/lib/Mgw/Util/Misc.dep, pkgFile=build/lib/Mgw/Util/Misc.pkg, oDir=build/lib
src/lib/Mgw/Util/Misc.hs ==> build/lib/Mgw/Util/Misc.f_o
scripts/my-ghc -hide-all-packages -package-id HTF-<IP_ADDRESS>-c4266c43aec6695a5c240bcafc204137 -package-id base-<IP_ADDRESS>-8aa5d403c45ea59dcd2c39f123e27d57 -package-id containers-<IP_ADDRESS>-ab1dae9a94cd3cc84e7b2805636ebfa2 -package-id directory-<IP_ADDRESS>-91a788fd88acd7f149f0f10f5f
1e23f2 -package-id filepath-<IP_ADDRESS>-b12cbe18566fe1532a1fda4c85e31cbe -package-id mtl-2.2.1-eda0a12427f66d975ce72a312c95b37a -package-id old-time-<IP_ADDRESS>-2f8ea093d0c7014780a8a5772f948883 -package-id time-<IP_ADDRESS>-10dc4804a19dc0000fab79908f1a9f50 -package-id unix-<IP_ADDRESS>-4
f219f792083f70ec440ce406f12ce57 -isrc/lib -isrc/progs -ibuild/lib -ibuild/progs -ibuild/gen-hs -ibuild/gen-o -hisuf f_hi -osuf f_o -Werror -W -fwarn-unused-imports -fwarn-unused-binds -fwarn-unused-matches -fwarn-unused-do-bind -fwarn-wrong-do-bind -pgmPcpphs -optP--cpp
-rtsopts -threaded -funbox-strict-fields -O0 -outputdir build/lib -c src/lib/Mgw/Util/Misc.hs -o build/lib/Mgw/Util/Misc.f_o
my-ghc (for build/lib/Mgw/Util/Misc.f_o build/lib/Mgw/Util/Misc.f_hi)
% Waiting -> Ready, build/lib/Mgw/Util/Misc.f_o build/lib/Mgw/Util/Misc.f_hi
% result (build/lib/Mgw/Util/Misc.f_o build/lib/Mgw/Util/Misc.f_hi) = (Files {mod=0xEF327FAD,size=0x26989,digest=VAL} {mod=0xB7C793A1,size=0x2947,digest=VAL})
% Waiting -> Ready, build/lib/Mgw/Util/Misc.f_o
% result build/lib/Mgw/Util/Misc.f_o = (File {mod=0xEF327FAD,size=0x26989,digest=VAL})
% Checking if build/lib/Mgw/Util/Misc.f_o is File {mod=0xEF327FAD,size=0x26989,digest=VAL}, passed
% Checking if build/lib/Mgw/Util/Misc.f_o build/lib/Mgw/Util/Misc.f_hi is Files {mod=0xEF327FAD,size=0x26989,digest=VAL} {mod=0xB7C793A1,size=0x2947,digest=VAL}, passed
% Checking if src/lib/Mgw/Util/Misc.hs is File {mod=0xC36A69F1,size=0x2D00,digest=VAL}, passed
Here is a link to the full shake.log file: https://dl.dropboxusercontent.com/u/10626736/shake-docker-problem.log
What has changed in shake in between 0.8 and now that could case such strange behavior under docker? Note that the problem only occurs the first time when entering the container. If I build the file again (in the same container), no recompilation is done:
root@dfa53f382eef:/DociGateway-local/server# ./shake build/lib/Mgw/Util/Misc.f_o
Starting build, targets: build/lib/Mgw/Util/Misc.f_o
Resolved targets: ["build/lib/Mgw/Util/Misc.f_o"]

So the change which I'm getting increasingly sure is to blame is that previously Shake had 1 second-resolution timestamps, and now it has 0.1 millisecond resolution timestamps. Looking at the logs (thanks for those) I get:

Shake ran and recorded a time of 0xB7FBB813 for Misc.f_o
You restarted docker
You manually sampled the time and it gave 0xB7FB9BF0

That equates to a 0.7s timestamp difference. Under the old Shake that might be considered "the same". Under the new Shake they won't.
Since the lint check succeeds that means that after a compile you still get given the exact same timestamp. My hypothesis is Docker is returning incorrect file times. Can you run the experiment:

Create a file in Docker.
Call modificationTimeHiRes on the file (not doing the weird Shake things, just the raw value).
Get the time again.
Restart docker.
Get the time.
Restart docker.
Get the time.

If my guess is right, 1 and 2 will be the same time, 3 will be different, and then I have no idea if 4 will match 3 or be different again (that will give me more info though).

Note that as a temporary workaround, changing to ChangeModtimeAndDigest would probably workaround the bug, as then you'd be relying on hashes not modtimes. Still interested in tracking it down properly though.

Neil, thanks for your help. Just one quick question: what do you mean by "restarting docker"? Creating a new container from the same image? Or something else?

So, did some more debugging. I couldn't verify your guess that docker returns inconsistent timestamps. But I noticed that the modification times inside a docker container never contain subseconds. The stat output always looks like this:
root@ac67985aa526:/DociGateway-local/server# stat build/lib/Mgw/Util/Misc.f_o
  File: ‘build/lib/Mgw/Util/Misc.f_o’
  Size: 158088    	Blocks: 312        IO Block: 4096   regular file
Device: 28h/40d	Inode: 389         Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-03-21 09:19:01.287189223 +0100
Modify: 2015-03-21 09:15:18.000000000 +0100
Change: 2015-03-21 09:19:01.287189223 +0100
 Birth: -

Docker does a lot of magic, so I could imagine that there are situations where the mtime include subseconds, but I never encountered such a situation. I tested it during a docker build and also inside a docker container.
I then tried your suggestion with ChangeModtimeAndDigest.We are still using GHC 7.6, so I was hit by a GHC RTS bug I reported two years ago (https://ghc.haskell.org/trac/ghc/ticket/7646). I guess that computing the md5sum in shake causes this bug to appear. I our part of the build system, we are using special functions for reading files to work around this bug
I solution to the whole problem would be to add a flag to shake telling it not to use the subseconds of the mtime. Would this be an option for you?

By restarting I meant shutting and reopenning the same Docker image. It does seem that Shake managed to record timestamps with some fractional seconds, since they're in the log. The fact that Shake doesn't rebuild after the first time also suggests it continued to return fractional seconds for some period of time. Perhaps the cache layer after writing out a file records the timestamps to more precision, but after that (once they are flushed to disk) it loses the subsecond part? I guess a single ghci in which you writeFile then immediately call modificationTimeHiRes might be interesting - if that gives fractional seconds then it would explain the issue.
Upgrading to 7.8 would probably be good, but we're still on 7.2, and also have nasty file reading functions to work around that bug...
If you compile Shake with the portable flag (cabal configure --flags=portable) I think that will only use second resolution. If it is more accurate than that, then I'm happy to change the meaning of the flag to ignore subseconds, since that would indeed give the most portable version.

Ok, then I understood correctly what you meant with "shutdown".
Your guess with the caching layer seems to to be good. It seems that docker looses timestamp precision when commiting a container's changes to a new image.
Inside docker container d33a797349c5:
root@d33a797349c5:/DociGateway-local/server# ghci -XOverloadedStrings
Prelude System.Posix.Files.ByteString> stat <- do writeFile "file.txt" "FOO"; getFileStatus "file.txt"
Prelude System.Posix.Files.ByteString> modificationTimeHiRes stat
1426949143.059797807s
root@d33a797349c5:/DociGateway-local/server# stat file.txt
  File: ‘file.txt’
  Size: 3         	Blocks: 8          IO Block: 4096   regular file
Device: 28h/40d	Inode: 1486        Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-03-21 17:24:30.581088152 +0100
Modify: 2015-03-21 17:24:30.581088152 +0100
Change: 2015-03-21 17:24:30.581088152 +0100
Birth: -

Then commit the changes:
$ docker commit d33a797349c5
0a269c7307900268dcbd796d5316a8cef8ea0d9e78f55a5ea9c41f71cc28c131

Create a container from this new image:
$ docker run -i -t --rm 0a269c7307900268dcbd796d5316a8cef8ea0d9e78f55a5ea9c41f71cc28c131
root@d8b4c7d47d41:/DociGateway-local/server# stat file.txt
  File: ‘file.txt’
  Size: 3         	Blocks: 8          IO Block: 4096   regular file
Device: 28h/40d	Inode: 89          Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-03-21 17:25:12.875520807 +0100
Modify: 2015-03-21 17:24:30.000000000 +0100
Change: 2015-03-21 17:25:12.875520807 +0100
Birth: -

I also tested with a patched version of shake, where timestamps do not have sub-seconds precision. With this version, everything is as with shake 0.8, that is there are no unnecessary rebuilds.
So, if shake could have an option that causes sub-seconds to be dropped, that would be great. For now, we are just using our patched version of shake, so there is no hurry.

Did you try cabal configure --flags=portable?

(way too late commenting on this issue)
Yes, it works with --flags=portable.

Thanks for the info. Leaving it open for the moment - there's something fishy going on here, and I want to have a bit more think.

I think a file system must have the invariant that modification time is correct and remains constant while things aren't being modified - and your Docker system seems not to, so (as far as I can tell) that's a bug in Docker or your particular Docker setup. Lots of people are using Shake with Docker successfully, so hopefully it's quite rare. There is the portable config flag workaround, and also you can use any of the shakeChange settings which also use digests, which is a generic workaround for any times of flakey modtime. If it happens to others I'll add it as an FAQ, and perhaps have some "test" mode, but otherwise I'll just ignore for now.