[MM-36659] Check for valid HomepageURL

Summary
Every apps needs to have a valid HomepageURL. This is now checked and verified. HTTP apps also need an HTTPRootURL.
The PR also fixes two build issue:

The minimum go version needs to be 1.6
All go test command should run against all packages, not only the ones in ./server

Ticket Link
https://mattermost.atlassian.net/browse/MM-36659

Codecov Report

Merging #211 (a760da1) into master (7a2443f) will decrease coverage by 2.46%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #211      +/-   ##
==========================================
- Coverage   25.53%   23.07%   -2.47%
==========================================
  Files          25       49      +24
  Lines        1402     2488    +1086
==========================================
+ Hits          358      574     +216
- Misses       1013     1858     +845
- Partials       31       56      +25

Impacted Files
Coverage Δ

apps/manifest.go
82.97% <100.00%> (ø)

utils/httputils/limited_readcloser.go
92.85% <0.00%> (ø)

apps/permissions.go
27.58% <0.00%> (ø)

upstream/upaws/provision_data.go
59.00% <0.00%> (ø)

upstream/upaws/upstream.go
14.00% <0.00%> (ø)

utils/byte_size.go
0.00% <0.00%> (ø)

upstream/upaws/client.go
0.00% <0.00%> (ø)

apps/app.go
100.00% <0.00%> (ø)

utils/utils.go
0.00% <0.00%> (ø)

upstream/upaws/clean.go
0.00% <0.00%> (ø)

... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7a2443f...a760da1. Read the comment docs.

@DHaussermann Given the good test coverage I'm inclined to merge this PR without QA review. The test steps would be:

Edit the manifest of an app of contain an invalid HomepageURL like ssl://example.org
Install the app
Notice that the installation fails with an error.

@mickmister Would you please merge this PR after QA is complete?

@DHaussermann inclined to merge without manual QA


Sound

So far your App works as stated. I have blue tooth hooked up to my Yamaha 6,5 and some issues with that. It works fine with Spotify on my Laptop or phone (no Problem). Roku works on the TV (no Problem), Roku has 2 modes for sound. TV/Mega boom 3 (no Problem)
The Roku will not work on the sound Video/Receiver. How do I get any Roku to work on my receiver too.


Does cityflow support vehicles with different size

If I simulate a bus having width of 2 lanes, a car (as usual with width of 1 lane) and a motorbike with a high speed, can cityflow support such difference in vehicles' sizes/speeds.
If yes, is there a way to get back the type (size) of vehicle back from simulator.

Cityflow does support different size and speed. You can take a look at examples/flow.json, and you can see that a vehicle has several properties such as length, width, and max speed.
To get back the type (size) of vehicles back from the simulator, I don't think it would be a problem because all vehicles are defined by you in flow.json, and the simulator will not change it. If you insist, you can find it in the replay file: replay.txt.

So, if I get want to check the size of a vehicle named flow_30_0, can I directly refer to the corresponding 31st (or 30th) vehicle in the flow.json (provided start/end times are same).
I want to get identify the vehicle during simulation, either by using size or speed or type etc. Replay might help after simulation ends. Can you suggest a way to have that done during the simulation.
Thanks

Yes, you can directly refer to the corresponding 31st (or 30th) flow in the flow.json;
Every vehicle has a unique name, such as flow_30_0, so I suggest you use the vehicle name to identify.
By the way, vehicles are named by their flow type(what you define in the flow.json) and their order. For example, flow_30_10 means the 11th vehicle of the 31st flow type.


[Improvement] Add data-kjb-element attributes to Sage React components

Description
UI test automation needs to be able to locate elements based on attributes in the DOM. Currently, most locators are based on CSS classes. This is unreliable and makes UI automation dependent on the presentation of the app. These locators are likely to break when minor UI changes are made to the app. The best practice for UI automation is to add test ID attribute to the elements that UI tests interact with. This attribute should be used only for automated UI tests. The attribute should only change when the functionality in the app changes. This allows developers to change the presentation (styles, element tag type, etc) without breaking UI automation.
We already use the attribute data-kjb-element on Ruby components. I would like to add them to the Sage React components as well.
I have a local branch where I've added them to the components that we use in our test automation. I can push the branch if you'd like to take a look.
Design
https://kajabi.atlassian.net/wiki/spaces/QA/pages/2681733155/WIP+Test+IDs+for+UI+automation+in+React
Additional Information
Please see the inline examples in the design document.

Issue has been added to Jira project: https://kajabi.atlassian.net/browse/DSS-1222
For future requests, please feel free to check out our IT Portal! https://kajabi.atlassian.net/servicedesk/customer/portal/51


when I upload bam or vcf， is iobio.io handel data which upload from local?

as title, I don't know if it is safe （maybe can protect my data to prevent my data to share anywhere）， when upload my personal data（bam or vcf）？

I'll add that technically using https://bam.iobio.io does give our servers full access to your data. As @AlistairNWard, we don't abuse that access, but there is some risk for example if someone managed to hack our servers they might be able to get access as well. It is possible to selfhost both bam.iobio and the gru backend (https://github.com/iobio/iobio-gru-backend), but selfhosting carries its own risks, which you'll want to be sure to understand.


[GLSL] Structure aliasing not working properly

input:
struct TestVaryingStruct
{
	float f;
};

struct TestStructUniform
{
	float x;
};

layout(binding = 0) uniform Uniforms
{
	mat4 projection;
	TestStructUniform test;
};

layout(location = 0) in vec4 in_Position;
layout(location = 1) in vec4 in_Color;

layout(location = 1) out vec4 out_Color;
layout(location = 4) out TestVaryingStruct out_TS;

void main()
{
	out_Color = in_Color;
	out_TS.f = 1.0;
	gl_Position = projection * in_Position;
}

Rename TestVaryingStruct and  TestStructUniform to _Struct_114174309
output:
struct _Struct_114174309
{
    float Member0;
};

struct _Struct_114174309_1
{
    float Member0;
};

layout(std140) uniform _UniformBufferBlock_S0_B2
{
    mat4 Member0;
    _Struct_114174309_1 Member1;
} _UniformBuffer_S0_B2;

out vec4 _Varying_L1;
layout(location = 1) in vec4 in_Color;

out _Struct_114174309 _Varying_L4;
layout(location = 0) in vec4 in_Position;

void main()
{
    _Varying_L1 = in_Color;
    _Varying_L4.Member0 = 1.0;
    gl_Position = _UniformBuffer_S0_B2.Member0 * in_Position;
    gl_Position.z = 2.0 * gl_Position.z - gl_Position.w;
}

Expected:
The _Varying_L4 and _UniformBufferBlock_S0_B2.Member1 must have the same type (_Struct_114174309)
Related issue: https://github.com/KhronosGroup/SPIRV-Cross/issues/1408

Oh, Sorry. It's different types. They shouldn't be aliased. One is block-like, second is not.


First line of template doesn't make sense

As seen here, the head of the generated file looks like this:
// Package docs GENERATED BY THE COMMAND ABOVE; DO NOT EDIT
// This file was generated by swaggo/swag
package docs

There is no command above that first line.

@TColl Swag is an open-source project, and you are welcome to fix this misleading comment by contributing with a PR.
Thanks!

@ubogdan done :)


feat: added typing for patterns and exposed logger

What did you implement:
I've created some typing for correlation-middleware, to add some clarity on where the correlationId and logger has been added, since the sqs capture correlationId puts the logger on the event and the capture streams exposes them on the context.
I've also exposed the Logger in correlation-middleware and appended it to the patterns. This just to reduce having to import both Logger and correlation-middleware. Due to the assumption that if I got the correlation-middleware I most likely will be using the logger.

How did you implement it:
Added a index.d.ts file and on the index.js exposed the logger as a named export.
How can we verify it:
Create a typescript project import logger and see ;)
Todos:

[ ] Write tests
[ ] Write documentation
[ ] Fix linting errors
[x] Make sure code coverage hasn't dropped
[ ] Provide verification config / projects / resources
[x ] Enable "Allow edits from maintainers" for this PR
[ ] Update the messages below

Is this ready for review?: NO
Is it a breaking change?: NO

Any opinions on the work done @gsingh1 ? I haven't exposed all of lambda Types, thinking it shouldn't be our responsibility to deal with all of them. So I've only exposed the types in which we've affected.

@simontabor or @gsingh1 can this be merged? The ability for custom logger shouldn't affect this story, since this this is just types for the current library.


[cpp/RE2] USE_RE2=ON fails to switch on RE2 regexp

Build Environment: gcc 4.8.1, RE2 installed from source
Steps:from libphonenumber/cpp
$ mkdir build
$ cd build
$ cmake -D USE_ICU_REGEXP=OFF -D USE_RE2=ON ..
$ make
Output: snipped successful output
...
Linking CXX executable libphonenumber_test
CMakeFiles/libphonenumber_test.dir/test/phonenumbers/regexp_adapter_test.cc.o: In function i18n::phonenumbers::ICURegExpFactory::ICURegExpFactory()': regexp_adapter_test.cc:(.text._ZN4i18n12phonenumbers16ICURegExpFactoryC2Ev[_ZN4i18n12phonenumbers16ICURegExpFactoryC5Ev]+0x1f): undefined reference tovtable for i18n::phonenumbers::ICURegExpFactory'
collect2: error: ld returned 1 exit status
make[2]: *** [libphonenumber_test] Error 1
make[1]: *** [CMakeFiles/libphonenumber_test.dir/all] Error 2
make: *** [all] Error 2
$
Actual: ICU is being switched on, RE2 is not.
Expected: RE2 should be switched on, not ICU.
Fix: In file cpp/test/phonenumbers/regexp_adapter_test.cc,
:%s/USE_RE2/I18N_PHONENUMBERS_USE_RE2
Post-fix: Output as Expected.

Ping.
This appears to be a simple fix.
Fix: In file cpp/test/phonenumbers/regexp_adapter_test.cc,
:%s/USE_RE2/I18N_PHONENUMBERS_USE_RE2
i.e. replace "USE_RE2" with "I18N_PHONENUMBERS_USE_RE2"
In case I am missing something here, please let me know.

This looks reasonable. Could you send a patch to fix this? Please also remember to sign the CLA first if you haven't done so:
https://github.com/googlei18n/libphonenumber/wiki/ContributeToTheProject

Hi @jias0001, would it be possible for someone on the libphonenumber team to please perform the string find and replace? I understand little about CLAs. Also, it would be considerably simpler for someone who already has everything setup.
On a related note, is it OK to use RE2 instead of ICU? I am trying to understand the trade off between the ICU and RE2 engines with respect to libphonenumber. In my very limited understanding, RE2 is the safer one, which is why I used USE_RE2=ON and this entire issue started.

Yes, it's OK to use RE2. They're both supported: https://github.com/googlei18n/libphonenumber/blob/master/cpp/README
I'll send a patch to fix this.

Oh nice feature! I said the PR fixes the issue and when I merged the PR the issue automatically close.
Note though that this is not yet released. It will be in the next GitHub tag.

Danke @keghani for the fix.
On a side note, would it be possible for you to share any insight/guidance regarding the tradeoff between ICU and RE2 engines with respect to libphonenumber? As mentioned earlier in the thread, in my limited understanding, RE2 seems to be the safer option.

I actually don't have context into ICU vs RE2 for regex matching in C++. would others be able to chime in?

And you're welcome :)


MAINT: integrate.qmc_quad: re-introduce qmc_quad

Reference issue
gh-17686
gh-16923
What does this implement/fix?
This brings back integrate.qmc_quad, which was previously merged but made private shortly before the release of 1.10.0 to allow time to reconsider the interface. This PR also makes the interface change discussed in gh-17628, which happens to be more consistent with integrate.nquad.
Additional Information
It's probably easier to review the two commits separately, since the first just reverts the PR that made qmc_quad private, and the second builds on top of that.

I am wondering. Would you consider changing n to d? Just a personal preference here.

Sure.


Failing nighty test: test_pixel2pixel (test_notebooks_single_gpu.StraightDopeSingleGpuTests)

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/NightlyTests_onBinaries/detail/NightlyTests_onBinaries/148/pipeline
FAIL: test_pixel2pixel (test_notebooks_single_gpu.StraightDopeSingleGpuTests)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/work/mxnet/tests/nightly/straight_dope/test_notebooks_single_gpu.py", line 255, in test_pixel2pixel

    assert _test_notebook('chapter14_generative-adversarial-networks/pixel2pixel')

AssertionError

The test is not Flaky.
This PR https://github.com/apache/incubator-mxnet/pull/12537 - reverts the change in the NDArrayIter
[WIP] PR - https://github.com/apache/incubator-mxnet/pull/12545 to capture edge case of empty label

Resolving this issue as the change is reverted and tests are passing.


Homebridge crashes if Nuki Smart Lock is not present

Hi Lukas,
I had to disassemble my Nuki Smart Lock yesterday because of an issue with my door. So it is now offline with its batteries removed.
I think the plugin doesn't like it when it cannot reach a configured lock. Is it possible ignore the lock temporarily when it cannot be reached?
Homebridge running into a crash loop is not optimal in this case, because I have to be able to change the batteries etc.
Thank you very much!
[2019-10-4 07:09:41] [NukiPlatform] Device with Nuki ID XXXXXXXXX and name Wohnungstür is a SmartLock. [2019-10-4 07:09:41] TypeError: Cannot read property 'state' of undefined     at NukiSmartLockDevice.update (/usr/local/lib/node_modules/homebridge-nuki/index.js:769:13)     at new NukiSmartLockDevice (/usr/local/lib/node_modules/homebridge-nuki/index.js:757:10)     at Object.callback (/usr/local/lib/node_modules/homebridge-nuki/index.js:147:31)     at Request._callback (/usr/local/lib/node_modules/homebridge-nuki/index.js:1026:12)     at Request.self.callback (/usr/local/lib/node_modules/homebridge-nuki/node_modules/request/request.js:185:22)     at emitTwo (events.js:126:13)     at Request.emit (events.js:214:7)     at Request.<anonymous> (/usr/local/lib/node_modules/homebridge-nuki/node_modules/request/request.js:1161:10)     at emitOne (events.js:116:13)     at Request.emit (events.js:211:7) [2019-10-4 07:09:41] Got SIGTERM, shutting down Homebridge...

@felixmueller can you try the development branch? Should prevent the error now.

Merged into master in #14.

Thank you very much for the fix! Sorry to not try it out in the first place, I didn’t want to recalibrate my lock again 😊


I know the build on Ubuntu fails anyway because of OpenSSL, but this error looks like because of this change.
In file included from QUICFrame.cc:24:
./QUICFrame.h:217:22: error: private field '_buf' is not used [-Werror,-Wunused-private-field]
      const uint8_t *_buf                                    = nullptr;
                     ^
1 error generated.

Yeah, I did not find in my local computer. Will fix.

Great !


feat: Added endpoint to mark notification as read

Description

This endpoint adds a new endpoint [PATCH] /api/v1/notifications/{id} which marks a notification as read. The notification can only be marked as read if the user making the request is the owner of the notification with the given id.
Related Issue (Link to issue ticket)

Related issue
How Has This Been Tested?

Screenshots (if appropriate - Postman, etc):

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

@Dev-wonderful

@Dev-wonderful


maxversiontested update, this should include multiple versions

@ChrisGuzak : Thanks for your contribution! The author(s) have been notified to review your proposed change.

This addresses https://dev.azure.com/microsoft/OS/_workitems/edit/33687330

@mcleanbyron : Thanks for your contribution! The author(s) have been notified to review your proposed change.

related: https://github.com/MicrosoftDocs/windows-uwp/pull/3183


ROX-23055: Add advanced filters bar to Platform CVE overview page

Description
As titled.
Checklist

[ ] Investigated and inspected CI test results
[ ] Unit test and regression tests added
[ ] Evaluated and added CHANGELOG entry if required
[ ] Determined and documented upgrade steps
[ ] Documented user facing changes (create PR based on openshift/openshift-docs and merge into rhacs-docs)

If any of these don't apply, please comment below.
Testing Performed
Visit Platform CVEs and apply filters:

Clear filters:

/retest


Fails to create /app/vendor mount

Something seems amiss with nanobox run when executed on a Linux (Debian) host with Docker native:
$ nanobox version
Nanobox version 2.0.4 (5d0aadd123e877eafe5cd122b081575f)

$ nanobox run
Preparing environment :

--------------------------------------------------------------------------------
+ HEADS UP:
+ This is the first build for this project and will take longer than usual.
+ Future builds will pull from the cache and will be much faster.
--------------------------------------------------------------------------------

Building runtime :
  ✓ Starting docker container
  ✓ Preparing environment for build
  ✓ Gathering requirements
  ! Mounting cache_dirs

! FAILED TO MKDIR -P /APP//VENDOR !

Exit
 1

Command
 su - gonano -c "mkdir -p /app//vendor"

Output
 mkdir: cannot create directory ‘/app//vendor’: Permission denied

Error   : failed to execute hook (mount) on 950cc75d98083b67230b6d505d54345c4b6b029afd021566b70fc48b2cb15413: util:Exec:/opt/nanobox/hooks/mount: bad exit code(1):
Context : failed to build the code

Due to the error itself I cannot enter the container and check the permissions. Any idea?

Interesting the mkdir is happening on the container. I thin @notxarb  might be able to shed some more  light on this. also it appears /app//vendor is trying to be created. Im assuming its because some variable is missing between /app/ and /vendor

So far, the only way I've been able to reproduce this is if I have the code in my current directory owned by root. I believe the reason this isn't an issue with virtualbox is because the code is mounted as the gonano user.
Possible solutions:

chown the code (not a great idea as it would modify it locally for the user)
bind mount the code to another location as user/group 1000 so the hooks work

Potential workarounds for users:

manually create directories as they are needed, and ensure they are writable by all
chown the code as user 1000 (chown -R 1000:1000 . from app directory)

Note: This will likely also be an issue if the code owner's id is not 1000

Stems from docker's inability to set owner/group of mounted volumes (this issue)

@mbrodala @rkkautsar and anyone else affected. I have created a test build image that should resolve this issue which I will redefine in a moment.  To test, ensure you have the following in your boxfile:
run.config:
  image: nanobox/build:bugfix_user

I still don't suggest running nanobox as root, but I believe this should even work if you still choose to do so (only tested one time as root).
The issue is bind mounting, and the fact that docker uses it. Bind mounts themselves are fantastic, but have limitations; such as the inability to set the owner/group. Since nanobox with docker native uses docker volumes (bind mounts) to your code, whoever owns them on your laptop will own them inside the container. Nanobox has created a user to run user applications and hooks, the user id of this user is 1000.
With default file permissions, users cannot write to something they do not own. Hence the Permission denied errors. This fix makes our nanobox user pretend to be you, so you can go ahead and use nanobox without worrying about what user you are logged into.

I don't see the issue ATM with Nanobox version 2.1.0 (5d0aadd123e877eafe5cd122b081575f) ...


Ensure "asyncRule" function can have custom Promise resolver

What
Need to ensure that asyncRule function can have custom Promise resolver.
Why
It's most likely developers would have their own resolve logic based on the WS response. It doesn't necessarily means that the request should resolve. It may have additional logic (i.e. depend on statusCode)
How
This is one of the expected usages of asyncRule:
<MyField asyncRule={() => fetch('https://backend.dev/ws', { ... }).then((payload) => {
  return (payload.statusCode !== 'ERROR')
}) } />

This reads: the field is considered valid unless payload.statusCode equals ERROR. In case the request throws, this should be handled automatically as invalid state of the field.

Also how to provide error message? As you can get message from WS or some error code...

@Vidlec yeap, need an easy way to bind async validations and their error messages.

@Vidlec the error messages for async validation have been provided within #25.
You can now handle async validation through FormProvider or directly in Form:
<FormProvider messages={{
  name: {
    email: {
      invalid: 'Please provide a proper email.', // sync validation
      async: { // async validation
        userExists: ({ payload, fieldProps }) => {
          if (payload.errorCode === '1234') {
            return `User with the email "${ fieldProps.value }" already exists."
          }
        }
      }
    }
  }
}}>
</FormProvider>

Each validation resolver has its unique key (userExists in the example above) for the sake of passing this key down to the custom UI component in the future. Maybe designers would want to show different UI behaviors depending on the validation outcome. This way we achieve granular control.

Update
Field.props.asyncRule is now expected to return an Object of a following shape:
{
  valid: boolean,
  ...rest
}

The validity of the field will be set based on the provided valid property. In case the field is invalid, all ...rest properties passed from the asyncRule prop will be available in the error message resolver:
const errorMessages = {
  name: {
    username: {
      async: {
        userExists: ({ value, errorCode }) => {
          return `User with the name "${value}" already exists. ${erroCode}.`;
        }
      }
    }
  }
};

<Field.Input
name="username"
asyncRule={({ value }) => fetch(...).then(res => res.json()).then((payload) => {
return {
valid: payload.statusCode === 'SUCCESS',
errorCode: payload.errorCode
};
}}
required />
Closing.


Update recording snippet

Update the Segment-FullStory integration with the most recent version of the FullStory recording snippet.

@segment-integrations/destinations, tagging you here so you can review this PR from our friends @ Fullstory. Thanks!

@sanjaybv @jameremo Thank you! We'll need to test this internally before merging it. Until then, can you provide some context on what is actually changing here? The code is minified so it's difficult to see what is changing here.

Thanks @tysonmote!  The integration is not changing with respect to how it integrates with Segment, but we are updating the recording script/snippet that bootstraps FullStory on the page.
There are 3 distinct changes to the snippet in this PR:

There is an additional check for whether the user needs to configure the namespace being used by FullStory, in case another library with the namespace "FS" is being used (line 35).  This is now a standard part of our snippet, although the log output it produces may mostly just be for debugging purposes with respect to Segment, since the namespace setting is not currently exposed by the integration.
We have a few new API calls being exposed (lines 38-39):  shutdown and restart (see the docs), as well as another GDPR-related call named consent for which there is no public documentation yet.
We are deprecating the clearUserCookie call, and replacing its original contents with a stub call that does nothing (line 41).

Hope that helps, let us know if you need any other info!
CC @sanjaybv

@jameremo Thanks for the context! I'll try to get this tested and deployed within the week. And just to double-check: There's nothing our mutual customers will need to do when we upgrade this snippet, correct?

@tysonmote Correct, this snippet should work transparently for all our customers with no additional config, and is in fact already deployed on our production servers.  We're working through all of our external integrations that contain a copy of the snippet to get those copies updated.  Thank you!!

Thank you @tysonmote!


how to make my angular library installs it's dependencies when using it ?

how to make my angular library installs it's deficiencies when using it ?
now I've made angular library that depends on ngx bootstrab and others
i don't want the library consumer to install all the dependencies first , but just install my library and that's it
so i wonder if that is possible to do

Hi
Is It Possible create multiple projects in a single Angular 6 Application?

Hello, we reviewed this issue and determined that it doesn't fall into the bug report or feature request category. This issue tracker is not suitable for support requests, please repost your issue on StackOverflow using tag angular-cli.
If you are wondering why we don't resolve support issues via the issue tracker, please check out this explanation.


Add CommandKit::Completion::Install

Add a CommandKit::Completion::Install module for installing a completion file into the completions directory of the current shell. Use CommandKit::Completions::Shell to determine the current shell type. Also use CommandKit::Env::Prefix to honor the PREFIX environment variable.

Implemented in cb2bbdd.


Ported Wireman to AdvDupe2 and fixed its E2

Why is that binary?

@TomyLobo I don't know why would you ask such a question.
It is recognized as binary file by GitHub.  Any Advanced Duplicator 2 file is treated as binary file on GitHub...

Because it contains non-textual data, and we all know only GLON would do such a horrid thing as that...

Well someone obviously didnt think things through when designing advdupe2...
The tiny amounts of space you save when storing those dupes are not worth the obfuscation, much less with a proprietary format like that.
Compressing for transfer, that's a different story and could be done on the fly.

@oldmud0

we all know only GLON would do such a horrid thing as that

If you think advdupe2 uses GLON, you're wrong then. From what I see (here), it uses FastLZ algorithm for compression of serialized dupe data..

I'm always wrong, so might as well ban me.

@oldmud0: You're right for the wrong reason.

Is there any advantage to using AdvDupe2 for this? (Wiremod only adopted development of AdvDupe2 because it was abandoned and we needed to fix things in it to be able to clean up AdvDupe2-specific hacks in Wiremod without people complaining at us.)

Current Wireman (AdvDupe1) is broken. This fixes its broken E2, and upgrades it to AdvDupe2. :dancer:

That is not an upgrade. AdvDupe and AdvDupe2, despite their names, are separate add-ons.

AdvDupe and AdvDupe2, despite their names, are separate add-ons.

Can you tell me what is Wireman doing here?


Trying to test in background

On simulator background works find but on device I puts console.logs but they not triggered
May you can help me ?

My code
console.log not triggered
Location.requestAlwaysAuthorization();
Location.startUpdatingLocation();
Location.setDistanceFilter(5.0);

var subscription = DeviceEventEmitter.addListener(
    'locationUpdated',
    (location) => {
      console.log('updated');
    }
);

Hi @ButuzGOL:  Are you using react-native v0.14 or greater?  I believe background JavaScript event support was added as of v0.14.

I am using ^0.18.0-rc

For me it is also not working on real device. Testing with RN 0.21.0-rc1. But working on simulator.

I have just discovered that another function is exposed in this implementation:
Location.setAllowsBackgroundLocationUpdates(true);
With adding this code, it is indeed working in background mode. But I think it was added only in iOs9 (link)
So I guess also documentation (about setAllowsBackgroundLocationUpdates) should be update.

thanks I will try


docs(pt) i18n: Translate reference/default-theme-badge.md

@viniciusdeliz could you assign me?

Just assigned this one to you as well, @savionascimentodev! Thank you

@viniciusdeliz, my friend will we translate the word 'badge' to 'emblem' or will we leave 'badge' anyway?

@viniciusdeliz, my friend will we translate the word 'badge' to 'emblem' or will we leave 'badge' anyway?

@savionascimentodev I used 'distintivo' in another instance. I recall seeing 'placa' for it as well somewhere. But I'm not a fan of both. Let's use 'emblema' as you suggested!

@viniciusdeliz, Should we translate the word ‘Custom Children’ too? If so, what would be the best 'Filho Personalizado'?

@viniciusdeliz, Should we translate the word ‘Custom Children’ too? If so, what would be the best 'Filho Personalizado'?

@savionascimentodev that's a tricky one, I'm reading here and it's a bit confusing and hard to translate. I'd write it as 'Filiação Personalizada'. Just a reminder that we should probably not translate elements, their attributes, CSS classes or related, such as <Badge />, --vp-badge, and else.

I'm sending the file and anything you let me know in pr
@viniciusdeliz One question, will you give permission to this repo, or should I create a fork too?

I'm sending the file and anything you let me know in pr
@viniciusdeliz One question, will you give permission to this repo, or should I create a fork too?

@savionascimentodev as you prefer. I added you as collaborator in this repo. Let me know if you need assistance


az_path Unicode Decode Error

Hi everyone,
I wanted to use the following line to start a game between the az-trained model and me.
python3 open_spiel/python/examples/mcts.py --game=connect_four --player1=az --player2=human --az_path=/mnt/d/Windows_to_Ubuntu/connect_four/checkpoint-240.pt
However, I ran into the following issue. I am using Ubuntu 20.04 LTS on my Windows 10. It passed 100% of the tests after I installed everything based on the instructions for Windows users. Also, I installed libtorch based on my colleague's suggestion. Unfortunately, this is still an issue. I hope someone could give me some suggestions. Thank you for your time and help in advance.
2021-05-18 19:01:49.871135: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-05-18 19:01:49.873442: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
File "open_spiel/python/examples/mcts.py", line 223, in
app.run(main)
File "/home/fengzhihao/.local/lib/python3.8/site-packages/absl/app.py", line 303, in run
_run_main(main, args)
File "/home/fengzhihao/.local/lib/python3.8/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "open_spiel/python/examples/mcts.py", line 197, in main
_init_bot(FLAGS.player1, game, 0),
File "open_spiel/python/examples/mcts.py", line 96, in _init_bot
model = az_model.Model.from_checkpoint(FLAGS.az_path)
File "/home/fengzhihao/open_spiel/open_spiel/python/algorithms/alpha_zero/model.py", line 202, in from_checkpoint
model = cls.from_graph(checkpoint, path)
File "/home/fengzhihao/open_spiel/open_spiel/python/algorithms/alpha_zero/model.py", line 215, in from_graph
saver = tf.train.import_meta_graph(metagraph)
File "/home/fengzhihao/.local/lib/python3.8/site-packages/tensorflow/python/training/saver.py", line 1465, in import_meta_graph
return _import_meta_graph_with_return_elements(meta_graph_or_file,
File "/home/fengzhihao/.local/lib/python3.8/site-packages/tensorflow/python/training/saver.py", line 1481, in _import_meta_graph_with_return_elements
meta_graph_def = meta_graph.read_meta_graph_file(meta_graph_or_file)
File "/home/fengzhihao/.local/lib/python3.8/site-packages/tensorflow/python/framework/meta_graph.py", line 643, in read_meta_graph_file
text_format.Merge(file_content.decode("utf-8"), meta_graph_def)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 70: invalid start byte

Hi @fzh6762788,
What produced checkpoint-240.pt ? The Python (TF-based) AlphaZero, or the Libtorch (C++-based) AlphaZero?
I believe that playing against a checkpoint in examples/mcts.py in your message was designed for the former. So if you're sending it LibTorch-generated checkpoints, that would certainly explain things.  I'm not 100% sure if we have something that can play against the torch-based checkpoints, but I know the original author is working on something related currently.
@tewalds, @christianjans can you weigh in on this?

Hi Lanctot,
Thank you so much for your response. That explains a lot. Yes, our az model was based on Libtorch. We thought that we could solve this by installing Libtorch. We are looking forward to receiving any updates on this matter.

I just remembered that @christianjans did add this recently: https://github.com/deepmind/open_spiel/pull/557
Specifically there is a C++ example that you can run called alpha_zero_game_example, here:

Hi @fzh6762788 and @lanctot, yes, to play a LibTorch model, run the alpha_zero_torch_game_example. The flags are slightly different than in examples/mcts.py, but hopefully they are clear. Please don't hesitate to ask if you have any questions!
And regarding the documentation for LibTorch AlphaZero, I have updated (and will continue update if things are unclear or need clarification) the alpha_zero_torch/README.md in #581 to show how to train, continue training, and play a saved model.

Hi @christianjans, this is a naive question. Do I still need to download or install something in my Ubuntu terminal? I am a little confused. Thank you so much for your time and help.

Do you see it in build/examples? i.e. can you run ls -l ./build/examples?
If not, you can try building it:
cd build
make alpha_zero_torch_game_example

If it doesn't work, can you paste the exact error?

@lanctot Thank you for your reply. I did not see that under /open_spiel/build/examples file. So I tried to build it under /open_spiel/build. It returned me "no rule to make target" as follows:
fengzhihao@Feng:~/open_spiel/build/examples$ ls
CMakeFiles           Makefile        cfr_example                   cmake_install.cmake  example  imperfect_recall_mccfr  mcts_example                                                                      policy_iteration_example    tabular_sarsa_example
CTestTestfile.cmake  benchmark_game  cfr_multi_equilibria_example  count_all_states     gtp      matrix_example         minimax_example  tabular_q_learning_example  value_iteration_example
fengzhihao@Feng:~/open_spiel/build/examples$ cd ..
fengzhihao@Feng:~/open_spiel/build$ make alpha_zero_torch_game_example
make: *** No rule to make target 'alpha_zero_torch_game_example'.  Stop.
fengzhihao@Feng:~/open_spiel/build$
Please let me know I made any mistakes.

Hi @fzh6762788, no worries, but yeah that is strange.
The updated README for LibTorch AlphaZero (https://github.com/deepmind/open_spiel/blob/49bfd9cb044316d58624bfd7a4436aab381322de/open_spiel/algorithms/alpha_zero_torch/README.md) may help. It has instructions on how to install, build, run training, and play a trained model for LibTorch AlphaZero. Did your setup process of LibTorch AlphaZero look similar to what this README is outlining?
Also, I'm wondering, the alpha_zero_torch_game_example.cc is a relatively new addition to OpenSpiel, if you run ls open_spiel/examples/ does alpha_zero_torch_game_example.cc show up?

@christianjans
Yes, I followed the instructions by running the following lines under the /open_spiel file,
export OPEN_SPIEL_BUILD_WITH_LIBTORCH="${OPEN_SPIEL_BUILD_WITH_LIBTORCH:-"ON"}"
and
export OPEN_SPIEL_BUILD_WITH_LIBTORCH_DOWNLOAD_URL="${OPEN_SPIEL_BUILD_WITH_LIBTORCH_DOWNLOAD_URL:-"https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.5.1%2Bcpu.zip"}"
and
./install.sh
Then, I got "command not found". Should I run those lines under another directory?
However, I saw alpha_zero_torch_game_example.cc under open_spiel/examples/.
Please let me know if it is solvable. Thank you for your help

Okay great @fzh6762788, glad you have alpha_zero_torch_game_example.cc 👍
And the commands look right. What command led to the "command not found" error? Did you also run ./open_spiel/scripts/build_and_run_tests.sh after running ./install.sh?

Thank you for your response. Here is what I got after I ran ./open_spiel/scripts/build_and_run_tests.sh under /open_spiel directory. Please let me know where I can improve.
fengzhihao@Feng:~/open_spiel$ ./open_spiel/scripts/build_and_run_tests.sh
Running ./open_spiel/scripts/build_and_run_tests.sh from /home/fengzhihao/open_spiel
Reusing virtualenv from ./venv.
Skipping installation of requirements.txt.
Building and testing in /home/fengzhihao/open_spiel/build using 'python' (version 3.8.5).

Could not find a package configuration file provided by "Torch" with any of
the following names:
TorchConfig.cmake
torch-config.cmake

Add the installation prefix of "Torch" to CMAKE_PREFIX_PATH or set
"Torch_DIR" to a directory containing one of the above files.  If "Torch"
provides a separate development package or SDK, be sure it has been
installed.
-- Configuring incomplete, errors occurred!
See also "/home/fengzhihao/open_spiel/build/CMakeFiles/CMakeOutput.log".
See also "/home/fengzhihao/open_spiel/build/CMakeFiles/CMakeError.log".
Exiting virtualenv

Hi @lanctot @christianjans,
My colleague helped me fix the issue above earlier. It turns out that it was a small issue that had nothing to do with open_spiel. This issue was because Unzip did not come with Ubuntu's installation. I never knew it had to be installed in the subsystem. After we installed Unzip and ran the open_spiel installation again, the AZ model with Pytorch worked! Thank you so much for your time and help.

Oh great to hear @fzh6762788! Glad this has been resolved, good find!


Add standard deviation to gateway status and widget

Before this pull request can be accepted you must first sign a CLA as described at https://www.pfsense.org/about-pfsense/#cla. Please read for more details.

I have previously signed the CLA, and had a PR accepted a few weeks ago. Is there a new version?

CLAs have been a little unreliable recently. Someone will fix this manually.

I couldn't find the CLA, maybe it was signed in the past and expired after 1 year. Nowadays it doesn't expire anymore. @jdillard can you take a look to make sure if CLA is signed or not?

@dennypage It looks like your old CLA expired on 1/5/2016. As @rbgarga mentioned the CLAs no longer automatically expire, so if you could log in and fill it out again (most of your info is saved) that would be appreciated: https://portal.pfsense.org/members/signup/ICLA
The automated system will pick it up shortly after that and autocorrect any PRs.

Thanks for checking Jared. Missed it by two weeks. :) I'll try to sign a new one tonight.

On Jan 19, 2016, at 10:06, Jared Dillard<EMAIL_ADDRESS>wrote:
@dennypage It looks like your old CLA expired on 1/5/2016. As @rbgarga mentioned the CLAs no longer automatically expire, so if you could log in and fill it out again (most of your info is saved) that would be appreciated: https://portal.pfsense.org/members/signup/ICLA
The automated system will pick it up shortly after that and autocorrect any PRs.
—
Reply to this email directly or view it on GitHub.

No worries. It should be smooth sailing from then on, but if you have problems feel free to ping me.
Thanks for all of your contributions!

Actually it looks like it worked doing it from my phone. Awesome. Please let me know if there is any issue. Thanks.

On Jan 19, 2016, at 11:34, Jared Dillard<EMAIL_ADDRESS>wrote:
No worries. It should be smooth sailing from then on, but if you have problems feel free to ping me.
Thanks for all of your contributions!
—
Reply to this email directly or view it on GitHub.

Thanks @dennypage !


Incorrect rotation of landscape cards

If the card are photographed in a landscape orientation (the card, that is) then the rotation of the card in the program will be incorrect.

[x] Make sure all cards are rotated to a portrait orientation (upside-down is OK, it's fixed later in the CardReader class).

Done!


Update addLinks to support a textview, with HTML tags in it.

closes: #31
Works when you use something like:
tvTaskDescription.text = HtmlCompat.fromHtml(task.description!!, HtmlCompat.FROM_HTML_MODE_LEGACY)
BetterLinkMovementMethod.linkify(Linkify.ALL, tvTaskDescription)

Thanks @cfsbhawkins, but I'm trying to keep this library a drop-in replacement for LinkMovementMethod. Adding linkify() functions were a mistake that I regret. A movement method library shouldn't be in the business of adding link spans to TextView, so I'll have to decline this. Sorry!
I should deprecate the linkify functions so that people stop using them and rely on framework APIs directly instead.


Deck manager crashes when mouse pointer moves over a card

Open the deck manager, everything works until the mouse overlaps a card image and then HSTracker crashes to the desktop.

what's your hstracker version, as well as osx version ?

HSTracker Version 0.12 (393)
OSX Yosemite 10.10.3


Rustfmt chokes on heavily nested code inside closures

Describe the bug
In deeply nested code, rustfmt is sometimes not able to get the code within the default max_width, so it just bails on formatting the entire statement.
The behavior seems to depend on the specific AST nodes being formatted. For example, it seems to happen in blocks inside of closures, but not in simple function blocks.
I don't want to simply bump up max_width, as that will change all the heuristics and join various things to a single line throughout my project.
To Reproduce
~Attempt to format this file and note that nothing happens: https://gist.github.com/umanwizard/36aae1346df5b567cf93f6e5d908f47b~
Edit: Actually, look at my comment below.
Expected behavior
~The println statement should have been de-indented, as it is at the wrong indentation.~
Edit: See my comment below.
Meta

rustfmt version: rustfmt 2.0.0-rc.2-nightly (87e96027 2021-03-01)
From where did you install rustfmt?: built from source
How do you run rustfmt: rustfmt path/to/file

Actually, I'm not sure if the linked repro file is demonstrating the particular bug I'm concerned about. (Rustfmt seems unable to format that even if I get rid of a bunch of the indentation... maybe that's a separate bug.)
A real-world example is here: https://github.com/MaterializeInc/materialize/blob/dba61fea9d6f0886ebff554b0353204ab47e1e25/src/dataflow/src/decode/csv.rs#L59
Try changing the indentation of that line 59 and note that Rustfmt can't fix it. It seems to be giving up on formatting anything in the let stream = stream.unary(... call.

Thanks for reaching out, but going to close this as a duplicate of several previous issues (e.g. #3863).
Your referenced snippet is actually a chain that starts up on line 49 via the read.unary( call
And you then have multiple lines of code in the deeply nested closures (which again are themselves part of a massive chain) which are simply not possible to reformat and fit within the max_width limit. One example is the string on line 134, and there are various cases of comments blowing way past the 100 column limit.
You can either modify your rustfmt config to give it permission to attempt to wrap your strings and comments to make things fit, or you can wrap them yourself and/or refactor things to reduce the requisite indentation levels.
e.g.
rustfmt src/dataflow/src/decode/csv.rs --check --config error_on_line_overflow=true,error_on_unformatted=true,format_strings=true,wrap_comments=true,comment_width=75

(note cli config overrides used for visibility, config file specification is fine too)

@calebcartwright is it not considered a bug for rustfmt to be unable to format code?
Perhaps the documentation should be changed, then. It suggests that this is a bug: "If this happens, then it is a bug in Rustfmt."
Anyway, what seems ideal here is a setting that allows rustfmt to make lines as long as it needs to if there is no other way to format them, but otherwise respect max_width and the other heuristics.

The comments and strings do not seem to be the issue, since filter_normal_code is run before the width is checked.

is it not considered a bug for rustfmt to be unable to format code?

Not really sure what this question is in reference to, but the answer is no. If it's my removal of the GitHub label, that's because we use that label for specific purposes (the description of that label is Panic, non-idempotency, invalid code, etc. )
If it's in general, then still no. There's plenty of macro calls that rustfmt is unable to format either. In these cases it's really a matter of deferring to the developer to format things that cannot be formatted within the provided constraints.

The comments and strings do not seem to be the issue, since filter_normal_code is run before the width is checked.

Please read the details in the previously linked issue and review the specific problem lines I pointed you to, as well as the command I've provided that you can run directly to view this in action for yourself. You have a string literal that ends at column 106, all within a chain, hence the duplicate of #3863. Not really keen on debating nor repeating any of those items.


Configure allowedAudiences for openIdConnectProviders

Is it possible to configure for allowed audiences for openIdConnectProviders when using file based authentication?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ca023618-58e6-d10c-d736-5fb42a028ec9
Version Independent ID: 2b9d2673-c57f-c5b7-a2e3-ff04ff16a31d
Content: File-based configuration of AuthN/AuthZ - Azure App Service
Content Source: articles/app-service/configure-authentication-file-based.md
Service: app-service
GitHub Login: @cephalin
Microsoft Alias: cephalin

@DavidJFowler Thank you for your feedback! We will review and update as appropriate.

@DavidJFowler Thanks for reaching here. As its mentioned in the document that here that following exhausts possible configuration options within the file.
Check this document link for more details: https://docs.microsoft.com/en-us/azure/app-service/configure-authentication-provider-openid-connect
Also, this channel is reserved for ‘Azure docs’ questions- For sharing doc feedback and suggesting content improvements, it is recommend to post the technical questions on the technical forum1 and technical forum2 where one of our specialized engineers in the subject or the Azure community members can provide the best possible answer to your scenario or question.


Adds deontic

Closes #22
Deontic is added on top of the line 2-3 for each statement.

Hi Sander, Thanky you for your prompt reply. I have added a further commit to address your comments.
In relation to the formatting of the text I agree, however I would keep the deontic in gray as it is visible in the .png if the background is black (and even white). E.g.


add trigger and run scripts

trigger scripts trigger build/test/release runs
run scripts, either build or release, testing is done via normal prow tests

@rkpagadala: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/daily-perf-tests.sh
b2fb6f4afca6a61248474d1f0c6ff567f24b9d77
link
/test daily-perf-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@rkpagadala: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/daily-perf-tests.sh
b2fb6f4afca6a61248474d1f0c6ff567f24b9d77
link
/test daily-perf-tests

prow/daily-upgrade-tests-1.0.1.sh
b2fb6f4afca6a61248474d1f0c6ff567f24b9d77
link
/test daily-upgrade-tests-1.0.1

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@rkpagadala: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/daily-perf-tests.sh
b2fb6f4afca6a61248474d1f0c6ff567f24b9d77
link
/test daily-perf-tests

prow/daily-upgrade-tests-1.0.1.sh
b2fb6f4afca6a61248474d1f0c6ff567f24b9d77
link
/test daily-upgrade-tests-1.0.1

prow/daily-upgrade-tests-1.0.2.sh
b2fb6f4afca6a61248474d1f0c6ff567f24b9d77
link
/test daily-upgrade-tests-1.0.2

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


PseudoSimple check incorrectly reports Simple database as skipped

Bug Report
General Troubleshooting steps

[ YES] Verified running the latest release of dbachecks?

Does (Find-Module dbachecks).Version match (Get-Module dbachecks).Version.ToString()
YES

[YES] Verified errors are not related to permissions?
[ YES] Can duplicate in new/clean PowerShell session (clean = powershell -NoProfile)?

Simple databases should not be checked for PseudoSimple

This can be fixed with
                if (-not($psitem.RecoveryModel -eq "Simple")){                         It "$($psitem.Name) has PseudoSimple recovery model equal false on $($psitem.Parent.Name)" {                             (Test-DbaFullRecoveryModel -SqlInstance $psitem.Parent -Database $psitem.Name).ActualRecoveryModel -eq "SIMPLE" | Should -BeFalse -Because "PseudoSimple means that a FULL backup has not been taken and the database is still effectively in SIMPLE mode"                         }                     }
at https://github.com/sqlcollaborative/dbachecks/blob/development/checks/Database.Tests.ps1#L510

merged


Customize buttons/actions between Create and Edit Forms

This is  a eature request.

Prerequisites

[x] Are you running the latest version?
[x] Are you reporting to the correct repository?
[x] Did you check the documentation?
[x] Did you perform a cursory search?

Description
Hi, I need to have different buttons visible for Create/Edit forms for a specific resource. For example on Create I want to have only back and Save and for Edit I want to have also the Delete button.
Since the configurations are saved in the same .json file, for example product.json, how can I differentiate between them? Is there an easy way that I did not see?
Example:
Edit: "actions": ["store", "destroy", "back"],
Create: "actions": "actions": ["store", "back"],

You can use the actions setter in your form object and pass an array of actions.

Thanks for the quick reply, is there any example in Enso code that I can follow?

I think I have found it:
return $this->form->actions(['destroy', 'update'])     ->edit($calendar);
I was looking on the FE side all this time.
Thanks!


V0.10.0 working

This will be a huge change. And this PR is going to be the living document.
https://docs.google.com/document/d/1G8CAn-gOg9wlH8XDUZq-hhmkpOpA5sm6L0PFUhjj4xE/edit?usp=sharing
Many thanks to @owulveryck, @cfgt @jokebroker @ynqa  who have given valueable input. The document is incomplete but will be completed as this branch gets completed.  Feel free to ping me for edit access to add your inputs.

Ugh and this is why you don't refactor at 3am. values types and shapes should be moved to internal

Coverage increased (+0.4%) to 63.354% when pulling 623ee66e186149e822542164cb059c20a49fe0ea on v0.10.0-working into 292e90c69f29f502c989685b0e4a5cc8144362aa on master.


Create upvm.txt

UPVM file

@BRNEM First stage of verification passed, official domain is upvm.edomex.gob.mx as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision. Proof of domain ownership: https://upvm.edomex.gob.mx/

@BRNEM Thank you for your patience while waiting for a review and processing your request. Pull request merged.


More open on the development process

First I want to thank you all for a very cool project.
AFAIK, the development process takes place mostly in china and you have a discussing channel on QQ. Do you have any plan to using another form a channel or forum that is more friendly to foreign people like me so that I can take part in or just to update on the engine development ? Does the project has a long term plan or it will end at some points in time ?
Thanks and keep up the good work

Can you use WeChat?
I will keep on developing this project.

Thanks for your answer. I haven't used WeChat. The dicussion in in English ?

Yes.

I installed WeChat, I searched for stackos group but I didn't found it. How can I join the group ? Thanks

try to search my id v_britannia

Thanks, friend request sent :)


is flowControl.maxMessages option honored in 0.14.x to limit calls to handler

Environment details

OS: Linux (ubuntu)
Node.js version: 8.5.0
npm version:  5.3.0
google-cloud-node version: @google-cloud/pubsub 0.14.2

Steps to reproduce
Is it possible to get or limit the inflight messages using the new 0.14.x pubsub library?
I'm calling something like the following
this.subscription = this.topic.subscription(this.options.name, this.options.subscription);
Even if I get a topic.subscription with the following options it appears to call the my handler with as many messages in in the queue.
{
  flowControl: {
    maxMessages: 1
  }
}

This happens because the PubSub API doesn't allow us to limit how many results it sends us, so we use those numbers as an indicator to stop making requests to the API if they are hit/exceeded. Unfortunately in this case its easy to exceed.
@lukesneeringer should we only emit the specified number of messages listed in maxMessages and store the rest in memory while modifying the ack deadline along the way?

/cc @jganetsk

Thanks @callmehiphop seems like this is a behavior change from the 0.13 client where I could specifically do 1 thing at a time if I wanted. For my specific use case I only can do one thing at a time due to the intensity of the jobs I'm doing.

should we only emit the specified number of messages listed in maxMessages and store the rest in memory while modifying the ack deadline along the way?

We should not do this. The point of flow control is to prevent the process from dying because it ran out of memory. Storing messages in memory defeats that purpose, and it's not very useful to the user to have a message in memory but not to emit it.

Is it possible to get or limit the inflight messages using the new 0.14.x pubsub library?

It sounds like the user wants flow control to be on, not off. But it's supposed to be on by default, so is there a bug in flow control?

It sounds like the user wants flow control to be on, not off. But it's supposed to be on by default, so is there a bug in flow control?

Maybe? @jganetsk I'm a little lost on how we are supposed to process payloads with message sizes that exceed flow control limits. As it is today all the flowControl knobs do are control when we pause/resume connections. If we don't want overflow messages in memory should we be nacking them?

We have a lower bound on flow control limits, iirc, can you take a look at what those are? I think it's ok to overflow by 1 message (which means we can only overflow by up to 10MB, since that's the max message size in Pub/Sub), but beyond that we probably should nack any additional overflowing messages.

I don't think there are bounds on any of the flow control limits. We have defaults of

maxBytes - 20% of available memory
maxMessages - Infinity

But I do not believe that there were any bounds specified in the design doc.

I might have been thinking about limits on batching settings. Well regardless, my point about overflowing 1 message still holds and nacking otherwise still holds.

@jganetsk sounds good, thanks for the input!

@jganetsk thanks for taking the time to comment on this issue. I'm not sure I follow completely with what you and @callmehiphop were discussing so maybe it's going to be resolved.... I wanted to clarify what i wanted to happen. I want maxMessages on and set to emit 1 at a time. It doesn't appear to honor that.
Thank you again.

@evilpacket I just cut a new release (0.14.4) that should make maxMessages behave as you intend. Thanks for reporting this!

hello @callmehiphop,
I encounter the same issue than @evilpacket with the version 0.14.4.
My simple testing code is
const options = { flowControl: { maxMessages: 1 }, ackDeadlineSeconds: 20 };
const [subscription] = await this.pubsub.createSubscription('any-topic', 'my-sub', options);
subscription.on('message', (message) => {
      console.log(‘MSG ID’, new Date(), message.id);
      setTimeout(() => {
        console.log(‘acking’, new Date(), message.id);
        msg.ack();
      }, 18000);
    });
  });

The librairy seems to well respect the ackDeadlineSeconds but not the maxMessages, as the output is:

MSG ID 2017-10-09T15:45:19.934Z<PHONE_NUMBER>63484
MSG ID 2017-10-09T15:45:21.167Z<PHONE_NUMBER>04853   // wrong, we got 2 messages at the same time
acking 2017-10-09T15:45:37.939Z<PHONE_NUMBER>63484
MSG ID 2017-10-09T15:45:37.941Z<PHONE_NUMBER>72984
acking 2017-10-09T15:45:39.167Z<PHONE_NUMBER>04853
MSG ID 2017-10-09T15:45:39.168Z<PHONE_NUMBER>72984
acking 2017-10-09T15:45:55.946Z<PHONE_NUMBER>72984
MSG ID 2017-10-09T15:45:55.947Z<PHONE_NUMBER>68159
acking 2017-10-09T15:45:57.173Z<PHONE_NUMBER>72984
MSG ID 2017-10-09T15:45:57.174Z<PHONE_NUMBER>78813
MSG ID 2017-10-09T15:45:57.174Z<PHONE_NUMBER>78813   // wrong, we now have 3 unacked messages in our hands
acking 2017-10-09T15:46:13.951Z<PHONE_NUMBER>68159
acking 2017-10-09T15:46:15.176Z<PHONE_NUMBER>78813
acking 2017-10-09T15:46:15.177Z<PHONE_NUMBER>78813

Any idea about how we can get this "flow control" working as expected?
Thanks you!

@thomas-hilaire the latest release has allowed for a single message to be overflowed, meaning at any given time you could have maxMessages + 1 unacked messages, which would explain your logs.

Maybe we can overflow by 1 message on maxBytes, but I don't think we should overflow ever on maxMessages.

@jganetsk ah, I may have misunderstood.

Hello,

I just cut a new release (0.14.4) that should make maxMessages behave as you intend.

@callmehiphop by reading this I understood that you removed the overflow on maxMessages. For me, having multiple messages in memory is not a real problem. However, I expect to get them one by one, it is a common pattern that most programmers require from a pubsub.
If you plan to nack overflowed messages, I would like to know where a nacked message will go in the subscription queue, to the head, to the tail or somewhere between that (I don't find this information in the documentation)? I don't expect a strict ordering but we can't send to the tail every overflowed message. In this case I prefer to keep the message in memory by the librairy while the first message still non acked.
Thanks for your help!

@thomas-hilaire There are no ordering guarantees with pub/sub. If you nack a message, it will be redelivered sooner or later. Probably sooner. And for example, if you have multiple processes pulling in parallel (each with maxMessages = 1), the message might be redelivered to another process as soon as that process finishes with whatever one message it was processing.
But maxMessages = 1 is not necessarily a common pattern with users of Cloud Pub/Sub, who often demand high throughput, and therefore need to process many messages simultaneously. Are you sure maxMessages = 1 is necessary for your use case?

Thanks for your explanations, the nack behavior that you explained fits my needs.
The maxMessages = 1 helps me to implement the task-queue/worker pattern, I don't really need high throughput. I may code this behavior by myself in each micro-service I have, but having it directly in the library would be much more comfortable as I have a lot of them already in production.

@callmehiphop Can you remind me why nacks are necessary to solve this problem, for maxMessages in particular? As soon as numMessages == maxMessages, the library should "slam the brakes" and immediately suspend all connection streams. Because Node is single-threaded, I wouldn't expect any more message events to be emitted after this point. But maybe my assumption is wrong here?
Note that in our code review, I was very particular that we deal with maxMessages  differently from maxBytes and make sure we don't have this kind of off-by-one bug. The cases around maxMessages all used == and would take action immediately on the arrival of the message that reached the limit, while the maxBytes cases would accept messages as long as numBytes < maxBytes, with the possibility of going over the limit.

I think because the API sends more than what the user wants: https://github.com/GoogleCloudPlatform/google-cloud-node/issues/2638#issuecomment-333385590

@callmehiphop Actually, I might again be confusing flow control and batching. We need to be pretty careful about all these off-by-one bugs in both batching and flow control.

Hello, thanks for your update but I continue unfortunately to encounter the issue with the version 0.14.5 of the lib.
I did the same simple test than for my previous comment (https://github.com/GoogleCloudPlatform/google-cloud-node/issues/2638#issuecomment-335206769) and I continue to get more messages than maxMessages. My limit is 1, I often have 2 but I can even have 3 or more of them at the same time.
I also noticed a more important issue, the same message can be delivered many times!
Note that in my tests, messages are acked after having waited for 18 seconds and I don't encounter these issues when the wait time is around 10ms (the ack deadline is at 60 seconds).

GET 2017-10-16T09:41:37.099Z<PHONE_NUMBER>79698
ACK 2017-10-16T09:41:55.099Z<PHONE_NUMBER>79698

Thanks you!

@thomas-hilaire the redelivery of acked messages is something that the API team is aware of and I believe a fix is coming soon. I'll update this issue (and likely put out a new release) when the change goes live.

@callmehiphop was the redelivery of acked messages fixed? Cheers in advance!

@rhodgkins it should be as of 0.14.8. I believe we just released a new version (0.15) as well.

I continue to have the redelivery of acked messages with the release 0.15. It seems that more the ack call is delayed, more the redelivery issue occurs. I don't understand why it's so easy to reproduce for me and not for others, this is really an annoying issue (I need to track in a DB which message has been already processed).
Do we open a new issue about that or we reopen this one?
Else the 0.15 release is great and I confirm that the flowControl.maxMessages option is well honored!

@thomas-hilaire I'm getting the same issue with redelivery, I raised a separate issue if that helps - #2756!

@thomas-hilaire @rhodgkins I've moved your issue to https://github.com/googleapis/nodejs-pubsub/issues/2, so we can track it there.


Add -L option to curl

The example in the README won't follow the github redirect without -L, so the example as is will download an html file instead of the actual fleet.zip. This fixes that.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Vaidman Detection Test

Hi all,
This example is based one we prepared during a demonstration from Dr. Corcoles and Dr. McClure at a summer school this past June. It's a simple demonstration of interaction free measurement, based on the Vaidman bomb testing circuit, which Dr. Corcoles suggested. I've updated it to reflect changes in the API structure and added some explanatory text/figures to match the format of other examples in the tutorial.
//Alex

Hi,
I think this is a cool tutorial that exploits the difference in probability computation: quantum computing uses probability amplitudes that when used with intermediate measurement can result in quadratic difference.
I have some details comment:

The rotation R_y(a) is actually a rotation by a/2. That is R_y(a) |0> results in cos(a/2) |0> + sin(a/2) |1>. Would you mind fixing this? Also, when setting epsilon = pi/nsteps, after nsteps of iteration, for the case dud, it will be flipped by pi/2, which is sin(pi/2) = 1 instead of flipping by pi.

you can use numpy.pi instead of 3.14159 (QASM also defines pi this way)

@rraymondhp the rotation angle is defined correctly using our convention.

@jaygambetta  But, according to qelib.inc1, R_y(theta) |0> = cos(theta/2) |0> + sin(theta/2) |1> ?
See Eq.2 of the qasm2.pdf here. That means, strictly speaking, after a rotation R_y(epsilon), the state will be approximately |0> + epsilon/2 |1> instead of |0> + epsilon |1> .
https://github.com/QISKit/openqasm/blob/master/spec/qasm2.pdf

I am not disagreeing with the math. I am disagreeing with the interpretation. If you map this to a probability or a Bloch sphere then putting a pi rotation gives you the excited state [sin(pi/2) = 1]. It is standard convention to call a pi pulse a pulse that makes 1 and a pi/2 that makes 0+1. This is a common (tiny) mistake often made and it depends if your definitions. We are defining a rotation by theta this way.

Thanks for the clarification. My comment is on the math, and I have corrected/omitted the rotation term.

I've updated the pull request to remove the custom ket command and correctly use numpy.pi. While I'm not intimately familiar with the angle definitions, I believe this is correct based on some testing.

@sabreitweiser  thanks for the swift response. I have summarized the matrices for gates available at QASM at the tutorial below. For small x, sin(x) approx x, and thus my comment.
https://nbviewer.jupyter.org/github/QISKit/qiskit-tutorial/blob/master/1_introduction/quantum_gates_and_linear_algebra.ipynb

@sabreitweiser  Also, please also update index.ipynb to link your tutorial and add your name.

Ah, I see now - the rotation is correct but my approximation is wrong. I've updated it accordingly, as well as adding the example and my name to index.ipynb.


Linux build fails

When trying to do a clean build (./runme.sh clean) building fails.
[160/312] Performing download step (git clone) for 'Freetype'
Cloning into 'Freetype'...
error: RPC failed; curl 92 HTTP/2 stream 0 was not closed cleanly: CANCEL (err 8)
error: 7754 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
Cloning into 'Freetype'...
error: RPC failed; curl 92 HTTP/2 stream 0 was not closed cleanly: CANCEL (err 8)
error: 8081 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
Cloning into 'Freetype'...
-- Had to git clone more than once: 3 times.

or dozens of
./fltk14/fltk_wrap.cpp:176860:13: error: invalid use of incomplete type ‘class Fl_PostScript_Graphics_Driver’
All dependencies are installed and updated.

For your first error, your network is failing (ie. git clone is not able to access the curl repository).  There have been intermittent errors since yesterday.
For your second error, there's an inconsistency between pyFLTK and FLTK and cmake seems not to be smart enough to detect the change.   You many need to clean the dir and start the build from scratch.
Try running:
git pull
./runme.sh clean

Also, note that the build now requires "meson" to be installed.

BTW:
You can see the latest build status at the top of the main page.  It says: "Full Builds: passing".  That status is from yesterday.
You can further verify the build is working in the Actions tab.  I just started a new build both locally (on Rocky Linux 8.9) and remotely (on Ubuntu 20.04) to verify all is working, just in case.  So far, everything is working.

Both the Rocky Linux 8.9 build finished successfully (I had to do./runme.shtwice as I also got the network error).
And the Ubuntu 20.04 remotely also finished successfully.

OK, thanks for your efforts. I'll give it another shot today and report back.

What's your swig version and Linux distro and version?
I can verify these versions work:
swig -version

SWIG Version 4.0.2

swig -version

SWIG Version 4.2.0

I think you may have either too old or too new swig version.

If you don't care about pyFLTK, you can disable it from the build with:
./runme.sh -D MRV2_PYFLTK=OFF
It is only used for the plug-in that allows you to update mrv2 from the menus (you might want to then move BUILD-Linux-amd64/Release/install/python/plug-ins/update-mrv2.py to another directory).

Mine is
SWIG Version 4.0.2

Compiled with g++ [x86_64-pc-linux-gnu]

Configured options: +pcre

And my OS is Linux Mint 21.3 x86_64

Still, I am puzzled, so I am going to bother you a bit more, if you don't mind.
Can you verify that:
git log
reports:
commit 15d1a00c598373a8f8fd30c955ee4da266aef70c (origin/main, origin/HEAD)

You're very welcome to bother me ;)
git log
gives me
commit 8cc996dc8fc7885ce135474800020b8143e69922 (HEAD -> main, origin/main, origin/beta, origin/HEAD)
Author: Gonzalo Garramuño<EMAIL_ADDRESS>Date:   Tue Feb 13 04:10:23 2024 -0300

    Added check for extension (format) for VP9 and AV1 codecs.

Ok.  You are a tad behind some changes I did to pyFLTK.  It should not have changed anything, but maybe I had screwed up...
git pull
./runme.sh

Don't clean the dir, it won't do anything to your already compiled mrv2 as pyFLTK is independent (well, it needs Python and FLTK, but you already compiled those, so it should be a relatively fast compile if it works).

Oh da**!  I know what happened.  I forgot to merge my "beta" branch into the "main" branch.  Try:
git pull
./runme.sh

and I swear it will work this time around!

Cool! I'm giving this a shot right now. Hang on...

Also, in case it fails, go to and see that it reads like this:

cd BUILD-Linux-amd64/Release/FLTK-prefix/src/FLTK
git log

commit a1d3bf182e6c085779b6e7e7315bf15f384f7112 (HEAD)
Author: Albrecht Schlosser<EMAIL_ADDRESS>Date:   Wed Feb 7 20:03:29 2024 +0100

    Fix GitHub CI build and minor CMake issues

    - GitHub CI: fix wrong CMake option name
    - fl_add_library: disable debug output (MSVC), fix formatting

It actually still fails with the same fltk error, but it gives me exactly this
steffen@V16m:~/mrv2$ cd BUILD-Linux-amd64/Release/FLTK-prefix/src/FLTK
steffen@V16m:~/mrv2/BUILD-Linux-amd64/Release/FLTK-prefix/src/FLTK$ git log
commit a1d3bf182e6c085779b6e7e7315bf15f384f7112 (HEAD)
Author: Albrecht Schlosser<EMAIL_ADDRESS>Date:   Wed Feb 7 20:03:29 2024 +0100

    Fix GitHub CI build and minor CMake issues

    - GitHub CI: fix wrong CMake option name
    - fl_add_library: disable debug output (MSVC), fix formatting

I'll be... What does this directory give you?
cd BUILD-Linux-amd64/Release/pyFLTK-prefix/src/pyFLTK
svn info

Working Copy Root Path: /media/gga/datos/code/applications/mrv2/BUILD-Linux-amd64/Release/pyFLTK-prefix/src/pyFLTK
URL: https://svn.code.sf.net/p/pyfltk/code/branches/fltk1.4
Relative URL: ^/branches/fltk1.4
Repository Root: https://svn.code.sf.net/p/pyfltk/code
Repository UUID: db8d5ec9-520e-0410-b818-d5ed6a5c6bea
Revision: 627
Node Kind: directory
Schedule: normal
Last Changed Author: andreasheld
Last Changed Rev: 627
Last Changed Date: 2024-01-27 18:10:00 -0300 (sáb, 27 ene 2024)

Hmm...
svn: E155007: '/home/steffen/mrv2/BUILD-Linux-amd64/Release/FLTK-prefix/src/FLTK' is not a working copy

svn: E155007: '/home/steffen/mrv2/BUILD-Linux-amd64/Release/FLTK-prefix/src/FLTK' is not a working copy

pyFLTK not FLTK.

FLTK - same version
pyFLTK - same version
swig - same version
One more thing to check:
g++ --version
and attach your latest compile.log (the one where pyFLTK fails):
BUILD-Linux-amd64/Release/compile.log

Thanks for your patience (I'll have to walk the dog right now, so I'll be AFK ~1h soon):
steffen@V16m:~/mrv2$ g++ --version
g++ (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

and I attached the compile.log here. compile.log

Thanks for your patience

No... Thanks to YOUR patience!
Ok.  After asking chatgpt and poking around, it seems the problem we are facing is a new "feature" of gcc/g++ 12.  I can try one more thing.
Attach these files:
BUILD-Linux-amd64/Release/pyFLTK-prefix/src/pyFLTK/fltk14/fltk_wrap.cpp
BUILD-Linux-amd64/Release/pyFLTK-prefix/src/pyFLTK/fltk14/fltk_wrap.h

I'll try to compile them with my older compiler and see if they compile without errors.

So just to confirm: I can't help you with this right now and you're currently trying to compile the fixes for a newer compiler with an older compiler (gcc 11?) and see if the old version still works?

No.  Never mind.  It is not a compiler issue, as I was able to check it another way.  I'm going to have you try again (this is the last one and then I give up).  Do the following:
rm -rf BUILD-Linux-amd64/Release/FLTK-prefix
rm -rf BUILD-Linux-amd64/Release/pyFLTK-prefix
git switch main
git pull
./runme.sh

I am having remove the directories in case there was something left behind that was incomplete.

OK, here I am again. Sorry for the late reply.
I just tried to rebuild and it still failed. I attached the compile.log (
compile.log ) but I don't have the pyFLTK-prefix folder / files:

I was working on the build scripts and I screwed up the download process.  That's why you did not get any pyFLTK-prefix files.  I am done testing it now and adding more debugging info.  Try it again:
rm -rf BUILD-Linux-amd64/Release/pyFLTK-prefix
git switch main
git pull
./runme.sh

Send me the compile.log if it still does not work.

OK, I rebuilt the latest main, it still fails but at least I have some files in the pyFLTK-prefix folder.
Here's the compile.log
And the 3 files from the pyFLTK-prefix folder:
fltk14.zip

OK, I rebuilt the latest main, it still fails but at least I have some files in the pyFLTK-prefix folder. Here's the compile.log
And the 3 files from the pyFLTK-prefix folder: fltk14.zip

Thanks for your patience.  Try it again, I think the issue is that you have an older version of FLTK installed in /usr/include and that's causing the conflict.  The swig script was re-arranging the paths in the wrong order and putting /usr/include before the include/ in the build location.
cd mrv2
git switch main
git pull
./runme.sh

Great! I just compiled from scratch and all is shiny again.
Thanks a lot!

Great! I just compiled from scratch and all is shiny again.
Thanks a lot!

Most excellent!!! It was a tricky one to fix.  I am closing the issue now.


Error: Error loading shared library libfontconfig.so.1: No such file or directory (needed by /app/node_modules/skia-canvas/lib/v6/index.node)

It was working well on local, but start getting error while deployed on Render
Node
FROM node:21.5.0-alpine
Shell uname
Linux srv-cir6bq5gkuvqadosit5g-7c9bc6977f-z6pcl 5.15.0-1032-gke #37-Ubuntu SMP Wed Apr 19 21:06:04 UTC 2023 x86_64 Linux
I have tried
apk add fontconfig
apk --no-cache add fontconfig
npm i skia-canvas in the project

I have checked  the skia-canvas doc, which a shared library of of @loskir/styled-qr-code-node
and found that to use skia-canvas you need to add fontconfig script while using docker
https://www.npmjs.com/package/skia-canvas?activeTab=readme
Thank you 😊.


Typescript interface at odds with README.md

Awesome tool. I'm excited to try it.
The readme indicates that you can call migrate with only two arguments. But, the typescript interface for 3.0.2 is
  export function migrate(
    dbConfig: MigrateDBConfig,
    migrationsDirectory: string,
    config: Config,
  ): Promise<Migration[]>

If you want typescript users to use migrate as indicated in the docs, you could make config optional, config?: Config.

Thanks, I reckon the docs have needed a once-over since the move to TypeScript. I'll have a look through and fix them up.

Should be solved with the new TypeScript changes and README updates.


What's the point of "Tray icon removed" message?

When I run taffybar from the terminal I see a lot of those because of https://github.com/travitch/taffybar/blob/97b4d2f6feffcab8e5d73523d7b6148b64177e31/src/System/Taffybar/Systray.hs#L21
@travitch is there any motivation for that message to appear in console?

It was to help me get the system tray working

Sorry, do you mean that without output to stdout system tray doesn't work?

No, I just mean that I never got around to removing the output.


Group jobs resource requirements not set in Snakemake runner

Snakemake version
5.8.2
Describe the bug
When grouping jobs that can run in parallel on the cluster, the grouped job will not set allocated resources in the Snakemake workflow leading to an overuse of resources.
Logs
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 64
Rules claiming more threads will be scaled down.
Job counts:
	count	jobs
	1	test_final
	1	test_temp
	1	test_temp2
	3

[Mon Dec 16 09:01:40 2019]
rule test_temp:
    output: results/test_temp_g1.txt
    log: logs/test_temp_g1.log
    jobid: 1
    wildcards: group=g1
    threads: 2
    resources: mem_mb=8000

        echo "bob2" > results/test_temp_g1.txt && sleep 10

[Mon Dec 16 09:01:40 2019]
rule test_temp2:
    output: results/test_temp2_g1.txt
    log: logs/test_temp2_g1.log
    jobid: 2
    wildcards: group=g1
    threads: 4
    resources: mem_mb=4000

        echo "bob" > results/test_temp2_g1.txt && sleep 10

[Mon Dec 16 09:01:50 2019]
Finished job 1.
1 of 3 steps (33%) done
[Mon Dec 16 09:01:50 2019]
Finished job 2.
2 of 3 steps (67%) done

[Mon Dec 16 09:01:50 2019]
rule test_final:
    input: results/test_temp_g1.txt, results/test_temp2_g1.txt
    output: results/test_final_g1.txt
    log: logs/test_final_g1.log
    jobid: 0
    wildcards: group=g1

        cat results/test_temp_g1.txt results/test_temp2_g1.txt > results/test_final_g1.txt

[Mon Dec 16 09:01:50 2019]
Finished job 0.
3 of 3 steps (100%) done

Minimal example

rule test_temp:
    output:
        "results/test_temp_{group}.txt"
    group: "test_{group}"
    threads: 2
    resources:
        mem_mb=8000
    log: "logs/test_temp_{group}.log"
    shell:
        '''
        echo "bob2" > {output} && sleep 10
        '''

rule test_temp2:
    output:
        "results/test_temp2_{group}.txt"
    group: "test_{group}"
    threads: 4
    resources:
        mem_mb=4000
    log: "logs/test_temp2_{group}.log"
    shell:
        '''
        echo "bob" > {output} && sleep 10
        '''

rule test_final:
    input:
        f1="results/test_temp_{group}.txt",
        f2="results/test_temp2_{group}.txt"
    output:
        f3="results/test_final_{group}.txt"
    group: "test_{group}"
    log: "logs/test_final_{group}.log"
    shell:
        '''
        cat {input.f1} {input.f2} > {output.f3}
        '''

Additional context
In this example, all jobs are grouped but the first two jobs can run in parallel with appropriate resources.  When submitted using the LSF profile (command: snakemake --profile lsf -j 1 --snakefile test_groups.smk results/test_final_g1.txt), it submits a job requesting 4 cores and 8000 MB of memory.  Instead of running test_temp and test_temp2 sequentially, it runs them in parallel (see log start/end times).  I believe this is Snakemake not recognizing resources limits due to the first couple lines in the output file, specifically:
Provided cores: 64
Rules claiming more threads will be scaled down.

One additional note: if I set the threads to a high value (for example, 48 cores on this 64 core machine), then I can confirm the jobs run sequentially as intended:
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 64
Rules claiming more threads will be scaled down.
Job counts:
	count	jobs
	1	test_final
	1	test_temp
	1	test_temp2
	3

[Mon Dec 16 09:18:04 2019]
rule test_temp:
    output: results/test_temp_g1.txt
    log: logs/test_temp_g1.log
    jobid: 0
    wildcards: group=g1
    threads: 48
    resources: mem_mb=8000

        echo "bob2" > results/test_temp_g1.txt && sleep 10

[Mon Dec 16 09:18:14 2019]
Finished job 0.
1 of 3 steps (33%) done

[Mon Dec 16 09:18:14 2019]
rule test_temp2:
    output: results/test_temp2_g1.txt
    log: logs/test_temp2_g1.log
    jobid: 1
    wildcards: group=g1
    threads: 48
    resources: mem_mb=4000

        echo "bob" > results/test_temp2_g1.txt && sleep 10

[Mon Dec 16 09:18:24 2019]
Finished job 1.
2 of 3 steps (67%) done

[Mon Dec 16 09:18:24 2019]
rule test_final:
    input: results/test_temp_g1.txt, results/test_temp2_g1.txt
    output: results/test_final_g1.txt
    log: logs/test_final_g1.log
    jobid: 2
    wildcards: group=g1

        cat results/test_temp_g1.txt results/test_temp2_g1.txt > results/test_final_g1.txt

[Mon Dec 16 09:18:24 2019]
Finished job 2.
3 of 3 steps (100%) done

Interesting.  I didn't know you could use wildcards in group names!
Can you confirm a few things:

Is your job getting all 64 cores from the scheduler or just the (presumably) 4 requested?
If no threads is set, what's the default?  Similarly, if you set threads on the last rule does this get resolved?
Can you set the submission command through --cluster and observe the same behavior?  There's a 'qsub exe' in the tests folder of this repo you can use to test.  Something like --cluster echo {threads} {mem_mb} ; ./qsub.

I'm trying to assess if it's a snakemake problem, the profile, or the cluster.

I can definitely confirm that it's only requesting the 4 (which is what I would expected). Here's a small snippet of the LSF job info:

...
                     >, Error File <logs/cluster/test_g1_809770ec.err>, 4 Proce
                     ssors Requested, Requested Resources <select[mem>8000] rus
                     age[mem=8000] span[hosts=1]>;
...

If I remove threads from all jobs, it just sets it to 1 (also as expected).  If I set the last rule to "4", I basically get identical performance as before (requests 4 processes to the cluster, but both of the jobs run in parallel utilizing 6 processes before the test_final job runs.
If I switch to using --cluster, I get the same issue as before (both jobs run in parallel & overuse resources).  Here's the command I used for the purpose of testing it:
snakemake --cluster "bsub -o logs/cluster/test.log -n {threads} -M {resources.mem_mb} -R \"span[hosts=1]\"" -j 1 --snakefile test_groups.smk results/test_final_g1.txt

Happy to try more things if you've got ideas.  I think the profile and cluster is operating correctly, but I'm fairly new to using profiles.  Historically, I've manually created the --cluster command, so it's possibly a profile issue that I'm not familiar with.

If you are requesting 4 cores it means snakemake is inferring the correct number of threads for the group, but then in the allocated node is detecting all cores on the machine instead of those given to it.  I'm having trouble determining where it picks up all the cores.
For non-group jobs, do you see the same message Provided cores: 64?

Yes, I do actually.  I commented out the group lines in the above.  Here's one of the outputs:
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 64
Rules claiming more threads will be scaled down.
Job counts:
	count	jobs
	1	test_temp2
	1

[Mon Dec 16 12:42:23 2019]
rule test_temp2:
    output: results/test_temp2_g1.txt
    log: logs/test_temp2_g1.log
    jobid: 0
    wildcards: group=g1
    threads: 4
    resources: mem_mb=4000

        echo "bob" > results/test_temp2_g1.txt && sleep 10

[Mon Dec 16 12:42:33 2019]
Finished job 0.
1 of 1 steps (100%) done

Looking through a bunch of unrelated logs from real pipelines, that number appears to always match the total number of cores on the machine it was assigned to.

Alright.  Can you check .snakemake/tmp* for any shell scripts?  They are usually cleaned up but some stick around if the workflow fails.  If so, just confirm that -j is an argument without a specified number of cores; that's how mine all look though in the code there is an option to supply a value.  Then it seems to be an issue with available_cpu_count.
Can you start an interactive job with like 2 cores and walk through some of that code to see how/where it's failing?  Is this a recent issue or seems to be consistent?  Also, if you need a quick fix you can have the second job depend on output of the first to force sequential execution
Maybe a better name for this issue is 'available_cpu_count incorrect in LSF cluster job'.

I caught one running and it appears to not specify a value as you predicted: ... --force -j --keep-target-files ...
I'm not entirely sure what you're wanting me to do with the interactive job.  Are you talking about trying to run the available_cpu_count() code manually to see why it's returning a high number or are you wanting me to try running Snakemake somehow?  As for recent or consistent, I'm not entirely sure.  I only recently had a use case for groups which is why I'm encountering it now.  However, all of the logs I've spot checked have the total number of cores in the log file.
Yea, my quick fix is just to set the threads to a high value (it works for the particular use case I have).
changed title of issue

Yeah, start an interactive job with 2 cpus and in the shell you can check the contents of /proc/self/status to see if Cpus_allowed: is in it and copy the line.  Then you can step through the code with an interactive python session or a couple of python -c {CODE} commands:
python -c 'print(bin(int({LINE}.replace(",", ""), 16)).count("1"))'  # LINE from /proc/self/status
python -c 'import multiprocessing ; print(multiprocessing.cpu_count())'

In my case, the first was set properly but the second was the total cores in the node.

Interesting, looks like it's not fixing it to particular cores which is making it think it has full access to the node:
...
Cpus_allowed:	ffffffff,ffffffff
Cpus_allowed_list:	0-63
...

I'm assuming this isn't a Snakemake issue then, but rather a lack of constraint on the job being created by our LSF instance right?

Seems like it.  I would ask your sys admin about this.  Hopefully it's a quick fix!

Any chance you know of a workaround via snakemake command line or otherwise?  If not, I'll mark it as closed (and thanks for the help debugging!)

Not without changing some other snakemake source code where it builds the submission.
I would add the output of the first rule as the input of the second rule.  You will then request the right number of cores in your job but the first rule will be executed first.  You can specify inputs without using their values.
Hope that helps!  I've been meaning to work out how snakemake gets the cores during submission, so this was a little extra motivation.

Yea, I could see that working if my use case matched the example.  In my real example, I basically have multiple copies of rule A feeding into a pipeline B, C, D...: 10xA -> B -> C -> D -> ....  There might be a way to order them if I can get creative, but it's non-obvious.
Anyways, thanks for the help and sorry for the false bug!


How do I play pokemon?

I have gone through numerous save files and none have worked, can I have some help?

How can I play Super Mario World if possible?

I have tried numerous download sites and none of them worked.

does anybody know how to play zelda minish cap on the gametime player?

does anyone have a pokemon leaf green file i can have that is OP

Is it normal for X and Y to take so long to load