gixy: 0.1.9 -> 0.1.20

Motivation for this change
latest release
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Fits CONTRIBUTING.md.

No attempt on x86_64-darwin (full log)
The following builds were skipped because they don't evaluate on x86_64-darwin: gixy
Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

Success on x86_64-linux (full log)
Attempted: gixy
Partial log (click to expand)
tests.plugins.test_simply.test_from_config('ssrf', '/build/source/tests/plugins/simply/ssrf/not_host_var_fp.conf', {u'severity': u'HIGH'}) ... ok
tests.plugins.test_simply.test_from_config('valid_referers', '/build/source/tests/plugins/simply/valid_referers/none_first.conf', {u'severity': u'HIGH'}) ... ok
tests.plugins.test_simply.test_from_config('valid_referers', '/build/source/tests/plugins/simply/valid_referers/none_middle.conf', {u'severity': u'HIGH'}) ... ok
tests.plugins.test_simply.test_from_config('valid_referers', '/build/source/tests/plugins/simply/valid_referers/wo_none_fp.conf', {u'severity': u'HIGH'}) ... ok
tests.plugins.test_simply.test_from_config('valid_referers', '/build/source/tests/plugins/simply/valid_referers/none_last.conf', {u'severity': u'HIGH'}) ... ok

----------------------------------------------------------------------
Ran 296 tests in 1.056s

OK

Success on aarch64-linux (full log)
Attempted: gixy
Partial log (click to expand)
tests.plugins.test_simply.test_from_config('add_header_multiline', '/build/source/tests/plugins/simply/add_header_multiline/more_set_headers_fp.conf', {u'severity': u'LOW'}) ... ok
tests.plugins.test_simply.test_from_config('add_header_multiline', '/build/source/tests/plugins/simply/add_header_multiline/more_set_headers.conf', {u'severity': u'LOW'}) ... ok
tests.plugins.test_simply.test_from_config('add_header_multiline', '/build/source/tests/plugins/simply/add_header_multiline/add_header_fp.conf', {u'severity': u'LOW'}) ... ok
tests.plugins.test_simply.test_from_config('add_header_multiline', '/build/source/tests/plugins/simply/add_header_multiline/add_header.conf', {u'severity': u'LOW'}) ... ok

----------------------------------------------------------------------
Ran 296 tests in 1.992s

OK
/nix/store/v4nb0g2r11ng3hl12b8jyjh83k83y76x-gixy-0.1.20


Gazebo: Failed to create robot simulation interface loader

Hello,
I am currently following the gazebo tutorials in the Franka Control Interface documentation.
I installed franka-ros from the ROS repositories with the following command:
sudo apt install ros-noetic-libfranka ros-noetic-franka-ros
The franka-gazebo package was missing so I installed myself.
When I now try to run:
roslaunch franka_gazebo panda.launch
I get the following error:

[ INFO] [1641851266.573617743]: gazebo_ros_control plugin is waiting for model URDF in parameter [/panda/robot_description] on the ROS param server.
[FATAL] [1641851266.676289907]: Failed to create robot simulation interface loader: Could not find library corresponding to plugin franka_gazebo/FrankaHWSim. Make sure the plugin description XML file has the correct name of the library and that the library actually exists.

Thank you for your help!

Seems like an error on our side.
As a quick fix I would suggest you to remove franka-ros completely with
sudo apt remove ros-noetic-franka-ros
and clone the github repo and build everything from source.

Hi,
I noticed the exact same error also with Melodic. But the source install seems to quickly fix this problem.

Probably related to https://github.com/frankaemika/franka_ros/issues/175. If so, this is already fixed on develop. Once we release the next version, this will be fixed on apt as well

Hi @hennking98, we recently released franka_ros 0.8.2 which contains the fix for your issue. So now also sudo apt update && sudo apt upgrade should fix your issue. The new version is already on Noetic, for Melodic it might take a while to go to upstream.
If problem persists, feel free  to reopen


[FLINK-14186][e2e] Skip Elastic Search 2.3.5 case when running with JDK11

What is the purpose of the change
This PR fixes the end-to-end test failure when testing elastic search link with ES 2.3.5 on JDK 11. Since Java has removed some GC options with the change on the GC algorithms, ES 2.3.5 will fail to start up with JDK 11 since it have included some removed GC options by default. Therefore, we should skip this case when testing with JDK 11.
Brief change log

18531c05a028f98d44a072705392b3c966493416 skips the Elastic Search 2.3.5 case when running with JDK11.

Verifying this change
This change is tested by manually run run-nightly-tests.sh under flink-end-to-end-tests.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature?no
If yes, how is the feature documented? not applicable

One more question. Is JAVA_OPTS="-XX:+IgnoreUnrecognizedVMOptions" a common pattern during we migrate to 10 or later? I am thinking of make it a high level option that could be referred to.

I would treat this as a separate issue.

We could use if [[ ${PROFILE} != *"jdk11"* ]]; then to filter out those tests.

Hi @tillrohrmann , we have also thought of remove this test from java 11 initially, but one concern is that if some users maintains one ES 2.3.5 cluster running on JDK 8 and also one Flink cluster running on JDK 11 to interact with that ES cluster, do we need also to cover this scenario ?

I think we do not have to cover this scenario atm. Testing all different Java versions times the number of external systems easily leads to a combinatorial explosion. The existing test also does not test this scenario since it runs ES with Java 11 anyway.

Got it, I am also agree with remove this test now. I will update this PR and the Jira.

Hi @tillrohrmann , I have updated the PR to skip the ES 2.3.5 when running with JDK11. I have manually triggered the tests  on Elastic Search and the ES 2.3.5 case was skipped indeed (The test script is modified to only run with Elastic Search cases, or the test will exceed 50 min and will be terminated by Travis).
Very thanks @tillrohrmann and @TisonKun for the review and suggestions! :)

Failing test case seems to be unrelated.


Shell and whitespace in file names

$ git rm $(git ls-files -d) is quite dangerous. You might accidentally delete other files if one of the files deleted in git contains whitespace in the file name.
Better:
$ git ls-files -d | while read f; do rm "$f"; done

That'll work better (still has issues with files containing new line characters but that's a lot less likely to happen).

Could you put this in a PR?


Repair stat reading

Ensures keys are defined and are numerical values, will improve Node memory reading

Coverage increased (+0.2%) to 89.787% when pulling 5d39d63c9e8043de10a34b780b916abb7b6aa572 on pselle:status_fix into 7d09b31ac38f7b913c4057f197a63c74e2abcb12 on iopipe:master.

Coverage increased (+0.2%) to 89.787% when pulling 51277925f266b6b68870c67f4e2dabfb02a9f138 on pselle:status_fix into 7d09b31ac38f7b913c4057f197a63c74e2abcb12 on iopipe:master.


No toggle between switching(on/off) auto execution during runtime

Based on https://github.com/hyperledger/aries-framework-go/pull/240#discussion_r324797346

This has been addressed as part of the same PR - https://github.com/hyperledger/aries-framework-go/pull/240#discussion_r325328752


Add categories to Cargo.toml

oops, didn't see #76.

Thanks @tmccombs, looks like we've merged #76.


fix accodion conditional

fixes issue #697 open only relevant accordions to the current user in the profile page

It works. Good job!


Problems with Variable Products

Hi,
We have problems editing variable products after last woocommerce update.
Product data indicates "Simple Product" but the product is variable.

Can you reproduce this issue on default Wordpress theme (eg Storefront)?
Can you reproduce this issue when all other plugins are disabled except WooCommerce, Polylang and Hyyan WooCommerce Polylang Integration?
What product versions and settings are you using when this issue occurs?

PHP:
WordPress:
WooCommerce:
Polylang: [state if using Polylang PRO]
Hyyan WooCommerce Polylang Integration:
Browser:

Steps to Reproduce

What I Expected
What Happened Instead
WordPress Environment

```
Copy and paste the system status report from **WooCommerce > System Status** in WordPress admin here.
```

I have the same problem, if I copy from primary languages, i show only simple products, without the possibility to copy variation. How can I solve this?

Can you please use the code from the repository, download zip directly from download code and install, i think it might be solved, i will be testing this a lot next week, but every feedback is welcome.

After downloading zip and installing from GitHub I have 2 problems:

If I copy a product from the primary language into the secondary it creates many variations, (example if I have a product with 2 colors, the second language creates 10 variations from the color)
After a customer's order, the product becomes out of stock even if there are still stocks.

1. If I copy a product from the primary language into the secondary it creates many variations, (example if I have a product with 2 colors, the second language creates 10 variations from the color)

I'm having the same problem with the GitHub version of the plugin

It seems that the latest PR #518 has screwed something related to variable products.

Ok i read topic, now the problem is a misspelling in a meta attribute. Can we solve with a update query and fix the name in src/Hyyan/WPI/Plugin.php?
@mrleemon  i saw you are a collaborator in this project, if you give me some guidelines on this error I can try a solution and make it available to the community.

Sorry, I only commuted some fixes for typos long time ago. My knowledge of the inner workings of this plugin is near zero.
I don't know what has to be done to fix this variations bug.

Right now, I think that the only one knowing how this plugin works is @Jon007

I did see that #518 included this typo fix which I already rejected on #450 as it would be likely to break existing sites, but @hyyan has accepted it...

I reverted #518 for now.  And I will take a deeper look later

Thanks!

Maybe the plugin should use the official Polylang pll_copy_taxonomies filter to sync WooCommerce taxonomies (product_type, product_visibility and others) instead of the current custom way, which seems vulnerable to the constant WooCommerce releases. With this filter, one can specify the taxonomies one wants to sync or copy and Polylang takes care of the synchronization/copy of said taxonomies and their related terms when one creates a new translation.

I looked at that but all these are getting dangerously obsolete, as woocommerce moves more towards its own api and its own tables you can't rely on treating products as a post and using generic post based apis.  At best there'll be bugs with the woocommerce caching mechanism.

Yes, I know that. I was thinking about it as a temporary solution while we wait for WC api changes.
Meanwhile, I think we can solve the incorrect selection of "Simple Product" when the product is variable changing the JS code in Meta.php to:
From:
$code = sprintf(
    '// <![CDATA[ %1$s'
    . ' addLoadEvent(function () { %1$s'
    . '  jQuery("#product-type option")'
    . '     .removeAttr("selected");%1$s'
    . '  jQuery("#product-type option[value=\"%2$s\"]")'
    . '    .attr("selected", "selected");%1$s'
    . '})'
    . '// ]]>', PHP_EOL, $type[0]
);

To:
$code = sprintf(
    '// <![CDATA[ %1$s'
    . ' addLoadEvent(function () { %1$s'
    . '  jQuery("#product-type option")'
    . '     .prop("selected", false);%1$s'
    . '  jQuery("#product-type option[value=\"%2$s\"]")'
    . '     .prop("selected", true);%1$s'
    . '})'
    . '// ]]>', PHP_EOL, $type[0]
);

Apparently, the use of attr() and removeAttr() for selecting/deselecting options is deprecated with the jQuery recent changes in WP.

From :
Breaking change: .removeAttr() no longer sets properties to false
Prior to jQuery 3.0, using .removeAttr() on a boolean attribute such as checked, selected, or readonly would also set the corresponding named property to false. This behavior was required for ancient versions of Internet Explorer but is not correct for modern browsers because the attribute represents the initial value and the property represents the current (dynamic) value.

It is almost always a mistake to use .removeAttr( "checked" ) on a DOM element. The only time it might be useful is if the DOM is later going to be serialized back to an HTML string. In all other cases, .prop( "checked", false ) should be used instead.

Also, right now, when the user clicks WC duplicate product link, the product type is not copied over the replica.
I "fixed" it by adding this to the unlinkOrginalProductTranslations() function in Duplicator.php:
$type = $product->get_type();
update_post_meta($duplicate->get_id(), '_translation_porduct_type', $type);

I don't know if there's anything wrong with this.

I'd like to get rid of '_translation_porduct_type' altogether, it shouldn't really be necessary, it's only there to cope with user interface quirk, however the simplest for now is to keep it and do as you suggest

I was going to ask you about this. Why is the translated product type stored in _translation_porduct_type? Can't it be retrieved from the original product directly?

I think it's complicated because when translating polylang is there trying to give you the translated product details when you ask for the old product, at least, copying it across in the metadata is easier

I think that in order to get rid of  the _translation_porduct_type meta, the plugin should use the WC API WC_Product:save() function instead of relying in a custom solution using the wp_insert_post action directly, but this means refactoring much of the code found in Meta.php and I don't know where to begin.

Hi. Have you solved this problem?

Hi, I committed some solutions to the Variable products issues including some explanation on #430 - it would be great if someone could test the latest code and confirm what issues remain - ideally on a new clean github issue, as there have been many partially or wholly duplicated tickets and getting hard to follow.

Thanks for your work! I'll test the latest code when I get a chance and I'll let you know if any issues arise.

I opened an issue after testing your latest code:
https://github.com/hyyan/woo-poly-integration/issues/526

@hyyan @mrleemon I checked in some important changes last week and labelled 5.1
In particular I have removed the workaround fix by @mrleemon on #408 by commenting out the meta.php call to:
$this->syncSelectedproductType( $ID );
Previously we had this problem because the save sequence was not correct at least in 5.0 it was:

WordPress saves post
woopoly synchronises post (but WooCommerce is not yet saved so Product Type is not saved and synchronised correctly)
WooCommerce saves product
[for variable product, WooCommerce saves variations and woopoly synchronises these]
When viewing product translation, the workaround fix  reset the form to show the product type in the _translation_porduct_type so that on saving the translation it is then correct

The code revisions now pick up the hooks just after WooCommerce save, and have been adjusted to work on quick edit (#549) and bulk edit too so the product type workarounds are no longer needed.
Of course this isn't the end of it - the whole thing should be re-reviewed to only use the woo api and avoid the wp api - but should remove some of the odd behaviours.

Great! That workaround fix was truly awful.
So, is the misspelled _translation_porduct_type meta no longer necessary, either?
I'm going to test this updated version when I have a minute.
Thanks!

I’ve left the  the misspelled _translation_porduct_type meta assignment in there for now but it isn’t used.   If this version goes ok a subsequent release could remove this and other redundant code

I tested this new version and I found an issue:

I have a site with three languages (Spanish, English and French). Spanish is the default one. The "Product-type" sync option in woopoly is checked.
I add a new product in Spanish
I select "Grouped product" (or any product type other than "Simple product") and save.
The product type dropdown shows the selected product type.
I click any "+" link in the "Languages" metabox to add a translation.
The product type dropdown in the new translation is disabled (the "Product-type" sync option is checked) but it shows "Simple product" instead of the selected product type in the default language product.

Ok yes I’ll take a look.
Changing product types (and any other property) on existing products should be fine.

@mrleemon checked in: on new translations needed to keep the wordpress hook as the woo hook is not fired yet

Ok, thanks!
I'll test it later when I have a moment and come back to you as soon as possible.

I tested this new version and my issue with new translations is fixed.
Thanks!


24 hour time

Can you please add an option for 24 hour (military) time ? Thanks
And maybe some future options require a switch for imperial vs metrical units.  ;-)

@sniperle I'll look into it. But as I'm incredibly busy right now, solving the specific bugs and implementing the new features might take a while so I can't specify a specific date as to when I'll push things live.

@sniperle does military time have PM and AM? I don't know if I should display it 23PM, 2 AM and so on or just 23, 24, 25. I was thinking instead of PM and AM, I would switch it to something like 23:00 or 2:00 or 13:00.
what do you think?

It doesn't have am and pm, just only the time in 24 hour notation, like you wrote: 02:47 or 21:56

@sniperle i implemented military time and it'll come in the next update which should be sometime this week. if you tap the settings icon, you will see an option for it.

Thanks :-)


TAG_INVALID in conformance tests is not present is spec

The TAG_INVALID error code is currently being checked by the conformance tests, but it's not mentioned in the specification.
Should TAG_INVALID be added to the spec, or should it be removed from the comformance tests?

This was removed from the distribution-spec back in #206. Is this throwing an error for you? It may be useful for UX when interacting with registries that implement the docker registry spec.

No errors no. I thought I'd bring it up because although they are accepted error codes in the conformance tests, they are not listed in the list of acceptable error codes so I was a bit confused.
i.e

The code field MUST be one of the following:
Except the code can also be TAG_INVALID or MANIFEST_UNVERIFIED.

Am I nitpicking?


Issue with locust master on Python and locust client on JAVA

Describe the bug
Hello I tried to write the actual code where I used Locust framework for performance testing and locust4j from github. I created the master on Python and slave on Java. The master and slave started, but master didn't see that slave was connected to master, actually is not connected. And in UI it wasn't worked.
Expected behavior
I changed the version of locust the worker was connected but i got errors:
Traceback (most recent call last):
File "src\gevent\greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "c:\users\maksim_kim\appdata\local\programs\python\python39\lib\site-packages\locust\runners.py", line 707, in client_listener
c.cpu_usage = msg.data["current_cpu_usage"]
KeyError: 'current_cpu_usage'
2021-07-03T12:27:24Z <Greenlet at 0x18eda9a4150: <bound method MasterRunner.client_listener of <locust.runners.MasterRunner object at 0x0000018EDA99A3D0>>> failed with KeyError
Actual behavior

Steps to reproduce
here is the Python code on master
from locust import User, TaskSet, task
class DummyTask(TaskSet):
@task(1)
def dummy(self):
pass
class Dummy(User):
tasks =[DummyTask]
I also used the command on master for opening the port: locust -f locust-master.py --master --master-bind-host=<IP_ADDRESS> --master-bind-port=5557 Here is the slave code on Java Main
import com.maksimkim.getData;

import com.github.myzhan.locust4j.Locust;
public class Main {
    public static void main(String[] args) {
        Locust locust = Locust.getInstance();
        locust.setMasterHost("<IP_ADDRESS>");
        locust.setMasterPort(5557); //some free port to run the Locust slave

        locust.run(
                new getData(50)
        );
    }
}

Here is the slave Code on Java getData
import com.github.myzhan.locust4j.AbstractTask;
import com.github.myzhan.locust4j.Locust;
import com.jayway.restassured.response.Response;
import static com.jayway.restassured.RestAssured.given;
public class getData extends AbstractTask {
private int weight;
@Override
public int getWeight() {
    return weight;
}

@Override
public String getName() {
    return "Open application task";
}

public getData(int weight) {
    this.weight = weight;
}

@Override
public void execute() {
    try {
        Response response = given().get("http://github.com");
        Locust.getInstance().recordSuccess("http", getName(), response.getTime(), 1);

    } catch (Exception ex) {
        ex.printStackTrace();
    }
}

}
Environment

OS: Windows 10
Python version: 3.9.6
Java: 16.0.1
Locust version: 1.6.0
Locust4j: 1.0.6
Locust command line that you ran: locust -f locust-master.py --master --master-bind-host=<IP_ADDRESS> --master-bind-port=5557 for master in python

Probably that locust4j version is not compatible with 1.6. File a ticket with them maybe? Or try an older locust version.


Fix task-collection endpoint error message

By setting the package name to asd.whl, in the request body, we can get this errro message:
"detail": "Invalid package or manifest. Original error: WARNING: Requirement 'asd.whl' looks like a filename, but the file does not exist\nERROR: asd.whl is not a valid wheel filename.\n\n[notice] A new release of pip is available: 23.2.1 -> 23.3.2\n[notice] To update, run: pip install --upgrade pip\n"

where the [notice] pip-related part should not be here.

See https://github.com/fractal-analytics-platform/fractal-server/pull/1122#issuecomment-1880872120

Here is where it comes from
$ python -m pip download --no-deps asd.whl -d .

WARNING: Requirement 'asd.whl' looks like a filename, but the file does not exist
ERROR: asd.whl is not a valid wheel filename.


[bug] failednodes wont remove from control panel, same as closed bug #318

Before submitting a bug please read: https://zwave-js.github.io/zwavejs2mqtt/#/troubleshooting/bug_report
Version
1.3.0
Build/Run method

[ X] Docker
[ ] PKG
[ ] Manually built (git clone - npm install - npm run build )

zwavejs2mqtt version: 1.3.0
zwavejs version: 6.4.0
Describe the bug
I have a number of nodes that are totally dead.  Tehy show up in the control panel when I enable "show hidden nodes" option.  I cannot remove them.  The "Is Failed Node" commands works succesfully (UI reports success), the Remove Failed Node options fails with "Error while calling api removeFailedNode: Node 2 was not found! "
To Reproduce
Steps to reproduce the behavior:

Show hidden nodes
Perform IS Failed Node option
Attempt Remove Failed Node option

Expected behavior
I would expect a success to eb returned after #3 above
Additional context
This appears to be the same as #318
In a (possibly misguided) attempt to get "clean" logs, I deleted the two log files in the control panel.  Performed a docker pull (and got latest version in proces) and the container started again.
In timestamp 21:22:54.020, the ISFailedNode command can be seen.  There is no Remove Failed Node after that that I can see (the REmove was requested in the UI withing seconds of the IsFailedNode)
The good nodes are not all showing complete as the battery ones havent worken up yet since container restart.  But I had this issue before I did the restart of the container as well. (I have another much larger set of logs available as well - pasted in as well)
zwavejs2mqtt-store - Logs from before 1.3.0 update.zip
zwavejs2mqtt-store Logs from after 1.3.0 update.zip

zwavejs2mqtt-store- SILLY log mode.zip
Just got new log dump in Silly mode.  Have done another IsFailedNode and attempted Remove Failed Node at end of log

Could you show me a screenshot of what you see? BTW you shouldn't enable that flag 'show hidden nodes' in most cases it just cause confusion, I will remove it in next major

Enclosed screengrab.  I an see the node is "REmoved" accoring to the column, but I was trying to get it to fully remove entirley.  I may be trying to do somethign that doesnt make sense.  my OCD just kicking in wanting things to be "clean".

Shold be fixed in #710

thanks

Isn't that inherent z-wave behaviour?
As far as I remember, it works (on a controller-level) like this:
When you remove a node, the ID is not "reused" initially - when adding a new node, it will get the highest "never-used" available node-ID - until you reach the limit of 254 nodes - then adding a new node will result in using the "once-used" node-IDs from the beginning

This is not really a bug, it's kind of an UI issue due to the fact on UI side and backend I was using an array for nodes where the index was the nodeId. As you said when a user removes/add a new node it takes the highest id (or he could use the replceFailed to give the node a specific id of a failed node), the node you see as removed are dummy nodes I add in the positions I don't have anything.


NPM Package: Unexpected token export

CodePen demo

https://codepen.io/pen?template=wGqJEz
Steps to reproduce the problem

Download the package via npm (npm install popper.js --save)
Import the library into your project /node_modules/popper.js/dist/popper.js
See the error in console

What is the expected behavior?
Library works without errors
What went wrong?
I suspect that the version builded in npm is wrong
Any other comments?
Nope

Read the usage instructions in the README.
Also. If you use npm you don't have to manually specify the path. Just use the package name and the right file will be picked from package.json


option to make popup not closeable by other popups

Make popups which stay on screen, even when other popups are triggered.
References #1853

Is this in RC3?


Weather show actual status, not a weather at ride date

Hi rostykk thanks for report
Do you have activity example?
Strange I'm giving the start date of act to windytv while embeding the weather.
Tom

example:
http://rostyk.net/test/stravix_weather.png
activities in february year ago: https://www.strava.com/activities/110729832
and recent: https://www.strava.com/activities/310770887
shows exactly the same data

So iframe links looks like
https://embed.windyty.com/?surface,wind,2014-02-02,52.24642075,4.5682495,20,,,,
and
https://embed.windyty.com/?surface,wind,2015-05-24,52.257484749999996,4.5575865,20,,,,

Sorry for mine not programmings skills.
What i notice:
for example same activity: https://www.strava.com/activities/310770887
it says:
startDateLocal:<PHONE_NUMBER>
so you function in stravistix/hook/extension/js/modifiers/WindyTyModifier.js
var date = new Date(pageView.activity().get('startDateLocal') * 1000); // will give date=1432473090000
and after:
date.toISOString().split('T')[0] give result "2015-05-24" not like inspected for windity yyyy-mm-dd**-hh**
and also windyty had only 00:00 / 06:00 / 12: 00 and 18:00 time slots
So probably you should roundUp  for to this values (or use by default "12")

Don't  worry add hour  is easilly fixable. With the  right and with good
hour ;) fixed  in  0.7.2
Tom
Le lun. 1 juin 2015 23:37, rostykk<EMAIL_ADDRESS>a écrit :

Sorry for mine not programmings skills.
What i notice:
for example same activity: https://www.strava.com/activities/310770887
it says:
startDateLocal:<PHONE_NUMBER>
so you function in
stravistix/hook/extension/js/modifiers/WindyTyModifier.js
var date = new Date(pageView.activity().get('startDateLocal') * 1000); //
will give date=1432473090000
and after:
date.toISOString().split('T')[0] give result "2015-05-24" not like
inspected for windity yyyy-mm-dd*-hh*
and also windyty had only 00:00 / 06:00 / 12: 00 and 18:00 time slots
So probably you should roundUp for to this values (or use by default "12")
—
Reply to this email directly or view it on GitHub
https://github.com/thomaschampagne/stravistix/issues/93#issuecomment-107719646
.

Cool. Thanks!

Dev notes
Fix bug with this simple computation:
var windyTyHour = Math.round(activityHour/6) * 6;

Should return the nearest windyty hour (0, 6, 12 or 18) for a given activity hour (0h to 23h)

Windity claims low space - no hostorical data then - need another service implementation...
https://community.windytv.com/topic/3487/no-more-historical-data-on-windytv-maybe-in-4q-2017

indeed. Do you know another one?

No free solution out of the box, as before.
Historical data without map
A time-machine request returns the observed weather at a given time (for many places, up to 60 years in the past):
https://api.darksky.net/forecast/[key]/[latitude],[longitude],[time]
https://darksky.net/dev/docs/time-machine

First thousand API calls you make every day are free
Every API call after that costs $0.0001 each
Credit needed: Credit us with a “Powered by Forecast”


multipleImage error

Laravel Version: 7
PHP Version: 7.4
Laravel-admin: #.#.#

i use multipleImage function to upload, but i can't editing my iteam.
Error Message:
  ModelNotFoundException In Builder.php line 398 :
     No query results for model [App\Gallery] 34

fixed


Add lightning-payment-pathfinding-for-reliability to adopting-bitcoin/2021

This PR adds Lightning Payment Pathfinding for Reliability transcript to the lightning-conference/2019 directory.

Great addition of chapters! Thanks for the contribution @gurwindersahota!


External Scripts, with and without document.write statements, and a Cancelled Callback

Hi
Thanks for the awesome library, really useful.
I'm trying to use PostScribe to manage script injection with external scripts, some of which may have document.write calls in them, some of which may not. At the moment I'm sending all of them through Postscribe because even if there's no document.write statement it's very important to maintain the order of the external scripts.
The problem I'm having is that when I send an external script without a document.write statement in it to Postscribe, I get back a cancel object and then Postscribe ceases to function, as you might expect.
In order to work out whether there's a document.write statement in the script, I would have to find some way of downloading and searching the cross-domain scripts with some sort of regex, I guess. It seems like you're already doing something better.
What would be even better from my point of view would be a cancelled  callback, which I can then work with and Postscribe would continue to perform its function for those external scripts that do have document write statements.
Tom

OK this may not be the use case, but if you add in this callback:
beforeWriteToken: a callback called before writing a token
and do nothing except log a message to the console, then the done function will fire even if there are no document.write statements in the external script.
Worth knowing maybe for someone.

Does the error callback not work for this? https://github.com/krux/postscribe/blob/master/src/postscribe.js#L65

Nope doesn't get fired

Are you running that on an HTTPS page? IF you give me an webpage I can test on then I'll take a look

Not https no. I'm just injecting Postscribe at the moment into this page:
http://www.noobmeter.com/compareplayers/eu/all/tomg1/gallam1
And then testing the code in the console.

As I said in the other thread, I'm running through a set of external scripts in an array, then adding each of those scripts in turn. I've just tested that page with an injection and then running in the console and it works fine. Could there be some issue with adding multiple Postscribe tags in quick succession, i.e. before the previous one has finished? It did all work when I was running them with the releaseAsync: true in the options object and adding async="async" to the script tags. Shall I post some code from my project?

var frag = document.createDocumentFragment();
console.log(source.substring(0, 50) + " (50 chars) now working via POSTSCRIBE with Unique ID of: " + arrayEntry[3]);
var postscribeScript = document.createElement('script');
if (arrayEntry[5]) { postscribeScript.type = arrayEntry[5]; } else { postscribeScript.type = 'text/javascript'; }
postscribeScript.charset = 'utf-8';
postscribeScript.async = false;

postscribeScript.textContent = "postscribe(\'#" + arrayEntry[3] + "\', \'<script src=" + source + "><\/script>\', { done: function() { console.info(\'Turbo Postscribe External Script (50 chars) " + source.substring(0, 50) + " has been Processed.\'); var marker = document.createElement(\'div\'); marker.setAttribute(\'data-turbo-script-type\', \'" + arrayEntry[0] + "\'); marker.setAttribute(\'data-turbo-script-content\', \'" + source.substring(0, 50) + "\'); marker.setAttribute(\'data-turbo-script-delivered\', \'" + arrayEntry[4] + "\'); (document.body||document.documentElement).appendChild(marker); }, error: function() { console.error(\'Turbo Postscribe External Script (50 chars) " + source.substring(0, 50) + " has hit an error. Starting Error Handler\'); var marker = document.createElement(\'div\'); marker.setAttribute(\'data-turbo-script-type\', '" + arrayEntry[0] + "\'); marker.setAttribute(\'data-turbo-script-content\', \'" + source.substring(0, 50) + "\'); marker.setAttribute(\'data-turbo-script-error-marker\', \'" + arrayEntry[4] + "\'); (document.body||document.documentElement).appendChild(marker); } });"
postscribeScript.setAttribute('data-optimised-script', 'true');
frag.appendChild(postscribeScript);
targetNode.appendChild(frag);

Can you setup a webpage to debug this with?

Just sent you an email.

Hi
OK so this was a timing issue. I was removing the first param targetnode from the page, after Postscribe checks for its presence but before the script was actually written to the page. So Postscribe just failed silently. Maybe another check for the target node before writing to the page? And maybe I'll leave tidying up until the whole thing is actually working. Anyway, thanks for your time @jnewman
Tom

I'll add a check in a future release


step 21 on linux

describe how to launch GoogleAppEngine on linux

Follow the instructions to install the SDK: https://cloud.google.com/appengine/downloads
There is a README file in the installation, where there are sections. called "Running the SDK" and "Using the App Engine Launcher"
Basically, you need to use dev_appserver.py and appcfg.py as there is no graphical tool.


[FLINK-9311] [pubsub] Added PubSub source connector with support for checkpointing (ATLEAST_ONCE)

What is the purpose of the change
Adding a PubSub connector with support for Checkpointing
Verifying this change
This change added tests and can be verified as follows:

Added unit tests.

Added integration tests to flink-end-to-end which runs against docker.

An example has been added in flink-examples which runs against the actual Google PubSub service. this has been manually verified.

Is there a need for integration tests?
We feel like there is and have added them.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): Yes, Google Cloud Sdk for PubSub  but because it is a connector this does not add any dependencies in flink itself.
The public API, i.e., is any changed class annotated with @Public(Evolving): No
The serializers: No
The runtime per-record code paths (performance sensitive): Nothing has been changed only a connector has been added.
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing,
Yarn/Mesos, ZooKeeper: No
The S3 file system connector: No

Documentation
Does this pull request introduce a new feature? Yes
If yes, how is the feature documented? JavaDocs, added an example in flink-examples and we updated the website docs.

Thanks you for your review @yanghua ! I made edits to fix most of them, I've posted 2 follow up questions on the points that were not entirely clear to me :)

I have fixed a subset of the review comments and rebased against the current master.

I check why the build failed and all I could find is this:
[FAIL] 'Kafka 0.10 end-to-end test' failed after 0 minutes and 54 seconds! Test exited with exit code 0 but the logs contained errors, exceptions or non-empty .out files

I have been unable to find out why this happens and what our changes did to cause this.
Please advise.

Thanks a lot for this pull request.
I will try to help you to get it into Flink soon.

@zentol What's your take on the "Sink / Source" name, vs "Consumer / Producer"

@Xeli and @nielsbasjes: Before we go really deep into the reviewing of this code, let's plan this contribution a bit.
I would really like this to go into Flink, ideally into the upcoming 1.8 release. I have the feeling that multiple people would like to use this connector, and maybe get the attention of Google and some Google Cloud users with this effort :)
But I also see a few problems with this effort
A) I haven't worked on the Flink codebase for over a year, so my reviewing skills are a bit rusty. I will try to mitigate this by asking for additional reviewing help once I've gone over the code once. But this means for you that there will be probably multiple rounds of reviews (and that I might comment on things which are not actually an issue).
B) There is not much time left till the 1.8 release. I don't know how many things we find in the code that need to be addressed, and how much time you have to address them.
C) I found already some issues in the code with respect to exception handling, logging, maybe performance. I also want to do some correctness testing (such as rescaling, failures etc.). Have you done such tests already?
Despite these problems, I'm committed to get this in. I'd be happy to support an approach where we decide to list "known issues" in the documentation. Once the release is out, we iteratively open small pull requests (maybe with the help from others) to resolve the known issues (+ the feedback we get after the release).
What do you think about this?

Hi @rmetzger
We are still eager to get the code in the flink repository, especially trying to get it in 1.8 sounds really good.
I am fine with putting some more effort into this as long as we can keep up the momentum and have a 1.8 release as the goal for now.
Regarding your points:
A) It would help us if you could make it clear which points I can improve (such as the exception logging comment) and which points are more of a design decision comment (source/sink vs subscription/publisher) and should not be changed just yet. I am fine with multiple review rounds as long as we keep the pace. :)
B) I don't see too many problems here for now.
C) We have done extensive tests using our own use-case as a base (a financial transaction processing system), so, for example, at-least-once guarantees under different failure scenarios has been tested quite extensively. And this PR also has some changes in existing code to make sure we handle rescaling and such properly.
All in all, I think we're on the same level, let's try to get this in 1.8 :)

Great!
I will then do the following:

I do a pass over the code, commenting on everything that seems suspicious
I will run some more tests with probably more feedback
I wait till you've addressed all outstanding items, then we do a final round of review

One thing to consider in the long run: Apache Beam / Google Cloud Dataflow's PubSub connector has a mode that manages subscriptions behind the scenes: https://cloud.google.com/dataflow/model/pubsub-io

Thanks a lot for addressing some of my comments. There are actually some more questions (I think GitHub has hidden them as "11 messages like this" in the activity stream of this PR)

Regarding the message pulling model:
The current approach is the following:

We set Flow control parameters, so that the API server knows how many messages we want to receive.
We launch a async listener, which gets messages pushed from the API server (according to our flow control)
The async listener forwards messages to a queue
The Source is removing messages from the queue in its while() loop

I propose to change the implementation to the following:

We use the Synchronous Pull API: https://cloud.google.com/pubsub/docs/pull#synchronous-pull
The Source while() loop is pull-ing data from the API server

Backpressure will be treated very naturally.
Flink's sourceContext.collect() will not return if there's backpressure, thus we won't pull any new data from the API server (other parallel sources can pick up the data).
Users don't need to manually configure a good set of checkpointing / flow control config parameters.
The memory footprint is lower, as we don't have data in a in-memory queue. Similarly, this queue is adding latency, as records are waiting there.

@rmetzger
Ah sorry, I missed those other questions because of the GitHub UI indeed :) I'll go over all questions today.
Regarding the message pulling proposal. I agree with you, the model you describe is a lot cleaner and I initially tried it implementing it that way. However, because this PR started back in June I cannot remember why I moved away from sync to async API...
I'm willing to give the sync API a try again, but I fear we won't be able to test it enough before the feature freeze of 1.8.
Are you able to help testing the stability on actual Flink clusters with Pubsub this week if I switch to the sync API? And do you think it will be feasible to get this stable before 1.8?

Thank you for all the comment addressing you've done!
Sorry for not being active on the PR yesterday. I had to take care of some other things.
I believe we should try to use the Sync API, because it is, as you say as well, the cleaner approach.
I will be able to test it as well on Google.
Whether we get it into the release or not is very hard to predict at this point. If we are "lucky", the release get's delayed. If not, I'm committed to keep pushing on this PR, so that we merge it as soon as we can  (even if it is 2 days after the 1.8 feature freeze).
So I suggest to keep pushing, maybe we manage to get it into 1.8.
My biggest headache are currently the changes to the existing codebase of Flink (outside this connector). If this PR would just add a connector without touching anything else, I would probably be more comfortable merging it. If we decide to keep those changes in, I will probably not merge this PR if I'm not getting a solid review by somebody who knows the RabbitMQ / acknowledging base code.

@rmetzger I've change from using the Async API to the Sync API and the code is so much cleaner now, thank you for the nudge towards Sync API !
I've run some quick tests on a 1.6 cluster and so far so good. I'll do a more thorough test on a 1.7.2 cluster this weekend, I am especially interested in the performance.
I have a few worries: The grpc calls catch all exceptions and rethrow them as unchecked exceptions. Which means I kind of hard to see what kind of exceptions are thrown that we should catch.
I currently see one exception quite often: DEADLINE_EXCEEDED which is basically a timeout. My guess is this tail latency. My flink cluster is behind some firewalls on a gcp kubernetes cluster and we just need some higher default timeouts. I am curious if you run into the same exception.
Do you think you can find someone to have a look at the MultipleIdsMessageAcknowledgingSourceBase (ping @StephanEwen who originally created it I think?)

Awesome! I'll try to do some testing this afternoon or on Monday!

@rmetzger @afedulov I think I've addressed all review points now.
Could you do a (final?) pass over the code and docs and close any comments that have either been solved or are not relevant anymore?

Thanks! I will do another check, the last two days have been pretty crazy with other work. Tomorrow looks better, I hope that I will get to it. Sorry.

I currently see one exception quite often: DEADLINE_EXCEEDED which is basically a timeout. My guess is this tail latency. My flink cluster is behind some firewalls on a gcp kubernetes cluster and we just need some higher default timeouts. I am curious if you run into the same exception.

I don't see that exception. What I see for the sink is the following:
15:34:34,803 INFO  io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder  - [id: 0x0568ec6c, L:/<IP_ADDRESS>:57417 - R:pubsub.googleapis.com/<IP_ADDRESS>:443] ignoring HEADERS frame for stream RST_STREAM sent. {}
15:34:34,803 INFO  io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder  - [id: 0x0568ec6c, L:/<IP_ADDRESS>:57417 - R:pubsub.googleapis.com/<IP_ADDRESS>:443] ignoring DATA frame for stream RST_STREAM sent. {}
15:34:34,803 INFO  io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder  - [id: 0x0568ec6c, L:/<IP_ADDRESS>:57417 - R:pubsub.googleapis.com/<IP_ADDRESS>:443] ignoring HEADERS frame for stream RST_STREAM sent. {}
15:34:34,803 INFO  io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder  - [id: 0x0568ec6c, L:/<IP_ADDRESS>:57417 - R:pubsub.googleapis.com/<IP_ADDRESS>:443] ignoring DATA frame for stream RST_STREAM sent. {}
15:34:34,803 INFO  io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder  - [id: 0x0568ec6c, L:/<IP_ADDRESS>:57417 - R:pubsub.googleapis.com/<IP_ADDRESS>:443] ignoring HEADERS frame for stream RST_STREAM sent. {}
15:34:34,803 INFO  io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder  - [id: 0x0568ec6c, L:/<IP_ADDRESS>:57417 - R:pubsub.googleapis.com/<IP_ADDRESS>:443] ignoring HEADERS frame for stream RST_STREAM sent. {}
15:34:34,803 INFO  io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder  - [id: 0x0568ec6c, L:/<IP_ADDRESS>:57417 - R:pubsub.googleapis.com/<IP_ADDRESS>:443] ignoring DATA frame for stream RST_STREAM sent. {}
15:34:34,803 INFO  io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder  - [id: 0x0568ec6c, L:/<IP_ADDRESS>:57417 - R:pubsub.googleapis.com/<IP_ADDRESS>:443] ignoring HEADERS frame for stream RST_STREAM sent. {}
15:34:34,803 INFO  io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder  - [id: 0x0568ec6c, L:/<IP_ADDRESS>:57417 - R:pubsub.googleapis.com/<IP_ADDRESS>:443] ignoring HEADERS frame for stream RST_STREAM sent. {}
15:34:34,803 INFO  io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder  - [id: 0x0568ec6c, L:/<IP_ADDRESS>:57417 - R:pubsub.googleapis.com/<IP_ADDRESS>:443] ignoring DATA frame for stream RST_STREAM sent. {}
15:34:34,803 INFO  io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder  - [id: 0x0568ec6c, L:/<IP_ADDRESS>:57417 - R:pubsub.googleapis.com/<IP_ADDRESS>:443] ignoring HEADERS frame for stream RST_STREAM sent. {}

I've tested the PR a bit again, and I talked with @StephanEwen about the MultipleIdsMessageAcknowledgingSourceBase.
I have the following observations to share:

Stephan is proposing to follow Apache Beam in the implementation of the PubSubSource, because the MultipleIdsMessageAcknowledgingSourceBase only provides exactly-once with a parallelism = 1. Apache Beam has a source for PubSub that only guarantees message delivery (aka at least once). After the sources, they have a keyed window operator, which deduplicates messages based on a unique id within a certain time-window (say 10 minutes), making this exactly-once within that time-window.
The PubSub source's performance is limited by the checkpointing frequency. When running the source with a checkpoint once every 1000ms, I achieve a throughput of 3000 msg/sec. With a checkpoint frequency of 50ms, I achieve a throughput of 20.000 msg/s.
The other parameter to tweak is the number of source instances. With 80 sources and a checkpoint interval of 1000ms, I achieve 65.000 msg/s.
The PubSub grpc system seems to be quite CPU intensive. The PubSub Sink can very easily burn my CPU. At a throughput of 15k msg/s, my CPU is already at 600%. I could not find anything in the code that would explain this behavior. The CPU sampler indicated that the grpc/netty code of the PubSub client is eating up many CPU cycles.

What’s your take on these observations?
I’m thinking the following for them:

We should investigate if this is easily possible to implement. If so, we could provide a fairly “bare” PubSubSource, which exposes the message ID downstream, so that we can provide a generic deduplication window (for this and other connectors).
If we go that route, I would propose to make this PR only about the Source itself, and provide the deduplication logic in a follow up PR.
Requiring users to run large topologies with very high checkpointing frequencies (or a high number of pubsub sources is not a good user experience. We should consider making the ack-behavior configurable. We either ack on checkpoint completion (current behavior, and my proposed default behavior), or we ack immediately. Acking immediately will basically make the source potentially loose records on failures, but it should be very fast.
I don’t think we can do anything about that.

Hello @rmetzger
Let me address them point by point:

I really like the idea of providing an out of the Box exactly-once implementation.
I am worried about the state size on high volume PubSub subscriptions. As in my experience, PubSub will be sending messages multiple times and this might happen as up to 7 days after the initial message. The 7 days being the upper bound as this is the maximum PubSub retention time. So this would mean we would be storing message ids for 7 days.
Implementation wise this should still be very doable though. We could replace the DeserializationSchema from Object deserialize(byte[]) to a class with something like: Object deserialize(PubSubMessage) so we can decide if and how we want to pass the messageId on to the next operator. I think if we can add this behavior in this PR we can always provide an Exactly-once implementation later on without becoming backward incompatible. What do you think?
I think either way it still makes sense to use the MultipleIdsMessageAcknowledgingSourceBase class for the Acknowledge-on-checkpoint behavior, just so we don't duplicate this behavior in code. The most elegant solution might be to split the acknowledge-on-checkpoint and exactly-once on parallelism 1 code. So RabbitMQ connector can use both parts while PubSub only uses the acknowledge-on-checkpoint part. But this would require some large changes for the RabbitMQ connector which I cannot test myself. I'm a bit hesitant to go this direction. What do you think?

I'm going to test this myself this weekend. In the previous (async) implementation I would for sure expect this behavior but the new implementation should only be bounded by memory (the amount of acknowledge Ids it has to store between checkpoints)
I fully agree we should not force users to lower their checkpoint frequency if they want a higher throughput. I think this is fixable, I'll come back to this!
When I first created the PR I has added modes to the PubSubSource. A NONE / ATLEAST_ONCE / EXACTLY_ONCE enum.
Where NONE would be the behavior you describe: acknowledge immediately with the risk of losing messages. If we can't seem to fix the performance for large checkpoint interval jobs we might consider adding these modes back again.

Could you play with the withMaxMessagesPerPull setting? I can imagine each pull has some overhead and reducing the number of pulls needed might be nicer for the CPU cycles.
Do you see this high CPU consumption on empty subscriptions as well?
We could always give the json api a try, but I would be suprised if this has performance..

Hi @rmetzger
I've done some performance tests to see if checkpoint interval limits the throughput and I am not seeing this happen. On a smallish kubernetes pod with parallelisms ranging from 1 to 5 I keep seeing around 1000 m/s throughput consistently using checkpoint intervals ranging from 50ms to 1000ms.
What setup / messages / parallism did you use to test it? And just for the double check: Are you sure the latest version of the connector was used?

Damn. You are right. For some reason, it seems that I have checked out your latest code, but my test project did not pull the build. Maybe I was running package instead of install.
I should have checked this more thoroughly :(
I was sick the last two days, that's why I've been so passive. I'll re-run my experiments.

Sorry to hear you were ill :) But I am glad to hear you used the wrong connector so the issue (hopefully) doesn't occur on the new version!
I can't wait to read about your experiments!

Hello @rmetzger
Let me address them point by point:
I really like the idea of providing an out of the box exactly-once implementation.
I am worried about the state size on high volume PubSub subscriptions. As in my experience, PubSub will be sending messages multiple times and this might happen up to 7 days after the initial message. The 7 days being the upper bound as this is the maximum PubSub retention time. So this would mean we would be storing message ids for 7 days.
Implementation wise this should still be very doable though. We could replace the DeserializationSchema from Object deserialize(byte[]) to a class with something like: Object deserialize(PubSubMessage) so we can decide if and how we want to pass the messageId on to the next operator. I think if we can add this behavior in this PR we can always provide an Exactly-once implementation later on without becoming backward incompatible. What do you think?
I think either way it still makes sense to use the MultipleIdsMessageAcknowledgingSourceBase class for the Acknowledge-on-checkpoint behavior, just so we don't duplicate this behavior in code. The most elegant solution might be to split the acknowledge-on-checkpoint and exactly-once on parallelism 1 code. So RabbitMQ connector can use both parts while PubSub only uses the acknowledge-on-checkpoint part. But this would require some large changes for the RabbitMQ connector which I cannot test myself. I'm a bit hesitant to go this direction. What do you think?
I'm going to test this myself this weekend. In the previous (async) implementation I would for sure expect this behavior but the new implementation should only be bounded by memory (the amount of acknowledge Ids it has to store between checkpoints)
I fully agree we should not force users to lower their checkpoint frequency if they want a higher throughput. I think this is fixable, I'll come back to this!
When I first created the PR I had added modes to the PubSubSource. A NONE / ATLEAST_ONCE / EXACTLY_ONCE enum.
Where NONE would be the behavior you describe: acknowledge immediately with the risk of losing messages. If we can't seem to fix the performance for large checkpoint interval jobs we might consider adding these modes back again.
Could you play with the withMaxMessagesPerPull setting? I can imagine each pull has some overhead and reducing the number of pulls needed might be nicer for the CPU cycles.
Do you see this high CPU consumption on empty subscriptions as well?
We could always give the json api a try, but I would be suprised if this has a better performance..

@Xeli I tend to agree with Stephan on getting rid of the MultipleIdsMessageAcknowledgingSourceBase. The fact that its memory consumption is unbounded worries me.

What's the status on this pull request?

@Xeli @rmetzger My apologies for being late on the review. I'll finish reviewing the testing part in one day or two.
BTW, some previous comments might need to be addressed or further discussed. e.g. the thread-safe issue in close() and stop(). And the usage of MultipleIdsMessageAcknowledgingSourceBase.

Hey @rmetzger  @becketqin
I'd like to prevent having to keep processing new review comments.
So I will wait until one of you has done a final run through of the code + tests and we can discuss what final changes should be done.

Very thorough review, indeed! Will there be a way to use this code in the currently released Flink, or will this be immediately released as 1.8 or 1.7.3?

Hi @becketqin
Thanks for the review! I've addressed most of them. Some I have not addressed yet because they are related to MultipleIdsMessageAcknowledgingSourceBase.
We'll need to make a decision about MultipleIdsMessageAcknowledgingSourceBase. I am oke with replacing it with a class inside of the PubSub connector. But do realize it will almost be identical to MultipleIdsMessageAcknowledgingSourceBase. If you're oke with a bit of code deduplication I will add it.

When I run this code with Flink 1.7, I get this critical error when things crash:
Apr 09, 2019 6:44:17 PM io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=5, target=pubsub.googleapis.com:443} was not shutdown properly!!! ~*~*~*
    Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
	at io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:103)
	at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
	at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
	at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:419)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:222)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:164)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:156)
	at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:157)
	at com.google.cloud.pubsub.v1.stub.GrpcSubscriberStub.create(GrpcSubscriberStub.java:260)
	at okr.sources.DefaultPubSubSubscriberFactory.getSubscriber(DefaultPubSubSubscriberFactory.java:30)
	at okr.sources.PubSubSource.open(PubSubSource.java:96)
	at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:424)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:290)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
	at java.lang.Thread.run(Thread.java:748)

Hi @haf
Apologies for the late response but thank you for trying/testing the connector!
Your second exception occurs because you are running the code on flink 1.7 (?). The class MessageAcknowledgingSourceBase has a bug that I've fixed here but as this PR hasn't been merged into the flink version you are using you do not have this fix.
I've written a slightly altered version of this class which we use at our company which also has this fix. I am considering releasing the connector outside of flink with this altered class, that would solve your problem too.
Regarding the first exception, note it is only a warning so for stability or correctness I do not think it has any impact, but I fully agree it isn't nice. I will look into reproducing and fixing it, thank you for letting me know about it! :)

@Xeli No worries as all, I'm getting it for free, right? :)
Happy to reference this from somewhere else.
Thanks for considering the exceptions and stacktraces. I think this can turn out really nice! I can then learn from your code and perhaps create a BigQuery sink ;)

@becketqin Hi! Any progress?
@Xeli Can you e-mail me the other connector you've build for yourself, pretty please? henrik - at - haf dot se.
Also @Xeli I'm successfully running this locally in the IDE, but when I submit it to a local "cluster" of a single TM and a single JM, I'm getting (tensor is the name of my laptop):
==> /usr/local/Cellar/apache-flink/1.7.2/libexec/log/flink-h-taskexecutor-1-tensor.out <==
	at com.google.api.gax.rpc.UnaryCallable$1.futureCall(UnaryCallable.java:126)
	at com.google.api.gax.rpc.UnaryCallable.futureCall(UnaryCallable.java:87)
	at okr.sources.PubSubSource.run(PubSubSource.java:126)
	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:94)
	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:58)
	at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:99)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
	at java.lang.Thread.run(Thread.java:748)

It ends up in the running state, without consuming any data, and with logs like this:
==> /usr/local/Cellar/apache-flink/1.7.2/libexec/log/flink-h-standalonesession-1-tensor.log <==
2019-04-16 09:42:27,232 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Triggering checkpoint 93 @<PHONE_NUMBER>232 for job 52dae7a7fa6e9caec8e34f2f837f2e97.
2019-04-16 09:42:27,243 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Completed checkpoint 93 for job 52dae7a7fa6e9caec8e34f2f837f2e97 (83957 bytes in 11 ms).
2019-04-16 09:42:28,231 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Triggering checkpoint 94 @<PHONE_NUMBER>231 for job 52dae7a7fa6e9caec8e34f2f837f2e97.

It also says:

2019-04-16 09:40:54,419 INFO  org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase  - No state to restore for the PubSubSource.

Is that correct? Isn't the previously used checkpoint used, or would I need to start ZK for that?

Will there be support for replying messages? https://cloud.google.com/pubsub/docs/replay-qs

I'm testing this source on spotty connections (as I do most of my work on the go), and when streaming a lot of messages over 4G, at some point, the source logs this:
14:40:33,649 INFO  io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder  - {} ignoring {} frame for stream {} {}
14:40:39,010 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Completed checkpoint 23 for job f2aed8990f452e357b8f15f6452e4244 (12951720 bytes in 18142 ms).
14:40:39,013 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Triggering checkpoint 24 @<PHONE_NUMBER>010 for job f2aed8990f452e357b8f15f6452e4244.
14:41:11,608 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Completed checkpoint 24 for job f2aed8990f452e357b8f15f6452e4244 (14193479 bytes in 32593 ms).
14:41:11,614 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Triggering checkpoint 25 @<PHONE_NUMBER>608 for job f2aed8990f452e357b8f15f6452e4244.
14:41:22,628 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Completed checkpoint 25 for job f2aed8990f452e357b8f15f6452e4244 (16366371 bytes in 11017 ms).
14:41:22,638 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Triggering checkpoint 26 @<PHONE_NUMBER>628 for job f2aed8990f452e357b8f15f6452e4244.
14:41:43,287 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Completed checkpoint 26 for job f2aed8990f452e357b8f15f6452e4244 (17107164 bytes in 20655 ms).
14:41:43,290 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Triggering checkpoint 27 @<PHONE_NUMBER>287 for job f2aed8990f452e357b8f15f6452e4244.
14:42:19,171 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Completed checkpoint 27 for job f2aed8990f452e357b8f15f6452e4244 (18502218 bytes in 35880 ms).
14:42:19,174 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Triggering checkpoint 28 @<PHONE_NUMBER>171 for job f2aed8990f452e357b8f15f6452e4244.
14:42:55,863 INFO  io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder  - {} ignoring {} frame for stream {} {}
14:43:00,242 INFO  io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder  - {} ignoring {} frame for stream {} {}
14:43:00,242 INFO  io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder  - {} ignoring {} frame for stream {} {}
1

And it would seem no further events are being pushed down the streaming pipeline.
Sometimes, I also get:
14:43:23,816 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job 2d2461182e268f6838e957972a9a0361 with allocation id AllocationID{8f8a93b52291c2fa883df2fa1a7735de}.
14:43:23,816 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor            - Allocated slot for AllocationID{fc0b24a95227bdbfa46a4119fb907d19}.
14:43:23,816 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService        - Add job 2d2461182e268f6838e957972a9a0361 for job leader monitoring.
14:43:23,816 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService        - Try to register at job manager akka://flink/user/jobmanager_1 with leader id 692682c0-30b2-4900-ad43-3d94acf9a2c6.
14:43:23,816 WARN  org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService  - Error notifying leader listener about new leader
java.lang.IllegalStateException: The RPC connection is already closed
	at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
	at org.apache.flink.runtime.registration.RegisteredRpcConnection.start(RegisteredRpcConnection.java:91)
	at org.apache.flink.runtime.taskexecutor.JobLeaderService$JobManagerLeaderListener.notifyLeaderAddress(JobLeaderService.java:327)
	at org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService$NotifyOfLeaderCall.run(EmbeddedLeaderService.java:430)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

All of this is running locally on my laptop without ZK, so there should be no problems talking between JM and TMs.

Hey @becketqin
I've pushed code which replaces the MultipleIdsMessageAcknowledgingSourceBase as we discussed via email. I redid my local tests and everything seems to be running good now.
These should cover all of your review points, could you have a look at them?
Hi @haf,
This latest push should make it possible to run it on older flink versions, as for your last 2 comments: Replay is supported in the sense that pubsub clients don't have to do anything special for replay to work as far as I know. Your stacktrace seems to be unrelated to this pubsub connector but rather the HA setup of flink. Maybe it's better to ask in the mailing list about this, to keep this PR clean(er). If you do think it's related to the PubSub connector I'd be happy to hear about it.

@xeli I'm getting pretty bad throughput from the source; about 20 messages per second with a lot of expired messages (200/s):

Any ideas why? I checked the "backpressure" tab in Flink's UI and can't see anything indicative of a problem there. I'm running outside the cloud on my own laptop with incremental snapshots to Google Cloud Storage.
This is what it looks like after running all night:

Hi @haf
The connector will acknowledge messages on every successful checkpoint. How often do you checkpoint? What might happen is that for some reason messages are not acknowledged fast enough.
Could you also show how you create the PubSubSource? So what configuration you've used.

The checkpoints take e.g. 12528 ms to write, but they are not large at all:

I put the checkpoint period to:
        env.enableCheckpointing(9000); // checkpoint every N seconds
        env.getCheckpointConfig().setMinPauseBetweenCheckpoints(2000);

But I've tried 1000 and 1500 respectively, as well with approximately the same results.
The creation looks like this:
        final String subscriptionName = params.get("pubsub-subscription", Env.getOrElse("ANALYTICS_PUBSUB_SUBSCRIPTION", "logs-analytics-dev"));

        String jsonFilePath =
            Optional
                .ofNullable(System.getenv("GOOGLE_APPLICATION_CREDENTIALS"))
                .orElse("./analytics-dev.json");

        if (! new File(jsonFilePath).exists()) {
            throw new Exception(String.format("Couldn't find path: '%s'", jsonFilePath));
        }
        Credentials credentials = GoogleCredentials.fromStream(new FileInputStream(jsonFilePath));
        logger.info("Using Credential JSON: {}, subscription name: {}", jsonFilePath, subscriptionName);

        PubSubSource<AppEvent> source = PubSubSource.newBuilder(new AppEventSerializer(), projectId, subscriptionName).withCredentials(credentials).build();
        return env.addSource(source).uid(String.format("pubsub-%s", subscriptionName));

I've spoken with @haf through email and it seems the low acknowledge rate is related to the checkpoint frequency. The actual throughput of messages is fine.

I have good and bad news
a) People seem to be using this code, even though it hasn't been merged yet
b) There are issues with it: https://issues.apache.org/jira/browse/FLINK-12376

@rmetzger I'm the people whose tickets you reference (btw)

Okay :) When I saw the screenshot again in this PR, I recognized it.
Thx for responding on the Jira ticket.
I assigned your Jira and the one for implementing PubSub to a new PubSub component in Jira.

I think this PR has become quite painful for the contributor to keep up with: it has been open for almost a year and we're nearing 300 messages in this conversation.
If there are no legal concerns, i.e. licensing is good I would suggest to merge this now and resolve any future work via Jira issues and future PRs. I believe it would be very nice to have this in Flink since there are users that actually want to use this and at least one user that has been using this in production for quite a while. Even more so because this PR adds documentation, adds testing, even adds end-to-end testing.
What do you think?
cc @nielsbasjes @becketqin

@aljoscha +1 from me.
I still have some (performance) improvements ready to be merged but I think it'll go a lot smoother if they can go via small PRs.

Yes, I agree.
Having a working "beta" in Flink will also allow others to try it and report the issues they run into.
+1

@aljoscha Yes, I agree. But before merge we should probably make sure the end-to-end test works.

@becketqin I can push the latest version of the connector (internally we use this). This has end-to-end test fixed, some of the performance improvements mentioned above and has been rebased on the latest master.
I kinda fear the rebase will reopen all the comments in this PR though...shall I push it never the less?

@Xeli It is great to hear that you've got the connector working in production. Thanks for letting us know. Do you know how much is your latest version differ from the current code in the PR? If it is not a big diff, say less 200 or 300 lines of code change, I can take a quick review and hopefully there won't be much concerns. But if the diff is huge, we may as well fix the test in the current PR and do a separate PR to get your latest code. What do you think?

@becketqin It is not too different, I've pushed it here: https://github.com/Xeli/flink/commits/pubsub-connector-latest (last 5 commits).
Let me know if you can give it a quick review. If you find any concerns we move it into a separate PR and just cherry-pick the end-to-end test fixes, if not we might as well merge it :)

+1 to merge it as a beta connector
The upcoming feature freeze of the 1.9 release reminded me of how much time has passed. I will open a pull request to pubsub-connector-latest with the license documentation.

There's actually no pull request needed, because the pubsub connector is not bundling (for example through shading) any dependencies. From a license point of view, @Xeli's branch is good to go

Great! Shall I then merge in the pubsub-connector-latest branch, if there are no other concerns? (cc @becketqin @rmetzger )

@Xeli Yeah, let's do that. I can give a final pass on the patch after you merge the latest changes. @rmetzger what do you think?

Yes, I agree

Alright, I've just pushed the last fixes and rebased everything onto master

Thanks for the updates. Checkstyle is still failing
04:05:19.022 [INFO] --- maven-checkstyle-plugin:2.17:check (validate) @ flink-connector-gcp-pubsub_2.11 ---
04:05:19.240 [INFO] There are 4 errors reported by Checkstyle 8.12 with /tools/maven/checkstyle.xml ruleset.
04:05:19.242 [ERROR] src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/PubSubSource.java:[74] (regexp) RegexpSinglelineJava: Line has leading space characters; indentation should be performed with tabs only.
04:05:19.248 [ERROR] src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/PubSubSource.java:[75] (regexp) RegexpSinglelineJava: Line has leading space characters; indentation should be performed with tabs only.
04:05:19.248 [ERROR] src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/PubSubSource.java:[76] (regexp) RegexpSinglelineJava: Line has leading space characters; indentation should be performed with tabs only.
04:05:19.248 [ERROR] src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/PubSubSource.java:[77] (regexp) RegexpSinglelineJava: Line has leading space characters; indentation should be performed with tabs only.
04:05:19.249 [INFO] ------------------------------------------------------------------------
04:05:19.249 [INFO] Reactor Summary:

@rmetzger it took a while for travis to run, but it just succeeded

@Xeli Just to let you know, I am reviewing the patch. Hopefully we can check this in before 1.9 release (code freeze on Jul 5). So we will be able to get more feedback as users onboard.

@becketqin thanks for the review! I've just addressed all your points.
As mentioned in one of the review points I have created https://issues.apache.org/jira/browse/FLINK-13083 for the follow-ups. Please don't hesitate to add any improvements to that jira issue

@Xeli Thanks for updating the patch. I made some minor modifications to it and created a PR against your repo. Can you take a look. Otherwise I think we can merge the patch and address other issues in followup PRs.

Great, I've merged your patch into this PR (+ added your builder improvement to the docs).

@rmetzger I have reviewed the PR and I think the major issues have been addressed. Do you have time to also take a look? We are trying to get this checked in before 1.9 feature freeze and do some minor clean-ups in a follow-up patch.

Thanks a lot for the review @becketqin
It seems that the change has passed the build in https://travis-ci.org/Xeli/flink/builds/554345672.
Are you ok with squashing all commits into one?

I've prepared this branch for merging: https://github.com/apache/flink/compare/master...rmetzger:pubsub
I'll merge it soon (after lunch to be precise :) )

Really glad to see this gets into Flink!
@Xeli Thanks a lot for your contribution and patience in addressing the comments.
@rmetzger thanks for helping merge the patch.

Thank you both for the implementation and review work.
I'm sorry that it took so long.


Added WithExpressionExpanding configuration extension for EF Core.

Well, added extensions which simplifies expanding EF Core queries. For now it is not required to call AsExpandable() but just registering with DbContextOptionsBuilder WithExpressionExpanding extension.
var builder = new DbContextOptionsBuilder<TestContext>();
builder
    .UseSqlite("DataSource=testdb;mode=memory;cache=shared")
    .WithExpressionExpanding();

And then you can create context which queries will be expanded automatically
var context = new TestContext(builder.Options);

This also work with Dependency Injection.

@sdanyliv
Can you please solve merge conflict?

Done, I'm glad that you are back ;)

@sdanyliv
Will this work for all EF versions?

For EF Core 3 and 5, For EF Core 6 should be created new library.

But maybe, it may work with EF Core 6 without any additional manipulations. Not tested.

Done

@sdanyliv will this work when Include is used?
At the moment when I don't use WithExprrssionExpanding, I have to call Include before AsExpandable?
Br


Invitation to Your First Open Source Project Organization

Hello, I am a senior Game Developer.
I want to contribute to this open source.

Hi @SirBang , great to hear! :-)
We welcome external PRs, please the the contribution guidelines: https://github.com/NVIDIAGameWorks/kaolin/blob/master/CONTRIBUTING.md

Closing inactive bug.


[Wireguard] Easier peer configuration

Hello, this is just suggestion for new feature. As network admin in these Covid days I need to setup many new peers for Wireguard. But this task is quite time demanding because you need manually

generate client private and public keys
set server public key (copy-paste from OPNsense) in client config
create new Endpoint in OPNsense with cliet public key (copy-paste from client)
add new peer in OPNsense Local settings of Wireguard

Can OPNsense provide "Quick add" of peer like this script? https://github.com/burghardt/easy-wg-quick
Quick add dialog should include only these fields

Endpoint Name
Endpoint Shared Secret
Endpoint AllowedIPs

After clicking Save, OPNsense should generate client config for this new peer. Config file should be downloadable and QR code should be shown for easier setting of mobile device. For security reasons, this config file should be deleted after download.

To simplify the feature request:
Change VPN - WireGuard - Endpoints - Edit Local Configuration form:

Add Endpoint public address (allow URL or IPv4 or IPv6)

Change VPN - WireGuard - Endpoints - Edit Endpoint form:

Add a drop-down box with all peering local endpoint (can stay blank)
Add DNS field
Add IP Address field
Add (and pre-select) a checkbox "Generate a new Key Pair"
Add (and keep blank) a checkbox "Generate a new Shared Secret"

When clicking Save button:

a new key pair for a remote endpoint is generated
the new public key is inserted is stored in the remote endpoint record
shared secret is generated  (or nothing if was blank) and stored in the remote endpoint record
chosen local endpoint gets a new Peer added to its Peers (or nothing if peering field was blank)
.config file with the name of remote Endpoint is generated:

new generated (remote) private key is stored in the .config
DNS field is stored in the .config
selected local endpoint public key is stored in the .config (or left with a placeholder if blank)
shared secret is stored in the .config (if selected and generated)
Public address of selected local endpoint is stored in the .config

The .config file is downloaded to the browser and deleted on OPNSense
Structure of downloaded .config file:
[Interface]
PrivateKey = <newly generated private key of Remote endpoint>
Address = <taken from the IP Address field>
DNS = <taken from DNS field>

[Peer]
PublicKey = <chosen local endpoint' public key - or dummy placeholder if blank>
PresharedKey = <newly generated shared secret - or ommit if blank>
AllowedIPs = <IP_ADDRESS>/0
Endpoint = <taken from Local Endpoint public address - or dymmy placeholder if blank>

To simplify the feature request:
Change VPN - WireGuard - Endpoints - Edit Local Configuration form:

Add Endpoint public address (allow URL or IPv4 or IPv6)

Change VPN - WireGuard - Endpoints - Edit Endpoint form:

Add a drop-down box with all peering local endpoint (can stay blank)
Add DNS field
Add IP Address field
Add (and pre-select) a checkbox "Generate a new Key Pair"
Add (and keep blank) a checkbox "Generate a new Shared Secret"

When clicking Save button:

a new key pair for a remote endpoint is generated
the new public key is inserted is stored in the remote endpoint record
shared secret is generated  (or nothing if was blank) and stored in the remote endpoint record
chosen local endpoint gets a new Peer added to its Peers (or nothing if peering field was blank)
.config file with the name of remote Endpoint is generated:

new generated (remote) private key is stored in the .config
DNS field is stored in the .config
selected local endpoint public key is stored in the .config (or left with a placeholder if blank)
shared secret is stored in the .config (if selected and generated)
Public address of selected local endpoint is stored in the .config

The .config file is downloaded to the browser and deleted on OPNSense
Structure of downloaded .config file:
[Interface]
PrivateKey = <newly generated private key of Remote endpoint>
Address = <taken from the IP Address field>
DNS = <taken from DNS field>

[Peer]
PublicKey = <chosen local endpoint' public key - or dummy placeholder if blank>
PresharedKey = <newly generated shared secret - or ommit if blank>
AllowedIPs = <IP_ADDRESS>/0
Endpoint = <taken from Local Endpoint public address - or dymmy placeholder if blank>

I find this bit the most confusing - it's bit me several times and did so again, hours of troubleshooting.
add new peer in OPNsense Local settings of Wireguard
If the endpoint is enabled, shouldn't that mean that it's a peer?  Nope.  It should though.. Or at least in the Peer section of the local, give an option for ALL enabled endpoints.

What about configurations that have multiple WG endpoints on different ports? I have two local endpoints and 7 peers - and not all peers are enabled for both endpoints...

Good point.. Then yea, an ALL option would be nice for those of us that have one.

Or maybe something in the endpoint itself, that allows you to add and remove it from different local peer configs.  It's just confusing that you have this in two places and there is that extra step of adding a client.


Automatic support for large ONNX models in ORTOptimizer

Previously we had errors as
Traceback (most recent call last):
  File "/home/user/miniconda3/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/user/miniconda3/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/user/scripts/optimum/optimum/exporters/onnx/__main__.py", line 255, in <module>
    main()
  File "/home/user/scripts/optimum/optimum/exporters/onnx/__main__.py", line 212, in main
    optimizer.optimize(save_dir=args.output, optimization_config=optimization_config, file_suffix="")
  File "/home/user/scripts/optimum/optimum/onnxruntime/optimization.py", line 192, in optimize
    optimizer.save_model_to_file(output_path.as_posix(), use_external_data_format, one_external_file)
  File "/home/user/miniconda3/lib/python3.10/site-packages/onnxruntime/transformers/models/gpt2/../../onnx_model.py", line 994, in save_model_to_file
    OnnxModel.save(self.model, output_path, use_external_data_format, all_tensors_to_one_file)
  File "/home/user/miniconda3/lib/python3.10/site-packages/onnxruntime/transformers/models/gpt2/../../onnx_model.py", line 984, in save
    save_model(model, output_path)
  File "/home/user/miniconda3/lib/python3.10/site-packages/onnx/__init__.py", line 231, in save_model
    s = _serialize(proto)
  File "/home/user/miniconda3/lib/python3.10/site-packages/onnx/__init__.py", line 75, in _serialize
    raise ValueError(
ValueError: The proto size is larger than the 2 GB limit. Please use save_as_external_data to save tensors separately from the model file.

because the argument use_external_data_format was not automatically set automatically for optimizer.save_model_to_file. Here we use a hopefully decent heuristic that will help to avoid errors in the optimization & export for large models. One slight issue is that optimizer.save_model_to_file saves as .data the external data instead of .onnx_data, the convention we took in the ONNX export. I believe this is a non-issue, just style.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


Moving files out of root to make things easier to find.

Linked Issue:

Checklist:

[ ] Is the linter reporting 0 errors?
[ ] Did you comply with our styleguide?
[ ] Is there adequate test coverage for your new code?
[ ] Does the CI pipeline pass?
[x] Did you update the documentation?
[ ] Did you update the sample app?
[ ] Do we need to increment the minor/major version?
[ ] Did you change the public API?
[ ] Have you done everything you can to make sure that errors that can occur are compile-time errors, and if they have to be runtime do you have adequate tests and documentation around those runtime errors? For more details.

If Applicable:

[ ] Did you test when the first item is skipped?
[ ] Did you test when the last item is skipped?
[ ] Did you test when middle items are skipped?
[ ] Did you test when incorrect data is passed forward?
[ ] Did you test proceeding backwards?

If Public API Has Changed:

[ ] Did you deprecate (rather than remove) any old methods/variables/etc? Our philosophy for deprecation.
[ ] Have you done the best that you can to make sure that the compiler guides people to changing to the new API? (Example: the renamed attribute)
[ ] If necessary, have you tested the upgrade path for at least N-1 versions? For example, if data persists between v1 and v2 then that upgrade should be tested and as easy as we can make it.

Codecov Report

Merging #60 (1a1ecae) into main (7f19d72) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main      #60   +/-   ##
=======================================
  Coverage   97.25%   97.25%
=======================================
  Files          35       35
  Lines         985      985
=======================================
  Hits          958      958
  Misses         27       27

Impacted Files
Coverage Δ

...SwiftCurrent/TypeErased/AnyFlowRepresentable.swift
87.23% <ø> (ø)

Sources/SwiftCurrent/TypeErased/AnyWorkflow.swift
85.29% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3ce7ad3...1a1ecae. Read the comment docs.


Input redesign

Rough idea: give Game a key map (something like a fn(Key) -> Option<Evt>), Game then forwards Evt to State, which does whatever it needs with it. This would eliminate the need for InputCache (at least for the keyboard side), potentially making things simpler, and make for a natural place for a key map (which I don't currently have at all, key functions are hardcoded which is bad).
I'm not 100% sure this is good, but definitely try it out.

Implemented a keymap on top of the current InputCache in Flamegrower and I'm pretty satisfied with how things are going with it. Putting this away


Be able to configure conditional branches

resolves manheim/terraform-pipeline#172

Codecov Report

Merging #325 (affa2f1) into master (30710d2) will increase coverage by 0.03%.
The diff coverage is 83.33%.

@@            Coverage Diff             @@
##           master     #325      +/-   ##
==========================================
+ Coverage   79.13%   79.17%   +0.03%
==========================================
  Files          43       43
  Lines        1107     1109       +2
  Branches      262      262
==========================================
+ Hits          876      878       +2
  Misses         99       99
  Partials      132      132

Impacted Files
Coverage Δ

src/ConditionalApplyPlugin.groovy
68.75% <83.33%> (+4.46%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 30710d2...affa2f1. Read the comment docs.

Codecov Report

Merging #325 (affa2f1) into master (30710d2) will increase coverage by 0.03%.
The diff coverage is 83.33%.

@@            Coverage Diff             @@
##           master     #325      +/-   ##
==========================================
+ Coverage   79.13%   79.17%   +0.03%
==========================================
  Files          43       43
  Lines        1107     1109       +2
  Branches      262      262
==========================================
+ Hits          876      878       +2
  Misses         99       99
  Partials      132      132

Impacted Files
Coverage Δ

src/ConditionalApplyPlugin.groovy
68.75% <83.33%> (+4.46%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 30710d2...affa2f1. Read the comment docs.

I tried to follow the initial suggestion on issue #172 e.g. ConditionalApplyPlugin.withBranchApplyEnabledFor('qa').init() but I'm new to the terraform-pipeline project and didn't understand the intent of calling .init() at the end. I figured I'd take a stab at it and get feedback afterwards.

I tried to follow the initial suggestion on issue #172 e.g. ConditionalApplyPlugin.withBranchApplyEnabledFor('qa').init() but I'm new to the terraform-pipeline project and didn't understand the intent of calling .init() at the end. I figured I'd take a stab at it and get feedback afterwards.

I wanted to design the library in a way that allowed plugins to be enabled or disabled, as the user sees fit.  I also wanted the enable/disable to be done as simply as possible - a static method fit the bill.
So init() is the convention that I came up with, to enable a plugin - without calling init(), a plugin should not have any effect on your pipeline, with the exception of a handful of "default" plugins (which includes ConditionalApplyPlugin) See: https://github.com/manheim/terraform-pipeline/blob/affa2f11c59bde1c388209b681b37d64327835a9/src/TerraformEnvironmentStage.groovy#L7
This is definitely an awkward implementation in hindsight.  A better implementation could have treated ConditionalApplyPlugin (and every default plugin), like any other plugin, and had it implement an init() method.  That init() method could then have been called in Jenkinsfile.init(), and achieved the same current behavior, without the code being any different from any other plugin.

I wanted to design the library in a way that allowed plugins to be enabled or disabled, as the user sees fit.  I also wanted the enable/disable to be done as simply as possible - a static method fit the bill.
So init() is the convention that I came up with, to enable a plugin - without calling init(), a plugin should not have any effect on your pipeline, with the exception of a handful of "default" plugins (which includes ConditionalApplyPlugin) See: https://github.com/manheim/terraform-pipeline/blob/affa2f11c59bde1c388209b681b37d64327835a9/src/TerraformEnvironmentStage.groovy#L7
This is definitely an awkward implementation in hindsight.  A better implementation could have treated ConditionalApplyPlugin (and every default plugin), like any other plugin, and had it implement an init() method.  That init() method could then have been called in Jenkinsfile.init(), and achieved the same current behavior, without the code being any different from any other plugin.

The implementation looks good.  Can you update the CHANGELOG with your Issue?  I'm also about to cut a release on master, and this change can go into v5.14, the next release.

The implementation looks good.  Can you update the CHANGELOG with your Issue?  I'm also about to cut a release on master, and this change can go into v5.14, the next release.

Hrm, I think I spent more time thinking/talking about the init method, that I didn't look into the implementation details enough.  I think what you implemented is different from the intent outlined in the issue.
Your change will allow apply to run on all environments so long as the branch names match.
The intent described in the issue is that apply can run on a specific environment, regardless of what the branch is named.
Those are two different things.  I pointed you in the direction of the existing Issue - is your UseCase different from what was outlined?  If so, we should open a different issue.

Hrm, I think I spent more time thinking/talking about the init method, that I didn't look into the implementation details enough.  I think what you implemented is different from the intent outlined in the issue.
Your change will allow apply to run on all environments so long as the branch names match.
The intent described in the issue is that apply can run on a specific environment, regardless of what the branch is named.
Those are two different things.  I pointed you in the direction of the existing Issue - is your UseCase different from what was outlined?  If so, we should open a different issue.

Per the discussion about the awkwardness of init and ConditionalApplyPlugin, I put in Issue #327

Per the discussion about the awkwardness of init and ConditionalApplyPlugin, I put in Issue #327

After chatting with @kmanning I realize I misinterpreted the goal of this issue. My PR does address half of issue #25 however, so I'm planning to update the PR for that issue.

After chatting with @kmanning I realize I misinterpreted the goal of this issue. My PR does address half of issue #25 however, so I'm planning to update the PR for that issue.

Other than the links in the ChangeLog, the changes look good.

Other than the links in the ChangeLog, the changes look good.


Debugging my cli+gateway setup

I set up the gateway on fly.io + Ably + Courier
I got the cli app running on my laptop.
The emails don't arrive but I can see them in the Courier logs so at least I was able to get the activation code.
After that, the timeld open my-timesheet command took a long time and then finally printed:
Fetch from https://frosty-hill-2761.fly.dev/api/key/michiel failed with 500: Internal Server Error

On https://fly.io/apps/frosty-hill-2761/monitoring I only see
2022-07-06T07:59:52.894 app[76391911] lhr [info] restify listening at http://[::]:8080

as the last log entry, so that's not helpful.
IIUC fly.io is heroku under the hood, so I would expect error messages about 500 errors to show up in the application log?

Ah, flyctl logs.

Hm, now when I try timeld open my-timesheet again it says:
Please enter your email address to register this device<EMAIL_ADDRESS>
again (probably because the first time it resulted in a 500?) but then it doesn't continue to "Check your email and paste the code", but instead immediately prints:
Fetch from https://frosty-hill-2761.fly.dev/api/jwe/michiel?email=michiel%40pondersource.com failed with 401: Unauthorized

Ah, worked around that by using timeld -u test open my-timesheet.

Hm, as expected that lead to:
Fetch from https://frosty-hill-2761.fly.dev/api/key/test failed with 500: Internal Server Error

But unfortunately no errors were logged in flyctl logs.

Addressed error handling in
https://github.com/m-ld/timeld/tree/42-server-error-reporting
&
https://www.npmjs.com/package/timeld-gateway/v/0.2.2-0
I have not yet tried the fresh deploy, so no problem if you don't have time to test it.

Addressed error handling in
https://github.com/m-ld/timeld/tree/42-server-error-reporting

That first link appears to be broken.  A quick search of the repo in case it had been moved revealed no similar content, hence it's either been renamed or deleted entirely.  Could you update the link accordingly?

Closing as stale


delta university for science and technology

@HqNw First stage of verification passed, official domain is deltauniv.edu.eg as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision.

@HqNw This domain is already added to the repository. I close the request as a duplicate.


Fix Issues in SavedHTTPAttachment

What
Correct SavedHTTPAttachment's expectations of what is present in a JSON object for attachments from couch.
How
Instead of the code path being based on stub being present, it is now based on the presence of data. length is correctly worked out based on the information provided in the data field if present or by
using the length field of the attachment objects.
reviewer @tomblench
reviewer @brynh

Typos in original commit comment, would also be good if you explicitly spelled out what we were mishandling previously in the comment.

+1 after my latest comments

+1


Can I create schema.json automatically using apollo-codegen?

i want to execute apollo-codegen at project build.
Is there a way? 🤔

but what do you want to get as the output of codegen?

He wants to have the code gen output a schema in a location that apollo will pickup.

I believe we have this gradle task ApolloSchemaIntrospectionTask.

Example described in description https://github.com/apollographql/apollo-android/pull/461

He wants to have the code gen output a schema in a location that apollo will pickup.

yes!
@sav007 thanks, i've created task!
by the way, Can I change url / output strings each ProductFlavor?

Don't know never tried :)

I don't think you will be able to do this with flavors due to flavors being an android specific construct while apollo gradle plugin does not depend on android plugin.  gradle.properties or cli is probably best you can do to drive the url