Added newegg.ca to supported stores

Description
Adds newegg.ca to supported stores
Discord is xdMatthewbx#1337
Testing
Tested running with newegg-ca added to stores list and am not encountering errors. Additionally stock alerts appear to be functioning but that may not be the case as none have actually come in stock yet.

LGTM. Should we add .CA stores to the section in readme maybe?

I believe I updated the readme to include newegg.ca

@xdMatthewbx since you are already in the Readme.md it might not be a terrible idea to add a note that this bot has the potential to use a lot of data, so people with data caps need to be mindful.  Someone posted a link of their data usage in discord and it's upwards of 80GB

LGTM. Should we add .CA stores to the section in readme maybe?

Thanks everyone!


Template parse errors: 'ion-tab-button' is not a known element:

Bug Report
Ionic Info
[ERROR] Error loading @ionic/angular-toolkit package.json: Error: Cannot find module '@ionic/angular-toolkit/package'

Ionic:

   ionic (Ionic CLI)             : 4.3.1 (C:\Users\name\AppData\Roaming\npm\node_modules\ionic)
   Ionic Framework               : @ionic/angular 4.0.0-beta.12
   @angular-devkit/build-angular : 0.7.5
   @angular-devkit/schematics    : 0.7.5
   @angular/cli                  : 6.1.5
   @ionic/angular-toolkit        : not installed

Cordova:

   cordova (Cordova CLI) : not installed
   Cordova Platforms     : not available
   Cordova Plugins       : cordova-plugin-ionic-keyboard 2.1.3, cordova-plugin-ionic-webview 2.1.4, (and 9 other plugins)

System:

   Android SDK Tools : 26.1.1 (C:\Users\name\AppData\Local\Android\sdk)
   NodeJS            : v10.1.0 (C:\Program Files\nodejs\node.exe)
   npm               : 5.6.0
   OS                : Windows 10

Describe the Bug
A. Console reports error after running "ionic serve":
"Template parse errors: 'ion-tab-button' is not a known element: ..." ;
B. Visual Studio Code reports:
"[Angular]
'ion-tab-bar' is not a known element:
1. If 'ion-tab-bar' is an Angular component, then verify that it is part of this module.
2. If 'ion-tab-bar' is a Web Component then add 'CUSTOM_ELEMENTS_SCHEMA' to the
<EMAIL_ADDRESS>of this component to suppress this message."
Steps to Reproduce
Steps to reproduce the behavior:

Go to https://beta.ionicframework.com/docs/api/tabs/  and copy paste the tab code found under the 'Usage' heading into any lazy or eagerly loaded component
run 'ionic serve'
see error in console
see error in visual studio code

Related Code
code gives error in any component:

Schedule

<ion-tab-button tab="speakers" href="/app/tabs/(speakers:speakers)">
  <ion-icon name="contacts"></ion-icon>
  <ion-label>Speakers</ion-label>
</ion-tab-button>

<ion-tab-button tab="map" href="/app/tabs/(map:map)">
  <ion-icon name="map"></ion-icon>
  <ion-label>Map</ion-label>
</ion-tab-button>

<ion-tab-button tab="about" href="/app/tabs/(about:about)">
  <ion-icon name="information-circle"></ion-icon>
  <ion-label>About</ion-label>
</ion-tab-button>

Expected Behavior
ion-tab-button should be a known component,.
Also there is an error after running ionic info "[ERROR] Error loading @ionic/angular-toolkit package.json: Error: Cannot find module '@ionic/angular-toolkit/package' " --> must i run npm install @ionic/angular-toolkit/package  ?
Additional Context
related issue: https://stackoverflow.com/questions/46612262/ionic3-template-parse-error-in-custom-component
related issue: https://stackoverflow.com/questions/39428132/custom-elements-schema-added-to-ngmodule-schemas-still-showing-error
related issue: https://github.com/ionic-team/ionic/issues/15942
related issue: https://github.com/ionic-team/ionic/issues/14874
my package.json:
"@ionic-native/core": "5.0.0-beta.21",
"@ionic-native/firebase": "^5.0.0-beta.21",
"@ionic-native/google-plus": "^5.0.0-beta.21",
"@ionic-native/splash-screen": "5.0.0-beta.21",
"@ionic-native/status-bar": "5.0.0-beta.21",
"@ionic/angular": "4.0.0-beta.12",

I ran npm install @ionic/angular --save but the package.json remains at "@ionic/angular": "^4.0.0-beta.12".
Please advise how to update to beta 15.

For anyone else who needs to know I ran npm install<EMAIL_ADDRESS>--save and that has updated the package.json

After running npm install<EMAIL_ADDRESS>--save the browser console is no longer giving an error but visual studio code is still giving errors?

This doesn't help


Functional genomic hypothesis generation and experimentation by a robot scientist

https://www.nature.com/articles/nature02236

@AlexanderTitus Check out the publication date on this one

@mwalsh52 it looks like people have been doing this for a WHILE. Good find.


redis.exceptions.ConnectionError in clashleaders/rq_worker.py:15

Error in ClashLeaders
redis.exceptions.ConnectionError in clashleaders/rq_worker.py:15
Error -2 connecting to redis:6379. Name or service not known.
View on Bugsnag
Stacktrace
clashleaders/rq_worker.py:15 - main

View full stacktrace
Created automatically via Bugsnag

An error linked to this issue has been reopened in Bugsnag
redis.exceptions.ConnectionError in clashleaders/rq_worker.py:15


Feature: Upgrade notification 2 - LTS

This is a follow up to #7652 and relates to issues #7336 and #7405.
This PR adds an option to show the upgrade notification only when the currently installed version is behind the latest LTS release. The actually latest version is ignored.
The following examples should illustrate the display behaviour of the upgrade notification with the new option set:

Current version
Latest version
Latest LTS version
Show upgrade notification
Description

2.10.2
2.10.3
2.7.13
False
LTS older than current

2.10.2
2.11
2.11
True
LTS is newer than current

2.10.2
2.12
2.11
True
LTS is newer than current, although older than latest

2.11 (LTS)
2.11.1
2.11.1
True
LTS is newer than current

2.11.1 (LTS)
2.12
2.11.1
False
LTS same as current, although older than latest

2.11.1 (LTS)
2.12
2.11.2
True
LTS newer than current, although older than latest

2.11.2 (LTS)
2.13
2.11.2
False
LTS same as current, although older than latest

2.11.2 (LTS)
2.14
2.11.2
False
LTS same as current, although older than latest

2.11.2 (LTS)
2.15
2.15
True
LTS newer than current

Closes #7336

Thanks for contributing to Wagtail! 🎉
Before submitting, please review the contributor guidelines https://docs.wagtail.io/en/latest/contributing/index.html and check the following:

Do the tests still pass? (https://docs.wagtail.io/en/latest/contributing/developing.html#testing) -> Yes
Does the code comply with the style guide? (Run make lint from the Wagtail root) -> Yes
For Python changes: Have you added tests to cover the new/fixed behaviour? -> Yes
For front-end changes: Did you test on all of Wagtail’s supported environments? -> No

Please list the exact browser and operating system versions you tested.
Please list which assistive technologies you tested.

For new features: Has the documentation been updated accordingly? -> Yes

cc @nimasmi

Note: frontend CI will fail but have these other PRs to resolve it
https://github.com/wagtail/wagtail/pull/8591
https://github.com/wagtail/wagtail/pull/8577

Thanks for checking,merging and fixing @lb-

for completeness - additional validations

default setting - showing the 2.16.2 as the latest version but linking to the 2.16 release notes

LTS setting - showing as 2.15.5 but linking to the 2.15 release notes

I have re-worded the release notes to be more clear about this difference 52e87968e2dad66eb0b35d47f7e95ddfcd1bf43e

Ah yes, good call. Thanks you @lb-


Unable to locate signup for xamarin newsletter

I am receiving a monthly newsletter (it is great) and I am looking to sign up another email address. But after some Googling and plenty of "smartphone browsing" I can not seem to find where to do this.
My guess is that it comes with my VS account, but when I log in there the box for newsletter is unchecked, so I am unsure.
The latest email read "Welcome to the February 2019 edition of the Xamarin newsletter."... And originated from<EMAIL_ADDRESS>It would be nice to be able to sign up for these xamarin newsletter updates without an account.
Thanks

@jkarlsso given that this is not directly to docs.microsoft.com, we cannot address this issue in this repository. I would recommend reaching out on the Xamarin Forums regarding your questions on the Xamarin platform: https://forums.xamarin.com/


Documentation release version mismatch

The version mentioned in the official documentation is 0.16.4 but the latest release/tag on Github is 0.16.3 which is also listed on Ansible Galaxy. Was 0.16.4 perhaps meant to be released but got stuck in the process?

You are looking at the docs for the main branch, the main branch contains the latest (unreleased/upcoming) changes

Yes, but it's also what the link on Ansible Galaxy redirects to: https://prometheus-community.github.io/ansible/

Ah, good point, I'll try to get that changed.


Added optional login hint parameter for OAuth2Helper constructor

I've only added the changes I needed for the Authorization Code grant to work using a Custom OAuth2Client and the OAuth2Helper. Additional development may be needed to support the Client Credentials grant.
The purpose for this feature is to allow a username field and login button to be displayed in the app.
Additionally, if your Authorization Server is setup in such a way, you could have it auto-route you to a particular Identity Provider based on the login_hint that was sent in the authorize request (e.g<EMAIL_ADDRESS>could auto-route to My Company IdP)
Forgive me if I've done something wrong as this is my first pull request ever. Thanks

Hi @rileypikus, sorry for the late reply...
You can achieve the same result by adding the "login_hint" parameter to the "authCodeParams" map, for example:
var client = MyOAuth2Client(...);
var tokenResp = client.getTokenWithAuthCodeFlow(
	clientId: 'your_client_id',
	scopes: ['scope1', 'scope2', ...],
        authCodeParams: {
             "login_hint": "your_login_hint"
        });

Or, if you're using the helper, you can set it this way:
var client = MyOAuth2Client(...);
var hlp = OAuth2Helper(client,
	grantType: OAuth2Helper.AUTHORIZATION_CODE,
	clientId: 'your_client_id',
	scopes: ['scope1', 'scope2', ...],
        authCodeParams: {
             "login_hint": "your_login_hint"
        });

Ahhh that is very helpful. I didn't see that anywhere on the README, so I just went forth with implementing it myself.


Add support for custom color maps, some predefined ones

I recently wanted to compare Viridis with Plasma for a plot and change the colormap to be white as the zero value.
Thought I'd clean this up and add support for easier handling of custom colormaps.
Instead of playing around with the JsonNode after converting to PlotJson (or serializing), one can either assign one of the predefined custom colormaps:

Plasma
Magma
Inferno

each with and without zero value as white

Viridis with zero as white
white to black instead of plotly's "Greys", which is black-to-white) by using the

trace.customCmap = getCustomMap(<field of PredefinedCustomMaps>)

or create a CustomColorMap object and assign it. The fields of it are rawColors, which are just a sequence of tuple[r, g, b: float] in the range of 0.0 .. 1.0. The custom maps can consist of N values. They will be taken to cover the whole data range.
The colormap range can additionally be restricted (or enlarged) via the zmin and zmax fields (only for heatmap, was not sure if I should add it for Countour?).
The normal colormap field has to be set to Custom for the custom map to be considered.
Alternatively the colormap sugar function can be used on a Plot[T] object. This is more convenient imo. One can hand one of:

ColorMap: enum: Plotly's default colormaps
PredefinedCustomMap: enum on of the predefined custom maps
CustomColorMap a custom colormap object
rawColors a sequence of raw colors as shown above

See the fig20 example and the test case for explicit usage.

Renamed the fields.


fix: no records behavior

PR https://github.com/projectdiscovery/retryabledns/pull/222 included in release https://github.com/projectdiscovery/retryabledns/releases/tag/v1.0.75 introduced a breaking change:
When no record are found, the error ErrRetriesExceeded is returned instead of an empty response.
This PR restore the previous behavior.

v1.0.8 has been released since my last message.
@calab33p as you are the author of the original PR, what do you think of this change?

@ehsandeep @Mzack9999 can we merge this for the next release?


Replace invalid DOI IRIs in xrefs with doi: CURIE strings.

These values were stored as IRIs with the invalid protocol DOI.

I am tagging @bvarner-ebi and @aleixpuigb as the core maintainers to check this too.


nixos/transmission: Enable configuring permissions of home dir

Motivation for this change
Continuing from @dsg22's work in https://github.com/NixOS/nixpkgs/commit/077934e1928f7cbfa7f4391f960dce3807515c5e, today I noticed that with a downloadDir pointing e.g to  ${homeDir}/Downloads (The default), it's impossible to cd to it since the ${homeDir} permissions are always set to 700. It's funny this hasn't bothered anyone up until now.. With these changes, the default allows 750 access to the ${homeDir} and the ${downloadDir} so at least by default users in the transmission group will be able to access it.
Also, I did some small variables renames I think improve the readability of the service, hope it's OK with you.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[x] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Notify maintainers
cc:

@dsg22 - The author of https://github.com/NixOS/nixpkgs/commit/077934e1928f7cbfa7f4391f960dce3807515c5e .
@coreyoconnor - you seem to be the maintainer of this module according to the test file.
@Lassulus - Since it seems you merged @dsg22's PR.

Another problem is when downloadDir is changed and if transmission user can not create the new directory, the service fails to start. This is especially important if the destination directory is owned by root (for example /home/mytransmission). I have made some changes to address the issue on top of your commit: https://github.com/bb2020/nixpkgs/commit/3f5e465bf8b3468fdb9f4a570b801b1f90376d3d

I'm not sure I figured out what you mean. Are you saying that the preStart script creating the necessary directories is being run by transmission as this is the user of the systemd service? Meaning that if the user transmission can't create these directories then the service will fail?
I'm not sure how exactly your changes improve that, nothing there elevates the permissions of the preStart script so that the mkdir command is being run root. Besides, I think if someone chooses such a non standard directory as a download-dir, it's rather reasonable that he will manually create it and then the preStart script will only set the permissions.

transmission user is not a super user, so it can not create necessary folders when non-default downloadDir is used. By running preStart as root, this can be solved. I've modified my comment above to include only two more lines.

Thanks for clarifying this @bb2020. The question is, are you sure the preStart script won't fail because the ExecStartPre is being run by transmission which is not privileged? Also what does this line do:
https://github.com/bb2020/nixpkgs/blob/c0cac417e4ad094dbd9a7ca2776d97b409722b8c/nixos/modules/services/torrent/transmission.nix#L141
?
As for this privileges doubt, according to systemd.service's man page (see this table), we should use the + as a prefix to the ExecStartPre and this will ensure the ExecStartPre script will be able to actually mkdir this directories.

This reminds me that @aszlig wrote about a "directories" module in https://github.com/NixOS/nixpkgs/pull/59988#issuecomment-533868004. I think having something like that in NixOS would be nice, and help with issues like this.

today I noticed that with a downloadDir pointing e.g to ${homeDir}/Downloads (The default), it's impossible to cd to it since the ${homeDir} permissions are always set to 700

If you ask me, introducing homeDirPermissions  is not necessary. Assigning homeDir dir to 770 and settingsDir to 700 should be sufficient.

what does this line do: https://github.com/bb2020/nixpkgs/blob/c0cac417e4ad094dbd9a7ca2776d97b409722b8c/nixos/modules/services/torrent/transmission.nix#L141

PermissionsStartOnly runs preStart as root, but not the service process itself. However it looks like it is deprecated and similar modules have been ported to systemd.tmpfiles.
I've made some changes here that fixes both issues, I presume. https://github.com/bb2020/nixpkgs/commit/a06e0a1408bba759b603d3f553320832308b06f1

Now that's the systemd way of handling this! Kind of makes @aszlig's thing redundant doesn't it? @bb2020 would you like to create a new PR? I'm off my laptop at the moment...

It is actually a discouraged method. However, I see it is widely used by many modules 😃
I don't have time to test it though. I was hoping you would test it and go on with this PR.

Discouraged? That's interesting.. Have you discovered it's discouraged in an RFC perhaps?

It was previously discussed in some nixpkgs issues.

It'd be nice to talk it over at nixos/RFCs, or the forums.. When I'll find the time I'll quote you so you could join the discussion :)
P.S I'll test your implementation as well.

BTW Speaking of improvements to this module, does anybody has a clear guess on how to fix this warnings I've been getting lately:
trace: warning: In file /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/services/torrent/transmission.nix
a list is being assigned to the option config.users.users.
This will soon be an error as type loaOf is deprecated.
See https://git.io/fj2zm for more information.
Do
  users.users =
    { transmission = {...}; ...}
instead of
  users.users =
    [ { name = "transmission"; ...} ...]

trace: warning: In file /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/services/torrent/transmission.nix
a list is being assigned to the option config.users.groups.
This will soon be an error as type loaOf is deprecated.
See https://git.io/fj2zm for more information.
Do
  users.groups =
    { transmission = {...}; ...}
instead of
  users.groups =
    [ { name = "transmission"; ...} ...]

I would prefer this over https://github.com/NixOS/nixpkgs/pull/78113 since i want transmission group members to be able to write/move files inside the transmission directory

This reminds me that @aszlig wrote about a "directories" module in #59988 (comment). I think having something like that in NixOS would be nice, and help with issues like this.

For thinks like eg. runtime or state directories, systemd can already handle this (as @doronbehar mentioned), but AFAIK it only applies to those standard directory layouts and not arbitrary directories. The module above however is fairly complex (mainly because it also supports POSIX ACLs), so I'm not really sure whether it would be worth introducing additional complexity if it's only used for like 2 modules.

The module above however is fairly complex (mainly because it also supports POSIX ACLs), so I'm not really sure whether it would be worth introducing additional complexity if it's only used for like 2 modules.

Perhaps a module like that might have more user out-of-tree than in-tree? I know that personally I would use it out-of-tree (for my nixos configs).

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/nixpkgs-policy-as-for-systemd-prestart-setup-scripts-vs-systemd-tmpfiles/5839/1

I'm closing this as I will (some when in the future) create a PR with broader changes for permissions of services/modules - as mentioned in that discourse thread^ (https://discourse.nixos.org/t/nixpkgs-policy-as-for-systemd-prestart-setup-scripts-vs-systemd-tmpfiles/5839/1).
In the meantime, I'm using:
    systemd.services.transmission.serviceConfig.ExecStartPre = pkgs.lib.mkForce null;


MySQL password should not be kept in repo

In recreate_mysql_root_user there is a password hanging around. Probably it's not the one you use in production but just in case i wanted to let you know :)

Yeah, I'm even wondering what is this repo :) I'll archive it, but thanks for the hint!


[SPARK-16921][PYSPARK] RDD/DataFrame persist()/cache() should return Python context managers

JIRA: https://issues.apache.org/jira/browse/SPARK-16921
Context managers are a natural way to capture closely related setup and teardown code in Python. It can be useful to apply this pattern to persisting/unpersisting RDDs and DataFrames.
This PR makes RDDs and DataFrames implement the context manager __enter__ and __exit__  functions, allowing code such as:
with labeled_data.persist():
    model = pipeline.fit(labeled_data)

How was this patch tested?
New doc tests.

Test build #63520 has started for PR 14579 at commit 2b4e56e.

Test build #63520 has finished for PR 14579 at commit 2b4e56e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63520/
Test PASSed.

cc @nchammas @holdenk @rxin
Note:
This is implemented by adding the __enter__ and __exit__ methods to RDD/DataFrame directly. This allows some potentially weird stuff, such as any instance of RDD/DF, including any method returning self, can be used in a with statement, e.g. this works:
with rdd.map(lambda x: x) as x:
    ...

Clearly this doesn't make a lot of sense. However, I looked at the 2 options of (a) a separate context manager wrapper class returned by persist; and (b) trying to dynamically add the methods (or at least __enter__) in persist.
The problem with (a) is that persist needs to return an RDD/DF instance, so this breaks chaining behavior such as rdd.cache().count() etc.
The problem with (b) is that the special method __enter__ is called in the context of with as type(rdd).__enter__(rdd) (see PEP 343). So it does not help to add a method dynamically to an instance, it must be done to the class. In this case, then after the first with statement usage, all existing and future instances of RDD/DF have the __enter__ method, putting us in the same situation as the approach in this PR of having the methods defined on the class (with associated allowed "weirdness").
So, if we want to avoid that, the only option I see is a variant of (a) above - adding a cached/persisted method that returns a context manager, so it would look like this:
with cached(rdd) as x:
    x.count()

This is less "elegant" but more explicit.
Any other smart ideas for handling option (b) above, please do shout!

Thanks @MLnick for taking this on and for breaking down what you've found so far.
I took a look through contextlib for inspiration, and I wonder if the source code for closing() offers a template we can follow that would let persist() return an RDD/DataFrame instance with the correct magic methods, without having to modify the class.
Have you taken a look at that?

@nchammas I looked at the @contextmanager decorator. It is an easy way to create a method that returns a context manager, but is is essentially only usable in a with statement as it returns a contextlib.GeneratorContextManager. For this use case it does not solve the issue that we need to return the RDD/DF instance from cache.
As far as I can see for the closing helper function, it is just a context manager itself, so the same as the final variant for option (a) mentioned above, i.e. with cached(rdd) as ...

Ah, you're right.
So if we want to avoid needing magic methods in the main RDD/DataFrame classes and avoid needing a separate utility method like cache(), I think one option available to us is to have separate PersistedRDD and PersistedDataFrame classes that simply wrap the base RDD and DataFrames classes and add the appropriate magic methods.
.persist() and .cache() would then return instances of these classes, which should satisfy the type(x).__enter__(x) behavior while still preserving backwards compatibility and method chaining.
What do you think of that?

None of our options seems great, but if I had to rank them I would say:

Add new Persisted... classes.
Make no changes.
Add separate persisted() or cached() utility method.
Modify base RDD and DataFrame classes.

Adding new internal classes for this use-case honestly seems a bit heavy-handed to me, so if we are against that then I would lean towards not doing anything.

cc @davies

One minor thing to keep in mind - the subclassing of RDD approach could cause us to miss out on pipelining if the RDD was used again after it was unpersisted - but I think that is a relatively minor issue.
On the whole I think modify base rdd and dataframe classes (option A / option 4) which is the one @MLnick has implemented here is probably one of the more reasonable options - the with statement doesn't add anything if the RDD/DataFrame isn't persisted but can do cleanup if it is.
But if there is a better way to do this I'd be excited to find out as well :)

the subclassing of RDD approach could cause us to miss out on pipelining if the RDD was used again after it was unpersisted

How so? Wouldn't __exit__() simply return the parent RDD or DataFrame object?

@nchammas so if we go with the subclassing approach but keep the current cache/persist interface (e.g. no special utility function) a user could easily write something like:
magic = rdd.persist() with magic as awesome:     awesome.count() magic.map(lambda x: x + 1)
I don't believe __exit__() could easily return a result that would updated magic to be rdd (infact __exit__() generally doesn't seem to return a result - instead its expected to do the teardown logic internally).

Sorry, you're right, __exit__()'s return value is not going to be consumed anywhere. What I meant is that unpersist() would return the base RDD or DataFrame object.
But I'm not seeing the issue with the example you posted. Reformatting for clarity:
magic = rdd.persist()

with magic as awesome:
    awesome.count()

magic.map(lambda x: x + 1)

Are you saying magic.map() will error? Why would it?
magic would be an instance of PersistedRDD, which in turn is a subclass of RDD, which has map() and all of the usual methods defined, plus the magic methods we need for the context manager.

yeah it would break pipelining - I don't think it will necessarily throw an error though.
e.g.
In [22]: rdd = sc.parallelize(["b", "a", "c"])

In [23]: type(rdd)
Out[23]: pyspark.rdd.RDD

In [24]: mapped = rdd.map(lambda x: x)

In [25]: type(mapped)
Out[25]: pyspark.rdd.PipelinedRDD

In [26]: mapped._is_pipelinable()
Out[26]: True

In [27]: p = mapped.cache()

In [28]: type(p)
Out[28]: pyspark.rdd.PersistedRDD

In [29]: p._is_pipelinable()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-29-02496125eccd> in <module>()
----> 1 p._is_pipelinable()

AttributeError: 'PersistedRDD' object has no attribute '_is_pipelinable'

In [30]: mapped2 = p.map(lambda x: x)

In [31]: type(mapped2)
Out[31]: pyspark.rdd.PipelinedRDD

So I think chaining will work, but the pipelined RDD thinks mapped2 is the 1st transformation, while it is actually the 2nd. I think this will just be an efficiency issue rather than a correctness issue however.
We could possibly work around it with some type checking etc but it then starts to feel like adding more complexity than the feature is worth...

Ah, I see. I don't fully understand how PipelinedRDD works or how it is used so I'll have to defer to y'all on this. Does the cached() utility method have this same problem?

We could possibly work around it with some type checking etc but it then starts to feel like adding more complexity than the feature is worth...

Agreed.
At this point, actually, I'm beginning to feel this feature is not worth it.
Context managers seem to work best when the objects they're working on have clear open/close-style semantics. File handles, network connections, and the like fit this pattern well.
In fact, the doc for with says:

This allows common try...except...finally usage patterns to be encapsulated for convenient reuse.

RDDs and DataFrames, on the other hand, don't have a simple open/close or try...except...finally pattern. And when we try to map one onto persist and unpersist, we get the various side-effects we've been discussing here.

Right I wouldn't expect it to error with subclassing - just not pipeline successfully - but only in a very long shot corner case.
I think the try/finally with persistance is not an uncommon pattern (we have something similar happen frequently inside of Spark ML/mllib but its in Scala code).

@nchammas a utility method (e.g. cached) - actually it will be a context manager class implemented in the same way as closing - will work because it only needs to return the context manager, not the RDD instance. So there is no chaining requirement, and it will only work in a with statement.

After looking at it and considering all the above, I would say the options are (1) do nothing; or (2) if we want to support this use case, then we implement a single utility method (I would say called persisted) that is a context manager.
Even though we could almost achieve things with subclassing, we do sort of break something and add too much risk/complexity vs reward of the feature IMHO.

So there is no chaining requirement, and it will only work in a with statement.

@MLnick - Couldn't we also create a scenario (like @holdenk did earlier) where a user does something like this?
persisted_rdd = persisted(rdd)
persisted_rdd.map(...).filter(...).count()

This would break pipelining too, no?
And I think the expectation would be for it not to break pipelining, because existing common context managers in Python don't have a requirement that they must be used in a with block.
For example, f = open(file) works fine, as does s = requests.Session(), and the resulting objects have the same behavior as they would inside a with block.

@nchammas to be clear - subclassing only breaks pipelining if the persisted_rdd is later unpersisted (e.g. used with a with statement or otherwise) - otherwise you can't pipeline on top of a persisted rdd anyways (which is why I say its a corner case).

Hmm, OK I see. (Apologies, I don't understand what pipelined RDDs are for, so the examples are going a bit over my head. 😅)

Sure - so at a bit of a high level and not like exactly on point - copying data out of the Python back to the JVM is kind of slow so if we have multiple python operations we can put together in the same task then we try and do this. Since caching is handled by storing the data inside of the JVM a cached RDD can't be pipelined since we need to copy the result to the JVM. You can see the details in the PipelinedRDD in rdd.py.

Thanks for the quick overview. That's pretty straightforward, actually! I'll take a look at PipelinedRDD for the details. 👍

@nchammas to answer your question above (https://github.com/apache/spark/pull/14579#issuecomment-239185438) - in short no.
The semantics of the utility method will be the same as for the closing example. persisted will return a wrapper class that implements the context manager methods. When entering the with statement, the __enter__ method is called, which returns the underlying rdd instance - this is in turn bound to the variable following as. So it would look something like this:
class persisted():
    def __init__(self, thing):
        self.thing = thing
    def __enter__(self):
        return self.thing
    def __exit__(self, *exc_info):
        self.thing.unpersist()

If someone tries to do persisted(rdd).map(...) it will throw an AttributeError.
The "file-like" version is what is currently implemented in this PR, and it works if __enter__ returns self which is what file et al do. Of course those classes seem to not tend to have other methods that return self so don't suffer the same chaining issue we run into with RDD.

@nchammas @holdenk @davies @rxin how about the approach of @MechCoder in https://github.com/apache/spark/pull/14579#discussion_r74813935?
I think this will work well, so we could raise an error to prevent (almost all I think) usages outside of the intended pattern of with some_rdd.cache() as x: or with some_rdd_already_cached as x:

I like it personally - if no one has a good reason why not it seems like a very reasonable approach.

Looks good to me. 👍

@MLnick still interested in updating this? (Just looking over the older Python PRs) :)

Yup! just been snowed under :( but will update with the approach above asap.

Just pinging to see how its going?

I hope to get to this soon - It's just the test cases that I need to get to also!

just a gentle ping - would be cool to add this for 2.1 we have the time :)

Is there any reason why it is not merged yet? I personally like this too.

Do you have time to update this @MLnick or maybe would it be OK if someone else made an updated PR based on this? It would be a nice feature to have for 2.2 :)

gentle ping.

@MLnick Hi, are you still working on this? If so, could you fix conflicts and update please?

@MLnick - or if you don't have a chance would it be ok for us to find someone (perhaps someone new to the project) to take this over and bring it to the finish line?

Hey all - sorry I haven't been able to focus on this. It shouldn't be tough to do, but it will need some tests. If we  can find someone who wants to take it over I think it makes a decent starter task :)


Updated truffle-verify + disabled verification for cloned contracts

I have removed the verification steps for cloned contracts, because when we verify the identity/proxy contract, the verification gets propagated to the cloned contracts. They have the exact same code.

Nice that makes sense. Thanks for fixing.

@jdville03 also tested with the deployment logs from museo, those contracts were all verified with success.


Automated cherry pick of #4295: Add unit test for pkg/agent/route
#4419: Set NO_FLOOD to IPsec tunnel ports
#4470: Fix that Service routes may get lost when starting on Windows
#4654: Restore NO_FLOOD to OVS ports after reconnecting the OVS
#4711: Fix route deletion for Service ClusterIP and LoadBalancerIP

Cherry pick of #4295 #4419 #4470 #4654 #4711 #4697 #4674 on release-1.8.
#4295: Add unit test for pkg/agent/route
#4419: Set NO_FLOOD to IPsec tunnel ports
#4470: Fix that Service routes may get lost when starting on Windows
#4654: Restore NO_FLOOD to OVS ports after reconnecting the OVS
#4711: Fix route deletion for Service ClusterIP and LoadBalancerIP
#4697: Fix windows image building error in jenkins pipeline
#4674:Ensure OVS port is valid before setting NO_FLOOD
For details on the cherry pick process, see the cherry pick requests page.

#4674 is missing, leading to same test failure

Golangci-lint check failed

/skip-e2e


Cleanup before tests

This PR changes the order between the cleanup of hdf5 files and the regression tests, so that the files are always cleaned up

bors r+


Update to update proxy sha with crash fix.

What this PR does / why we need it:
The latest proxy fixed envoy crash.
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
Release note:

NONE

@qiwzhang: Adding do-not-merge/release-note-label-needed because the release note process has not been followed.

One of the following labels is required "release-note", "release-note-action-required", or "release-note-none".
Please see: https://github.com/kubernetes/community/blob/master/contributors/devel/pull-requests.md#write-release-notes-if-needed.
Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

/lgtm

@qiwzhang: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-e2e.sh
7c60f6ae301b022eed0b3aa81a9f42187736c3c9
link
/test istio-pilot-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@qiwzhang: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-e2e.sh
7c60f6ae301b022eed0b3aa81a9f42187736c3c9
link
/test istio-pilot-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test istio-pilot-e2e

/retest


Oxyl forgets about the current song

Oxyl sometimes forgets a song is playing if there is nothing else in the queue, commands like stop and skip do not work then.

I need steps to reproduce


protorpc source distributions (.tar.gz) do not include subdirectories, e.g. "wsgi"

This results in errors like the following when using the App Engine "endpoints" library:
ERROR    2018-05-21 14:00:19,811 wsgi.py:263]
Traceback (most recent call last):
  File "/root/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/root/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/root/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
  File "/opt/root/src/appengine-cloudsql-benchmark/main.py", line 27, in <module>
    import endpoints
  File "/opt/root/src/appengine-cloudsql-benchmark/lib/endpoints/__init__.py", line 27, in <module>
    from .apiserving import *
  File "/opt/root/src/appengine-cloudsql-benchmark/lib/endpoints/apiserving.py", line 81, in <module>
    from protorpc.wsgi import service as wsgi_service
ImportError: No module named wsgi

The solution is to do make the following change to setup.py:
diff --git a/setup.py b/setup.py
index 2c83901..0ff14c0 100644
--- a/setup.py
+++ b/setup.py
@@ -18,7 +18,7 @@

 import platform

-from setuptools import setup
+from setuptools import find_packages, setup

 # Configure the required packages and scripts to install, depending on
 # Python version and OS.
@@ -34,9 +34,6 @@ if py_version < '2.6':
   REQUIRED_PACKAGES.append('simplejson')

 _PROTORPC_VERSION = '0.12.0'
-packages = [
-    'protorpc',
-]

 setup(
     name='protorpc',
@@ -46,7 +43,7 @@ setup(
     author='Google Inc.',
<EMAIL_ADDRESS>     # Contained modules and scripts.
-    packages=packages,
+    packages=find_packages(),
     entry_points={
         'console_scripts': CONSOLE_SCRIPTS,
         },

I'm using the exact same dependencies mentioned in the "official" GCP Python tutorial and I'm hitting this error as well.
import service_backend.main File "/Users/dgaedcke/dev/TouchstoneMicroservices/service_backend/main.py", line 7, in import endpoints as google_cloud_endpoints File "/Users/dgaedcke/dev/TouchstoneMicroservices/lib/endpoints/init.py", line 31, in from .apiserving import * File "/Users/dgaedcke/dev/TouchstoneMicroservices/lib/endpoints/apiserving.py", line 73, in from protorpc.wsgi import service as wsgi_service ImportError: No module named wsgi
All suggestions appreciated!


Proactive connection management

Features
Minimum connection pool size per endpoint
Using a system config, the application developer can specify the minimum no. of connections to be set for all endpoints for all physical partitions across all containers whose connections are opened using the openConnectionsAndInitCaches flow either through CosmosClient / CosmosAsyncClient or CosmosContainer / CosmosAsyncContainer. This can help alleviate the cold start latency associated with establishing connections on-demand when requests are made by the SDK.
Setting the minimum connection pool size per endpoint through a system config
// initialize the min. no. of connections required for all endpoints
int minConnectionPoolSizePerEndpoint = 5;
System.setProperty("COSMOS.MIN_CONNECTION_POOL_SIZE_PER_ENDPOINT", String.valueOf(minConnectionPoolSizePerEndpoint));

Aggressive connection establishment time duration
In the openConnectionsAndInitCaches flow for CosmosClient / CosmosAsyncClient, a duration can be specified within which connections are established aggressively in a blocking manner. Once this duration elapses, connections are established defensively but in a non-blocking manner. This improves on the previous flow which only opened connections in a blocking manner therefore preventing the SDK from making requests until all connections were established.
Setting the aggressive proactive connection establishment duration through the public API
// containers to which connections are to be proactively opened
CosmosContainerIdentity containerIdentity1 = new CosmosContainerIdentity("sample_db_id", "sample_container_id_1");
CosmosContainerIdentity containerIdentity2 = new CosmosContainerIdentity("sample_db_id", "sample_container_id_2");

// no. of regions to which connections are to be proactively opened
int proactiveConnectionRegionsCount = 1;

// duration for which connections are to be aggressively opened in a blocking manner
// beyond this duration connections will be defensively opened in a non-blocking manner
Duration sampleAggressiveConnectionEstablishDuration = Duration.ofSeconds(1);

// building the client along with opening connections
CosmosAsyncClient clientWithOpenConnections = new CosmosClientBuilder()
          .endpoint("<account URL goes here")
          .key("<account key goes here>")
          .endpointDiscoveryEnabled(true)
          .preferredRegions(Arrays.asList("sample_region_1", "sample_region_2"))
          .openConnectionsAndInitCaches(new CosmosContainerProactiveInitConfigBuilder(Arrays.asList(containerIdentity1, containerIdentity2))
                .setProactiveConnectionRegionsCount(proactiveConnectionRegionsCount)
                .setAggressiveProactiveConnectionEstablishmentDuration(sampleAggressiveConnectionEstablishDuration)
                .build())
          .directMode()
          .buildAsyncClient();

Proactive connection management during connection close and reset
This PR helps proactively re-establish connections which are reset by the server or if they have been gracefully closed or in cases where the channel was closed due to it being unhealthy or idle.
Design considerations

Fairness: This done to ensure open connection tasks for a given endpoint are spaced out as much as possible.
Retries: A connection attempt could fail and such failures could be transient. Therefore, a backoff retry is used to reattempt unsuccessful connection attempts.
Concurrency control: Since the SDK could open connections in the background, it is important to set the concurrency here as low as possible (set as 1 by default). This could be too low for an application and hence a system config - COSMOS.OPEN_CONNECTIONS_DEFENSIVE_CONCURRENCY can be used to modify the defensive concurrency.

Benchmarking results
Insights

Cold start latency has reduced for workloads with proactive connection management.
Cold start latency differences is lesser between workloads with and without proactive connection management at lower concurrencies.
Cold start latencies are higher when the aggressive connection establishment duration is lower.

Reads (Provisioned throughput : 1 million RUs, Concurrency : 512, Documents to read: 10 thousand, Operations count: 1 million)

Reads (Provisioned throughput : 1 million RUs, Concurrency : 128, Documents to read: 10 thousand, Operations count: 1 million)

Reads (Provisioned throughput : 1 million RUs, Concurrency : 512, Documents to read: 10 thousand, Operations count: 1 million, proactive connection management with aggressive connection establishment duration)

Reads (Provisioned throughput : 1 million RUs, Concurrency : 128, Documents to read: 10 thousand, Operations count: 1 million, proactive connection management with aggressive connection establishment duration)

Reads (Provisioned throughput : 20 thousand RUs, Concurrency : 10, Documents to read: 1 thousand, Operations count: 100 thousand)

Writes (Provisioned throughput : 1 million RUs, Concurrency : 512, Operations count: 1 million)

Writes (Provisioned throughput : 1 million RUs, Concurrency : 128, Operations count: 1 million)

Writes (Provisioned throughput : 1 million RUs, Concurrency : 512, Operations count: 1 million, proactive connection management with aggressive connection establishment duration)

Writes (Provisioned throughput : 1 million RUs, Concurrency : 128, Operations count: 1 million, proactive connection management with aggressive connection establishment duration)

Writes (Provisioned throughput : 20 thousand RUs, Concurrency : 10, Operations count: 100 thousand)

/azp run java - cosmos - tests

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-cosmos

/azp run java - cosmos - tests

/azp run java - cosmos - tests

/azp run java - cosmos - tests

/azp run java - cosmos - tests

/azp run java - cosmos - tests

/azp run java - cosmos - tests

/azp run java - cosmos - tests

/azp run java - cosmos - tests

/azp run java - cosmos - tests

/azp run java - cosmos - tests

/azp run java - cosmos - tests

/azp run java - cosmos - tests

/azp run java - cosmos - tests

/azp run java - cosmos - tests

/azp run java - cosmos - tests

/azp run java - cosmos - tests

/azp run java - cosmos - tests

/azp run java - cosmos - tests

/check-enforcer override

fixes #33082
fixes #33079
fixes #33080
fixes #33087


[outreachy] Registry : Add to favorites #5447

Fixes

Fixes #5447 by @svrnm

Description

This PR introduces a "Add to Favorites" feature, allowing users to quickly mark registry entries as favorites. Favorites are stored locally in local storage, ensuring persistence across sessions.

Implemented a button for adding/removing items from favorites.
Favorites are stored in local storage for better user experience.
Integrated the functionality using a favorite-btn that updates dynamically.

Technical details

Partial for JavaScript: Moved JavaScript functionality to a separate js/addToFavorites.js file, loaded via a partial in the layout for better maintainability and performance.

Local Storage Logic: The local storage checks if an item is already marked as a favorite, and updates the button's appearance accordingly. This avoids the need for server-side calls, keeping it fast and client-side focused.

Screenshots

@mailsg: I see that I can add favorites, have you planned any way that I can use the favorites, e.g. only see the favorites or any other way?

@mailsg please make sure that you register with the outreachy platform after that I will provide a review

thanks for registering your contributions in the outreachy platform. Following up with my previous feedback:
I see that I can add favorites, have you planned any way that I can use the favorites, e.g. only see the favorites or any other way?

thanks for registering your contributions in the outreachy platform. Following up with my previous feedback:
I see that I can add favorites, have you planned any way that I can use the favorites, e.g. only see the favorites or any other way?

Hi @svrnm,
In addition to providing users with the ability to add and remove items from their favorites, there are several practical ways to make this feature more useful. These can be the planned usage of the Add to Favorites feature :

Dedicated "Favorites" Page

Users will be able to view all of their favorite marked items in one place by navigating to a dedicated "Favorites" page. This page will aggregate all marked items for easy access.
This allows users to quickly locate their preferred resources without having to search through the entire registry each time.

Filter by Favorites

An additional filter could be added to the registry that allows users to view only their favorite marked items while browsing the main catalog. This would integrate the favorites functionality directly into the registry interface, making it more user-friendly and efficient for users looking to engage with their curated resources.

Persistence Across Sessions

Since the favorites are stored in local storage or session cookies, users can access their list of favorites even after logging out or closing the browser. This ensures that the favorites list persists across sessions, improving the user experience by providing continuity.

Improved Personalization

With a user's favorites data available, the site could later introduce personalized content suggestions based on their favorites, such as related tools, newly added components, or relevant updates. This could lead to better user satisfaction and engagement without requiring significant changes to the initial feature implementation.

Future Expansions

In the future, favorites could serve as the foundation for more advanced features, such as exporting favorite lists, sharing with others, or receiving notifications when an item in their favorites is updated. These expansions would make the feature even more valuable to users while aligning with the overall goal of improving user experience.

To conclude, the Add to Favorites feature is designed to make the registry more user-friendly and personalized. By giving users a dedicated page for their favorites, a filter to view only their selected items, and ensuring everything is saved across sessions, we are making it easier for users to engage with the platform in a way that works for them. This will not only make the registry more useful on a day-to-day basis, but it also sets the stage for future improvements like personalized recommendations or sharing favorites with others. Overall, I think this can be a step towards making the site more interactive and valuable for the community.
Thanks.

Please implement "Filter by Favorites". Thanks.

Please implement "Filter by Favorites". Thanks.

Description

The JavaScript functionality enhances user interaction with the registry by allowing users to filter their view to display only their favorite entries. This feature simplifies navigation by letting users focus on items they are most interested in.

Technical details

The implementation consists of event listeners that manage user interactions with the Show Favorites button. When a user selects to view only their favorites, the system checks against local storage to display the appropriate items and toggling Show Favorites to Show All.
When the user selects Show All, they can view all the registry items.

Thanks.

overall lgtm, a few comments

Hi @svrnm,
I have made the suggested changes.

Thanks.

I consider this as done! After the outreachy application phase we will take another look and see if we can merge this PR. Thank you!

I will close this for now, since I raised a follow up discussion here: https://github.com/open-telemetry/opentelemetry.io/issues/5759


Dont call virtual method GetStatelessResetToken  in ctor QuicSession::QuicSession

QuicSession class calls the virtual method GetStatelessResetToken() token in constructor

Virtual methods should not be called in constructors, as the virtual call mechanism is disabled.
This might lead to a wrong tokens being returned.

Thank you.  This proposal was merged internally with flag protection at https://github.com/google/quiche/commit/db4dca88c57fa053f38ea4139c991e7a09094651.


[Error: extract docx unzip exec error: Error: stdout maxBuffer exceeded.]

Hi David,
I'm getting this error when trying to textract a big .docx file (1.8MB). I tried increasing the maxBuffer setting by doing $ textract big-file.docx --exec.maxBuffer 512000 but it's not working (tried many values, but none seem to work).
Do you know a possible fix?
Thanks!

1.8MB may just be too big for the process?  This'll end up being a node process problem more than anything I think.

Closing this out, not sure textract really has much it can do here.


Added Ukrainian language

Did you translate those yourself or with translation tools?

@mjauvin  Yes, I translated

Thanks!


Implement react hooks where appropriate

Will use react hooks where appropriate for future development, but won't redo anything that exists for no reason unless react-redux implements react hooks.


Fix build with documentation.nixos.includeAllModules = true;

This includes the options of all modules used in the evaluation, not just the ones from <nixpkgs/nixos> in the local manual.
Right now this breaks with
error: attribute '"‹name›"' missing

because the submodule environment.persistence doesn't have actual declarations when building the manual, but a dummy only to evaluate the sub-options (including their defaults which is the cause of the error) and generate documentation from that.
Using defaultText prevents that because there's no need anymore to evaluate the default values of the options.
cc @lovesegfault @talyz

Thanks!


NoMethodError in CamaleonCms::Admin#dashboard

I have integrated Camaleon cms with the existing app.But after I have created my admin panel, there is the following issue when try to log in to admin panel.

Did anyone get the same issue?
Please help me.

Please put the full error message

Hi, owen.
I have got the following trouble in terminal.
ActionView::Template::Error (undefined method name' for {:name=>"Dashboard", :url=>"/en/admin", :options=>{}}:Hash):     3:      4:      5:     <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>     6:     <title><%= raw cama_admin_title_draw %></title>     7:     <meta http-equiv="X-UA-Compatible" content="IE=edge"/>     8:     <meta name="viewport" content="width=device-width, initial-scale=1"/>     9:     <%= stylesheet_link_tag "camaleon_cms/admin/admin-manifest", media: "all" %>   lib/rack/seoredirect.rb:20:in call'
Thanks.

Fixed! pls test using github version

Owen, Thanks a lot,
After github version is installed, the bug is fixed.
But I have another problem now.
ActionView::Template::Error (undefined method html_safe' for {:tag=>:li, :separator=>""}:Hash):     1: <% if @breadcrumbs.present? %>     2:   <ul class="breadcrumb <%= options[:class] %>">     3:     <% separator = divider.html_safe %>     4:     <% @breadcrumbs[0..-2].each do |crumb| %>     5:       <li class="<%= options[:item_class] %>">     6:         <%= link_to crumb[:name], crumb[:url], crumb[:options] %>   lib/rack/seoredirect.rb:20:in call'
Greetings for your help.

I have fixed that.
Thanks for great CMS.

Hi, owen.
After I have fixed the problem, there is the following issue.

I have noticed that there is the trouble in breadcrumb_on_rails gem, and fixed the issue by revising
the <%= render_breadcrumbs :tag => :li, :separator => "" %> to <%= render_breadcrumbs %>.
Regards.

Can you review the logs to check request params and verify if there is slug value?
If it is not present, pls review if there is an javascript error by javascript console which is affecting the slug generator.

I have reviewed the javascript error in console.
That is :

Thanks a lot.

I am also experiencing a breadcrumb error since the last commit. Relevant stack trace:
FATAL -- : [e941c34e-f0cb-44fe-b1fb-bc91b3a2d0b3] ActionView::Template::Error (undefined method `[]' for #<BreadcrumbsOnRails::Breadcrumbs::Element:0x00007fe0aa1d6f90>):
FATAL -- : [e941c34e-f0cb-44fe-b1fb-bc91b3a2d0b3]     3:
     4:
     5:     <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
     6:     <title><%= raw cama_admin_title_draw %></title>
     7:     <meta http-equiv="X-UA-Compatible" content="IE=edge"/>
     8:     <meta name="viewport" content="width=device-width, initial-scale=1"/>
     9:     <%= stylesheet_link_tag "camaleon_cms/admin/admin-manifest", media: "all" %>
FATAL -- : [e941c34e-f0cb-44fe-b1fb-bc91b3a2d0b3]
FATAL -- : [e941c34e-f0cb-44fe-b1fb-bc91b3a2d0b3] vendor/bundle/ruby/2.4.0/bundler/gems/camaleon-cms-c98ed9f56b15/app/helpers/camaleon_cms/admin/breadcrumb_helper.rb:5:in `block in cama_admin_title_draw'
vendor/bundle/ruby/2.4.0/bundler/gems/camaleon-cms-c98ed9f56b15/app/helpers/camaleon_cms/admin/breadcrumb_helper.rb:5:in `each'
vendor/bundle/ruby/2.4.0/bundler/gems/camaleon-cms-c98ed9f56b15/app/helpers/camaleon_cms/admin/breadcrumb_helper.rb:5:in `cama_admin_title_draw'
vendor/bundle/ruby/2.4.0/bundler/gems/camaleon-cms-c98ed9f56b15/app/views/layouts/camaleon_cms/admin.html.erb:6:in `_vendor_bundle_ruby_______bundler_gems_camaleon_cms_c__ed_f__b___app_views_layouts_camaleon_cms_admin_html_erb___3685030159700069274_70301466949620'

Thanks again for all your work on this CMS @owen2345! It really is fantastic.

1: About js error: the problem is jquery version (3x deprecated .size() and other methods), pls use  jquery-migrate.xx
2: apparently there is a conflict with breadcrumb_on_rails gem, can you verify for a other breadcrumb gem?
Regards!!!

Perhaps I should have made a separate issue. I commented here because my problem is with the same commit, but it is different than the issue described above. I can't even access the admin portions of my site with the latest version. I see no javascript error because the page doesn't even load.
I'm not using any other breadcrumb gem, just breadcrumbs_on_rails as a dependency of Camaleon.

mmm ok! this weekend I will review with the last version of breadcrumb on rails. Please share me your version of breadcrumb_on_rails gem.

I'm using breadcrumbs_on_rails v3.0.1. Currently I've locked Camaleon to commit c5c9019 and it works fine, so it's only the most recent change that caused the issue.

I found the bug causing my issue and submitted a PR. I didn't reference this issue because I'm not sure whether it will affect @kriskoli's issue or not.
There was no issue with breadcrumbs_on_rails. It hasn't changed since last year. Ruby just wasn't parsing the new ternary statement as intended, easily fixed with parentheses.
@kriskoli sorry I hijacked your issue. I thought mine was related, but it appears not.

Hi, owen.
Thank for your help.
I 've changed the .size() method to length for jquery version 3.2.1 on my issue.
It works fine.

Great! pls be careful with other jquery deprecations on 3.x
Note: I solved the breadcrumb issue


Updating to newer ways to Concourse likes to be used

Broke config out to separate file
Updating to use PCF Dev
Updated tasks that were not working with newer version of Concourse
Updated README for ci

need to update docs


fix(subject): Faster subscriptions management in Subject

Description:
The current implementation of the Subject was backing itself on an array of observers. When one observer unsubscribed itself, it was performing a linear scan on the array of observers and dropping the item from the array of observers. The algorithm complexity of this operation was O(n) with n being the number of observers connected to the Subject.
It caused code like:
const allSubscriptions = [];
const source = new Subject();
for (let index = 0 ; index !== 1_000_000 ; ++index) {
  allSubscriptions.push(source.subscribe()); // rather quick
}
for (const subscription of allSubscriptions) {
  subscription.unsubscribe(); // taking very long
}

The proposed approach consists into changing our backing collection (an array) into a Map. But we lose somehow the set capability on subject.observers (might possibly be patched in some way).
The command cross-env TS_NODE_PROJECT=tsconfig.mocha.json mocha --config spec/support/.mocharc.js "spec/**/Subje*-spec.ts" with the new test added:

Now: 452ms
Before: 10s

BREAKING CHANGE:
Subject.observers has been impacted. If needed we can probably make it backward compatible in some ways. Here is what changed:

Now a getter property, was previously a basic property
The value change from one call to another -we recompute it anytime we access it)
There is no setter anymore
The value is never null

Related issue (if exists):

I believe we had similar sort of PR and did not made changes, but search fails me tonfind exact one.
Also in my opinion this seems for extreme cases: if you really have more than even 10,000 subscriptions I'd expect app's architecture need to be revisited. What are the real world problem we'd like to fix with breaking changes?

Also in my opinion this seems for extreme cases: if you really have more than even 10,000 subscriptions I'd expect app's architecture need to be revisited. What are the real world problem we'd like to fix with breaking changes?

Good point! On my side, I patched the issue locally by packing all my subscriptions behind a single one, itself pipping data back to my original subscribers. My case was a grid containing thousands of cells in which each cell subscribes to a shared store to see updates. My local fix has been to drop the subscribe done by the cell to an home-made version of a subscribe dealing with adding and removing subscribed callbacks into a Set that will be read by the single real subscription done on the source subject.
I though about submitting this potential fix as actually even if the problem is not that horrible in small cases, we still have extra time taken by a linear look-up while we could just have a immediate delete by switching to a Map. Idea was to improve a lot very huge case but also smaller ones.

I can probably start to work onto measuring the performance impact on various cases including: 1/2/10/100 subscribers and time to subscribe/emit/unsubscribe. So that we can check whether or not it could bring negative (clearly a no-go) or positive performance impact on small and nominal cases 🤔

Before doing any additional work may need some archaeology to confirm if there was a public / internal discussions for this and why we didn't. It would be bad if you spend amount of time to prepare PR then we find out some historical reason we can't land this.

I added a test that passes in master locally just to make sure this didn't break anything. However, the tests have been running for > 13 minutes now, so I suspect something is wrong. This is a decent idea, but it makes the changes in the wrong spots, and removes other optimizations.

lol...using the in-browser IDE is great, except I can't run the tests. I should really start using Stackblitz for this. 🤔

Actually... wait, I'm moving too fast because I was excited about this optimization. This particular code won't work. Using the Subscription as the key means that in some cases the same Subscription ends up trampling things. These cases are non-obvious because we're returning a Subscriber instance as the Subscription in a lot of cases (as an optimization). That Subscriber instance is often passed through the subscribe call as-is (another optimization), this means that if I rollback to @dubzzz's original commit, a large number of tests fail. The test I added also fails.
We can't accept this as-is... but I'll think about this a bit. This can probably be optimized.

Since I destroyed this one with the Web IDE, I'm going to close it and reopen it in it's original state at #7234 .


Allow renaming of custom event parameters (#48)

Description & motivation
If a custom parameter conflicts with one of the standard column names exported by the staging views, it's currently not possible to access it. unnest_key supports renaming so we should ideally support this in stage_custom_parameters too.
Checklist

[x] I have verified that these changes work locally
[x] I have updated the README.md (if applicable)
[ ] I have added tests & descriptions to my models (and macros if applicable)
[ ] I have run dbt test and python -m pytest . to validate exists tests

I can't even run the test suite on master:
python3 -m pytest .
============================================================================================================================ test session starts =============================================================================================================================
platform darwin -- Python 3.9.13, pytest-7.1.2, pluggy-1.0.0
rootdir: /Users/will/Sekuda/Client work/Delivereasy/dbt-ga4
collected 0 items / 1 error

=================================================================================================================================== ERRORS ===================================================================================================================================
_______________________________________________________________________________________________________________________ ERROR collecting test session ________________________________________________________________________________________________________________________
Defining 'pytest_plugins' in a non-top-level conftest is no longer supported:
It affects the entire test suite instead of just below the conftest as expected.
  /Users/will/Sekuda/Client work/Delivereasy/dbt-ga4/integration_tests/conftest.py
Please move it to a top level conftest file at the rootdir:
  None
For more information, visit:
  https://docs.pytest.org/en/stable/deprecations.html#pytest-plugins-in-non-top-level-conftest-files
========================================================================================================================== short test summary info ===========================================================================================================================
ERROR
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================================================================== 1 error in 0.11s ==============================================================================================================================

Just merged in a fix for the broken unit tests. Thanks for flagging!

Looks good. Very elegant solution.


2.2.6 and above, images take up all available space

Going from 2.2.5 to 2.2.6, CachedImage wants to take up all available space.  This is happening on Android and iOS.
<ffimage:CachedImage Source="ic_desktop_windows_white_24dp"
    HorizontalOptions="Center"
    VerticalOptions="Center"
    DownsampleToViewSize="true"
    TransparencyEnabled="false" >
</ffimage:CachedImage>

Using 2.2.5, Forms <IP_ADDRESS>

Using 2.2.6, Forms <IP_ADDRESS>

Images used:

White one below

Yes, there were some changes. It would be nice if someone could help me with that:

Xamarin.Forms code: https://github.com/xamarin/Xamarin.Forms/blob/master/Xamarin.Forms.Core/Image.cs#L62-L118
https://github.com/luberda-molinet/FFImageLoading/blob/master/source/FFImageLoading.Forms/CachedImage.cs#L490-L523

Basically if I put the same code as Xamarin.Forms Image CachedImage doesn't work as suppose to in expandable layouts. I don't know why. Do anyone knows how could we improve that?

Let's continue here: https://github.com/luberda-molinet/FFImageLoading/issues/545


Downloading .csv for an address downloads incomplete data if history is <30 transactions

Describe the bug
The export as .csv function will download the last 30 entries per default. It will only include more if you click "Load more transactions" to reveal additional transactions.
Any address will do to reproduce.

To Reproduce
Steps to reproduce the behavior:

Go to explorer.hiro.so
visit any address with more than 30 transactions for example: https://explorer.hiro.so/address/SP2YADQRAJ4468KEX4CYD4MQPF0S6QYFT5BRA22J0?chain=mainnet
click export as csv
view the csv and see only the last 30 transactions.

Expected behaviour
The expected behaviour is that users can specify the entries they want to include (for example by year for tax purposes) or to get the complete history of the address.

Perhaps a filter option can be added here to display "Last years" history, or 2022 history etc (from date X - to date Y). for tax purposes:

Users don't expect the csv export to only contain entries currently displayed on screen, rather that they get the complete history.

Adding filters is a good idea, To start with we should fix the issues with the download action: it has a bug where it can only download the tx that are shown on the page (30 tx) and it doesn't show FT tx.
Until we fix it the workaround is using https://fatstx.github.io/transactions


Fix #3015

After this fix, the logic for determining the outcome of docker-machine active will be:

If any active docker host or swarm is found, return it as active
If no active host is found, and any host is in state state.Timeout, then the error is errActiveTimeout: "Error getting active host: timeout"
If no active host is found, and no host is in state state.Timeout, then the error is errNoActiveHost: "No active host found"

I have implemented step 2, and I have added one unit test and modified old unit tests to check 1-3 above.
Signed-off-by: Patrik Erdes<EMAIL_ADDRESS>

LGTM

LGTM


Deselect items by setting selected to falsy or out-of-range value

Fixes #20

This change is

Review status: 0 of 2 files reviewed at latest revision, 3 unresolved discussions.

vaadin-list-mixin.html, line 85 at r1 (raw file):

        } else if (itemToSelect === undefined) {
          items.forEach(item => item.selected = false);
        }

This code might work in all cases
items.forEach(item => item.selected = item === itemToSelect);
if (itemToSelect && !itemToSelect.disabled) {
       this._scrollToItem(selected);
}

vaadin-list-mixin.html, line 151 at r1 (raw file):

      const totalItems = this._navItems.length;
      const isValidIndex = idx !== undefined && idx !== null && !isNaN(idx);
      for (let i = 0; isValidIndex && i < totalItems; i++, idx += (increment || 1)) {

for (let i = 0; typeof idx == 'number' && i < totalItems; i++, idx += (increment || 1)) {'
is simpler and covers more cases, like idx being a string, object, etc.

test/vaadin-list-mixin.html, line 182 at r1 (raw file):

        });

        it('should clear selection when `selected` property is set to negative value', () => {

There are so much tests here for a simple condition, I would just test by sending a not number, and use the typeof condition I mentioned

Comments from Reviewable

Review status: 0 of 2 files reviewed at latest revision, 3 unresolved discussions.

vaadin-list-mixin.html, line 85 at r1 (raw file):
Previously, manolo (Manuel Carrasco Moñino) wrote…
This code might work in all cases
items.forEach(item => item.selected = item === itemToSelect);
if (itemToSelect && !itemToSelect.disabled) {
       this._scrollToItem(selected);
}

Done.

vaadin-list-mixin.html, line 151 at r1 (raw file):
Previously, manolo (Manuel Carrasco Moñino) wrote…
for (let i = 0; typeof idx == 'number' && i < totalItems; i++, idx += (increment || 1)) {'
is simpler and covers more cases, like idx being a string, object, etc.

Done.

test/vaadin-list-mixin.html, line 182 at r1 (raw file):
Previously, manolo (Manuel Carrasco Moñino) wrote…
There are so much tests here for a simple condition, I would just test by sending a not number, and use the typeof condition I mentioned

Done.

Comments from Reviewable

Reviewed 2 of 2 files at r2.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable

Review status: :shipit: all files reviewed at latest revision, all discussions resolved, all commit checks successful.

Comments from Reviewable


Building an IoT weather station

Objective
I've wanted a weather station since I can remember and whilst we have been locked up socially distancing during the COVID-19 pandemic I decided to make this a reality. This was mainly driven by the amazing weather, blue skies, and clean air. The idea being I could use this unique moment to capture a dataset and compare it against whatever normal is when we return to living our lives after the pandemic. I've often looked at the different types of weather stations available on the market and always been drawn to the Bresser weather centre for the design and build quality. The Bresser looks to have a solid frame and isn't unpleasant to the eye. compared to some on the market.
Not stopping there I wanted to make the data digitally available for me to access as I liked. Through integrating the data with Home Assistant I can drive all kinds of automation and integration with other products. Over and above the typical data that weather stations capture, I want to include other weather-related sensors such as air quality (co2, no2, 2.5PM, and 10PM etc) and a lightning sensor and learn more about the weather and environment quality where I live.

[x] Sniff 838mHz data from the weather station using RTL_433
[x] Send the data over to Home Assistant via MQTT
[x] Include an air pressure sensor
[ ] Integrate air quality PM, VOC, NO₂ and CO₂ sensors (including 3D printed  Stevenson screen and fixings)
[ ] Research building RTL_433 on ESPHome on an ESP32 board
[ ] Add additional weather and environmental sensors
[x] Rain wet sensor #22
[ ] Indoor display
[ ] Stevenson screen

Ingredients

Bresser 5-in-1 weather station
868mHz software-defined radio receiver
Raspberry Pi or ESP32 for hosting the radio receiver
MQTT broker

Fortunately, there is a framework call RTL_433 available on GitHub which allows you to easily receive the data from some 150+ devices. Unlike the name, the framework works on 433, 868, 315, 345, and 915 MHz bands.
I used this framework with a Raspberry Pi and an 868 MHz SDR receiver to test if I was able to receive data from my weather station. Luckily I was.

Sharing the project on Twitter

Here is the Bresser weather station after it arrived and it mounted in the garden shortly afterwards.

[ ] Wind direction offset.
You are expected to point the weather station North. In my view, this looks odd as it looks better parrel to the roofline rather than skewed. To mitigate this I plan to include an option to offset the wind direction to allow for this.

As part of my research, I created a list of sensors I wanted to consider adding to the weather station to provide additional environmental data. Remember this was not just about having access to basic weather station data, this project was for me to learn about the weather and have an interesting dataset to analyse over time.

Temperature (surface/ground, air, soil)
Air quality (2.5PM, 10PM, VOC, NO₂, CO₂, Ozone)
Wind direction, speed and gusts (daily, weekly average etc)
Rainfall
Lightening detector
UV and LUX sensor
Solar radiation
Sky camera

I plan to come back and add my reasoning about why I wanted to capture this data.

Yay. 🥳
RTL_433 is now supplying Home Assistant with data over MQTT. I still plan to look at running RTL_433 on an ESP board if possible but if this is not possible I might look at some alternatives to get the data over to Home Assistant from the Raspberry Pi it is currently running on. I just want to avoid my dependency on MQTT when other APIs exist.
Due to how I mounted my weather station it doesn't point North. Therefore my wind direction was skewed and doesn't report correctly. As highlighted above my North was 296° so I wanted to build something using Home Assistant to correct this but at the same time also create the wind direction as a heading rather than a value. For this, I chose to use the names from a 16-wind compass rose. What was nice was reaching the same solution others had created prior to finding such articles. 🤓
I took off to the trusted template editor available in Dev Tools to test some YAML for me to use in a number of new sensors.
Wind RAW from Weather Station: {{ states('sensor.wind_direction') }}°
{% set calibrate = 296 | int %}
Weather station mounted North at: {{ calibrate | int }}°
{% set adjusted = (states('sensor.wind_direction') | float(2) + calibrate) %}
Adjusted Wind (pre-calculation): {{ adjusted }}°
{% if adjusted > 360 %}{% set wind = adjusted - 360 %}{% elif adjusted < 0 %}{% set wind = 0 + adjusted %}{% else %}{% set wind = adjusted %}{% endif %}
Wind offset: {{ wind }}
{% if wind >= 348.75 or wind <= 11.25 %}
Wind heading: North (N)
{% elif wind >= 11.25 and wind <= 33.75 %}
Wind heading: North Northeast (NNE)
{% elif wind >= 33.75 and wind <= 56.25 %}
Wind heading: North East (NE)
{% elif wind >= 56.25 and wind <= 78.75 %}
Wind heading: East Northeast (ENE)
{% elif wind >= 78.75 and wind <= 101.25 %}
Wind heading: East (E)
{% elif wind >= 101.25 and wind <= 123.75 %}
Wind heading: East Southeast (ESE)
{% elif wind >= 123.75 and wind <= 146.25 %}
Wind heading: South East (SE)
{% elif wind >= 146.25 and wind <= 168.75 %}
Wind heading: South Southeast (SSE)
{% elif wind >= 168.75 and wind <= 191.25 %}
Wind heading: South (S)
{% elif wind >= 191.25 and wind <= 213.75 %}
Wind heading: South Southwest (SSW)
{% elif wind >= 213.75 and wind <= 236.25 %}
Wind heading: South West (SW)
{% elif wind >= 236.25 and wind <= 258.75 %}
Wind heading: West Southwest (WSW)
{% elif wind >= 258.75 and wind <= 281.25 %}
Wind heading: West (W)
{% elif wind >= 281.25 and wind <= 303.75 %}
Wind heading: West Northwest (WNW)
{% elif wind >= 303.75 and wind <= 326.25 %}
Wind heading: North West (NW)
{% elif wind >= 326.25 and wind <= 348.75 %}
Wind heading: North Northwest (NNW)
{% else %}
Wind heading: ?
{% endif %}

This provided me with the following data.

I created an input_number entity to store my offset value. This is the new North bearing of weather station.
input_number:
  weather_station_wind_north_offset:
    name: North Offset
    min: 0
    max: 360
    step: 0.1
    mode: box
    unit_of_measurement: '°'
    icon: mdi:compass

I then created a template sensor to convert the compass value to the 16-point wind direction.
sensor:
  platform: template
  sensors:
    wind_bearing:
      friendly_name_template: "Wind Direction"
      value_template: >-
        {% set wind = states('sensor.wind_direction_calibrated') | int %}
        {% if wind >= 348.75 or wind <= 11.25 %}
        North (N)
        {% elif wind >= 11.25 and wind <= 33.75 %}
        North Northeast (NNE)
        {% elif wind >= 33.75 and wind <= 56.25 %}
        North East (NE)
        {% elif wind >= 56.25 and wind <= 78.75 %}
        East Northeast (ENE)
        {% elif wind >= 78.75 and wind <= 101.25 %}
        East (E)
        {% elif wind >= 101.25 and wind <= 123.75 %}
        East Southeast (ESE)
        {% elif wind >= 123.75 and wind <= 146.25 %}
        South East (SE)
        {% elif wind >= 146.25 and wind <= 168.75 %}
        South Southeast (SSE)
        {% elif wind >= 168.75 and wind <= 191.25 %}
        South (S)
        {% elif wind >= 191.25 and wind <= 213.75 %}
        South Southwest (SSW)
        {% elif wind >= 213.75 and wind <= 236.25 %}
        South West (SW)
        {% elif wind >= 236.25 and wind <= 258.75 %}
        West Southwest (WSW)
        {% elif wind >= 258.75 and wind <= 281.25 %}
        West (W)
        {% elif wind >= 281.25 and wind <= 303.75 %}
        West Northwest (WNW)
        {% elif wind >= 303.75 and wind <= 326.25 %}
        North West (NW)
        {% elif wind >= 326.25 and wind <= 348.75 %}
        North Northwest (NNW)
        {% else %}
        ?
        {% endif %}
      unit_of_measurement: '°'

Next up is creating this as a real-time graphic in Home Assistant.
Useful guidance I found during setting up these templates sensors.

Points on the compass
How to Convert Magnetometer Data into Compass Heading
How to Hookup Davis Anemometer to Arduino
Java: method to convert degrees into directions (16-wind compass rose)
Compass graphic

Last night I tried to find reasonable quality images for my compass and needle. Whilst many exist few split the gauge and needle as layers so I use one that was available for free that I liked made by ShaneTurcotte.Murazik and took off in Photoshop to recreate it.
Using the CSS rotate feature I am easily able to rotate the need to represent the wind direction. To compensate for the change in X-axis I used an additional attribute with the rotate transform styling.
.example{
  transform: translateX(180px) rotate(45deg);
}

Well this evening, after recreating the compass and needle as two layers I uploaded them to my Home Assistant images directory in my www folder. I then used the picture elements card to display the compass gauge as the background image and added an image element to control the needle.
elements:
  - type: image
    entity: sensor.wind_bearing
    tap_action:
      action: toggle
    image: /local/images/compass_needle.png
    style:
      top: 1%
      left: 25%
      width: 15%
      transform: translateX(85px) rotate(318deg)
image: /local/images/compass_bg.png
type: picture-elements

My next challenge is updating the rotate transform attribute with the sensor value or alternatively using the state_filter with the image element.

Really happy with how accurate the weather station temperature and humidity are. I stuck a Zigbee Aqara temperature sensor directly underneath the weather station and they both have been reporting almost identical values.
Weather station: humidity blue; temperature red;

Time to get the Particulate Matter sensor working. I have a working Arduino sketch so just need to get this converted and working in ESPHome.

Laser PM2.5 Air Quality Sensor
Arduino sketch

@DinoTech suggested on Discording using a state-switch and matching 16 or 64 of the 360 degrees by overlaying the needle image of the top of a picture-elements cards and having the state match and image 'swapped' for that templated match. (e.g. 25°, NNE, 225° SW)

My SGP30 TVOC and eCO2 will be on the same board as my PM sensor so time to get this working too.

SGP30 Air Quality Sensor Breakout
ESPHome

The VOC sensor allows temperature and humidity compensation. To achieve this I need to incorporate my BME680.

BME680 Digital Temperature Humidity Pressure Sensor
ESPHome
Adafruit
Datasheet

BH1750 LUX sensor

Amazon
ESPHome
[DF Robot] (https://wiki.dfrobot.com/Bh1750)
Datasheet

Soil temperature and moisture

Dallas temperature probe
Science Direct
ESPHome
Instructables

Notes:

Depth 30cm
Soil temperature definition
Photosynthetic activity was more highly correlated with soil temperature at 30 cm than with air temperature or soil temperature at shallower depths.

Environmental noise (dB)

[WS1361C Decibel Meter](https://www.amazon.co.uk/gp/product/B01M32WX3H/
Wensn WS1361 Sound Pressure Level (SPL) Meter over USB

Notes
I had planned to use an Omnidirectional Microphone Module I2S Interface INMP441 but the I2S interface seems to have poor documentation and community traction. Instead, I opted for a more sensitive decibel meter which I think will perform better for this project.

Weather/sky camera

ESP32-CAM Camera
ESPHome|

Notes

This could provide a live on-demand RTSP stream
Daily timelapse (gif?)
Could be triggered to record if the weather data suggests a weather event is happening or object detection in the video (Blue Iris)

Lightning detector

AS3935 I2C SPI Lightning Sensor
ESPHome

Ultraviolet light sensor

MP8511 UV Sensor
ESPHome

Dew point
Given I have data for air temperature and humidity I can calculate the dew point.

Draft:
I started to solder the sensors to a PCB board this evening.
Two hypothesis.

The glue gun glue sealing the UV sensors to the top of the Stevenson screen will not endure high temperature and melt away leaving the sensor exposed.

The sensors won’t work once connected because I have difficulty removing the breakout board pin connectors.

Air gases

MICS-4514 MICS-6814 Carbon Monoxide Nitrogen Oxygen Gas Sensor Module CO/NO2/NH3
MICS-4514 Sensor Module
Brand Name: CJMCU
Carbon monoxide CO 1-1000ppm
Nitrogen dioxide NO2 0.05-10ppm
Ethanol C2H5OH 10-500ppm
Hydrogen H2 1-1000ppm
Ammonia NH3 1-500ppm
Methane CH4 >1000ppm
Propane C3H8 >1000ppm
Iso-butane C4H10 >1000ppm

Air quality - Air gases LPG, CO, Smoke

MQ-2 sensor

Air quality - OZone

MQ-131 sensor

Air quality - OZone

MQ-131 sensor

Display
Cool idea to use ESPHome and a display to show weather station data.
https://home-assistant-guide.com/2020/10/08/the-esphome-weather-station-with-a-nextion-display/

KKmoon 100KHz-1.7GHz Full Band UV HF RTL-SDR USB Tuner Receiver/ R820T+RTL2832U AM CW FM DSB LSB
https://www.amazon.co.uk/gp/product/B0716CB1PM

https://esphome.io/components/tca9548a.html

Hi, Really interesting project!
Have you made any headway with using the ESP32 in place of the Pi? I'm hoping to use a Bresser in a remote location where i already have a suite of rain gauges connected rto IoT vi two soplar powered gateways. I want to power the weather station reporting device from the gateway battery, so minimal power usage is essential, and as i dont need any of the Pi's extra capabilities i think the ESP32 is ideal

Hello,
I am interested in the purchase of a Bresser weather station (a 5 in 1 or a 7 in 1, but does not matter right now) and I ended up here because of the following reason.
RTL_443 is good to gather the values of the outdoor sensors because of RF, but I would like to gather, store and monitor the indoor temperatures and humidity % from the central base station inside home. How could I accomplish this task? I had searched for many days but I got nothing useful.
Another question I have I related to the Wind direction offset.
I would setup the external unit approx to 022° North. Reading your post here, I did not understand how and where I should offset the North.
Many thanks.

Hello,
First of all really interesting project, thank you for documenting it this much.
I have a Bresser weather station and a Raspberry Pi, same use case as you. I would like to know which 868 MHz SDR receiver did you used?
Thank you,


host cloud anchor only after anchor is placed.

Hello, very new to android development and development in general. I'm following a tutorial on placing objects and hosting cloud anchors; I've got everything working but I'd like to modify things a bit. How can I place a single model in a scene (currently just using shapes from the ModelFactory) and then only host the anchor once the user has placed the object where and how they'd like it?
The onTapListener code I'm currently running:
        MaterialFactory.makeOpaqueWithColor(this, new Color(android.graphics.Color.BLUE))
                .thenAccept(material -> {
                    sphereRenderable = ShapeFactory.makeSphere(0.1f, new Vector3(0.0f,0.15f, 0.0f ),material);
                });

        arFragment = (CustomArFragment) getSupportFragmentManager().findFragmentById(R.id.ux_fragment);
        assert arFragment != null;
        arFragment.getArSceneView().getScene().addOnUpdateListener(this::onUpdateFrame);

        assert arFragment != null;
        arFragment.setOnTapArPlaneListener(
                (HitResult hitResult, Plane plane, MotionEvent motionEvent) -> {
                    if (plane.getType() != Plane.Type.HORIZONTAL_UPWARD_FACING ||
                            appAnchorState != AppAnchorState.NONE) {
                        return;
                    }
                    Anchor anchor = arFragment.getArSceneView().getSession().hostCloudAnchor(hitResult.createAnchor());

                    setCloudAnchor(anchor);

                    appAnchorState = AppAnchorState.HOSTING;
                    Toast.makeText(this,"Now Hosting...", Toast.LENGTH_LONG).show();
                    snackbarHelper.showMessage(this, "Now hosting...");

                    AnchorNode anchorNode = new AnchorNode(anchor);
                    anchorNode.setParent(arFragment.getArSceneView().getScene());

                    TransformableNode sphere = new TransformableNode(arFragment.getTransformationSystem());
                    sphere.setParent(anchorNode);
                    sphere.setRenderable(sphereRenderable);
                    isPlaced = true;
                    sphere.select();

                }
        );

I believe I would first have to set the initial anchor to hitResult.createAnchor(); but how would I allow the anchor to be hosted only after it is placed in a scene? This code begins hosting the anchor immediately. I planned to have a button revealed once the object is placed called "Finish" which would host the anchor. Thank you for any help.

Hi!
It is recommended to first make sure that the feature map quality is sufficient before hosting the Cloud Anchor: https://developers.google.com/ar/develop/java/cloud-anchors/developer-guide#check_the_mapping_quality_of_feature_points
For example, this ARCore sample application has a UI that helps the user to scan the anchor from different points of view (you don't need to set up Firebase to see the UI): https://github.com/google-ar/arcore-android-sdk/tree/master/samples/persistent_cloud_anchor_java
To allow the user to host the Cloud Anchor after he or she confirms the object position you can simply use a button that becomes visible or enabled after the object is placed.


Cleaning up Settings, Profile, etc.

make the options now available on the profile screen be on the settings screen

Solved in #602


[TAJO-2085] Add parseDateTime method with TimeZone in DateTimeFormat class

I added the UDF for my application like below.
TimeMeta tm = DateTimeFormat.parseDateTime(pastDateText, "YYYYMMDD");
DateTimeUtil.toUserTimezone(tm, timezone);
I expected to return the date to apply a time zone but It wasn't working.
So I hope to add parseDateTime method with TimeZone.
Thanks.

It was to resolve useing DateTimeUtil.toUTCTimezone(tm, tz);
Thanks a lot.


Missing desktop file after upgrade

Operating system: Linux Ubuntu 14.04 x86_64
Messenger for Desktop version: 2.0.6

I used to be able to launch messenger for desktop with my launcher simply by typing 'messenger' but after upgrade it doesn't work anymore and the only way to launch is from terminal.

Have you identified which .desktop file is missing?

@aluxian No way of knowing for me since it doesn't exist anymore. If you can tell me how I can install an older version of messenger I could see what file was being used in the earlier version

you can get older versions here: https://github.com/aluxian/Messenger-for-Desktop/releases
Or if you use apt, you can search for a package's versions (can't remember the command though)

in the 2.0.4 version it's there.
cat /usr/share/applications/messengerfordesktop.desktop
Name=Messenger for Desktop
Comment=A simple & beautiful desktop client for Facebook Messenger.
GenericName=Internet Messenger
Exec=/opt/messengerfordesktop/messengerfordesktop %U
Icon=messengerfordesktop
Terminal=false
Type=Application
StartupNotify=true
StartupWMClass=messengerfordesktop
Keywords=Facebook;Messenger;chat;desktop;
Categories=Network;InstantMessaging;
X-GNOME-UsesNotifications=true```

That's weird, I can't reproduce it. Does apt-get install (or dpkg) show any errors? Maybe some permissions are wrong, somewhere?
Does it work if you install the latest version and then manually replace the .desktop file? This is the most up-to-date version:
Name=Messenger for Desktop
Comment=A simple & beautiful desktop client for Facebook Messenger.
GenericName=Internet Messenger
Exec=/opt/messengerfordesktop/messengerfordesktop %U
Icon=messengerfordesktop
Terminal=false
Type=Application
StartupNotify=true
StartupWMClass=Messenger for Desktop
Keywords=Facebook;Messenger;chat;desktop;
Categories=Network;InstantMessaging;
X-GNOME-UsesNotifications=true

okay after doing a clean installation of the 2.0.6 version it's working now, I'm not sure what was wrong, maybe something went wrong with the update but it's working now so I will close this issue :)