Notes while reading

Not sure this is the most useful way to record these, but I figure it's better than my just thinking them to myself and then forgetting about them :-). Let me know if they should be moved to some other repo or mailing list or something.

IMO the SNI callback should receive the hostname as a bytes object containing the A-label version of the hostname, which for the stdlib wrapper means there should be a call to .encode("idna") in pep543_callback.

OpenSSL has an annoying thing where when you set a new context from the SNI callback, it only updates some parts of its internal configuration from that context, while ignoring others. Possibly the PEP 543 wrapper callback should work around this by noticing when these other things have changed and updating them manually, if it can, and otherwise error out. (ref)

We need at least two kinds of TLSError: one that means "definitely an error", and one that means "definitely an error, unless you're running in HTTP-compatibility mode, in which case pretend it's a clean EOF". (For the stdlib wrapper, SSLEOFError should give you the first one, and everything else the second one.) [Edit: I see that PEP 543 says that's RaggedEOF; so then the note here is that _error_converter needs to do SSLEOFError → RaggedEOF.]

I'm pretty sure the stdlib never raises SSLZeroReturnError. Though I can't prove it. But it certainly tries to convert it into a return b"" internally.

Writing to a ssl.MemoryBIO definitely never returns short. The API allows for the possibility because it might happen for other BIOs, but MemoryBIOs in particular will grow to whatever size they need to, so the nervous comment in receive_from_network is unnecessary.

You do need to watch out though that MemoryBIO.write(b"") "helpfully" gets interpreted as indicating an EOF, which may or may not match your API convention for receive_from_network.

Speaking of unbounded memory buffers, it's also impossible to get an SSLWantWriteError. I think we should consider removing it from the PEP 543 abstract API.

You might want to enforce that do_handshake is called given https://bugs.python.org/issue30141

A few more thoughts:

PEP 543 should probably be very explicit about what happens on the frontend side if you try to call read/write before calling do_handshake. And I guess the answer should probably be "you get an error". (The other option is "the backend magically does the handshake for you". Either can work, the stdlib does a third thing that's broken, and probably we want this in the spec so we don't have some backends picking one option and other backends picking the other.)

PEP 543's shutdown semantics are currently way underspecified.  OpenSSL provides SSL_shutdown, which sort of munges them together into one function call, which does something like (assuming a non-blocking BIO here):
if close_notify_not_sent:
    send_close_notify()
    return bool(close_notify_received)
else:
    process_incoming_bio()
    if close_notify_received:
        return True
    else:
        signal SSL_WANT_READ

Then there's the stdlib unwrap method, which does something like:
if close_notify_not_sent:
    send_close_notify()
process_incoming_bio()
if not close_notify_received:
    raise SSLWantReadError

The stdlib semantics are actually pretty reasonable, so long as you know what they are – if you just want to send close_notify and move on then you call it once and ignore the WantReadError, or otherwise you keep calling it like any other non-blocking SSL function. But I'm guessing that the other backends will have native semantics that are different again, so, definitely needs to be specified :-).
(If this becomes complicated then I guess it'd also be reasonable for the first version to only support "send close_notify and move on", since there are very few cases where you need to wait for a close_notify after sending one.)

I wonder if the bytes_buffered/peek/consume-style API is overkill? Trio, Twisted, asyncio all want to pull out all the bytes every time, I believe. So maybe this buffering code should move into the shared TLSWrappedSocket implementation, instead of being reimplemented inside each backend? (And I'm not sure, but I think for the case of TLSWrappedSocket + the stdlib ssl backend, there are actually more copies this way than if TLSWrappedSocket did its own buffering?) I don't have a strong opinion either way here, just noticed this.

Speaking of unbounded memory buffers, it's also impossible to get an SSLWantWriteError. I think we should consider removing it from the PEP 543 abstract API.

No, because wrapped sockets.

I wonder if the bytes_buffered/peek/consume-style API is overkill?

Yeah, the answer is "maybe". But the reasoning for it is simple: to reduce the number of places that have to buffer. That may be unnecessary, in the end, of course.

No, because wrapped sockets.

Oh, sure. But maybe we should drop it from the set of allowable exceptions raised by the abstract sans-IO interface.

@njsmith BTW, where there are things that are underspecified it'd be super-useful to see a PR that would better specify them. I'm acutely aware that I've basically written all of this document by myself, which means it's seen a substantial lack of coverage in areas that aren't popping up in my mind.
(Also, if we rely on me doing all of this it'll take ages to get done. ;) )

Where would such a PR go? directly against the peps repository?

Hrm, yeah, good point. I have an in-progress copy of the PEP somewhere: I'll add it to this repo.

Ok I've brought in an updated copy of PEP 543. Feel free to make PRs against it.


Convert to TypeScript and Add Tests

Using TypeScript allows some better completion for OpenRCT2 functionality, and helps this plugin maintain its stability over time, especially as we're about to make some big changes in the road to v1.0.

:tada: This PR is included in version 0.5.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Fix warning with torch.meshgrid

Motivation
torch.meshgrid() has started raising a warning when not called with an explicit indexing parameter:
https://pytorch.org/docs/stable/generated/torch.meshgrid.html
python demo/image_demo.py demo/demo.jpg configs/yolox/yolox_s_8x8_300e_coco.py https://download.openmmlab.com/mmdetection/v2.0/yolox/yolox_s_8x8_300e_coco/yolox_s_8x8_300e_coco_20211121_095711-4592a793.pth

gives
/home/allgeuer/anaconda3/envs/mmdev/lib/python3.9/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ../aten/src/ATen/native/TensorShape.cpp:2157.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]

Modification
Provide indexing='ij' to all calls to torch.meshgrid that don't already specify indexing.

Hi @pallgeuer ,
Thanks for your kind contribution. Since PyTorch only support indexing argument of meshgrid 1.10. Simply update the interface will cause mmdet cannot run on PyTorch<1.10. However, we need to support PyTorch >=1.5. Could you implement a meshgrid function to wrap torch.meshgrid? In this function, you can handle both cases so that  mmdet can still run on PyTorch < 1.10.
For example, implement and use the function below:
def meshgrid(x, y, indexing=None):
    # if pytorch version < 1.10:
    # use torch.meshgrid
    # else: use torch.meshgrid(x, y, indexing=xx) to avoid warning.

Good point. This affects many packages (mmdetection, mmpose, mmclassification, etc). How to best deal with that?
Create a wrapper for torch.meshgrid in mmcv?

Good point. This affects many packages (mmdetection, mmpose, mmclassification, etc). How to best deal with that?
Create a wrapper for torch.meshgrid in mmcv?

Since we do not want to increase the dependency on mmcv so fast, we can add a wrapper in mmcv first, then we can also copy the wrapper in mmdet and use that in mmcv if the mmcv version meets our demand, otherwise we can use that in mmdet.

I have updated the PR with the same solution as was merged for mmpose:
https://github.com/open-mmlab/mmpose/pull/1402

Hi. @pallgeuer ,
Thanks for your kind contribution.
We have discussed the naming issue and wish the code to be changed as below:

_torch_version_meshgrid_indexing = (
    'parrots' not in TORCH_VERSION
    and digit_version(TORCH_VERSION) >= digit_version('1.10.0a0'))

def torch_meshgrid(*tensors):
    """A wrapper of torch.meshgrid to compat different PyTorch versions.

    Since PyTorch 1.10.0a0, torch.meshgrid supports the arguments ``indexing``.
    So we implement a wrapper here to avoid warning when using high-version PyTorch and avoid compatibility issues when using previous versions of PyTorch.

    Args:
        xxx: (add description here for each argument)

    Return:

    """
    if _torch_version_meshgrid_indexing:
        return torch.meshgrid(*tensors, indexing='ij')
    else:
        return torch.meshgrid(*tensors)  # Uses indexing='ij' by default

To reduce the repeat work on your side, we suggest you directly update the PR in MMCV and we will review and merge it quickly.
Then we can help to update those interfaces in the remaining repositories like mmdet/mmpose/mmocr.
Thank you!

I have updated the mmcv PR.
I guess this means that the mmpose/etc packages will all have to depend on the latest version of mmcv soon then.

Hi @pallgeuer ,
Thanks for your kind and patient contribution. I can imagine that it is not easy to process multiple similar PR across repos.
We will take over the following progress in MMCV and downstream repos. Furthermore, we will update our contribution documentation to make it more friendly to the community.

MMCV already supports torch_meshgrid wrapper since 1.6.0 https://github.com/open-mmlab/mmcv/releases/tag/v1.6.0. We can upgrade the minimum requirement of MMCV since this PR.


DB Migrations fail from 2.9 -> 2.10

What I'm trying to achieve
I want to migrate to 2.10 from 2.9. However db migrations fail:
Running migrations:
  Applying plugins.0001_initial... OK
  Applying plugins.0002_auto_20200417_0335... OK
  Applying product.0110_auto_20191108_0340... OK
  Applying account.0035_staffnotificationrecipient... OK
  Applying account.0036_auto_20191209_0407... OK
  Applying account.0037_auto_20191219_0944... OK
  Applying warehouse.0001_initial... OK
  Applying product.0111_auto_20191209_0437... OK
  Applying product.0112_auto_20200129_0050... OK
  Applying product.0113_auto_20200129_0717... OK
  Applying product.0114_auto_20200129_0815... OK
  Applying product.0115_auto_20200221_0257... OK
  Applying order.0077_auto_20191118_0606... OK
  Applying order.0078_auto_20200221_0257... OK
  Applying checkout.0022_auto_20191219_1137... OK
  Applying checkout.0023_checkout_country... OK
  Applying checkout.0024_auto_20200120_0154... OK
  Applying checkout.0025_auto_20200221_0257... OK
  Applying account.0038_auto_20200123_0034... OK
  Applying account.0039_auto_20200221_0257... OK
  Applying core.0001_migrate_metadata... OK
  Applying account.0040_auto_20200415_0443... OK
  Applying account.0041_permissions_to_groups... OK
  Applying account.0040_auto_20200225_0237... OK
  Applying account.0041_merge_20200421_0529... OK
  Applying account.0042_merge_20200422_0555... OK
  Applying account.0043_rename_service_account_to_app... OK
  Applying webhook.0003_unmount_service_account... OK
  Applying account.0044_unmount_app_and_app_token... OK
  Applying account.0045_auto_20200427_0425... OK
  Applying app.0001_initial... OK
  Applying discount.0019_auto_20200217_0350... OK
  Applying django_prices_openexchangerates.0005_auto_20190124_1008... OK
  Applying giftcard.0003_auto_20200217_0350... OK
  Applying menu.0016_auto_20200217_0350... OK
  Applying warehouse.0002_auto_20200123_0036... OK
  Applying warehouse.0003_warehouse_slug...Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 328, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 369, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 231, in handle
    post_migrate_state = executor.migrate(
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/app/saleor/warehouse/migrations/0003_warehouse_slug.py", line 17, in create_unique_slug_for_warehouses
    first_char = warehouse.name[0].lower()
IndexError: string index out of range

Steps to reproduce the problem
Bring up empty installation of Saleor 2.9 (commit 877e5c285bac9a4b3a35a45063e3451f4bfbad52) and populatedb

docker-compose build api
docker-compose run --rm api python3 manage.py migrate
docker-compose run --rm api python3 manage.py populatedb --createsuperuser
Update to 2.10 (commit a1ec77677e1a769ad7995e3163c7c35295c13ed6) and run migrations

What I expected to happen
Migrations complete successfully.
Screenshots
System information
Operating system:
docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:21:11 2020
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:29:16 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Should be fixed by https://github.com/mirumee/saleor/pull/5611 not yet released though

Was facing the same issue.
Created a new DB and did migrations and it was successful.
Though this might be a bug.


remove rdf files, prefer jpm to build

review? @vvuk
This doesn't address #2 yet, simply makes a build that is submittable to AMO.  The old RDF was being rejected.  It's simpler to npm i -g jpm && jpm xpi.
Submitted here: https://addons.mozilla.org/en-US/firefox/addon/mozilla-webvr-enabler/

lgtm


[versioning] Consider using semantic versionin

https://github.com/AharonSambol/PrettyPrintTree/blob/master/setup.py#L9
Would be able to provide better-implied meaning if it was instead suffixed with .0
Just in case you haven't run into this before, here's the semantic versioning documentation.
With a quick summary in the first paragraph:

Given a version number MAJOR.MINOR.PATCH, increment the:
MAJOR version when you make incompatible API changes,
MINOR version when you add functionality in a backwards compatible manner, and
PATCH version when you make backwards compatible bug fixes.

I just didn't want to change anything about the version until I actually changed it. But no worries I'm about to do just that.

Just realized I probably just versioned this wrong 😅


Missing insert(at) and remove(at) methods

Hey,
I thought I could use this to do day 20 of Advent of Code but the absence of these methods makes it a bit annoying.

Be able to remove and retrieve any item just like remove(), but probably on Cursor
Be able to insert an item at any point in the list, which I can now do probably by splicing lists in weird ways

Isn't the idea of a linked list that these kind of mutations should be relatively cheap?
Looks great otherwise!

Hey,
Thanks for the issue ! It gave me motivation to implement these missing features. I used the day 20 problem of AoC as an inspiration to determine what I would need to solve it.


Use tcnative's new setVerify modes

Modifications:
tcnative made some fixes and API changes related to setVerify. We should absorb these changes in Netty.
Modifications:

Use tcnatives updated APIs
Add unit tests to demonstrate correct behavior

Result:
Updated to latest tcnative code and more unit tests to verify expected behavior.

@jchambers - FYI. Also let me know if you are aware of a way to create an intermediate CA with out a configuration file (e.g. openssl.cnf).

This is necessary for https://github.com/netty/netty-tcnative/pull/232

Thanks for the heads-up!

Also let me know if you are aware of a way to create an intermediate CA with out a configuration file.

I strongly suspect I may be missing your point, but if you omit the -extfile argument, openssl should just use its default configuration, and I think pretty much every default configuration within reason knows about the v3_ca extension.

I think pretty much every default configuration within reason knows about the v3_ca extension.

I tried this however at least on my machine basicConstraints = CA:true was not set, and so the Intermediate CA was generated in a way that doesn't permit it to sign other certificates. I added a unit test to check the two cases (invalid intermediate CA [1], and valid intermediate CA [2]).
[1] https://github.com/netty/netty/pull/6369/files#diff-56d12ae4c3938365ef518ae82a30591cR10
[2] https://github.com/netty/netty/pull/6369/files#diff-56d12ae4c3938365ef518ae82a30591cR18

@normanmaurer - You put LGTM on https://github.com/netty/netty-tcnative/pull/232 (which is the counterpart to this) ... is this good too?

@Scottmitch Yep LGTM ..

4.1 (fdcad31) 4.0 (a5386c3)


aem-upload from behind proxy

Expected Behaviour
aem-upload supports specification and/or autodetection of proxy configuration.
Actual Behaviour
aem-upload times out when attempting to upload from an environment with proxy requirement.
Reproduce Scenario (including but not limited to)
Windows host with Proxy requirement
Windows host with Fiddler or equivalent, monitoring for traffic via proxy to AEM Cloud.
Steps to Reproduce
Configure host machine to block default traffic and require http proxy or set up dev monitorable proxy (Fiddler, etc)
Attempt aem-upload to cloud.
Upload will time out or bypass proxy.
Platform and Version
Windows hosts, any AEMaaCS
Sample Code that illustrates the problem
Any upload script that works in a non-proxy environment
Logs taken while reproducing problem
2021-02-02T16:18:48.319Z [] error: Error while Uploading Error: connect ETIMEDOUT [[redacted]]:443

Thank you for the issue report, I can confirm that this is an issue. A potential resolution would be to add a new withProxy() method to DirectBinaryUploadOptions, and ensure this value is passed through to the proxy option when submitting requests with axios.
No ETA for a fix at the moment, but we do accept PRs if a fix is required sooner.

Axios does respect environment variables for proxy, so that is potentially
an option worth documenting. For authenticated proxies, from what I can
tell, this requires a plaintext password in the environment variable, which
is a potential security risk.
On Mon, Feb 22, 2021, 1:31 PM Mark Frisbey<EMAIL_ADDRESS>wrote:

Thank you for the issue report, I can confirm that this is an issue. A
potential resolution would be to add a new withProxy() method to
DirectBinaryUploadOptions, and ensure this value is passed through to the
proxy option when submitting requests with axios
https://www.npmjs.com/package/axios.
No ETA for a fix at the moment, but we do accept PRs if a fix is required
sooner.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/adobe/aem-upload/issues/35#issuecomment-783688294,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABDZGWY3WITGFAY2E3F2Q7LTALEKJANCNFSM4W7A3HBA
.

Version 1.3.0 has introduced a withProxy() option to DirectBinaryUploadOptions. When specified, all requests sent by the library will go through the specified proxy.


browset.get() method sometimes freezes Protractor

Dear Protractor team,
We have this very confusing issue. When running a test suite, Protractor occasionally freezes at the step that calls the browser.get() method. To make matters worse, it doesn't happen always, but it happens frequently enough to be painful (about every third or fourth run of the test suite). Here is what we could figure out so far:

we use the Cucumber.js framework, and the freezing episodes occur only when the whole suite (of 18 scenarios) is run; never when running individual scenarios;
in our test suite, browser.get is called multiple times — yet in those ~30% of test runs that result in Protractor's freezing, this freezing occurs in the middle of the test suite, after browser.get has been successfully called several times (also, it is always the same step during which the freezing occurs);
before Protractor hangs, the code up to browser.get in the problematic step gets executed without any problems;
the browser does not visit the url passed as an argument to browser.get, but just hangs indefinitely;
the Node console shows that no GET request has been sent
console loggers placed in the Protractor.prototype.get method (in node_modules/protractor/lib/protractor.js) show that:

the method gets called
it receives the correct url as the destination argument
it starts the first for loop (for (var i = 0; i < this.mockModules_.length; ++i))
it, however, does not execute Protractor's driver.wait method: the following console logger does not get registered

  this.driver.wait(function() {
        console.log('1. I am inside Protractor. And I was told to visit:', destination);

At this point, we are completely mystified. Could you please suggest which parts of Protractor's code we could check further to narrow down the possible reason of this strange behavior?
P.S.: We use Protractor v. 1.8, but v. 2.0 behaves the same.

I have the same issue.
Protractor stopped working.
It opens browser successfully but not sends url to it.

Updated: Please disregard the P.S. in my message above — I realized that I have not tested v.2.0 properly. V.2.0 is now giving me a different kind of trouble (it fails some tests complaining that UnknownError: <unknown>: Failed to set the 'cookie' property on 'Document': Cookies are disabled inside 'data:' URLs.), but it hasn't frozen on me yet.

Are you able to reproduce using another test framework? That would help us narrow it down.
I'm guessing it's some sort of control flow synchronization issue - usually when things are hanging unpredictably that's the culprit. There was an issue that is fixed for the next release of selenium-webdriver - https://github.com/SeleniumHQ/selenium/issues/444

We haven't tried any other frameworks. However I've been experimenting with Protractor v.2.0.0 over the past 3 days, and these hanging episodes never occurred when testing with v.2.0.0. So I guess my original question may be closed now.
Protractor v.2.0 seems to have a more fragile control flow than its predecessor though, because it would sporadically fail the tests that worked fine on v.1.8. We are basically coding the control flow manually now, explicitly writing all the then's, and still certain steps would fail from time to time, usually the same ones, but quite inconsistently.

I'm hoping that the resolution to the issue I linked above will help with
the fragility.
On Mon, May 18, 2015 at 7:47 PM, Andrey Azov<EMAIL_ADDRESS>wrote:

We haven't tried any other frameworks. However I've been experimenting
with Protractor v.2.0.0 over the past 3 days, and these hanging episodes
never occurred when testing with v.2.0.0. So I guess my original question
may be closed now.
Protractor v.2.0 seems to have a more fragile control flow than its
predecessor though, because it would sporadically fail the tests that
worked fine on v.1.8. We are basically coding the control flow manually
now, explicitly writing all the then's, and still certain steps would
fail from time to time, usually the same ones, but quite inconsistently.
—
Reply to this email directly or view it on GitHub
https://github.com/angular/protractor/issues/2144#issuecomment-103310253
.

I am currently having this problem in version  2.1.0.
The hang occurs on driver.browser.get(url), and occurs consistently at the start of the second scenario in a feature file. This means that the first scenario plays through to completion, the current browser is closed, a new one is opened, and the first step in the next scenario (the get) hangs. The only configuration I've done is:
  var seleniumAddress = 'http://localhost:4444/wd/hub';
  var options = { browser : 'firefox', timeout : 100000};
  this.World = pc.world(seleniumAddress, options);

  this.After(function(callback) {
	    this.quit(callback);
	  });

@djlaplante Are you using grunt-protractor-runner? If so, try without it. I had the same problem and was able to solve it, well, now at least most of the time all the suits run without hanging. Anyway, due to the most of the time I've got to +1 here.

@borisdiakur Nope, just regular ol' protractor-cucumber run with protractor I'm afraid :/

Using Jasmine. Still hanging occasionally.

Having the same issue using protractor 2.1 with Jasmine

+1

+1

Running with the --troubleshoot flag I see that the WebDriver session is not started successfully in our case. So -1 in favour of #2419

We had the same problem using protractor 2.2.
I edited config.json in the protractor root folder, changed chromedriver version to 2.19 and downloaded  the new version of cromedriver with webdriver-manager update. That solved the freeze-problem for us.

Closing this because we split cucumber off with https://github.com/angular/protractor/pull/2710

I still encounter this problem with protractor 5.3.2, geckodriver 0.20.1 and Firefox 60.0.1. Was there a solution for this issue? I don't really get why #2710 whould fix it. Can you explain @sjelin ?

@timnederhoff
First of all this is an old issue, so nobody will look at this (instead of me ;-)).
Secondly @sjelin currently isn't working on the project, so the change he will react is small.
If this is still an issue please create a new issue with a detailed description in it so we can reproduce it
Tnx in advance

Hi @wswebcreation Thanks for noticing! I'll create a new issue then

@wswebcreation , sorry that ask you here but does @sjelin supports webdriver-js-extender now? In case he doesn't, who I should ask for issue review?

Hi @IgorSasovets
I don't know who's picking this up. This issue is already closed.


Error building: Source option 6 is no longer supported. Use 7 or later.

BUG REPORT

Please describe the issue you observed:

I'm unable to run the built version of RocketMQ for some reason (will file a separate issue for that), so I've tried to compile it using Maven as described at https://rocketmq.apache.org/docs/quick-start/. However, the project won't build, something about a "source option". I have basically zero Java skill so I don't know what that means - I just need it fixed so I can run RocketMQ to work on a Golang project.

What did you do (The steps to reproduce)?

$ pwd
/Users/jah/Downloads/rocketmq-all-4.7.1-source-release
$ mvn -Prelease-all -Dskiptests clean install -U
# <snip>
[INFO] -------------------------------------------------------------
[WARNING] COMPILATION WARNING :
[INFO] -------------------------------------------------------------
[WARNING] bootstrap class path not set in conjunction with -source 6
[INFO] 1 warning
[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] Source option 6 is no longer supported. Use 7 or later.
[ERROR] Target option 6 is no longer supported. Use 7 or later.
[INFO] 2 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache RocketMQ 4.7.1 4.7.1:
[INFO]
[INFO] Apache RocketMQ 4.7.1 .............................. SUCCESS [  2.247 s]
[INFO] rocketmq-logging 4.7.1 ............................. FAILURE [  0.419 s]
[INFO] rocketmq-remoting 4.7.1 ............................ SKIPPED
[INFO] rocketmq-common 4.7.1 .............................. SKIPPED
[INFO] rocketmq-client 4.7.1 .............................. SKIPPED
[INFO] rocketmq-store 4.7.1 ............................... SKIPPED
[INFO] rocketmq-srvutil 4.7.1 ............................. SKIPPED
[INFO] rocketmq-filter 4.7.1 .............................. SKIPPED
[INFO] rocketmq-acl 4.7.1 ................................. SKIPPED
[INFO] rocketmq-broker 4.7.1 .............................. SKIPPED
[INFO] rocketmq-tools 4.7.1 ............................... SKIPPED
[INFO] rocketmq-namesrv 4.7.1 ............................. SKIPPED
[INFO] rocketmq-logappender 4.7.1 ......................... SKIPPED
[INFO] rocketmq-openmessaging 4.7.1 ....................... SKIPPED
[INFO] rocketmq-example 4.7.1 ............................. SKIPPED
[INFO] rocketmq-test 4.7.1 ................................ SKIPPED
[INFO] rocketmq-distribution 4.7.1 ........................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.820 s
[INFO] Finished at: 2020-10-31T00:31:31-06:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.5.1:compile (default-compile) on project rocketmq-logging: Compilation failure: Compilation failure:
[ERROR] Source option 6 is no longer supported. Use 7 or later.
[ERROR] Target option 6 is no longer supported. Use 7 or later.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :rocketmq-logging

What did you expect to see?
Successful build message

What did you see instead?
An error in building the logging component, "Source option 6 is no longer supported. Use 7 or later."

Please tell us about your environment:

$ echo $JAVA_HOME
/Library/Java/JavaVirtualMachines/jdk-15.0.1.jdk/Contents/Home
$ mvn --version
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /Users/jah/.brew/Cellar/maven/3.6.3_1/libexec
Java version: 15.0.1, vendor: Oracle Corporation, runtime: /Library/Java/JavaVirtualMachines/jdk-15.0.1.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.15.7", arch: "x86_64", family: "mac"
$ java -version
java version "15.0.1" 2020-10-20
Java(TM) SE Runtime Environment (build 15.0.1+9-18)
Java HotSpot(TM) 64-Bit Server VM (build 15.0.1+9-18, mixed mode, sharing)
$ uname -a
Darwin nova.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64

@jahio Plz uses java8 instead of java15.

How did you solve this problem? I met the same one

I had the same issue today. I realized it would take too much effort to fix it; fortunately, you can download RocketMQ binary from their site. It's more convenient to do so if you're not planning to do any changes in source code.


Adding DatePicker option on edit & Drilldown option in table row

Is there any way we can add datepicker option on edit to select a date?
Also we have a scenario to drilldown data by multiple levels in data table?
I have added screenshots for the same,
DatePicker Editable field:

Drilldown data:

Closing due to age and release of https://www.predix-ui.com/#/elements/data-grid/px-data-grid


Type parser needs to support multi-dimensional arrays

bv.parse_type_string("char[8][64]") Fails. This is something we should support.

Fixed in 2.3.2760


Port fix from #5807 to 1.1.X - Race condition in ControllerActionInvoker causes NullRef

See #5807
We need to patch this in 1.1.X

This patch fix is approved. Please follow the normal code review / pull request process and make sure you make the change in the correct branch.

@rynowak - here's the bug we were talking about. Apparently I forgot to assign to you, sorry!
Please get the fix in ASAP.

@rynowak ping.

Is this still waiting on @pranavkm to unblock something?

55873bcbfc17d215237d8fbb4a082fb2ff8a0fdf

Preview builds of this patch fix should be available on the following feeds:

https://dotnet.myget.org/gallery/aspnet-1-0-5-may2017-patch-public (for fixes in the 1.0.x release)

Feed URL to add to NuGet.config: https://dotnet.myget.org/F/aspnet-1-0-5-may2017-patch-public/api/v3/index.json

https://dotnet.myget.org/gallery/aspnet-1-1-2-may2017-patch-public (for fixes in the 1.1.x release)

Feed URL to add to NuGet.config: https://dotnet.myget.org/F/aspnet-1-1-2-may2017-patch-public/api/v3/index.json

If you have a chance, please try out these preview builds and let us know if you have any feedback!


Bug: can not decompress for some compressed string

Hi,
I have a case to compress json string, bug can not decompress correctly;
input:
const data = {"tabs":[],"config":{"editable":false,"pinnedTab":false}}

then call LZString.compress(JSON.stringify(data))
output:
'㞂ࡠ䘌쁭˨ڄذ㶀急胦઀⦀☆䔀㙤➄듆稀㠓躔ʭ⍦갂﬊\x00'

then JSON.parse(LZString.decompress(str))
got null, can not get back to origin data.
can you help me?

Please do not use the compress method, which produces an output that cannot be reliably stored in some some browsers, as specified in the documentation. Use compressToUTF16 instead (and decompressFromUTF16 to decompress)

Hi, Ive had issues with compressToUTF16, instead you should use compressToBase64 if you are sending the request using HTTP.

I have had issues with UTF16,
check this issue:
https://github.com/pieroxy/lz-string/issues/92

@yarden-app There is no issue with compressToUTF16 if you feed decompress with the exact output of compress. If, however, you convert that UTF16 to something else, such as UTF8 then back to UTF16, chances are the UTF16 string you get back is not what you had in the beginning, so obviously you cannot decompress it.


BSDF model reset to Disney BRDF after using the BSDF picking mode

Picking an object using the BSDF mode and starting a region render resets the BSDF to the Disney BSDF.

Fixed.


Difference from Ackee?

This project looks interesting, though fairly similar from what I can tell to electerious/Ackee. Do you have plans to add any different features?

There are plenty of similar products in the space like Fathom, Plausible and Matomo. I really just wanted to focus on a fast user experience. Where you get to your data with few clicks as possible.


Task Scheduling

Not sure if it is already a feature or not, I am fairly new at using rclone or rclone browser.
Is there a possibility to schedule individual tasks for a consistent syncing? Like every hour or half an hour. Although this would create an inconsistency if say, I modify a file on my other device before that half hour is complete. In which case, the optimal option would be for the task to keep running forever. Is that an optimal solution or am I better off using google's app on windows?
Edit: I think the other option could be to trigger the sync upload based on the application launch, and then sync download when exit.

There is no scheduler in Rclone Browser. I see it more as a tool to transfer data to/from cloud. And try to focus development on making it working well.
I am not sure it would be useful. There are many other backup/archive solutions available providing functionality beyond simple file copy e.g. versioning or incremental backup.
Me personally if I want to use cloud as a backup destination I use Arq or syncthing. Everything depends on your requirement.

On 12 Feb 2020, at 14:26, RaXorX<EMAIL_ADDRESS>wrote:

Not sure if it is already a feature or not, I am fairly new at using rclone or rclone browser.
Is there a possibility to schedule individual tasks for a consistent syncing? Like every hour or half an hour. Although this would create an inconsistency if say, I modify a file on my other device before that half hour is complete. In which case, the optimal option would be for the task to keep running forever. Is that an optimal solution or am I better off using google's app on windows?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

I know this is closed but I wanted to throw it out there that I would really like this as well. If it had a simple scheduler to run a task once an hour, day, week, on file change, etc, there would be no need for me to have another application. I already have this setup with my NAS as a destination, as well as my cloud locations. It already has tasks. The only thing missing from making this a very well rounded backup solution is a way to schedule those tasks.

Perfect timing. I am working on this for the last few weeks. It will be included in the next release soon. Scheduler and many other improvements

This is great news! Very glad to hear. I look forward to the release!

In a similar vein, being able to schedule run all tasks (with an option to do 1 by 1 or n at a time in parallel) would be great.

Yes, Scheduler feature should be added as it will help user to do backup on regular basis.

Glad to see it was reopened and already in wip. :)


nu_to_M different results for same inputs in degree or radians

The function nu_to_M gives different results when inputting nu in degrees or radians.
Doesn't say the input can not be degrees.

nuu=150 (in degrees)
ecc=0.08
np.rad2deg(nu_to_M((np.deg2rad(nuu)*u.rad), (ecc)))%360
OUT:<PHONE_NUMBER>1222044
(nu_to_M(((nuu*u.deg)), (ecc)))%360
OUT:<PHONE_NUMBER>7328553
💡 Possible solutions

add error message
add check for degrees or radians within function
add radians requirement in documentation

Hi @WolfsSky,
the poliastro.angles.nu_to_M function from the high-level API (the one accepting astropy.units.Quantity inputs) works fine. Furthermore, if not passing a Quantity instance it raises an error. If you are using the poliastro.core.angles.nu_to_M your input should be always in radians.
I wrote a small gist to show this behaviour.

Ah! That makes sense, I did indeed use the core, and not the angles version. Thanks for clarifying :)

Ups, sorry! In fact it is poliastro.twobody.angles the one for the high-level API :relaxed:

Yes I tried from poliastro.angles but it failed then, but thanks!


Not able to export the project to firebase

Getting below exception. I have noticed that in the readme, "-vcm"  is mentioned along with the project id. I am not sure, do we really need to keep "-vcm" as i can see that my bucket url and project id doesn't contain "-vcm" anywhere.
Error Exception: Error while initiating importing the dataset: {"message":"3 INVALID_ARGUMENT: Cannot import from gs://projectId/dataset/labels.csv; it is not inside ['^projectId-vcm$', '^ips_perm_20343534080625_575976234244573$']"}

Don't know why it is used for but able to resolve it by adding "-vcm" with project id in constants


.gz doesn't return correct file size for content above 2^32 B = 4 GB

1.0.0-beta13
Linux 64bit

Taking it out from #629, apparently the gzip file format cannot accurately return the size of
files above 4GB (2^32 bytes), but returns the modulo.
Looks like on the command line people recommended something like zcat file.gz | wc -c or gzip -dc file | wc -c which give the correct value - though then decompresses the the file twice. Might have to do that for gzip in the end, though, since likely >4GB files are common for Etcher's use case.
This might let images to start to be burned onto cards that are too small (in worst case), or affects the progress bar.
From testing with a 4100MiB > 4096MiB image, indeed .gz version lets to select a 512MB SD card, while the same file's .xz archive does not.
For the progress bar, the MB/s reading seems to be affected (shows very low speed, eg. 0.01MB/s)  but the progress percentage does not (shows correctly for the burning process), so it's not too bad.

So as far as I understand, the user will eventually hit ENOSPC, without any further weird behaviour, right?
I would love to be able to reliably get the drive size (or at least closely estimate), however decompressing the whole thing twice sounds like a very bad solution.
I'll research on this and see if there is a way we can do it.

@jviotti I think your description is still mixing up two uncovered issues, broken out to two parts to here and to #629. In that other issue (as described the reproducible ENOSPC) that happens regardless of compression.
In this issue:

ENOSPC would happen if there's a gz image with SIZE > 2^32 bytes,  and the user is trying to burn onto a card which has CAPACITY < SIZE but CAPACITY > SIZE mod 2^32. Then it would cause an issue, because the initial capacity check in etcher couldn't figure out the correct size

If using a card that CAPACITY > SIZE, then the only effect is that the "speed" bar is wrong, but everything else works properly (including the progress bar), and the user won't run into ENOSPC.

Decompressing things twice might be a bad solution, but judging by the comments, it's just gz not designed for these big files (nor to return correct size estimate either), so curious to see if there's any other solution than run through the file twice. Should not be too bad, especially if that has some UI display such as "checking archive contents", so people know it not all just hung. I think doing things correctly for gz is more important than taking a bit longer time. Decompression itself with no data storage just byte counting seems to be pretty fast.

If using a card that CAPACITY > SIZE, then the only effect is that the "speed" bar is wrong, but everything else works properly (including the progress bar), and the user won't run into ENOSPC.

I see. I wonder why the speed is wrong. I can't think of a way this
issue could affect the speed (unless GZ files >4GB are very slow to
decompress?).

Decompressing things twice might be a bad solution, but judging by the
comments, it's just gz not designed for these big files (nor to
return correct size estimate either), so curious to see if there's any
other solution than run through the file twice. Should not be too bad,
especially if that has some UI display such as "checking archive
contents", so people know it not all just hung. I think doing things
correctly for gz is more important than taking a bit longer time.
Decompression itself with no data storage just byte counting seems to
be pretty fast.

Yeah, could be, but needs more thought. I'd love to push a bit more to
see if we can find an alternative solution. We're putting an enourmous
amount of effort to reduce the time from getting an image to the drive,
and it sounds counter-intuitive to be willing to spend time
decompressing things twice.
On Sun, Aug 21, 2016 at 07:57:09PM -0700, Gergely Imreh wrote:

@jviotti I think your description is still mixing up two uncovered issues, broken out to two parts to here and to #629. In that other issue (as described the reproducible ENOSPC) that happens regardless of compression.
In this issue:

ENOSPC would happen if there's a gz image with SIZE > 2^32 bytes,  and the user is trying to burn onto a card which has CAPACITY < SIZE but CAPACITY > SIZE mod 2^32. Then it would cause an issue, because the initial capacity check in etcher couldn't figure out the correct size

If using a card that CAPACITY > SIZE, then the only effect is that the "speed" bar is wrong, but everything else works properly (including the progress bar), and the user won't run into ENOSPC.

Decompressing things twice might be a bad solution, but judging by the comments, it's just gz not designed for these big files (nor to return correct size estimate either), so curious to see if there's any other solution than run through the file twice. Should not be too bad, especially if that has some UI display such as "checking archive contents", so people know it not all just hung. I think doing things correctly for gz is more important than taking a bit longer time. Decompression itself with no data storage just byte counting seems to be pretty fast.
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/resin-io/etcher/issues/638#issuecomment-241304873

--
Juan Cruz Viotti
Software Engineer

As has already been mentioned, if the uncompressed file is over 4GB, gzip returns its size modulo 4GB (because it only has a 32bit size field) - gzip would report a 4.5GB file as 0.5GB, a 9.3GB file as 1.3GB, and a 15.9GB file as 3.9GB. So therefore the only way to correctly get the size is to uncompress the whole file first.
However I wonder if you could use some kind of heuristic to 'guess' when gzip has wrapped the size of the compressed file, based on the size of the uncompressed file? (I'm guessing the disk images used with Etcher probably have vaguely similar compression ratios).
E.g. if the compressed .gz file is bigger than N GB, and gzip reports the uncompressed file as being M GB, perhaps you could 'deduce' that the size of the uncompressed file is actually M+4 GB?
Obviously you'd need to do some experimentation with different disk-images to work out a reliable value for N.
(and similarly if the compressed .gz file is bigger than N*2 GB, you could deduce that the uncompressed size is actually M+8 GB)
Note that this heuristic is still likely to fail though if you have a compressed disk-image with a lot of blank unpartitioned space (because blank space compresses really well.).

I see. I wonder why the speed is wrong.

Well, I guess if Etcher thinks the uncompressed file is only 0.5GB, but it's actually 4.5GB, perhaps the speed is calculated based on the 0.5GB figure? (4.5GB obviously takes a lot longer to write than 0.5GB would!)
P.S. Obviously where I've said '4GB' everywhere above, it's just a shorthand for '2^32 bytes'.

Revisiting this issue, I am with @lurch on this. He is probably right about the heuristic approach and about the speed explanation. @jviotti?

Sounds good, lets experiment with it.

Obviously where I've said '4GB' everywhere above, it's just a shorthand for '2^32 bytes'.

I'd suggest that you guys watch your units more closely, especially when building UI or write logic. 2^32 bytes is actually 4GiB (gibibytes), not GB (gigabytes).

I can never remember which way around they are :-/
And as Etcher is aimed at novices, I expect they might get confused if we started reporting everything in GiB instead of just GB ?

I can never remember which way around they are :-/

The simple way to remember it is to compare it to metric measurements. If it has an SI prefix like metric units do, you know it's a power of ten (as with metric).

And as Etcher is aimed at novices, I expect they might get confused if we started reporting everything in  GiB  instead of just GB?

As I see it, if a novice doesn't know what gibibytes are, they'll probably just assume that GiB is "Gigabytes". That's a close enough approximation for said novice's uses, I'd imagine.

@jviotti did anyone end up doing anything with this? It really should be a fairly simple fix for 99% of cases

(this is our oldest bug still open)

Is there actually anyone using gzip to compress images over 4GB in size? I.e. does anyone have some example images that can be used for testing, or is this just an edge-case that we'll (probably) never hit in practice?

I've seen some out there, although its definitely rare. It shouldn't be hard to fix the speed + percentage issues though. I'm happy if we treat gzip sizes with a grain of salt and eventually throw ENOSPC if there is no remaining space in the drive, like we do with bzip2.

@jhermsmeier Can you help me out with this one? Try flashing an gz image directly with the Etcher CLI. The flash progress quickly reaches 99%, and remains there for quite some time while there's more data coming through the stream chain. Maybe we're not calculating the size correctly somewhere?

Oh, the sizes seem to be fine. The issue is that for compressed images, the stream chain looks like this:

Input compressed file
Calculate progress based on compressed size
Apply decompression transform
Write to drive

It looks like the decompression is very fast compared to the drive writing, and therefore the progress reaches 99% much sooner.
I'm not sure what would be the solution here. In some cases we only have the uncompressed size, so we should make use of it to show the progress. Maybe there is a way to make the decompressor stream wait for the drive writing?

Looks like pausing/resuming the readable stream should do it.

I investigated the slow speed issue in more detail, and it looks to be resolved in master. There is a speed penalty (~2.0 MB/s) when decompressing large files though, but its not even closer to @imrehg initial report (~0.01MB/s).
The slow speed seem to happen on larger compressed images, and also depends on the compression level. I ran various experiments with images of several sizes (from ~1 GB to ~4 GB), using compression levels 1 to 9, inclusive, and the decompression time drastically increases on larger images, which can be reproduced with the gzip tool as well.
In summary:

Images that don't fit into the drive will eventually cause ENOSPC, leading a friendly message being presented to the user
The absurd speed times issue was fixed
We're hitting a UX issue where the decompression time is much faster than the flashing time (for certain images)

I think we can close this issue once the last one is fixed.

@jhermsmeier Check http://linorg.usp.br/OpenELEC/OpenELEC-Generic.x86_64-6.0.3.img.gz for an image that showcases the 99% UX issue.

the decompression time drastically increases on larger images

Isn't that expected?!?  I'd expect a streaming gzip decompressor to have a constant(ish) decompression speed, so of course a larger image will take longer to decompress.
Or are you saying that larger images decompress at a slower speed (bitrate) than smaller images, in which case that sounds like it might be a resource-leak somewhere?
I'm sure @jhermsmeier , our streaming expert, will have some better ideas ;-)

I mean that decompression time and image size are not directly proportional. As the image gets larger (mainly if you compress with the highest compression level), decompression starts getting ridiculously slow.
I ran all my experiments with the gzip CLI tool, I didn't test on Etcher at all.

Interesting - I'd expect the streaming block-based nature of gzip (based on top of zlib) would mean there'd be a direct proportion between decompression time and image size? Perhaps it depends on the dictionary size, or perhaps I'm completely misunderstanding this ;-)

Yeah, I don't know. Maybe it also depends on the image itself? I'm a compression noob, so I have no clue apart from what I saw on my experiments.

Beyond all this though, we should have a heuristic that basically says this:

gzip compresses images within a certain range (e.g. 1.5x to 3x)
if an image claims to be much out of that range (e.g. says it's 300mb but
the archive is 2.5gb) we should assume it's actually 4.3gb instead.
Essentially, we should add 2^32 bytes to the estimated size again and
again, until the compression ratio gets within a realistic range.

I think an algorithm like this, used only for gzip files (maybe bzip too?),
should fix the vast majority of the cases. We should still fail well when
we're wrong, but we should try hard to be right :)
--
Alexandros Marinos
Founder & CEO, Resin.io
+1<PHONE_NUMBER>
@alexandrosm
On Fri, Mar 3, 2017 at 3:51 PM, Juan Cruz Viotti<EMAIL_ADDRESS>wrote:

Yeah, I don't know. Maybe it also depends on the image itself? I'm a
compression noob, so I have no clue apart from what I saw on my experiments.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/resin-io/etcher/issues/638#issuecomment-284102753,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABLUCKLN91vbMP2jKNkGcuVVVW_-u52jks5riKeIgaJpZM4Joo50
.

I agree. Let's add that estimation algorithm to https://github.com/resin-io/etcher/blob/master/lib/image-stream/handlers.js#L84

Looking at #1171 I've just had another idea about an alternative approach to this...
The only "proper" way to get the full size of a .gz or .bz2 compressed image, is to extract the whole thing, and count how many bytes get spat out (i.e. something like gunzip -c someimage.img.gz | wc -c on the command-line). However this can be a slow process for larger images, so we don't want to wait to extract the whole image to count how big it is, before we start flashing it to the user's SD card.
However the time taken to extract the image in the background is obviously going to be less than the time taken to actually write the image to the SD card, so I guess one approach could be to start decompressing and writing the image to the SD card as normal, but then in another background thread, decompress the image in its entirety to see how big it is. Once that size-counting is finished, the 'foreground' thread that's actually writing the image to the SD card could be updated with how big the image actually is, and the writing process could then update the percentage-bar to display the true progress, based on the image's actual size (or it could early-abort the writing process if it discovers the image is actually bigger than the card).
It'd look a bit visually-jarring with the percentage bar suddenly taking a jump backwards (if the true size is discovered to be bigger than the estimated size), but IMHO that'd still be much better than the "write speed slows to a crawl" behaviour that we currently have.
Thoughts? (have I explained that clearly enough?)

I guess it depends on how much more reliable it will be than the heuristic
method. I think it won't be much better, so I'd rather do the heuristic and
see how far we get before we resort to extreme measures...
--
Alexandros Marinos
Founder & CEO, Resin.io
+1<PHONE_NUMBER>
@alexandrosm
On Thu, Mar 9, 2017 at 8:55 AM, Andrew Scheller<EMAIL_ADDRESS>wrote:

Looking at #1171 https://github.com/resin-io/etcher/issues/1171 I've
just had another idea about an alternative approach to this...
The only "proper" way to get the full size of a .gz or .bz2 compressed
image, is to extract the whole thing, and count how many bytes get spat out
(i.e. something like gunzip -c someimage.img.gz | wc -c on the
command-line). However this can be a slow process for larger images, so we
don't want to wait to extract the whole image to count how big it is,
before we start flashing it to the user's SD card.
However the time taken to extract the image in the background is
obviously going to be less than the time taken to actually write the image
to the SD card, so I guess one approach could be to start decompressing and
writing the image to the SD card as normal, but then in another
background thread, decompress the image in its entirety to see how big it
is. Once that size-counting is finished, the 'foreground' thread that's
actually writing the image to the SD card could be updated with how big the
image actually is, and the writing process could then update the
percentage-bar to display the true progress, based on the image's
actual size (or it could early-abort the writing process if it discovers
the image is actually bigger than the card).
It'd look a bit visually-jarring with the percentage bar suddenly taking a
jump backwards (if the true size is discovered to be bigger than the
estimated size), but IMHO that'd still be much better than the "write speed
slows to a crawl" behaviour that we currently have.
Thoughts? (have I explained that clearly enough?)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/resin-io/etcher/issues/638#issuecomment-285410734,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABLUCOceT5J6bQqsDM8kXfXDoHPhnOYCks5rkC7ngaJpZM4Joo50
.

As I explain in https://github.com/resin-io/etcher/issues/1171#issuecomment-285402214 (and see also the table in a comment further down that page) we already have cases where even the best-tuned heuristic will still give incorrect results. I can't see any scenario (unless I'm missing something obvious?) where what I describe above would be "unreliable".
Is spawning a background thread to do an extra decompress of the image really an "extreme measure"? ;-) (although I agree that the cross-thread communication may be tricky - that's something I've never done in NodeJS)
And what I describe above could also work in addition to the heuristic, it's not an either-or scenario. i.e.

user selects gzip-compresssed image
get uncompressed size from gzip
apply heuristic to try to get a 'better' uncompressed size
start writing image to disk
start background decompression thread
(...some time later...) background-decompression-thread gives us the true uncompressed size (which may or may not match the heuristic size)
Adjust the progress-bar if necessary, and continue writing the image

I think you're mixing up two different problems. The heuristic IS NOT used at all for the writing, but only to prevent a possibly too small drive from being selected. When the writer kicks off, it uses the compressed size only. The reason for the "write speed slows to a crawl" is not related to the uncompressed heuristic.

Sorry for getting confused. But surely it would make sense for the writer to report write-progress as the percentage of data written to disk out of the total size of the (uncompressed) image? Isn't that what most people would expect? :confused:

No such thing as threads in Node userland ;)
The equivalent of gunzip file > /dev/null while counting bytes and keeping time in Node:
function gunzipSize( filename, callback ) {

  var size = 0

  fs.createReadStream( filename )
    .on( 'error', callback )
    .pipe( zlib.createUnzip() )
    .on( 'readable', function() {
      var chunk = null
      while( chunk = this.read() ) {
        size = size + chunk.length
      }
      chunk = null
    })
    .on( 'error', callback )
    .on( 'end', function() {
      callback( null, size )
    })

}

var filename = path.join( process.env['HOME'], 'Downloads', 'android-ver6.0-20170112-pine64-32GB.img.gz' )
var stats = fs.statSync( filename )
var time = process.hrtime()

gunzipSize( filename, ( error, size ) => {

  time = process.hrtime( time )

  var ms = ( time[0] * 1e3 ) + ( time[1] / 1e6 )
  var minutes = ( ms / 1000 / 60 ) | 0
  var seconds = ( ms / 1000 ) - ( minutes * 60 )

  console.log( 'Finished in %s min %s s', minutes, seconds.toFixed(1) )

  if( error ) {
    return console.log( error )
  }

  console.log( 'Uncompressed size: %s GB (%s bytes)', (size / 1024 / 1024 / 1024).toFixed(1), size )
  console.log( 'Compressed size: %s GB (%s bytes)', (stats.size / 1024 / 1024 / 1024).toFixed(1), stats.size )

})

Ends up chewing through at 100% CPU (1 core, obv.), and about 115 MB RAM during a run of it on my Macbook Air from 2012:
Finished in 3 min 21.7 s
Uncompressed size: 28.8 GB<PHONE_NUMBER>4 bytes)
Compressed size: 0.8 GB (815284283 bytes)

Which means, if this were to run in parallel, we'd be bottlenecking the CPU (or I/O if the source is slow). I think heuristic + counting bytes while writing is probably the less resource-intensive path.

The current implementation calculates the percentage based on how much was decompressed from the file. For most decompression algorithms we're using, writing to the drive is faster than decompressing, so the progress bar displays fine, however gzip seems to be a special case, because its quite fast.
I avoided heuristics and relied on that approach given that for some compression methods (e.g: bzip2), its impossible to get even an estimate, and we'd be really guessing in the dark.
I believe that we can pause decompression if we're getting slow on writes, and that should work fine.

No such thing as threads in Node userland ;)

From what I remember reading about Electron when I first started working on Etcher, I thought it was designed around having separate co-operating processes?

if this were to run in parallel, we'd be bottlenecking the CPU

I wonder if we can assign threads/processes to have a lower priority than the main thread? Given that we're going to have lots of I/O waits waiting for the SD card to write, there should be plenty of 'spare' CPU time-slots.

or I/O if the source is slow

Well, I'd still expect even reading from a 'slow' disk to be faster than writing to a SD card?

For most decompression algorithms we're using, writing to the drive is faster than decompressing

Yikes! :-(  Is that because they're implemented in pure-javascript, and JS isn't designed for manipulating binary data? How much is it slowing things down by - would it be worth trying to find native-binding equivalents? (if I'm using the right terminology)
A benchmark from 12 years ago (wow!) shows even the most highly-compressed bzip2 file decompressing at over 5MB/s. (A more recent benchmark shows bzip2 decompressing at over 20MB/s)

however gzip seems to be a special case, because its quite fast.

Maybe that points to gzip being implemented in C rather than JS?

given that for some compression methods (e.g: bzip2), its impossible to get even an estimate

Yeah, the background-decompression-thread approach I suggested above would also work for .bz2 files (although I guess it means the progress-indicator would have to switch part-way through from being a "rough estimate, based on how much of the input file has been decompressed", to "an exact figure, based on how much of the output file has been written to disk so far").

I think heuristic + counting bytes while writing is probably the less resource-intensive path.

I believe that we can pause decompression if we're getting slow on writes, and that should work fine.

Fair enough, looks like I've been out-voted ;-)

Andrew -- just to clarify, I said that the heuristic approach will be less
reliable but much much simpler to write and execute, not the other way
around.
--
Alexandros Marinos
Founder & CEO, Resin.io
+1<PHONE_NUMBER>
@alexandrosm
On Thu, Mar 9, 2017 at 12:01 PM, Juan Cruz Viotti<EMAIL_ADDRESS>wrote:

The current implementation calculates the percentage based on how much was
decompressed from the file. For most decompression algorithms we're using,
writing to the drive is faster than decompressing, so the progress bar
displays fine, however gzip seems to be a special case, because its quite
fast.
I avoided heuristics and relied on that approach given that for some
compression methods (e.g: bzip2), its impossible to get even an estimate,
and we'd be really guessing in the dark.
I believe that we can pause decompression if we're getting slow on writes,
and that should work fine.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/resin-io/etcher/issues/638#issuecomment-285464517,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABLUCFdvsHaHIJu6yYJ4P5BbLNQ3P_iuks5rkFqqgaJpZM4Joo50
.

The threads of electron are exactly two: one for ui/browser stuff, and one
for "server"/node stuff.
--
Alexandros Marinos
Founder & CEO, Resin.io
+1<PHONE_NUMBER>
@alexandrosm
On Thu, Mar 9, 2017 at 12:33 PM, Andrew Scheller<EMAIL_ADDRESS>wrote:

No such thing as threads in Node userland ;)
From what I remember reading about Electron when I first started working
on Etcher, I thought it was designed around having separate co-operating
processes?
if this were to run in parallel, we'd be bottlenecking the CPU
I wonder if we can assign threads/processes to have a lower priority than
the main thread? Given that we're going to have lots of I/O waits waiting
for the SD card to write, there should be plenty of 'spare' CPU time-slots.
or I/O if the source is slow
Well, I'd still expect even reading from a 'slow' disk to be faster than
writing to a SD card?
For most decompression algorithms we're using, writing to the drive is
faster than decompressing
Yikes! :-( Is that because they're implemented in pure-javascript, and JS
isn't designed for manipulating binary data? How much is it slowing things
down by - would it be worth trying to find native-binding equivalents? (if
I'm using the right terminology)
A benchmark http://tukaani.org/lzma/benchmarks.html from 12 years ago
(wow!) shows even the most highly-compressed bzip2 file decompressing at
over 5MB/s. (A more recent benchmark
https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-comparison/
shows bzip2 decompressing at over 20MB/s)
however gzip seems to be a special case, because its quite fast.
Maybe that points to gzip being implemented in C rather than JS?
given that for some compression methods (e.g: bzip2), its impossible to
get even an estimate
Yeah, the background-decompression-thread approach I suggested above
would also work for .bz2 files (although I guess it means the
progress-indicator would have to switch part-way through from being a
"rough estimate, based on how much of the input file has been
decompressed", to "an exact figure, based on how much of the output file
has been written to disk so far").
I think heuristic + counting bytes while writing is probably the less
resource-intensive path.
I believe that we can pause decompression if we're getting slow on writes,
and that should work fine.
Fair enough, looks like I've been out-voted ;-)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/resin-io/etcher/issues/638#issuecomment-285474526,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABLUCDVlo-Ih407SAOM4MQbXReSlT0I4ks5rkGIFgaJpZM4Joo50
.

Ahhhh-hhaaaa!!! I've just realised something significant :-D
As seen in https://github.com/resin-io/etcher/issues/1171#issuecomment-285445931 the 29476.5MiB of the android-ver6.0-20170112-pine64-32GB.img gets compressed to just 778MiB in android-ver6.0-20170112-pine64-32GB.img.gz. However (here's the important part) the data isn't distributed evenly throughout the .gz file - there's maybe 2 or 3 GB of actual disk data, and then 20+ GB of binary zeroes (which is why the image compresses so well). However it only takes a few blocks to store those 20+ GB of binary zeroes in the .gz file. So by the time all the 'real data' has been decompressed and written to the SD card, it's totally believable that we are now 95% of the way through reading the gzip file (as @jviotti explains above, the progress-indicator is based on how much of the gzip file has been read so far).
So even if the stream-backpressure stuff is working correctly, the progress indicator will display 95% progress, even though we've actually only written maybe 3GB out of the actual 25GB that we need to write to disk. And of course if the progress bar thinks we're already 95% done, but we've still got another 20+ GB of zeroes to write, that'll lead to the "apparent write speed" dropping so dramatically.
So it's not necessarily just the "gzip being quite fast" that is causing this problem, but the fact that disk images tend to have lots of empty-space at the end, which compresses super-efficiently, and takes up only a very small fraction of the compressed file, but still takes a very long time (much longer than the "real data" in the example in #1171 ) to actually write out to disk.
Does that make sense?

...and on the subject of background processes in Electron, I just found this - @jviotti that's a similar approach to your child-writer stuff isn't it?

Yeah, we can always create child processes and communicate through IPC, which can provide us with some sort of multi-threading.

How about displaying something like a wild guess, and saying so,  if the compressed size is bigger than gzip claims the output file will be. Although it is possible that a compressed file is truly bigger than the output it would be much better than reporting so really bogus number like with a 5GB input reporting 866MB output!
Here's an off the wall idea: Ask the user if they know what the output size is supposed to be! I would expect in most cases they do.
Interesting  side note: 2 years ago lurch commented "Is there actually anyone using gzip to compress images over 4GB in size". Today why would anyone buy a SD card smaller than 16GB!

Dear all,
if anyone is still following this bug ... in bioinformatics we have thousands of ASCII files that are (when compressed) well over 4GiB. This is a very common occurrence, and therefore, the inability of GZIP of listing the correct size of the uncompressed file is a routine nuisance.
As one of many potential examples: ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR277/ERR277077/
Thank you for supporting this amazing utility!

Recently also faced this issue with gzip 1.9
$ gzip -l filename.gz
compressed        uncompressed  ratio uncompressed_name
5627740354<PHONE_NUMBER> -156.0% filename

@tipuraneo It's a limitation of the .gz file-format itself, rather than a 'bug' in any particular implementation of gzip.

I know. So you could say gzip is not the best choice for files > 4 GB? I prefer xz over gzip.


x/tools/imports: TestFindImportVendor and TestProcessVendor fail

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version devel +c6e8342 Mon Mar 14 14:28:49 2016 +1300 linux/amd64
(but also 1.6)

What operating system and processor architecture are you using (go env)?

linux/amd64

What did you do?

go test golang.org/x/tools/imports

What did you expect to see?

tests passing

What did you see instead?

(master)mwhudson@aeglos:tools$ go test golang.org/x/tools/imports
--- FAIL: TestFindImportVendor (0.05s)
	fix_test.go:909: findImportGoPath("hpack", HuffmanDecode ...)="internal/golang.org/x/net/http2/hpack", false, want "golang.org/x/net/http2/hpack", false
--- FAIL: TestProcessVendor (0.05s)
	fix_test.go:942: Process("/opt/opensource/go/src/math/x.go") did not add expected hpack import:
		package http

		import (
			"strings"

			"internal/golang.org/x/net/http2/hpack"
		)

		func f() { strings.NewReader(); hpack.HuffmanDecode() }
FAIL
FAIL	golang.org/x/tools/imports	0.192s

(this happens on build.golang.org too)

I think these fail because Go 1.6's stdlib does not, in the end, use any vendoring but this test still assumes they do.

It will use vendoring again soon.

This was fixed recently.


chore(sharding): possible sharding fix

:tada: This PR is included in version 9.7.6 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Add a message to NotConfigured on ScrapyZyteAPIDownloadHandler

When you don't add a ZyteAPI key to the project, it raises NotConfigured without giving any context.
Please add a message so users can get a better idea of what is going on.
Arigatou.

@felipecustodio said he will help us out


python310Packages.readme_renderer: 36.0 -> 37.0

Description of changes
https://github.com/pypa/readme_renderer/releases/tag/37.0

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 187847 run on x86_64-linux 1

3 packages marked as broken and skipped:

apache-airflow
python310Packages.apache-airflow
python39Packages.apache-airflow

9 packages failed to build:

home-assistant-component-tests.google_translate
python310Packages.restview
python310Packages.spacy
python310Packages.spacy-transformers
python310Packages.textacy
python39Packages.restview
python39Packages.spacy
python39Packages.spacy-transformers
python39Packages.textacy

56 packages built:

check_systemd
dialect
dtrx
evdevremapkeys
keepkey_agent (python310Packages.keepkey_agent)
ledger_agent (python310Packages.ledger_agent)
luigi
mailman-web
mnemosyne
onlykey-agent
pypiserver (python310Packages.pypiserver)
python310Packages.ansible-kernel
python310Packages.ansible-runner
python310Packages.gtts
python310Packages.invocations
python310Packages.langcodes
python310Packages.language-data
python310Packages.libagent
python310Packages.marisa-trie
python310Packages.nagiosplugin
python310Packages.ndjson
python310Packages.python-daemon
python310Packages.readme
python310Packages.readme_renderer
python310Packages.salmon-mail
python310Packages.textx
python310Packages.toggl-cli
trezor_agent (python310Packages.trezor_agent)
twine (python310Packages.twine)
python310Packages.wheel-inspect
python310Packages.wordfreq
python39Packages.ansible-kernel
python39Packages.ansible-runner
python39Packages.gtts
python39Packages.invocations
python39Packages.keepkey_agent
python39Packages.langcodes
python39Packages.language-data
python39Packages.ledger_agent
python39Packages.libagent
python39Packages.marisa-trie
python39Packages.nagiosplugin
python39Packages.ndjson
python39Packages.pypiserver
python39Packages.python-daemon
python39Packages.readme
python39Packages.readme_renderer
python39Packages.salmon-mail
python39Packages.textx
python39Packages.toggl-cli
python39Packages.trezor_agent
python39Packages.twine
python39Packages.wheel-inspect
python39Packages.wordfreq
strictdoc
wlc

Result of nixpkgs-review pr 187847 run on x86_64-linux 1

3 packages marked as broken and skipped:

apache-airflow
python310Packages.apache-airflow
python39Packages.apache-airflow

7 packages failed to build:

home-assistant-component-tests.google_translate
python310Packages.spacy
python310Packages.spacy-transformers
python310Packages.textacy
python39Packages.spacy
python39Packages.spacy-transformers
python39Packages.textacy

58 packages built:

check_systemd
dialect
dtrx
evdevremapkeys
keepkey_agent (python310Packages.keepkey_agent)
ledger_agent (python310Packages.ledger_agent)
luigi
mailman-web
mnemosyne
onlykey-agent
pypiserver (python310Packages.pypiserver)
python310Packages.ansible-kernel
python310Packages.ansible-runner
python310Packages.gtts
python310Packages.invocations
python310Packages.langcodes
python310Packages.language-data
python310Packages.libagent
python310Packages.marisa-trie
python310Packages.nagiosplugin
python310Packages.ndjson
python310Packages.python-daemon
python310Packages.readme
python310Packages.readme_renderer
python310Packages.restview
python310Packages.salmon-mail
python310Packages.textx
python310Packages.toggl-cli
trezor_agent (python310Packages.trezor_agent)
twine (python310Packages.twine)
python310Packages.wheel-inspect
python310Packages.wordfreq
python39Packages.ansible-kernel
python39Packages.ansible-runner
python39Packages.gtts
python39Packages.invocations
python39Packages.keepkey_agent
python39Packages.langcodes
python39Packages.language-data
python39Packages.ledger_agent
python39Packages.libagent
python39Packages.marisa-trie
python39Packages.nagiosplugin
python39Packages.ndjson
python39Packages.pypiserver
python39Packages.python-daemon
python39Packages.readme
python39Packages.readme_renderer
python39Packages.restview
python39Packages.salmon-mail
python39Packages.textx
python39Packages.toggl-cli
python39Packages.trezor_agent
python39Packages.twine
python39Packages.wheel-inspect
python39Packages.wordfreq
strictdoc
wlc


CRC Error

Hi biancode,
I got "CRC Error" when I try to read more than 1 register over Modbus serial RS485 (sparkfun shield). If I read just 1, it works flawlessly.

Hi, please I need more information about your device or a manual link. CRC is Big Endian by default - some implementations on devices are wrong in that case. If it is to configure check that CRC configuration at your device, please.

The device is called PM800 by Schneider Electric. You can find the modbus register list here
In the previous version 0.8.2, I can select the endian (big or little) but the option is not there anymore since I updated to version 1.0.1.
I cannot change device's CRC configuration.
"Manufacturing Data CRC" value is 48251. I don't know what this means.

please, post your flow

[{"id":"32a26606.f81a2a","type":"modbus-read","z":"6bedd68d.a19a68","name":"R1","showStatusActivities":false,"unitid":"","dataType":"HoldingRegister","adr":"1699","quantity":"2","rate":"10","rateUnit":"s","server":"a833778d.035c68","x":152.00003051757812,"y":554.6000213623047,"wires":[["1b0d63dc.53368c"],[]]},{"id":"1b0d63dc.53368c","type":"debug","z":"6bedd68d.a19a68","name":"","active":false,"console":"false","complete":"payload","x":357.0000915527344,"y":548.7999877929688,"wires":[]},{"id":"a833778d.035c68","type":"modbus-client","z":"","name":"PM800","clienttype":"serial","bufferCommands":false,"stateLogEnabled":false,"tcpHost":"<IP_ADDRESS>","tcpPort":"502","serialPort":"/dev/serial0","serialType":"RTU","serialBaudrate":"38400","serialDatabits":"8","serialStopbits":"1","serialParity":"none","serialConnectionDelay":"200","unit_id":"1","commandDelay":"","clientTimeout":"10000","reconnectTimeout":"2000"}]

please try:
[{"id":"8d9cebcf.0f4f48","type":"modbus-read","z":"c751892a.2202a8","name":"R1","showStatusActivities":false,"unitid":"","dataType":"HoldingRegister","adr":"1699","quantity":"2","rate":"1","rateUnit":"s","server":"5cfbae9.bc3595","x":350,"y":320,"wires":[["4202368e.b6b098"],["771964f5.0ef35c"]]},{"id":"4202368e.b6b098","type":"debug","z":"c751892a.2202a8","name":"","active":false,"console":"false","complete":"payload","x":550,"y":300,"wires":[]},{"id":"771964f5.0ef35c","type":"modbus-response","z":"c751892a.2202a8","name":"","registerShowMax":20,"x":570,"y":340,"wires":[]},{"id":"5cfbae9.bc3595","type":"modbus-client","z":"c751892a.2202a8","name":"PM800","clienttype":"serial","bufferCommands":true,"stateLogEnabled":false,"tcpHost":"<IP_ADDRESS>","tcpPort":"502","serialPort":"/dev/serial0","serialType":"RTU-BUFFERD","serialBaudrate":"38400","serialDatabits":"8","serialStopbits":"1","serialParity":"none","serialConnectionDelay":"200","unit_id":"1","commandDelay":"","clientTimeout":"3000","reconnectTimeout":"5000"}]

please take a look at the Wiki-Pages

@yonas124 try https://www.npmjs.com/package/node-red-contrib-isa - you can map modbus flat data to structured OPC UA - it is not only isa - see examples

Still not connecting, with or without queue

I can connect normally using this one

send your flow from the working package, please

That package is exactly the same but uses alway RTU buffered

This is the flow from the working modbus serial
[{"id":"11914137.f913cf","type":"function","z":"6bedd68d.a19a68","name":"DecodeBuffer1","func":"msg.payload = ((msg.payload.readUInt16BE(6)*1000000000000+msg.payload.readUInt16BE(4)*100000000+msg.payload.readUInt16BE(2)*10000+msg.payload.readUInt16BE(0))/1000);\nreturn msg;","outputs":1,"noerr":0,"x":327,"y":539,"wires":[["5d45c760.41b238"]]},{"id":"d63d1491.f2d0e8","type":"function","z":"6bedd68d.a19a68","name":"DecodeBuffer2","func":"msg.payload = ((msg.payload.readUInt16BE(14)*1000000000000+msg.payload.readUInt16BE(12)*100000000+msg.payload.readUInt16BE(10)*10000+msg.payload.readUInt16BE(8))/1000);\nreturn msg;","outputs":1,"noerr":0,"x":329,"y":592,"wires":[["13b8c221.bc339e"]]},{"id":"8135f409.775f78","type":"modbusSerial in","z":"6bedd68d.a19a68","port":"7d3b04a0.33890c","slaves":"1","start":"1699","count":"8","dtype":"holding","period":"2","topic":"","name":"","x":107,"y":540,"wires":[["11914137.f913cf","d63d1491.f2d0e8"]]},{"id":"7d3b04a0.33890c","type":"modbusSerialConfig","z":"","port":"/dev/serial0","baud":"38400","data":"8","parity":"none","stop":"1","name":"PM800"}]
Also the isa shows warning:
warning : the expected version of node-opcua  is 0.0.56 - actual version is 0.0.60
then it crashed:
2 Feb 09:48:28 - [red] Uncaught Exception:
2 Feb 09:48:28 - Error: Cannot find module 'underscore'
    at Function.Module._resolveFilename (module.js:325:15)
    at Function.Module._load (module.js:276:25)
    at Module.require (module.js:353:17)
    at require (internal/module.js:12:17)
    at Object.<anonymous> (/home/pi/.node-red/node_modules/node-red-contrib-isa/isa                                                                                /helpers/isa95_demo_address_space.js:4:9)
    at Module._compile (module.js:409:26)
    at Object.Module._extensions..js (module.js:416:10)
    at Module.load (module.js:343:32)
    at Function.Module._load (module.js:300:12)
    at Module.require (module.js:353:17)
    at require (internal/module.js:12:17)
    at post_initialize (/home/pi/.node-red/node_modules/node-red-contrib-isa/isa/ISA95-OPCUA-Server.js:158:51)
    at Immediate._onImmediate (/home/pi/.node-red/node_modules/node-red-contrib-isa/node_modules/node-opcua/lib/server/opcua_server.js:442:9)
    at processImmediate [as _immediateCallback] (timers.js:383:17)

ah OK, then I've the underscore package in global install - I'll check that next week, but for now I did not see a difference - very interesting - please try new v1.0.2 with static 4.5.1 of modbus-serial package

Hello biancode,
we have the same problem like yonas124. We use the v1.0.1 and the modbus-serial package v4.5.1. When we readout only 1 input-register with modbus RTU then it works perfectly. But when we change to readout 2 input-registers, then we get the crc check error.
The device we try to readout is DL-100T485-W/DL-100TM485-W from ICPDAS.
Do you have an idea how we can solve this?

please try the v1.0.5 package with modbus-serial 4.5.1 - v4.6.1 may has a problem or I've to change some for that version

@yonas124 how about your tests?

please test with<EMAIL_ADDRESS>

Hello Biancode,
thanks for your answer I will test it and give you feedback.

Now we have v1.0.8 does it work for you?

iam getting crc error. how to resolve this???

@kalyanTDS  normally you have wrong settings in you config node. Try to connect via modbus-serial with a tiny script. Does it work? and what are the parameters in the script to succeed?


Deprecate usage of ! symbol to control path escaping.

Usage of ø.GET.URL becomes difficult to understand when ! symbol is used. It is not obvious when api makes escaping when not. The ø.GET.URL should support following use-case:

Host is know and well defined to the template
Host is not know and it is derived from config
Host is not known and it is derived during I/O
Path segments are defined either from const or dynamically.

fixed by #42


[FEAT] - Add 'DATA_PIPELINE_MACHINE_TYPE' environ example to quick start notebook

We were hitting out of memory errors during the 'Prepare Features' step.  We were able to track down in the code where the machine type was defined, but it was a bit hidden.
I would suggest adding this environ as a commented out example with the rest of the environs that can be set.  Example below:

This is a good suggestion.
Would you create a pull request so I can merge to main?

Fixed.


Update what-is-kubernetes.md

minor changes

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please delete this note before submitting the pull request.
For 1.15 Features: set Milestone to 1.15 and Base Branch to dev-1.15
For Chinese localization, base branch to release-1.14
For Korean Localization: set Base Branch to dev-1.14-ko.<latest team milestone>
Help editing and submitting pull requests:
https://kubernetes.io/docs/contribute/start/#improve-existing-content.
Help choosing which branch to use:
https://kubernetes.io/docs/contribute/start#choose-which-git-branch-to-use.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Hi @pelligrag thanks for your contribution! Before we can have a look, can you sign the CNCF CLA? Instructions on how to do that are described here
After that, @micheleberardi can you have a look at this PR?

Thanks @micheleberardi!
@pelligrag before we are able to merge this, you need to sign the CNCF CLA following the instructions above. After that, I'll /approve this PR :)

@pelligrag were you able to follow the steps for signing the CNCF CLA?

@pelligrag I'm closing this PR because the CLA status check has not been resolved for over a month. Please

feel free to reopen this if you are still working on it, and
try follow instructions here to debug your CLA status if you have already signed it.


Unnecessary API method definitions?

As I started working on a PR for #48, I started thinking about all of these method definitions that seem unnecessary.
For example:
public func af_setImageWithURL(URL: NSURL, placeholderImage: UIImage? = nil)
and
public func af_setImageWithURL(URL: NSURL, placeholderImage: UIImage? = nil, filter: ImageFilter)
Since we have the ability to give parameters default values in Swift, these could be combined into one function:
public func af_setImageWithURL(URL: NSURL, placeholderImage: UIImage? = nil, filter: ImageFilter? = nil)
Is there a reason that I'm not seeing why all of these functions exist? It would make the APIs much cleaner to remove than and just use default parameter values. Keeping all these functions also makes it much harder to add new features.
While trying to add the progress parameter for #48, I'm finding that the number of combinations of the function definitions we need to handle this is growing far too large.

I've created a pull request for this refactor already. #81

This is a great question @AnthonyMDev. The variations were originally created to make it easier to pick the exact API you wanted in auto-complete. That approach continues to lose popularity as more and more people learn how to write Swift. I think it's probably time to squash down those APIs as well. I'm assuming #81 contains two APIs, one that takes a URL, and one that takes a URLRequestConvertible.
I'm going to close this out for now since I'm guessing that answers your question. Happy to re-open if necessary. 🍻


[CF1] Limitations for lb tunnels

Addresses https://jira.cfops.it/browse/PCX-1730

Screenshot of output


Rename crds and create release channels

Make these names consistent with what is uploaded. Any new versions should be named with versions in them like this.

Waiting on https://github.com/aws/eks-distro-build-tooling/pull/107 I guess

@TerryHowe #107 is merged now

/retest


Yard documentation check isn't working on 2.7.0

On Ruby 2.7.0 there is an incompatibility with our version of YARD, this causes:
bin/yard stats --list-undoc
Traceback (most recent call last):
	64: from bin/yard:14:in `<main>'
	63: from bin/yard:14:in `load'
	62: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/bin/yard:13:in `<top (required)>'
	61: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/cli/command_parser.rb:54:in `run'
	60: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/cli/command_parser.rb:72:in `run'
	59: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/cli/command.rb:14:in `run'
	58: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/cli/stats.rb:40:in `run'
	57: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard.rb:20:in `parse'
	56: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/parser/source_parser.rb:113:in `parse'
	55: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/logging.rb:182:in `enter_level'
	54: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/parser/source_parser.rb:114:in `block in parse'
	53: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/parser/source_parser.rb:371:in `parse_in_order'
	52: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/parser/source_parser.rb:45:in `parse'
	51: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/logging.rb:82:in `capture'
	50: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/parser/source_parser.rb:46:in `block in parse'
	49: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/parser/source_parser.rb:443:in `parse'
	48: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/parser/source_parser.rb:497:in `post_process'
	47: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/processor.rb:111:in `process'
	46: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/processor.rb:111:in `each_with_index'
	45: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/processor.rb:111:in `each'
	44: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/processor.rb:112:in `block in process'
	43: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/processor.rb:112:in `each'
	42: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/processor.rb:114:in `block (2 levels) in process'
	41: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/ruby/class_condition_handler.rb:23:in `block in <class:ClassConditionHandler>'
	40: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/ruby/class_condition_handler.rb:84:in `parse_then_block'
	39: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/ruby/base.rb:136:in `parse_block'
	38: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/base.rb:387:in `push_state'
	37: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/ruby/base.rb:138:in `block in parse_block'
	36: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/processor.rb:111:in `process'
	35: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/processor.rb:111:in `each_with_index'
	34: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/processor.rb:111:in `each'
	33: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/processor.rb:112:in `block in process'
	32: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/processor.rb:112:in `each'
	31: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/processor.rb:114:in `block (2 levels) in process'
	30: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/ruby/class_handler.rb:27:in `block in <class:ClassHandler>'
	29: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/ruby/base.rb:136:in `parse_block'
	28: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/base.rb:387:in `push_state'
	27: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/ruby/base.rb:138:in `block in parse_block'
	26: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/processor.rb:111:in `process'
	25: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/processor.rb:111:in `each_with_index'
	24: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/processor.rb:111:in `each'
	23: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/processor.rb:112:in `block in process'
	22: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/processor.rb:112:in `each'
	21: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/processor.rb:114:in `block (2 levels) in process'
	20: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/ruby/comment_handler.rb:8:in `block in <class:CommentHandler>'
	19: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/handlers/base.rb:453:in `register_docstring'
	18: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/docstring_parser.rb:119:in `parse'
	17: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/docstring_parser.rb:320:in `call_directives_after_parse'
	16: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/docstring_parser.rb:320:in `each'
	15: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/tags/directives.rb:369:in `after_parse'
	14: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/tags/directives.rb:417:in `create_object'
	13: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/tags/directives.rb:417:in `new'
	12: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/tags/overload_tag.rb:10:in `initialize'
	11: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/tags/overload_tag.rb:59:in `parse_signature'
	10: from /home/travis/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
	 9: from /home/travis/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
	 8: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/parser/ruby/legacy/token_list.rb:2:in `<top (required)>'
	 7: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/parser/ruby/legacy/token_list.rb:3:in `<module:YARD>'
	 6: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/parser/ruby/legacy/token_list.rb:4:in `<module:Legacy>'
	 5: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/parser/ruby/legacy/token_list.rb:5:in `<class:TokenList>'
	 4: from /home/travis/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
	 3: from /home/travis/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
	 2: from /home/travis/build/rspec/bundle/ruby/2.7.0/gems/yard-0.9.20/lib/yard/parser/ruby/legacy/ruby_lex.rb:1:in `<top (required)>'
	 1: from /home/travis/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
/home/travis/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require': cannot load such file -- e2mmap (LoadError)
Traceback (most recent call last):
	1: from -e:8:in `<main>'
-e:8:in `Float': can't convert nil into Float (TypeError)

It is currently suppressed but should be restored when it can... of interest is:
require': cannot load such file -- e2mmap (LoadError)

Seems to be related to this https://github.com/lsegal/yard/pull/1296

Yard has been released with the patch https://github.com/lsegal/yard/releases/tag/v0.9.21
In rspec-core:
$ bin/yard stats --list-undoc
Files:          33
Modules:        18 (    0 undocumented)
Classes:        36 (    0 undocumented)
Constants:       5 (    0 undocumented)
Attributes:     85 (    0 undocumented)
Methods:       252 (    0 undocumented)
 100.00% documented