Feature Request - add ability to hold of 1.0.0 version bump

Hey friends!
What do you think about adding an option that prevents a major version?
Often when writing a new library you want to wait with the 1.0.0 release
prerelease has it's issues where you need to specify the prerelease prefix to get automatic updates
I thought about creating a PR that adds an option to make Majors into minors
So whenever semver returns "major" as the suggested bump, it'll turn it into minor
WDYT?

Hey @shairez

Often when writing a new library you want to wait with the 1.0.0 release

0.x.y versions are for the initial development stage, where you can introduce breaking changes, but actually, if a breaking change is introduced in the commit history, semver automatically bumps to a major which results in v1.0.0. This is your problem, right?

What do you think about adding an option that prevents a major version?

It feels like a workaround because if we add this new option, ignoring the fact that there are already too many options, you will need to remove it manually when you want to go v1.0.0, which is not ideal for automation, but on other hand, I understand the need, maybe we can find another approach to solve this?

I think the ability to introduce breaking changes without bumping to 1.0.0 is something needed, it could help dealing with the initial development phase. I'm just not enterly sure what's the clean way to do it.
An option could be detecting a special commit body to tells to semver when to bump to the 1.0.0 major version. For example we could detect BREAKING CHANGE 1.0.0: <message> while BREAKING CHANGE: <message> does not bump 0.x.y to 1.0.0. This would allow full automation as well as breaking changes in 0.x.y initial phase. The only thing I don't like is that we need to extend the semver spec. Do people will understand it?
In any case this feature is a breaking change which could only be introduced in the next major, but it's good to think about it now.

I think it's a common enough use case that worth maybe updating the original spec :)
or creating SEMVER for open source 😅
If you take for example - astro, sveltekit, solid start and qwik etc.. they all took their time before announcing 1.0
BTW, a suggestion for a commit message keyword could be: FIRST_MAJOR

Note for #655:
// semver.config.js
if (versionable.name === 'xxx') {
  versionable.prereleasePrefix = 'alpha'
}

Hey brother!!!
It's pretty funny... because guess what? I fell into the same trap today and released @jscutlery/operators 1.0.0 instead of 0.3.0 because it introduced a breaking change. https://github.com/jscutlery/devkit/releases
Well, we just had a discussion about this with @edbzn, and as he mentioned, with @jscutlery/semver 3.0 we are planning to get out of the false-feature rich situations of heaving so many options and solve this with strategies.
👉 TL;DR:  Meanwhile, the workaround that we've been using is that we simply don't use "BREAKING CHANGE" before hitting 1.0.0.
In fact, conceptually, there is no backward-compatibility in zero-version minors. So as anything could break at any moment.
That said, this is the theory and it would be nice to provide a nice changelog informing users of breaking changes in zero minors.
As an additional note to #655, this good be solved using either:

(ideally) bump strategy: bump = zeroBumpStrategy()
(not ideal but let's note this as it might help other situations) changelog strategy that would group changes using conventional commit types + tags! Cf. https://github.com/jscutlery/semver/issues/692

🤗

Thanks @yjaaidi
What I ended up doing is to define few targets, one for minor and one for patch, and we manually decide between them based on the changes we're doing 😅

I think the plugin need to hold any version not just the 1.0.0 version. Imagine if your add support for releasing versions from other branches than the main branch in the future. In this situation I want to hold the same version in that branches for fixes and small features.

Related:
https://github.com/jscutlery/semver/issues/444#issuecomment-1185510458
(set initial version on the package.json)


Utility classes check does not fail the build if Utility classes are in project

In readme we have section describing how we analyzing "Utility classes" and failing the build if code has them.
However this code is still valid with the latest version of oop-cop:
final class HttpHelper {
  static String buildPath(final String name) {
    ...
  }
}

when I invoke the plugin I expect to see something like this:
...
'HttpHelper.java' is a utilityClass, its prohibited, here is why: ... (UtilityClassCheck)
BUILD FAILURE

But for now its all clean.

@h1alexbel thank you for reporting this. I'll assign someone to take care of it soon.


Add download page for 22.02 [skip ci]

Add download page for 22.02
Signed-off-by: Hao Zhu<EMAIL_ADDRESS>

@jlowe we are preparing the doc updates PRs in advance this time as per @sameerz .
Especially for download page PR, it would be nice we can collect the feedback from you and the team to add anything missing.
This initial version is based on PRD doc, so it might not be fully complete.

I'm fine with iterating here on the PR, but I don't think we should merge this until relatively late.  Marking this as a draft until we're ready so it doesn't get accidentally merged early if someone swings by and notices it's approved.

I'm fine with iterating here on the PR, but I don't think we should merge this until relatively late. Marking this as a draft until we're ready so it doesn't get accidentally merged early if someone swings by and notices it's approved.

Cool. I just added a line for mixed join feature:
* Support LeftOuter, RightOuter, FullOuter, Inner, hash semi and anti joins using libcudf's mixed join feature

Feel free to re-word if it is not accurate.

build

build


⚠ ClamAV has degraded performance

In 84d9215, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 187 ms

Resolved: ClamAV performance has improved in 41e7afe.


MSID: 90182 Version: 2 DOI: 10.1101/2<IP_ADDRESS>0760

MSID: 90182
Version: 2
Preprint DOI: https://doi.org/10.1101/2<IP_ADDRESS>0760
Step 1. Awaiting reviews
Editorial to post reviews via hypothesis
Useful links:

DocMap: https://data-hub-api.elifesciences.org/enhanced-preprints/docmaps/v2/by-publisher/elife/get-by-manuscript-id?manuscript_id=90182
New model tracking: https://docs.google.com/spreadsheets/d/1_fHaoOy7hjyocptKtVJRijeNpUY4hBS7Ck_aVmx6ZJk/
Reviews on sciety: https://sciety.org/articles/activity/10.1101/2<IP_ADDRESS>0760

For trouble shooting (e.g. no Docmaps available):

DocMap issue addressing: https://miro.com/app/board/uXjVNCwK6EI=/
Explore DataHub DocMaps API: https://lookerstudio.google.com/reporting/4c2f0368-babb-4beb-b5b3-497e7e7b0f08/page/ejphD
Unmatched submissions and preprints: https://lookerstudio.google.com/u/0/reporting/9f86204f-3bf7-477c-9b18-5c5ef141bf69/page/p_gxi57ha93c
Unmatched manuscripts spreadsheet: https://docs.google.com/spreadsheets/d/15QcK8w-ssB7109RQEDtFpJPZ0J5HTGxoHa_2TtpMBbg/edit#gid=1336081641

Step 2. Preview reviewed preprint
Production QC content ahead of publication
Instructions:

QC preview: https://prod--epp.elifesciences.org/previews/90182v2
Update ticket with any problems (add blocked label)
When QC OK, add QC OK label to ticket and add publication date and time to https://docs.google.com/spreadsheets/d/1amAlKvdLcaDp5W8Z8g77NmkwbMF5n_u89ArSqPMO8jg
Move card to next column
(At end of the day post link in #enhanced-preprint and ask for PDF to be generated)

Useful links:

Preprint DOI :  https://doi.org/10.1101/2<IP_ADDRESS>0760
Confirm reviews returned by EPP: https://prod--epp.elifesciences.org/api/reviewed-preprints/90182/v2/reviews
To update the MECA path in the docmap: https://docs.google.com/spreadsheets/d/1mctCQuNFBjSn97Lihy7_vBO6z7-N-oqyLv4clyi6zHg

Step 3: Awaiting search reindex
This step adds the reviewed preprint to the homepage: https://elifesciences.org
The search reindex is triggered once an hour. We need the reviewed preprint to be indexed as the search application serves the journal homepage.
Useful links:

Jenkins pipeline to reindex search can be triggered sooner or monitored here: https://alfred.elifesciences.org/job/process/job/process-reindex-reviewed-preprints/

Step 4: Published! PDF requested
Waiting for PDF to be generated
Useful links:

PDF tracking: https://docs.google.com/spreadsheets/d/106_XeDjmuBae7gexOTNzg60lapeqjl2aRn9DzupGyS8/

Step 5: Introduce PDF to data folder and git repo
Upload PDF to relevent folder in git repo https://github.com/elifesciences/enhanced-preprints-data/
Step 6: Done!

[ ] Kettle is on!

(Issues with placement of figures/tables in the appendix need resolving)


Fixes to 0.2

better code.

I can't merge, there's a conflict.
What about tests?

Ill merge. Will add tests later

ok. We'll use it once it has tests.


Implement Driver::hardware_address

embassy-net-driver recently changed the Driver trait, removing the ethernet_address fn and adding a hardware_address fn: https://github.com/embassy-rs/embassy/pull/1704
This pull request updates esp-wifi in line with the embassy-net-driver changes.

Thank you for the PR. We probably can convert this to draft until there is a suitable embassy-net-driver release on crates.io


Add missing ErrorMessage size large

After examining components that are used together we identified that ErrorMessage is missing a size Large to pair with other form text options.
This will need to be added to Figma, tokens and React.

Could one of you guys add this soon-ish? @mrosvik @Febakke ErrorMessage currently defaults to medium size when large is selected for the new Textfield.

yup.

https://github.com/digdir/designsystem/pull/839

Is this added in Figma? @mrosvik @Febakke

Seems like it is: https://www.figma.com/file/vpM9dqqQPHqU6ogfKp5tlr/Felles-komponenter?type=design&node-id=20750%3A76353&mode=design&t=t24ZXTXKIWpqxnad-1


add doc to locate a customer instance

Current dependencies on/for this PR:

main

PR #5784   👈

This comment was auto-generated by Graphite.


Fix signIn/signUp typo

I think the example should be for signUp here.

By the way, is there a way to have the function return values documented as well?

Hey @MarcelloTheArcane, we introduced our documentation auto-generator tool recently. Could you please either let our pre-commit hook generate the documentation (by amending your commit) or run pnpm docgen in the project root?

Don't bother, I pushed the autogenerated documentation.
Thanks Marcello!


Add user to tenstorrent-metal team in Cloud website

Problem/Purpose
Following the guide:
https://github.com/tenstorrent-metal/metal-internal-workflows/wiki/Creating-a-virtual-machine-on-the-TT-Cloud#create-a-base-virtual-machine

File an issue on https://github.com/tenstorrent-metal/tt-metal to be added to the "Tenstorrent Metal" team on the Cloud website. You cannot proceed to the next step until someone completes this request for you.

[not urgent]

Closing this, let me know privately if you still see access problems


Ignore unrecognized NDISC options (as per RFC 4861).

https://tools.ietf.org/html/rfc4861#section-4

Future versions of this protocol may define new option types.
Receivers MUST silently ignore any options they do not recognize
and continue processing the message.

@m-labs-homu r+
Thanks!

:pushpin: Commit ecedf6d has been approved by whitequark

:broken_heart: Test failed - status-travis

@m-labs-homu r+

:pushpin: Commit 3398964 has been approved by whitequark

:hourglass: Testing commit 339896446bff3a37737c18a92660fba9f7061ab8 with merge 1bde6e7ec45684019f191714c158e2f496eb49d0...


org.ektorp.UpdateConflictException: document update conflict: id: unknown rev: unknown

Hi,
Is the 'unknown' below a bug or is there a way to return the conflicting id/rev?
org.ektorp.UpdateConflictException: document update conflict: id: unknown rev: unknown
Thanks,
-Nestor

It is set to unknown in situations where it is hard to resolve which revision and id caused the conflict. If you get an conflict during db.update you will have better info in the exception

Hi,
This happens when adding a new document wich ID already exists FYI.
Not sure if it is hard in that case.
I can always catch the exception and provide a better message like
'Document id a123d46..... conflict' one thing known for sure is the id
I believe.
Thanks!
-Nestor
On Oct 13, 2011, at 3:09 AM, Henrik Lundgren
<EMAIL_ADDRESS>wrote:

It is set to unknown in situations where it is hard to resolve which revision and id caused the conflict. If you get an conflict during db.update you will have better info in the exception
--
Reply to this email directly or view it on GitHub:
https://github.com/helun/Ektorp/issues/49#issuecomment-2391888

It might not be hard in that case, you are welcome to contribute a patch that address this

hey helun...
_id:"12314"
_rev id:"1-35637877878899yeye"
userId:"swe123"
age:"45"
this is my fields...but when i try to create a document with attachments...only id ,rev id and attachment id is being created...not other fields of my documents

Could you please provide some steps to replicate this? I can't pinpoint what is causing this issue in our system


Do not overwrite any existing cabal file if it has been modified manually

LGTM.  Thanks a bunch for working on this!

I made some minor modifications in https://github.com/sol/hpack/pull/215/commits/c8a20651d2f40dead37fb3982bcb3e7075029bdd:

Always overwrite when created with hpack < 0.20.0
Never overwrite when hash is missing

In addition I added --force to the hpack executable.  I think this is good to go from hpack's perspective.
@mgsloan @snoyberg I'll cut a release as soon as you guys give me an other thumbs-up and CI passes.

On Hackage as 0.20.0.

Thanks, PR to Stack sent to update. Looks like this'll make it into Stack 1.6.0 🎆


Factory vs EnvironmentObject

Hey @hmlongco, thank you for publishing Factory and all your open source work in the Swift/iOS community!
I'm working on a SwiftUI project for which we've been using singletons for services so far. It's gaining complexity now, so I'm looking at a better option for dependency injection. Reading through the Factory readme, I'm wondering in which cases to go with a Factory-injected service and when to use a vanilla EnvironmentObject. Do you have any pointers that can help with this decision?

Many of my more complex views in SwiftUI look similar to the following pattern...
class SomeViewModel: ObservableObject {
    @Injected(Container. textLoadService) var textLoadService
    @Published var text: String = ""
    @MainActor
    func load(id: String) async {
        text = await textLoadService.load(id: id)
    }
}

struct SomeView: View {
    let id: String
    @StateObject var viewModel = SomeViewModel()
    var body: some View {
        Text(viewModel.text)
            .task {
                await viewModel.load(id: id)
            }
    }
}

I use Factory to provide services to the ViewModel for the views that have them. (Or to provide dependencies for the services, in this case TextLoadService would probably have its own dependency on a network service.)
I have quite a few articles on Medium that pertain to this, check out: https://betterprogramming.pub/swiftui-view-models-are-not-protocols-8c415c0325b1
In fact, you might look at the entire SwiftUI series.
This help?

Awesome, thanks you for sharing the pattern and link to your Medium, @hmlongco!
Reading through the articles, the main problems I see with using an EnvironmentObject (instead of a Factory) as (micro)services are:

EnvironmentObjects are not passed to sheets (new view hierarchies): https://medium.com/swlh/swiftui-and-the-missing-environment-object-1a4bf8913ba7
ObservableObjects can't depend on each other, making it harder to split out services to achieve high cohesion (loose coupling and high cohesion are goals for SoC)


Segfault on pure Lua code

I've got some Lua code run from hslua that gives a SIGSEGV (Address boundary error). Running directly from the Lua interpreter, it works fine (although my code is somewhat dependent on Haskell, so the test ran is smaller than the whole thing).
Gdb's backtrace gives:
#0  0x00007ffff601756c in luaD_poscall () from /usr/local/lib/x86_64-linux-ghc-7.8.4/hslua-0.3.13/libHShslua-0.3.13-ghc7.8.4.so
#1  0x00007ffff6017863 in luaD_precall () from /usr/local/lib/x86_64-linux-ghc-7.8.4/hslua-0.3.13/libHShslua-0.3.13-ghc7.8.4.so
#2  0x00007ffff60278cf in luaV_execute () from /usr/local/lib/x86_64-linux-ghc-7.8.4/hslua-0.3.13/libHShslua-0.3.13-ghc7.8.4.so
#3  0x00007ffff6017c6d in luaD_call () from /usr/local/lib/x86_64-linux-ghc-7.8.4/hslua-0.3.13/libHShslua-0.3.13-ghc7.8.4.so
#4  0x00007ffff6016ffb in luaD_rawrunprotected () from /usr/local/lib/x86_64-linux-ghc-7.8.4/hslua-0.3.13/libHShslua-0.3.13-ghc7.8.4.so
#5  0x00007ffff6017ddb in luaD_pcall () from /usr/local/lib/x86_64-linux-ghc-7.8.4/hslua-0.3.13/libHShslua-0.3.13-ghc7.8.4.so
#6  0x00007ffff60103ac in lua_pcall () from /usr/local/lib/x86_64-linux-ghc-7.8.4/hslua-0.3.13/libHShslua-0.3.13-ghc7.8.4.so
#7  0x00007ffff5ffc277 in hsluazm0zi3zi13_ScriptingziLua_zdwa26_info () from /usr/local/lib/x86_64-linux-ghc-7.8.4/hslua-0.3.13/libHShslua-0.3.13-ghc7.8.4.so
#8  0x0000000000000000 in ?? ()

sooo, any ideas?

It's hard to say something with only this stack trace, can you write a program to reproduce the error?

Oh, my bad, didn't post any code =P
I don't really know how to reproduce the error now, but I can look into it sometime.
The program is in this repo, it's a build system based on Builders, which represents a pattern for building something.
As it is, the error occurs at "build_install.lua:251", in the BI.makeClean function, when calling the program with the clean command
But seems all the Builders created after the ones I had are having this problem...
I'll try to debug some more =]

Hey, any news on this?
I've been trying to debug another segfault error recently and after wasting several hours, I thought, this was working a few months ago, what has changed since then? I realized that the program works fine with GHC 7.8.1 and 7.8.2, but fails with 7.8.3 and 7.8.4.
I'll continue working on this but in the meantime, can you try your program with GHC 7.8.1 and 7.8.2 and tell me if it's working with any of them?

Update: The stacktrace I'm getting is the same as what you're getting, so using older GHC should probably solve this too.

Wow, you're right, it was a wrong argument (string expected, got nil) ¬¬
didn't even need to use older ghc...
I thought it was the Lua code segfaulting, but it called a Haskell function, which gave the problem. I thought haskell functions (not raw functions) had argument type checking...
Thanks, it's working now (and if it happens again, I know where to look at) =]

I thought haskell functions (not raw functions) had argument type checking...

They have, and it's currently broken with GHC 7.8.3 and 7.8.4. I'm trying to fix this in #24.

Ah, ok, I kinda didn't read #24 entirely...


Add lift to FutureEffect

Brilliant, thanks!

Depend on 1.7-20160526110310-1704d42 if you want to use your nice new toy :-).

👍


[BUG] Thermostat cluster is missing a command implementation mechanism

Reproduction steps
The thermostat cluster implementation is missing some code - it is marked ToDo
https://github.com/project-chip/connectedhomeip/blob/master/src/app/clusters/thermostat-server/thermostat-server.cpp#L471
There is no ThermostatDataProvider object defined, and then those commands would be mapped to member functions on the ThermostatDataProvider() object.  From there I can implement the object by overriding the virtual methods.
I am not looking for an implementation of these commands, what I need is the mapping onto a data provider object which I can then override with my implementation.  Either that, or these command functions need to be marked _weak so that I can replace them.
Bug prevalence
always
GitHub hash of the SDK that was being used
5bb5c9e23d532cea40476fc0bd1d3008522792ba
Platform
core
Platform Version(s)
No response
Anything else?
No response

The implementation in app/clusters/thermostat-server does not follow the spec. Many of those stubbed out commands are commands which don't exist, and the SetpointRaiseLower command is missing.


Adding documentation: QuoVadis main concepts.

Thank you, @samuelkgutierrez ! Please feel free to make changes as you see fit.


Phone not jailbreaking

Hey,
My phone isn't jailbreaking on 3.2.0, I have no idea why. It successfully "jailbreaks", but doesn't respring - just goes blank. Left it for 10 mins, still nothing.
iPhone SE - iOS 12.0.1
Works on 3.0.1 and 3.1.3., not 3.2.0(tried with RSD off)
Tweaks kept at a minimum, only snowboard and eclipse 12

I have the same problem. Any ideas?


Add CODEOWNERS

Proposing using a CODEOWNERS file to help with maintaining the project. I will add whatever github handles make sense and alphabetize them before merging.
CODEOWNERS can make it easy to request reviews. We can also require a review from a code owner before merging.
There are likely other ways to accomplish this and I don't care if we choose a different path, but I'd like to add some structure/policy to this repo.
Per conversation in https://github.com/parquet-go/parquet-go/issues/36

Please add myself, @thorfour, @asubiotto and @metalmatze from Polar Signals side.

For transparency, I reached out to @kevinburke when I saw the initial issue since I didn't want the repo to be abandoned. I've been granted maintainer permissions as well, but we only use the library internally and have no commercial offering based on it. I would not mind if my permissions are taken away as long as there are enough people to review PRs against the repository.

From this comment https://github.com/parquet-go/parquet-go/issues/36#issuecomment-1648063492 I don't think its appropriate to include me in this file.

Please add myself as well. I will help to maintain the repo and triage issues.

@gernest
I will gladly remove you from this PR as desired, but please know that your contributions were welcome. If you have interest in the future on collaborating in this repo the door is always open.

@fpetkovski Added your handle. I think at this point anyone willing to put time into the repo should be included. In 6 or 12 months we will likely get a better feel for who really has the time to invest and can whittle things down appropriately.

"Unknown owner on line 1: make sure @metalmatze exists and has write access to the repository"

I went ahead and merged this, it will be helpful to scale the maintenance of this repository, we can figure out the permission issues independently.


API unresponsive for 2 minutes after successfully downloading a few dataframes

See the following example:
"""
API minimum reproducible example
"""
from multiprocessing import Pool
from functools import partial
from retrying import retry
import time

from vds_api_client import VdsApiV2

def wrap_roi_ts(api_vds, start, end, regions, n_proc=4):
    # Define mapping function to parallelize over
    map_func = partial(get_roi_ts, api_vds, start, end)
    # The option exists to loop over the ROIs and not parallelize the operation.
    if n_proc is None or n_proc is False or n_proc == 1:
        for roi in regions:
            map_func(roi)
    else:
        pool = Pool(n_proc)
        pool.map(map_func, regions)

@retry(stop_max_attempt_number=4)
def get_roi_ts(api_vds, start, end, region_id):
    sho = time.perf_counter()
    df = api_vds.get_roi_df(product='SM-SMAP-LN-DESC_V3.1_100::QC', roi=region_id, start_date=start, end_date=end)
    tam = time.perf_counter() - sho
    print('Downloaded data for {}'.format(region_id))
    print('{} [sec]'.format(tam))

vds =<EMAIL_ADDRESS>'******')
rois = vds.rois.filter(name_regex='SwissRe_Ukraine').ids_to_list()[:50]
start_date = '2021-03-02'
end_date = '2021-04-01'
num_proc = 4

sho = time.perf_counter()
wrap_roi_ts(vds, start_date, end_date, rois, n_proc=num_proc)
tam = time.perf_counter() - sho
print('{} [sec]'.format(tam))

The script runs fine until suddenly it outputs this:
/opt/miniconda3/envs/vds_insurance/bin/python /snap/pycharm-community/236/plugins/python-ce/helpers/pydev/pydevd.py --multiproc --qt-support=auto --client <IP_ADDRESS> --port 35241 --file /home/soroshshoaie/PycharmProjects/vds_insurance/personal_scripts/api_mre.py
Connected to pydev debugger (build 211.6693.115)
2021-04-16 12:21:19 - vds_api -     INFO - __init__ @ thr<PHONE_NUMBER>90784  -  ================== VDS_API initialized ==================

Downloaded data for 9283
0.13546402399879298 [sec]
Downloaded data for 9287
0.13837170300030266 [sec]
Downloaded data for 9284
0.10519520800153259 [sec]
Downloaded data for 9294
0.16638344600141863 [sec]
Downloaded data for 9289
0.12269279300016933 [sec]
Downloaded data for 9298
0.16886850999799208 [sec]
Downloaded data for 9295
0.12342391399943153 [sec]
Downloaded data for 9285
0.1654799149982864 [sec]
Downloaded data for 9290
0.11234415199942305 [sec]
Downloaded data for 9300
0.14624839299722225 [sec]
Downloaded data for 9296
0.1229136389993073 [sec]
Downloaded data for 9286
0.12153443799979868 [sec]
Downloaded data for 9291
0.14344482799788238 [sec]
Downloaded data for 9301
0.10994474700055434 [sec]
Downloaded data for 9297
0.10774154899991117 [sec]
Downloaded data for 9303
0.11728104299982078 [sec]
Downloaded data for 9302
0.1201207180019992 [sec]
Downloaded data for 10273
0.12486524800260668 [sec]
Downloaded data for 10277
0.15764935799961677 [sec]
Downloaded data for 9304
0.15550299900132813 [sec]
Downloaded data for 10281
0.12943152799925883 [sec]
Downloaded data for 10274
0.16696305599907646 [sec]
Downloaded data for 9305
0.10445150399755221 [sec]
Downloaded data for 10278
0.12531482300255448 [sec]
Downloaded data for 10282
0.13500779699825216 [sec]
Downloaded data for 10275
0.12725345699800528 [sec]
Downloaded data for 10271
0.13242110400096863 [sec]
Downloaded data for 10279
0.11631401900012861 [sec]
Downloaded data for 10276
0.12106328699883306 [sec]
Downloaded data for 10283
1.1600049810003839 [sec]
Downloaded data for 10289
0.1533900590002304 [sec]
Downloaded data for 10280
0.2779080200016324 [sec]
Downloaded data for 10290
0.1526672090003558 [sec]
Downloaded data for 10285
0.4117055070018978 [sec]
Downloaded data for 10291
0.11837836099948618 [sec]
Downloaded data for 10293
0.17365858899938758 [sec]
Downloaded data for 10286
0.12394987099833088 [sec]
Downloaded data for 10292
0.1267180799986818 [sec]
Downloaded data for 10294
0.14531772999907844 [sec]
Downloaded data for 10287
0.1547291110000515 [sec]
Downloaded data for 10297
0.12478498199925525 [sec]
Downloaded data for 10295
0.1627491569997801 [sec]
Downloaded data for 10288
0.1419056320009986 [sec]
Downloaded data for 10298
0.13286103900099988 [sec]
Downloaded data for 10301
0.1275661610015959 [sec]
Downloaded data for 10296
0.1686530209990451 [sec]
Downloaded data for 10299
0.19449762699878193 [sec]
Downloaded data for 10302
0.10437436000211164 [sec]
Downloaded data for 10300
0.2356658429998788 [sec]
Downloaded data for 10284
0.13083889400149928 [sec]
135.8246309189999 [sec]

Process finished with exit code 0

So it is clear that for these 50 regions that all last less than 0.3 seconds with 1 exception; the whole procedure should last less 15 seconds, but it takes 135 seconds. How should we investigate where the issue lies?

When I execute this piece of code, I get the following results for the last -overall- performance benchmark:

3.486206902001868 [sec]
2.982811229005165 [sec]
3.327922716998728 [sec]
3.3890383929974632 [sec]
2.926594144999399 [sec]

I notice you run it with the debugger enabled. You don't have any breakpoints enabled? I cannot reproduce this behavior

So if you run through all 50 rois, they all take about the same duration to
download?
For me about half way suddenly one takes much longer.
On Fri, Apr 16, 2021, 3:35 PM teije01 @.***> wrote:

When I execute this piece of code, I get the following results for the
last -overall- performance benchmark:

3.486206902001868 [sec]
2.982811229005165 [sec]
3.327922716998728 [sec]
3.3890383929974632 [sec]
2.926594144999399 [sec]

I notice you run it with the debugger enabled. You don't have any
breakpoints enabled? I cannot reproduce this behavior
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/vandersat/vds-api-client/issues/39#issuecomment-821182409,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AP7GOELCGJYMQJQSCJXO6MDTJA4J7ANCNFSM43BPQWQA
.

Indeed they do, but from your logging I cannot see that any of the rois take much longer than they should. Only the last number, the total time for all of them combined, is much longer but I don't see anything holding you back.
Let me ask you this: which of the rois takes longer than it should in the logging you posted previously? The longest I could identify was 1.1600049810003839 sec for roi_id = 10283

See the following results.  Just to be sure that is not causing the issue

I extended this test to 649 rois, and all I get is around 29-30 seconds consistently. Please try this outside pycharm to rule that out. one thing you can try is adding a timeout to the request:
Add
timeout=10
to this line
r = self.get(uri, timeout=10)

that should speed things up a bit. If it works for you I'm open to a pull request for this.
Note that the timeout should be set to a value that is reasonable enough for delivering a very long request as well, otherwise this will bite you back later. I estimate that 10 seconds should be more than enough for that.
Furthermore, you can add an exponential backoff delay to your retry to prevent the requests to pile up upon failures. See https://github.com/rholder/retrying#examples for examples

using the issue

I tried it outside of Pycharm and the result is the same.
soroshshoaie@sorosh:~/PycharmProjects/vds_insurance/personal_scripts$ conda activate vds_insurance
(vds_insurance) soroshshoaie@sorosh:~/PycharmProjects/vds_insurance/personal_scripts$ python api_mre.py
Traceback (most recent call last):
(vds_insurance) soroshshoaie@sorosh:~/PycharmProjects/vds_insurance/personal_scripts$ python api_mre.py
2021-04-21 04:40:01 - vds_api -     INFO - __init__ @ thr<PHONE_NUMBER>59712  -  ================== VDS_API initialized ==================

Downloaded data for 61958
Elapsed time: 0.3878 seconds
Downloaded data for 61962
Elapsed time: 0.3120 seconds
Downloaded data for 61966
Elapsed time: 0.2390 seconds
Downloaded data for 61959
Elapsed time: 0.2895 seconds
Downloaded data for 61970
Elapsed time: 0.3517 seconds
Downloaded data for 61963
Elapsed time: 0.3866 seconds
Downloaded data for 61971
Elapsed time: 0.2786 seconds
Downloaded data for 61967
Elapsed time: 0.5187 seconds
Downloaded data for 61960
Elapsed time: 0.4897 seconds
Downloaded data for 61964
Elapsed time: 0.3021 seconds
Downloaded data for 61965
Elapsed time: 0.2272 seconds
Downloaded data for 61972
Elapsed time: 0.3871 seconds
Downloaded data for 61968
Elapsed time: 0.3815 seconds
Downloaded data for 61961
Elapsed time: 0.5523 seconds
Downloaded data for 61973
Elapsed time: 0.4201 seconds
Downloaded data for 61969
Elapsed time: 0.4217 seconds
Downloaded data for 61975
Elapsed time: 0.2265 seconds
Downloaded data for 61976
Elapsed time: 0.3030 seconds
Downloaded data for 61979
Elapsed time: 0.2785 seconds
Downloaded data for 61983
Elapsed time: 0.2552 seconds
Downloaded data for 61980
Elapsed time: 0.2444 seconds
Downloaded data for 61977
Elapsed time: 0.3323 seconds
Downloaded data for 61987
Elapsed time: 0.2503 seconds
Downloaded data for 61984
Elapsed time: 0.2480 seconds
Downloaded data for 61978
Elapsed time: 0.2993 seconds
Downloaded data for 61981
Elapsed time: 0.3930 seconds
Downloaded data for 61982
Elapsed time: 129.7787 seconds
Downloaded data for 61991
Elapsed time: 129.6079 seconds
Downloaded data for 61985
Elapsed time: 130.0478 seconds
Downloaded data for 61988
Elapsed time: 130.1042 seconds
Downloaded data for 61989
Elapsed time: 0.2173 seconds
Downloaded data for 61992
Elapsed time: 0.2756 seconds
Downloaded data for 61986
Elapsed time: 0.4325 seconds
Downloaded data for 61995
Elapsed time: 0.2419 seconds
Downloaded data for 61990
Elapsed time: 0.2851 seconds
Downloaded data for 61993
Elapsed time: 0.2661 seconds
Downloaded data for 61996
Elapsed time: 0.3394 seconds
Downloaded data for 61994
Elapsed time: 0.3129 seconds
Downloaded data for 61997
Elapsed time: 0.2711 seconds
Downloaded data for 61998
Elapsed time: 0.2167 seconds
Downloaded data for 61999
Elapsed time: 0.6336 seconds
Downloaded data for 62008
Elapsed time: 0.2097 seconds
Downloaded data for 62003
Elapsed time: 0.5194 seconds
Downloaded data for 62000
Elapsed time: 0.3309 seconds
Downloaded data for 62009
Elapsed time: 0.3352 seconds
Downloaded data for 62004
Elapsed time: 0.4191 seconds
Downloaded data for 62001
Elapsed time: 0.3116 seconds
Downloaded data for 62005
Elapsed time: 0.3069 seconds
Downloaded data for 62002
Elapsed time: 0.2062 seconds
Downloaded data for 62007
Elapsed time: 0.3909 seconds
Elapsed time: 135.9136 seconds

With the suggestion of r = self.get(uri, timeout=10) the code resulted in a requests.exceptions.ConnectTimeout
(vds_insurance) soroshshoaie@sorosh:~/PycharmProjects/vds_insurance/personal_scripts$ python api_mre.py
2021-04-21 04:47:26 - vds_api -     INFO - __init__ @ thr<PHONE_NUMBER>05152  -  ================== VDS_API initialized ==================

Downloaded data for 61958
Elapsed time: 0.2569 seconds
Downloaded data for 61962
Elapsed time: 0.3156 seconds
Downloaded data for 61970
Elapsed time: 0.2489 seconds
Downloaded data for 61959
Elapsed time: 0.4947 seconds
Downloaded data for 61963
Elapsed time: 0.2750 seconds
Downloaded data for 61966
Elapsed time: 0.5994 seconds
Downloaded data for 61971
Elapsed time: 0.2500 seconds
Downloaded data for 61960
Elapsed time: 0.2254 seconds
Downloaded data for 61964
Elapsed time: 0.2523 seconds
Downloaded data for 61972
Elapsed time: 0.2579 seconds
Downloaded data for 61961
Elapsed time: 0.2551 seconds
Downloaded data for 61965
Elapsed time: 0.2815 seconds
Downloaded data for 61973
Elapsed time: 0.2303 seconds
Downloaded data for 61967
Elapsed time: 0.5412 seconds
Downloaded data for 61968
Elapsed time: 0.3183 seconds
Downloaded data for 61975
Elapsed time: 0.2547 seconds
Downloaded data for 61979
Elapsed time: 0.2315 seconds
Downloaded data for 61969
Elapsed time: 0.2258 seconds
Downloaded data for 61976
Elapsed time: 0.2702 seconds
Downloaded data for 61983
Elapsed time: 0.2678 seconds
Downloaded data for 61980
Elapsed time: 0.2601 seconds
Downloaded data for 61984
Elapsed time: 0.2419 seconds
Downloaded data for 61987
Elapsed time: 0.2056 seconds
Downloaded data for 61981
Elapsed time: 0.2755 seconds
Downloaded data for 61977
Elapsed time: 0.6536 seconds
Downloaded data for 61985
Elapsed time: 0.3794 seconds
Elapsed time: 40.0612 seconds
Elapsed time: 40.0750 seconds
Elapsed time: 40.0360 seconds
Elapsed time: 40.0636 seconds
Downloaded data for 61991
Elapsed time: 37.4415 seconds
Elapsed time: 40.0479 seconds
Elapsed time: 40.0641 seconds
Elapsed time: 40.0532 seconds
Downloaded data for 61992
Elapsed time: 27.3574 seconds
Downloaded data for 61993
Elapsed time: 0.2598 seconds
Downloaded data for 61994
Elapsed time: 0.2744 seconds
Downloaded data for 62008
Elapsed time: 37.5202 seconds
Downloaded data for 62009
Elapsed time: 0.3037 seconds
Elapsed time: 122.0494 seconds
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/urllib3/connection.py", line 159, in _new_conn
    conn = connection.create_connection(
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/urllib3/connection.py", line 309, in connect
    conn = self._new_conn()
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/urllib3/connection.py", line 164, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x7f463419ad90>, 'Connection to maps.vandersat.com timed out. (connect timeout=10)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='maps.vandersat.com', port=443): Max retries exceeded with url: /api/v2/products/SM-SMAP-LN-DESC_V3.1_100::QC/roi-time-series-sync?roi_id=61988&start_time=2021-03-02&end_time=2021-04-01&climatology=true&avg_window_direction=backward&avg_window_days=20&format=csv (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f463419ad90>, 'Connection to maps.vandersat.com timed out. (connect timeout=10)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/retrying.py", line 212, in call
    raise attempt.get()
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/six.py", line 703, in reraise
    raise value
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "api_mre.py", line 100, in get_roi_ts
    df = api_vds.get_roi_df(product='SM-SMAP-LN-DESC_V3.1_100::QC', roi=region_id, start_date=start, end_date=end)
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/vds_api_client/api_v2.py", line 478, in get_roi_df
    r = self.get(uri, timeout=10)
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/vds_api_client/requester.py", line 119, in get
    r = requests.get(uri, verify=True, stream=True,
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/site-packages/requests/adapters.py", line 504, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='maps.vandersat.com', port=443): Max retries exceeded with url: /api/v2/products/SM-SMAP-LN-DESC_V3.1_100::QC/roi-time-series-sync?roi_id=61988&start_time=2021-03-02&end_time=2021-04-01&climatology=true&avg_window_direction=backward&avg_window_days=20&format=csv (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f463419ad90>, 'Connection to maps.vandersat.com timed out. (connect timeout=10)'))
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "api_mre.py", line 111, in <module>
    wrap_roi_ts(vds, start_date, end_date, rois, n_proc=num_proc)
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "api_mre.py", line 93, in wrap_roi_ts
    pool.map(map_func, regions)
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/miniconda3/envs/vds_insurance/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
requests.exceptions.ConnectTimeout: None: Max retries exceeded with url: /api/v2/products/SM-SMAP-LN-DESC_V3.1_100::QC/roi-time-series-sync?roi_id=61988&start_time=2021-03-02&end_time=2021-04-01&climatology=true&avg_window_direction=backward&avg_window_days=20&format=csv (Caused by None)

Then as a last option, I followed the examples of adding an exponential backoff delay like this:
@retry(wait_exponential_multiplier=5000, wait_exponential_max=15000,
           stop_max_attempt_number=7)
    def get_roi_df(self, product, roi, start_date, end_date):
        """
        Method to querry streamed json output and transform this into
        a pandas DataFrame

        Parameters
        ----------
        product: str
            Product to extract dataframe from
        roi: int or str
            Region of interest to retrieve DataFrame from
        start_date: str
            start date to retrieve data yyyy-mm-dd
        end_date: str
            end date to retrieve data yyyy-mm-dd (inclusive)

        Returns
        -------
        dfs_list: list of pd.DataFrame

        """
        roi_id = self.rois[roi].id
        uri = (f'https://{self.host}/api/v2/products/{product}/roi-time-series-sync?'
               f'roi_id={roi_id}&start_time={start_date}&end_time={end_date}&climatology=true&'
               f'avg_window_direction=backward&avg_window_days=20&format=csv')
        r = self.get(uri, timeout=10)
        csv = BytesIO()
        for chunk in r.iter_content(2048):
            csv.write(chunk)
        csv.seek(0)
        df = pd.read_csv(csv, index_col=0, parse_dates=True, dayfirst=True, comment='#')
        csv.close()
        return df

Where I wait up to 10 seconds and stop trying after 7 times. Note that the r = self.get(uri, timeout=10) is still set.  There is a off chance that something is weird because you are running inside our google network.

I see that. Unfortunately this cannot be reproduced easily. I'm requesting the exact same products and rois with the same code as you do. Not once did that take longer than it should have. Also the retries do not seem to have the desired effect. If you look at your trace, you'll find that 4 retries were tried for the same roi (61988) and none of them succeeded. The request is valid though, and when I make that same request it works.
Have you tried this locally, or only from the google network?

Have you tried this locally, or only from the google network?

I have only tried this on the VM (did not think to do it locally since the functionality needed direct access to the database). However, it is an interesting point. I will perform the check and see if any difference occurs.
@teije01 did you ran your requests only locally?

Yes, I only requested from my home and the office network. Both fine

I ran the minimal example locally and it ran in 7 seconds! There is a difference from running this on a VM.  It is a networking issue on google cloud, so we cannot solve that in the api client. As long as the problem is only when using VMs on our internal network, we needn't worry.


Missing some custom events to hook in

The lib is nice but it should takes some callbacks as option to allow custom js hookin or trigger some custom events to hook in
cheers

Hi,
I've merged your PR. Thanks a lot ;)


add file transfer download support for remote payload

@sagarjain13  FYI I updated your branch to install the azure FS lib as a python package dep rather than via Dockerfile. Tests are now passing. You will need to rebase this onto master as there are some merge conflicts, but it should be straightforward.

@bcwaldon thanks, I have rebased and resolved all comments, if you don't have any other suggestions can you approve the PR so we can go ahead with the merge


app_update: fix incorrect first byte from esp_ota_get_app_elf_sha256

At -O2 optimization level, GCC seems to optimize out the copying of the
first byte of the checksum, assuming it is zero. This "miscompilation"
happens because the esp_app_desc struct is declared const, but then modified
post-compilation. Casting to volatile disables the optimization.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for your contribution.

Thanks for your contribution.

Hi @itayperl.
Thank you for the fix. Really the first byte is 0.
I (175) cpu_start: ELF file SHA256:  005a3a4c72ec7bd0...
...
ELF file SHA256: 005a3a4c72ec7bd0

> sha256sum  build/unit-test-app.elf
e25a3a4c72ec7bd0ff4afeaa832b449e74ec699b256c32634360bfad95c86c1a  build/unit-test-app.elf

We will merge this fix into master and create backports.

Hi @itayperl.
Thank you for the fix. Really the first byte is 0.
I (175) cpu_start: ELF file SHA256:  005a3a4c72ec7bd0...
...
ELF file SHA256: 005a3a4c72ec7bd0

> sha256sum  build/unit-test-app.elf
e25a3a4c72ec7bd0ff4afeaa832b449e74ec699b256c32634360bfad95c86c1a  build/unit-test-app.elf

We will merge this fix into master and create backports.


Wrong zIndex container

The z-index is wrongfully inserted on the container with the class iziToast it should be inserted on the div containing the class   iziToast-wrapper

Yes, it has already been fixed. Wait for the next update. Thanks


⚠ Science Web Services - QState has degraded performance

In 4c426ca, Science Web Services - QState (https://sws.labbricks.com/qstate/) experienced degraded performance:

HTTP code: 523
Response time: 1096 ms

Resolved: Science Web Services - QState performance has improved in 69fff8d after 12 minutes.


Add formatter and some other stuff

This closes #2 , #3 , #4

Cool! I will review this today.

:+1:

Thanks for this PR, I think its really close!
As noted in the comments, I'd like to avoid falling back to standard-format as a formatter. I don't quite think it makes sense to do so as it most folks using this package would be deviating from standard :)
Would you be willing to make these changes?

Sure no problem :smile:

Updated with the changes you suggested, please review :+1:

Thanks! this was manually merged and released as 1.5.0, I'm  attempting to update semistandard now :)


Full list of implemented and possible operators

List of operators
It seemed as though the list was going quite quickly, so I've compiled them here.

✔ Many Term+
Matches 1 or more occurences of a term

✔ Any Term* { Terms }
Matches 0 or more occurences of a term ( perhaps this could just be replaced with Term+? or Term?+ ? )

✔ Maybe Term? [ Terms ]
Matches 0 or 1 occurences of a term

✔ Or Term1 | Term2 < Terms >
Matches any of the terms in the 'list' of options

✔ Except Term1~Term2
Matches a term, but only if its not a part of another term

❌ Select @Term
Labels a term for later use, serves no functional purpose

❌ Then Term THEN Term Term, Term
Matches multiple terms in sequence

❌ Count N Term ( perhaps can be simplified to NTerm, if you restrict terms to only be defined with alphabetic characters)
Matches a specific number (N) of terms in sequence

The following are suggestions for possible operators

❔ But / Not #Term NOT Term ( { Terms } if you end up removing the Any term, as mentioned above )
Matches anything that is not a Term

❔ Anything ... ANY
Any term. After thinking about this, it'd probably be a bad idea to use this in conjunction with Many or Any, unless it becomes lazy like in regex

❔ Built-ins :name: ( inspired by Character Classes in the regex POSIX standard Source )
This is more like a group of built in terms, but I thought I could put it in here anyway
Types of characters which turn up it lots of places, so its helpful to have a group that can be used from the beginning, including:

:alnum:
Alphanumeric characters
:punct:
Punctuation characters
:xdigit:
Characters that are hexadecimal digits

I'd be happy to tick off the operators when they are completed by updating my comment

It's helpful to see it all laid out like this!
I have the 'any' operator as its own thing because it's from 'EBNF'. This makes me realise that MotherTode's 'or' operator works differently to EBNF. Maybe it would make more sense if it worked the same way...
And maybe built-ins! That would be a nice way of differentiating then. I wonder what built-ins would be helpful. I guess EOF could be a built-in too.

Yeah! The EOF operator would make sense as an in-built term, and if your going with that, you could also implement a SOF (Start Of File) term too.
EBNF separates it's terms with commas, and you've decided against that, so in a similar sense you could leave the ORperator (bah dum tss) as it is. But it is your language, so you can do what you you think would work best.

That's true! I wonder if commas could be optional actually! Start of file makes sense too. I think close to EBNF would be good, with a few liberties. Arbitrary things like the or operator, I see no reason not to be the same as EBNF

Hmm, yeah I can undertand you reasoning. Go for it! After all, the more similarities it has to other languages, the easier it'll be for people to learn!

Closing because of upcoming rewrite


How did you sniff the Notion API ?

How did you sniff the Notion API ?
Do they have a REST API ?
Or do they have a very secured API they gave you the keys for ?

You can use the Notion.so web view and open the network tab in any web developer toolset. Here's an example where I'm looking for the details of the file upload API.

You can see a ton of different endpoints by refreshing the page, filtering only to XHR requests, performing the action you are looking to check, and then noting where they hit and what data they send in the network tab.

Oh yeah, I already tried that, but I wasn't seeing the column you mention.
Thank you for the help, I hope I'll find out the methods I'm interested in thanks to this method!
I was wondering, is there any app that can hide these method calls in any way ? So that guys like us can't reverse-engineer their API ?

No problem
No, there's no way to hide this sort of thing. You can obfuscate it and
structure your API to make it harder to edit things, but at the end of the
day anything you put on the client's computer can be reverse
engineered/taken apart and simulated back to your API.
The complexity involved in trying to build such an obfuscated API also
usually involved a lot more confusion and communication on the business
side so no one really does it.
On Mon, Dec 2, 2019, 6:08 PM Callixte Girard<EMAIL_ADDRESS>wrote:

Oh yeah, I already tried that, but I wasn't seeing the column you mention.
Thank you for the help, I hope I'll find out the methods I'm interested in
thanks to this method!
I was wondering, is there any app that can hide these method calls in any
way ? So that guys like us can't reverse-engineer their API ?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/jamalex/notion-py/issues/72?email_source=notifications&email_token=ABTSGCDSFXPDFNP6XPDPJSTQWWIO3A5CNFSM4JP3MTY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFW6AWQ#issuecomment-560848986,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABTSGCFO3YRK7JZIPXAB7KTQWWIO3ANCNFSM4JP3MTYQ
.

Thank you for your answers. I think they perfectly make sense. Now I know how to display the different requests used by the API.
After filtering on XHR and locating only the exact request I need with the time graph, and I found the request made to add a block : submitTransaction.
But the content of the request, which is in cookie if I understood well, seems not significant at all to me ; how did you decrypt it ?

Let me clarify.
If you look at my screenshot, these are attributes in the headers section of the submitTransaction request :

_cfduid
logglytrackingsession
intercom-session-gpfdrxfd
amplitude_id_af
But I don't understand :
what purpose do they serve ?
where is the part that says "insert a new empty block at this place"

Hope I'm clear enough, and thank you in advance for your patience !

Cookie is the browser cookie for the domain you're requesting to. It's sent
to the server for specific requests as an HTTP Header. It's the same as the
data in document.cookie. But this isn't the data that's sent in the
request.
For POST, PUT, and other requests, the data is sent in the body of the
message. That's usually at the very bottom of the Headers panel. It maybe
be behind a little '>' accordion menu arrow, as Chrome will automatically
parse JSON sent to the server and display it as a nice object for you to
browse. The data you're looking for should not be in the Headers section.
On Sat, Dec 7, 2019, 5:19 PM Callixte Girard<EMAIL_ADDRESS>wrote:

Thank you for your answers. I think they perfectly make sense. Now I know
how to display the different requests used by the API.
After filtering on XHR and locating only the exact request I need with the
time graph, and I found the request made to add a block :
submitTransaction.
But the content of the request, which is in cookie if I understood well,
seems not significant at all to me ; how did you decrypt it ?
[image: Screenshot 2019-12-07 at 23 09 54]
https://user-images.githubusercontent.com/45435811/70381242-3de4d200-1947-11ea-8fb5-fa683e5e2315.png
Let me clarify.
If you look at my screenshot, these are attributes in the headers section
of the submitTransaction request :

_cfduid
logglytrackingsession
intercom-session-gpfdrxfd
amplitude_id_af
But I don't understand :
what purpose do they serve ?
where is the part that says "insert a new empty block at this place"

Hope I'm clear enough, and thank you in advance for your patience !
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/jamalex/notion-py/issues/72?email_source=notifications&email_token=ABTSGCHRJRBOT3SLCDSBAHTQXQOOXA5CNFSM4JP3MTY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGGQWYA#issuecomment-562891616,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABTSGCG32NJTWOH7O2UXEKLQXQOOXANCNFSM4JP3MTYQ
.

Okay, so I supposed this is what you're talking about (Request Payload), indeed at the very bottom of the Headers section.

So I just have to generate a random requestId and transactionId, then generate an id for the new block and set its attributes like in the screenshot, alright ?
How should I turn this JSON into a URL-based HTTP request ? I know how to parse an HTTP response in the form of a JSON but not the opposite.
When you search something on a website, it does something like : https://www.thewebsite.com/?search=your-search
Is it something similar I'm supposed to do ? How can I fit such a large request in just a single URL ?

GET requests normally pass their parameters as "query string" parameters,
the ones after the ? delimited by & and using = for key value relationship.
To pass the JSON object like this you cant use a URL, you have to use a
request with a body (like POST or PUT) though the server might be expecting
a specific kind.
Also, requestID and transaction ID are not random but are guids. They also
might not be random guids but they might come from some previously called
endpoint to say open a transaction. Notion-py does do all this sort of
Notion handshaking behind the scenes so if you study the code you should
figure out how to do it (I haven't looked deep enough in that part of the
library to know the answer though).
On Sat, Dec 7, 2019, 6:32 PM Callixte Girard<EMAIL_ADDRESS>wrote:

Okay, so I supposed this is what you're talking about (Request Payload),
indeed at the very bottom of the Headers section.
[image: Screenshot 2019-12-08 at 00 23 58]
https://user-images.githubusercontent.com/45435811/70381812-6671c980-1951-11ea-9466-baf72d0ba7b7.png
So I just have to generate a random requestId and transactionId, then
generate an id for the new block and set its attributes like in the
screenshot, alright ?
How should I turn this JSON into a URL-based HTTP request ? I know how to
parse an HTTP response in the form of a JSON but not the opposite.
When you search something on a website, it does something like :
https://www.thewebsite.com/?search=your-search
Is it something similar I'm supposed to do ? How can I fit such a large
request in just a single URL ?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/jamalex/notion-py/issues/72?email_source=notifications&email_token=ABTSGCETB7Y2PQFLOQILC7TQXQXBFA5CNFSM4JP3MTY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGGRZWI#issuecomment-562896089,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABTSGCBMDL7M2IBHDPPVGY3QXQXBFANCNFSM4JP3MTYQ
.

Okay I think I understand. Thank you again for your time !
I just have to dig into all of this now... ;-)


bug: VCF5.0- Isolated WLD- Unable to find (svc-vra-vsphere) in vCenter Server (sfo-w01-vc01): PRE_VAILIDATION_FAILED

Code of Conduct

[X] I have read and agree to the Code of Conduct.
[ ] Vote on this issue by adding a 👍 reaction to the original issue initial description to help the maintainers prioritize.
[ ] Do not leave "+1" or other comments that do not add relevant information or questions.
[ ] If you are interested in working on this issue or have submitted a pull request, please leave a comment.

VMware Cloud Foundation
5.0
Module Version
<IP_ADDRESS>1
PowerShell Version
7.3.6
PowerCLI Version
13.1
PowerVCF Version
<IP_ADDRESS>4
Guest Operating System
windows server 2019
Environment Details
VCF 5.0 with isolated WLD
Description
https://docs.vmware.com/en/VMware-Cloud-Foundation/services/vcf-private-cloud-automation-v1/GUID-47B20817-952F-4820-BDA6-F2CC4B74E66A.html
Set-vCenterPermission fails with below error message.

PS F:\platformtools> Set-vCenterPermission -server $sddcManagerFqdn -user $sddcManagerUser -pass $sddcManagerPass -domain $domainAlias -workloadDomain $sddcDomainName -principal $vraServiceAccount -role "NoAccess" -folderName $nsxEdgeVMFolder -folderType "VM"
Set-vCenterPermission: Unable to find (svc-vra-vsphere) in vCenter Server (sfo-w01-vc01): PRE_VAILIDATION_FAILED
PS F:\platformtools>

Error or Debug Output

PS F:\platformtools> Set-vCenterPermission -server $sddcManagerFqdn -user $sddcManagerUser -pass $sddcManagerPass -domain $domainAlias -workloadDomain $sddcDomainName -principal $vraServiceAccount -role "NoAccess" -folderName $nsxEdgeVMFolder -folderType "VM"
Set-vCenterPermission: Unable to find (svc-vra-vsphere) in vCenter Server (sfo-w01-vc01): PRE_VAILIDATION_FAILED
PS F:\platformtools>

Expected Behavior
should be successful.
Actual Behavior

PS F:\platformtools> Set-vCenterPermission -server $sddcManagerFqdn -user $sddcManagerUser -pass $sddcManagerPass -domain $domainAlias -workloadDomain $sddcDomainName -principal $vraServiceAccount -role "NoAccess" -folderName $nsxEdgeVMFolder -folderType "VM"
Set-vCenterPermission: Unable to find (svc-vra-vsphere) in vCenter Server (sfo-w01-vc01): PRE_VAILIDATION_FAILED
PS F:\platformtools>

Steps to Reproduce
Deploy VCF 5.0 with WLD in isolated mode.
Deploy vRSLCM and VRA
Perform the steps https://docs.vmware.com/en/VMware-Cloud-Foundation/services/vcf-private-cloud-automation-v1/GUID-47B20817-952F-4820-BDA6-F2CC4B74E66A.html
Log Fragments and Files
No response
Screenshots
No response
References
No response

Reason: At the current time no VMware Validated Solutions have been fully validated with an Isolated Workload Domain, automation failures are to be expected but before we can address any automation issues there is significant work required to review and update the manual procedures in the context of an Isolated Workload Domain.
Marking this issues as Declined/Closed but tracking issue internally pending further analysis / review of the Isolated Workload Domain use case.


How to omit / bypass shift parameters in the [ Flux Sampler Parameters ] node ?

In 🔧 Flux Sampler Parameters, leaving the shift parameters unfilled, has the same effect as using default values ( 1.15 / 0.5 ).
This also gives the same effect as using the KSampler + ModelSamplingFlux, with default shift values ( 1.15 / 0.5 ).
But, bypassing ModelSamplingFlux, gives a different image.
So, how can we simulate bypassing ModelSamplingFlux, with a 🔧 Flux Sampler Parameters workflow ?

A way to do this could be pre-filling 🔧 Flux Sampler Parameters, with the default shift values.
And if we delete these values, we have the same effect as bypassing ModelSamplingFlux.

I'll look into it, I thought 1.15 was the default anyway. I'll double check


Deprecation warnings under JRuby 9.2.x

For a Rails app running under Tomcat via Warbler, upgrading from JRuby <IP_ADDRESS> (Ruby 2.3.3 compatibility) to JRuby <IP_ADDRESS> (Ruby 2.5.7 compatibility) with jruby-rack 1.1.21 gives me the following new deprecation warnings on application startup:
uri:classloader:/jruby/rack/response.rb:294: warning: constant ::Fixnum is deprecated
uri:classloader:/jruby/rack/core_ext.rb:26: warning: constant ::NativeException is deprecated

These correspond to:

https://github.com/jruby/jruby-rack/blob/1.1.21/src/main/ruby/jruby/rack/response.rb#L294
https://github.com/jruby/jruby-rack/blob/1.1.21/src/main/ruby/jruby/rack/core_ext.rb#L26

Apparently, Fixnum has been replaced by Integer, and seems to already be fixed on master (via removal), just not in any jruby-rack release. I'm not sure on the best path forward for NativeException, or the best way to be compatible with both Ruby 2.3.3 and 2.5.7+ if that's desired.

the issue is expected to be resolved, please upgrade to >= 1.1.22


Allow tree structure for punchbag DigiROMs

So we can have more without too much scrolling.

Resolve #186 first. Figure out why we're using so much RAM at the moment.


Get-IshFolderContent parameter VersionFilter defaults to LATEST but doesn't allow empty to retrieve all versions

When two or more versions exist on an object, Get-IshFolderContent can only retrieve the latest or specific but not all versions.
Get-IshFolderContent -IshSession $ishSession -VersionFilter "" -LanguagesFilter "en"
Suggest to remove [ValidateNotNullOrEmpty] on VersionFilter which would allow a deliberate empty string. Do note that we cannot allow other metadata filter options here. For that use the default LATEST on version filter and use Get-IshDocumentObj to tune.

Suggestion doesn't work, and results in [-145] The version number must be supplied. [145;VersionIsMandatory] as the API call doesn't allow it. As an extra API call is required anyway, the suggestion is to move/pipe into Get-IshDocumentObj. Otherwise all those filters have to be added on Get-IshFolderContent to be consistent.


Kernel driver not loading on Windows Release Preview (21H2)

Describe the bug
Recently on each startup I started to receive an error message from ProcessHacker:
[Window Title]
Process Hacker

[Main Instruction]
Unable to verify the kernel driver signature.

[Content]
Do usługi lub funkcji przekazano nieprawidłowy parametr.

You will be unable to use more advanced features, view details about system processes or terminate malicious software.

[Close]

Message in the middle in English should probably be: An invalid parameter was passed to a service or function.
I thought that it might be problem with new version, but I tried older one (3.0.4373) and it is affected as well.
It seems related to:

https://github.com/processhacker/processhacker/issues/997
But I don't have core isolation enabled

https://github.com/processhacker/processhacker/issues/340
But I don't have any AV, and Windows Defender is disabled.

It seems directly related to the fact that I'm on release preview version of Windows 10.
I checked it on mainstream version of Windows 10 (20H2 build 19042.1387) and no error appears.
I think it's worth investigating now, to avoid letting it break when the update arrives to mainstream.
To Reproduce
Steps to reproduce the behavior:

Install newest Release preview of Windows 10 (21H2)
Install latest nightly version of ProcessHacker (currently v3.0.4399)
Enable kernel-mode driver in options
Restart system
See error

Expected behavior
Error does not appear, kernel driver loads normally.
Screenshots

Desktop:

OS: Windows 10
Version: Release preview 21H2  build 19044.1387

21H2 build 19044.1387

I checked that version and there were no errors, so it would be something on your machine modifying the executable or the signature file causing the verification failure.
There was a similar report last year about this and they found a rootkit/bitcoin miner that would suspend when running task manager or PH so it's probably that same variant - Microsoft and Antivirus companies have never once done anything to stop malicious software attacking PH, so your AV probably isn't showing the rootkit/bitcoin miner during scans... you'll have to check manually.

I updated another computer to the same Windows version and it works, so you're right.. Sorry ;)
Thanks for the heads-up.  I'll try to isolate this issue and if I find anything I'll let you know.

I'll try to isolate this issue and if I find anything I'll let you know

I've come across a few kernel caching issues with binaries containing CFG bitmaps on Windows 10 but nothing recent.

I just upgraded Nvidia drivers (496.76 -> 497.09) and MsiAfterburner (4.6.3 beta -> 4.6.4) along with RivaTuner (7.3.2 beta -> 7.3.3) and the problem has gone away.
I was already thinking that RivaTuner probably could cause such issues as it hooks itself to pretty much everything, but I recently added ProcessHacker to its blacklist, so I thought I was safe.
Also I don't have MsiAfterburner and RivaTuner on my second computer, so that pretty much confirms it - MsiAfterbuner (4.6.3 beta) or RivaTuner (beta 7.3.2) was the culprit ;)


Crash on creating new world: Missing sound file

The sound ultracraft:music.limbo.calm referred in data\ultracraft\worldgen\biome\limbo.json is not present, causing the crash.

More issue came out. https://mclo.gs/H4On8Ld

that by itself shouldn't cause a crash; can you send a full crash report ?

I did more tests later. There's another mod causing problems which can actually crash the game so I guess for Ultracraft it's only this small issue.


Bug - Cannot Click "Initiative" to roll for Initiative

Currently, you cannot click the word "Initiative" to roll for combat initiative. You have to switch back to the default 5e sheet or enter in a /r 1d20 + dex into chat.

I have also disabled all other modules to make sure nothing conflicts.

Dang. I thought I fixed that. Thanks for the report! I'll take a look asap.

Thanks!

Fixed in 0.5.6


Embedded Ansible - Service provisioning based on Ansible playbook stuck in check_completed state

I activated the Embedded Ansible role on the lasted and freshly installed version of MIQ.
I have also added a public repository, all playbooks were discovered.
The issue, is when provisioning a simple service(create a new user on MIQ), the job is stucking in check_completed state and then just retrying endless.
I followed this tutorial:
https://cloudformsblog.redhat.com/2017/07/24/my-first-ansible-service-video/
My service is using exactly the same playbook as in the tutorial "create_user.yml" from this repository:
https://github.com/jonnyfiveiq/Ansible_Playbooks/blob/master/ManageIQ/create_user.yml
No errors in the evm.log or in the automation.log files.
Thanks in advance for your help.

I am facing exactly similar issue. After retrying, throws this ERROR in evm.log:
[----] E, [2018-03-09T13:49:16.564855 #1602:11c3134] ERROR -- : Q-task_id([service_template_provision_task_2]) MIQ(MiqAeEngine.deliver) Error delivering {"dialog_credential"=>nil, "dialog_hosts"=>"localhost", "request"=>"clone_to_service", :service_action=>"Provision", "Service::Service"=>2} for object [ServiceTemplateProvisionTask.2] with state [check_completed] to Automate:

@kekimiann is this still an issue for you? If not, please close it.


[BW Clock] don't draw over clock info boarder when updating time

Fixes #2459

This should be obsolete with my new PR #2471 which allows showing only the time etc.

Closing this then 🙂


_tkinter.TclError: couldn't connect to display "localhost:14.0"

I am trying to set up hgtector to run on a set of bacterial genomes in a remote server. I created a conda environment as per the instructions and I was able to run the search part of the second run example. However, when I try to run the analysis part I get the error below
Analysis started at 2020-06-12 15:16:01.101009.
Reading local taxonomy database...
Done. Read 92 taxa.
Reading homology search results...
  o55h7: 5045 proteins.
Done. Read search results of 1 samples.
Auto-inferring plausible taxIds for input genomes based on taxonomy of search results...
  o55h7: 562 (Escherichia coli) (covering 93.2933% best hits).
Refining taxonomy database...
Done. Retained 92 taxa.
All input genomes belong to 562 (species Escherichia coli).
Auto-inferred self group:
  562 (species Escherichia coli)
Self group has 5 taxa.
Auto-inferred close group:
  543 (family Enterobacteriaceae)
Close group has 15 taxa.
Calculating protein scores by group...
  o55h7
Done.
Summarizing scores of all proteins... done.
Protein scores written to scores.tsv.
Removed 687 ORFans.
Removed 89 outliers.
Predicting HGTs...
Calculating thresholds for clustering...
Close group:
Traceback (most recent call last):
  File "/home/groups/hbfraser/modules/packages/conda/4.6.14/envs/hgtector/bin/hgtector", line 96, in <module>
    main()
  File "/home/groups/hbfraser/modules/packages/conda/4.6.14/envs/hgtector/bin/hgtector", line 35, in main
    module(args)
  File "/home/groups/hbfraser/modules/packages/conda/4.6.14/envs/hgtector/lib/python3.8/site-packages/hgtector/analyze.py", line 160, in __call__
    self.predict_hgt()
  File "/home/groups/hbfraser/modules/packages/conda/4.6.14/envs/hgtector/lib/python3.8/site-packages/hgtector/analyze.py", line 720, in predict_hgt
    self.plot_hist(self.df[group].tolist(),
  File "/home/groups/hbfraser/modules/packages/conda/4.6.14/envs/hgtector/lib/python3.8/site-packages/hgtector/analyze.py", line 1027, in plot_hist
    fig = plt.figure(figsize=(5, 5))
  File "/home/groups/hbfraser/modules/packages/conda/4.6.14/envs/hgtector/lib/python3.8/site-packages/matplotlib/pyplot.py", line 540, in figure
    figManager = new_figure_manager(num, figsize=figsize,
  File "/home/groups/hbfraser/modules/packages/conda/4.6.14/envs/hgtector/lib/python3.8/site-packages/matplotlib/backend_bases.py", line 3337, in new_figure_manager
    return cls.new_figure_manager_given_figure(num, fig)
  File "/home/groups/hbfraser/modules/packages/conda/4.6.14/envs/hgtector/lib/python3.8/site-packages/matplotlib/backends/_backend_tk.py", line 876, in new_figure_manage
r_given_figure
    window = tk.Tk(className="matplotlib")
  File "/home/groups/hbfraser/modules/packages/conda/4.6.14/envs/hgtector/lib/python3.8/tkinter/__init__.py", line 2261, in __init__
    self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use)
_tkinter.TclError: couldn't connect to display "localhost:14.0"

My guess is that it is trying to show me some plots but there is no graphical interface in the remote server so it is crashing. Is that correct? can I disable the graphical output?
Thanks

Hello @surh Sorry for the late response. I believe that your guess is correct. I modified the code (#56 ) so that matplotlib will no longer attempt to open a graphical interface which is not available in many remote servers, but it is still capable of generating figures. Please update the program by:
pip install -U<EMAIL_ADDRESS>
And let me know if it now works for you! If so I will update my master branch. Thank you!

thanks for looking into this @qiyunzhu, your fix worked!
Thanks,
Sur


Windows installation page lists wrong cudnn version

The windows installation pages specifically asks to use cuDNN 6 - link. Then, when running tensorflow it looks specifically for cuDNN7.
% (build_info.cudnn_dll_name, build_info.cudnn_version_number))

ImportError: Could not find 'cudnn64_7.dll'. TensorFlow requires that this DLL be installed in a directory that is named in your %PATH% environment variable. Note that installing cuDNN is a separate step from installing CUDA, and this DLL is often found in a different directory from the CUDA DLLs. You may install the necessary DLL by downloading cuDNN 7 from this URL: https://developer.nvidia.com/cudnn
Would be great if the page modified this to specifically ask for cuDNN 7.

Thanks for the note.
@gunan @av8ramit - does the Windows page need to be updated?

FYI @markdaoust
I thought we updated the website with the correct information, could you confirm?

Thanks for the reminder!
This is fixed now: https://www.tensorflow.org/install/install_windows


Fix error in parsing incorrectly declared attributes.

When parsing incorrectly declared attributes, HtmlAgilityPack currently omits one character from the attribute-name.
Consider the following example:
<html>
   <body>
    <img src='http://boo'/onerror='moo'>
  </body>
</html>

When parsed this will result in two attributes for the img node:

src
nerror (note missing leading o)

Expected outcome should be:

src
onerror

This patch fixes the output to match what you would expect (and how browser handle similarly formatted data). This closes https://github.com/zzzprojects/html-agility-pack/issues/16.
Note: In lack of a test-suite for this project, I cannot reliably tell if I've broken other expected behaviour in the process.
Are there any recommended test-procedures I should follow, or is this good as is?

Hello @josteink ,
Thank you for this pull request, I will try to review & merge it today or tomorrow.
There are no test-procedures to follow currently.  I'm normally a big fan of unit test (by example, Entity Framework Extensions has over 10k unit tests!) however this library lack of it. Probably during the summer, we will choose if we re-write a v2.0 for this library and create unit test on this version or add unit test on the v1.0
Best Regards,
Jonathan

Awesome. Thanks!


proposal: mark MarshalPKIXPublicKey deprecated in favor of key-type specific functions

What version of Go are you using (go version)?

$ go version
go1.12.5

What did you do?
Used MarshalPKIXPublicKey with the wrong format.
What did you expect to see?
Expected the program not to compile as this should be typed, or at least a verbose error
What did you see instead?
panic: x509: only RSA and ECDSA public keys supported

Proposal:
In my opinion, MarshalPKIXPublicKey is too generic (interface{} takes in anything) and should be marked as deprecated in the comment. To resolve this we should add 2 new functions with the follow signature:

func MarshalPKIXRSAPublicKey(key *rsa.PublicKey) ([]byte, error)
func MarshalPKIXECPublicKey(key *ecdsa.PublicKey) ([]byte, error)

func MarshalPKIXPublicKey can then be rewritten as such, this maintains the functionality:
func MarshalPKIXPublicKey(pub interface{}) ([]byte, error) {
	switch pub := pub.(type) {
	case *rsa.PublicKey:
		return MarshalPKIXRSAPublicKey(pub)
	case *ecdsa.PublicKey:
		return MarshalPKIXECPublicKey(pub)
	default:
		return nil, errors.New("x509: only RSA and ECDSA public keys supported")
	}
}

Furthermore we should drop marshalPublicKey from the whole codebase, rewriting this to the following: marshalRSAPublicKey and marshalECPublicKey.
These take the code from the original (marshalPublicKey) switch case, but this time, each in their own function.
The two new functions would be:
func MarshalPKIXRSAPublicKey(key *rsa.PublicKey) ([]byte, error) {
        var publicKeyBytes []byte
	var publicKeyAlgorithm pkix.AlgorithmIdentifier
	var err error

	publicKeyBytes, publicKeyAlgorithm, err = marshalRSAPublicKey(key)
	if err != nil {
		return nil, err
	}

	pkix := pkixPublicKey{
		Algo: publicKeyAlgorithm,
		BitString: asn1.BitString{
			Bytes:     publicKeyBytes,
			BitLength: 8 * len(publicKeyBytes),
		},
	}

	ret, _ := asn1.Marshal(pkix)
	return ret, nil
}

and
func MarshalPKIXECPublicKey(key *ecdsa.PublicKey) ([]byte, error) {
        var publicKeyBytes []byte
	var publicKeyAlgorithm pkix.AlgorithmIdentifier
	var err error

	publicKeyBytes, publicKeyAlgorithm, err = marshalECPublicKey(key)
	if err != nil {
		return nil, err
	}

	pkix := pkixPublicKey{
		Algo: publicKeyAlgorithm,
		BitString: asn1.BitString{
			Bytes:     publicKeyBytes,
			BitLength: 8 * len(publicKeyBytes),
		},
	}

	ret, _ := asn1.Marshal(pkix)
	return ret, nil
}

This prevents users from compiling code which is totally out of typed scope, since interfacer{} takes anything. This is not exactly type safe / correct.
Finally, since we drop marshalPublicKey (which, again, is too generic) from the codebase, we will need to edit CreateCertificate and CreateCertificateRequest to reflect the change, but these should just run a simple switch case, e.g. CreateCertificate becomes:
	var publicKeyBytes []byte
	var publicKeyAlgorithm pkix.AlgorithmIdentifier
	switch pub := pub.(type) {
	case *rsa.PublicKey:
		publicKeyBytes, publicKeyAlgorithm, err = marshalRSAPublicKey(pub)
	case *ecdsa.PublicKey:
		publicKeyBytes, publicKeyAlgorithm, err = marshalECPublicKey(pub)
	default:
		return nil, errors.New("x509: only RSA and ECDSA public keys supported")
	}

WDYT? Highlighting @FiloSottile and @andrewmbenton in regards to earlier discussions about this topic.

This is reasonable I think, depending on how MarshalPKIXPublicKey is most commonly used. If callers most-often don't know the key type, then this just forces the type check and switch statement outside of the standard library. Maybe that's desirable on its own, but it would be more compelling if lots of callers actually know the key type they want to marshal already.
Then there's also the fact that MarshalPKIXPublicKey is the inverse of ParsePKIXPublicKey.
In any case this proposal doesn't satisfy my particular needs RE marshaling of public keys. My goal was to get at the underlying publicKeyBytes without the PKIX overhead. So the most useful thing for me would be to export your new marshalRSAPublicKey and marshalECPublicKey funcs.

Well, since marshalRSAPublicKey and marshalECPublicKey are perfectly typed I do not see a problem making these public. However I would rename it to be more precise (less overlap with other names):

MarshalRSAPublicKeyRaw
MarshalECPublicKeyRaw

^ this would clarify that these are raw parts, other naming schemes should be considered as well.
Thank you for pointing out the ParsePKIXPublicKey, I woud actually suggest we do the exact same as with MarshalPKIXPublicKey ==> Mark it deprecated and split the functionality in 3 separate functions (RSA, DSA, ECDSA).
Any reason why MarshalPKIXPublicKey cannot marshal a DSA public key btw?

I don't think it's fair to expect that every caller of ParsePKIXPublicKey will know what type of key it is asking to be parsed. In fact I expect almost none will.
How would you know what func to call when attempting to parse a PEM-encoded key like the following?
MIG2AgEAMBAGByqGSM49AgEGBSuBBA...
-----END PUBLIC KEY-----```

Took some time to think about this, and I don't think the type safety advantages justify adding three (see below) more functions to the already dizzying array of crypto/x509.
func MarshalECPrivateKey(key *ecdsa.PrivateKey) ([]byte, error)
func MarshalPKCS1PrivateKey(key *rsa.PrivateKey) []byte
func MarshalPKCS1PublicKey(key *rsa.PublicKey) []byte
func MarshalPKCS8PrivateKey(key interface{}) ([]byte, error)
func MarshalPKIXPublicKey(pub interface{}) ([]byte, error)

PKCS#8 and PKIX are generic encapsulation formats, so it makes sense that they would have untyped arguments. In Go 1.13 we are adding support for Ed25519, and it was nice not to have to add a new exported function for it.
The argument for the parsing functions is even weaker, as @andrewmbenton pointed out.
We definitely wouldn't want to deprecate the existing MarshalPKIXPublicKey, too. I want to keep deprecation in a crypto package as a strong signal that action is required for safety or security. This is not a safety or security issue.
The original issue looks like it would be addressed well by improving the error message, which is #32640.


load_preprocessed error - TypeError: unhashable type: 'dict'

I got this error when trying to train on a preprocessed input.
The error comes from here: https://github.com/sherjilozair/char-rnn-tensorflow/blob/master/utils.py#L42
self.chars is a tuple with (1) a list of chars and (2) a dict of char to integer mapping. I changed the function to just get it running:
def load_preprocessed(self, vocab_file, tensor_file):
    with open(vocab_file) as f:
        self.chars = cPickle.load(f)[0][0]
    print self.chars
    self.vocab_size = len(self.chars)
    self.vocab = dict(zip(self.chars, range(len(self.chars))))
    self.tensor = np.load(tensor_file)
    self.num_batches = self.tensor.size / (self.batch_size * self.seq_length)

self.chars is just a list of characters, and that's what's being stored in the vocab_file. Check this: https://github.com/sherjilozair/char-rnn-tensorflow/blob/master/utils.py#L34
I'm not sure why are you getting this error. Are you sure you haven't changed something which is causing this problem?

No, I did not change anything. I will let you know shortly how to replicate the error.

@sherjilozair I found out why.
Normally, during preprocessing, two vocab.pkl are created, one in data_dir and the other in save_dir. The two files are different.
https://github.com/sherjilozair/char-rnn-tensorflow/blob/master/utils.py#L34
https://github.com/sherjilozair/char-rnn-tensorflow/blob/master/train.py#L48
In my experiment I set --data_dir and --save_dir to be the same folder, resulting in one vocab.pkl overwriting the other. When I keyboard interrupt the training and try to restart it, the content in the remaining vocab.pkl (generated by train.py) does not match what is expected (generated by utils.py).

Whoops. Quite a nasty bug. I'll fix this later today. Thanks for figuring this out.


Difference from another ADMM pruning work published at ECCV 2018

I just wanted to know the difference between this paper and the one published years ago. Thank you. Please refer to this paper https://openaccess.thecvf.com/content_ECCV_2018/papers/Tianyun_Zhang_A_Systematic_DNN_ECCV_2018_paper.pdf

Our work is about layer-wise pruning of LLMs.
ECCV work is about using ADMM as heuristic for pruning the whole network.


chore(main): release 0.9.0

:robot: I have created a release beep boop
0.9.0 (2023-02-15)
Features

add information redirect of sed command in uninstall.sh script (b47b759)
add information redirect of sed command in uninstall.sh script (271f434)
add system parameters of net.core.somaxconn (c989c10)
ngxway cannot guarantee that the command will take effect in the global environment during installation (baa48ce)

Bug Fixes

fix failed to create ngxway lobal command (89fa561)
fix the bug No such file or directory: uninstall.sh (c7b92e0)
fix the bug of adjusting system parameters (d9bba20)
fix the bug of adjusting system parameters (17f72e3)
fix the install.sh source not work (48164d2)
fix the install.sh source not work (45d16a2)
fix the install.sh source not work (8a0cb15)
fix the md5 command not found (f37d3eb)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/WGrape/ngxway/releases/tag/v0.9.0 :sunflower:


15min vs 60min data

Thank you for your effort!
I think this mixes up the 15min and 60min data. Looks like it first parses the 15min data (as if it were 60min data) and then overwrites the first 24 positions with 60min data.
I think you need another parameter with resolution=PT15MIN/PT60MIN and search for
pe_child.tag.endswith("resolution")

and filter accordingly. Then it will work.
Edit: I just realized that you are properly testing with a country that only has 60min data. You can try '10Y1001A1001A82H' (DE_LU) to see the problem.

Hey. what parameters are you using? Because when i run this for NL I get a
list of hourly data back. I confirmed this by running the URL directly. I
don't see 15m data.
Op vr 4 okt 2024 om 22:59 schreef Olaf Lüke @.***>:

Thank you for your effort!
I think this mixes up the 15min and 60min data. Looks like it first parses
the 15min data (as if it were 60min data) and then overwrites the first 24
positions with 60min data.
I think you need another parameter with resolution=PT15MIN/PT60MIN and
search for
pe_child.tag.endswith("resolution")
and filter accordingly. Then it will work.
—
Reply to this email directly, view it on GitHub
https://github.com/GeneralCP/entsoe2/issues/1, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABURJDFQEHJ55TX2I5XRYFTZZ36SDAVCNFSM6AAAAABPMTBIKWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGU3DOMZWHEYTQMA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Try 10Y1001A1001A82H for the country (DE_LU)

I fixed this in the latest version. https://github.com/EnergieID/entsoe-py is also fixed now so might want to go back to that. This version is still a lot lighter though.


App Launch is failing

Since one of the newer updates I get an error message saying "You must install or update .NET to run this application. Framework: 'Microsoft.AspNetCore.App', version '7.0.0' (x64)".  The downloader always worked without problems but now I get this error even after reinstalling the latest .NET 7.0 Desktop Runtime.

Try installing it manually from https://dotnet.microsoft.com/en-us/download/dotnet/7.0
make sure you use version 7.0.0 and not any later version

Since one of the newer updates I get an error message saying "You must install or update .NET to run this application. Framework: 'Microsoft.AspNetCore.App', version '7.0.0' (x64)". The downloader always worked without problems but now I get this error even after reinstalling the latest .NET 7.0 Desktop Runtime.

its not working because your downloading the wrong one, you need to install " 'Microsoft.AspNetCore.App', version '7.0.0' (x64)"  aka  aspnetcore-runtime-7.0.0-win-x64

Yes this did the trick, thank you!


PLT - add visualization of objective as function of stop_val

This adds a toggle to enable visualizing the objectives as function stop_val, namely n_iter.
Link to a dummy benchmark  to preview the feature. (credit to @mathurinm for the idea of the benchmark).
A link to the benchmark repo  to reproduce.

PR related to #287

any thoughts @mathurinm?

Thanks for having a look @Badr-MOUFAD ! I can't see anything for the other objective, is it normal ?
https://hal.archives-ouvertes.fr/hal-01941152v2/document

How does this handle combination of solvers that use iteration stop_val and tol stop val ?

Thanks for having a look @Badr-MOUFAD ! I can't see anything for the other objective, is it normal ?

@mathurinm, the values of the objective are negative and hence can't be visualized using a log-y scale.
Try linear scale or in the sidebar set the objective to Affine[with_exp=True]

Some minor comments after @mathurinm's invitation (to take with a grain of salt):

Stopping Criterion is not intuitive for the name of such a toggle, an alternative name would at least include "X-axis" somewhere
Stop_value is also unintuitive #iterations is better
I would avoid scientific notation for the x-axis in the case of iterations and use the integer prior over the indices to have a better visualization (as in matplotlib MaxNLocator(integer=True))
The default scale for a plot function of iterations should be linear

(Not linked to this PR, but with benchopt the kind of ZigZag pattern that can be observed in the current result plot is not rare and a little bit puzzling)

Actually, I am not sure we want to add a switch.
I would be more in favor of a solution to have a selector that allows to put any metric as x and any as y? (keeping the default to time and first objective value).
This would make the UI very explicit that one is choosing the axis value.
And it would also solve extra use cases, where people want to look for pareto front with multiple objectives.
Also, this would allow to separate iterations/tolerance, which I think is a pain to handle right now (and probably why it is called stop_value in the PR).
WDYT?

I would like to reiterate that the completion of this PR would be extremely valuable for a lot of us in the community.
I side with @tomMoral on the selector for x instead of a toggle. It would be very easy to add more resampling options in the long-term.
I also remain adamant that at least on the display html, names should reflect what the user is interested in and not variable names from the codebase i.e. iterations/tolerance and not StoppingCriterion or stop_val.
It would be really great as well if we can have integer ticks if x == iterations.
Cheers.

As discussed with @mathurinm IRL, I propose the following solution to move on with this PR:

We add a drop-down menu with ['iteration', 'tolerance'] on the page.
In the solver_data[solver]['scatter'] dictionary, we add a value 'strategy' in iteration, tolerance.
Depending on the drop-down menu, we display only the solver with the strategy that matches the one selected in the drop-down menu.

I would say this looks like a good first step, without adding too much complexity? WDYT @Badr-MOUFAD ?

Codecov Report

Merging #479 (d673c7b) into main (1487639) will decrease coverage by 0.02%.
The diff coverage is 0.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##             main     #479      +/-   ##
==========================================
- Coverage   54.64%   54.62%   -0.02%
==========================================
  Files          43       43
  Lines        2796     2797       +1
  Branches      511      511
==========================================
  Hits         1528     1528
- Misses       1153     1154       +1
  Partials      115      115

Yes, it sounds good to me.

@tomMoral, I have a little concern about this point

In the solver_data[solver]['scatter'] dictionary, we add a value 'strategy' in iteration, tolerance.

In fact, we don't store the strategy in the data frame of results and hence we cannot determine the stopping strategy at the plotting stage: generate_html.py.
One workaround is to add the stopping strategy attribute in meta when running the benchmark. That way the stopping_startegy would be available in the final data frame
https://github.com/benchopt/benchopt/blob/1487639e156a9acec809cb3a86fd2d2e7960115f/benchopt/runner.py#L187-L192
WDYT @tomMoral? Do you see another workaround?

WDYT @tomMoral? Do you see another workaround?

I would have suggested the same so perfect :) As the dataframe is stored as a parquet (thus compressed), the impact on the output file's size should be minim so works for me.

With the latter remarks being taken into account here is a preview of the features.
The xaxis type drop menu acts also as a filter: only the solver with a stopping strategy matching xaxis type will be shown.
@tomMoral, @jeandut I am open to suggestions.

With the latter remarks being taken into account here is a preview of the feature.
The xaxis type drop menu acts also as a filter: only the solver with a stopping strategy matching xaxis type will be shown.
@tomMoral, @jeandut I am open to suggestions.

Very cool work it looks very good !
A few disorganised comments on the preview:

I would remove the word type after X-axis.
Tolerance axis does not seem to work in this example
would love to have integer tick labels when switching to iterations but this is a detail
Overall it's very nice !

Thanks, @jeandut for the feedback! Here is the updated preview of the feature.

Could not check on my computer right now but any idea why the x-axis is not visible on mobile?

I will review it tomorrow but the changes look good and good idea to create the drop down menu dynamically, it is better to only have non empty option 😁

Could not check on my computer right now but any idea why the x-axis is not visible on mobile?

I have just become aware that the responsiveness (mobile support) was implemented by duplicating the sidebar's content and styling it separately.
https://github.com/benchopt/benchopt/blob/d520e74b09eed7b3bf6c87bc2a4ef90e5d36655b/benchopt/plotting/html/templates/result.mako.html#L22-L25
https://github.com/benchopt/benchopt/blob/d520e74b09eed7b3bf6c87bc2a4ef90e5d36655b/benchopt/plotting/html/templates/result.mako.html#L97-L99
Hence to fix https://github.com/benchopt/benchopt/pull/479#issuecomment-1454283066, we should make the same changes we did for the sidebar of the big screen to the one on the mobile screen.
Responsiveness is usually implemented through breakpoints that specify the style to be applied according to the screen size.
A follow-up TODO of this PR is to remove the content duplication and implement properly the responsiveness.

Thanks, @jeandut for the feedback! Here is the updated preview of the feature.

Awesome !!!

Although there might be a bug see values in iteration axis when converting to log-log ?:

Could not check on my computer right now but any idea why the x-axis is not visible on mobile?

I have just become aware that the responsiveness (mobile support) was implemented by duplicating the sidebar's content and styling it separately.
https://github.com/benchopt/benchopt/blob/d520e74b09eed7b3bf6c87bc2a4ef90e5d36655b/benchopt/plotting/html/templates/result.mako.html#L22-L25
https://github.com/benchopt/benchopt/blob/d520e74b09eed7b3bf6c87bc2a4ef90e5d36655b/benchopt/plotting/html/templates/result.mako.html#L97-L99
Hence to fix #479 (comment), we should make the same changes we did for the sidebar of the big screen to the one on the mobile screen.
Responsiveness is usually implemented through breakpoints that specify the style to be applied according to the screen size. A follow-up TODO of this PR is to remove the content duplication and implement properly the responsiveness.
P.S. I updated the feature preview

Thanks a lot for the explanation, indeed it would be much better to implement it with the right synthax.

Although there might be a bug see values in iteration axis when converting to log-log ?: 1-2-5-10-2-5-100-2-5-1000 etc.

Looking at Plotly log scale doc, I don't think it is a bug.
Looks fine to me. Do you think it might be misleading @jeandut?

Although there might be a bug see values in iteration axis when converting to log-log ?: 1-2-5-10-2-5-100-2-5-1000 etc.

Looking at Plotly log scale doc, I don't think it is a bug. Looks fine to me. Do you think it might be misleading @jeandut?

You are totally right. The tick values do make sense somehow on log-scale. Looks like it's ready to be merged ! Great work !

merged, thanks a lot for making this feature @Badr-MOUFAD !


the raw data

Hello,
I would like to ask, the data set in the raw data link you gave is not the same as the data set which only contains the direction sequence information given in the paper on Deep Fingerprinting, so how did you get the data set in the raw data link? Is it obtained by processing the dataset of Deep Fingerprinting?

Hi, we collected the raw dataset from the authors of the Deep Fingerprinting paper. We get the timing and direction information by processing the dataset of Deep Fingerprinting. Because of the storage limitation, we discarded the raw data and kept only the necessary timing and directional information.

Thank you very much for your reply, you have shown that https://drive.google.com/drive/folders/1k6X8PjKTXNalCiUQudx-HyqoAXVXRknL is the raw data given in the Deep Fingerprinting article. But when I checked the Deep Fingerprinting article's publicly available code repository https://github.com/deep-fingerprinting/df and found the publicly available dataset chain https://drive.google.com/drive/folders/ The dataset contained in 1kxjqlaWWJbMW56E3kbXlttqcKJFkeCy6 is not the same as the raw data you gave, this one contains only orientation sequences. Where did you get the raw data containing the time information?

I see the confusion. Because we worked in the same research group, we have had the .pcap files at that time and for Tik-Tok paper we extracted the timing information from the same raw files. We do not have those .pcap files available now.

I understand and thank you for your patience.