[Home] Adds some null protection to the home / auctions pages

In looking at https://sentry.io/artsynet/eigen-staging/issues/406930867/ we had wondered if it was related to an expired access token. TBH, I don't think it does right now, (and our access tokens are 25 years by default, you'd need to change your password) but this at least exposes a bunch of easy to grab nil crashes bugs.
#trivial

👍 Yeah I had removed this guard from here in a previous PR assuming that metaphysics would, no matter what, return an empty array from edges. Was I wrong?

You're not wrong, wrong - it's more that the related me request could fail to resolve if you have no access token - so then everything is a bit broke :D
It's a trade-off on whether these should come in, but it does mean that all root screens won't crash if you  happen to change your password. Might assign to @alloy to have final say

I’m not fully following what issues you were able to reproduce and how, could you elaborate on that? Without that info it’s kinda hard to judge the code.

Seeing as the MP schema changed and the merge conflict needs to be resolved, I’d also like us to take some time to re-test that it all works as expected after doing that. As such, I’ll leave this one for 4.0.1


Roadmap for implementing support for OSes

In some projects I am working on an ability to use this build system would be very helpful. To enable any planing around this I would appreciate if team would share feature implementation roadmap indicating approximate times when support for .NET supported OSes will be available in this repo. This could potentially save some duplicate work on my side.

There are separate issues tracking support for different OS'es now:

Windows: https://github.com/dotnet/source-build/issues/2910
macOS: https://github.com/dotnet/source-build/issues/2909
aribitrary Linux distros: https://github.com/dotnet/source-build/issues/297

Is there anything else that needs to be answered? Can this issue be closed as a "duplicate" of these other issues (sorry for the terminology, I know the other issues were filed later).

[Triage] Closing in favor of tracking with the previously mentioned issues.


Feature: refactors the transformer subsystem to be more generic

What this PR changes/adds
This PR moves the transformer interfaces into spi:web-spi and adds marker interfaces for api and ids.
Further, I added implementations for the API transformer and -registry, which should be used by all transformers and controllers.
Why it does that
All API controllers should use the ApiTransformerRegistry to perform conversions between business objects and DTOs.
Before, the transformer subsystem resided in the ids module, so it needed to be made more generic.
Further notes

most of the plethora of changed files is due to the refactoring (imports)
unrelated: the entry for the contractagreement-API was added in the extensions/api/data-management/build.gradle.kts BOM
an example of how this should be used can be found here

Linked Issue(s)
Closes #779
Checklist

[x] added appropriate tests?
[x] performed checkstyle check locally?
[x] added/updated copyright headers?
[x] documented public classes/methods?
[ ] added/updated relevant documentation?
[x] added relevant details to the changelog? (skip with label no-changelog)
[x] formatted title correctly? (take a look at the CONTRIBUTING and styleguide for details)

@paullatzelsperger how generic are the transformers intended to be? For example, would they be used to transform types in non HTTP endpoints? For example, over other transports?
It doesn't seem the web module is the correct place for them.

@paullatzelsperger how generic are the transformers intended to be? For example, would they be used to transform types in non HTTP endpoints? For example, over other transports?
It doesn't seem the web module is the correct place for them.

I should clarify, the core transformer types, not the API and ids related ones.

@paullatzelsperger how generic are the transformers intended to be? For example, would they be used to transform types in non HTTP endpoints? For example, over other transports?
It doesn't seem the web module is the correct place for them.

I think they should be used to also transform, say, gRPC protocol messages. Then, they should be n some sort of yet-to-be-created 'transport-spi' or so. Wouldn't put them in 'core-spi'

@Izzzu added some example code to the README.md

@paullatzelsperger how generic are the transformers intended to be? For example, would they be used to transform types in non HTTP endpoints? For example, over other transports?
It doesn't seem the web module is the correct place for them.

I think they should be used to also transform, say, gRPC protocol messages. Then, they should be n some sort of yet-to-be-created 'transport-spi' or so. Wouldn't put them in 'core-spi'

+1 transport-spi. Can you include that in this PR?

+1 transport-spi. Can you include that in this PR?

done.


Don't understand why need to Register the app in the Portal.

I am trying to modify the code for the Ago-Assistant, and find that I need to register the app in my Portal to obtain an 'appId' for OAuth dialog to log in first. But I don't know how and why should I do this.
To try to register it in Portal, from the Portal's 'Add Item', I chose 'An Application', and then it give me several optional types: Web Mapping, Mobile, Desktop, Application, and Application Extension. I don't know how should I fill the info in this page.
Besides, I have some hard time to put up the project in Visual Studio, particularly on how to use NPM and Grunt in Visual Studio 2015.
Thanks,
Wei

I need to register the app in my Portal [...] I don't know how and why should I do this.

I'd recommend checking out our documentation on OAuth2 and Implementing Named User Login.
The gist of it is that when you create an application item and register it you're creating a placeholder to make sure our platform knows that its safe to pass back short-lived tokens.

and then it give me several optional types:

Since ago-assistant is a web application, 'Web Mapping' is the right choice here.

how to use NPM and Grunt in Visual Studio 2015.

npm and grunt are both invoked via the command-line.

Also, you don't actually need to do the app registration step to use this app with your Portal. That's only if you want to use OAuth. You can use simple authentication by choosing the "Log in to Portal for ArcGIS" option.

Thanks, Evan. We are using SAML to authenticate our user. I think I still
need to register the app in this case, correct? I heard SAML is pretty
similar as OAuth. And what URL should I specify while registering it on
Portal with choosing 'Web Mapping' option?
Many thanks,
Wei
On Tue, May 16, 2017 at 9:59 AM, Evan Caldwell<EMAIL_ADDRESS>wrote:

Also, you don't actually need to do the app registration step to use this
app with your Portal. That's only if you want to use OAuth. You can use
simple authentication by choosing the "Log in to Portal for ArcGIS" option.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Esri/ago-assistant/issues/184#issuecomment-301809898,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AG8o3ShAlNLOTLF53RpgtjwAHZiRBasLks5r6bnegaJpZM4Nb8JS
.

to be clear, the master branch of this repo doesn't currently support OAuth2 logins to portals (whether they utilize SAML or not).

We are using SAML to authenticate our user. I think I still
need to register the app in this case, correct?

@ecaldwell is correct that an older school technique to login to Portals is supported, the catch is that SAML identity providers require an OAuth2 approach and that that approach requires you to register your own app.

And what URL should I specify while registering it on Portal with choosing 'Web Mapping' option?

Its not crystal clear in the doc i linked previously, but it doesn't matter what url you specify when you create the item.  in contrast, it is crucial that you specify redirect uris that correspond to the actual location of your app.
i have isolated some changes in https://github.com/Esri/ago-assistant/pull/180 that so far have proven sufficient to get folks logged into a SAML Portal using OAuth2 (via the Log in to ArcGIS Online). if you're interested in helping test that work, that'd be great.

@jgravois we are working on supporting OAuth across both source and destination (for Portal) in a separate branch which should resolve this issue shortly..

Addressed by PR #194.


flatMerge & flatCombine

This is more of a helper/optimization suggestion that maybe should go in some form of "best practice/helper" section?
Often we come in a situation where we need to merge the sinks of a stream of sinks. This especially common when working with nested children whom need to be either combined (DOM) or merged (usually everything but DOM).
Usually you would do this:
const items$ = state$
  .pluck('items')
  .map(items => items.map(item => Item({ ...sources, props$: Observable.of(item))
  .shareReplay(1)

return {
   DOM: items$.flatMapLatest(items => Observable.combineLatest(...items.map(item => item.DOM), (...items) => view({ items }))
   HTTP: items$.flatMapLatest(items => Observable.merge(...items.map(item => item.HTTP))
}

which is rather inefficient since flatMapLatest will not reuse existing subscriptions. Instead I would suggest something like:
const items$ = state$
  .pluck('items')
  .flatCombine(item => Observable.of(Item({ ...sources, props$: Observable.of(item))), (...items) => items)
  .shareReplay(1)

return {
   DOM: items$.flatCombine(item => item.DOM, (...items) => view({ items })),
   HTTP: items$.flatMerge(item => item.HTTP)
}

export function flatMerge(selector) {
  const pairs$ = this
    .startWith([])
    .pairwise()
    .share()

  const add$ = pairs$
    .flatMap(([prev, curr]) => curr.filter(x => prev.indexOf(x) === -1))
  const remove$ = pairs$
    .flatMap(([prev, curr]) => prev.filter(x => curr.indexOf(x) === -1))

  return add$
    .flatMap(x => selector(x).takeUntil(remove$.filter(remove => remove === x)))
}

export function flatCombine(selector, combiner) {
  // TODO: Optimize
  return this
    .flatMapLatest(items => Observable
      .combineLatest(
        Observable.of(null),
        ...items.map(selector),
        (...items) => combiner(...items.filter(x => x))),
    )
}

Thoughts?
The next thing to get the performance with nested components story finished is to solve the re-evaluation inside const items$ = ....
We need something similar for optimizing flatMapLatest + combineLatest the same way this optimises flatMapLatest + merge.

Related https://github.com/cyclejs/core/issues/204

flatMapLatest will not reuse existing subscriptions.

Why not use just flatMap() ?

@TylorS: Because flatMap while not cancel running subscriptions... has to be flatMapLatest.


Is there way to close a reveal card with a custom button?

You can add an event listener on your custom button and trigger a close event. It not the perfect solution but will do the work.
Here is the codepen link which will resolve you issue : https://codepen.io/anchit1501/pen/Xwwooy


Align our sync resolving behavior with async

@defunctzombie When you have a sec, can you please review this?  Thanks!


Added getInputRef prop

Hey @andreaswilli, thanks for a great library!
I need to get an input ref to be able to reset its value depending on the flow in my app. It seems that it's not possible at the moment. This PR adds an ability to access input prop outside of the component.

Wrong base branch repo. Closing this one.


App stops when com.google.android.webview is updated

Bug description:
My Android app keeps showing the sticky notification for a foreground service, but the react native service stopped working.
After a bug report dump from developer settings, I see this:
  package: com.myapp.mobile
    Historical Process Exit for uid=10549
        ApplicationExitInfo #0:
          timestamp=2023-10-05 01:05:58.005 pid=8310 realUid=10549 packageUid=10549 definingUid=10549 user=0
          process=com.myapp.mobile reason=10 (USER REQUESTED) subreason=21 (FORCE STOP) status=0
          importance=125 pss=231MB rss=209MB description=stop com.google.android.webview due to installPackageLI state=empty trace=null
        ApplicationExitInfo #1:
          timestamp=2023-10-04 20:20:05.759 pid=8440 realUid=99256 packageUid=10549 definingUid=10171 user=0
          process=com.google.android.webview:sandboxed_process0:org.chromium.content.app.SandboxedProcessService0:0 reason=13 (OTHER KILLS BY SYSTEM) subreason=17 (ISOLATED NOT NEEDED) status=0
          importance=300 pss=53MB rss=143MB description=isolated not needed state=empty trace=null

It seems to me that Android is updating com.google.android.webview and with that stopping the app.
Is there a way to work around this issue? Or maybe even a clean solution?
To Reproduce:
I am unable to reproduce it so far.
Expected behavior:
That the app is not killed when webview is stopped
Screenshots/Videos:
n/a
Environment:

OS: Android
OS version: 13
react-native version: 0.71.8
react-native-webview version: 11.26.0

should be reproducible with following adb command:
adb shell am force-stop com.google.android.webview
BTW, There is following issue in react native to be handled prior to investigation of this issue https://github.com/facebook/react-native/issues/37211
I just tested a blank project: npx react-native@latest init AwesomeProject  (0.74.5) the app crashes with this reproduction step


Sync with upstream for new release

I exclude the new logging system of NNG which requires more attention to modify and sync.


Issue33/stitch solids examples

Fixes: https://github.com/krzysztofarendt/building3d/issues/33

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

7 of 7 (100.0%)  changed or added relevant lines in 3 files are covered.
3 unchanged lines in 2 files lost coverage.
Overall coverage increased (+0.05%) to 91.652%

Files with Coverage Reduction
New Missed Lines
%

geom/polygon.py
1
90.12%

geom/operations/stitch_solids.py
2
97.78%

Totals

Change from base Build<PHONE_NUMBER>:
0.05%

Covered Lines:
5292

Relevant Lines:
5774

💛  - Coveralls


[UI] Notification Center: Success notifications not displaying in correct filter

Current Behavior
See the recording
https://user-images.githubusercontent.com/7570704/128276081-13d8e504-98ae-4232-92f3-fb9fae605a52.mov
Desired Behavior
Success notifications should display under "Success" filter.
Environment

Browser: Safari
Host OS: Mac
Meshery Server Version: stable-v0.5.42
Platform: Kubernetes

Contributor Guides and Resources

🎨 Wireframes and designs for Meshery UI in Figma
🖥 Contributing to Meshery UI
🙋🏾🙋🏼 Questions: Layer5 Discussion Forum and Layer5 Community Slack

I would like to try this.
/assign

Sure @afzal442 go ahead 🚀

@sayantan1413 could u help me reproduce this issue? thanks

for now I am getting null on my notification section as this

I believe that this has been fixed.

I believe that this has been fixed.


buildHtml does not capture closure variables

here I used a example provided in readme.md file, modified slightly:
include karax / prelude

var lines: seq[kstring] = @[] # CASE 1

proc createDom(): VNode =
  # var lines: seq[kstring] = @[] # CASE 2

  result = buildHtml(tdiv):
    button:
      text "Say hello!"
      proc onclick(ev: Event; n: VNode) =
        lines.add "Hello simulated universe"
    for x in lines:
      tdiv:
        text x

setRenderer createDom

note that CASE 1 works fine, but in CASE 2 no line is added when you click on the button [no error happens]

This is not an issue with closure variables, I think maybe just a slight misunderstanding of how Karax works.
Here's what is happening from Karax's point of view, when you use the code in case 2:

It's told that whenever something changes, it should call createDom to get the latest vdom tree (you tell it this with the setRenderer call)
It calls createDom to get the initial vdom tree to render. createDom initializes lines as an empty seq, then returns a tree of VNodes representing the desired state of the dom. This initial tree does not include any text from lines, as it is empty
It renders that initial tree to the page, in doing so attaches an onclick handler to the button element. This handler is wrapped so that it includes a call to redraw()
You click the button, firing the onclick event handler
The event handler adds a string to the lines seq, then calls redraw()
To redraw the app, Karax calls createDom to get the latest updated version of the vdom tree
createDom initializes lines as an empty seq, then returns a tree of vnodes. This tree again does not include any text from lines, as it is empty
Karax diffs the two trees, but does not find any changes as they are the same, so nothing happens

This is not a bug in Karax, so there is no error message expected


Update Use_certificate-based_client-server_authentication

@damlys What is you reason to add this link at this place?


[UT]AbstractMetadataReportFactoryTest is not stable

[ ] I have searched the issues of this repository and believe that this is not a duplicate.
[ ] I have checked the FAQ of this repository and believe that this is not a duplicate.

Environment

Dubbo version: *
Operating System version: *
Java version: *

Steps to reproduce this issue
Run it in different system having multiple network adapters.
Expected Result
Stably successful.
Actual Result
Fail sometimes.

How about the different results?

How about the different results?

The root cause is due to the different logic on fetching the local hostname.
Please refer to #5303


add documentation about dependencies

We should have a central place in the documentation that lists all the packages on which Qiskit depends on. This info should be consolidated from the requirements.txt files of terra, ignis, aer, aqua. It should be updated with each release.
The requirements-dev.txt files list the extra dependencies needed for developers of qiskit, we can list those separately.

You want to have a doc that includes a list of requirements? That doesn't seem like something that should be written in a doc, it will go stale very quickly. It's also a really big list (mostly because of aqua):
qiskit-terra==0.8.0                Software for developing quantum computing programs
├── jsonschema<2.7,>=2.6           An implementation of JSON Schema validation for Python
├── marshmallow-polyfield<4,>=3.2  An unofficial extension to Marshmallow to allow for polymorphic fields
│   └── marshmallow>=2.0.0         A lightweight library for converting complex datatypes to and from native Python datatypes.
├── marshmallow<3,>=2.17.0         A lightweight library for converting complex datatypes to and from native Python datatypes.
├── networkx>=2.2                  Python package for creating and manipulating graphs and networks
│   └── decorator>=4.3.0           Better living through Python with decorators
├── numpy>=1.13                    NumPy is the fundamental package for array computing with Python.
├── pillow>=4.2.1                  Python Imaging Library (Fork)
├── ply>=3.10                      Python Lex & Yacc
├── psutil>=5                      Cross-platform lib for process and system monitoring in Python.
├── pylatexenc>=1.4                Python library for encoding unicode to latex and for parsing LaTeX to generate unicode text
├── scipy>=1.0                     SciPy: Scientific Library for Python
│   └── numpy>=1.8.2               NumPy is the fundamental package for array computing with Python.
└── sympy>=1.3                     Computer algebra system (CAS) in Python
    └── mpmath>=0.19               Python library for arbitrary-precision floating-point arithmetic

qiskit-aer==0.2.0  Qiskit Aer - High performance simulators for Qiskit
└── numpy>=1.13    NumPy is the fundamental package for array computing with Python.

qiskit-ibmq-provider==0.2.1            Qiskit provider for accessing the quantum devices and simulators at IBMQ
├── qiskit-terra>=0.8.0                Software for developing quantum computing programs
├── requests-ntlm>=1.1.0               This package allows for HTTP NTLM authentication using the requests library.
│   ├── cryptography>=1.3              cryptography is a package which provides cryptographic recipes and primitives to Python developers.
│   │   ├── asn1crypto>=0.21.0         Fast ASN.1 parser and serializer with definitions for private keys, public keys, certificates, CRL, OCSP, CMS, PKCS#3, PKCS#7, PKCS#8, PKCS#12, PKCS#5, X.509 and TSP
│   │   ├── cffi!=1.11.3,>=1.8         Foreign Function Interface for Python calling C code.
│   │   │   └── pycparser              C parser in Python
│   │   └── six>=1.4.1                 Python 2 and 3 compatibility utilities
│   ├── ntlm-auth>=1.0.2               Creates NTLM authentication structures
│   └── requests>=2.0.0                Python HTTP for Humans.
│       ├── certifi>=2017.4.17         Python package for providing Mozilla's CA Bundle.
│       ├── chardet<3.1.0,>=3.0.2      Universal encoding detector for Python 2 and 3
│       ├── idna<2.9,>=2.5             Internationalized Domain Names in Applications (IDNA)
│       └── urllib3<1.25,>=1.21.1      HTTP library with thread-safe connection pooling, file post, and more.
├── requests>=2.19                     Python HTTP for Humans.
│   ├── certifi>=2017.4.17             Python package for providing Mozilla's CA Bundle.
│   ├── chardet<3.1.0,>=3.0.2          Universal encoding detector for Python 2 and 3
│   ├── idna<2.9,>=2.5                 Internationalized Domain Names in Applications (IDNA)
│   └── urllib3<1.25,>=1.21.1          HTTP library with thread-safe connection pooling, file post, and more.
└── websockets<8,>=7                   An implementation of the WebSocket Protocol (RFC 6455 & 7692)

qiskit-ignis==0.1.1                    Qiskit tools for quantum information science
├── numpy>=1.13                        NumPy is the fundamental package for array computing with Python.
├── qiskit-terra>=0.8.0                Software for developing quantum computing programs
└── scipy!=0.19.1,>=0.19               SciPy: Scientific Library for Python
    └── numpy>=1.8.2                   NumPy is the fundamental package for array computing with Python.

qiskit-aqua==0.5.0                         Qiskit Aqua: An extensible library of quantum computing algorithms
├── cvxopt                                 Convex optimization package
├── dlx                                    Implementation of Donald Knuth's Dancing Links algorithm.
├── docplex                                The IBM Decision Optimization CPLEX Modeling for Python
│   ├── docloud>=1.0.315                   The IBM Decision Optimization on Cloud Python client
│   │   ├── requests                       Python HTTP for Humans.
│   │   │   ├── certifi>=2017.4.17         Python package for providing Mozilla's CA Bundle.
│   │   │   ├── chardet<3.1.0,>=3.0.2      Universal encoding detector for Python 2 and 3
│   │   │   ├── idna<2.9,>=2.5             Internationalized Domain Names in Applications (IDNA)
│   │   │   └── urllib3<1.25,>=1.21.1      HTTP library with thread-safe connection pooling, file post, and more.
│   │   └── six                            Python 2 and 3 compatibility utilities
│   ├── requests                           Python HTTP for Humans.
│   │   ├── certifi>=2017.4.17             Python package for providing Mozilla's CA Bundle.
│   │   ├── chardet<3.1.0,>=3.0.2          Universal encoding detector for Python 2 and 3
│   │   ├── idna<2.9,>=2.5                 Internationalized Domain Names in Applications (IDNA)
│   │   └── urllib3<1.25,>=1.21.1          HTTP library with thread-safe connection pooling, file post, and more.
│   └── six                                Python 2 and 3 compatibility utilities
├── fastdtw                                Dynamic Time Warping (DTW) algorithm with an O(N) time and memory complexity.
│   └── numpy                              NumPy is the fundamental package for array computing with Python.
├── jsonschema<2.7,>=2.6                   An implementation of JSON Schema validation for Python
├── numpy>=1.13                            NumPy is the fundamental package for array computing with Python.
├── psutil>=5                              Cross-platform lib for process and system monitoring in Python.
├── pyeda                                  Python Electronic Design Automation
├── qiskit-ignis<0.2,>=0.1.0               Qiskit tools for quantum information science
├── qiskit-terra<0.9,>=0.8.0               Software for developing quantum computing programs
├── quandl                                 Package for quandl API access
│   ├── inflection>=0.3.1                  A port of Ruby on Rails inflector to Python
│   ├── more-itertools<=5.0.0              More routines for operating on iterables, beyond itertools
│   │   └── six<2.0.0,>=1.0.0              Python 2 and 3 compatibility utilities
│   ├── ndg-httpsclient                    Provides enhanced HTTPS support for httplib and urllib2 using PyOpenSSL
│   │   ├── PyOpenSSL                      Python wrapper module around the OpenSSL library
│   │   │   ├── cryptography>=2.3          cryptography is a package which provides cryptographic recipes and primitives to Python developers.
│   │   │   │   ├── asn1crypto>=0.21.0     Fast ASN.1 parser and serializer with definitions for private keys, public keys, certificates, CRL, OCSP, CMS, PKCS#3, PKCS#7, PKCS#8, PKCS#12, PKCS#5, X.509 and TSP
│   │   │   │   ├── cffi!=1.11.3,>=1.8     Foreign Function Interface for Python calling C code.
│   │   │   │   │   └── pycparser          C parser in Python
│   │   │   │   └── six>=1.4.1             Python 2 and 3 compatibility utilities
│   │   │   └── six>=1.5.2                 Python 2 and 3 compatibility utilities
│   │   └── pyasn1>=0.1.1                  ASN.1 types and codecs
│   ├── numpy>=1.8                         NumPy is the fundamental package for array computing with Python.
│   ├── pandas>=0.14                       Powerful data structures for data analysis, time series, and statistics
│   │   ├── numpy>=1.12.0                  NumPy is the fundamental package for array computing with Python.
│   │   ├── python-dateutil>=2.5.0         Extensions to the standard Python datetime module
│   │   │   └── six>=1.5                   Python 2 and 3 compatibility utilities
│   │   └── pytz>=2011k                    World timezone definitions, modern and historical
│   ├── pyOpenSSL                          Python wrapper module around the OpenSSL library
│   │   ├── cryptography>=2.3              cryptography is a package which provides cryptographic recipes and primitives to Python developers.
│   │   │   ├── asn1crypto>=0.21.0         Fast ASN.1 parser and serializer with definitions for private keys, public keys, certificates, CRL, OCSP, CMS, PKCS#3, PKCS#7, PKCS#8, PKCS#12, PKCS#5, X.509 and TSP
│   │   │   ├── cffi!=1.11.3,>=1.8         Foreign Function Interface for Python calling C code.
│   │   │   │   └── pycparser              C parser in Python
│   │   │   └── six>=1.4.1                 Python 2 and 3 compatibility utilities
│   │   └── six>=1.5.2                     Python 2 and 3 compatibility utilities
│   ├── pyasn1                             ASN.1 types and codecs
│   ├── python-dateutil                    Extensions to the standard Python datetime module
│   │   └── six>=1.5                       Python 2 and 3 compatibility utilities
│   ├── requests>=2.7.0                    Python HTTP for Humans.
│   │   ├── certifi>=2017.4.17             Python package for providing Mozilla's CA Bundle.
│   │   ├── chardet<3.1.0,>=3.0.2          Universal encoding detector for Python 2 and 3
│   │   ├── idna<2.9,>=2.5                 Internationalized Domain Names in Applications (IDNA)
│   │   └── urllib3<1.25,>=1.21.1          HTTP library with thread-safe connection pooling, file post, and more.
│   └── six                                Python 2 and 3 compatibility utilities
├── scikit-learn>=0.20.0                   A set of python modules for machine learning and data mining
│   ├── joblib>=0.11                       Lightweight pipelining: using Python functions as pipeline jobs.
│   ├── numpy>=1.11.0                      NumPy is the fundamental package for array computing with Python.
│   └── scipy>=0.17.0                      SciPy: Scientific Library for Python
│       └── numpy>=1.8.2                   NumPy is the fundamental package for array computing with Python.
├── scipy!=0.19.1,>=0.19                   SciPy: Scientific Library for Python
│   └── numpy>=1.8.2                       NumPy is the fundamental package for array computing with Python.
├── setuptools>=40.1.0                     Easily download, build, install, upgrade, and uninstall Python packages
├── sympy>=1.3                             Computer algebra system (CAS) in Python
│   └── mpmath>=0.19                       Python library for arbitrary-precision floating-point arithmetic
└── torch                                  Tensors and Dynamic neural networks in Python with strong GPU acceleration
    └── numpy                              NumPy is the fundamental package for array computing with Python.

There are overlaps but I only removed the qiskit sub-dependencies when there is a cross project dependency (ie ignis depends on terra). It's also worth pointing out that pip still doesn't have a dep solver so overlaps are a potential point of concern with a list this big.

Yeah you're right. This is going to be hard to maintain. This was an external request, but I now agree that the docs aren't the right place. Something like pydeptree might be an option


Ugly URLs with override for HTML eat taxonomy term pages

When using ugly URLs, but overriding that for HTML, hugo doesn't create any output for taxonomy term pages. See https://github.com/factopolis/factopolis/tree/uglyurls for an example (claims/index.html and groups/index.html, which are the taxonomy term pages, are missing).

See https://github.com/factopolis/factopolis/tree/uglyurls for an example

There are no uglyURL config in your example.

On Thu, 2017-08-24 at 23:04 +0000, Bjørn Erik Pedersen wrote:

See https://github.com/factopolis/factopolis/tree/uglyurls for an
example

There are no uglyURL config in your example.

Are you looking at the "uglyurls" branch? That's what I linked to, but
I definitely should have been more clear about it since it's pretty
subtle on GitHub. Sorry about that :(
Lines 9 thru 11 of config.toml in that branch are
uglyurls = true
[outputFormats.HTML]
    nougly = true

If you are looking at the uglyurls branch, I'm confused… what are you
expecting to see?


Update INSTALL.md

Update list of dependencies to successfully build on Fedora 38
Checklist

[x] Language changes are updated in man/adoc/bpftrace.adoc and if needed in docs/reference_guide.md
[x] User-visible and non-trivial changes updated in CHANGELOG.md
[x] The new behaviour is covered by tests

Interesting sidenote...
build-libs.sh reports:
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.13")
-- Found LLVM: /usr/include 16.0.4 (Use LLVM_ROOT envronment variable for another version of LLVM)
-- Found BISON: /usr/bin/bison (found version "3.8.2")
-- Found FLEX: /usr/bin/flex (found version "2.6.4")
-- Found LibElf: /usr/lib64/libelf.so
-- Performing Test ELF_GETSHDRSTRNDX
-- Performing Test ELF_GETSHDRSTRNDX - Success
-- Found LibDebuginfod: /usr/lib64/libdebuginfod.so
-- Found LibLzma: /usr/lib64/liblzma.so
While the CMake for bpftrace reports:
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.13")
-- Found LibBcc: /home/wcunning/projects/bpf/bpftrace/build/build-libs/lib64/libbcc.a
-- Found LibBpf: /home/wcunning/projects/bpf/bpftrace/build/build-libs/lib64/libbpf.a
-- Found LibElf: /usr/lib64/libelf.so
-- Please install the liblzma package (missing: LIBLZMA_LIBRARIES)
-- Please install the libz package (missing: LIBZ_LIBRARIES)
Why can't this cmake find liblzma or libz? This is especially weird since the BCC cmake has no trouble finding the correct libraries.

What cmake flags are you configuring bpftrace with?
In general I've found build-libs.sh to be difficult to make work across distros. I am currently working on nix-ifying the CI so that we no longer need to use build-libs.sh. (See https://github.com/iovisor/bpftrace/blob/master/docs/nix.md). I am quite optimistic nix flakes will solve a number of pain points in this area.

@danobi -- I followed the Fedora build instructions exactly, so just -DCMAKE_BUILD_TYPE=Release.

Ah, that is b/c bpftrace has custom cmake modules for locating liblzma and libz:

https://github.com/iovisor/bpftrace/blob/446109acdef586fb0f0b5d40196dedb946b613ed/cmake/FindLibLzma.cmake#L9
https://github.com/iovisor/bpftrace/blob/446109acdef586fb0f0b5d40196dedb946b613ed/cmake/FindLibZ.cmake#L9

It looks like liblzma and libz and primarily used from a static build context:
https://github.com/iovisor/bpftrace/blob/446109acdef586fb0f0b5d40196dedb946b613ed/cmake/FindLibBcc.cmake#L99
And all that leads back to build-libs.sh and how we are trying to statically link against bcc and libbpf. Which again the nix work should address.

Awesome, sounds like you have a plan to make the whole thing more robust. Would you like me to do a test build with that system? Is it ready for that kind of test?

You can if you'd like (see https://github.com/iovisor/bpftrace/blob/master/docs/nix.md#build-bpftrace ). But I am fairly confident it will work on your system (the power of nix). The next step on my side is to hook up the nix build to CI.


upgrade to grpc-gateway/v2

upgrade protoc to 3.13
upgrade grpc-gateway to v2.0.1
use protocomp.Transform() for cmd.Diff with proto
&runtime.JSONPb{} in v2/runtime defaults to the previously called in v1 EmitDefaults
use of protocmp.IgnoreFields(proto.MessageV2(&pb.Note{}), "create_time", "update_time"), instead of cmd.Path approach
adopted v1 generate.go to work with higher tools
adopted v1beta1 generate.go to work with higher tools
adopted v1alpha to work with higher version tools, and to use openapi2 annotations instead of swagger (dues to grpc-gateway/v2)
changed go/v1/{note,occurrences} and its tests to return the proper (pb.X, error) due to *resp = *n resulting in ssignment copies lock value to req (as google.golang.org/protobuf/internal/impl.MessageState contains sync.Mutex)
updated circleci config to use Golang:1.14

FYI: @aysylu @wkozlik

Wow! That is quite a bit of work you did. I will look it over later this week.

@wkozlik it only looks big, because of the autogenerated files
the generate.go for the  protoc/swagger have given a small headache though (especially for the v1)... but the one committed now (without relative paths) is functional.

quite busy times towards the end of year...
@wkozlik got any chance so far to zoom in to my changes?

Busy as well. I will try to find time to review it today or tomorrow.


Bigtable: 14. Implement Bulk Mutations

Please note that this is only the initial implementation. There will be follow up PRs for:

once a new version of gax is cut, I will move retries behind th BatchingCallable
Once there is flushing implementation, it will be swapped in

rebased


How to access tink cli after deploying the tinkerbell stack on k8s?

I have install the tink stack helm on k8s. But I dont see any pod/deployment created for tink-cli.

Expected Behaviour
Maybe there should be a seperate pod created for access to tink-cli
OR Do I need to install tink-cli some other way?
Environment

Ubuntu 18.04 + K3s

Running tinkerbell stack on k8s; deployed via tink stack helm chart

Hi @rishi97. tink-cli has been obsolete and removed from the code base for quite some time. You interact with your data via kubectl. E.g. kubectl get hardware.


interaction_in_progress: Interaction is currently in progress.

I'm attempting to use the loginRedirect. When calling the function I receive the following error:
interaction_in_progress: Interaction is currently in progress. Please ensure that this interaction has been completed before calling an interactive API.
Code Path:
export const App = () =>
{
  const [auth] = useState(new AuthStore());

  useEffect(() => {initializeIcons();}, [])

  try
  {
    auth.msalClient.loginRedirect();
  }
  catch(err)
  {
    console.log(err)
  }
  ....
  }

AuthStore is a simple class wrapper around the msalClient and configuration objects, as well as the register call for the redirect callback.
AuthStore.tsx
import * as msal from "@azure/msal-browser";

export class AuthStore
{
    accountId?: string;
    idToken?: string;
    msalClient: msal.PublicClientApplication = new msal.PublicClientApplication(
        {
            auth:
            {
                clientId: '...',
                authority: 'https://login.microsoftonline.com/...'
            }
        }
    );

    constructor()
    {
        // Register Callbacks for Redirect flow
        this.msalClient.handleRedirectPromise()
            .then(this.handleResponse)
            .catch((error) =>
            {
                // TODO: Logging!
                console.log("Handle Redir: " + error);
            }
        );
    }

    handleResponse(resp: msal.AuthenticationResult | null)
    {
        if (resp !== null && resp.account !== null)
        {
            this.accountId = resp.account.homeAccountId;
            this.idToken = resp.idToken;
        }
        else
        {
            const currentAccounts = this.msalClient.getAllAccounts();

            if (!currentAccounts || currentAccounts.length < 1)
            {
                this.msalClient.loginRedirect({scopes: ["User.Read"]});
            }
        }
    }
}

The response from MSAL is null, this causes the error case in AuthStore to re-call the login redirect, despite having already been redirected.

The response from MSAL is null, this causes the error case in AuthStore to re-call the login redirect, despite having already been redirected.

@tnorling Can you help here?

@tnorling Can you help here?

@ignatzami I would recommend putting your login call into a useEffect hook. You don't want loginRedirect to be invoked on every render. Additionally, I would not recommend calling loginRedirect inside handleResponse as this can cause loops.
In case you are not yet aware we also have a react wrapper which solves a lot of these problems for you and should be mostly plug and play. Check it out here

@ignatzami I would recommend putting your login call into a useEffect hook. You don't want loginRedirect to be invoked on every render. Additionally, I would not recommend calling loginRedirect inside handleResponse as this can cause loops.
In case you are not yet aware we also have a react wrapper which solves a lot of these problems for you and should be mostly plug and play. Check it out here

@tnorling , the issue that @ignatzami  describes seems to be exactly what another dev describes in issue #2149. I'm experiencing the issue in this thread by using the repro steps in the one I referenced. It seems to have been fixed in version 2.0.2, but it appeared again in 2.10.0. I walked through upgrading my msal-browser version-by-version from 2.0.1 to 2.10.0. Here is what I found:
2.0.1: this issue exists
2.0.1 - 2.9.0: this is not an issue
2.10.0: this issue reappears
Here is a very simple React component that demonstrates this issue. I realize there is a React wrapper being built, but it's in alpha and I hope to have a production-ready app prior to the library being officially released.
`
import React from 'react';
import { PublicClientApplication } from '@azure/msal-browser'; //-- https://www.npmjs.com/package/@azure/msal-browser
const Login = () => {
const msalPublicClientApp = new PublicClientApplication(APP_SETTINGS.msalConfig);
const handleLogin = () => {
	msalPublicClientApp.loginRedirect();
};

return (
	<>

Login

		<button onClick={handleLogin}>Log In</button>
	</>
);

};
export default Login;
`

@tnorling , the issue that @ignatzami  describes seems to be exactly what another dev describes in issue #2149. I'm experiencing the issue in this thread by using the repro steps in the one I referenced. It seems to have been fixed in version 2.0.2, but it appeared again in 2.10.0. I walked through upgrading my msal-browser version-by-version from 2.0.1 to 2.10.0. Here is what I found:
2.0.1: this issue exists
2.0.1 - 2.9.0: this is not an issue
2.10.0: this issue reappears
Here is a very simple React component that demonstrates this issue. I realize there is a React wrapper being built, but it's in alpha and I hope to have a production-ready app prior to the library being officially released.
`
import React from 'react';
import { PublicClientApplication } from '@azure/msal-browser'; //-- https://www.npmjs.com/package/@azure/msal-browser
const Login = () => {
const msalPublicClientApp = new PublicClientApplication(APP_SETTINGS.msalConfig);
const handleLogin = () => {
	msalPublicClientApp.loginRedirect();
};

return (
	<>

Login

		<button onClick={handleLogin}>Log In</button>
	</>
);

};
export default Login;
`

@AGDevX Thanks for the information. I've identified the source of the regression and will make sure a fix is included in the next release.

@AGDevX Thanks for the information. I've identified the source of the regression and will make sure a fix is included in the next release.

@tnorling this is an issue that also happens in Angular.

All<EMAIL_ADDRESS>has just been released, which contains the fix for this bug. Closing, please open a new issue if you have further questions. Thanks!

@tnorling I'm still able to reproduce this using loginRedirect with<EMAIL_ADDRESS>

@fjuriolli Can you please open a new issue and include your config and steps to reproduce? Thanks!


Misleading documentation about uploading gradle plugins & manual approval

On this page there's a section about manual review: https://plugins.gradle.org/docs/submit

Some plugins require manual acceptance by the Plugin Portal maintainers, and will not be made available immediately when they are published. This is only the case when you have specified a custom “group ID” for your plugin artifact. The plugin publishing plugin will tell you if your plugin is pending acceptance when you publish.

Also here: https://plugins.gradle.org/docs/publish-plugin

If you would like to have a non-prefixed coordinate group, you can do so by setting the group in a mavenCoordinates block. Note that non-prefixed groups require manual approval the first time one is published. See the Full Example below.

Documentation says that if I do not change group id, plugin will be enabled automatically. However, it's not happening as policy was changed in 2020, all plugins are reviewed manually now: https://discuss.gradle.org/t/plugin-pending-approval-questions/35017
These sections should be removed from documentation, it should say that all new plugins are manually reviewed.
P.S. Raising this issue because I spent some time figuring out why my plugin wasn't enabled right away. Most likely somebody else also wasted time due to this issue.

Issue moved to gradle/plugin-portal-requests #94 via ZenHub


Fix: check participant status to make sure the participant is onboarded

Currently, the crawler will fetch catalog data from participants that have a DENIED status.
What this PR changes/adds
Check participant status
Why it does that
You only want to crawl participants that are ONBOARDED

There might be a valid case for listing participants that are not in the ONBOARDED state. As the onboarding is an asynchronous operation, it is I think currently the only way to check your onboarding status.
But anyhow, as the status is part of the Dto, I think the client should check its value before accepting a participant as valid. Even when the RegistrationService would commit to only list ONBOARDED participants.

There might be a valid case for listing participants that are not in the ONBOARDED state. As the onboarding is an asynchronous operation, it is I think currently the only way to check your onboarding status.

A participant can get their own status using the get participant API operation (also available in CLI).

But anyhow, as the status is part of the Dto, I think the client should check its value before accepting a participant as valid. Even when the RegistrationService would commit to only list ONBOARDED participants.

Per our contributing guide, please create an issue to discuss the idea, a PR is not the best way to achieve this.


Simplify specs

@charlierudolph @allewun
Some specs create commits that aren't really necessary. I have gone through all feature specs and removed the ones that I think are not adding to the feature spec.

It seems necessary for some specs. I have another PR that introduces it properly.

bump

All comments addressed

:ship:


fix: PermissionError when using tesseract_ocr_cli_model

Make sure that the tesseract_ocr_cli_model.py does not open the png image file twice (before it was opened once with tempfile.NamedTemporaryFile  and again by passing the file name rather than the file object itself to high_res_image.save;
Ensure that _run_tesseract is executed after the file is no longer open by python. This otherwise results in a "PermissionError: [Errno 13] Permission denied" error on Windows.

Thanks @gaspardpetit for this fix. Here a few notes

Please review the test failures. I think you are missing the pre-commit styling, which could be fixed with pre-commit run --all-files
I was reading a bit more about the issue in the python docs

it would be better to use delete_on_close=False and avoid the extra try/finally. Can you please give it a try?

Thank you for reviewing this pull request. I have fixed the import order according the the pre-commit linter.
I disagree with using delete_on_close=False (with delete=True and without the try/finally blocks):
With delete_on_close=False, temporary files will survive until the script terminates. On a script being used to convert a large number of pdf files, each containing potentially hundreds of pages, this would resulting thousand of temporary image files being deleted only on termination. Within containers with limited storage, this could be a huge disadvantage. It may also make the termination of the script appear to hang and possibly timeout in some scenarios, leading to the process being killed and the files leaking.
Let me know if you would still prefer to use delete_on_close=False and I will amend my pull request accordingly.

Note: the CI is currently reporting

one of your commit is missing the DCO sign-off. Can you please fix it with git rebase HEAD~15 --signoff?
the diff currently looks weird. I'm wondering if something went wrong in merging with the latest main branch.

Hi! Thank you for reviewing this.
You are right, I might have merged incorrectly - I will probably end up submitting again from a clean branch.
Just to confirm that I understand correctly - using delete_on_close will also bump the python requirements to 3.12 and I think the docling currently supports 3.9. Do we want that?

using delete_on_close will also bump the python requirements to 3.12

Good catch. Sorry, I didn't notice it before. Then it is definitely a no-go.
Then I fully support your solution with delete=False.

Brilliant, I'll resubmit as a clean branch in a couple of hours then, thank !

Replaced by clean merge request https://github.com/DS4SD/docling/pull/496


[SPARK-47316][SQL] Fix TimestampNTZ in Postgres Array

What changes were proposed in this pull request?

For Postgres, TimestampNTZ works well for plain TimestampNTZ types but not for nested ones, typically for now: array.
This PR makes it work.
Why are the changes needed?

feature and data consistency
Does this PR introduce any user-facing change?

preferTimestampNTZ jdbc option now works for array([timestamp|time|...]) in postgres
How was this patch tested?

new tests
Was this patch authored or co-authored using generative AI tooling?
no

cc @cloud-fan

Merged to master.
Thank you very much @cloud-fan @dongjoon-hyun


Hosting in a .Net Core 2.1 Windows Service on Windows Server 2016

I've followed the instructions here "https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/windows-service?view=aspnetcore-2.1" to the letter and my .Net Core 2.1 Windows Service works perfectly when I create and start the service on my Windows 10 Pro machine.
The problem arises when I deploy on to our Windows Server 2016 Datacenter OS.  It creates the service as expected, but when I go to start the service I immediately get the following error:
The service did not respond to the start or control request in a timely fashion.
To emphasize, the service instantly fails to start with the above error.
If I try to run the exe itself I get the following error:
Error:
An assembly specified in the application dependencies manifest (TestService.deps.json) was not found:
package: 'Microsoft.AspNetCore.Hosting.WindowsServices', version: '2.1.1'
path: 'lib/netstandard2.0/Microsoft.AspNetCore.Hosting.WindowsServices.dll'
I suspect the two error are interlinked somehow.
I'm fairly convinced that the code is not being hit in Program.Main as I've added logging in the first line of code to be ran but nothing happens.  I've tried adding the ServicesPipeTimeout registry entry to extend the time given for the service to start but I still get an instant fail.  I've ensured that the server has the required versions of .Net Core installed and they are:

Microsoft. NET Core 2.1.1 - Windows Server Hosting - v2.1.13007.0
Microsoft .NET Core Runtime - 2.1.1 (x64) - <IP_ADDRESS>606

I've also ensured the IIS is enabled on the system.
I've been struggling with this for 2 days and cannot find the answers to resolve these issues anywhere.
Can this document be updated with instructions on how to deploy to the server to ensure all the pre-requisites are covered so that these issues don't occur?
I'd really like an answer to this, thanks in advance :)

Dup of https://github.com/aspnet/Docs/issues/7373


Chart with arc mark (pie chart) using Selenium backend errors out

I am using 'master' and I fail to create PNG exports using Selenium using example data 'https://vega.github.io/vega-lite/examples/layer_arc_label.html'.
Note that the whole setup works perfectly for any other type of chart.
I am invoking altair_saver.save method with the following options {"method": "selenium", "scale_factor": 2.0}, and forcing output to PNG using a filename with a PNG extension, and I get the following error:
Traceback (most recent call last):
  File "./scripts/generate_chart.py", line 231, in <module>
    sys.exit(main())
  File "./scripts/generate_chart.py", line 145, in main
    package_name.save_chart(data_def, args.output_directory, "name")
  File "/Users/user/Workspace/package-name/package_name/__init__.py", line 49, in save_chart
    altair_saver.save(chart, **option_set)
  File "/Users/user/.virtualenvs/venv36/src/altair-saver/altair_saver/_core.py", line 172, in save
    return saver.save(fp=fp, fmt=fmt)
  File "/Users/user/.virtualenvs/venv36/src/altair-saver/altair_saver/savers/_saver.py", line 121, in save
    content = self._serialize(fmt, "save")
  File "/Users/user/.virtualenvs/venv36/src/altair-saver/altair_saver/savers/_selenium.py", line 284, in _serialize
    out = self._extract(fmt)
  File "/Users/user/.virtualenvs/venv36/src/altair-saver/altair_saver/savers/_selenium.py", line 280, in _extract
    raise JavascriptError(result["error"])
altair_saver.savers._selenium.JavascriptError: TypeError: Cannot read property 'orient' of undefined

The error pops up only with the selenium backend (node backend works alright, but resolution of the image if quite sub-optimal, hence the choice of the selenium backend)
I am using the following dependencies (requirements.txt file):
-e git+https://github.com/altair-viz/altair.git@81b40755c51b5821b078d36702b3cccadfc84071#egg=altair
-e git+https://github.com/altair-viz/altair_saver.git@3c8e1102776a7acd2c971354190a283c394d1cfd#egg=altair-saver
altair-data-server==0.4.1
altair-viewer==0.3.0
attrs==20.3.0
entrypoints==0.3
importlib-metadata==3.3.0
Jinja2==2.11.2
jsonschema==3.2.0
MarkupSafe==1.1.1
numpy==1.19.5
pandas==1.1.5
portpicker==1.3.1
pyrsistent==0.17.3
python-dateutil==2.8.1
pytz==2020.5
selenium==3.141.0
six==1.15.0
toolz==0.11.1
tornado==6.1
typing-extensions==<IP_ADDRESS>
urllib3==1.26.2
vega-datasets==0.9.0
vegascope==1.0.14
zipp==3.4.0

Can you share the code you are using? The bundled vega libraries do not support pie charts (radial marks were added in a later release) so I wouldn't expect it to work unless you're using JS libraries loaded from CDN.

Can you share the code you are using? The bundled vega libraries do not support pie charts (radial marks were added in a later release) so I wouldn't expect it to work unless you're using JS libraries loaded from CDN.

I am passing to altair_saver.save the Vega-Lite JSON definition below as python dictionary with the options I mentioned in the previous post. Nothing more than that. Let me know if you need more info.
{
  "$schema": "https://vega.github.io/schema/vega-lite/v4.json",
  "description": "A simple pie chart with labels.",
  "data": {
    "values": [
      {"category": "a", "value": 4},
      {"category": "b", "value": 6},
      {"category": "c", "value": 10},
      {"category": "d", "value": 3},
      {"category": "e", "value": 7},
      {"category": "f", "value": 8}
    ]
  },
  "encoding": {
    "theta": {"field": "value", "type": "quantitative", "stack": true},
    "color": {"field": "category", "type": "nominal", "legend": null}
  },
  "layer": [{
    "mark": {"type": "arc", "outerRadius": 80}
  }, {
    "mark": {"type": "text", "radius": 90},
    "encoding": {
      "text": {"field": "category", "type": "nominal"}
    }
  }],
  "view": {"stroke": null}
}

I am passing to altair_saver.save the Vega-Lite JSON definition below as python dictionary with the options I mentioned in the previous post. Nothing more than that. Let me know if you need more info.
{
  "$schema": "https://vega.github.io/schema/vega-lite/v4.json",
  "description": "A simple pie chart with labels.",
  "data": {
    "values": [
      {"category": "a", "value": 4},
      {"category": "b", "value": 6},
      {"category": "c", "value": 10},
      {"category": "d", "value": 3},
      {"category": "e", "value": 7},
      {"category": "f", "value": 8}
    ]
  },
  "encoding": {
    "theta": {"field": "value", "type": "quantitative", "stack": true},
    "color": {"field": "category", "type": "nominal", "legend": null}
  },
  "layer": [{
    "mark": {"type": "arc", "outerRadius": 80}
  }, {
    "mark": {"type": "text", "radius": 90},
    "encoding": {
      "text": {"field": "category", "type": "nominal"}
    }
  }],
  "view": {"stroke": null}
}

As I mentioned, arc marks are not supported by the bundled javascript libraries. You'll have to save with offline=False and specify the versions of vega & vega-lite that you'd like to be used. The relevant arguments can be seen here: https://github.com/altair-viz/altair_saver/blob/3c8e1102776a7acd2c971354190a283c394d1cfd/altair_saver/savers/_selenium.py#L163-L174

As I mentioned, arc marks are not supported by the bundled javascript libraries. You'll have to save with offline=False and specify the versions of vega & vega-lite that you'd like to be used. The relevant arguments can be seen here: https://github.com/altair-viz/altair_saver/blob/3c8e1102776a7acd2c971354190a283c394d1cfd/altair_saver/savers/_selenium.py#L163-L174

Thanks @jakevdp for the snippet, that makes sense, feel free to close this.
By the way, is there any reason for the lower pace of altair/altair_saver releases?
I see there are many little issues that require using the package in master and not a proper release.

Thanks @jakevdp for the snippet, that makes sense, feel free to close this.
By the way, is there any reason for the lower pace of altair/altair_saver releases?
I see there are many little issues that require using the package in master and not a proper release.

It's mainly that Vega-Lite 4.9 made some significant changes that will require some significant adaption of Altair, and that happened to coincide with the time where a global pandemic caused me to lose childcare & non-home schooling for my kids.

It's mainly that Vega-Lite 4.9 made some significant changes that will require some significant adaption of Altair, and that happened to coincide with the time where a global pandemic caused me to lose childcare & non-home schooling for my kids.

Since Altair 5.2, the functionality of Altair Saver is now available in Altair via the vl-convert package. Most of the functionality has been available since 5.0, and the main addition in 5.2 was PDF export. See the docs on how to save charts for more details
We are going to archive this repo, so I'm closing all the open issues and PRs before doing so. Try out the new options for saving charts mentioned above and if you run into issues, please open an issue directly in the altair or vl-convert repo.


Comma-separate large numbers

It would be great if large numbers were comma-separated anywhere they appear (ie MPS).

+1, but in other words, localize and delimit numbers. Commas are not universal.
Thanks!


paperless: Fix a few papercuts

Description of changes
Resolves a few papercuts I experienced while setting up paperless.

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

There is still the following error visible in the NixOS tests.
vm-test-run-paperless> subtest: Documents are consumed
vm-test-run-paperless> machine: waiting for success: (($(curl -u admin:admin -fs localhost:28981/api/documents/ | jq .count) == 2))
vm-test-run-paperless> machine # [   29.821756] paperless-ngx[1071]: [2022-09-04 12:21:28,810] [INFO] [paperless.consumer] Consuming doc.png
vm-test-run-paperless> machine # [   30.666513] paperless-ngx[1071]: [2022-09-04 12:21:29,655] [WARNING] [paperless.parsing.tesseract] Error while getting DPI from image /var/lib/paperless'
vm-test-run-paperless> machine # [   33.225031] paperless-ngx[1071]: [2022-09-04 12:21:32,213] [WARNING] [ocrmypdf._exec.tesseract] [tesseract] Warning: Invalid resolution 48 dpi. Using 70.
vm-test-run-paperless> machine # [   33.226829] paperless-ngx[1071]: [2022-09-04 12:21:32,213] [WARNING] [ocrmypdf._exec.tesseract] [tesseract] Warning. Invalid resolution 48 dpi. Using 70.
vm-test-run-paperless> machine # [   33.229178] paperless-ngx[1071]: [2022-09-04 12:21:32,213] [ERROR] [ocrmypdf._exec.tesseract] [tesseract] Error during processing.
vm-test-run-paperless> machine # [   34.208424] paperless-ngx[1071]: [2022-09-04 12:21:33,197] [WARNING] [ocrmypdf._exec.tesseract] [tesseract] Warning: Invalid resolution 48 dpi. Using 70.
vm-test-run-paperless> machine # [   34.633477] paperless-ngx[1153]: convert: FailedToExecuteCommand `'gs' -sstdout=%stderr -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=5000000.
vm-test-run-paperless> machine # [   34.638537] paperless-ngx[1153]: convert: no images defined `/tmp/paperless/paperless-aid36n32/convert.webp' @ error/convert.c/ConvertImageCommand/3342.
vm-test-run-paperless> machine # [   34.656765] paperless-ngx[1071]: [2022-09-04 12:21:33,645] [WARNING] [paperless.parsing] Thumbnail generation with ImageMagick failed, falling back to g!
vm-test-run-paperless> machine # [   35.650601] paperless-ngx[1071]: [2022-09-04 12:21:34,639] [INFO] [paperless.consumer] Document 2005-10-16 doc consumption finished
vm-test-run-paperless> machine # [   35.653660] paperless-ngx[1071]: 12:21:34 [Q] INFO Process-1:1 stopped doing work
vm-test-run-paperless> machine # [   35.671751] paperless-ngx[1073]: 12:21:34 [Q] INFO Processed [doc.png]
vm-test-run-paperless> machine # [   38.267439] paperless-ngx[1070]: 12:21:37 [Q] INFO recycled worker Process-1:1
vm-test-run-paperless> machine # [   38.270700] paperless-ngx[1180]: 12:21:37 [Q] INFO Process-1:4 ready for work at 1180
vm-test-run-paperless> machine # [   38.271560] paperless-ngx[1180]: 12:21:37 [Q] INFO Process-1:4 processing [webdoc.png]
vm-test-run-paperless> machine # [   38.890814] paperless-ngx[1180]: [2022-09-04 12:21:37,879] [INFO] [paperless.consumer] Consuming webdoc.png
vm-test-run-paperless> machine # [   39.224800] paperless-ngx[1180]: [2022-09-04 12:21:38,213] [WARNING] [paperless.parsing.tesseract] Error while getting DPI from image /tmp/paperless/pap'
vm-test-run-paperless> machine # [   40.001992] paperless-ngx[1180]: [2022-09-04 12:21:38,990] [WARNING] [ocrmypdf._exec.tesseract] [tesseract] Warning: Invalid resolution 48 dpi. Using 70.
vm-test-run-paperless> machine # [   40.002783] paperless-ngx[1180]: [2022-09-04 12:21:38,990] [WARNING] [ocrmypdf._exec.tesseract] [tesseract] Warning. Invalid resolution 48 dpi. Using 70.
vm-test-run-paperless> machine # [   40.004080] paperless-ngx[1180]: [2022-09-04 12:21:38,990] [ERROR] [ocrmypdf._exec.tesseract] [tesseract] Error during processing.
vm-test-run-paperless> machine # [   40.756717] paperless-ngx[1180]: [2022-09-04 12:21:39,745] [WARNING] [ocrmypdf._exec.tesseract] [tesseract] Warning: Invalid resolution 48 dpi. Using 70.
vm-test-run-paperless> machine # [   41.041213] paperless-ngx[1224]: convert: FailedToExecuteCommand `'gs' -sstdout=%stderr -dQUIET -dSAFER -dBATCH -dNOPAUSE -dNOPROMPT -dMaxBitmap=5000000.
vm-test-run-paperless> machine # [   41.044443] paperless-ngx[1224]: convert: no images defined `/tmp/paperless/paperless-h13cqq46/convert.webp' @ error/convert.c/ConvertImageCommand/3342.
vm-test-run-paperless> machine # [   41.066114] paperless-ngx[1180]: [2022-09-04 12:21:40,054] [WARNING] [paperless.parsing] Thumbnail generation with ImageMagick failed, falling back to g!
vm-test-run-paperless> machine # [   41.985678] paperless-ngx[1180]: [2022-09-04 12:21:40,974] [INFO] [paperless.consumer] Document 2005-10-16 webdoc consumption finished
vm-test-run-paperless> machine # [   41.988824] paperless-ngx[1180]: 12:21:40 [Q] INFO Process-1:4 stopped doing work
vm-test-run-paperless> machine # [   41.998447] paperless-ngx[1073]: 12:21:40 [Q] INFO Processed [webdoc.png]
vm-test-run-paperless> (finished: waiting for success: (($(curl -u admin:admin -fs localhost:28981/api/documents/ | jq .count) == 2)), in 14.13 seconds)

The documentation suggests that it is by default set to /usr/bin/convert and /usr/bin/gs respectively and thereby non-empty, so I guess it should be clarified if that is the case.
https://paperless-ngx.readthedocs.io/en/latest/configuration.html#binaries

That's a documentation error, I'll fix it upstream.
Besides convert and gs, there are many more binaries that are picked up via PATH.

@ofborg eval

It seems as if commit 81a17f7352bd7acc438c298f317c190d2fbdcc35 from this PR somehow broke my setup.
error: cannot coerce a set to a string

       at /nix/store/wibagkzj3knj4269x289x0jx0kcmgf5c-source/nixos/modules/services/misc/paperless.nix:27:88:

           26|   manage = let
           27|     setupEnv = lib.concatStringsSep "\n" (mapAttrsToList (name: val: "export ${name}=\"${val}\"") env);
             |                                                                                ^
           28|   in pkgs.writeShellScript "manage" ''
(use '--show-trace' to show detailed location information)

with time.timeZone = "Europe/Berlin".


Hide part pricing history

Due to the many outstanding issues with part "pricing", the "Part" detail page can take a very long time (> 10 seconds) just to load templates if it has a complex (multi-level) Bill of Materials.
Pricing is getting a major refactor in 0.7.x release.
For now, this PR adds an option to hide pricing history (and calculation of pricing data) from the part detail page.

@matmair you may want to be aware of this one. I'm (by default) disabling part pricing display as it is really killing the browser (and the server!) for any level of BOM complexity..
We will have to re-address this in 0.7.x when we refactor the pricing. Going to be a daunting task I fear.


Correctly link szip for conda-forge

track progress and success https://github.com/conda-forge/imagecodecs-feedstock/pull/82

Thank you.


feat: kubefirst.dev local domain

good call, ill do a better fix before merging. we're just tinkering with the idea, might be kind of neat!


Serde Implementation with graphql_object

I've implemented the following User structure and graphql_object :
#[derive(Serialize, Deserialize, Debug)]
pub struct User {
    #[serde(rename = "_id")]
    pub id: Option<ObjectId>,

    #[serde(default)]
    pub email: Option<String>,

    #[serde(default)]
    pub username: Option<String>,

    #[serde(default)]
    pub role: Option<String>,
}

graphql_object!(User: Context |&self| {
    field id() -> String { if let Some(ref id) = self.id { id.to_string() } else { "".into() } }
    field email() -> &str { if let Some(ref email) = self.email { email.as_str() } else { "".into() } }
    field username() -> &str { if let Some(ref username) = self.username { username.as_str() } else { "".into() } }
    field role() -> &str { if let Some(ref role) = self.role { role.as_str() } else { "".into() } }
});

I need the struct User to interact with my database MongoDB. Is it possible to translate my structure to a graphql_object using serde or an other serialization process to avoid duplicating my structure ?
Thanks for your support.

@AurelienFT I would highly recommend to keep your models for DB and GraphQL objects separately, so don't event try to write GraphQL objects into DB.
Let me explain why...
GraphQL objects are part of a GraphQL schema, which does not represent a data model, but rather a data interface for external clients. In terms of mainstream REST API, we can say that GraphQL schema is a Controller layer, not a Model/Domain layer.
This is quite obvious if you think/look about/at how GraphQL schema may evolve in time. Let's start with a simple:
#[derive(juniper::GraphQLObject)]
struct Chat {
    id: Uuid,
    members: Vec<Member>,
}

#[derive(juniper::GraphQLObject)]
struct Member {
    id: Uuid,
    name: String,
}

This schema is quite obvious and each field has trivial resolution (just returning reference to its value).
However, at some point, we want be able to query Messages of this chat. But we cannot just pull all the Messages stored in our DB and paster them into Chat struct as Vec. So, what we want is a dynamic field resolution. We want Messages being loaded from DB only if they were requested. With Juniper it can be easily done as following:
struct Chat {
    id: Uuid,
    members: Vec<Member>,
}

#[juniper::object(Context = Db)]
impl Chat {
    fn id(&self) -> Uuid { &self.id }

    fn members(&self) -> &[Member] { self.members.as_slice() }

    fn messages(
        &self,
        after: Option<Cursor>,
        first: Option<i32>,
        context: &Db,
    ) -> Result<Vec<Message>, Error> {
        Ok(context
            .load_messages_of_chat(self.id, after, first)?
            .into_iter()
            .map(Into::into)
            .collect())
    }
}

#[derive(juniper::GraphQLObject)]
struct Message {
    id: Uuid,
    text: String,
}

#[derive(juniper::GraphQLObject)]
struct Member {
    id: Uuid,
    name: String,
}

Now we won't trigger Messages loading at all in situations like this:
queries {
    chat(id: "....") {
        members,
    }
}

So, as you see, Chat has nothing common with DTOs or domain Models. It's just a queryable API interface.

So, you problem can be solved by introducing a separate structs which store the actual data and are able to ser/de for your DB and are From/Into-convertible to GraphQL objects of your schema for ergonomics.

@tyranron Thanks a lot man ! Your response is awesome.


README: Focus on local runtime (create/start/stop)

These actions need consistent tooling, and they're the sweet spot for
tools like runc.  Once you map the container spec to the filesystem,
the other actions (copy, snapshot, upload, download) can all be
handled by existing tools (e.g. POSIX's 'cp', 'btrfs subvolume
snapshot', netcat, etc.).  If existing tooling for those actions is
not sufficiently portable, new portable tools can be written, but that
should be separate from the container-specific stuff here.
I'm not sure about the intended meaning of the 'tagged' action.  Was
that intended for discoverability ("I want a container named 'Nginx'")
or auth ("Is my local /srv/nginx the same container 'Nginx' that Bob
audited?").  I think both are independent of the container runtime
though.

Sure, this change seems fine to me.

lgtm

LGTM


rect exceeds domain

Reducing the domain for a plot with Plot.rect() leads to a rectangle larger than the domain
https://observablehq.com/d/cf99f3f3d9471fc7

This is the intended behavior. You can enable the clamp scale option if you like. See also #165 for a proposed feature regarding clipping.


Silence auditing formula revisions deprecations

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[x] Have you successfully run brew tests with your changes locally?

Add a Homebrew.raise_deprecation_exceptions method as a global setting to allow raising exceptions on odeprecated even for non-developers.
Fixes #681.
Checked that this doesn't reproduce #696
CC @DomT4

Sorry for the delay. A belated 👍, can't reproduce the original problem we fell into the other day.


Calendar not updating selected date from binding

Avalonia 0.9.9
The Calendar control displays the selected date correctly when first displayed. But when when SelectedDate changes via view model binding it does not update the selected day in the UI until I interact with the control; e.g. move to the next month.
<Calendar SelectedDate="{Binding Date}"/>

Here is the UI after the bound Date is changed from 21st to the 15th. As you can see, the selected date is still 21 after the bound Date changed to 15.

Another issue is the selected day background colour makes the day number unreadable in both light and dark themes on some displays.

Works as expected with v11.


Fix ordering of properties in static spacing override classes

We incorrectly shipped static spacing override classes in the format
govuk-!-padding-static and govuk-!-margin-static, when we intended
them to be in the format govuk-!-static.
Our documentation refers to govuk-!static. We should update the classes to match,
but keep the existing classes for now as people may already be using them.

@36degrees does this match what you were expecting in terms of aliases and deprecating the old class?

@domoscargin not yet, but I have got a checklist item to do that on the issue card


test: errorcheck support for intraline errors

Now that the compiler supports reporting column position for errors, we need a mechanism in test/run.go for testing this.
I propose we use /* ERROR "foo" */ placed just before where we expect the error to be emitted, except to allow for white space in between.
E.g., for #19576, the test would be
package p

func f() int { /* ERROR "missing return" */ }

Rationale:

run.go currently only looks for // ERROR annotations, which can only occur at the end of the line, so there's no risk of collision using /* ERROR */ for column-precise error matching.

Not explicitly having to write column numbers into the errors makes them easier to maintain, just like how we don't currently have to worry about line numbers.

We emit errors at the start of tokens, so putting the error expectation before that lets us unambiguously match it to an expected column. Putting afterwards would require more complex logic for figuring out how many characters to rewind past.

Allowing whitespace after the error comment allows the code to be gofmt'd.

/cc @griesemer @ianlancetaylor @bradfitz

My main comment is that I would prefer that we not go heavily into testing column values.  gccgo uses the same testsuite, and it's inevitable that there will be some disagreements about column position, and that will force the use of GC_ERROR and GCCGO_ERROR, which makes the tests that much less readable.
I think your suggestion sounds good for cases where we do explicitly want to test column values.

The go/types test suite uses this mechanism, except that (for a marginally simpler implementation) it requires the ERROR comments to be after the offending token. @mdempsky 's suggestion is better (and I wanted to change this for go/types in the past) because as is in go/types, the ERROR comment often ends up in the middle of some construct, for instance a selector expression:
... x /* ERROR "x has no field or method f" */ .f

One way to make column testing not destroy the readability of code is to use an extra comment and leave the ERROR comments where they are. That would also make it easier to convert existing tests:
... /*1*/ x.f  // ERROR "x has no field or method f"

The number in /*1*/ could be the index of the respective ERROR comment at the end of the line. If there's no index comment, we just check the line.

You could also do something like
// ERROR TOKEN 5 "x has no field or method f"
Which would require that the error occur at the same column as the 5th token if you tokenize that line.  Hopefully all errors occur at a token, right?
It's not quite testing columns, but it is probably a lot easier to count tokens than columns and nicer than interstitial comments.

Errors are positioned at tokens because the compiler only sees tokens and their positions. But using a comment in place where the error happens is much more maintainable: We don't want to re-arrange all token numbers just because we changed the test code a little bit.

@ianlancetaylor Agreed, I think for most tests line position information is adequate.
@griesemer Using /*1*/ seems reasonable too. Perhaps less intrusive to implement too. Also has the benefit for gccgo that we could include multiple /*1*/ markers and accept the error at any of them, or explicitly annotate them like /*GC:1*/ or /*GCCGO:1*/.
@randall77 Yeah, errors are always at the start of a token. But if we're going to require test authors to explicitly record position information, I'd lean towards just writing the column number directly.

Just a drive-by post, @mdempsky sent me here. The regress test from https://go-review.googlesource.com/c/go/+/76150 could use a cleanup once we've implemented this feature.

The /* ERROR "msg" */ format can make line harder to read (subjectively):
/* ERROR "n declared and not used" */ n, /* ERROR "err declared and not used" */ err := fmt.Println(1)

Numbered comments make things better:
/*1*/ n, /*2*/ err := fmt.Println(1) // ERROR "n declared and not used" "err declared and not used"

But after you run gofmt, it will look like this:
/*1*/ n /*2*/, err := fmt.Println(1) // ERROR "n declared and not used" "err declared and not used"

I find it somewhat confusing.
If indexed comments are placed after the token, there is no such problem:
n /*1*/, err /*2*/ := fmt.Println(1) // ERROR "n declared and not used" "err declared and not used"

By the way, test for issue21317 could be written without any changes to errorcheck:
// errorcheck

package main

import "fmt"

func main() {
	n, // ERROR "n declared and not used"
		err := // ERROR "err declared and not used"
		fmt.Println(1)
}

@Quasilyte It is already the case that we do not gofmt the test directory (at least as a whole) since some tests rely on the specific layout of the code. I don't expect that to change.


add new WD

Added joeyderuiter.me

Tried to go back and do a separate pull request for each addition, but it looks like it's using the latest commit for the PR since the changes were done under a single branch. Sorry!

Hi Ben, thank you so much for your submissions, it's no problem at all.
These projects are lit, thanks again!

Very welcome, I always enjoy checking your site/blog and seeing the project evolve. Cheers!


add ContinueOnError to Automation API

Similar to what we did for Go, Python and NodeJS, add ContinueOnError to the dotnet automation api.

Changie changelog?

whoops I forgot. Added now.


Typo in Linux device enumeration

Hello,
I think there are a couple of mistakes in the enumeration of devices on Linux. There appear to be two places where each DefactoryStandardResolution frame width is incorrectly compared to the enumerated frame max_height - instead of the frame max_width.
https://github.com/kekyo/FlashCap/blob/854186a445e494548480ae0d6b607c1bf009eb57/FlashCap.Core/Devices/V4L2Devices.cs#L66
and
https://github.com/kekyo/FlashCap/blob/854186a445e494548480ae0d6b607c1bf009eb57/FlashCap.Core/Devices/V4L2Devices.cs#L80

I think there is a second issue with the filtering of the DefactoStandardResolutions.
The following lines don't seem to take operator precedence into account correctly. The modulo division has a higher order of precedence than the subtraction meaning it will happen before the subtraction. I think there should be parenthesis around the subtraction to ensure it occurs before the modulo division.
https://github.com/kekyo/FlashCap/blob/854186a445e494548480ae0d6b607c1bf009eb57/FlashCap.Core/Devices/V4L2Devices.cs#L67
and
https://github.com/kekyo/FlashCap/blob/854186a445e494548480ae0d6b607c1bf009eb57/FlashCap.Core/Devices/V4L2Devices.cs#L70
Currently I believe this will result in no characteristics being returned for devices that report a Stepwise frame size capability
(specifically this issue along with https://github.com/kekyo/FlashCap/issues/127 seems to be stopping my Raspberry PI onboard camera from being enumerated successfully)

@gplwhite I will review it in a couple of days, just wait and see.

Merged, thanks!


Fix numerous issues in fullnode docs based upon user feedback

Specify storage requirements for fullnode
Mark fullnode.yaml editing as optional
Replace Local with Custom RPC URL set to port 9000


Not optimal sql query when using jdbc_page_size

I have prepared following logstash config
input {
 jdbc {
   jdbc_driver_library => "mysql-connector-java-5.1.39-bin.jar"
   jdbc_driver_class => "com.mysql.jdbc.Driver"
   jdbc_connection_string => "jdbc:mysql://dbhost:3306/mydb"
   jdbc_user => "user"
   jdbc_password => "password"
   statement => "SELECT * FROM test"
   jdbc_paging_enabled => "true"
   jdbc_page_size => "10000"
 }
}
output {
  stdout {  codec => json_lines  }
}

When I run logstash and check SHOW PROCESSLIST; on my db, I see following sql statements:
SELECT * FROM (SELECT * FROM test) as j LIMIT 10000 OFFSET 10000

This query reads all rows from table on each page (a few milions rows in my case) so this paging strategy is not optimal. It should do queries presented below.
SELECT * FROM test LIMIT 10000 OFFSET 10000

And it is too slow! Is there a proper way to perform sql "select * from tbl" when the amount of table is large?

is there any workaround for this?

There is a solution for this:
jdbc_connection_string => "jdbc:mysql://localhost:3306/mydb?useCursorFetch=true"
jdbc_fetch_size => 50000

useCursorFetch will use server-side cursors, which in MySQL are implemented using temporary tables

In MySQL, a table like this:
CREATE TABLE `videos` (
  `video_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`video_id`),
  KEY `ts` (`ts`)
) ENGINE=InnoDB

the optimal queries should be, for count and fetch respectively:
SELECT COUNT(ts) FROM videos WHERE ts >= :sql_last_value LIMIT 1

SELECT * FROM videos WHERE ts >= :sql_last_value ORDER BY ts ASC LIMIT 1000

Is there any way to achieve this? I can add a limit to the statement clause, but then the count will be wrong and presumably stop import after one iteration. I'd like to avoid high offsets and large temporary tables.

After poking around the issues (#200), it appears the count query is only used for debug. In that case, the latter query of mine should partially work; it would be preferable if it continued to fetch rows until exhausting the input.

@axhiao we found this issue also. Posted topic here: https://discuss.elastic.co/t/logstash-jdbc-plugin/84874
We have a large dataset.
What you recommend to performance tune?

@jbboehr count query also used in Sequel pager

@izhigalko In MySQL at least, queries with high offsets are slow. If possible, it's better to page using a where on an indexed column.
mysql> select video_id from videos force index (ts) order by ts asc limit 3665722, 1;
1 row in set (1.15 sec)

mysql> explain select video_id from videos order by ts asc limit 3665722, 1;
+----+-------------+--------+-------+---------------+------+---------+------+---------+-------------+
| id | select_type | table  | type  | possible_keys | key  | key_len | ref  | rows    | Extra       |
+----+-------------+--------+-------+---------------+------+---------+------+---------+-------------+
|  1 | SIMPLE      | videos | index | NULL          | ts   | 4       | NULL | 3472469 | Using index |
+----+-------------+--------+-------+---------------+------+---------+------+---------+-------------+

mysql> select * from videos force index (ts) order by ts asc limit 3665722, 1;
1 row in set (11.72 sec)

mysql> explain select * from videos order by ts asc limit 3665722, 1;
+----+-------------+--------+------+---------------+------+---------+------+---------+----------------+
| id | select_type | table  | type | possible_keys | key  | key_len | ref  | rows    | Extra          |
+----+-------------+--------+------+---------------+------+---------+------+---------+----------------+
|  1 | SIMPLE      | videos | ALL  | NULL          | NULL | NULL    | NULL | 3472472 | Using filesort |
+----+-------------+--------+------+---------------+------+---------+------+---------+----------------+

mysql> select * from videos where ts >= '2017-05-12 17:45:20' order by ts asc limit 1;
1 row in set (0.00 sec)

mysql> explain select * from videos where ts >= '2017-05-12 17:45:20' order by ts asc limit 1;
+----+-------------+--------+-------+---------------+------+---------+------+------+-----------------------+
| id | select_type | table  | type  | possible_keys | key  | key_len | ref  | rows | Extra                 |
+----+-------------+--------+-------+---------------+------+---------+------+------+-----------------------+
|  1 | SIMPLE      | videos | range | ts            | ts   | 4       | NULL | 1127 | Using index condition |
+----+-------------+--------+-------+---------------+------+---------+------+------+-----------------------+

The solution I'm going with for now is to use a higher limit without paging and just wait for the scheduler.

Here is the cause of the issue:
Like explained here, Sequel (the library used to interact with the DB) do not parse the raw SQL that is provided to it. So it cannot add a LIMIT or an OFFSET clause to the statement. The only thing it can and does do is run the statement provided by the user as a subquery.
Imo, the solution to our problem is simple but lower level: take care of the pagination ourselves (by adding manually the LIMIT and OFFSET clauses to the statement).

Disclaimer: We use MySQL, so this may only apply to that.
I'm not sure you're replying to my comment or not, but my suggestion is to have an option to not use offset for paging at all. Each subsequent page should be fetched using the previous :sql_last_value until their are no more records remaining.
Using an offset of 1m involves a key or table scan of 1m rows, whereas using an indexed where involves a simple btree lookup. Note that, in the former case, the last iteration will involve a full table or index scan (assuming MySQL doesn't pull of some dirty tricks, although I'm pretty sure it doesn't).
I've mostly set this up already in my own configuration, the only issue is each page has to wait for the next cron iteration. Because of this, I've set the cron to once per 15 seconds with a high limit clause, but it should be faster to import if it automatically fetches each page after the previous is done.
If the cron is set to once per second and it takes more than one second to import, does it ignore the next iteration or double up (parallel or serial)?

@jbboehr I was not replying specifically to your comment. I was explaining the root cause of the issue mentioned by OP. I also opened a PR to solve that issue.
However, I implemented what you described here. I needed to migrate several tables with millions of rows. Using this approach, I noticed a performance increase of several order of magnitude. I will try to clean up the code and make it mergeable shortly.

The second approach works wonders for importing a 1+ billion row MySQL database into elasticsearch. The bottleneck that's left is the paged MySQL query can only use 1 thread so logstash waits however long until it finishes for every paged query. On a 48 core machine this is highly inefficient and resources are barely used. Is there a way to execute multiple paged queries at a time to in essence multithread it?

Sorry if I'm missing something, but can someone explain why c7c91c4 closes this issue? The docs note that paging is not enabled when using prepared statements, so I'm not sure how that helps.
I'm currently running into a similar issue. I have a well-optimized query that becomes extremely slow (takes several minutes to execute, as opposed to a fraction of a second) when it is wrapped in the SELECT * FROM () LIMIT X OFFSET Y when using paging. However, I need to use paging as the dataset size causes an out of memory error in logstash when paging is off.
What is the recommended solution?


Set custom installation paths for dependencies

This is needed to have wordpress plugins installed in their default wp-content/plugins and tonik/gin inside vendor

These paths are too much "project specific" and adding it by default may be a problem for someone with different WordPress setup (like based on Bedrock or WP Starter)
Can you describe more how your WordPress installation structure and workflow is like and what problem these changes solves?

Well I'm working in a normal WordPress installation, using Redux Framework as single dependency to manage the theme options plus content, then RF and Gin were being installed in wp-content/plugins (which also gave me issues having Gin a missing vendor directory because it just cloned into plugins folder).
It is true that this brings issues with other setups, so I guess this is a task for the CLI tool, just to insert a convenient default configuration in composer.json depending on WordPress setup and if user accepts it of course. What do you think?

After the second look, I still say that it should be adjusted to a specific project. Trying to cover all use cases in CLI will be really hard.


Problem send message error after 8 - 12 hours

Afters 8 - 12 hours, When i try to connect to WebSocket server from the client, after connect success, client send message but the server respond with the following error Error during WebSocket handshake.
I'm using autobahn js from client side, but i tested the connection from Simple Websocket Client and websocket and same thing appened.
My server code:
        $port = 8080;
        $server = IoServer::factory(
            new HttpServer(
                new WsServer(
                    new WampServer(
                        new Wamp
                    )
                )
            )
          , $port
        );
        $server->run();

I run 8PM yesterday, and tested 9AM today. Two day, i don't have any solution.
Thanks all.

plz, add Waml class code

@Gemorroj I get code from link https://github.com/ratchetphp/socketo.me/blob/master/src/Ratchet/Website/Chat/ChatRoom.php

sr, I found error.
New day, wamp class can't connect mysql.
Code error: 2006
Message code: MySQL server has gone away ..

Keep in mind that when working with a database, you should use an asynchronous model.
https://github.com/amphp/mysql for example

@Gemorroj Thank you.
I will confirm it.


Make admin web app accessible with users with different roles and permissions

Problem
Currently, admin web app is not developed to restrict different sections based on the scopes. Only the workflow sections check for the workflow-related scopes [1] before loading and because of it, we can't allow different roles of users to access the admin web app.
[1] - https://github.com/wso2/apim-apps/blob/main/portals/admin/src/main/webapp/source/src/app/components/AdminPages/Dashboard/Dashboard.jsx#L43
Solution
Evaluate the scopes of each section before render.
Affected Component
APIM
Version
No response
Implementation
No response
Related Issues
No response
Suggested Labels
No response

We had a design review to finalize the design.
Admin web app is not developed to restrict different sections based on the scopes. Currently any user with apim:admin scope can view all the sections (Rate limiting policies, Gateways, API Categories, Key Managers, Settings except Tasks). We can restrict each section by checking each user's scopes from the UI. But if we can go with an in- built role(internal role) for common scenarios such as task managing, rate limiting, settings ..etc. we can reduce the overhead of assigning multiple scopes for the users. We will have to create internal roles for each operation as in below.

Admin portal Section
Scopes
Suggested internal Role

Rate Limiting policies
apim:tier_view, apim:policies_import_export, apim:tier_manage, apim:bl_manage, apim:bl_view
Policy manager

Gateways
apim:environment_manage, apim:environment_read
Gateway manager

API categories
admin_operations
Category manager

KeyManagers
admin_operations
Keymanager admin

Tasks
apim:api_workflow_view, apim:api_workflow_approve, apim:tenantInfo ,apim:admin_settings
Workflow manager

Settings: Applications
apim:app_owner_change, apim:app_import_export, apim:admin_application_view
application manager

Settings: Scope Assignment
apim:scope_manage, apim:role_manage
scope manager

In case of the design, the user experience may be easy as follows.Customers want to restrict admin users for the work-flow functionalities, they just assign admin users to internal/workflowManger roles. When an admin user logs to the admin portal, he/she can only see the tasks left side menu.

By considering the user experience and the maintainability,  we discussed to go with an internal/manager role which is bound to most common user scenarios such as tasks and rate limiting policies. It is the default role which will go with the pack. Besides that, if someone wants to add other functionalities to it or else if the user wants to restrict only for one operation, he/she has to follow the doc and create custom roles and map related scopes to it.

We had a code review and asked to create a new constant file and put the necessary constants in there without adding it to the public accessible settings.js.

I created separate file and still the PR is in review

Please find the doc [1].
[1]. https://docs.google.com/document/d/1a4hrjD3VrPJ81BpN21-x1DX8y54pXXByqgqAVeVk6uA/edit?usp=sharing

Additional Fixes
https://github.com/wso2/carbon-apimgt/pull/12182
https://github.com/wso2/apim-apps/pull/513

Integration test fixes
https://github.com/wso2/product-apim/pull/13287
https://github.com/wso2/product-apim/pull/13290

Remove invalid scopes https://github.com/wso2/apim-apps/pull/514


Fix ISSUE#142(Incorrect bean property)

Fix #142

Hi, @kbilsted
Thank you for your advice. I'll refactor this code.

Hi, @kbilsted
After trying, only "\uFEFF" BOM information is left in Java, no matter the file is utf8 with BOM, utf16le, utf16be. Therefore, it only need to deal with the situation of "\uFEFF"

@dota17  many thanks for your input. it is really strange since my understanding of the BOM's correspond to https://en.wikipedia.org/wiki/Byte_order_mark

Hi, @kbilsted
I agree, the BOM in the following is the same as you understand. However, after the code reading is processed through the Java read API, "\ufeff" will be left in Java at the first of the string. Because the Java read API only converts the files with different encoding into the unified Java Unicode code, but does not remove the BOM.
BufferedReader reader = new BufferedReader(new InputStreamReader(
                                                  new FileInputStream(UTF8_FILE), "UTF-8"));
BufferedReader reader = new BufferedReader(new InputStreamReader(
                                                  new FileInputStream(UTF16BE_FILE), "UTF-16"));
BufferedReader reader = new BufferedReader(new InputStreamReader(
                                                  new FileInputStream(UTF16LE_FILE), "UTF-18"));

customers_utf8.csv:

customers_utf16be.csv:

customers_utf16le.csv:

Hi, @kbilsted
What about this PR? Close?

nice detail you only try to subtract on the first line


model result

In the results, there will only be one for each category. Is there any solution to solve the problem?

Hi @uuu686,
I am not sure if I understand your question correctly. The grasp detection branch predicts multiple grasp candidates for the objects in the scene. For visualization and evaluation, we take only the one grasp candidate with the highest confidence score for each category.
Now, if there are multiple objects of the same category in the scene (e.g. two bananas), the final output will be a grasp candidate for only one banana, as both belong to the same category. If you think about grasping in practice, after successfully removing the one banana, the network will again predict grasp candidates for the other one which is left.
However, if you want to predict possible grasp candidates and segmentation masks for all objects in the scene regardless of the object category, you should look at our  BMVC paper, where we perform class (or category) agnostic object instance segmentation and grasp detection, to predict grasp candidates and segmentation masks for all graspable objects in the scene.
Let me know if this answers your question.

Thanks for your reply, and the answer solved my question.
Moreover, I want to ask another question: If I want to train the model on my dataset, I should prepare the three folders: "Annotation", "rgb" and "seg_mask_labeled_combi". The "rgb" folder contains the original images; the "seg_mask_labeled_combi" folder contains the labels of segmentation; the "Annotation" folder contains grasp candidates,and each grasp candidate contains four coordinates. I'm confused that is there any requirement for the order of these four points. And I also want to ask what software did you use to do this annotation?
Thank you, and I look forward to your apply.

Yes, you are right, you need the rgb images, as well as the annotations for grasp candidates and segmentation. If you use the same folder structure as OCID_grasp, you can re-use most parts of the provided OCID_grasp dataloader.
The four coordinates represent the corners of the grasp candidate. The order is important, whereas the first point represents the top-left corner, then continuing in clock-wise order (same as Cornell grasp dataset).
We implemented our own annotation tool, which is not part of this repository. However, there is software available which might be interesting for you, e.g. here.

Thank you very much, and I find a tool https://github.com/cgvict/roLabelImg.The tool can be used to do the task.
And I want to ask,when i annotate the grasp candidate, how should I annotate the rectangle, such as the width, the height, and the border.
For example, when I'm annotating a pen, it has many grasp candidates, and these rectangles will have overlapping parts. How should I control its width, height, and the size of the overlapping area.

It is up to you how you annotate your data.
Overlapping grasp candidates are not a problem, as each grasp candidate is independent.
The height of the rectangle depends on the graspable object; for the width you should keep in mind that it represents the parallel plates of your gripper.
Hope this information helps. I am closing this issue now, feel free to reopen if needed, or create a new issue for other questions.