DRILL-5068: Create sys.profiles and sys.profiles_json tables

(Thanks to Hongze Zhang for the initial work)
Introduced a non-distributed (i.e. the foreman Drillbit executes a single fragment query) system table to list the completed profiles that are visible to it in the persistent store. This allows for querying the profiles on the following common metrics:
queryId
startTime
foreman
fragments
user
queue
planTime
queueTime
executeTime
totalTime
state
query
An additional system table - sys.profiles_json has also been introduced. This only has the following fields:
queryId
json
This allows for more detail metrics to be queried on the profile as a JSON document. Users are expected to use sys.profiles to identify profiles on the available metrics (such as long executeTime or fragments), and use the queryId to find the corresponding JSON document representation for further drill down into the profiles.
Here is what a sample output looks like:
[root@kk127 ~]# /opt/drill/apache-drill-1.13.0/bin/sqlline -u "jdbc:drill:drillbit=`hostname -i`"
...
apache drill 1.13.0-SNAPSHOT
"just drill it"
0: jdbc:drill:drillbit=<IP_ADDRESS>> select * from sys.profiles where startTime < date '2017-12-18' limit 10;
+---------------------------------------+--------------------------+---------------+------------+------------+--------+-----------+------------+--------------+------------+------------+--------------------------------------------------------+
|                queryId                |        startTime         |    foreman    | fragments  |    user    | queue  | planTime  | queueTime  | executeTime  | totalTime  |   state    |                         query                          |
+---------------------------------------+--------------------------+---------------+------------+------------+--------+-----------+------------+--------------+------------+------------+--------------------------------------------------------+
| 25cdcede-410d-6399-97f9-00ec2048f247  | 2017-12-14 08:06:57.062  | kk127.qa.lab  | 1          | anonymous  | -      | 35        | 0          | 7718         | 7753       | COMPLETED  | select queryId from profiles limit 1                   |
| 25cdd1a2-7928-bcf9-a8cd-8e0fa653e2ce  | 2017-12-14 07:55:09.551  | kk127.qa.lab  | 1          | anonymous  | -      | 41        | 0          | 7580         | 7621       | COMPLETED  | select queryId from profiles limit 1                   |
| 25cdd1b6-3a56-e2b3-a6a1-4d1006ec8fac  | 2017-12-14 07:54:49.403  | kk127.qa.lab  | 1          | anonymous  | -      | 77        | 0          | 7452         | 7529       | COMPLETED  | select queryId from profiles limit 1                   |
| 25cdd1bd-963f-7a57-6a2d-cf7a3eac3f00  | 2017-12-14 07:54:41.065  | kk127.qa.lab  | 0          | anonymous  | -      | 0         | 0          | 0            | 0          | FAILED     | select profileJson from profiles limit 1               |
| 25cdd1e5-de23-0cfe-5c01-86621b31d8d5  | 2017-12-14 07:54:01.576  | kk127.qa.lab  | 1          | anonymous  | -      | 1184      | 0          | 10860        | 12044      | COMPLETED  | select profileJson from profiles_json limit 1          |
| 25cdd1e7-7c3a-5a25-0860-a86246bbc9ed  | 2017-12-14 07:54:00.186  | kk127.qa.lab  | 1          | anonymous  | -      | 102       | 0          | 231          | 333        | COMPLETED  | use sys                                                |
| 25cdd1f0-4473-55aa-a5eb-c85194321118  | 2017-12-14 07:53:51.966  | kk127.qa.lab  | 0          | anonymous  | -      | 0         | 0          | 0            | 0          | FAILED     | select profileJson from profiles_json limit 1          |
| 25cdd442-04cc-65fe-7a25-ac075fa22467  | 2017-12-14 07:43:57.251  | kk127.qa.lab  | 1          | anonymous  | -      | 40        | 0          | 10553        | 10593      | COMPLETED  | select profileJson from profiles_json limit 1          |
| 25cdd7c8-614d-8bcc-3b89-08df12761a3a  | 2017-12-14 07:28:55.359  | kk127.qa.lab  | 1          | anonymous  | -      | 170       | 0          | 10508        | 10678      | COMPLETED  | select count(*) from (select * from profiles_json)     |
| 25cdd7e0-34ea-9040-ab2a-ef8c6a136440  | 2017-12-14 07:28:31.464  | kk127.qa.lab  | 0          | anonymous  | -      | 0         | 0          | 0            | 0          | FAILED     | select count(*) from (pwdselect * from profiles_json)  |
+---------------------------------------+--------------------------+---------------+------------+------------+--------+-----------+------------+--------------+------------+------------+--------------------------------------------------------+
10 rows selected (8.336 seconds)
0: jdbc:drill:drillbit=<IP_ADDRESS>> select * from sys.profiles_json where queryId = '25cdd1e7-7c3a-5a25-0860-a86246bbc9ed';
+---------+------+
| queryId | json |
+---------+------+
| 25cdd1e7-7c3a-5a25-0860-a86246bbc9ed | {"id":{"part1":2724064141780867621,"part2":603667490114619885},"type":1,"start":1513238040186,"end":1513238040519,"query":"use sys","foreman":{"address":"kk127.qa.lab","userPort":31010,"controlPort":31011,"dataPort":31012,"version":"1.12.0-SNAPSHOT","state":0},"state":2,"totalFragments":1,"finishedFragments":0,"fragmentProfile":[{"majorFragmentId":0,"minorFragmentProfile":[{"state":3,"minorFragmentId":0,"operatorProfile":[{"inputProfile":[{"records":1,"batches":2,"schemas":1}],"operatorId":0,"operatorType":26,"setupNanos":0,"processNanos":67070036,"peakLocalMemoryAllocated":4530688,"waitNanos":0},{"inputProfile":[{"records":0,"batches":0,"schemas":0}],"operatorId":0,"operatorType":13,"setupNanos":0,"processNanos":0,"peakLocalMemoryAllocated":4530176,"waitNanos":0},{"inputProfile":[{"records":1,"batches":1,"schemas":1}],"operatorId":0,"operatorType":13,"setupNanos":0,"processNanos":15787437,"peakLocalMemoryAllocated":4530176,"metric":[{"metricId":0,"longValue":41}],"waitNanos":18796208}],"startTime":1513238040329,"endTime":1513238040482,"memoryUsed":0,"maxMemoryUsed":9060864,"endpoint":{"address":"kk127.qa.lab","userPort":31010,"controlPort":31011,"dataPort":31012,"version":"1.12.0-SNAPSHOT","state":0},"lastUpdate":1513238040487,"lastProgress":1513238040487}]}],"user":"anonymous","optionsJson":"[ ]","planEnd":1513238040288,"queueWaitEnd":1513238040288,"totalCost":0.0,"queueName":"-"} |
+---------+------+
1 row selected (11.667 seconds)

@sohami
Applied the checks for security to deny unauthorized access.
This is a screenshot with an admin user (mapr) querying the table.

This is a screenshot with a non-admin user (kuser1) querying the table.


docs(pf-button): add href for Link Variant

What I did

Add example of how to include href values to pf-button link variant.

Testing Instructions

Observe and Test the pf-button link variant with href value.

Notes to Reviewers
@adamjohnson I Had some questions regarding A11y:-

When I use TAB Key and try clicking the pf-button with a href link, the Focus gets inside the button. Would that be a Accessibility concern ?

Closes: #2769

When I use TAB Key and try clicking the pf-button with a href link, the Focus gets inside the button. Would that be a Accessibility concern ?

Yes. This will fail WCAG Success Criterion 4.1.2 Name, Role Value — Level A.
This is a commonly seen issue most often referred to as "nested interactive controls". When an interactive control has another interactive control as a child element, it disrupts how the child element is communicated to the accessibility tree. This issue can prevent users of assistive technology from finding the child element, getting its correct name and role information, or interacting with it.
The best way forward would be to not nest interactive controls.
It looks like Lumar, a QA and accessibility tool Red Hat uses, has a check for nested interactive controls.

When I use TAB Key and try clicking the pf-button with a href link, the Focus gets inside the button. Would that be a Accessibility concern ?

Yes. This will fail WCAG Success Criterion 4.1.2 Name, Role Value — Level A.
This is a commonly seen issue most often referred to as "nested interactive controls". When an interactive control has another interactive control as a child element, it disrupts how the child element is communicated to the accessibility tree. This issue can prevent users of assistive technology from finding the child element, getting its correct name and role information, or interacting with it.
The best way forward would be to not nest interactive controls.
It looks like Lumar, a QA and accessibility tool Red Hat uses, has a check for nested interactive controls.

When I use TAB Key and try clicking the pf-button with a href link, the Focus gets inside the button. Would that be a Accessibility concern ?

Yes. This will fail WCAG Success Criterion 4.1.2 Name, Role Value — Level A.
This is a commonly seen issue most often referred to as "nested interactive controls". When an interactive control has another interactive control as a child element, it disrupts how the child element is communicated to the accessibility tree. This issue can prevent users of assistive technology from finding the child element, getting its correct name and role information, or interacting with it.
The best way forward would be to not nest interactive controls.
It looks like Lumar, a QA and accessibility tool Red Hat uses, has a check for nested interactive controls.

Thanks @adamjohnson for the information and guidance for the A11y on this issue.
Hi @bennypowers, Since I can't nest the anchor tag inside the pf-button.
Should I try creating a href prop for pf-button "link-variant" for adding links ? Your thoughts on this.

yeah i think changing the template for pf-button depending on whether there's an href attribute makes sense
@adamjohnson, considering that pfv4 has a <Button component="a" href={}> component , and considering our mandate in this repo to reproduce as closely as possible upstream patterns, do you concur that we can have <pf-button href="...">, which would produce an anchor in shadow dom?

@bennypowers As long as there are no nested interactive controls, that works for me. 👍

Closing this PR, further changes can be tracked under PR #2814


WaPo & MPV Data / Fatality

Solution description

Chose to use python to import data from Washington Post and MPV(Mapping Police Violence) in an effort to move forward with connecting the frontend to relevant data until the point which we have them coming from DS and BE in the needed formats.
Created simplified JSON files which can act as filters in the Fatality and Incident pages and also make visualizations in charts less bloated and easier to accomplish.

Covered unit test cases
Were unit test cases recorded for this fix?
Code is clear and maintainable:

[x] My code follows the style guidelines of this project
[x] My code is consistently formatted.
[x] No instance on duplicate code.

Tested this issue fix on all browser
[x] Chrome
[x] Safari
[x] Internet Explorer
[x] Firefox
Keep Pull Request Simple
Pull Request is relatively small. It does not contain multiple changes or solve multiple problems.
Reviewed by Team members
Pull Request is reviewed by one or more team members
Idiomatic Code
Code follows industry standards for language, framework, and libraries.

Great work, I need to do more research on Python Pandas!


Support broker plugins

The ActiveMQArtemis CR doesn't support broker plugins. The support could be added defining a new field i.e.
apiVersion: broker.amq.io/v1beta1
kind: ActiveMQArtemis
metadata:
  name: artemis-id-cache
spec:
  brokerPlugins:
    - className: some.plugin.UserPlugin
      properties:
        - key: property1
          value: val_1
        - key: property2
          value: val_2

I think the existing broker properties may be sufficient but what would be missing is some way to add the plugin jar to the classspath. see https://github.com/artemiscloud/activemq-artemis-operator/issues/562

https://issues.apache.org/jira/browse/ARTEMIS-4443


feat: add script to add missing countries to db

Description
The purpose of this PR is to create a script that will add missing countries from a service to as a row in the lists database
Trello ticket: https://trello.com/c/sRF8i7tc/1166-admin-users-to-setup-lists-and-prevent-operational-actions-on-list-items
Note!!
I haven't committed my package.json file but to get this to run you'll need to add this line to the scripts section
"add-missing-countries": "npx -y ts-node ./src/server/utils/missing-countries.ts --esm"
https://user-images.githubusercontent.com/1377253/184678416-aba0e6e0-e4fa-4b23-8eeb-23c972ac7de5.mp4

hey @alisalaman-cyb - I'm not sure this is the right approach, or perhaps I'm not following your intention here. let's discuss when you're back.
A script / process to retrieve List records for all countries for a given service type, compare with the existing country list in metadata.ts and create any that are missing. The inputs to the process should be

serviceType (mandatory)
sericeTypeToCopyFrom (optional): specifying this will copy the email addresses specified in the administrators, validators and publishers fields that are defined in the source List to copy from into the destination (newly created or existing) List for the serviceType specified in serviceType (argument a).

Did you want a manually run script via CLI? who would do this (let's say in future when this is supported by IT/IDD, I don't know if they'd be comfortable auth and running these scripts, and I'm not entirely sure they should?)
point 2 sounds like this should be a GUI feature, i.e. for a superAdmin, they can create a serviceType and copy from an existing one from the LMS interface.
point 1 sounds like a migration script. I would do this by

manually generating a CSV of countries in metadata.ts
import this CSV and insert new countries, ON CONFLICT DO NOTHING.
then do the same for the missing lists
going forward, create a migration script each new provider that we deploy
that creates a new list for each country
and copies over permissions (although I'm not sure we should do this via a script anyway, since we release selectively)

Not to self: Re-create this branch and apply changes to avoid conflicts when a decision has been made


Aliyun

aliyun的资源

aliyun

I'm not really sure what this PR is all about. If you'd like to submit content for inclusion, please break the PR down into smaller, more consumable (and more easily reviewable) chunks. Because there are too many files and too many changes to review, I'm closing this PR.


Single page API

Add an API that creates a single page version of Parchment for a given file

The MVP is finished, thanks @dfabulich!


Can't upload more than reserveFileSizeThreshold into NFS mount: fallocate: operation not supported

I'm uploading files into a NFS filesystem (Linux), which doesn't support the fallocate syscall, which result in the request failing with status 500. The fix is simple: SizeWillBe should not fail at all if the error from fallocate is ENOTSUP.

I'm not sure about how the release procedure work for caddy plugins. Will this fix be available on the upcoming April 20's Caddy release? If not, when and how does it get released?
Thanks

It'll be available for the next release, or the 21ˢᵗ at the latest.
As part of the CI/CD pipeline a binary is built with the latest version of this plugin. To download it click on the »build: success« icon. Pick the topmost result, and click on the button on its right side.


feat: support OpenTelemetry

Signed-off-by: Andy Lok<EMAIL_ADDRESS>

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

40 of 46 (86.96%)  changed or added relevant lines in 3 files are covered.
7 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.4%) to 78.722%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

minitrace-opentelemetry/src/lib.rs
36
42
85.71%

Files with Coverage Reduction
New Missed Lines
%

minitrace-datadog/src/lib.rs
7
71.79%

Totals

Change from base Build<PHONE_NUMBER>:
-0.4%

Covered Lines:
1491

Relevant Lines:
1894

💛  - Coveralls


Keep getting  Method not found: 'System.Object Akka.Persistence.SnapshotOffer.get_Snapshot()'. when trying to read from snapshot offer

Snapshots are being saved just fine, but I get this error every time I try to get the snapshot offer from Mongo.
All my message classes are registered using BsonClassMap.RegisterClassMap. I even tried to register SnapshotMetadata and SnapshotOffer just to try something else out but nothing. Keep getting the same error:
[ERROR][12/3/2017 5:36:21 p. m.][Thread 0003][akka://UserManagerActorSystem/user/UserManagerActor/srv_qatoolstester] Method not found: 'System.Object Akka.Persistence.SnapshotOffer.get_Snapshot()'.
Cause: System.MissingMethodException: Method not found: 'System.Object Akka.Persistence.SnapshotOffer.get_Snapshot()'.
at Moravia.Ahoj.Server.System.Users.Actors.UserActor.b__5_8(SnapshotOffer offer)
at lambda_method(Closure , Object , Action1 , Action1 , Action1 )    at Akka.Tools.MatchHandler.PartialHandlerArgumentsCapture4.Handle(T value)
at Akka.Persistence.ReceivePersistentActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Persistence.ReceivePersistentActor.OnRecover(Object message)
at Akka.Persistence.UntypedPersistentActor.ReceiveRecover(Object message)
at Akka.Persistence.Eventsourced.b__88_0(Object message)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Persistence.Eventsourced.<>c__DisplayClass88_0.b__1(Receive receive, Object message)
at Akka.Persistence.Eventsourced.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)
Here is my hocon file configuration. Very simple:
    persistence
    {
      publish-plugin-commands = on

      journal
      {
        plugin = "akka.persistence.journal.mongodb"
        mongodb
        {
          class = "Akka.Persistence.MongoDb.Journal.MongoDbJournal, Akka.Persistence.MongoDb"
          connection-string = "mongodb://localhost/EventStore"
          collection = "EventJournal"
          metadata-collection = "Metadata"
          auto-initialize = on
          plugin-dispatcher = "akka.actor.default-dispatcher"
        }
      }

      snapshot-store
      {
        plugin = "akka.persistence.snapshot-store.mongodb"
        mongodb
        {
          class = "Akka.Persistence.MongoDb.Snapshot.MongoDbSnapshotStore, Akka.Persistence.MongoDb"
          connection-string = "mongodb://localhost/EventStore"
          collection = "SnapshotStore"
          auto-initialize = on
          plugin-dispatcher = "akka.actor.default-dispatcher"
        }
      }
    }

Events are replayed OK. The only problem seems to be when trying to read the snapshot offer

Had to use an older version of Akks.Persistence


[ENH] Added build for git

Why?

Because as we run Debian 0.1 the stock version is really old ....
Best wishes,
Stuart
On Aug 18, 2015, at 00:24, Thomas A Caswell <notifications<EMAIL_ADDRESS>wrote:
Why?
—
Reply to this email directly or view it on GitHubhttps://github.com/NSLS-II/conda-prescriptions/pull/59#issuecomment-132069179.

Can you please add a 2.5.0 subfolder to put the files in?  This layout is to play nicely with some automated tools that we will be bringing on line 'soon'.

I added the folder and merged locally.


Button and TextField Selection

After auto re sizing  Button and Text Field width, I found the selection highlighter have not covered for the full width.
Is there any way to fix this issue.
Below I have added the screenshot for your reference.

Facing same Issue here also

+1


Fixes error message for an existing cached page

Could you change it on the test as well: https://github.com/nagelflorian/flow-router/blob/ssr/server/__tests__/route.js#L30

Done. Anything else I've been missing @arunoda ?

this is great.


We might want a proper BYU page footer

Now we just have minimal links to privacy policy and cookie preferences.

See issue #94 which has a link to a sample page.


[PE-D][Tester A] Typo in UG

As attached, fied has a typo.

Labels: severity.VeryLow type.DocumentationBug
original: Khaleelur-Rahman/ped#17

done


Support for Now CLI v17

Since version 17 the Now CLI is now interactive by default. It can be worked around using environment variables - https://github.com/zeit/now/pull/3588 Perhaps this action could take these as options and set the variables.

@honzajavorek  v2 released
name: deploy website
on: [pull_request]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: amondnet/now-deployment@v2
        with:
          zeit-token: ${{ secrets.ZEIT_TOKEN }} # Required
          now-args: '--prod'
          now-org-id: 'YOUR_ORG'
          now-project-id: 'YOUR_PROJECT_ID'

Hello. Thanks for the update! I'm having trouble finding the Org and Project ID values in the Now dashboard. How can these be retrieved?

@amondnet awesome, thanks 👍
@thelucre yup, see my comment at https://github.com/zeit/now/pull/3588#issuecomment-586149881


None of the commands are working inside an angular-cli project

Version tested: 832b83b6c8e81a3af40a5b35df4340bec5fcd081
Steps to reproduce
1- Create an angular-cli project:
$ ng new test339
2- Step into it and try to run any of the commands:
$ cd test339
$ ng
Result
/Users/cironunes/work/angular-cli/bin/ng:27
  cli({
  ^
TypeError: object is not a function
    at /Users/cironunes/work/angular-cli/bin/ng:27:3
    at /Users/cironunes/work/angular-cli/node_modules/resolve/lib/async.js:44:21
    at ondir (/Users/cironunes/work/angular-cli/node_modules/resolve/lib/async.js:187:31)
    at /Users/cironunes/work/angular-cli/node_modules/resolve/lib/async.js:153:39
    at onex (/Users/cironunes/work/angular-cli/node_modules/resolve/lib/async.js:93:22)
    at /Users/cironunes/work/angular-cli/node_modules/resolve/lib/async.js:24:18
    at FSReqWrap.oncomplete (fs.js:95:15)

This is because the angular-cli npm package is not currently pointing to this repo.
I'll have to publish a new version of the package to overwrite it.

This is fixed now as I published the new version of angular-cli to npm.


Request to move plugin to FilePond Instance instead of FilePond Object

Summary
Possibility of moving the option to register a plugin to the FilePond Instance instead of the FilePond Object.  This request is due to have multiple instances of FilePond on a page which require different plugins.
How to reproduce
Plugins can only be registered through FilePond Object
Expected behaviour
Able to register plugin through FilePond Instance

You can enable/disable plugins per instance using their respective toggle properties. For example allowImagePreview.

Thanks.  This is exactly what I saw looking for.


bigquery: rename name field of Dataset to dataset_id

Rename the former dataset_id property to full_dataset_id.
Also rename Table.dataset_name to Table.dataset_id.
Perform other renamings (of various variables and constants).
These names match usage better. The API's Dataset.id field is
"project:dataset_id", which is confusing and basically useless,
so it's a mistake to call that dataset_id.

Having done this, I'm a bit bothered by the stutter of dataset.dataset_id. What about renaming the field to id? Is that idiomatic Python?

Having done this, I'm a bit bothered by the stutter of dataset.dataset_id. What about renaming the field to id? Is that idiomatic Python?

Yes. I would also like that it neatly solves the problem I mentioned in my previous comment.

Having done this, I'm a bit bothered by the stutter of dataset.dataset_id. What about renaming the field to id? Is that idiomatic Python?

We have historically avoided using bare id as an attribute, both to avoid the (appearance of) shadowing the id built-in, and also because it makes the semantics clearer.
Re: stutter:  there is no requirement that instances be assigned to variables named dataset.  E.g., consider a notional "copy" operation, with source and destination as the names for the two dataset instances.

OK, will leave as-is. Waiting for @tswast to approve.


Refactored AppBar using flexbox

Used flexbox in app bar which is more reliable

Bonus Refactored svg to SvgIcon
closes #16
closes #17

Awesome, thanks @pantharshit00 !


(maint) Fix typo Error -> Errno

Fixes typo in the hypervisor/vmpooler.rb

:sparkles:

Refer to this link for build results (access rights to CI server needed):
http://jenkins-beaker.delivery.puppetlabs.net//job/qe_beaker_btc-intn/916/
Test FAILed.

Debian 7 sneeze, test failures unrelated to patch.


[BUG] The vocab_size of tokenizer is mismatched with the vocab_size of the model config

是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this?

[X] 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答？ | Is there an existing answer for this in FAQ?

[X] 我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior
Qwen tokenizer vocab_size: 151851
Qwen vocab_size in config.json: 151936
What are the additional 85 (151936 - 151851 = 85) tokens used for? Will this cause any unexpected issues?
For example, if we add 100 new tokens in the pretrained tokenizer, the 85 tokens are already in the model's embedding space (un-trainable as we do not want to change the LLM's performance), while the new 15 tokens should be added by the model's resize_token_embeddings() function (trainable).
期望行为 | Expected Behavior
The Qwen tokenizer vocab_sizeshould be matched with the Qwen vocab_size in config.json. Then if we add new tokens, the model's resize_token_embeddings behavior will make sure the new token embeddings are trainable.
复现方法 | Steps To Reproduce
No response
运行环境 | Environment
- OS:
- Python:
- Transformers:
- PyTorch:
- CUDA (`python -c 'import torch; print(torch.version.cuda)'`):

备注 | Anything else?
No response

What are the additional 85 (151936 - 151851 = 85) tokens used for?

Please see https://github.com/QwenLM/Qwen/issues/419 and https://github.com/QwenLM/Qwen/issues/482 for our previous responses. resize_token_embeddings has an argument named pad_to_multiple_of for the exact same reason.

Will this cause any unexpected issues?

For one, I don't think transformers support your example natively. (Correct me if I'm wrong.) resize_token_embeddings will replace the old embedding module with the new embedding module, which means that it is, as a whole, either trainable or not trainable.
If customized code is needed, you could use the vocab_size of the tokenizer to determine which parts needs to be trained. The embedding parameters corresponding to tokens out of the tokenizer are not meaningful (not in the model's embedding space) and can be used for new tokens. For example, run resize_token_embeddings(new_tokenizer_vocab_size, pad_to_multiple_of=128) and then make sure the first old_tokenizer_vocab_size tokens is not trainable.

Thanks for your kind reply. I see, 151851 is the real old vocab size, while 151936 (7B) and 152064 (14B) are for efficient training.
FYI, we can make the whole embedding or lm_head module trainable and then masking out old embeddings' gradients, e.g.,
mask = torch.zeros_like(self.lm_head.weight)
mask[self.config.old_vocab_size:] = 1.
self.lm_head.weight.register_hook(lambda grad: grad*mask)


remove case ordering section

Ordering cases by having the simplest thing be first is common practice in functional programming. I see two reasons to do so:

Tail recursion
Understanding the base case makes it easier to read the more complicated case.

On the other hand, I understand the desire to have Failure cases be at the bottom, so it looks more like a try/catch block.
To avoid coming up with complicated rules I suggest removing this clause altogether.

I'm in favor of this change - this section hasn't been overly useful to us in practice. Better to use one's judgement.

No objections from me.

LGTM

forgot about this PR. we should merge it so I can update #10 properly. @ayakushev99 can you do the honors?

I'm fine with removing this from Scalastyle, but I still believe that we should follow this in almost all cases. We've certainly discussed it when deciding how to structure code.


Feature request: allow using the target page's displayed page number as the link text in index entries

I have a document where the first three pages are unnumbered, and numbering stats at 1 on the fourth actual page of the PDF (using #counter(page).update(1)). Since in-dexter uses loc.position() for both the link destination and the link text, the links go to the right location but have the wrong text.
This patch file will add a new field to make-index called use-page-counter that will make it use the page counter for display numbers, but I'm fine to send it as a PR instead if that would be easier for you, or also fine if you don't want to accept drive-by contributions.
diff --git a/in-dexter.typ b/in-dexter.typ
index e8f2044..4b515ec 100644
--- a/in-dexter.typ
+++ b/in-dexter.typ
@@ -14,6 +14,7 @@
         fmt: fmt,
         initial: initial,
         location: loc.position(),
+        page-counter: counter(page).display(),
         entry: entry,
     ))<jkrb_index>
 ])
@@ -65,7 +66,7 @@
     let register = (:)
     let initials = (:)
     for indexed in query(<jkrb_index>, loc) {
-        let (entry, fmt, initial, location) = indexed.value
+        let (entry, fmt, initial, location, page-counter) = indexed.value
         let entries = entry.pos().map(as-text)
         if entries.len() == 0 {
             panic("expected entry to have at least one entry to add to the index")
@@ -90,7 +91,7 @@

             }
             let reg-entry = register.at(initial-letter, default: (:))
-            register.insert(initial-letter, make-entries(entries, (page: location.page, fmt: fmt), reg-entry))
+            register.insert(initial-letter, make-entries(entries, (page: location.page, fmt: fmt, page-counter: page-counter), reg-entry))
         }
     }
     (register: register, initials: initials)
@@ -98,21 +99,22 @@

 // Internal function to format a page link
-#let render-link((page, fmt)) = {
-    link((page: page, x: 0pt, y: 0pt), fmt[#page])
+#let render-link(use-page-counter, (page, fmt, page-counter)) = {
+    link((page: page, x: 0pt, y: 0pt), fmt[#if use-page-counter { page-counter } else { page }])
 }

 // Internal function to format a plain or nested entry
-#let render-entry(idx, entries, lvl) = {
+#let render-entry(idx, entries, lvl, use-page-counter) = {
     let pages = entries.at("pages", default: ())
+    let render-function = render-link.with(use-page-counter)
     let rendered-pages = [
-        #box(width: lvl * 1em)#idx#box(width: 1fr)#pages.map(render-link).join(", ") \
+        #box(width: lvl * 1em)#idx#box(width: 1fr)#pages.map(render-function).join(", ") \
     ]
     let sub-entries = entries.at("nested", default: (:))
     let rendered-entries = if sub-entries.keys().len() > 0 [
         #for entry in sub-entries.keys().sorted() [
-            #render-entry(entry, sub-entries.at(entry), lvl + 1)
+            #render-entry(entry, sub-entries.at(entry), lvl + 1, use-page-counter)
         ]
     ]
     [
@@ -125,7 +127,8 @@
 // Inserts the index into the document
 // @param title (default: none) sets the title of the index to use
 // @param outlined (default: false) if index is shown in outline (table of contents)
-#let make-index(title: none, outlined: false) = locate(loc => {
+// @param use-page-counter (default: false) use the value of the page counter for page number text
+#let make-index(title: none, outlined: false, use-page-counter: false) = locate(loc => {
     let (register, initials) = references(loc)

     if title != none {
@@ -141,7 +144,7 @@
         heading(level: 2, numbering: none, outlined: false, letter)
         let entry = register.at(letter)
         for idx in entry.keys().sorted() {
-            render-entry(idx, entry.at(idx), 0)
+            render-entry(idx, entry.at(idx), 0, use-page-counter)
         }
     }
 })
\ No newline at end of file

Nice, I hadn't this scenario yet.
I will give it a try. Can you please send a PR for it? I will test and document it then.
Thank you!

Created as #8 :)


inactive save buttons

I modified the docking route and the map a bit, but I can't save it.

Screen recording 2024-04-30 16.02.20.webm
Maybe the error is in my head :)

For some reason after clicking move points your made changes has been reseted to legacy. It cold happened if your shapes aren't valid. Try it again and pls log everything in debug level.

For some reason after clicking move points your made changes has been reseted to legacy. It cold happened if your shapes aren't valid. Try it again and pls log everything in debug level.

2024-04-30 17 02 20 INFO Backend  Starting backend.txt

Some shape is intersecting itself
2024-04-30 17:03:08 DEBUG TopologyException: Input geom 1 is invalid: Self-intersection at 5.618942576603521 16.17930306185232

unfortunately, this is the same problem when moving any point on the map. I exported from Sunray.

Your import is already faulty, you have to fix this coordinates before you can use it. Just check the coordinates mentioned above

16.1793

I looked for the point in question in the .json file, deleted it, everything is OK. Thank you very much.

"16.1793
I looked for the point in question in the .json file, deleted it, everything is OK. Thank you very much."
It was written by you, didn't? I thought it is everything alright

"16.1793
I looked for the point in question in the .json file, deleted it, everything is OK. Thank you very much."
It was written by you, didn't? I thought it is everything alright

yes, I retaliated. I deleted all 16.17 entries, after pressing the button on the right, it lets me save, but the old map is saved. I'll map again tomorrow.

Log? I think it is another issue

Don't use the same name for new map, in this case your changes will be ignored without warnings. Still on to do list...

map (1).json
2024 05 01 10 43 32 INFO.txt
The attached sunray file does not have the incorrect position. What to do? THX

Can you attach here your sunray export. I will check that

Ah sorry, didn't see the map json before

Here, just remove this point by marking it an clicking "-" after that is your map valid

That is the self intersection.

I was able to delete it in the sunray application, and downloaded it again. Now it works perfectly. Thank you very much. :)

 That is the self intersection.

I was able to delete it in the sunray application, and downloaded it again. Now it works perfectly. Thank you very much. :)


remove quotes from CSS URL functions

Hello! Personally, I prefer to write CSS url functions as url('#xxx') as opposed to url(#xxx) in my SVG source. It would be cool if SVGO could be able to identify these and convert them to the shorter (latter) form.
I was able to write a custom plugin that did the trick for me, but I feel like some modifications would be appropriate to make it accept double quotes, as well as URL functions that do not take up the full attribute value.
{
	type: "perItem",
	fn: item =>
	{
		if(!item.isElem()) return

		for(let name in item.attrs)
		{
			let attr = item.attrs[name]
			let value = attr.value
			if(value.startsWith("url('") && value.endsWith("')"))
				attr.value = `url(${value.slice(5, -2)})`
		}
	},
}

@TrySound this doesn't happen in the latest csso:
.foo { background: url("#xxx") }
.bar { background: url('#xxx') }

.foo{background:url(#xxx)}.bar{background:url(#xxx)}

Is there any other place that this could be missed?

SVGO always works. Probably because I bumped CSSO version.
const { extendDefaultPlugins } = require('./lib/svgo-node.js');
module.exports = {
  plugins: extendDefaultPlugins([
    {name:'inlineStyles', active: false}
  ])
}

<svg id="test" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100">
    <style>
        .st0{ fill:url("#ssdsd"); padding-top: 1em; padding-right: 1em; padding-bottom: 1em; padding-left: 1em; }
    </style>
    <rect width="100" height="100" class="st0" style="stroke-width:3; margin-top: 1em; margin-right: 1em; margin-bottom: 1em; margin-left: 1em;"/>
</svg>


CreateAccountRequest and create account sometimes bugs out, with NO error message

I have a problem with some users that if they create an account with their email, and get the code. They enter the code, and I directly do this function:
bool success = await userAuth.createAccountRequest(
      data.additionalSignupData!["first_name"].toString().replaceAll(" ", "") + " " + data.additionalSignupData!["last_name"].toString().replaceAll(" ", "") ,
      data.name!,
      data.password!,
    );

When I do that function I get a false returned, but in the logs there are NO error or any things I can see that are off. When I use the serverpod analytics application I see this:

Not really an error and their is a record in the database?
But for some reason the function returns false?
And then, if I skip this step, and go to verify the code I run this function:
 var userInfo = await userAuth.validateAccount(data.email, code);
Their is no userInfo returned just null and this is the record in the analytics:

But NO error? And now the user is not even added as a record in the table.
So I don't know what is going on here, I have no freaking idea....
And the problem is that some user have this problem and I don't know how to fix it and we are in testing with the application but this is a total deal breaker, if not all users can sign-up.
To Reproduce
Just use the serverpod guide to email signup. I have EXACTLY those steps.
Expected behavior
Just the normal sign-up process
Serverpod versions
The version is 1.2.3 on all sides.
Platform information
Their is no difference with iOS or android they both have this issue.
I am hosting the server elsewhere at www.transip.nl.

Hey @Tristannos,
Let's first try and find out why userAuth.createAccountRequest(...) might return false. I'm assuming that the method is called from on the client.
There are a couple of reasons the method returns false.

There already exists a users with the provided email (the data.name! parameter that you provide).
The second parameter where you provide data.name! is not a valid email.
The password provided does not satisfy the configured min and max password length.
The user name is empty after trim() is called, that is the first parameter provided.

Could you validate that these are not true for the users that are experiencing issues?
Some additional questions:

Not really an error and their is a record in the database?

In what table can you observe a record for the user in the database?
If the trace that you have provided in the first image is the only one connected with the endpoint call I would suspect we get stuck on one of the steps mentioned. If this was not the case we would have seen an additional database lookup to EmailCreateAccountRequest.findFirstRow that is performed if all checks pass.

I'm experiencing the same behavior as well.
await _auth!.createAccountRequest(userName, email, password);

Returns false even it has successfully created a new record in the serverpod_email_create_request table. Every input entered was validated and there is nothing wrong on those variables.

Found the cause why it is not working on my end. I forgot to implement callbacks in the server.dart file in the my_pod_server directory.

The feedback in this process is lacking in many ways, we need to improve our error messaging here to give the developer more information about why account creation failed.


CSS Variables

Hallo, is there any timeline for CSS3 Variables getting supported any time soon? Or any guidance on where I could implement it my self and contribute it back to the repository? Thanks in advance.

Hello @JailbreakPapa,
CSS3 Variables are not planned soon. It doesn't seem to be a high priority at the current stage.
I could write instructions on how to create certain CSS modules into a project. But with CSS3 Variables you need to think how to implement them so that they work quickly.

I have a very heavy workload at my main job right now, and unfortunately I don't have time for a lot of things.
That will change soon, but for now, that's the way it is.

I have a very heavy workload at my main job right now, and unfortunately I don't have time for a lot of things.
That will change soon, but for now, that's the way it is.

Ok, I'll dig around in the codebase and see if I can help in that regard. I'll create a PR once I'm ready.


horizon/docs: Minor aesthetic, phrasing, and grammatical improvements

Mostly created so that we can use the PR UI to comment on places where we can make improvements.
Summarized list of changes / their rationale:

Most Markdown renderers prefer paragraphs to be all on one line (let your
editor do the wrapping) as opposed to with manual breaks. This can be
especially problematic with additional formatting e.g. [links broken over a newline](#), etc.
GitHub's renderer is solid, but better safe than sorry for portability.
Dropping whitespace changes from the diff should help w/ viewing.

Unifying usage of "Captive Core" or "Captive Stellar-Core" (consistent
capitalization, hyphenation, etc.).

Drop the example in the CAPTIVE_CORE_CONFIG_APPEND_PATH and link to the
below one, instead, to avoid them diverging or otherwise getting invalid.

Fixed a few comma splices and other punctuation pedantry

Prefer the systemctl method of managing Horizon

Specify syntax highlighting methods where possible (e.g. the Bash script)

Niiice!

I guess this is ready to merge? Ship it!!

I guess this is ready to merge? Ship it!!

I also guess we should retarget the PR to master or onto a new 2.0 (non-beta) release branch.

I also guess we should retarget the PR to master or onto a new 2.0 (non-beta) release branch.

@cla-bot check

@cla-bot check


fix: link in pdf export - print all slides into one page

This is currently a Work In Progress to try to solve the following issue https://github.com/slidevjs/slidev/issues/483
Currently this version adds a new route /print that displays all slides.
I created a new built-in Link component that uses anchor when we are in print mode.
When exporting to PDF, it uses the new route and the PDF is generated in one shot (no need to merge all pages together anymore).
Pros:

Links works inside the PDF
Big performance gain when generating the PDF (about 230 slides generated in 20 seconds against more than 11 minutes with the previous version)

Problems:

We need to rethink the nav logic because we can't rely on the global context to gives informations about currentPage...Etc.
Currently the currentPage is 1 when printing, but this causes issue with the <toc/> and can cause issue with global components used in global-top.vue or global-bottom.vue.
I propose we use provide/inject to give access to all those informations in child components.
This way we can inject the correct values for each slides when we print them all together.
We need to duplicate some slides if we want to keep the --with-clicks print feature.
It will need some work but I don't think this is an issue if we can use the previous proposal and send the click information with provide/inject.

@antfu What is your though about this ?
Thanks in advance for your answer.

So I made a new commit to switch from the $slidev global property to a provide / inject context.
This way I am able to provide a context for each slides while I am rendering them all in the /print page.
To avoid breaking anything, I created a loader to automatically inject this context into all slides and all vue components so you can still use $slidev without worrying.
I updated the <toc> component to use this context instead of importing from logic/nav.
For now I did not update other components, but if we want to have them compatible with the print mode we should update them as well.
I will now check the --with-clicks feature.

I added back the --with-clicks feature. :partying_face:
I also fixed some issue I add with the <Tweet> and <Youtube> components.
And I switch the PNG export to use the same process (using Locator.screenshot() from Playwright).
Now export time is around 20-30s for both format (for slides in demo/starter folder).
What still needs to be done:

probably add a --with-drawing CLI option to export the drawings (currently it is enabled by default)
maybe we should refactor components to use inject instead of importing from logic/nav? I only refactored the <Toc> because components like the <Editor> does not make sense in print mode, but maybe we should follow the same logic for every components?
I kept methods and computed values in logic/nav for retro-compatibility purpose but maybe we should remove them or mark them as deprecated?
What to do with the progress bar displayed on the CLI while exporting ? We can't use the same logic now as everything is printed in one shot (we can use it for PNG as I am iterating over the slide items but not for PDF, or in an other way...).

Do you have any opinion @antfu ?

Fixing merge conflicts

What to do with the progress bar displayed on the CLI while exporting ? We can't use the same logic now that everything is printed in one shot (we can use it for PNG as I am iterating over the slide items but not for PDF, or in an other way...).

Possible to use a generator or event callback to notify the progress?

Possible to use a generator or event callback to notify the progress?

The problem is more with the steps.
As I render all the pages I an't use the slide number as the steps anymore.
So what steps should we use ?
I can use fixed steps:

increment step after await page.goto()
increment step after await page.waitForLoadState()
increment step after await page.emulateMedia()
increment step after await page.locator('.tweet')
...etc.

Or I can update the progress with an indeterminate option that will only displays the {spin} and 'rendering' parts.

Or I can update the progress with an indeterminate option that will only displays the {spin} and 'rendering' parts.

Sounds good to me!

Sounds good to me!

Good, I pushed all changes.

I'm receiving updates on this so I take the opportunity to add a request/question: would there be a way to not print some slides? is it just a question of adding a custom class like no-print and some CSS or should it be embedded in the print logic?

I think you can use class: print:hidden in the Front Matter block.

Thanks for the suggestion.
I had a quick try (on slidev without this PR) and the slide is still printed in the pdf :(

I tested on this branch and it seems to work

Ok. I'll retry at some point. Sorry for polluting the discussion.

Awesome work! @tonai Thanks for the contributions you have made to Slidev, I have sent you the invitation to join the org as a part of the team! 🙌


ci/integration-tests: Name the container to notify it about cancellation

Name the integration-tests container to notify it about the cancellation
The CI is capturing the signal sent by GH when a cancellation occurs, but it is not able to forward it to the integration-tests container because it does not have the expected name:
CI is notifying the container, but it is not named as expected:
docker kill --signal="SIGINT" ig-integration-tests

This error comes from a wrong rebased of https://github.com/kinvolk/inspektor-gadget/pull/653 where this modification got lost somehow.
How to use
Force-pushing a commit on the same branch while running the integration tests or manually cancelling it on the Actions section of the repo.
Testing done
Before this PR
The cancellation request is captured, but it is not forwarded to the integration-tests container. Therefore, the ARO cluster is not cleaned up. Example here:
IntegrationTestsJob: Workflow run is being cancelled: SIGINT was received
IntegrationTestsJob: Start the clean-up...
IntegrationTestsJob: Notifying the integration tests container about the cancellation
Error response from daemon: Cannot kill container: ig-integration-tests: No such container: ig-integration-tests

It implies that the next execution of the integration-tests on the ARO cluster (which is always the same) will fail while trying to deploy IG and SPO because they are still there. Example here:
customresourcedefinition.apiextensions.k8s.io/traces.gadget.kinvolk.io configured
namespace/gadget unchanged
serviceaccount/gadget unchanged
role.rbac.authorization.k8s.io/gadget-role unchanged
rolebinding.rbac.authorization.k8s.io/gadget-role-binding unchanged
clusterrole.rbac.authorization.k8s.io/gadget-cluster-role unchanged
clusterrolebinding.rbac.authorization.k8s.io/gadget-cluster-role-binding unchanged
daemonset.apps/gadget configured
invalid command output(DeployInspektorGadget): output didn't match the expected regexp: gadget created

After this PR
The cancellation request is captured and forwarded to the integration-tests container, which starts with the cleanup process. Example here:
IntegrationTestsJob: Workflow run is being cancelled: SIGINT was received
IntegrationTestsJob: Start the clean-up...
IntegrationTestsJob: Notifying the integration tests container about the cancellation
Handling cancellation...
Stop init commands (if they are still running)...

Start command(DeleteRemainingTestNamespace): kubectl delete ns -l scope=ig-integration-tests
Start command(CleanupInspektorGadget): $KUBECTL_GADGET undeploy
Start command(RemoveSecurityProfilesOperator):
	kubectl delete seccompprofile -n security-profiles-operator --all
	kubectl delete -f https://raw.githubusercontent.com/kubernetes-sigs/security-profiles-operator/v0.4.2/deploy/operator.yaml
	kubectl delete -f https://github.com/jetstack/cert-manager/releases/download/v1.7.2/cert-manager.yaml

Approving as this was a "lost" change in a previous PR.

Yeah, I am sorry for the noise. After the rebased, I only tested https://github.com/kinvolk/inspektor-gadget/pull/653 locally, so I didn't realise the mistake.


chore: bdd-gh-1568061930 to 0.0.1

chore: Promote bdd-gh-1568061930 to version 0.0.1

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


i18n & sidebar

sidebar title with internationalization generate an error. (v. 0.3.2)

sorry, it's all ok ---> but works I18n.t() and not only t()


Saving and loading an array of strings changes datatype to object

What is your issue?
See the code below
import xarray as xr
# Make an xarray with an array of strings
da = xr.DataArray(
    data=[["a", "b"], ["c", "d"]],
    dims=["x", "y"],
    coords={"x": [0, 1], "y": [0, 1]},
)
da.to_netcdf("test.nc", mode='w')
# Load the xarray back in
da_loaded = xr.load_dataarray("test.nc")
assert da.dtype == da_loaded.dtype, "Dtypes don't match"

Now da_loaded.dtype is dtype('O').

Similar problem where saving, loading, saving, loading changes the dtype:
import xarray as xr
da = xr.DataArray(data=[True], dims=["x"], coords={"x": [0]})
da.to_netcdf("test.nc", engine="h5netcdf", mode="w")
da_loaded = xr.load_dataarray("test.nc", engine="h5netcdf")
da_loaded.to_netcdf("test.nc", engine="h5netcdf", mode="w")
da_loaded = xr.load_dataarray("test.nc", engine="h5netcdf")
assert da.dtype == da_loaded.dtype, "Dtypes don't match"

Another fun one where int64 -> int32:
import xarray as xr

da = xr.DataArray(data=[1], dims=["x"], coords={"x": [0]})
da.to_netcdf("test.nc", mode="w")
da2 = xr.load_dataarray("test.nc")
da.dtype, da2.dtype

A similar problem where saving, loading, saving, loading changes the dtype bool -> int8:

That's an issue with netcdf file format, too, it has no bool-dtype.
XRef: https://github.com/pydata/xarray/issues/1500
data = np.array([True], dtype=bool)
with nc.Dataset("test-bool-netcdf4.nc", mode="w") as ds:
    ds.createDimension("x", size=1)
    var = ds.createVariable("da", data.dtype.str,  dimensions=("x"))
    var[:] = data

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[42], line 4
      2 with nc.Dataset("test-bool-netcdf4.nc", mode="w") as ds:
      3     ds.createDimension("x", size=1)
----> 4     var = ds.createVariable("da", data.dtype.str,  dimensions=("x"))
      5     var[:] = data

File src/netCDF4/_netCDF4.pyx:2945, in netCDF4._netCDF4.Dataset.createVariable()

File src/netCDF4/_netCDF4.pyx:4121, in netCDF4._netCDF4.Variable.__init__()

TypeError: illegal primitive data type, must be one of dict_keys(['S1', 'i1', 'u1', 'i2', 'u2', 'i4', 'u4', 'i8', 'u8', 'f4', 'f8']), got bool

Another fun one where int64 -> int32:

Can't reproduce this one with my environment. See above for details.

A similar problem where saving, loading, saving, loading changes the dtype bool -> int8:

@basnijholt I'd appreciate if you could test #7654 for that particular case.

Thanks a lot @kmuehlbauer!
I replied in your PR :smile:

Great, much appreciated, thanks! Let's iterate over there then.

OK, I've finally gotten to the bottom of this, so I'm writing my findings here:
int64 -> int32
This works with h5netcdf/netcdf4-backends in any case. That's a feature^1 of NETCDF3-format which will be used if scipy is installed and netCDF4 is not installed. It has not notion of int64 so this is silently cast to int32 on write.
<U1 -> object
This works with the changes applied in #7654 for h5netcdf/netcdf4-backends in normal cases (writing out as VLEN string). Again, NETCDF3 format does not have a notion of string so all strings have to be converted to an internal NC_CHAR representation. I'm not sure it will do any good to try to make this one work.
My suggestion would be, just use NETCDF4-format and one of the capable backends.

@kmuehlbauer this is amazing!
It would be very valuable to convert this list of limitations to the documentation: https://docs.xarray.dev/en/stable/user-guide/io.html#netcdf

@kmuehlbauer, great!
I can confirm that both problems are indeed fixed on my end when using h5netcdf and the code from your PR :tada:

@kmuehlbauer this is amazing!
It would be very valuable to add this list of limitations to the documentation: https://docs.xarray.dev/en/stable/user-guide/io.html#netcdf

I've added a bit to this over at #7654.


The message Prowl received just show one line and the newline part is gone

I installed the regrowl version by snicker. Don’t know if it’s the problem of gntp or the problem of regrowl. The information received by Prowl can only display one line, and the line break part is gone. The newline part has been removed in the Notification-Text of debug info and only the rest was forwarded to Prowl. It is suspected that gntp has a problem with the processing of line breaks after receiving the information.

I have not really worked on this library in many years, especially since Growl itself has been retired, 1
However, if you want to investigate yourself, you need to keep in mind that the GNTP protocol uses \r\n for lines, so as long as your line uses \n for new line, I think it should probably work.

@kfdm Great thanks for your reply! I know Growl is a bit old, but I think it is still strong. 😹
I test it just now, but the part after the line break is still missing.
[00:00:00] INFO -- : Sending Growl notification 'test': 'abc
123'
And I received:

Notification Packet:
===================================
GNTP/1.0 NOTIFY NONE SHA512:xxx
Notification-Callback-Context-Type: type
Notification-ID: xxx
Application-Name: HuginnGrowl
Notification-Text: abc

[00:00:00] INFO -- : Sending Growl notification 'test': 'abc\n123'
And I received:

Notification Packet:
==================================
GNTP/1.0 NOTIFY NONE SHA512:xxx
Notification-Callback-Context-Type: type
Notification-ID: xxx
Application-Name: HuginnGrowl
Notification-Text: abc\n123

Is the prowl part something you wrote? I don't see anything prowl related in the regrowl library. Currently I don't have this library, regrowl, or growl installed, so it would be a little challenging for me to debug it without a more detailed explanation of your setup and how everything is connected together.

I didn't write anything. Wait for me to speak slowly. 😸
I send messages with Growl Agent of huginn. Huginn is a very popular project on Github. After deployed huginn, you can see the Growl Agent on a page something like this:

And I installed regrowl version by snicker on the same server. The regrowl related to Prowl was written by him. You mention it here in 2014. 😹
After deployed those above your Prowl can receive messages send by Growl Agent.

I'm sorry, I don't really have any of this setup to test, and have not used it myself in many years 🙇


Problems with reading mff files

Description of the problem
Hi,
I am trying to open data files that recorded with EGI system, with mne.io.read_raw_egi.
for some recordings it works perfectly fine and for others I get an error message.
MNE version: 1.1.1
tried also 1.2.dev0 version and the error persisted.
operating system: Windows 10
here is an example code and files. for sub3003 I can read the file and for sub3004 I get an error message.
Steps to reproduce
import mne

sub3003 = r"C:\Users\paltiele.WISMAIN\Dropbox (Weizmann Institute)\PhD\Learning_faces\EEG experiment\data\EEG\EL3003\EL3003_SLEEP_20220221_015208.mff"
raw3003 = mne.io.read_raw_egi(sub3003)

sub3004 = r"C:\Users\paltiele.WISMAIN\Dropbox (Weizmann Institute)\PhD\Learning_faces\EEG experiment\data\EEG\EL3004\EL3004_SLEEP_20220309_125216.mff"
raw3004 = mne.io.read_raw_egi(sub3004)

Link to data
subject 3003:
https://www.dropbox.com/sh/8qt0nslqdjsh1ik/AAB3I6eZBvvUS3rncILJbVlra?dl=0
subject 3004:
(https://www.dropbox.com/sh/y6q5txmcteubf8u/AAD3AY7vu4MgIzrrW3krWvqHa?dl=0
Expected results
read the mff file smoothly
Actual results
Error message:
'''
sub3004 = r"C:\Users\paltiele.WISMAIN\Dropbox (Weizmann Institute)\PhD\Learning_faces\EEG experiment\data\EEG\EL3004\EL3004_SLEEP_20220309_125216.mff"
raw = mne.io.read_raw_egi(sub3004)
Reading EGI MFF Header from C:\Users\paltiele.WISMAIN\Dropbox (Weizmann Institute)\PhD\Learning_faces\EEG experiment\data\EEG\EL3004\EL3004_SLEEP_20220309_125216.mff...
Traceback (most recent call last):
File "C:\Users\paltiele.WISMAIN\AppData\Local\Temp\ipykernel_12524\2134038754.py", line 2, in
raw = mne.io.read_raw_egi(sub3004)
File "", line 12, in read_raw_egi
File "C:\Users\paltiele.WISMAIN\Anaconda3\envs\for_visbrain\lib\site-packages\mne\io\egi\egi.py", line 157, in read_raw_egi
exclude, preload, channel_naming, verbose)
File "", line 12, in _read_raw_egi_mff
File "C:\Users\paltiele.WISMAIN\Anaconda3\envs\for_visbrain\lib\site-packages\mne\io\egi\egimff.py", line 390, in _read_raw_egi_mff
preload, channel_naming, verbose)
File "", line 12, in init
File "C:\Users\paltiele.WISMAIN\Anaconda3\envs\for_visbrain\lib\site-packages\mne\io\egi\egimff.py", line 404, in init
egi_info = _read_header(input_fname)
File "C:\Users\paltiele.WISMAIN\Anaconda3\envs\for_visbrain\lib\site-packages\mne\io\egi\egimff.py", line 212, in _read_header
mff_hdr = _read_mff_header(input_fname)
File "C:\Users\paltiele.WISMAIN\Anaconda3\envs\for_visbrain\lib\site-packages\mne\io\egi\egimff.py", line 93, in _read_mff_header
list(epochs['last_samps'])))
RuntimeError: EGI epoch first/last samps could not be parsed:
[0]
[26586364]
'''
Additional information
Platform:         Windows-10-10.0.19041-SP0
Python:           3.7.13 (default, Mar 28 2022, 08:03:21) [MSC v.1916 64 bit (AMD64)]
Executable:       C:\Users\paltiele.WISMAIN\Anaconda3\envs\for_visbrain\python.exe
CPU:              Intel64 Family 6 Model 140 Stepping 1, GenuineIntel: 8 cores
Memory:           31.7 GB
mne:              1.2.dev0
numpy:            1.21.6 {MKL 2022.1-Product with 4 threads}
scipy:            1.7.3
matplotlib:       3.5.3 {backend=module://matplotlib_inline.backend_inline}
sklearn:          1.0.2
numba:            0.56.2
nibabel:          Not found
nilearn:          Not found
dipy:             Not found
openmeeg:         Not found
cupy:             Not found
pandas:           1.3.5
pyvista:          Not found
pyvistaqt:        Not found
ipyvtklink:       Not found
vtk:              Not found
qtpy:             Not found
ipympl:           Not found
pyqtgraph:        Not found
pooch:            v1.6.0
mne_bids:         Not found
mne_nirs:         Not found
mne_features:     Not found
mne_qt_browser:   Not found
mne_connectivity: Not found
mne_icalabel:     Not found

Hi @elba1212 - can you zip the .mff files before you upload them to dropbox? Do you have any smaller files that you can share and which also cause the issue? The files you shared appear to be so large that dropbox won't even let me download them.

@elba1212 I also just wanted to highlight that the patient video stream is included in your file. If you are not allowed to disclose patient information you can remove the .mov files inside the .mff directory (which may also slightly decrease your overall file size).
Thanks!

One of those .bin files is 26.5GB for example -- I'm guessing this as a 32-bit int problem. I should be able to check this week hopefully

Hi @scott-huberty and @larsoner,
Thank you for your reply!
I tried replicating the error with some of my smaller files, but it only replicates with my large files.
(when I try to read the smaller files for some of them, I get a different error: IndexError when it reads the event channel).
I removed the videos, zipped the mff files, and updated the links in the original message as you suggested.
Thanks,
Ella

@scott-huberty @larsoner there is an off-by-one error in subject 3004:
ipdb> p n_samps_epochs
26586364
ipdb> p n_samps_block
26586365

I can't access subject 3003, Dropbox says "zip files are supported but something went wrong"

We have an off-by-one-bug check elsewhere:
https://github.com/mne-tools/mne-python/blob/706aea1eb723801c43ae5fc5f68bbb7b53184c29/mne/io/egi/egimff.py#L559-L563
I wonder if we need to introduce one here, too.

I corrected the "3003" link; it works now.
Anyway, this is the file that I can successfully read.
If you need more examples of problematic files, tell me.


Upgrade completions to WordPress 4.2.2

Hi,
I created completions for WP 4.2.2 deleting deprecateds functions, etc.. and separated in different files for actions, filters, constants and classes and append completions for capabilities

@kallookoo Why did you close this?

Because the author does not seem to want to keep updating it and I decided to make the package WordPress Completions


build oci image for every oCIS service

Description

Related Issue

Fixes <issue_link>

Motivation and Context

How Has This Been Tested?

test environment:
test case 1:
test case 2:
...

Screenshots (if appropriate):
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Technical debt
[ ] Tests only (no source changes)

Checklist:

[ ] Code changes
[ ] Unit tests added
[ ] Acceptance tests added
[ ] Documentation ticket raised:

This PR comes from a fork.
@micbar is CI still disabled for PRs from forks?


Grocy and Picnic API

Hi Mike,
Nice to see your work on the picnic api. Just tried it and works awesome! I want to integrate my picnic orders in Home Assistant with the addon Grocy.
My plan is as follows:

Barcode scanner
Keep track of own foodstock in the house
Every time I use a product, lower stock in Grocy.

As an alternative it could say to my Google Home assistant:

Hey Google "I'm cutting an onion " -> Trigger lower stock in Grocy

But it would be nice to use a barcode with an IP camera in my pantry.
Unfortanly I can't find the barcodes in from picnic products in the app. I tried searching my kippendijfilet but the kip wasn't found :)
Do you know if there is a way to connect the barcode to a product? Searching for the product ID in the app is working fine..
(Example is from barcode tomatos in plastic..)

Hi koenieee,
Glad you are enjoying the repo. I started this project with the exact same idea, sadly life required me to spend my time elsewhere and put this project on hold.
Picnic does not provide any way to use the barcode, so a self made dataset with barcodes and product ids would probably the only solution. I tried some tinkering with EAN-API services (this one for example) and use the output from those sites to query the products, but it proved to be unreliable at best.
I'd love to join and help you continuing this idea, please keep me posted on any updates!

Hi koenieee,
Glad you are enjoying the repo. I started this project with the exact same idea, sadly life required me to spend my time elsewhere and put this project on hold.
Picnic does not provide any way to use the barcode, so a self made dataset with barcodes and product ids would probably the only solution. I tried some tinkering with EAN-API services (this one for example) and use the output from those sites to query the products, but it proved to be unreliable at best.
I'd love to join and help you continuing this idea, please keep me posted on any updates!

Hi Mike,
Thanks for pointing out. Indeed I have the same idea. I am going to use the AH api/Jumbo api (Supermarktconnector python) to find the barcode with the product name and match the name in my picnic order list using fuzzy text matching.
I will start a repository with some code and scripts for homeassistant. Just bought a barcode scanner to have some scanning fun haha! I also had to idea to match a picture of an union using opencv with a image of picnic product item. So you can hold something in front of a webcam to find out what it is.. Maybe my Google Home assistant can tell me what product it is (using TTS and HomeAssistant).
This will be a long term project, but well I have some time at the moment :)

Hi Mike,
Thanks for pointing out. Indeed I have the same idea. I am going to use the AH api/Jumbo api (Supermarktconnector python) to find the barcode with the product name and match the name in my picnic order list using fuzzy text matching.
I will start a repository with some code and scripts for homeassistant. Just bought a barcode scanner to have some scanning fun haha! I also had to idea to match a picture of an union using opencv with a image of picnic product item. So you can hold something in front of a webcam to find out what it is.. Maybe my Google Home assistant can tell me what product it is (using TTS and HomeAssistant).
This will be a long term project, but well I have some time at the moment :)

Hi! Just wanting to know I'm too thinkering with this idea. I made a PHP-client several months ago. But found out that urls arent working anymore. Thanks to this repo I could find the right url again (and User-Agent... ;) )!
Back to the idea.. I wasted some hours creating a barcode scanner myself in Android based on the Google Vision barcode scanner. I now have a app which scans the EAN and sends it back to a backend. This is basically a POC.  If the mobile version works I'm going to embed a barcode scanner in my kitchen, maybe using a Arduino. Now just where you guys were, I'm at this point where I wanted to match the EAN with the Picnic products. I saw several services which provides a EAN-database but they are all paid. So thats a no-go. I did found out that the site of Coop uses the ean in the url... So maybe that's a entry point. The only way you could match it is by name then and that's not going to be fail proof. I can imagine that I'm going to map the products by myself since my basket is basically the same every week.
It's very funy but also very scarry that I had the same ideas as @koenieee  last week... Maybe we can join forces and create something out of this?

Hi! Just wanting to know I'm too thinkering with this idea. I made a PHP-client several months ago. But found out that urls arent working anymore. Thanks to this repo I could find the right url again (and User-Agent... ;) )!
Back to the idea.. I wasted some hours creating a barcode scanner myself in Android based on the Google Vision barcode scanner. I now have a app which scans the EAN and sends it back to a backend. This is basically a POC.  If the mobile version works I'm going to embed a barcode scanner in my kitchen, maybe using a Arduino. Now just where you guys were, I'm at this point where I wanted to match the EAN with the Picnic products. I saw several services which provides a EAN-database but they are all paid. So thats a no-go. I did found out that the site of Coop uses the ean in the url... So maybe that's a entry point. The only way you could match it is by name then and that's not going to be fail proof. I can imagine that I'm going to map the products by myself since my basket is basically the same every week.
It's very funy but also very scarry that I had the same ideas as @koenieee  last week... Maybe we can join forces and create something out of this?

Hi Mike and Fan1200,
I just started a repository to import picnic deliveries in grocy. With all the product information, such as weight and how much. I have two friends who are also interested in this project. Good idea to join forces. Indeed fan1200, I think we acutally have to set the barcodes ourself. Maybe barcodebuddy can help with that. Would also be great to share recipes and the product database!
https://github.com/koenieee/Grocy-Picnic
The code is't nice yet, normally I program in C++ and I'm learning python with this project

Hi Mike and Fan1200,
I just started a repository to import picnic deliveries in grocy. With all the product information, such as weight and how much. I have two friends who are also interested in this project. Good idea to join forces. Indeed fan1200, I think we acutally have to set the barcodes ourself. Maybe barcodebuddy can help with that. Would also be great to share recipes and the product database!
https://github.com/koenieee/Grocy-Picnic
The code is't nice yet, normally I program in C++ and I'm learning python with this project

I just stubled upon this repo. First of all: kudo's that it already exists. I was already making preperations to make this my first py + github thing ever (I'm a PowerShell guy, so py should be doable). I'm exactly thinking the same way. I'm not using Grocy yet, but if Picnic integration is available, then it would be awesome.
As far as barcodes: GS1 is an international company who has a lot of barcodes is their system. I'm not sure if there is a free api available though.
If there is anything I can do regarding testing or whatever, I'm very happy to help!

I just stubled upon this repo. First of all: kudo's that it already exists. I was already making preperations to make this my first py + github thing ever (I'm a PowerShell guy, so py should be doable). I'm exactly thinking the same way. I'm not using Grocy yet, but if Picnic integration is available, then it would be awesome.
As far as barcodes: GS1 is an international company who has a lot of barcodes is their system. I'm not sure if there is a free api available though.
If there is anything I can do regarding testing or whatever, I'm very happy to help!

Ok, I think I've a project for this weekend :)
I just looked at GS1 as @marti28 suggested but that only brings up information about the issuer/company behind the EAN. The only information you can use to match the product is by matching them with the company information provided on the PDP. So that's little to no use.
After another try to figure out how to identifying the product based on what Picnic gives it leaves with me with only one option and that is matching by name. Given an EAN I could try to render it through the website of Coop. It uses the EAN in the url, so that makes it easy to find something. If not available I could to try to use other sources. For example I found https://nl.openfoodfacts.org/. It seems to be a open source project to link any food related information to a product. Such as the ingredients. You could enter a EAN and it could you give more product information. The database seems to be completly free of access as you could download a nightly export. You could even use the API for a single search. With the information given on openfoodfacts you could match it against the name, 'ingredients_blob' and 'nutritional_values'.
It's not failproof but I think that even if you try to match by name it could result into a really nice dataset from which you could kickstart. Anyways, tomorrow I'm going to try this. I'll keep you posted!
@koenieee is there a link available for barcodebuddy? I'm willing to enter my share of codes and products ;)

Ok, I think I've a project for this weekend :)
I just looked at GS1 as @marti28 suggested but that only brings up information about the issuer/company behind the EAN. The only information you can use to match the product is by matching them with the company information provided on the PDP. So that's little to no use.
After another try to figure out how to identifying the product based on what Picnic gives it leaves with me with only one option and that is matching by name. Given an EAN I could try to render it through the website of Coop. It uses the EAN in the url, so that makes it easy to find something. If not available I could to try to use other sources. For example I found https://nl.openfoodfacts.org/. It seems to be a open source project to link any food related information to a product. Such as the ingredients. You could enter a EAN and it could you give more product information. The database seems to be completly free of access as you could download a nightly export. You could even use the API for a single search. With the information given on openfoodfacts you could match it against the name, 'ingredients_blob' and 'nutritional_values'.
It's not failproof but I think that even if you try to match by name it could result into a really nice dataset from which you could kickstart. Anyways, tomorrow I'm going to try this. I'll keep you posted!
@koenieee is there a link available for barcodebuddy? I'm willing to enter my share of codes and products ;)

Hello @fan1200 and @marti28.
Thanks for wanting to help with this project.
I can add you and @MikeBrink to my repository so we can work together on this. Next week i have some extra time to continue the product import for picnic.
Barcodebuddy can be found here
Have a look at: https://github.com/Forceu/barcodebuddy

Hello @fan1200 and @marti28.
Thanks for wanting to help with this project.
I can add you and @MikeBrink to my repository so we can work together on this. Next week i have some extra time to continue the product import for picnic.
Barcodebuddy can be found here
Have a look at: https://github.com/Forceu/barcodebuddy

So, after some hours wasted. I managed to create a "system" that looks up eans and finds product names. Pretty much like Barcodebuddy does. It only takes 2 sources: Coop and OpenFoodFacts. From the 26 random products I scanned in my kitchen and bathroom it found 17 names. The products that could't been found were either not Googleable or found on some website I can't really scrape.
With the names found it was time to match it up against the product database from Picnic. That became harder than I thought. An simple solution you could think of was using the "like" function in Mysql. Well, that was not really helpful (as expected). Then I knew I could use the PHP-function "similar_text". The function could match two strings and gives back a percentage of how similar one is to another. The function seems to work fine on paper but I'm not going use it since I have to check all products against the available ean results. Since "similar_text" is a levenshtein distance method I found a Mysql procedure function that I could use. That would give me a option to match the given product name against the Picnic ones. It worked! I got 10 out of 16 results right. But it's painfully slow. Each calculation took around a minute to process. So I trashed that idea as well.
Now still with those thoughts of matching product names I only had one option left. Using the Elasticsearch. The major benefit of this method is that you have more control on searching thru several aspects and within a matter of a few seconds.  After indexing the products (name and description) I could make some queries. I used a multi match query and a boosted one. Both seem to have it's advantages but still could bring up some false-positives results. For example: "Zuivelmeester Yoghurt Griekse stijl 10% vet" would give me "Optimel Griekse stijl yoghurt kers 0% vet" as a multi match result and "Bio+ yoghurt Griekse stijl 10%" as a boosted result. While "Palmolive Vloeibare zeep amandel" gives back "Palmolive handzeep melk & amandel" as a multi match result. And that's the right one!
So after all those tries. I still came to the conclusion that you can't automate this process with enough satisfaction. It could give some good results but most of the time it's not reliable. I still want to proceed with these ideas but I'm going to wrap it in a GUI. It would find some results. Take the top 5 products. Show them and I've to choose which product it is. Doing it in a web UI could give me some joy thru all the CLI-darkness. And it could be a nice project to learn VueJS.
As for BarcodeBuddy and Grocery... I can't really find any use of it. The Grocery app does too much and the UI looks outdated. It's a shame that BarcodeBuddy doesn't stand on its own and relies on Grocery. Otherwise it could have been great for this purpose. And the fact that you have run locally or host it yourself doesn't help us either to complete our goal ;)
For now I added my results: product names, eans and product ids, in a json file stored within the repo of @koenieee https://github.com/koenieee/Grocy-Picnic maybe we can create a webservice where we can put our results per request and pulling it out with a simple request?

So, after some hours wasted. I managed to create a "system" that looks up eans and finds product names. Pretty much like Barcodebuddy does. It only takes 2 sources: Coop and OpenFoodFacts. From the 26 random products I scanned in my kitchen and bathroom it found 17 names. The products that could't been found were either not Googleable or found on some website I can't really scrape.
With the names found it was time to match it up against the product database from Picnic. That became harder than I thought. An simple solution you could think of was using the "like" function in Mysql. Well, that was not really helpful (as expected). Then I knew I could use the PHP-function "similar_text". The function could match two strings and gives back a percentage of how similar one is to another. The function seems to work fine on paper but I'm not going use it since I have to check all products against the available ean results. Since "similar_text" is a levenshtein distance method I found a Mysql procedure function that I could use. That would give me a option to match the given product name against the Picnic ones. It worked! I got 10 out of 16 results right. But it's painfully slow. Each calculation took around a minute to process. So I trashed that idea as well.
Now still with those thoughts of matching product names I only had one option left. Using the Elasticsearch. The major benefit of this method is that you have more control on searching thru several aspects and within a matter of a few seconds.  After indexing the products (name and description) I could make some queries. I used a multi match query and a boosted one. Both seem to have it's advantages but still could bring up some false-positives results. For example: "Zuivelmeester Yoghurt Griekse stijl 10% vet" would give me "Optimel Griekse stijl yoghurt kers 0% vet" as a multi match result and "Bio+ yoghurt Griekse stijl 10%" as a boosted result. While "Palmolive Vloeibare zeep amandel" gives back "Palmolive handzeep melk & amandel" as a multi match result. And that's the right one!
So after all those tries. I still came to the conclusion that you can't automate this process with enough satisfaction. It could give some good results but most of the time it's not reliable. I still want to proceed with these ideas but I'm going to wrap it in a GUI. It would find some results. Take the top 5 products. Show them and I've to choose which product it is. Doing it in a web UI could give me some joy thru all the CLI-darkness. And it could be a nice project to learn VueJS.
As for BarcodeBuddy and Grocery... I can't really find any use of it. The Grocery app does too much and the UI looks outdated. It's a shame that BarcodeBuddy doesn't stand on its own and relies on Grocery. Otherwise it could have been great for this purpose. And the fact that you have run locally or host it yourself doesn't help us either to complete our goal ;)
For now I added my results: product names, eans and product ids, in a json file stored within the repo of @koenieee https://github.com/koenieee/Grocy-Picnic maybe we can create a webservice where we can put our results per request and pulling it out with a simple request?

Wow I love the progress guys! Sadly I don't have much time to contribute to this effort yet, I am currently wrapping up my masters degree.
However, I do have an idea for a possible approach:

I need to extend this API wrapper to make it possible to traverse the menus (e.g. Fruit > Banaan, appel & meer).
Write a python script to scrape all products (also makes it possible to log price changes)
Use the Coop website that @fan1200 mentioned and try to literal match on name (I tried some and it mostly works)
In case that there isn't an exact match: try to match all the ingredients that are both available at Coop and Picnic

There is also a huge interest for the retrieval of images (getting multiple emails a month asking for this), but I did not intent this wrapper for that purpose initially (also picnic might not be thrilled with us scraping huge amounts of image data). However, it might be interesting for a "product recognition AI"? If so I can dedicate some time in two weeks maybe.
I was hoping that some of the people emailing me about the images would pick up this endeavor, but it doesn't seem so. It also takes some time for me to reverse engineer the API.

Wow I love the progress guys! Sadly I don't have much time to contribute to this effort yet, I am currently wrapping up my masters degree.
However, I do have an idea for a possible approach:

I need to extend this API wrapper to make it possible to traverse the menus (e.g. Fruit > Banaan, appel & meer).
Write a python script to scrape all products (also makes it possible to log price changes)
Use the Coop website that @fan1200 mentioned and try to literal match on name (I tried some and it mostly works)
In case that there isn't an exact match: try to match all the ingredients that are both available at Coop and Picnic

There is also a huge interest for the retrieval of images (getting multiple emails a month asking for this), but I did not intent this wrapper for that purpose initially (also picnic might not be thrilled with us scraping huge amounts of image data). However, it might be interesting for a "product recognition AI"? If so I can dedicate some time in two weeks maybe.
I was hoping that some of the people emailing me about the images would pick up this endeavor, but it doesn't seem so. It also takes some time for me to reverse engineer the API.

@MikeBrink I could save you time by sharing my code. I already made some scripts to scrape products and categories. The only thing is that it was written PHP. But I think Picnic is not going to be so thrilled about this. It's like 7000 products 😅. And Coop can be easier, they also have a app and thus an API that you could use.

@MikeBrink I could save you time by sharing my code. I already made some scripts to scrape products and categories. The only thing is that it was written PHP. But I think Picnic is not going to be so thrilled about this. It's like 7000 products 😅. And Coop can be easier, they also have a app and thus an API that you could use.

@fan1200 thanks for the first product list, I added the a python function inside the picnic import script to search for EAN connected to the PicnicID. So whenever someone adds an item to the list it will be automatically found.
I made my import script a little more user friendly. But it's working quite nice already, @MikeBrink pictures are also imported in Grocy from picnic. You can just append the img_id url to a base url and get the picture.

@fan1200 thanks for the first product list, I added the a python function inside the picnic import script to search for EAN connected to the PicnicID. So whenever someone adds an item to the list it will be automatically found.
I made my import script a little more user friendly. But it's working quite nice already, @MikeBrink pictures are also imported in Grocy from picnic. You can just append the img_id url to a base url and get the picture.

@fan1200 @MikeBrink @marti28 I updated my repository with some new code, would be nice if someone could test it and give me feedback. I tried to make it a little bit more user friendly, also with installing and setting up the grocy database :)
It should work with adding the products, importing and increasing stock.
We still have to scan more products to get some barcodes...
Now I'm going to try to see if scraping a recipe website and match them with grocy products is possible...

@fan1200 @MikeBrink @marti28 I updated my repository with some new code, would be nice if someone could test it and give me feedback. I tried to make it a little bit more user friendly, also with installing and setting up the grocy database :)
It should work with adding the products, importing and increasing stock.
We still have to scan more products to get some barcodes...
Now I'm going to try to see if scraping a recipe website and match them with grocy products is possible...

@koenieee I'm afraid I can't test your implementation since I don't use Python but it seems fine. I made some progress of making a "Barcode Searcher"-package. Since I need to use it for my personal use I thought it would be nice to share it with the rest of the world. It is written PHP and kind of relies on the Laravel Framework which I mostly use. The available providers are: Albert Heijn, CodeChecker, Coop, Jumbo, OpenFoodFact, UpcDatabase and UpcItemDb. I took some inspiration out of all the available code that was already Github. And added some of my own.
Tomorrow I'll will update the ean list with codes and product ids ;)
It's still in development but the code is here: https://github.com/fan1200/barcode_seacher/tree/develop

@koenieee I'm afraid I can't test your implementation since I don't use Python but it seems fine. I made some progress of making a "Barcode Searcher"-package. Since I need to use it for my personal use I thought it would be nice to share it with the rest of the world. It is written PHP and kind of relies on the Laravel Framework which I mostly use. The available providers are: Albert Heijn, CodeChecker, Coop, Jumbo, OpenFoodFact, UpcDatabase and UpcItemDb. I took some inspiration out of all the available code that was already Github. And added some of my own.
Tomorrow I'll will update the ean list with codes and product ids ;)
It's still in development but the code is here: https://github.com/fan1200/barcode_seacher/tree/develop

@koenieee I updated the product list with 37 new products.

I'm not sure if I'm of any help to you guys, but I created a very messy but working Python script containing a hunting mode 🥚 This is the way I connect ean codes to picnic products :) I am in no way a professionall Python developer, so don't expect to much.

I'm not sure if I'm of any help to you guys, but I created a very messy but working Python script containing a hunting mode  This is the way I connect ean codes to picnic products :) I am in no way a professionall Python developer, so don't expect to much.

https://github.com/larsverp/picnic-easy-inventory

So, after a year I'm going to bump this thread into 2022 with some news. By accident I noticed that a product package that I recently bought had a QR-code. It was a Picnic Branded product. So when scanning the QR I got greeted with a URL that included the EAN and it would have redirected me to the PDP of the app.
The url is: https://picnic.app/qr/gtin/8720326001681 it redirects to https://picnic.app/nl/deeplink/?path=store/storefront/product-detail;id=11455430 which would give us the product id. It seems like they finally integrated some support for EAN usage. It also works on non-picnic branded products.
So I'm just leaving this message here. Do what you like with it. I kind of abandoned the project due to lack of time and constraints. I may continue some kind of project  since it now should be easy to integrate EAN based operations into some home automation without the hassle of using third party searching or keeping up an database.

If there are any Go users that want to quickly get a picnic ID from an eancode, I created a very small but nice package. You can find it at https://pkg.go.dev/github.com/larsverp/go-eantopicnic
A big thanks to @fan1200 for sharing this information! ❤

To make the work of others easier:
import json
from python_picnic_api import PicnicAPI

p = PicnicAPI("XXX","XXX",'de');

def getProductIdFromEAN(p, EAN):
    response = p.session.get("https://picnic.app/qr/gtin/" + EAN)
    url = response.url
    if ";id=" in url:
        return url.split(";id=",1)[1]
    else:
        return None

id = getProductIdFromEAN(p, "4037400434315")
print(id)
if (id):
    print(json.dumps(p.get_article(id), indent=4))

The issue might be there are a few more redirects. I see the following redirects in the chrome debugger:

This library should follow the Location header till there is something like product-detail;id=.
If that's too complex, you could change the base url to https://gtin-resolver-prod.nl.picnicinternational.com/qr/gtin/[EAN]/.
But picnic could change the underlying infra, so it's best to keep using https://picnic.app/qr/gtin/ because that's what they put in the QR's and should be supported.


Call eval with file and line nb so seeds fail with proper stracktraces

Otherwise the stacktrace starts with .../gems/seed-fu-2.3.6/lib/seed-fu/runner.rb:46:ineval':` which is far from helpful.
This also adds a basic test of the runner with a file containing # BREAK EVAL

@mbleigh please consider this. There is a very old blog post from Ola Bini (JRuby core) explaining the concept: http://ola-bini.blogspot.nl/2008/01/ruby-antipattern-using-eval-without.html

There has been a commit in over a year, this project looks abandoned :sob:

@aserafin This is a very simple PR, any chance of it getting merged?


docs: Add react-native-blurhash

Adds the React Native UI component implementation

cc @jerry-git


Distance and Direction Rings tab: upon enter, Center Point should appear and zoom

Currently, when navigating in the rings tab and adding a coordinate pair to the Center Point box, I expect to click enter and see the map zoom to this location and see a circular preview. However, I can only see a center point when I click away from the map, and I can't get the zoom to work at all.
To reproduce:
navigate to rings tab
enter specific coordinate pair as center point (33, 6 is in Algeria) and click enter. You'll see nothing happen.
Click on a subsequent box and you'll see the center point appear.

@Dbarnes1 @kgonzago
Dan I have fixed this issue and made sure it is consistent across all 4 tabs (Lines,Circle,Ellipse,Rings). On hitting enter the point is added and the map pans to center on that coord  (I haven't put a zoom on this as the user could find it irritating if the map zooms in, then they have to zoom back out to perform next action).

this has been fixed, except for:

Not sure why (UNKNOWN) is appearing in the UI. Assigning back to @adgiles to take a look. @kgonzago fyi.

@Dbarnes1
Issue fixed please can you verify

@adgiles verified:

closing


User Agent incorrectly set by Client

When looking at requests created by the java client in Hello Sign the User Agent is not properly resolved and shows like hellosign-java-sdk/${project.version}

Looks like when this was created the maven build filtered over the /src/main/resources directory
https://github.com/hellosign/hellosign-java-sdk/blob/4a72ae2d3566d612ef3e6a1d21009b28aca2c242/pom.xml#L57
This must not have been carried over when the build was converted to gradle

@jspaetzel I'm not a gradle guy so this is probably the extent of what I can do.  I did a PR but I can't tell if the test suite is actually being run by github actions

Thanks for the PR @codylerum! Next time I or someone on my team makes changes to this SDK I'll be sure to verify this is working the way we're expecting and merge it in.


java.lang.NoClassDefFoundError thrown sometimes when returning to idle desktop game window

hi,
actually I'm not sure if it's a libgdx issue or a kotlin issue (since the game core is implemented in kotlin) but I occasionally get the mentioned error when I trigger any ingame ClickListener of a non-focused desktop window.

https://github.com/libgdx/libgdx/blob/master/CONTRIBUTING.md#bug-reports
At least a complete stack trace would be nice, this is like reporting "it is broken, sometimes" :)

yeah sorry, I read the guidelines before, but I thought I will rather get a response like "no it's not libgdx related", why I skipped some information.
one stack trace here:
com.badlogic.gdx.utils.GdxRuntimeException: java.lang.NoClassDefFoundError: de/lmgt/quiz/view/screens/DuelStateScreen$refreshLayout$2
	at com.badlogic.gdx.backends.lwjgl.LwjglApplication$1.run(LwjglApplication.java:127)
Caused by: java.lang.NoClassDefFoundError: de/lmgt/quiz/view/screens/DuelStateScreen$refreshLayout$2
	at de.lmgt.quiz.view.screens.DuelStateScreen.refreshLayout(DuelStateScreen.kt:100)
	at de.lmgt.quiz.view.screens.DuelStateScreen.show(DuelStateScreen.kt:51)
	at com.badlogic.gdx.Game.setScreen(Game.java:61)
	at de.lmgt.quiz.LmgtQuiz.setScreen(LmgtQuiz.kt:219)
	at de.lmgt.quiz.services.DuelModeManager.showDuelStateOf(DuelModeManager.kt:398)
	at de.lmgt.quiz.view.screens.DuelModeScreen$refreshLayout$2$4.invoke(DuelModeScreen.kt:279)
	at de.lmgt.quiz.view.screens.DuelModeScreen$refreshLayout$2$4.invoke(DuelModeScreen.kt:34)
	at de.lmgt.quiz.misc.MiscPackage$SoundedClick$becdf8c1$addSoundedClickListener$1.invoke(SoundedClick.kt:27)
	at de.lmgt.quiz.misc.MiscPackage$SoundedClick$becdf8c1$addSoundedClickListener$1.invoke(SoundedClick.kt)
	at de.lmgt.quiz.misc.MiscPackage$SoundedClick$becdf8c1$soundedClick$1.clicked(SoundedClick.kt:17)
	at de.lmgt.quiz.misc.SoundedClick.clicked(SoundedClick.kt:41)
	at com.badlogic.gdx.scenes.scene2d.utils.ClickListener.touchUp(ClickListener.java:89)
	at com.badlogic.gdx.scenes.scene2d.InputListener.handle(InputListener.java:58)
	at com.badlogic.gdx.scenes.scene2d.Stage.touchUp(Stage.java:348)
	at com.badlogic.gdx.InputMultiplexer.touchUp(InputMultiplexer.java:96)
	at com.badlogic.gdx.backends.lwjgl.LwjglInput.processEvents(LwjglInput.java:306)
	at com.badlogic.gdx.backends.lwjgl.LwjglApplication.mainLoop(LwjglApplication.java:207)
	at com.badlogic.gdx.backends.lwjgl.LwjglApplication$1.run(LwjglApplication.java:120)
Caused by: java.lang.ClassNotFoundException: de.lmgt.quiz.view.screens.DuelStateScreen$refreshLayout$2
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 18 more

Please use the forum or irc if you need help debugging your code. If you are actually reporting an issue with libgdx, then at least include the information required to reproduce the issue. https://github.com/libgdx/libgdx/wiki/Getting-Help

well, it's not about debugging. I just reported this error, since I faced it several times. The huge problem is that it's happening occasionally. I have no idea what might be the reason. I will provide more information if I'm able to. I've already checked the forums and issues, nothing similar found. The only thing I haven't tried yet are the nightly builds (currently using 1.6.4).

There is pretty much no possible way this could be due to libgdx.

ok thanks for the response, I think I should report this in the kotlin world.


IP150-MQTTv2.py modifications to python v3

Hi all.
I am trying to make the necessary modifications for this topic to run the script in python3.
Any ideas, since I am stuck with some errors in IP150-MQTTv2.py & the client.py?
ParadoxIP150v2-master.zip
Or any other ideas how to run the whole process in Python2?

I’m still running in 2.7 side by side with 3.9, keen to see if there are updates in this thread!

I’m still running in 2.7 side by side with 3.9, keen to see if there are updates in this thread!

is there any way that I run the whole process from 2.7?
when I open python s/w from raspberry, I have only option to compile with v3.

You should be able to use apt-get to install 2.7, then you can update alternatives if I recall if you want to run it commandline. I've got it running as a service and specify 2.7 in there:
`[Unit]
Description=Starts and stops the Paradox IP150 Python script
Documentation=https://github.com/Tertiush/ParadoxIP150v2
Wants=network-online.target
After=network-online.target
[Service]
EnvironmentFile=/opt/paradoxip150v2/config.ini
User=homeassistant
Group=homeassistant
WorkingDirectory=/opt/paradoxip150v2/
PermissionsStartOnly=true
ExecStart=/usr/bin/python2.7 /opt/paradoxip150v2/IP150-MQTTv2.py
Shutdown delay in seconds, before process is tried to be killed with KILL (if configured)
TimeoutStopSec=20
[Install]
WantedBy=multi-user.target`


Optionally Prepopulate IP Address when adding Firewall Exception

It would be a productivity boost to prepopulate one's public IP when adding a firewall exception. We could make a call to a free service like https://icanhazip.com and use that to prepopulate the input or add as a suggestion below it. This could be exposed under the parameters as optional along with the option to configure the URL if the user prefers a different service.
If you're open to this idea, I'd be happy to do a PR on it.

Hi, welcome to the repo and thanks for the suggestion. Sure, sounds like fun, let's do it 👍

@milnet92 Thanks for the pr (the 100th btw. 🎉 ), looking good, this is now available in version 0.52.0
@tnoor-kr Let us know if this works for you.

Thank you for getting to this sooner than I could have. I just tried it out and it populated the IP instantly.